4 Commits

Author	SHA1	Message	Date
Michael Yang	333e360422	model: handle multiple eos tokens (#10577) ... * get eos_token_id from generation_config.json * refactor * include both ids and strings in trace * comments * remove special case for gemma3 special vocab (#10743)	2025-05-16 13:40:23 -07:00
Michael Yang	4456012956	fix unmarshaling merges	2024-12-04 09:21:56 -08:00
Michael Yang	60e47573a6	more tokenizer tests	2024-08-27 14:51:10 -07:00
Michael Yang	3eb08377f8	detect chat template from configs that contain lists	2024-08-27 10:49:33 -07:00