highperfocused/ollama
1
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2025-11-11 10:57:36 +01:00
Code Issues Packages Projects Releases Wiki Activity
4,543 Commits 269 Branches 329 Tags
pdevine/parser-tidy
Commit Graph

2 Commits

Author SHA1 Message Date
Jeffrey Morgan
e093db92c4 sample: temporarily use grammars for constrained generation in new engine (#9586) 2025-03-10 16:17:39 +01:00
Parth Sareen
0682dae027 sample: improve ollama engine sampler performance (#9374) 
This change bring in various interface cleanups along with greatly improving the performance of the sampler.

Tested with llama3.2 on local machine.
Improves performance from ~ 70 tokens/s -> 135 tokens/s with topK(40) enabled.
Without topK performance is ~ 110 tokens/s
 2025-03-07 12:37:48 -08:00