Jesse Gross e119783e66 llm: Clamp batch size to context size ...

The context must always be able to store the current batch, so
if the user requests a small context then we should also shrink
the batch to match. This also fixes the TestLongInputContext
test on the new engine. (The old engine already has this behavior.)

2025-09-08 20:40:11 -07:00

..

llm_darwin.go

…

llm_linux.go

…

llm_windows.go

…

memory_test.go

…

memory.go

…

server_test.go

…

server.go

llm: Clamp batch size to context size

2025-09-08 20:40:11 -07:00

status.go

…