jmorganca b42970063d kvcache: Add check for values that fall out of sliding window cache ...

The sliding window cache trims entries that are outside the window for
the latest token. This works when we are extending the cache, such as
when the conversation continues. However, if we have a partial overlap
in conversation (including the BOS tokens), then we resume from a past
point in the conversation and the needed tokens are no longer stored
in memory. This verifies that the new window overlaps with the old one
before reusing the cache.

Co-authored-by: Jesse Gross <jesse@ollama.com>

2025-04-02 11:55:48 -07:00

..

cache.go

kvcache: Add check for values that fall out of sliding window cache

2025-04-02 11:55:48 -07:00

causal_test.go

kvcache: Add check for values that fall out of sliding window cache

2025-04-02 11:55:48 -07:00

causal.go

kvcache: Add check for values that fall out of sliding window cache

2025-04-02 11:55:48 -07:00

encoder.go

kvcache: Add check for values that fall out of sliding window cache

2025-04-02 11:55:48 -07:00

wrapper.go

kvcache: Add check for values that fall out of sliding window cache

2025-04-02 11:55:48 -07:00