highperfocused/ollama
1
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2025-09-27 20:27:05 +02:00
Code Issues Packages Projects Releases Wiki Activity

ggml: Disable flash attention for gemma2

Browse Source 
Our new engine implementation of gemma2 doesn't support flash
attention, which means that it also doesn't support KV cache
quantization. Currently, it is possible to turn these two on,
which will result in a crash.
This commit is contained in:
Jesse Gross
2025-09-09 10:48:34 -07:00
committed by Jesse Gross
parent 71cb86af3e
commit 29ddfc2cab
1 changed files with 4 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
fs/ggml/ggml.go
View File

@@ -883,6 +883,10 @@ func (f GGML) SupportsFlashAttention() bool {
return false
}
if arch := f.KV().Architecture(); slices.Contains([]string{"gemma2"}, arch) {
return false
}
// Check head counts match and are non-zero
headCountK := f.KV().EmbeddingHeadCountK()
headCountV := f.KV().EmbeddingHeadCountV()