highperfocused/ollama
1
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2025-04-08 03:48:21 +02:00
Code Issues Packages Projects Releases Wiki Activity

patch: use default locale in wpm tokenizer (#3034)

Browse Source
This commit is contained in:
Jeffrey Morgan 2024-03-09 21:12:12 -08:00 committed by GitHub
parent cdf65e793f
commit 908005d90b
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 13 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
llm/patches/03-locale.diff Normal file
View File

@ -0,0 +1,13 @@
diff --git a/llama.cpp b/llama.cpp
index b19616e8..519b9602 100644
--- a/llama.cpp
+++ b/llama.cpp
@@ -9938,7 +9938,7 @@ struct llm_tokenizer_wpm {
}
uint32_t to_lower(uint32_t code) {
- static const std::locale locale("en_US.UTF-8");
+ static const std::locale locale("");
#if defined(_WIN32)
if (code > 0xFFFF) {
return code;