devin-ai-integration[bot] b82123563b

Fix Unicode sanitization for Vespa document indexing (#3831) ...

* Add support for filtering 0xFDD0-0xFDEF Unicode range

- Update remove_invalid_unicode_chars to handle 0xFDD0-0xFDEF range
- Add comprehensive test cases for Unicode character sanitization
- Fix issue with illegal code point 0xFDDB in Vespa indexing

Co-Authored-By: Chris Weaver <chris@onyx.app>

* Remove unused pytest import

Co-Authored-By: Chris Weaver <chris@onyx.app>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Weaver <chris@onyx.app>

2025-01-29 18:32:00 +00:00

..

test_utils.py

Fix Unicode sanitization for Vespa document indexing (#3831)

2025-01-29 18:32:00 +00:00