Contextual Retrieval (#4029)

* contextual rag implementation

* WIP

* indexing test fix

* workaround for chunking errors, WIP on fixing massive memory cost

* mypy and test fixes

* reformatting

* fixed rebase
This commit is contained in:
evan-danswer
2025-03-30 11:49:09 -07:00
committed by GitHub
parent cb5bbd3812
commit 56f8ab927b
42 changed files with 1007 additions and 68 deletions

View File

@@ -87,6 +87,9 @@ def _create_indexable_chunks(
metadata_suffix_keyword="",
mini_chunk_texts=None,
large_chunk_reference_ids=[],
doc_summary="",
chunk_context="",
contextual_rag_reserved_tokens=0,
embeddings=ChunkEmbedding(
full_embedding=preprocessed_doc["content_embedding"],
mini_chunk_embeddings=[],