highperfocused/danswer
1
0
Fork 0
mirror of https://github.com/danswer-ai/danswer.git synced 2025-09-27 20:38:32 +02:00
Code Issues Packages Projects Releases Wiki Activity

Updating some Prompts (#3894)

Browse Source
This commit is contained in:
Yuhong Sun
2025-02-04 12:23:15 -08:00
committed by GitHub
parent 91f0650071
commit 59b1d4673f
2 changed files with 510 additions and 413 deletions
Download Patch File Download Diff File Expand all files Collapse all files

918
backend/onyx/prompts/agent_search.py
View File

@@ -53,8 +53,8 @@ Answer:
# Step/Utility Prompts # Step/Utility Prompts
# Note this one should always be used with the ENTITY_TERM_EXTRACTION_PROMPT_JSON_EXAMPLE # Note this one should always be used with the ENTITY_TERM_EXTRACTION_PROMPT_JSON_EXAMPLE
ENTITY_TERM_EXTRACTION_PROMPT = f""" ENTITY_TERM_EXTRACTION_PROMPT = f"""
Based on the original question and some context retrieved from a dataset, please generate a list of Based on the original question and some context retrieved from a dataset, please generate a list of \
entities (e.g. companies, organizations, industries, products, locations, etc.), terms and concepts entities (e.g. companies, organizations, industries, products, locations, etc.), terms and concepts \
(e.g. sales, revenue, etc.) that are relevant for the question, plus their relations to each other. (e.g. sales, revenue, etc.) that are relevant for the question, plus their relations to each other.
Here is the original question: Here is the original question:
@@ -98,445 +98,541 @@ ENTITY_TERM_EXTRACTION_PROMPT_JSON_EXAMPLE = """
""".strip() """.strip()
HISTORY_CONTEXT_SUMMARY_PROMPT = ( HISTORY_CONTEXT_SUMMARY_PROMPT = f"""
"{persona_specification}\n\n" {{persona_specification}}
"Your task now is to summarize the key parts of the history of a conversation between a user and an agent."
" The summary has two purposes:\n" Your task now is to summarize the key parts of the history of a conversation between a user and an agent. \
" 1) providing the suitable context for a new question, and\n" The summary has two purposes:
" 2) To capture the key information that was discussed and that the user may have a follow-up question about.\n\n" 1) providing the suitable context for a new question, and
"Here is the question:\n" 2) To capture the key information that was discussed and that the user may have a follow-up question about.
f"{SEPARATOR_LINE}\n"
"{question}\n" Here is the question:
f"{SEPARATOR_LINE}\n\n" {SEPARATOR_LINE}
"And here is the history:\n" {{question}}
f"{SEPARATOR_LINE}\n" {SEPARATOR_LINE}
"{history}\n"
f"{SEPARATOR_LINE}\n\n" And here is the history:
"Please provide a summarized context from the history so that the question makes sense and can" {SEPARATOR_LINE}
" - with suitable extra information - be answered.\n\n" {{history}}
"Do not use more than three or four sentences.\n\n" {SEPARATOR_LINE}
"History summary:"
).strip() Please provide a summarized context from the history so that the question makes sense and can \
- with suitable extra information - be answered.
Do not use more than three or four sentences.
History summary:
""".strip()
# INITIAL PHASE # INITIAL PHASE
# Sub-question # Sub-question
# Intentionally left a copy in case we want to modify this one differently # Intentionally left a copy in case we want to modify this one differently
INITIAL_QUESTION_DECOMPOSITION_PROMPT = ( INITIAL_QUESTION_DECOMPOSITION_PROMPT = f"""
"Decompose the initial user question into no more than 3 appropriate sub-questions that help to answer the" Decompose the initial user question into no more than 3 appropriate sub-questions that help to answer the \
" original question. The purpose for this decomposition may be to:\n" original question. The purpose for this decomposition may be to:
" 1) isolate individual entities (i.e., 'compare sales of company A and company B' ->" 1) isolate individual entities (i.e., 'compare sales of company A and company B' -> \
" ['what are sales for company A', 'what are sales for company B'])\n" ['what are sales for company A', 'what are sales for company B'])
" 2) clarify or disambiguate ambiguous terms (i.e., 'what is our success with company A' ->" 2) clarify or disambiguate ambiguous terms (i.e., 'what is our success with company A' -> \
" ['what are our sales with company A','what is our market share with company A'," ['what are our sales with company A','what is our market share with company A', \
" 'is company A a reference customer for us', etc.])\n" 'is company A a reference customer for us', etc.])
" 3) if a term or a metric is essentially clear, but it could relate to various components of an entity and you" 3) if a term or a metric is essentially clear, but it could relate to various components of an entity and you \
" are generally familiar with the entity, then you can decompose the question into sub-questions that are more" are generally familiar with the entity, then you can decompose the question into sub-questions that are more \
" specific to components (i.e., 'what do we do to improve scalability of product X', 'what do we to to improve" specific to components (i.e., 'what do we do to improve scalability of product X', 'what do we to to improve \
" scalability of product X', 'what do we do to improve stability of product X', ...])\n" scalability of product X', 'what do we do to improve stability of product X', ...])
" 4) research an area that could really help to answer the question.\n\n" 4) research an area that could really help to answer the question.
"Here is the initial question to decompose:\n"
f"{SEPARATOR_LINE}\n" Here is the initial question to decompose:
"{question}\n" {SEPARATOR_LINE}
f"{SEPARATOR_LINE}\n\n" {{question}}
"{history}\n\n" {SEPARATOR_LINE}
"Do NOT include any text in your answer outside of the list of sub-questions!"
"Please formulate your answer as a newline-separated list of questions like so:\n" {{history}}
" <sub-question>\n"
" <sub-question>\n" Do NOT include any text in your answer outside of the list of sub-questions!
" <sub-question>\n" Please formulate your answer as a newline-separated list of questions like so:
" ...\n\n" <sub-question>
"Answer:" <sub-question>
).strip() <sub-question>
...
Answer:
""".strip()
# TODO: combine shared pieces with INITIAL_QUESTION_DECOMPOSITION_PROMPT # TODO: combine shared pieces with INITIAL_QUESTION_DECOMPOSITION_PROMPT
INITIAL_DECOMPOSITION_PROMPT_QUESTIONS_AFTER_SEARCH = ( INITIAL_DECOMPOSITION_PROMPT_QUESTIONS_AFTER_SEARCH = f"""
"Decompose the initial user question into no more than 3 appropriate sub-questions that help to answer the" Decompose the initial user question into no more than 3 appropriate sub-questions that help to answer the \
" original question. The purpose for this decomposition may be to:\n" original question. The purpose for this decomposition may be to:
" 1) isolate individual entities (i.e., 'compare sales of company A and company B' ->" 1) isolate individual entities (i.e., 'compare sales of company A and company B' -> \
" ['what are sales for company A', 'what are sales for company B'])\n" ['what are sales for company A', 'what are sales for company B'])
" 2) clarify or disambiguate ambiguous terms (i.e., 'what is our success with company A' ->" 2) clarify or disambiguate ambiguous terms (i.e., 'what is our success with company A' -> \
" ['what are our sales with company A','what is our market share with company A'," ['what are our sales with company A','what is our market share with company A', \
" 'is company A a reference customer for us', etc.])\n" 'is company A a reference customer for us', etc.])
" 3) if a term or a metric is essentially clear, but it could relate to various components of an entity and you" 3) if a term or a metric is essentially clear, but it could relate to various components of an entity and you \
" are generally familiar with the entity, then you can decompose the question into sub-questions that are more" are generally familiar with the entity, then you can decompose the question into sub-questions that are more \
" specific to components (i.e., 'what do we do to improve scalability of product X', 'what do we to to improve" specific to components (i.e., 'what do we do to improve scalability of product X', 'what do we to to improve \
" scalability of product X', 'what do we do to improve stability of product X', ...])\n" scalability of product X', 'what do we do to improve stability of product X', ...])
" 4) research an area that could really help to answer the question.\n\n" 4) research an area that could really help to answer the question.
"To give you some context, you will see below also some documents that may relate to the question. Please only"
" use this information to learn what the question is approximately asking about, but do not focus on the details" To give you some context, you will see below also some documents that may relate to the question. Please only \
" to construct the sub-questions! Also, some of the entities, relationships and terms that are in the dataset may" use this information to learn what the question is approximately asking about, but do not focus on the details \
" not be in these few documents, so DO NOT focussed too much on the documents when constructing the sub-questions!" to construct the sub-questions! Also, some of the entities, relationships and terms that are in the dataset may \
" Decomposition and disambiguations are most important!\n\n" not be in these few documents, so DO NOT focussed too much on the documents when constructing the sub-questions! \
"Here are the sample docs to give you some context:\n" Decomposition and disambiguations are most important!
f"{SEPARATOR_LINE}\n"
"{sample_doc_str}\n" Here are the sample docs to give you some context:
f"{SEPARATOR_LINE}\n\n" {SEPARATOR_LINE}
"And here is the initial question to decompose:\n" {{sample_doc_str}}
f"{SEPARATOR_LINE}\n" {SEPARATOR_LINE}
"{question}\n"
f"{SEPARATOR_LINE}\n\n" And here is the initial question to decompose:
"{history}\n\n" {SEPARATOR_LINE}
"Do NOT include any text in your answer outside of the list of sub-questions!" {{question}}
"Please formulate your answer as a newline-separated list of questions like so:\n" {SEPARATOR_LINE}
" <sub-question>\n"
" <sub-question>\n" {{history}}
" <sub-question>\n"
" ...\n\n" Do NOT include any text in your answer outside of the list of sub-questions!\
"Answer:" Please formulate your answer as a newline-separated list of questions like so:
).strip() <sub-question>
<sub-question>
<sub-question>
...
Answer:
""".strip()
# Retrieval # Retrieval
QUERY_REWRITING_PROMPT = ( QUERY_REWRITING_PROMPT = f"""
"Please convert the initial user question into a 2-3 more appropriate short and pointed search queries for" Please convert the initial user question into a 2-3 more appropriate short and pointed search queries for \
" retrieval from a document store. Particularly, try to think about resolving ambiguities and make the search" retrieval from a document store. Particularly, try to think about resolving ambiguities and make the search \
" queries more specific, enabling the system to search more broadly.\n" queries more specific, enabling the system to search more broadly.
"Also, try to make the search queries not redundant, i.e. not too similar!\n\n"
"Here is the initial question:\n" Also, try to make the search queries not redundant, i.e. not too similar!
f"{SEPARATOR_LINE}\n"
"{question}\n" Here is the initial question:
f"{SEPARATOR_LINE}\n\n" {SEPARATOR_LINE}
"Do NOT include any text in your answer outside of the list of queries!" {{question}}
"Formulate the queries separated by newlines (Do not say 'Query 1: ...', just write the querytext) as follows:\n" {SEPARATOR_LINE}
"<query 1>\n"
"<query 2>\n" Do NOT include any text in your answer outside of the list of queries!\
"...\n\n" Formulate the queries separated by newlines (Do not say 'Query 1: ...', just write the querytext) as follows:
"Queries:" <query 1>
) <query 2>
...
Queries:
""".strip()
DOCUMENT_VERIFICATION_PROMPT = ( DOCUMENT_VERIFICATION_PROMPT = f"""
"Determine whether the following document text contains data or information that is potentially relevant " Determine whether the following document text contains data or information that is potentially relevant \
"for a question. It does not have to be fully relevant, but check whether it has some information that " for a question. It does not have to be fully relevant, but check whether it has some information that \
"would help - possibly in conjunction with other documents - to address the question.\n\n" would help - possibly in conjunction with other documents - to address the question.
"Be careful that you do not use a document where you are not sure whether the text applies to the objects "
"or entities that are relevant for the question. For example, a book about chess could have long passage " Be careful that you do not use a document where you are not sure whether the text applies to the objects \
"discussing the psychology of chess without - within the passage - mentioning chess. If now a question " or entities that are relevant for the question. For example, a book about chess could have long passage \
"is asked about the psychology of football, one could be tempted to use the document as it does discuss " discussing the psychology of chess without - within the passage - mentioning chess. If now a question \
"psychology in sports. However, it is NOT about football and should not be deemed relevant. Please " is asked about the psychology of football, one could be tempted to use the document as it does discuss \
"consider this logic.\n\n" psychology in sports. However, it is NOT about football and should not be deemed relevant. Please \
"DOCUMENT TEXT:\n" consider this logic.
f"{SEPARATOR_LINE}\n"
"{document_content}\n" DOCUMENT TEXT:
f"{SEPARATOR_LINE}\n\n" {SEPARATOR_LINE}
"Do you think that this document text is useful and relevant to answer the following question?\n\n" {{document_content}}
"QUESTION:\n" {SEPARATOR_LINE}
f"{SEPARATOR_LINE}\n"
"{question}\n" Do you think that this document text is useful and relevant to answer the following question?
f"{SEPARATOR_LINE}\n\n"
"Please answer with exactly and only a 'yes' or 'no'. Do NOT include any other text in your response:\n\n" QUESTION:
"Answer:" {SEPARATOR_LINE}
).strip() {{question}}
{SEPARATOR_LINE}
Please answer with exactly and only a '{YES}' or '{NO}'. Do NOT include any other text in your response:
Answer:
""".strip()
# Sub-Question Anser Generation # Sub-Question Anser Generation
SUB_QUESTION_RAG_PROMPT = ( SUB_QUESTION_RAG_PROMPT = f"""
"Use the context provided below - and only the provided context - to answer the given question. " Use the context provided below - and only the provided context - to answer the given question. \
"(Note that the answer is in service of answering a broader question, given below as 'motivation'.)\n\n" (Note that the answer is in service of answering a broader question, given below as 'motivation').
"Again, only use the provided context and do not use your internal knowledge! If you cannot answer the "
f'question based on the context, say "{UNKNOWN_ANSWER}". It is a matter of life and death that you do NOT ' Again, only use the provided context and do not use your internal knowledge! If you cannot answer the \
"use your internal knowledge, just the provided information!\n\n" question based on the context, say "{UNKNOWN_ANSWER}". It is a matter of life and death that you do NOT \
"Make sure that you keep all relevant information, specifically as it concerns to the ultimate goal. " use your internal knowledge, just the provided information!
"(But keep other details as well.)\n\n"
"It is critical that you provide inline citations in the format [D1], [D2], [D3], etc! " Make sure that you keep all relevant information, specifically as it concerns to the ultimate goal. \
"It is important that the citation is close to the information it supports. " (But keep other details as well.)
"Proper citations are very important to the user!\n\n"
"For your general information, here is the ultimate motivation:\n" It is critical that you provide inline citations in the format [D1], [D2], [D3], etc! \
f"{SEPARATOR_LINE}\n" It is important that the citation is close to the information it supports. \
"{original_question}\n" Proper citations are very important to the user!
f"{SEPARATOR_LINE}\n\n"
"And here is the actual question I want you to answer based on the context above (with the motivation in mind):\n" For your general information, here is the ultimate motivation:
f"{SEPARATOR_LINE}\n" {SEPARATOR_LINE}
"{question}\n" {{original_question}}
f"{SEPARATOR_LINE}\n\n" {SEPARATOR_LINE}
"Here is the context:\n"
f"{SEPARATOR_LINE}\n" And here is the actual question I want you to answer based on the context above (with the motivation in mind):
"{context}\n" {SEPARATOR_LINE}
f"{SEPARATOR_LINE}\n\n" {{question}}
"Please keep your answer brief and concise, and focus on facts and data.\n\n" {SEPARATOR_LINE}
"Answer:"
).strip() Here is the context:
{SEPARATOR_LINE}
{{context}}
{SEPARATOR_LINE}
Please keep your answer brief and concise, and focus on facts and data.
Answer:
""".strip()
SUB_ANSWER_CHECK_PROMPT = ( SUB_ANSWER_CHECK_PROMPT = f"""
"Determine whether the given answer addresses the given question. " Determine whether the given answer addresses the given question. \
"Please do not use any internal knowledge you may have - just focus on whether the answer " Please do not use any internal knowledge you may have - just focus on whether the answer \
"as given seems to largely address the question as given, or at least addresses part of the question.\n\n" as given seems to largely address the question as given, or at least addresses part of the question.
"Here is the question:\n"
f"{SEPARATOR_LINE}\n" Here is the question:
"{question}\n" {SEPARATOR_LINE}
f"{SEPARATOR_LINE}\n\n" {{question}}
"Here is the suggested answer:\n" {SEPARATOR_LINE}
f"{SEPARATOR_LINE}\n"
"{base_answer}\n" Here is the suggested answer:
f"{SEPARATOR_LINE}\n\n" {SEPARATOR_LINE}
f'Does the suggested answer address the question? Please answer with "{YES}" or "{NO}".' {{base_answer}}
).strip() {SEPARATOR_LINE}
Does the suggested answer address the question? Please answer with "{YES}" or "{NO}".
""".strip()
# Initial Answer Generation # Initial Answer Generation
INITIAL_ANSWER_PROMPT_W_SUB_QUESTIONS = ( INITIAL_ANSWER_PROMPT_W_SUB_QUESTIONS = f"""
"{persona_specification}\n\n" {{persona_specification}}
"Use the information provided below - and only the provided information - to answer the provided main question.\n\n"
"The information provided below consists of:\n" Use the information provided below - and only the provided information - to answer the provided main question.
" 1) a number of answered sub-questions - these are very important to help you organize your thoughts and your answer\n"
" 2) a number of documents that deemed relevant for the question.\n\n" The information provided below consists of:
"{history}\n\n" 1) a number of answered sub-questions - these are very important to help you organize your thoughts and your answer
"It is critical that you provide prover inline citations to documents in the format [D1], [D2], [D3], etc.!\n" 2) a number of documents that deemed relevant for the question.
"It is important that the citation is close to the information it supports. If you have multiple citations that support\n"
"a fact, please cite for example as [D1][D3], or [D2][D4], etc.\n" {{history}}
"Feel free to also cite sub-questions in addition to documents, but make sure that you have documents cited with the "
"sub-question citation. If you want to cite both a document and a sub-question, please use [D1][Q3], or " It is critical that you provide prover inline citations to documents in the format [D1], [D2], [D3], etc.! \
"[D2][D7][Q4], etc.\n" It is important that the citation is close to the information it supports. If you have multiple citations that support \
"Again, please NEVER cite sub-questions without a document citation! " a fact, please cite for example as [D1][D3], or [D2][D4], etc. \
"Proper citations are very important for the user!\n\n" Feel free to also cite sub-questions in addition to documents, but make sure that you have documents cited with the \
"IMPORTANT RULES:\n" sub-question citation. If you want to cite both a document and a sub-question, please use [D1][Q3], or [D2][D7][Q4], etc. \
" - If you cannot reliably answer the question solely using the provided information, say that you cannot reliably answer.\n" Again, please NEVER cite sub-questions without a document citation! Proper citations are very important for the user!
" You may give some additional facts you learned, but do not try to invent an answer.\n"
f' - If the information is empty or irrelevant, just say "{UNKNOWN_ANSWER}".\n' IMPORTANT RULES:
" - If the information is relevant but not fully conclusive, specify that the information is not conclusive and say why.\n\n" - If you cannot reliably answer the question solely using the provided information, say that you cannot reliably answer. \
"Again, you should be sure that the answer is supported by the information provided!\n\n" You may give some additional facts you learned, but do not try to invent an answer.
"Try to keep your answer concise. But also highlight uncertainties you may have should there be substantial ones,\n" - If the information is empty or irrelevant, just say "{UNKNOWN_ANSWER}".
"or assumptions you made.\n\n" - If the information is relevant but not fully conclusive, specify that the information is not conclusive and say why.
"Here is the contextual information:\n"
f"{SEPARATOR_LINE_LONG}\n\n" Again, you should be sure that the answer is supported by the information provided!
"*Answered Sub-questions (these should really matter!):\n"
f"{SEPARATOR_LINE}\n" Try to keep your answer concise. But also highlight uncertainties you may have should there be substantial ones, \
"{answered_sub_questions}\n" or assumptions you made.
f"{SEPARATOR_LINE}\n\n"
"And here are relevant document information that support the sub-question answers, " Here is the contextual information:
"or that are relevant for the actual question:\n" {SEPARATOR_LINE_LONG}
f"{SEPARATOR_LINE}\n"
"{relevant_docs}\n" *Answered Sub-questions (these should really matter!):
f"{SEPARATOR_LINE}\n\n" {SEPARATOR_LINE}
"And here is the question I want you to answer based on the information above:\n" {{answered_sub_questions}}
f"{SEPARATOR_LINE}\n" {SEPARATOR_LINE}
"{question}\n"
f"{SEPARATOR_LINE}\n\n" And here are relevant document information that support the sub-question answers, or that are relevant for the actual question:
"Please keep your answer brief and concise, and focus on facts and data.\n\n" {SEPARATOR_LINE}
"Answer:" {{relevant_docs}}
).strip() {SEPARATOR_LINE}
And here is the question I want you to answer based on the information above:
{SEPARATOR_LINE}
{{question}}
{SEPARATOR_LINE}
Please keep your answer brief and concise, and focus on facts and data.
Answer:
""".strip()
# Used if sub_question_answer_str is empty # Used if sub_question_answer_str is empty
INITIAL_ANSWER_PROMPT_WO_SUB_QUESTIONS = ( INITIAL_ANSWER_PROMPT_WO_SUB_QUESTIONS = f"""
"{answered_sub_questions}{persona_specification}\n\n" {{answered_sub_questions}}{{persona_specification}}
"Use the information provided below - and only the provided information - to answer the provided question. "
"The information provided below consists of a number of documents that were deemed relevant for the question.\n" Use the information provided below - and only the provided information - to answer the provided question. \
"{history}\n\n" The information provided below consists of a number of documents that were deemed relevant for the question.
"IMPORTANT RULES:\n"
" - If you cannot reliably answer the question solely using the provided information, say that you cannot reliably answer. " {{history}}
"You may give some additional facts you learned, but do not try to invent an answer.\n"
f' - If the information is irrelevant, just say "{UNKNOWN_ANSWER}".\n' IMPORTANT RULES:
" - If the information is relevant but not fully conclusive, specify that the information is not conclusive and say why.\n\n" - If you cannot reliably answer the question solely using the provided information, say that you cannot reliably answer. \
"Again, you should be sure that the answer is supported by the information provided!\n\n" You may give some additional facts you learned, but do not try to invent an answer.
"It is critical that you provide proper inline citations to documents in the format [D1], [D2], [D3], etc! " - If the information is irrelevant, just say "{UNKNOWN_ANSWER}".
"It is important that the citation is close to the information it supports. If you have multiple citations, " - If the information is relevant but not fully conclusive, specify that the information is not conclusive and say why.
"please cite for example as [D1][D3], or [D2][D4], etc. Citations are very important for the user!\n\n"
"Here is the relevant context information:\n" Again, you should be sure that the answer is supported by the information provided!
f"{SEPARATOR_LINE}\n"
"{relevant_docs}\n" It is critical that you provide proper inline citations to documents in the format [D1], [D2], [D3], etc! \
f"{SEPARATOR_LINE}\n\n" It is important that the citation is close to the information it supports. \
"And here is the question I want you to answer based on the context above:\n" If you have multiple citations, please cite for example as [D1][D3], or [D2][D4], etc. \
f"{SEPARATOR_LINE}\n" Citations are very important for the user!
"{question}\n"
f"{SEPARATOR_LINE}\n\n" Here is the relevant context information:
"Please keep your answer brief and concise, and focus on facts and data.\n\n" {SEPARATOR_LINE}
"Answer:" {{relevant_docs}}
).strip() {SEPARATOR_LINE}
And here is the question I want you to answer based on the context above:
{SEPARATOR_LINE}
{{question}}
{SEPARATOR_LINE}
Please keep your answer brief and concise, and focus on facts and data.
Answer:
""".strip()
# REFINEMENT PHASE # REFINEMENT PHASE
REFINEMENT_QUESTION_DECOMPOSITION_PROMPT = ( REFINEMENT_QUESTION_DECOMPOSITION_PROMPT = f"""
"An initial user question needs to be answered. An initial answer has been provided but it wasn't quite " An initial user question needs to be answered. An initial answer has been provided but it wasn't quite good enough. \
"good enough. Also, some sub-questions had been answered and this information has been used to provide " Also, some sub-questions had been answered and this information has been used to provide the initial answer. \
"the initial answer. Some other subquestions may have been suggested based on little knowledge, but they " Some other subquestions may have been suggested based on little knowledge, but they were not directly answerable. \
"were not directly answerable. Also, some entities, relationships and terms are given to you so that " Also, some entities, relationships and terms are given to you so that you have an idea of how the available data looks like.
"you have an idea of how the available data looks like.\n\n"
"Your role is to generate 2-4 new sub-questions that would help to answer the initial question, considering:\n\n" Your role is to generate 2-4 new sub-questions that would help to answer the initial question, considering:
"1) The initial question\n"
"2) The initial answer that was found to be unsatisfactory\n" 1) The initial question
"3) The sub-questions that were answered\n" 2) The initial answer that was found to be unsatisfactory
"4) The sub-questions that were suggested but not answered\n" 3) The sub-questions that were answered
"5) The entities, relationships and terms that were extracted from the context\n\n" 4) The sub-questions that were suggested but not answered
"The individual questions should be answerable by a good RAG system. " 5) The entities, relationships and terms that were extracted from the context
"So a good idea would be to use the sub-questions to resolve ambiguities and/or to separate the "
"question for different entities that may be involved in the original question, but in a way that does " The individual questions should be answerable by a good RAG system. So a good idea would be to use the sub-questions to \
"not duplicate questions that were already tried.\n\n" resolve ambiguities and/or to separate the question for different entities that may be involved in the original question, \
"Additional Guidelines:\n" but in a way that does not duplicate questions that were already tried.
"- The sub-questions should be specific to the question and provide richer context for the question, "
"resolve ambiguities, or address shortcoming of the initial answer\n" Additional Guidelines:
"- Each sub-question - when answered - should be relevant for the answer to the original question\n" - The sub-questions should be specific to the question and provide richer context for the question, resolve ambiguities, \
"- The sub-questions should be free from comparisons, ambiguities,judgements, aggregations, or any " or address shortcoming of the initial answer
"other complications that may require extra context.\n" - Each sub-question - when answered - should be relevant for the answer to the original question
"- The sub-questions MUST have the full context of the original question so that it can be executed by " - The sub-questions should be free from comparisons, ambiguities,judgements, aggregations, or any other complications that \
"a RAG system independently without the original question available\n" may require extra context
" (Example:\n" - The sub-questions MUST have the full context of the original question so that it can be executed by a RAG system \
' - initial question: "What is the capital of France?"\n' independently without the original question available
' - bad sub-question: "What is the name of the river there?"\n' Example:
' - good sub-question: "What is the name of the river that flows through Paris?")\n' - initial question: "What is the capital of France?"
"- For each sub-question, please also provide a search term that can be used to retrieve relevant " - bad sub-question: "What is the name of the river there?"
"documents from a document store.\n" - good sub-question: "What is the name of the river that flows through Paris?"
"- Consider specifically the sub-questions that were suggested but not answered. This is a sign that they are not " - For each sub-question, please also provide a search term that can be used to retrieve relevant documents from a document store.
"answerable with the available context, and you should not ask similar questions.\n\n" - Consider specifically the sub-questions that were suggested but not answered. This is a sign that they are not answerable \
"Here is the initial question:\n" with the available context, and you should not ask similar questions.
f"{SEPARATOR_LINE}\n"
"{question}\n" Here is the initial question:
f"{SEPARATOR_LINE}\n" {SEPARATOR_LINE}
"{history}\n\n" {{question}}
"Here is the initial sub-optimal answer:\n" {SEPARATOR_LINE}
f"{SEPARATOR_LINE}\n" {{history}}
"{base_answer}\n"
f"{SEPARATOR_LINE}\n\n" Here is the initial sub-optimal answer:
"Here are the sub-questions that were answered:\n" {SEPARATOR_LINE}
f"{SEPARATOR_LINE}\n" {{base_answer}}
"{answered_sub_questions}\n" {SEPARATOR_LINE}
f"{SEPARATOR_LINE}\n\n"
"Here are the sub-questions that were suggested but not answered:\n" Here are the sub-questions that were answered:
f"{SEPARATOR_LINE}\n" {SEPARATOR_LINE}
"{failed_sub_questions}\n" {{answered_sub_questions}}
f"{SEPARATOR_LINE}\n\n" {SEPARATOR_LINE}
"And here are the entities, relationships and terms extracted from the context:\n"
f"{SEPARATOR_LINE}\n" Here are the sub-questions that were suggested but not answered:
"{entity_term_extraction_str}\n" {SEPARATOR_LINE}
f"{SEPARATOR_LINE}\n\n" {{failed_sub_questions}}
"Please generate the list of good, fully contextualized sub-questions that would help to address the main question.\n" {SEPARATOR_LINE}
"Specifically pay attention also to the entities, relationships and terms extracted, as these indicate what type of "
"objects/relationships/terms you can ask about! Do not ask about entities, terms or relationships that are not " And here are the entities, relationships and terms extracted from the context:
"mentioned in the 'entities, relationships and terms' section.\n\n" {SEPARATOR_LINE}
"Again, please find questions that are NOT overlapping too much with the already answered " {{entity_term_extraction_str}}
"sub-questions or those that already were suggested and failed.\n" {SEPARATOR_LINE}
"In other words - what can we try in addition to what has been tried so far?\n\n"
"Generate the list of questions separated by one new line like this:\n" Please generate the list of good, fully contextualized sub-questions that would help to address the main question. \
"<sub-question 1>\n" Specifically pay attention also to the entities, relationships and terms extracted, as these indicate what type of \
"<sub-question 2>\n" objects/relationships/terms you can ask about! Do not ask about entities, terms or relationships that are not mentioned in the \
"<sub-question 3>\n" 'entities, relationships and terms' section.
"..."
).strip() Again, please find questions that are NOT overlapping too much with the already answered sub-questions or those that \
already were suggested and failed. In other words - what can we try in addition to what has been tried so far?
Generate the list of questions separated by one new line like this:
<sub-question 1>
<sub-question 2>
<sub-question 3>
...""".strip()
REFINED_ANSWER_PROMPT_W_SUB_QUESTIONS = ( REFINED_ANSWER_PROMPT_W_SUB_QUESTIONS = f"""
"{persona_specification}\n\n" {{persona_specification}}
"Your task is to improve on a given answer to a question, as the initial answer was found to be lacking in some way.\n\n"
"Use the information provided below - and only the provided information - to write your new and improved answer.\n\n" Your task is to improve on a given answer to a question, as the initial answer was found to be lacking in some way.
"The information provided below consists of:\n"
" 1) an initial answer that was given but found to be lacking in some way.\n" Use the information provided below - and only the provided information - to write your new and improved answer.
" 2) a number of answered sub-questions - these are very important(!) and definitely should help you to answer "
"the main question. Note that the sub-questions have a type, 'initial' and 'refined'. The 'initial' " The information provided below consists of:
"ones were available for the creation of the initial answer, but the 'refined' were not, they are new. So please use " 1) an initial answer that was given but found to be lacking in some way.
"the 'refined' sub-questions in particular to update/extend/correct/enrich the initial answer and to add " 2) a number of answered sub-questions - these are very important(!) and definitely should help you to answer the main \
"more details/new facts!\n" question. Note that the sub-questions have a type, 'initial' and 'refined'. The 'initial' ones were available for the \
" 3) a number of documents that were deemed relevant for the question. This is the context that you use largely for " creation of the initial answer, but the 'refined' were not, they are new. So please use the 'refined' sub-questions in \
"citations (see below). So consider the answers to the sub-questions as guidelines to construct your new answer, but " particular to update/extend/correct/enrich the initial answer and to add more details/new facts!
"make sure you cite the relevant document for a fact!\n\n" 3) a number of documents that were deemed relevant for the question. This is the context that you use largely for citations \
"It is critical that you provide proper inline citations to documents in the format [D1], [D2], [D3], etc! " (see below). So consider the answers to the sub-questions as guidelines to construct your new answer, but make sure you cite \
"It is important that the citation is close to the information it supports. " the relevant document for a fact!
"DO NOT just list all of the citations at the very end. "
"Feel free to also cite sub-questions in addition to documents, but make sure that you have documents cited with the " It is critical that you provide proper inline citations to documents in the format [D1], [D2], [D3], etc! \
"sub-question citation. If you want to cite both a document and a sub-question, please use [D1][Q3], or [D2][D7][Q4], etc. " It is important that the citation is close to the information it supports. \
"and always place the document citation before the sub-question citation. " DO NOT just list all of the citations at the very end. \
"Again, please NEVER cite sub-questions without a document citation!\n" Feel free to also cite sub-questions in addition to documents, \
"Proper citations are very important for the user!\n\n" but make sure that you have documents cited with the sub-question citation. \
"{history}\n\n" If you want to cite both a document and a sub-question, please use [D1][Q3], or [D2][D7][Q4], etc. and always place the \
"IMPORTANT RULES:\n" document citation before the sub-question citation. Again, please NEVER cite sub-questions without a document citation! \
" - If you cannot reliably answer the question solely using the provided information, say that you cannot reliably answer. " Proper citations are very important for the user!
"You may give some additional facts you learned, but do not try to invent an answer.\n"
f' - If the information is empty or irrelevant, just say "{UNKNOWN_ANSWER}".\n' {{history}}
" - If the information is relevant but not fully conclusive, provide an answer to the extent you can but also "
"specify that the information is not conclusive and why.\n" IMPORTANT RULES:
" - Ignore any existing citations within the answered sub-questions, like [D1]... and [Q2]! " - If you cannot reliably answer the question solely using the provided information, say that you cannot reliably answer. \
"The citations you will need to use will need to refer to the documents (and sub-questions) that you are explicitly " You may give some additional facts you learned, but do not try to invent an answer.
"presented with below!\n\n" - If the information is empty or irrelevant, just say "{UNKNOWN_ANSWER}".
"Again, you should be sure that the answer is supported by the information provided!\n\n" - If the information is relevant but not fully conclusive, provide an answer to the extent you can but also specify that \
"Try to keep your answer concise. But also highlight uncertainties you may have should there be substantial ones, " the information is not conclusive and why.
"or assumptions you made.\n\n" - Ignore any existing citations within the answered sub-questions, like [D1]... and [Q2]! The citations you will need to \
"Here is the contextual information:\n" use will need to refer to the documents (and sub-questions) that you are explicitly presented with below!
f"{SEPARATOR_LINE_LONG}\n\n"
"*Initial Answer that was found to be lacking:\n" Again, you should be sure that the answer is supported by the information provided!
f"{SEPARATOR_LINE}\n"
"{initial_answer}\n" Try to keep your answer concise. But also highlight uncertainties you may have should there be substantial ones, \
f"{SEPARATOR_LINE}\n\n" or assumptions you made.
"*Answered Sub-questions (these should really help you to research your answer! They also contain questions/answers "
"that were not available when the original answer was constructed):\n" Here is the contextual information:
"{answered_sub_questions}\n\n" {SEPARATOR_LINE_LONG}
"And here are the relevant documents that support the sub-question answers, and that are relevant for the actual question:\n"
f"{SEPARATOR_LINE}\n" *Initial Answer that was found to be lacking:
"{relevant_docs}\n" {SEPARATOR_LINE}
f"{SEPARATOR_LINE}\n\n" {{initial_answer}}
"Lastly, here is the main question I want you to answer based on the information above:\n" {SEPARATOR_LINE}
f"{SEPARATOR_LINE}\n"
"{question}\n" *Answered Sub-questions (these should really help you to research your answer! They also contain questions/answers that \
f"{SEPARATOR_LINE}\n\n" were not available when the original answer was constructed):
"Please keep your answer brief and concise, and focus on facts and data.\n\n" {{answered_sub_questions}}
"Answer:"
).strip() And here are the relevant documents that support the sub-question answers, and that are relevant for the actual question:
{SEPARATOR_LINE}
{{relevant_docs}}
{SEPARATOR_LINE}
Lastly, here is the main question I want you to answer based on the information above:
{SEPARATOR_LINE}
{{question}}
{SEPARATOR_LINE}
Please keep your answer brief and concise, and focus on facts and data.
Answer:
""".strip()
# sub_question_answer_str is empty # sub_question_answer_str is empty
REFINED_ANSWER_PROMPT_WO_SUB_QUESTIONS = ( REFINED_ANSWER_PROMPT_WO_SUB_QUESTIONS = f"""
"{answered_sub_questions}{persona_specification}\n\n" {{answered_sub_questions}}{{persona_specification}}
"Use the information provided below - and only the provided information - to answer the provided question.\n\n"
"The information provided below consists of:\n" Use the information provided below - and only the provided information - to answer the provided question.
" 1) an initial answer that was given but found to be lacking in some way.\n"
" 2) a number of documents that were also deemed relevant for the question.\n\n" The information provided below consists of:
"It is critical that you provide proper inline citations to documents in the format [D1], [D2], [D3], etc! " 1) an initial answer that was given but found to be lacking in some way.
"It is important that the citation is close to the information it supports. " 2) a number of documents that were also deemed relevant for the question.
"DO NOT just list all of the citations at the very end of your response. Citations are very important for the user!\n\n"
"{history}\n\n" It is critical that you provide proper inline citations to documents in the format [D1], [D2], [D3], etc! \
"IMPORTANT RULES:\n" It is important that the citation is close to the information it supports. \
" - If you cannot reliably answer the question solely using the provided information, say that you cannot reliably answer. " DO NOT just list all of the citations at the very end of your response. Citations are very important for the user!
"You may give some additional facts you learned, but do not try to invent an answer.\n"
f' - If the information is empty or irrelevant, just say "{UNKNOWN_ANSWER}".\n' {{history}}
" - If the information is relevant but not fully conclusive, provide an answer to the extent you can but also "
"specify that the information is not conclusive and why.\n\n" IMPORTANT RULES:
"Again, you should be sure that the answer is supported by the information provided!\n\n" - If you cannot reliably answer the question solely using the provided information, say that you cannot reliably answer. \
"Try to keep your answer concise. But also highlight uncertainties you may have should there be substantial ones, " You may give some additional facts you learned, but do not try to invent an answer.
"or assumptions you made.\n\n" - If the information is empty or irrelevant, just say "{UNKNOWN_ANSWER}".
"Here is the contextual information:\n" - If the information is relevant but not fully conclusive, provide an answer to the extent you can but also specify that \
f"{SEPARATOR_LINE_LONG}\n\n" the information is not conclusive and why.
"*Initial Answer that was found to be lacking:\n"
f"{SEPARATOR_LINE}\n" Again, you should be sure that the answer is supported by the information provided!
"{initial_answer}\n"
f"{SEPARATOR_LINE}\n\n" Try to keep your answer concise. But also highlight uncertainties you may have should there be substantial ones, \
"And here are relevant document information that support the sub-question answers, " or assumptions you made.
"or that are relevant for the actual question:\n"
f"{SEPARATOR_LINE}\n" Here is the contextual information:
"{relevant_docs}\n" {SEPARATOR_LINE_LONG}
f"{SEPARATOR_LINE}\n\n"
"Lastly, here is the question I want you to answer based on the information above:\n" *Initial Answer that was found to be lacking:
f"{SEPARATOR_LINE}\n" {SEPARATOR_LINE}
"{question}\n" {{initial_answer}}
f"{SEPARATOR_LINE}\n\n" {SEPARATOR_LINE}
"Please keep your answer brief and concise, and focus on facts and data.\n\n"
"Answer:" And here are relevant document information that support the sub-question answers, \
).strip() or that are relevant for the actual question:
{SEPARATOR_LINE}
{{relevant_docs}}
{SEPARATOR_LINE}
Lastly, here is the question I want you to answer based on the information above:
{SEPARATOR_LINE}
{{question}}
{SEPARATOR_LINE}
Please keep your answer brief and concise, and focus on facts and data.
Answer:
""".strip()
INITIAL_REFINED_ANSWER_COMPARISON_PROMPT = ( INITIAL_REFINED_ANSWER_COMPARISON_PROMPT = f"""
"For the given question, please compare the initial answer and the refined answer and determine if " For the given question, please compare the initial answer and the refined answer and determine if the refined answer is \
"the refined answer is substantially better than the initial answer, not just a bit better. Better could mean:\n" substantially better than the initial answer, not just a bit better. Better could mean:
" - additional information\n" - additional information
" - more comprehensive information\n" - more comprehensive information
" - more concise information\n" - more concise information
" - more structured information\n" - more structured information
" - more details\n" - more details
" - new bullet points\n" - new bullet points
" - substantially more document citations ([D1], [D2], [D3], etc.)\n\n" - substantially more document citations ([D1], [D2], [D3], etc.)
"Put yourself in the shoes of the user and think about whether the refined answer is really substantially "
"better and delivers really new insights than the initial answer.\n\n" Put yourself in the shoes of the user and think about whether the refined answer is really substantially better and \
"Here is the question:\n" delivers really new insights than the initial answer.
f"{SEPARATOR_LINE}\n"
"{question}\n" Here is the question:
f"{SEPARATOR_LINE}\n\n" {SEPARATOR_LINE}
"Here is the initial answer:\n" {{question}}
f"{SEPARATOR_LINE}\n" {SEPARATOR_LINE}
"{initial_answer}\n"
f"{SEPARATOR_LINE}\n\n" Here is the initial answer:
"Here is the refined answer:\n" {SEPARATOR_LINE}
f"{SEPARATOR_LINE}\n" {{initial_answer}}
"{refined_answer}\n" {SEPARATOR_LINE}
f"{SEPARATOR_LINE}\n\n"
"With these criteria in mind, is the refined answer substantially better than the initial answer?\n\n" Here is the refined answer:
f'Please answer with a simple "{YES}" or "{NO}".' {SEPARATOR_LINE}
).strip() {{refined_answer}}
{SEPARATOR_LINE}
With these criteria in mind, is the refined answer substantially better than the initial answer?
Please answer with a simple "{YES}" or "{NO}"
""".strip()

5
backend/onyx/prompts/chat_prompts.py
View File

@@ -2,8 +2,9 @@ from onyx.prompts.constants import GENERAL_SEP_PAT
from onyx.prompts.constants import QUESTION_PAT from onyx.prompts.constants import QUESTION_PAT
REQUIRE_CITATION_STATEMENT = """ REQUIRE_CITATION_STATEMENT = """
Cite relevant statements INLINE using the format [1], [2], [3], etc to reference the document number, \ Cite relevant statements INLINE using the format [1], [2], [3], etc. to reference the document number. \
DO NOT provide a reference section at the end and DO NOT provide any links following the citations. DO NOT provide any links following the citations. To cite multiple documents, use [1], [2] format \
instead of [1, 2]. Try to cite inline as opposed to leaving all citations until the very end of the response.
""".rstrip() """.rstrip()
NO_CITATION_STATEMENT = """ NO_CITATION_STATEMENT = """