697 Commits

Author SHA1 Message Date
rkuo-danswer
2ac41c3719
Feature/celery beat watchdog (#4534)
* upgrade celery to release version

* make the watchdog script more reusable

* use constant

* code review

* catch interrupt

---------

Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
2025-04-15 22:05:37 +00:00
evan-danswer
a8cba7abae
extra logging for uncommon permissions cases (#4532)
* extra logging for uncommon permissions cases

* address CW comments
2025-04-15 18:56:17 +00:00
evan-danswer
ae9f8c3071
checkpointed confluence (#4473)
* checkpointed confluence

* confluence checkpointing tested

* fixed integration tests

* attempt to fix connector test flakiness

* fix rebase
2025-04-14 23:59:53 +00:00
Weves
ca2aeac2cc Fix black 2025-04-14 15:53:09 -07:00
ThomaciousD
f7543c6285 Fix #3764: Dynamically handle default branch in GitLab connector 2025-04-14 15:52:10 -07:00
rkuo-danswer
7c4487585d
rollback properly on exception (#4073)
* rollback properly on exception

* rollback on exception

* don't continue if we can't set the search path

* cleaner handling via context manager

---------

Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>
2025-04-14 21:48:35 +00:00
evan-danswer
68c6c1f4f8
refactor to use stricter typing (#4513)
* refactor to use stricter typing

* older version of ruff
2025-04-14 17:23:07 +00:00
Chris Weaver
a5edc8aa0f
Fix default log level (#4501)
* Fix default log level

* fix
2025-04-14 16:40:11 +00:00
evan-danswer
a377f6ffb6
Unify document deduping (#4520)
* minor cleanup

* cleanup doc deduping and add unit tests
2025-04-14 16:33:00 +00:00
joachim-danswer
2683207a24
Expanded basic search (#4517)
* initial working version

* ranking profile

* modification for keyword/instruction retrieval

* mypy fixes

* EL comments

* added env var (True for now)

* flipped default to False

* mypy & final EL/CW comments + import issue
2025-04-13 23:13:01 -07:00
Chris Weaver
e3aab8e85e
Improve index attempt display (#4511) 2025-04-13 15:57:47 -07:00
pablonyx
65fd8b90a8
add image indexing tests (#4477)
* address file path

* k

* update

* update

* nit- fix typing

* k

* should path

* in a good state

* k

* k

* clean up file

* update

* update

* k

* k

* k
2025-04-11 22:16:37 +00:00
Chris Weaver
6eaa774051
Confluence timeout fix? (#4509) 2025-04-11 20:06:27 +00:00
evan-danswer
60da282dd1
ensure individual search tool runs do not affect each other (#4503)
* ensure individual search tool runs do not affect each other

* small bug fixes

* nit
2025-04-11 17:24:57 +00:00
rkuo-danswer
493e5386ec
Bugfix/salesforce correctness (#4497)
* refactor salesforce sqlite db access

* more refactoring

* refactor again

* refactor again

* rename object

* add finalizer to ensure db connection is always closed

* avoid unnecessarily nesting connections and commit regularly when possible

---------

Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
2025-04-11 08:41:22 +00:00
rkuo-danswer
bc74bcae3a
updating more packages (#4502)
* updating more packages

* mypy fixes

---------

Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
2025-04-10 20:53:36 -07:00
evan-danswer
e51e4b33b6
fix max 10 drives issue (#4505) 2025-04-11 02:22:38 +00:00
rkuo-danswer
1d7d5e1809
fix scheduler init (#4504) 2025-04-10 18:21:47 -07:00
Weves
6d48b9b4fd fix drive permission sync 2025-04-10 10:41:40 -07:00
rkuo-danswer
77e60b9812
remove try update in init ... we really don't need the init to access the db or do any work. (#4498)
Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
2025-04-10 10:15:28 -07:00
rkuo-danswer
24184024bb
Bugfix/dependency updates (#4482)
* bump fastapi and starlette

* bumping llama index and nltk and associated deps

* bump to fix python-multipart

* bump aiohttp

* update package lock for examples/widget

* bump black

* sentencesplitter has changed namespaces

* fix reorder import check, fix missing passlib

* update package-lock.json

* black formatter updated

* reformatted again

* change to black compatible reorder

* change to black compatible reorder-python-imports fork

* fix pytest dependency

* black format again

* we don't need cdk.txt. update packages to be consistent across all packages

---------

Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
Co-authored-by: Richard Kuo <rkuo@rkuo.com>
2025-04-10 08:23:02 +00:00
evan-danswer
e79134eaa0
don't yield expected auth errors (#4494)
* don't yield expected auth errors

* only catch 403s
2025-04-10 01:53:02 +00:00
evan-danswer
b5be1fb948
important clarity comment (#4492) 2025-04-10 01:28:34 +00:00
evan-danswer
1718b8f677
fix claude bug (#4493)
* fix claude bug

* fixed tests
2025-04-10 00:59:18 +00:00
rkuo-danswer
3fc8027e73
pass through various id's and log them in the model server for better… (#4485)
* pass through various id's and log them in the model server for better tracking

* fix test

---------

Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
2025-04-10 00:40:57 +00:00
pablonyx
caa9b106e4
k (#4487) 2025-04-10 00:19:47 +00:00
pablonyx
8db80a6bb1
Add latency metrics (#4472)
* k

* update

* Update chat_backend.py

nit

---------

Co-authored-by: evan-danswer <evan@danswer.ai>
2025-04-08 21:23:26 +00:00
rkuo-danswer
9b6c7625fd
Bugfix/cloud checkpoint cleanup (#4478)
* use send_task to be consistent

* add pidbox monitoring task

* add logging so we can track the task execution

* log the idletime of the pidbox

---------

Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
2025-04-08 19:47:07 +00:00
Chris Weaver
634d990cb8
Fix startup w/ seed_db (#4481) 2025-04-08 19:46:41 +00:00
pablonyx
5792261a4f
Minor doc set fix (#4480)
* update

* update

* update

* k
2025-04-08 19:14:56 +00:00
Chris Weaver
71839e723f
Add stuff to better avoid bot-detection in web connector (#4479)
* Add stuff to better avoid bot-detection in web connector

* Switch to exception log
2025-04-08 12:31:30 -07:00
evan-danswer
10f1ac5da1
use persona info when creating tool args (#4397)
* use persona info when creating tool args

* fixed unit test

* include system message

* fix unit test

* nit
2025-04-08 02:55:36 +00:00
Weves
1f80ed11d9 Fix black 2025-04-07 20:33:15 -07:00
Emerson Gomes
ba80191f5b
Handle exception for token cost calculation (#4474)
The code for token cost calculation fails when using a LiteLLM proxy due to mismatch with the provider naming. For now, just handle this exception and assume cost 0 when that happens instead of breaking the flow - A more precise, LiteLLM proxy based cost calculation (relying in the `/model/info`) LiteLLM Proxy method will be needed
2025-04-07 20:30:50 -07:00
Raunak Bhagat
206daa6903
feat: Vertex AI support (#4458)
* Add gemini well-known-llm-provider

* Edit styling of anonymous function

* Remove space

* Edit how advanced options are displayed

* Add VertexAI to acceptable llm providers

* Add new `FileUploadFormField` component

* Edit FileUpload component

* Clean up logic for displaying native llm providers; add support for more complex `CustomConfigKey` types

* Fix minor nits in web app

* Add ability to pass vertex credentials to `litellm`

* Remove unused prop

* Change name of enum value

* Add back ability to change form based on first time configurations

* Create new Error with string instead of throwing raw string

* Add more Gemini models

* Edit mappings for Gemini models

* Edit comment

* Rearrange llm models

* Run black formatter

* Remove complex configurations during first time registration

* Fix nit

* Update llm provider name

* Edit temporary formik field to also have the filename

* Run reformatter

* Reorder commits

* Add advanced configurations for enabled LLM Providers
2025-04-08 00:56:47 +00:00
evan-danswer
17562f9b8f
Id not set in checkpoint2 (#4468)
* unconditionally set completion

* drive connector improvements

* fixing broader typing issue

* fix tests, CW comments

* actual test fix
2025-04-07 17:00:42 -07:00
evan-danswer
9c73099241
Drive smart chip indexing (#4459)
* WIP

* WIP almost done, but realized we can just do basic retrieval

* rebased and added scripts

* improved approach to extracting smart chips

* remove files from previous branch

* fix connector tests

* fix test
2025-04-07 21:52:45 +00:00
Emerson Gomes
88d4a65e7b Fix hardcoded temperature 2025-04-07 13:56:11 -07:00
SubashMohan
157da24504
update test expectations for Highspot connector (#4464) 2025-04-07 05:12:22 +00:00
Evan Lohn
989dab51b9 unconditionally set completion 2025-04-06 22:39:42 -07:00
Chris Weaver
aadd4f212a
Adjust pg engine intialization (#4408)
* Adjust pg engine intialization

* Fix mypy

* Rename var

* fix typo

* Fix tests
2025-04-06 12:44:49 -07:00
Ferdinand Loesch
8b05f98d54
Thinking mode UI. (#4370)
* Update web connector implementation and fix line length issues

* Update configurations and fix connector issues

* Update Slack connector

* Update connectors and add jira_test_env to gitignore, removing sensitive information

* Restore checkpointing functionality and remove sensitive information

* Fix agent mode to properly handle thinking tokens

* up

* Enhance ThinkingBox component with improved content handling and animations. Added support for partial thinking tokens, refined scrolling behavior, and updated CSS for better visual feedback during thinking states.

* Create clean branch with frontend thinking mode changes only

* Update ThinkingBox component to include new props for completion and streaming states. Refactor smooth scrolling logic into a dedicated function for improved readability. Add new entry to .gitignore for jira_test_env.

* Remove autoCollapse prop from AIMessage component for improved flexibility in message display.

* Update thinking tokens handling in chat utils

* Remove unused cleanThinkingContent import from Messages component to streamline code.

---------

Co-authored-by: ferdinand loesch <f.loesch@sportradar.com>
Co-authored-by: EC2 Default User <ec2-user@ip-10-73-128-233.eu-central-1.compute.internal>
Co-authored-by: Your Name <you@example.com>
Co-authored-by: Chris Weaver <25087905+Weves@users.noreply.github.com>
2025-04-05 17:31:02 -07:00
Weves
1c16c4ea3d Adjusting default search assistant 2025-04-05 16:00:47 -07:00
pablonyx
09450010cd
refresh token limit (#4456) 2025-04-05 01:27:57 +00:00
pablonyx
0acd50b75d docx bugfix 2025-04-04 18:20:31 -07:00
pablonyx
c3c9a0e57c
Docx parsing (#4455)
* looks okay

* k

* k

* k

* update values

* k

* quick fix
2025-04-04 23:36:43 +00:00
pablonyx
ef978aea97
Additional ACL Tests + Slackbot fix (#4430)
* try turning drive perm sync on

* try passing in env var

* add some logs

* Update pr-integration-tests.yml

* revert "Update pr-integration-tests.yml"

This reverts commit 76a44adbfed4d94282bc99450ab05b2759e81521.

* Revert "add some logs"

This reverts commit ab9e6bcfb1825b4f5d701f1b758d92b2a0bc1850.

* Revert "try passing in env var"

This reverts commit 9c0b6162ea856d53b82bd3daac71642266714426.

* Revert "try turning drive perm sync on"

This reverts commit 2d35f61f427432defd775459d55cfd93be0646f5.

* try slack connector

* k

* update

* remove logs

* remove more logs

* nit

* k

* k

* address nits

* run test with additional logs

* Revert "run test with additional logs"

This reverts commit 1397a2c4a0a554da66a825299eaa6ddf18cdcc39.

* Revert "address nits"

This reverts commit d5e24b019dfaf41e88d4e2bd41734e92d64b929d.
2025-04-04 22:00:17 +00:00
rkuo-danswer
15ab0586df
handle gong api race condition (#4457)
* working around a gong race condition in their api

* add back gong basic test

* formatting

* add the call index

---------

Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
2025-04-04 19:33:47 +00:00
rkuo-danswer
839c8611b7
Bugfix/salesforce (#4335)
* add some gc

* small refactoring for temp directories

* WIP

* add some gc collects and size calculations

* un-xfail

* fix salesforce test

* loose check for number of docs

* adjust test again

* cleanup

* nuke directory param, remove using sqlite db to cache email / id mappings

---------

Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
2025-04-04 16:21:34 +00:00
joachim-danswer
68f9f157a6
Adding research topics for better search context (#4448)
* research topics addition

* allow for question to overwrite research area
2025-04-04 09:53:39 -07:00