* wip checkpointing/continue on failure
more stuff for checkpointing
Basic implementation
FE stuff
More checkpointing/failure handling
rebase
rebase
initial scaffolding for IT
IT to test checkpointing
Cleanup
cleanup
Fix it
Rebase
Add todo
Fix actions IT
Test more
Pagination + fixes + cleanup
Fix IT networking
fix it
* rebase
* Address misc comments
* Address comments
* Remove unused router
* rebase
* Fix mypy
* Fixes
* fix it
* Fix tests
* Add drop index
* Add retries
* reset lock timeout
* Try hard drop of schema
* Add timeout/retries to downgrade
* rebase
* test
* test
* test
* Close all connections
* test closing idle only
* Fix it
* fix
* try using null pool
* Test
* fix
* rebase
* log
* Fix
* apply null pool
* Fix other test
* Fix quality checks
* Test not using the fixture
* Fix ordering
* fix test
* Change pooling behavior
* dedupe make_private_persona and update test
* add comment
* comments, and just have duplicate user id's for the test instead of modifying edit
* found the magic word
---------
Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>
* add timings for syncing
* add more logging
* more debugging
* refactor multipass/db check out of VespaIndex
* circular imports?
* more debugging
* add logs
* various improvements
* additional logs to narrow down issue
* use global httpx pool for the main vespa flows in celery. Use in more places eventually.
* cleanup debug logging, etc
* remove debug logging
* this should use the secondary index
* mypy
* missed some logging
* review fixes
* refactor get_default_document_index to use search settings
* more missed logging
* fix circular refs
---------
Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>
Co-authored-by: pablodanswer <pablo@danswer.ai>
* Added ability to use a tag to insert the current datetime in prompts
* made tagging logic more robust
* rename
* k
---------
Co-authored-by: Yuhong Sun <yuhongsun96@gmail.com>
* Combined Persona and Prompt API
* quality
* added tests
* consolidated models and got rid of redundant fields
* tenant appreciation day
* reverted default
* checkpoint
* add celery termination of the task
* rename to RedisConnectorPermissionSyncPayload, add RedisLock to more places, add get_active_search_settings
* rename payload
* pretty sure these weren't named correctly
* testing in progress
* cleanup
* remove space
* merge fix
* three dots animation on Pausing
* improve messaging when connector is stopped or killed and animate buttons
---------
Co-authored-by: Richard Kuo <rkuo@rkuo.com>
* add creator id to cc pair
* fix alembic head
* show email instead of UUID
* safer check on email
* make foreign key relationships optional
* always allow creator to edit (per hagen)
* use primary join
* no index_doc_batch spam
* try this again
---------
Co-authored-by: Richard Kuo <rkuo@rkuo.com>
* Make curators able to create permission synced connectors
* removed editing permission synced connectors for curators
* updated tests to use access type instead of is_public
* update copy
* in progress PoC
* working limited user, needs routes to be marked next
* make selected endpoint available to limited user role
* xfail on test_slack_prune
* add comment to sync function
---------
Co-authored-by: Richard Kuo <rkuo@rkuo.com>
* cloud auth referral source
* minor clarity
* k
* minor modification to be best practice
* typing
* Update ReferralSourceSelector.tsx
* Update ReferralSourceSelector.tsx
---------
Co-authored-by: hagen-danswer <hagen@danswer.ai>
* doc_sync is refactored
* maybe this works
* tested to work!
* mypy fixes
* enabled integration tests
* fixed the test
* added external group sync
* testing should work now
* mypy
* confluence doc id fix
* got group sync working
* addressed feedback
* renamed some vars and fixed mypy
* conf fix?
* added wiki handling to confluence connector
* test fixes
* revert google drive connector
* fixed groups
* hotfix
* refactor RedisConnectorDeletion into RedisConnector
* refactor redis stop and deletion
* port pruning
* nest pruning
* port deletion
* port indexing
* refactor into individual files
* refactor redis connector index to take search settings at init
* move back to debug level log
* refactor doc set and user group (mostly)
* mypy fixes
* check for index swap
* initial bones
* kk
* k
* k:
* nit
* nit
* rebase + update
* nit
* minior update
* k
* minor integration test fixes
* nit
* ensure we build test docker image
* remove one space
* k
* ensure we wipe volumes
* remove log
* typo
* nit
* k
* k
* fresh indexing feature branch
* cherry pick test
* Revert "cherry pick test"
This reverts commit 2a624220687affdda3de347e30f2011136f64bda.
* set multitenant so that vespa fields match when indexing
* cleanup pass
* mypy
* pass through env var to control celery indexing concurrency
* comments on task kickoff and some logging improvements
* use get_session_with_tenant
* comment out all of update.py
* rename to RedisConnectorIndexingFenceData
* first check num_indexing_workers
* refactor RedisConnectorIndexingFenceData
* comment out on_worker_process_init
* fix where num_indexing_workers falls back
* remove extra brace
* check last_pruned instead of is_pruning
* try using the ThreadingHTTPServer class for stability and avoiding blocking single-threaded behavior
* add startup delay to web server in test
* just explicitly return None if we can't parse the datetime
* switch to uvicorn for test stability
