rkuo-danswer
6913efef90
fresh indexing feature branch ( #2790 )
...
* fresh indexing feature branch
* cherry pick test
* Revert "cherry pick test"
This reverts commit 2a624220687affdda3de347e30f2011136f64bda.
* set multitenant so that vespa fields match when indexing
* cleanup pass
* mypy
* pass through env var to control celery indexing concurrency
* comments on task kickoff and some logging improvements
* use get_session_with_tenant
* comment out all of update.py
* rename to RedisConnectorIndexingFenceData
* first check num_indexing_workers
* refactor RedisConnectorIndexingFenceData
* comment out on_worker_process_init
* fix where num_indexing_workers falls back
* remove extra brace
2024-10-18 22:40:05 +00:00
rkuo-danswer
12cbbe6cee
use with for update instead of serializable ( #2848 )
...
* use with for update instead of serializable
* remove tenant logic handled now by get_session_with_tenant
* remove usage of begin_nested ... it's not necessary
2024-10-18 20:35:23 +00:00
Chris Weaver
36134021c5
Refactor + add global timeout env variable ( #2844 )
...
* Refactor + add global timeout env variable
* remove model
* mypy
* Remove unused
2024-10-18 18:25:27 +00:00
rkuo-danswer
5b78299880
use native rate limiting in the confluence client ( #2837 )
...
* use native rate limiting in the confluence client
* upgrade urllib3 to v2.2.3 to support retries in confluence client
* improve logging so that progress is visible.
2024-10-18 18:15:43 +00:00
Richard Kuo (Danswer)
59364aadd7
Revert "no serializable, use with_for_update to lock the row."
...
This reverts commit e12785d277296cd9749c8ee7d8959b4df62f957e.
2024-10-18 11:10:09 -07:00
Richard Kuo (Danswer)
e12785d277
no serializable, use with_for_update to lock the row.
2024-10-18 11:07:54 -07:00
pablodanswer
7906d9edc8
Add all-tenants migration for K8 job ( #2846 )
...
* add migration
* update migration logic for tenants
* k
* k
* k
* k
2024-10-18 02:55:05 +00:00
pablodanswer
6e54c97326
multitenant setup ( #2845 )
2024-10-17 17:54:02 -07:00
pablodanswer
61424de531
add sentry ( #2786 )
...
* add sentry
* nit
* nit
* add requirement to ee
* try to ensure sentry is installed in integration tests
2024-10-17 23:20:37 +00:00
rkuo-danswer
4c2cf8b132
always finalize the serialized transaction so that it doesn't leak ou… ( #2843 )
...
* always finalize the serialized transaction so that it doesn't leak outside the function
* re-raise the exception and log it
2024-10-17 23:13:57 +00:00
pablodanswer
b169f78699
Push multi tenancy for slackbot ( #2828 )
...
* push multi tenancy for slackbot
* move to utils
* k
* k
---------
Co-authored-by: hagen-danswer <hagen@danswer.ai>
2024-10-17 21:04:48 +00:00
pablodanswer
e48086b1c2
add slack markdown formatting ( #2829 )
...
* add slack markdown formatting
* nit
* k
2024-10-17 20:27:57 +00:00
hagen-danswer
deb66a88aa
dont fail flaky tests
2024-10-17 13:37:50 -07:00
hagen-danswer
90bd535c48
Merge pull request #2836 from danswer-ai/flakey-test-run-but-dont-fail
...
Make flakey test still run but not fail CI
2024-10-17 13:31:00 -07:00
rkuo-danswer
0de487064a
lock to avoid rare serializable errors ( #2818 )
...
Co-authored-by: Richard Kuo <rkuo@rkuo.com>
2024-10-17 19:44:51 +00:00
rkuo-danswer
114326d11a
fix sync to use update_single ( #2822 )
2024-10-17 19:43:34 +00:00
rkuo-danswer
389c7b72db
Bugfix/monitor exceptions ( #2830 )
...
* do a rollback before more db work
* warn if not all doc_by_cc_pair entries were deleted
---------
Co-authored-by: Richard Kuo <rkuo@rkuo.com>
2024-10-17 19:43:19 +00:00
hagen-danswer
28ad01a51a
py
2024-10-17 12:37:34 -07:00
hagen-danswer
0c102ebb5c
simplified the document search function
2024-10-17 12:13:42 -07:00
hagen-danswer
5063b944ec
Make flakey test still run but not fail CI
2024-10-17 11:36:59 -07:00
pablodanswer
15afe4dc78
bump litellm ( #2827 )
2024-10-17 18:05:35 +00:00
pablodanswer
a159779d39
prevent alembic from configuring logger ( #2826 )
...
* k
* k
2024-10-17 16:31:17 +00:00
hagen-danswer
938a65628d
rearrange logging
2024-10-17 09:01:51 -07:00
hagen-danswer
5d390b65eb
Added logging for when a member has no email or username
2024-10-17 08:47:46 -07:00
Chris Weaver
33974fc12c
Add support for passthrough auth for custom tool calls ( #2824 )
...
* Add support for passthrough auth for custom tool calls
* Fix formatting
2024-10-16 22:50:16 +00:00
pablodanswer
db0779dd02
Session id: int -> UUID ( #2814 )
...
* session id: int -> UUID
* nit
* validated
* validated downgrade + upgrade + all functionality
* nit
* minor nit
* fix test case
2024-10-16 22:18:45 +00:00
rkuo-danswer
0a0215ceee
check last_pruned instead of is_pruning ( #2748 )
...
* check last_pruned instead of is_pruning
* try using the ThreadingHTTPServer class for stability and avoiding blocking single-threaded behavior
* add startup delay to web server in test
* just explicitly return None if we can't parse the datetime
* switch to uvicorn for test stability
2024-10-16 18:52:27 +00:00
pablodanswer
1a9921f63e
Redirect with query param ( #2811 )
...
* validated
* k
* k
* k
* minor update
2024-10-16 17:26:44 +00:00
pablodanswer
a385234c0e
Parsing ( #2734 )
...
* k
* update chunking limits
* nit
* nit
* clean up types
* nit
* validate
* k
2024-10-16 16:44:19 +00:00
Yuhong Sun
c148fa5bfa
Notion Recurse Empty Final Field ( #2819 )
2024-10-15 23:03:37 -07:00
pablodanswer
11372aac8f
Add custom tool headers ( #2773 )
...
* add custom tool headers
* simplify
* k
* k
* k
* nit
2024-10-16 04:37:00 +00:00
Yuhong Sun
f23a89ccfd
Notion Empty Property Fix ( #2817 )
2024-10-15 21:52:00 -07:00
pablodanswer
e022e77b6d
Simpler azure embedding ( #2751 )
...
* functional but janky
* nit
* adapt for azure
* nit
* minor updates
* nits
* nit
* nit
* ensure access to litellm
* k
2024-10-15 23:23:11 +00:00
pablodanswer
bfe963988e
various multi tenant improvements ( #2803 )
...
* various multi tenant improvements
* nit
* ensure consistent db session operations
* minor robustification
2024-10-15 20:10:57 +00:00
pablodanswer
0e6c2f0b51
add ca option ( #2774 )
2024-10-15 19:23:04 +00:00
rkuo-danswer
efe2e79f27
Rate limiting confluence through redis ( #2798 )
...
* try rate limiting through redis
* fix circular import issue
* fix bad formatting of family string
* Revert "fix bad formatting of family string"
This reverts commit be688899e5b4dd189dc13d9fec1d0f3ade07ad4f.
* redis usage optional
* disable test that doesn't match with new design
2024-10-14 23:51:24 +00:00
pablodanswer
6f9740d026
Ensure warmup occurs once ( #2777 )
...
* ensure shared chats are shared
* ensure warm occurs once
* nit
* ensure warmup occurs once
* Revert "ensure shared chats are shared"
This reverts commit 8be887f3ee651e64b7f2eb3d45f7c36b53471a20.
2024-10-14 22:53:02 +00:00
rkuo-danswer
dee197570d
Bugfix/mediawiki ( #2800 )
...
* fix formatting
* fix poorly structured doc id, fix empty page id, fix family_class_dispatch invalid name (no spaces), fix setting id with int pageid
* fix mediawiki test
2024-10-14 22:48:06 +00:00
Chris Weaver
f8a7749b46
Fix file too large error ( #2799 )
...
* Fix file too large error
* Add cannotDownloadFile
2024-10-14 14:47:36 -07:00
hagen-danswer
494fda906d
Confluence permission sync fix for server deployment ( #2784 )
...
* initial commit
* Made perm sync with with cql
* filter fix
* undo connector changes
* fixed everything
* whoops
2024-10-14 20:52:57 +00:00
Weves
9537a2581e
Handle 'cannotExportFile' + fix forms
2024-10-14 09:49:54 -07:00
Weves
3ccd951307
Fix stopping of indexing runs when pausing a connector
2024-10-14 09:47:53 -07:00
Yuhong Sun
ba712d447d
Notion Connector Improvements ( #2789 )
2024-10-13 23:01:17 -07:00
pablodanswer
a9bcc89a2c
Add cursor to cql confluence ( #2775 )
...
* add cursor to cql confluence
* k
* k
* fixed space indexing issue
* fixed .get
---------
Co-authored-by: hagen-danswer <hagen@danswer.ai>
2024-10-14 02:09:17 +00:00
pablodanswer
ded42e2036
nit ( #2787 )
2024-10-14 01:22:14 +00:00
Yuhong Sun
b393af676c
Mypy ( #2785 )
2024-10-13 14:35:56 -07:00
Chris Weaver
26bdb41e8f
Fix parallel tool calls ( #2779 )
...
* Fix parallel tool calls
* remove comments
2024-10-13 03:29:18 +00:00
pablodanswer
20df20ae51
Multi tenant vespa ( #2762 )
...
* add vespa multi tenancy
* k
* formatting
* Billing (#2667 )
* k
* data -> control
* nit
* nit: error handling
* auth + app
* nit: color standardization
* nit
* nit: typing
* k
* k
* feat: functional upgrading
* feat: add block for downgrading to seats < active users
* add auth
* remove accomplished todo + prints
* nit
* tiny nit
* nit: centralize security
* add tenant expulsion/gating + invite user -> increment billing seat no.
* add cloud configs
* k
* k
* nit: update
* k
* k
* k
* k
* nit
2024-10-12 23:53:11 +00:00
pablodanswer
d892203821
fix typo ( #2768 )
2024-10-12 00:39:09 +00:00
Chris Weaver
35d32ea3b0
Fix indexing model server port for warmup ( #2767 )
2024-10-11 04:24:34 +00:00
