* WIP
* implement hard timeout
* fix callbacks
* put back the timeout
* missed a file
* fixes
* try installing playwright deps
* Revert "try installing playwright deps"
This reverts commit 42174275682e4479f301209deddb6f52762ea0a4.
---------
Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>
Co-authored-by: Richard Kuo <rkuo@rkuo.com>
* ignore result when using send_task on lightweight tasks
* fix ignore_result
---------
Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>
Co-authored-by: Richard Kuo <rkuo@rkuo.com>
* no thread local locks in callbacks and raise permission sync timeout by a lot based on empirical log observations
* more fixes
---------
Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>
* move indexing
* all monitor work moved
* reacquire lock more
* remove monitor task completely
* fix import
* fix pruning finalization
* no multiplier on system/cloud tasks
* monitor queues every 30 seconds in the cloud
---------
Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>
* add validation for pruning
* fix missing class
* get external group sync validation working
* backport fix for pruning check
* fix pruning
* log the payload id
* remove scan_iter from pruning
* missed removed scan_iter, also remove other scan_iters and replace with sscan_iter of the lookup table
* external group sync needs active signal. h
* log the payload id when the task starts
* log the payload id in more places
* use the replica
* increase primary pool and slow down beat
* scale sql pool based on concurrency
* fix concurrency
* add debugging for external group sync and tenant
* remove debugging and fix payload id
---------
Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>
* WIP for external group sync lock fixes
* prototyping permissions validation
* validate permission sync tasks in celery
* mypy
* cleanup and wire off external group sync checks for now
* add active key to reset
* improve logging
* reset on payload format change
* return False on exception
* missed a return
* add count of tasks scanned
* add comment
* better logging
* add return
* more return
* catch payload exceptions
* code review fixes
* push to restart test
---------
Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>
* try using a redis replica in some areas
* harden up replica usage
* comment
* slow down cloud dispatch temporarily
* add ignored syncing list back
* raise multiplier to 8
* comment out per tenant code (no longer used by fanout)
---------
Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>
* signal from the watchdog so that the monitor task doesn't try to clean up before it can exit
* ttl constants
* improve comment
---------
Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>
* WIP
* WIP
* try spinning out check for indexing into a system task
* check for the correct delimiter
* use constants
---------
Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>
Co-authored-by: Richard Kuo <rkuo@rkuo.com>
* more debugging
* test reacquire outside of loop
* more logging
* move lock_beat test outside the try catch so that we don't worry about testing locks we never took
* use a larger scan_iter value for performance
* batch stale document sync batches
* add debug logging for a particular timeout issue
---------
Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>
* allow beat tasks to expire. it isn't important that they all run
* validate fences are in a good state and cancel/fail them if not
* add function timings for important beat tasks
* optimize lookups, add lots of comments
* review changes
---------
Co-authored-by: Richard Kuo <rkuo@rkuo.com>
Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>
