* first cut at redis
* some new helper functions for the db
* ignore kombu tables in alembic migrations (used by celery)
* multiline commands for readability, add vespa_metadata_sync queue to worker
* typo fix
* fix returning tuple fields
* add constants
* fix _get_access_for_document
* docstrings!
* fix double function declaration and typing
* fix type hinting
* add a global redis pool
* Add get_document function
* use task_logger in various celery tasks
* add celeryconfig.py to simplify configuration. Will be used in a subsequent commit
* Add celery redis helper. used in a subsequent PR
* kombu warning getting spammy since celery is not self managing its queue in Postgres any more
* add last_modified and last_synced to documents
* fix task naming convention
* use celeryconfig.py
* the big one. adds queues and tasks, updates functions to use the queues with priorities, etc
* change vespa index log line to debug
* mypy fixes
* update alembic migration
* fix fence ordering, rename to "monitor", fix fetch_versioned_implementation call
* mypy
* switch to monotonic time
* fix startup dependencies on redis
* rebase alembic migration
* kombu cleanup - fail silently
* mypy
* add redis_host environment override
* update REDIS_HOST env var in docker-compose.dev.yml
* update the rest of the docker files
* in flight
* harden indexing-status endpoint against db changes happening in the background. Needs further improvement but OK for now.
* allow no task syncs to run because we create certain objects with no entries but initially marked as out of date
* add back writing to vespa on indexing
* actually working connector deletion
* update contributing guide
* backporting fixes from background_deletion
* renaming cache to cache_volume
* add redis password to various deployments
* try setting up pr testing for helm
* fix indent
* hopefully this release version actually exists
* fix command line option to --chart-dirs
* fetch-depth 0
* edit values.yaml
* try setting ct working directory
* bypass testing only on change for now
* move files and lint them
* update helm testing
* some issues suggest using --config works
* add vespa repo
* add postgresql repo
* increase timeout
* try amd64 runner
* fix redis password reference
* add comment to helm chart testing workflow
* rename helm testing workflow to disable it
* adding clarifying comments
* address code review
* missed a file
* remove commented warning ... just not needed
* fix imports
* refactor to use update_single
* mypy fixes
* add vespa test
* multiple celery workers
* update logs as well and set prefetch multipliers appropriate to the worker intent
* add db refresh to connector deletion
* add some preliminary locking
* organize tasks into separate files
* celery auto associates tasks created inside another task, which bloats the result metadata considerably. trail=False prevents this.
* code review fixes
* move monitor_usergroup_taskset to ee, improve logging
* add multi workers to dev_run_background_jobs.py
* update supervisord with some recommended settings for celery
* name celery workers and shorten dev script prefixing
* add configurable sql alchemy engine settings on startup (needed for various intents like API server, different celery workers and tasks, etc)
* fix comments
* autoscale sqlalchemy pool size to celery concurrency (allow override later?)
* supervisord needs the percent symbols escaped
* use name as primary check, some minor refactoring and type hinting too.
* stash merge (may not function yet)
* remove dead code
* more cleanup
* remove dead file
* we shouldn't be checking for deletion attempts in the db any more
* print cc_pair_id
* print status on status mismatch again
* add logging when cc_pair isn't present
* don't indexing any ingestion type connectors, and don't pause any connectors that aren't active
* add more specific check for deletion completion
* remove flaky mediawiki test site
* move is_pruning
* remove unused code
* remove old function
---------
Co-authored-by: Richard Kuo <rkuo@rkuo.com>
* add tenant provisioning to data plane
* minor typing update
* ensure tenant router included
* proper auth check
* update disabling logic
* validated basic provisioning
* use new kv store
* Added permission sync tests for Slack
* moved folders
* prune test + mypy
* added wait for indexing to cc_pair creation
* commented out check
* should fix other tests
* added slack channel pool
* fixed everything and mypy
* reduced flake
* first cut at redis
* some new helper functions for the db
* ignore kombu tables in alembic migrations (used by celery)
* multiline commands for readability, add vespa_metadata_sync queue to worker
* typo fix
* fix returning tuple fields
* add constants
* fix _get_access_for_document
* docstrings!
* fix double function declaration and typing
* fix type hinting
* add a global redis pool
* Add get_document function
* use task_logger in various celery tasks
* add celeryconfig.py to simplify configuration. Will be used in a subsequent commit
* Add celery redis helper. used in a subsequent PR
* kombu warning getting spammy since celery is not self managing its queue in Postgres any more
* add last_modified and last_synced to documents
* fix task naming convention
* use celeryconfig.py
* the big one. adds queues and tasks, updates functions to use the queues with priorities, etc
* change vespa index log line to debug
* mypy fixes
* update alembic migration
* fix fence ordering, rename to "monitor", fix fetch_versioned_implementation call
* mypy
* switch to monotonic time
* fix startup dependencies on redis
* rebase alembic migration
* kombu cleanup - fail silently
* mypy
* add redis_host environment override
* update REDIS_HOST env var in docker-compose.dev.yml
* update the rest of the docker files
* in flight
* harden indexing-status endpoint against db changes happening in the background. Needs further improvement but OK for now.
* allow no task syncs to run because we create certain objects with no entries but initially marked as out of date
* add back writing to vespa on indexing
* actually working connector deletion
* update contributing guide
* backporting fixes from background_deletion
* renaming cache to cache_volume
* add redis password to various deployments
* try setting up pr testing for helm
* fix indent
* hopefully this release version actually exists
* fix command line option to --chart-dirs
* fetch-depth 0
* edit values.yaml
* try setting ct working directory
* bypass testing only on change for now
* move files and lint them
* update helm testing
* some issues suggest using --config works
* add vespa repo
* add postgresql repo
* increase timeout
* try amd64 runner
* fix redis password reference
* add comment to helm chart testing workflow
* rename helm testing workflow to disable it
* adding clarifying comments
* address code review
* missed a file
* remove commented warning ... just not needed
* fix imports
* refactor to use update_single
* mypy fixes
* add vespa test
* multiple celery workers
* update logs as well and set prefetch multipliers appropriate to the worker intent
* add db refresh to connector deletion
* add some preliminary locking
* organize tasks into separate files
* celery auto associates tasks created inside another task, which bloats the result metadata considerably. trail=False prevents this.
* code review fixes
* move monitor_usergroup_taskset to ee, improve logging
* add multi workers to dev_run_background_jobs.py
* update supervisord with some recommended settings for celery
* name celery workers and shorten dev script prefixing
* add configurable sql alchemy engine settings on startup (needed for various intents like API server, different celery workers and tasks, etc)
* fix comments
* autoscale sqlalchemy pool size to celery concurrency (allow override later?)
* supervisord needs the percent symbols escaped
* use name as primary check, some minor refactoring and type hinting too.
* addressing code review
* fix import
* fix prune_documents_task references
---------
Co-authored-by: Richard Kuo <rkuo@rkuo.com>
* Added permission syncing on the backend
* Rewored to work with celery
alembic fix
fixed test
* frontend changes
* got groups working
* added comments and fixed public docs
* fixed merge issues
* frontend complete!
* frontend cleanup and mypy fixes
* refactored connector access_type selection
* mypy fixes
* minor refactor and frontend improvements
* get to fetch
* renames and comments
* minor change to var names
* got curator stuff working
* addressed pablo's comments
* refactored user_external_group to reference users table
* implemented polling
* small refactor
* fixed a whoopsies on the frontend
* added scripts to seed dummy docs and test query times
* fixed frontend build issue
* alembic fix
* handled is_public overlap
* yuhong feedback
* added more checks for sync
* black
* mypy
* fixed circular import
* todos
* alembic fix
* alembic
* first cut at redis
* some new helper functions for the db
* ignore kombu tables in alembic migrations (used by celery)
* multiline commands for readability, add vespa_metadata_sync queue to worker
* typo fix
* fix returning tuple fields
* add constants
* fix _get_access_for_document
* docstrings!
* fix double function declaration and typing
* fix type hinting
* add a global redis pool
* Add get_document function
* use task_logger in various celery tasks
* add celeryconfig.py to simplify configuration. Will be used in a subsequent commit
* Add celery redis helper. used in a subsequent PR
* kombu warning getting spammy since celery is not self managing its queue in Postgres any more
* add last_modified and last_synced to documents
* fix task naming convention
* use celeryconfig.py
* the big one. adds queues and tasks, updates functions to use the queues with priorities, etc
* change vespa index log line to debug
* mypy fixes
* update alembic migration
* fix fence ordering, rename to "monitor", fix fetch_versioned_implementation call
* mypy
* switch to monotonic time
* fix startup dependencies on redis
* rebase alembic migration
* kombu cleanup - fail silently
* mypy
* add redis_host environment override
* update REDIS_HOST env var in docker-compose.dev.yml
* update the rest of the docker files
* in flight
* harden indexing-status endpoint against db changes happening in the background. Needs further improvement but OK for now.
* allow no task syncs to run because we create certain objects with no entries but initially marked as out of date
* add back writing to vespa on indexing
* actually working connector deletion
* update contributing guide
* backporting fixes from background_deletion
* renaming cache to cache_volume
* add redis password to various deployments
* try setting up pr testing for helm
* fix indent
* hopefully this release version actually exists
* fix command line option to --chart-dirs
* fetch-depth 0
* edit values.yaml
* try setting ct working directory
* bypass testing only on change for now
* move files and lint them
* update helm testing
* some issues suggest using --config works
* add vespa repo
* add postgresql repo
* increase timeout
* try amd64 runner
* fix redis password reference
* add comment to helm chart testing workflow
* rename helm testing workflow to disable it
* adding clarifying comments
* address code review
* missed a file
* remove commented warning ... just not needed
* fix imports
* refactor to use update_single
* mypy fixes
* add vespa test
* add db refresh to connector deletion
* code review fixes
* move monitor_usergroup_taskset to ee, improve logging
---------
Co-authored-by: Richard Kuo <rkuo@rkuo.com>
* Move StandardAnswer to EE section of danswer/db/models
* Move StandardAnswer DB layer to EE
* Add EERequiredError for distinct error handling here
* Handle EE fallback for slack bot config
* Migrate all standard answer models to ee
* Flagging categories for removal
* Add missing versioned impl for update_slack_bot_config
---------
Co-authored-by: danswer-trial <danswer-trial@danswer-trials-MacBook-Pro.local>
* persona
* all prepared excluding configuration
* more sensical model structure
* update tstream
* type updates
* rm
* quick and simple updates
* minor updates
* te
* ensure typing + naming
* remove old todo + rebase update
* remove unnecessary check
* migration: add column "match_any_keywords" to StandardAnswer
* Implement any/all keyword matching for standard answers
* Add match_any_keywords to non-searchable fields
* Remove stray print
* Simplify Slack messages for any and all cases
---------
Co-authored-by: danswer-trial <danswer-trial@danswer-trials-MacBook-Pro.local>
* Migrate standard answers implementations to ee/
* renaming
* Clean up slackbot non-ee standard answers import
* Move backend api/manage/standard_answer route to ee
* Move standard answers web UI to ee
* Hide standard answer controls in bot edit page
* Kwargs for fetch_versioned_implementation
* Add docstring explaining return types for handle_standard_answers
* Consolidate blocks into ee/handle_standard_answers
---------
Co-authored-by: Hyeong Joon Suh <hyeongjoonsuh@Hyeongs-MacBook-Pro.local>
Co-authored-by: danswer-trial <danswer-trial@danswer-trials-MacBook-Pro.local>
* first cut at redis
* some new helper functions for the db
* ignore kombu tables in alembic migrations (used by celery)
* multiline commands for readability, add vespa_metadata_sync queue to worker
* typo fix
* fix returning tuple fields
* add constants
* fix _get_access_for_document
* docstrings!
* fix double function declaration and typing
* fix type hinting
* add a global redis pool
* Add get_document function
* use task_logger in various celery tasks
* add celeryconfig.py to simplify configuration. Will be used in a subsequent commit
* Add celery redis helper. used in a subsequent PR
* kombu warning getting spammy since celery is not self managing its queue in Postgres any more
* add last_modified and last_synced to documents
* fix task naming convention
* use celeryconfig.py
* the big one. adds queues and tasks, updates functions to use the queues with priorities, etc
* change vespa index log line to debug
* mypy fixes
* update alembic migration
* fix fence ordering, rename to "monitor", fix fetch_versioned_implementation call
* mypy
* switch to monotonic time
* fix startup dependencies on redis
* rebase alembic migration
* kombu cleanup - fail silently
* mypy
* add redis_host environment override
* update REDIS_HOST env var in docker-compose.dev.yml
* update the rest of the docker files
* harden indexing-status endpoint against db changes happening in the background. Needs further improvement but OK for now.
* allow no task syncs to run because we create certain objects with no entries but initially marked as out of date
* add back writing to vespa on indexing
* update contributing guide
* backporting fixes from background_deletion
* renaming cache to cache_volume
* add redis password to various deployments
* try setting up pr testing for helm
* fix indent
* hopefully this release version actually exists
* fix command line option to --chart-dirs
* fetch-depth 0
* edit values.yaml
* try setting ct working directory
* bypass testing only on change for now
* move files and lint them
* update helm testing
* some issues suggest using --config works
* add vespa repo
* add postgresql repo
* increase timeout
* try amd64 runner
* fix redis password reference
* add comment to helm chart testing workflow
* rename helm testing workflow to disable it
* adding clarifying comments
* address code review
* missed a file
* remove commented warning ... just not needed
---------
Co-authored-by: Richard Kuo <rkuo@rkuo.com>
* Add user when they interact outside of UI (e.g. Slack bot)
* fix mypy errors
* don't use user manager to avoid async messiness
* fix email is none scenario
* fix mypy
* make code slightly clearer
* PR comments
* get slack email in generate button as well
* fix alembic migration
* update name to be more descriptive
---------
Co-authored-by: Hyeong Joon Suh <hyeongjoonsuh@Hyeongs-MacBook-Pro.local>
* initial commit
* almost done
* finished 3 tests
* minor refactor
* built out initial permisison tests
* reworked test_deletion
* removed logging
* all original tests have been converted
* renamed user_groups to user_group
* mypy
* added test for doc set permissions
* unified naming for manager methods
* Refactored models and added new deletion test
* minor additions
* better logging+fixed input variables
* commented out failed tests
* Added readme
* readme update
* Added auth to IT
set auth_type to basic and require_email_verification to false
* Update run-it.yml
* used verify and added to readme
* added api key manager
* ran bump-pydantic
* replace root_validator with model_validator
* mostly working. some alternate assistant error. changed root_validator and typing_extensions
* working generation chat. changed type
* replacing .dict with .model_dump
* argument needed to bring model_dump up to parity with dict()
* fix a fewremaining issues -- working with llama and gpt
* updating requirements file
* more requirement updates
* more requirement updates
* fix to make search work
* return type fix:
* half way tpyes change
* fixes for mypy and pydantic:
* endpoint fix
* fix pydantic protected namespaces
* it works!
* removed unecessary None initializations
* better logging
* changed default values to empty lists
* mypy fixes
* fixed array defaulting
---------
Co-authored-by: hagen-danswer <hagen@danswer.ai>
* add new user provider hook
* account for additional logic
* add users
* remove is loading
* Curator polish
* useeffect -> provider + effect
* squash
* use use user for user default models
* squash
* Added ability to add users to groups among other things
* final polish
* added connection button to groups
* mypy fix
* Improved document set clarity
* string fixes
---------
Co-authored-by: pablodanswer <pablo@danswer.ai>
* Added backend support for curator role
* modal refactor
* finalized first 2 commits
same as before
finally
what was it for
* added credential, cc_pair, and cleanup
mypy is super helpful hahahahahahahahahahahaha
* curator support for personas
* added connector management permission checks
* fixed the connector creation flow
* added document access to curator
* small cleanup added comments and started ui
* groups and assistant editor
* Persona frontend
* Document set frontend
* cleaned up the entire frontend
* alembic fix
* Minor fixes
* credentials section
* some credential updates
* removed logging statements
* fixed try catch
* fixed model name
* made everything happen in one db commit
* Final cleanup
* cleaned up fast code
* mypy/build fixes
* polish
* more token rate limit polish
* fixed weird credential permissions
* Addressed chris feedback
* addressed pablo feedback
* fixed alembic
* removed deduping and caching
* polish!!!!
