* add utility function
* add utility functions to DocExternalAccess
* refactor db access out of individual celery tasks and put it directly into the heavy task
* code review and remove leftovers
* fix circular imports
---------
Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
* don't hardcode -1
* extra spaces
* fix binary data in blurb
* add note to binary handling
---------
Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
* friendlier handling of slack channel retrieval
* retry on downgrade_postgres deadlock
* fix comment
* text
---------
Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
* Return default value instead of throwing error
* Add default parameter
* Move logic around
* Use dummy value for max_input_tokens in testing flow
* Remove unnecessary assignment
* Add multi text array field
* Add multiple values to model configuration for a custom LLM provider
* Fix reference to old field name
* Add migration
* Update all instances of model_names / display_model_names to use new schema migration
* Update background task
* Update endpoints to not throw errors
* Add test
* Update backend/alembic/versions/7a70b7664e37_add_models_configuration_table.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update backend/onyx/background/celery/tasks/llm_model_update/tasks.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Fix list comprehension nits
* Update web/src/components/admin/connectors/Field.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update web/src/app/admin/configuration/llm/interfaces.ts
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Implement greptile recommendations
* Update backend/onyx/db/llm.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update backend/onyx/server/manage/llm/api.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update backend/onyx/background/celery/tasks/llm_model_update/tasks.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update backend/onyx/db/llm.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Fix more greptile suggestions
* Run formatter again
* Update backend/onyx/db/models.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Add relationship to `LLMProvider` and `ModelConfigurations` classes
* Use sqlalchemy ORM relationships instead of manually populating fields
* Upgrade migration
* Update interface
* Remove all instances of model_names and display_model_names from backend
* Add more tests and fix bugs
* Run prettier
* Add types
* Update migration to perform data transformation
* Ensure native llm providers don't have custom max input tokens
* Start updating frontend logic to support custom max input tokens
* Pass max input tokens to LLM class (to be passed into `litellm.completion` call later)
* Add ModelConfigurationField component for custom llm providers
* Edit spacing and styling of model configuration matrix
* Fix error message displaying bug
* Edit opacity of `FiX` field for first index
* Change opacity back
* Change roundness
* Address comments on PR
* Perform fetching of `max_input_tokens` at the beginning of the callgraph and rope it throughout the entire callstack
* Change `add` to `execute`
* Move `max_input_tokens` into `LLMConfig`
* Fix bug with error messages not being cleared
* Change field used to fetch LLMProvider
* Fix model-configuration UI
* Address comments
* Remove circular import
* Fix failing tests in GH
* Fix failing tests
* Use `isSubset` instead of equality to determine native vs custom LLM Provider
* Remove unused import
* Make responses always display max_input_tokens
* Fix api endpoint to hit
* Update types in web application
* Update object field
* Fix more type errors
* Fix failing llm provider tests
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* bump fastapi and starlette
* bumping llama index and nltk and associated deps
* bump to fix python-multipart
* bump aiohttp
* update package lock for examples/widget
* bump black
* sentencesplitter has changed namespaces
* fix reorder import check, fix missing passlib
* update package-lock.json
* black formatter updated
* reformatted again
* change to black compatible reorder
* change to black compatible reorder-python-imports fork
* fix pytest dependency
* black format again
* we don't need cdk.txt. update packages to be consistent across all packages
---------
Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
Co-authored-by: Richard Kuo <rkuo@rkuo.com>
* add prometheus metrics endpoints via helper package
* model server specific requirements
* mark as public endpoint
---------
Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
* sanitize llm keys and handle updates properly
* fix llm provider testing
* fix test
* mypy
* fix default model editing
---------
Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>
Co-authored-by: Richard Kuo <rkuo@rkuo.com>
* work in progress
* work in progress
* WIP
* refactor, use inline attachment for image (base64 encoding doesn't work)
* pretty sure this belongs behind a multi_tenant check
* code review / refactor
---------
Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>
* remove title for slack
* initial working code
* simplification
* improvements
* name change to information_content_model
* avoid boost_score > 1.0
* nit
* EL comments and improvements
Improvements:
- proper import of information content model from cache or HF
- warm up for information content model
Other:
- EL PR review comments
* nit
* requirements version update
* fixed docker file
* new home for model_server configs
* default off
* small updates
* YS comments - pt 1
* renaming to chunk_boost & chunk table def
* saving and deleting chunk stats in new table
* saving and updating chunk stats
* improved dict score update
* create columns for individual boost factors
* RK comments
* Update migration
* manual import reordering
* rename agent test script to prevent pytest autodiscovery
* first cut
* fix log message
* fix up typing
* add a sample test
---------
Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>
* functional initial auth modal
* k
* k
* k
* looking good
* k
* k
* k
* k
* update
* k
* k
* misc bunch
* improvements
* k
* address comments
* k
* nit
* update
* k
A new setting 'is_ephemeral' has been added to the Slack channel configurations.
Key features/effects:
- if is_ephemeral is set for standard channel (and a Search Assistant is chosen):
- the answer is only shown to user as an ephemeral message
- the user has access to his private documents for a search (as the answer is only shown to them)
- the user has the ability to share the answer with the channel or keep private
- a recipient list cannot be defined if the channel is set up as ephemeral
- if is_ephemeral is set and DM with bot:
- the user has access to private docs in searches
- the message is not sent as ephemeral, as it is a 1:1 discussion with bot
- if is_ephemeral is not set but recipient list is set:
- the user search does *not* have access to their private documents as the information goes to the recipient list team members, and they may have different access rights
- Overall:
- Unless the channel is set to is_ephemeral or it is a direct conversation with the Bot, only public docs are accessible
- The ACL is never bypassed, also not in cases where the admin explicitly attached a document set to the bot config.
* first cut at slack oauth flow
* fix usage of hooks
* fix button spacing
* add additional error logging
* no dev redirect
* early cut at google drive oauth
* second pass
* switch to production uri's
* try handling oauth_interactive differently
* pass through client id and secret if uploaded
* fix call
* fix test
* temporarily disable check for testing
* Revert "temporarily disable check for testing"
This reverts commit 4b5a022a5f.
* support visibility in test
* missed file
* first cut at confluence oauth
* work in progress
* work in progress
* work in progress
* work in progress
* work in progress
* first cut at distributed locking
* WIP to make test work
* add some dev mode affordances and gate usage of redis behind dynamic credentials
* mypy and credentials provider fixes
* WIP
* fix created at
* fix setting initialValue on everything
* remove debugging, fix ??? some TextFormField issues
* npm fixes
* comment cleanup
* fix comments
* pin the size of the card section
* more review fixes
* more fixes
---------
Co-authored-by: Richard Kuo <rkuo@rkuo.com>
Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>
