* Fixed indexing when no sites are specificed
* Added test for Sharepoint all sites index
* Accounted for paginated results.
* Typing
* Typing
---------
Co-authored-by: Wenxi Onyx <wenxi-onyx@Wenxis-MacBook-Pro.local>
* Add replies to document construction and edit tests
* Update tests
* Add replies processing to teams
* Fix test
* Add try-except block around potential failure
* Update entity-id during ConnectorFailure raise
* Add basic foundation for teams checkpointing classes
* Fix slack connector main entrypoint
* Saving changes
* Finish teams checkpointing impl
* Remove commented out code
* Remove more unused code
* Move code around
* Add threadpool to process requests in parallel
* Fix mypy errors / warnings
* Move test import to main function only
* Address nits on PR
* Remove unnecessary check prior to entering while-loop
* Remove print statement
* Change exception message
* Address more nits
* Use indexing instead of destructuring
* Add back invocation of `run_with_timeout` instead of a direct call
* Revert slack testing code
* Move early return to before second API call
* Pull fetch to team outside of loop
* Address nits on PR
* Add back client-side filtering
* Updated connector to return after a team's indexing is finished
* Add type ignore
* Implement proper datetime range fetching
* Address comment on PR
* Rename function
* Change exception type when no team with the given id was found
* Address nit on PR
* Add comment on why `page_loaded` is needed to be specified explicitly
* Remove duplicated calls to fetching channels
* Use helper function for thread-based yielding instead of manual logic
* Move datetime filtering to message-level instead
* Address more comments on PR
* Add new utility function for yielding sections
* Add additional utility function
* Add teams tests
* Edit error message
* Address nits on PR
* Promote url-prefix to be a class level constant
* Fix mypy error
* Remove start/end parameters from function that doesn't use them anymore; move around comments
* Address more nits on PR
* Add comment
* Add more logging for confluence perm-sync + handle case where permissions are removed from the access token
* Make required permissions are explicit
* more
* Add slim fetch limit + mark all cc pairs of source type as successful upon group sync
* Add to dev compose
* Small teams fix
* Add file
* Add single limit pagination for confluence
* Restrict to server only
* more logging
* cleanup
* Cleanup
* Remove CONFLUENCE_CONNECTOR_SLIM_FETCH_LIMIT
* Handle teams error
* Fix ut
* Remove db dependency from confluence_doc_sync
* move stuff back to debug
* it will never happen again.
* fix perm sync issue
* fix perm sync issue2
* ensure member emails map is populated
* other fix for perm sync
* address CW comments
* nit
* memory optimize task generation for connector deletion
* test
* fix up integration test docker file
* more no-cache
---------
Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
* tolerance of confluence api weirdness
* remove checkpointing
* remove skipping logic from checkpointing
* add back checkpointing
* switch confluence checkpointing to be based on page starts
* address CW comments and fix unit tests
* some mitigations of bad confluence api
* new checkpointing approach and testing fixes
* fix test
* CW comments
* friendlier handling of slack channel retrieval
* retry on downgrade_postgres deadlock
* fix comment
* text
---------
Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
* tool to generate vespa schema variations for our cloud
* extraneous assign
* use a real templating system instead of search/replace
* fix float
* maybe this should be double
* remove redundant var
* template the other files
* try a spawned process
* move the wrapper
* fix args
* increase timeout
* run multitenant reset operations out of process as well
---------
Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
Co-authored-by: Richard Kuo <rkuo@rkuo.com>
* add emails to retry with on 403
* attempted fix for connector test
* CW comments
* connector test fix
* test fixes and continue on 403
* fix tests
* fix tests
* fix concurrency tests
* fix integration tests with llmprovider eager loading
* Add multi text array field
* Add multiple values to model configuration for a custom LLM provider
* Fix reference to old field name
* Add migration
* Update all instances of model_names / display_model_names to use new schema migration
* Update background task
* Update endpoints to not throw errors
* Add test
* Update backend/alembic/versions/7a70b7664e37_add_models_configuration_table.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update backend/onyx/background/celery/tasks/llm_model_update/tasks.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Fix list comprehension nits
* Update web/src/components/admin/connectors/Field.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update web/src/app/admin/configuration/llm/interfaces.ts
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Implement greptile recommendations
* Update backend/onyx/db/llm.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update backend/onyx/server/manage/llm/api.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update backend/onyx/background/celery/tasks/llm_model_update/tasks.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update backend/onyx/db/llm.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Fix more greptile suggestions
* Run formatter again
* Update backend/onyx/db/models.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Add relationship to `LLMProvider` and `ModelConfigurations` classes
* Use sqlalchemy ORM relationships instead of manually populating fields
* Upgrade migration
* Update interface
* Remove all instances of model_names and display_model_names from backend
* Add more tests and fix bugs
* Run prettier
* Add types
* Update migration to perform data transformation
* Ensure native llm providers don't have custom max input tokens
* Start updating frontend logic to support custom max input tokens
* Pass max input tokens to LLM class (to be passed into `litellm.completion` call later)
* Add ModelConfigurationField component for custom llm providers
* Edit spacing and styling of model configuration matrix
* Fix error message displaying bug
* Edit opacity of `FiX` field for first index
* Change opacity back
* Change roundness
* Address comments on PR
* Perform fetching of `max_input_tokens` at the beginning of the callgraph and rope it throughout the entire callstack
* Change `add` to `execute`
* Move `max_input_tokens` into `LLMConfig`
* Fix bug with error messages not being cleared
* Change field used to fetch LLMProvider
* Fix model-configuration UI
* Address comments
* Remove circular import
* Fix failing tests in GH
* Fix failing tests
* Use `isSubset` instead of equality to determine native vs custom LLM Provider
* Remove unused import
* Make responses always display max_input_tokens
* Fix api endpoint to hit
* Update types in web application
* Update object field
* Fix more type errors
* Fix failing llm provider tests
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* tool to generate vespa schema variations for our cloud
* extraneous assign
* use a real templating system instead of search/replace
* fix float
* maybe this should be double
* remove redundant var
* template the other files
* try a spawned process
* move the wrapper
* fix args
* increase timeout
---------
Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
Co-authored-by: Richard Kuo <rkuo@rkuo.com>
