* wip checkpointing/continue on failure
more stuff for checkpointing
Basic implementation
FE stuff
More checkpointing/failure handling
rebase
rebase
initial scaffolding for IT
IT to test checkpointing
Cleanup
cleanup
Fix it
Rebase
Add todo
Fix actions IT
Test more
Pagination + fixes + cleanup
Fix IT networking
fix it
* rebase
* Address misc comments
* Address comments
* Remove unused router
* rebase
* Fix mypy
* Fixes
* fix it
* Fix tests
* Add drop index
* Add retries
* reset lock timeout
* Try hard drop of schema
* Add timeout/retries to downgrade
* rebase
* test
* test
* test
* Close all connections
* test closing idle only
* Fix it
* fix
* try using null pool
* Test
* fix
* rebase
* log
* Fix
* apply null pool
* Fix other test
* Fix quality checks
* Test not using the fixture
* Fix ordering
* fix test
* Change pooling behavior
* better propagation of exceptions up the stack
* remove debug testing
* refactor the watchdog more to emit data consistently at the end of the function
* enumerate a lot more terminal statuses
* handle more codes
* improve logging
* handle "-9"
* single line exception logging
* typo/grammar
---------
Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>
* Fix airtable connector w/ mt cloud + move telem logic to match new standard
* Address Greptile comment
* Small fixes/improvements
* Revert back monitoring frequency
* Small monitoring fix
* feat: add option to treat all non-attachment fields as metadata in Airtable connector
- Added new UI option 'treat_all_non_attachment_fields_as_metadata'
- Updated backend logic to support treating all fields except attachments as metadata
- Added tests for both default and all-metadata behaviors
Co-Authored-By: Chris Weaver <chris@onyx.app>
* fix: handle missing environment variables gracefully in airtable tests
Co-Authored-By: Chris Weaver <chris@onyx.app>
* fix: clean up test file and handle environment variables properly
Co-Authored-By: Chris Weaver <chris@onyx.app>
* fix: add missing test fixture and fix formatting
Co-Authored-By: Chris Weaver <chris@onyx.app>
* chore: fix black formatting
Co-Authored-By: Chris Weaver <chris@onyx.app>
* fix: add type annotation for metadata dict in airtable tests
Co-Authored-By: Chris Weaver <chris@onyx.app>
* fix: add type annotation for mock_get_api_key fixture
Co-Authored-By: Chris Weaver <chris@onyx.app>
* fix: update Generator import to use collections.abc
Co-Authored-By: Chris Weaver <chris@onyx.app>
* refactor: make treat_all_non_attachment_fields_as_metadata a direct required parameter
- Move parameter from connector_config to direct class parameter
- Place parameter right under table_name_or_id argument
- Make parameter required in UI with no default value
- Update tests to use new parameter structure
Co-Authored-By: Chris Weaver <chris@onyx.app>
* chore: fix black formatting
Co-Authored-By: Chris Weaver <chris@onyx.app>
* chore: rename _METADATA_FIELD_TYPES to DEFAULT_METADATA_FIELD_TYPES and clarify usage
Co-Authored-By: Chris Weaver <chris@onyx.app>
* chore: fix black formatting in docstring
Co-Authored-By: Chris Weaver <chris@onyx.app>
* test: make airtable tests fail loudly on missing env vars
Co-Authored-By: Chris Weaver <chris@onyx.app>
* style: fix black formatting in test file
Co-Authored-By: Chris Weaver <chris@onyx.app>
* style: add required newline between test functions
Co-Authored-By: Chris Weaver <chris@onyx.app>
* test: update error message pattern in parameter validation test
Co-Authored-By: Chris Weaver <chris@onyx.app>
* style: fix black formatting in test file
Co-Authored-By: Chris Weaver <chris@onyx.app>
* test: fix error message pattern in parameter validation test
Co-Authored-By: Chris Weaver <chris@onyx.app>
* style: fix line length in test file
Co-Authored-By: Chris Weaver <chris@onyx.app>
* test: simplify error message pattern in parameter validation test
Co-Authored-By: Chris Weaver <chris@onyx.app>
* test: add type validation test for treat_all_non_attachment_fields_as_metadata
Co-Authored-By: Chris Weaver <chris@onyx.app>
* fix: add missing required parameter in test
Co-Authored-By: Chris Weaver <chris@onyx.app>
* fix: remove parameter from test to properly validate it is required
Co-Authored-By: Chris Weaver <chris@onyx.app>
* fix: add type validation for treat_all_non_attachment_fields_as_metadata parameter
Co-Authored-By: Chris Weaver <chris@onyx.app>
* style: fix black formatting in airtable_connector.py
Co-Authored-By: Chris Weaver <chris@onyx.app>
* fix: update type validation test to handle mypy errors
Co-Authored-By: Chris Weaver <chris@onyx.app>
* fix: specify mypy ignore type for call-arg
Co-Authored-By: Chris Weaver <chris@onyx.app>
* Also handle rows w/o sections
* style: fix black formatting in test assertion
Co-Authored-By: Chris Weaver <chris@onyx.app>
* add TODO
* Remove unnecessary check
* Fix test
* Do not break existing airtable connectors
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Weaver <chris@onyx.app>
Co-authored-by: Weves <chrisweaver101@gmail.com>
* Added Permission Syncing for Salesforce
* cleanup
* updated connector doc conversion
* finished salesforce permission syncing
* fixed connector to batch Salesforce queries
* tests!
* k
* Added error handling and check for ee and sync type for postprocessing
* comments
* minor touchups
* tested to work!
* done
* my pie
* lil cleanup
* minor comment
* discord: frontend and backend poll connector
* added requirements for discord installation
* fixed the mypy errors
* process messages not part of any thread
* minor change
* updated the connector; this logic works & am able to docs when i print
* minor change
* ability to enter a start date to pull docs from and refactor
* added the load connector and fixed mypy errors
* local commit test
done!
* minor refactor and properly commented everything
* updated the logic to handle permissions and index active/archived threads
* basic discord test template
* cleanup
* going away with the danswer discord client class ; using an async context manager
* moved to proper folder
* minor fixes
* needs improvement
* fixed discord icon
---------
Co-authored-by: hagen-danswer <hagen@danswer.ai>
