joachim-danswer
c5adbe4180
Knowledge Graph v1 ( #4626 )
...
* db setup
* transfer 1 - incomplete
* more adjustments
* relationship table + query update
* temp view creation
* restructuring
* nits
* updates
* separate read_only engine
* extraction revamp
* focus on metadata relatonships 1
* dev
* migration downgrade fix
* rebase migration change
* a3+
* progress
* base
* new extraction
* progress
* fixed KG extraction
* nits
* updates
* simplifications & cleanup
* fixes
* updates
* more feature flag checks
* fixes
* extraction process fix
* read-only user creation as part of setup
* fix for missing entity attributes
* kg read-only user creation as part of migration
* typo
* EL initial comments
* initial Account/SF Connector chnges
* SF Connector update
- include account information
* base w/ salesforce
* evan updates + quite a bit more
* kg-filtered search
* EL changes pt 2
* migrations and env vars
* quick migration fix
* migration update
* post_rebase fixes
* mypy fixes
* test fixes
* test fix
* test fix
* read_only pool + misc
* nf
* env vars
* test improvements
* salesforce fix
* test update
* small changes
* small adjustments
* SF Connector fix & kg_stage removal for one table
* mypy fix
* small fixes
* EL + RK (pt 1) comments
* nit
* setting updated
* Salesforce test update
* EL comments
* read-only user replacement & cleanup
* SQL View fix
* converting entity type-name separators
* sql view group ownership
* view fix
* SQL tweak
* dealing with docs that were skipped by indexing
* increased error handling
* more error handling
* Output formatting fix
* kg-incremental-reindexing
* 0-doc found improvement
* celery
* migration correction
* timeout adjustments
* nit
* Updated migration
* Entity Normalization for KG Dev 1 (#4746 )
* feat: trigrams column
* fix: reranking and db
* feat: v1
* fix: convert to orm
* feat: parallel
* fix: default to id_name
* fix: renamed semantic_id and semantic_id_trigrams
* fix: scalar subquery
* fix: tuning + redundancy
* fix: threshold
* fix: typo
* fix: shorten names
* wip
* fix: reverted
* feat: config
* feat: works but it was dumb
* feat: clustering works
* fix: mypy
* normalization <-> language awareness for SQL generation
* small type fixes
---------
Co-authored-by: joachim-danswer <joachim@danswer.ai >
* mypy
* typo and dead code
* kg_time_fencing
* feat: remove temp views on migration downgrade
* remove functions and triggers for now
* rebase adjustments
* EL code review results
* quick fix + trigger/funcs for single tenant
* fix: typo, mypy, dead code
* fix: autoflake
* small updatesd
* nit
* fix: typo
* early + faster view creation
* Extension creation in MT migration
* nit changes to default ETs
* Incremental Clustering and KG Refactor V1 (#4784 )
Optimized/restructured incremental clustering. New pipeline actually that moves vespa updates to clustering.
Also, celery configuration has been updated.
---------
Co-authored-by: joachim-danswer <joachim@danswer.ai >
* prompt tweak & ET extraction reset
* more general hierarchical structure
* feat: better vespa reset logic
* prompt optimization and entity replacemants
* small prompt changes
* KG Refactor V2 (#4814 )
Clustering & Extraction improvements & various nits
Co-authored-by: joachim-danswer <joachim@danswer.ai >
* add connector-level coverage days
* fix: nit
* initial EL responses
* refactor: helper functions for formatting
* fix: more helper fns & comments
* fix: comment code that's been implemented elsewhere
* fix: tenant_id missing arg
* fix: removed debugging stuff
* fix: moved kg_interactions db query to helper fn
* fix: tenant_id
* fix: tenant_id & removed outdated helper fn
* fix always set entity class
* fix: typo
* fix alembic heads
* fix: celery logging
* fix: migrations fix
* fix: multi tenant permissions
* fix: temp connector fix
* fix: downgrade
* Fix upgrade migration
* fix: tenant for normalization
* added additional acl
* stray EL comments
* fix: connector test
* fix mypy
* fix: temporary connector test fix
* fix: jira connector test
* nit
* small nits
* fix: black
* fix: mypy
* fix: mypy
---------
Co-authored-by: Rei Meguro <36625832+Orbital-Web@users.noreply.github.com >
2025-06-07 23:14:20 +00:00
rkuo-danswer
51be9000bb
Feature/vespa bump ( #4819 )
...
* bump cloudformation
* update kubernetes
* bump helm chart
* bump docker compose
* update chart.lock
* ai accident!
* bump vespa helm chart for fix
* increase timeout
---------
Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app >
2025-06-04 00:03:01 +00:00
Evan Lohn
4c71a5f5ff
drive perm sync logs + misc deployment improvements ( #4788 )
...
* some logs
* give postgress more memory
* give postgress more memory
* give postgress more memory
* revert
* give postgress more memory
* bump external access limit
* vespa timeout
* deployment consistency
* bump vespa version
* skip upgrade check
* retry permission by ids
* logs
* fix temp docx file issue
* fix drive file deduping
* RK comments
* mypy
* aggregate logs
2025-06-01 23:36:57 +00:00
rkuo-danswer
ab7a4184df
Feature/helm k8s probes 2 ( #4766 )
...
* add probes
* lint fixes
* add beat probes
---------
Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app >
2025-05-28 05:20:24 +00:00
rkuo-danswer
e822afdcfa
add probes ( #4762 )
...
* add probes
* lint fixes
---------
Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app >
2025-05-23 02:24:54 +00:00
Chris Weaver
95ae6d300c
Fix slack bot kubernetes template ( #4734 )
...
* Fix slack path for kubernetes files
* Add env variables
2025-05-19 21:25:57 +00:00
rkuo-danswer
2282f6a42e
fix restart ( #4726 )
...
Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app >
2025-05-17 17:02:27 +00:00
Richard Kuo (Onyx)
daa1746b4a
just readme fixes
2025-05-13 09:56:07 -07:00
rkuo-danswer
d8068f0a68
Feature/helm separate workers ( #4679 )
...
* add test
* try breaking out background workers
* fix helm lint complaints
* rename disabled files more
* try different folder structure
* fix beat selector
* vespa setup should break on success
* improved instructions for basic helm chart testing
---------
Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app >
2025-05-13 02:23:32 +00:00
rkuo-danswer
7965fd9cbb
run testing ( #4681 )
...
* run testing
* need to break on success
* add a readme
* raise vespa to 6GB
* allow test to retry
* add 20 attempts
* put memory limits back to normal
* restore chart testing on changes only
* increase retries to 40
---------
Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app >
2025-05-09 11:49:43 -07:00
Chris Weaver
70df685709
Non default schema fix ( #4667 )
...
* Use correct postgres schema
* Remove raw Session() use
* Refactor + add test
* Fix comment
2025-05-06 20:35:59 -07:00
evan-danswer
dc62d83a06
File connector tests ( #4561 )
...
* danswer to onyx plus tests for file connector
* actually add test
2025-04-19 15:54:30 -07:00
Weves
c3ffaa19a4
Small no-letsencrypt improvement
2025-04-15 18:29:07 -07:00
Weves
72ce2f75cc
Add env var to docker compose file
2025-04-13 23:14:06 -07:00
Chris Weaver
e3aab8e85e
Improve index attempt display ( #4511 )
2025-04-13 15:57:47 -07:00
Chris Weaver
89688f0cef
Fix naming of volume ( #4491 )
2025-04-09 23:46:10 +00:00
Weves
614d0f8d72
Add more options to dev compose file
2025-04-07 10:03:34 -07:00
Weves
cf6ff3ce4a
Fix run-nginx
2025-04-05 16:00:10 -07:00
Weves
86d9f5d9dd
Update resource limits
2025-04-05 16:00:10 -07:00
Weves
c04c1ea31b
Fix onyx_config.jsonl
2025-04-03 22:44:56 -07:00
Chris Weaver
2380c2266c
Infra and Deployment for ECS Fargate ( #4449 )
...
* Infra and Deployment for ECS Fargate
---------
Co-authored-by: jpb80 <jordan.buttkevitz@gmail.com >
2025-04-03 22:43:56 -07:00
Chris Weaver
7648627503
Save all logs + add log persistence to most Onyx-owned containers ( #4368 )
...
* Save all logs + add log persistence to most Onyx-owned containers
* Separate volumes for each container
* Small fixes
2025-03-26 22:25:39 -07:00
Tim Rosenblatt
0f6cf0fc58
Fixes docker logs helper text in run-nginx.sh ( #3678 )
...
The docker container name is slightly wrong, and this commit fixes it.
2025-03-26 09:03:35 -07:00
Chris Weaver
0d3c72acbf
Add basic memory logging ( #4234 )
...
* Add basic memory logging
* Small tweaks
* Switch to monotonic
2025-03-08 03:49:47 +00:00
rkuo-danswer
2c0af0a0ca
Feature/helm updates ( #4201 )
...
* add ingress for api and web
* helm setup docs
* add letsencrypt. close blocks
* use pathType ImplementationSpecific as Prefix is deprecated
* fix backend labels. configure nginx routes. update annotations
* fix linting
---------
Co-authored-by: Sajjad Anwar <sajjadkm@gmail.com >
Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app >
2025-03-06 19:48:20 +00:00
Chris Weaver
f731beca1f
Add ONYX_QUERY_HISTORY_TYPE to the dev compose files ( #4196 )
2025-03-05 17:34:55 +00:00
rkuo-danswer
60bd9271f7
Bugfix/model tests ( #4092 )
...
* trying out a fix
* add ability to manually run model tests
* add log dump
* check status code, not text?
* just the model server
* add port mapping to host
* pass through more api keys
* add azure tests
* fix litellm env vars
* fix env vars in github workflow
* temp disable litellm test
---------
Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app >
2025-02-25 04:53:51 +00:00
Weves
bdaa293ae4
Fix nginx for prod compose file
2025-02-21 16:57:54 -08:00
pablonyx
5a131f4547
Fix integration tests ( #4059 )
2025-02-21 15:56:11 -08:00
Chris Weaver
ad4efb5f20
Pin xmlsec version + improve SAML flow ( #4054 )
...
* Pin xmlsec version
* testing
* test nginx conf change
* Pass through more
* Cleanup + remove DOMAIN across the board
2025-02-19 16:02:05 -08:00
Chris Weaver
f1fc8ac19b
Connector checkpointing ( #3876 )
...
* wip checkpointing/continue on failure
more stuff for checkpointing
Basic implementation
FE stuff
More checkpointing/failure handling
rebase
rebase
initial scaffolding for IT
IT to test checkpointing
Cleanup
cleanup
Fix it
Rebase
Add todo
Fix actions IT
Test more
Pagination + fixes + cleanup
Fix IT networking
fix it
* rebase
* Address misc comments
* Address comments
* Remove unused router
* rebase
* Fix mypy
* Fixes
* fix it
* Fix tests
* Add drop index
* Add retries
* reset lock timeout
* Try hard drop of schema
* Add timeout/retries to downgrade
* rebase
* test
* test
* test
* Close all connections
* test closing idle only
* Fix it
* fix
* try using null pool
* Test
* fix
* rebase
* log
* Fix
* apply null pool
* Fix other test
* Fix quality checks
* Test not using the fixture
* Fix ordering
* fix test
* Change pooling behavior
2025-02-16 02:34:39 +00:00
Yuhong Sun
ab8081c36b
k
2025-02-15 13:42:43 -08:00
pablonyx
48ac690a70
Multi tenant tests ( #3919 )
...
* ensure fail on multi tenant successfully
* attempted fix
* udpate ingration tests
* minor update
* improve
* improve workflow
* fix migrations
* many more logs
* quick fix
* improve
* fix typo
* quick nit
* attempted fix
* very minor clean up
2025-02-07 01:24:00 +00:00
trial-danswer
2fc41cd5df
Helm Chart Fixes ( #3900 )
...
* initial commit for helm chart refactoring
* Continue refactoring helm. I was able to use helm to deploy all of the apps to a cluster in aws. The bottleneck was setting up PVC dynamic provisioning.
* use default storage class
* Fix linter errors
* Fix broken helm test
* update
* Helm chart fixes
* remove reference to ebsstorage
* Fix linter errors
---------
Co-authored-by: jpb80 <jordan.buttkevitz@gmail.com >
2025-02-06 10:41:09 -08:00
trial-danswer
715359c120
Helm chart refactoring ( #3797 )
...
* initial commit for helm chart refactoring
* Continue refactoring helm. I was able to use helm to deploy all of the apps to a cluster in aws. The bottleneck was setting up PVC dynamic provisioning.
* use default storage class
* Fix linter errors
* Fix broken helm test
---------
Co-authored-by: jpb80 <jordan.buttkevitz@gmail.com >
2025-02-03 10:56:07 -08:00
pablodanswer
6b31e2f622
remove cloud_kubes from public repo
2025-01-30 09:52:57 -08:00
Chris Weaver
3b7695539f
Add monitoring worker ( #3677 )
...
* Add monitoring worker
* Add locks
* Add tenant id to lock
* Remove unneeded tenant postfix
2025-01-15 01:39:56 +00:00
Weves
af2061c4db
Add Linear OAuth env variables to dev compose
2025-01-04 16:02:04 -08:00
Chris Weaver
d64464ca7c
Add support for OAuth connectors that require user input ( #3571 )
...
* Add support for OAuth connectors that require user input
* Cleanup
* Fix linear
* Small re-naming
* Remove console.log
2024-12-31 18:03:33 -08:00
pablonyx
e2700b2bbd
Remove left over yaml errors ( #3527 )
...
* remove left over vim command
* additional misconfigurations
* ensure all regions updated
2024-12-29 01:45:07 +00:00
pablonyx
64b6f15e95
AWS extraneous error fix ( #3529 )
...
* remove left over vim command
* aws fix
* k
* remove double
2024-12-20 19:31:04 +00:00
pablonyx
aa4bfa2a78
Forgot password feature ( #3437 )
...
* forgot password feature
* improved config
* nit
* nit
2024-12-20 04:53:37 +00:00
Chris Weaver
23ecf654ed
Add support for custom LLM error messages ( #3501 )
...
* Add support for custom LLM error messages
* Fix mypy
2024-12-17 22:58:17 -08:00
pablonyx
8db6d49fe5
IAM Auth for RDS ( #3479 )
...
* k
* functional iam auth
* k
* k
* improve typing
* add deployment options
* cleanup
* quick clean up
* minor cleanup
* additional clarity for db session operations
* nit
* k
* k
* update configs
* docker compose spacing
2024-12-17 22:02:37 +00:00
pablodanswer
c85157f734
k
2024-12-13 12:48:50 -08:00
pablodanswer
a6ab8a8da4
organize fe tests
2024-12-13 12:06:26 -08:00
pablodanswer
0ba77963c4
update nit references
2024-12-13 11:49:27 -08:00
pablodanswer
21ec5ed795
welcome to onyx
2024-12-13 09:56:10 -08:00
pablonyx
9cacb373ef
let users specify resourcing caps ( #3403 )
...
* let users specify resourcing caps
* functioanl resource limits
* improve defaults
* k
* update
* update comment + refer to proper resource
* self nit
* update var names
2024-12-12 00:59:41 +00:00
rkuo-danswer
c667d28e7a
update helm charts for onyx-dot-app rebrand ( #3412 )
...
* update helm charts for onyx-dot-app rebrand
* fix helm chart testing config
---------
Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app >
2024-12-11 18:08:39 +00:00
