16 Commits

Author	SHA1	Message	Date
pablonyx	eff433bdc5	Reduce errors in workers (#3962)	2025-02-13 15:59:44 -08:00
rkuo-danswer	61058e5fcd	merge monitoring with kickoff tasks (#3953) ... * move indexing * all monitor work moved * reacquire lock more * remove monitor task completely * fix import * fix pruning finalization * no multiplier on system/cloud tasks * monitor queues every 30 seconds in the cloud --------- Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>	2025-02-13 02:35:41 +00:00
rkuo-danswer	bd08e6d787	alert if revisions are null or query fails (#3910) ... * alert if revisions are null or query fails * comment * mypy --------- Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>	2025-02-05 23:45:38 +00:00
pablodanswer	42a0f45a96	update	2025-02-04 12:06:11 -08:00
pablodanswer	125e5eaab1	various mypy improvements	2025-02-04 12:06:10 -08:00
Chris Weaver	a5d2f0d9ac	Fix airtable connector w/ mt cloud + move telem logic to match new st… (#3868) ... * Fix airtable connector w/ mt cloud + move telem logic to match new standard * Address Greptile comment * Small fixes/improvements * Revert back monitoring frequency * Small monitoring fix	2025-01-31 16:29:04 -08:00
Chris Weaver	95701db1bd	Add more sync records + fix small bug in monitoring task causing deletion metrics to never be emitted (#3837) ... Double check we don't double-emit + fix pruning metric Add log Fix comment rename	2025-01-29 18:03:49 -08:00
rkuo-danswer	4fe99d05fd	add timings for syncing (#3798) ... * add timings for syncing * add more logging * more debugging * refactor multipass/db check out of VespaIndex * circular imports? * more debugging * add logs * various improvements * additional logs to narrow down issue * use global httpx pool for the main vespa flows in celery. Use in more places eventually. * cleanup debug logging, etc * remove debug logging * this should use the secondary index * mypy * missed some logging * review fixes * refactor get_default_document_index to use search settings * more missed logging * fix circular refs --------- Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app> Co-authored-by: pablodanswer <pablo@danswer.ai>	2025-01-29 23:24:44 +00:00
rkuo-danswer	fa78f50fe3	Bugfix/celery ignore result (#3770) ... * try using a redis replica in some areas * harden up replica usage * ignore results --------- Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>	2025-01-27 08:53:01 +00:00
pablonyx	70795a4047	Sync status improvements (#3782) ... * minor improvments / clarity * additional comment for clarity * typing * quick updates to monitoring * connector deletion * quick nit * fix typing * update values * quick nit * functioning * improvements to monitoring * update * minutes -> seconds	2025-01-26 17:35:26 +00:00
pablonyx	ccb16b7484	Indexing latency check fix (#3747) ... * add logs + update dev script * update conig * remove prints * temporarily turn off * va * update * fix * finalize monitoring updates * update	2025-01-23 17:14:26 +00:00
rkuo-danswer	69c60feda4	cloud check for migrations (#3734) ... * cloud check for migrations * fix table declaration * change back interval * Fix usage of POSTGRES_DEFAULT_SCHEMA --------- Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>	2025-01-22 22:41:28 +00:00
Weves	a72bd31f5d	Small background telemetry fix	2025-01-18 16:19:28 -08:00
Chris Weaver	a05addec19	Add is_cloud info to telemetry + get consistent customer_uuid's for a… (#3684) ... * Add is_cloud info to telemetry + get consistent customer_uuid's for a given tenant * Address Richard's comments	2025-01-16 02:43:21 +00:00
Chris Weaver	3b7695539f	Add monitoring worker (#3677) ... * Add monitoring worker * Add locks * Add tenant id to lock * Remove unneeded tenant postfix	2025-01-15 01:39:56 +00:00
Chris Weaver	9ea2ae267e	Performance monitoring (#3658) ... * Initial scaffolding for metrics * iterate * more * More metrics + SyncRecord concept * Add indices, standardize timing * Small cleanup * Address comments	2025-01-13 12:36:45 -08:00