1487 Commits

Author SHA1 Message Date
hagen-danswer
a96cea2ce0 logging improvements 2024-09-01 16:21:35 -07:00
hagen-danswer
8d443ada5b
Integration tests (#2256)
* initial commit

* almost done

* finished 3 tests

* minor refactor

* built out initial permisison tests

* reworked test_deletion

* removed logging

* all original tests have been converted

* renamed user_groups to user_group

* mypy

* added test for doc set permissions

* unified naming for manager methods

* Refactored models and added new deletion test

* minor additions

* better logging+fixed input variables

* commented out failed tests

* Added readme

* readme update

* Added auth to IT

set auth_type to basic and require_email_verification to false

* Update run-it.yml

* used verify and added to readme

* added api key manager
2024-09-01 22:21:00 +00:00
Yuhong Sun
580848cf8c
mypy (#2283) 2024-08-30 18:02:18 -07:00
Yuhong Sun
f01027cfb7
Catch LLM Eval Failures (#2272) 2024-08-30 17:42:58 -07:00
pablodanswer
76db4b765a
Detect GPU on startup for default multi-pass indexing value (#2242) 2024-08-30 17:38:31 -07:00
pablodanswer
5800c7158e
Add typing to pdf extraction (#2280) 2024-08-30 17:16:56 -07:00
Weves
21af852073 Add connector creation docs 2024-08-30 16:43:42 -07:00
pablodanswer
4181124e7a
add metadata to pdf extraction (#2278) 2024-08-30 15:14:02 -07:00
pablodanswer
f2e8680955
Account for edge case in indexing times with connectors #2190 (#2190) 2024-08-30 14:07:07 -07:00
pablodanswer
b952dbef42
Minor search formatting updates (#2276) 2024-08-30 14:02:35 -07:00
pablodanswer
183569061b
Minor search UX improvements + Critical connector fixes (#2259) 2024-08-30 11:47:52 -07:00
hagen-danswer
1734a4a18c
Added DanswerBot response limit environment variables (#2266)
* Added DanswerBot response limit environment variables

* mypy fix

* changed defaults
2024-08-29 19:25:11 +00:00
rkuo-danswer
766652de14
ignore kombu tables used by celery in alembic (#2261) 2024-08-29 18:49:35 +00:00
pablodanswer
00fa36d591
Get accurate model output max (#2260)
* get accurate model output max

* squash

* udpated max default tokens

* rename + use fallbacks

* functional

* remove max tokens

* update naming

* comment out function to prevent mypy issues
2024-08-29 18:01:56 +00:00
pablodanswer
3b596fd6a8
Default rerank API key to None (new Pydantic compatibility) (#2258)
* default to None

* rm
2024-08-28 16:02:06 +00:00
hagen-danswer
e4e67c61ef
Some additional curator polish (#2253) 2024-08-28 02:44:24 +00:00
Chris Weaver
8afa53c6bf
Confluence improvements (#2248)
* Confluence improvements

* Improve CONFLUENCE_CONNECTOR_INDEX_ONLY_ACTIVE_PAGES
2024-08-28 02:16:10 +00:00
Yuhong Sun
1e67332078
Remove warning on user signup (#2252) 2024-08-27 18:49:05 -07:00
josvdw
50c17438d5
Litellm bump (#2195)
* ran bump-pydantic

* replace root_validator with model_validator

* mostly working. some alternate assistant error. changed root_validator and typing_extensions

* working generation chat. changed type

* replacing .dict with .model_dump

* argument needed to bring model_dump up to parity with dict()

* fix a fewremaining issues -- working with llama and gpt

* updating requirements file

* more requirement updates

* more requirement updates

* fix to make search work

* return type fix:

* half way tpyes change

* fixes for mypy and pydantic:

* endpoint fix

* fix pydantic protected namespaces

* it works!

* removed unecessary None initializations

* better logging

* changed default values to empty lists

* mypy fixes

* fixed array defaulting

---------

Co-authored-by: hagen-danswer <hagen@danswer.ai>
2024-08-28 00:00:27 +00:00
Yuhong Sun
657d2050a5
Confluence Internal Error Handling (#2247) 2024-08-27 15:23:02 -07:00
Yuhong Sun
3640d0c550
Better Web Connector Logging (#2246) 2024-08-27 15:06:24 -07:00
Chris Weaver
8614cd8934
Handle missing email more gracefully (#2244) 2024-08-27 20:29:25 +00:00
pablodanswer
97ba71e1b3
Db search (#2235)
* k

* update enum imports

* add functional types + model swaps

* remove a log

* remove kv

* fully functional + robustified for kv swap

* validated with hosted + cloud

* ensure not updating current search settings when reindexing

* add instance check

* revert back to updating search settings (will need a slight refactor for endpoint)

* protect advanced config override1

* run pretty

* fix typing

* update typing

* remove unnecessary function

* update model name

* clearer interface names

* validated foreign key constaint

* proper migration

* squash

---------

Co-authored-by: Yuhong Sun <yuhongsun96@gmail.com>
2024-08-27 04:26:51 +00:00
Chris Weaver
a873fc6483
Fix Confluence freezing (#2239) 2024-08-26 19:44:01 -07:00
Chris Weaver
c0e1a02e8e
Add it on merge queue (#2112)
* Github action to run integration tests

* Improve

* Fix build

* Add pull

* Fix readiness script

* Add IT runner

* Add IT runner

* Add logs

* update

* Fix

* Fix path

* file path

* test

* fix

* fix

* fix

* test

* network

* fix

* cleanup

* fix

* test

* Fix downgrade

* Add OpenAI API key

* Add VESPA_HOST

* test pulling first

* Add API server host

* Cache tweak

* Fix pull/push settings:

* Stop pushing to latest tag

* test cache change

* test

* test

* test

* remove cache temporarily

* Fix

* Enable EE

* test

* Remove duplicate funcs

* add back build

* Update all

* Fix stop cmd

* Add to merge queue

* Cleanup image tag
2024-08-26 07:20:28 +00:00
hagen-danswer
205c3c3fc8
Combined the get document set endpoints (#2234)
* Combined the get document set endpoints

* removed unused function

* fixed permissioning for document sets
2024-08-25 19:02:27 +00:00
hagen-danswer
c21b0ee3f5
Curator polish (#2229)
* add new user provider hook

* account for additional logic

* add users

* remove is loading

* Curator polish

* useeffect -> provider + effect

* squash

* use use user for user default models

* squash

* Added ability to add users to groups among other things

* final polish

* added connection button to groups

* mypy fix

* Improved document set clarity

* string fixes

---------

Co-authored-by: pablodanswer <pablo@danswer.ai>
2024-08-25 01:10:24 +00:00
hagen-danswer
c1c35b00cb
Fixed slack bot auto filters for document sets (#2231) 2024-08-24 18:54:21 +00:00
Weves
7d201f67d4 Fix typing for custom tool response 2024-08-23 13:34:53 -07:00
pablodanswer
c152123ef4
alembic once again (#2221) 2024-08-23 05:28:13 +00:00
Chris Weaver
5cb9c17ddf
Add better logging for connectors (#2219)
* Add better logging for connectors

* fix
2024-08-23 03:58:29 +00:00
Chris Weaver
b1302303b2
Add chat_session_id + message_pair_num (#2220) 2024-08-22 20:55:21 -07:00
pablodanswer
e89dc67e5d
Update embedding interface (#2205)
* squash

* simplify interface

* some updates to typing

* cloud provider type

* update typing to be even clearer

* push local commit (squash)

* cleaner interfaces

* another quick pass

* squash

* cleaner alembic

* cleaner

* remove trailing whitespace

* add sequence

* quick circle back to double check

* update

* update naming

* update naming
2024-08-23 03:52:02 +00:00
hagen-danswer
c042a19c00
Curator role (#2166)
* Added backend support for curator role

* modal refactor

* finalized first 2 commits

same as before

finally

what was it for

* added credential, cc_pair, and cleanup

mypy is super helpful hahahahahahahahahahahaha

* curator support for personas

* added connector management permission checks

* fixed the connector creation flow

* added document access to curator

* small cleanup added comments and started ui

* groups and assistant editor

* Persona frontend

* Document set frontend

* cleaned up the entire frontend

* alembic fix

* Minor fixes

* credentials section

* some credential updates

* removed logging statements

* fixed try catch

* fixed model name

* made everything happen in one db commit

* Final cleanup

* cleaned up fast code

* mypy/build fixes

* polish

* more token rate limit polish

* fixed weird credential permissions

* Addressed chris feedback

* addressed pablo feedback

* fixed alembic

* removed deduping and caching

* polish!!!!
2024-08-23 01:39:37 +00:00
Chris Weaver
99db27d989
Add metadata for simple doc (#2212) 2024-08-22 12:30:28 -07:00
pablodanswer
197b62aed1
Regenerate (branch of stop) (#2157)
* add regenerate

* functional once again post rebase but quite ugly

* validated + cleaner UI

* more robust implementation for first messages

* squash

* remove parameter

* proper margin

* clarify for future programmers

* remove some logs

* self nit pick - smoother ux

* more self-nits

* stroke line cap

* rebase
2024-08-22 19:06:44 +00:00
Yuhong Sun
9d5db05e4b
Add Migration (#2213) 2024-08-22 10:44:42 -07:00
Yuhong Sun
1a9e5da7c0
Enable Surrounding Context (#2210) 2024-08-22 09:59:13 -07:00
Chris Weaver
8afcb03f3c
Fix OIDC expiry issues (#2206)
* Fix oidc expiry issues

* fix

* fix
2024-08-22 03:15:17 +00:00
Chris Weaver
9bf42d2303
Fix connectors running while deleting (#2204)
* Fix connectors running while deleting

* fix
2024-08-22 02:18:01 +00:00
rkuo-danswer
584eae17e3
fix message param to use query instead of rephrased query (#2199) 2024-08-21 18:00:55 +00:00
rkuo-danswer
b9b633bb74
support indexing attachments as separate docs when not part of a page (#2194)
* support indexing attachments as separate docs when not part of a page

* fix time filter, fix batch handling, fix returned number of attachments processed
2024-08-21 17:15:13 +00:00
Yuhong Sun
bb1916d5d0
Warm Up Models Prep (#2196) 2024-08-20 20:53:02 -07:00
pablodanswer
048cb8dd55
update alembic version (for rebase) (#2193) 2024-08-21 02:58:22 +00:00
Yuhong Sun
3b035d791e
Fix Model Server (#2191) 2024-08-20 17:57:09 -07:00
pablodanswer
53387ab3eb
Simplify index and model name swap logic (#2188) 2024-08-20 17:31:00 -07:00
Yuhong Sun
ec6e2369a1
Log YQL (#2189) 2024-08-20 17:03:57 -07:00
hagen-danswer
075eacdd91
added collection and collection type to Guru metadata (#2187)
* added collection and collection type to metadata

* removed collection type
2024-08-20 23:29:40 +00:00
hagen-danswer
f77b1ebd87
Updated pruning defaults (#2186)
* Updated pruning defaults

* changed minutes to days
2024-08-20 23:29:19 +00:00
rkuo-danswer
1ddb4b2025
normalize emails on bulk invite, normalize/lowercase emails on invite… (#2184)
* normalize emails on bulk invite, normalize/lowercase emails on invite matching

* fix validate_email import
2024-08-20 22:15:42 +00:00