121 Commits

Author SHA1 Message Date
Yuhong Sun
ba712d447d
Notion Connector Improvements (#2789) 2024-10-13 23:01:17 -07:00
pablodanswer
20df20ae51
Multi tenant vespa (#2762)
* add vespa multi tenancy

* k

* formatting

* Billing (#2667)

* k

* data -> control

* nit

* nit: error handling

* auth + app

* nit: color standardization

* nit

* nit: typing

* k

* k

* feat: functional upgrading

* feat: add block for downgrading to seats < active users

* add auth

* remove accomplished todo + prints

* nit

* tiny nit

* nit: centralize security

* add tenant expulsion/gating + invite user -> increment billing seat no.

* add cloud configs

* k

* k

* nit: update

* k

* k

* k

* k

* nit
2024-10-12 23:53:11 +00:00
rkuo-danswer
7f788e4b1e
bump celery to 5.5.0b4 (#2681) 2024-10-04 05:54:32 +00:00
Chris Weaver
b8232e0681
Update litellm to fix bedrock models (#2649) 2024-10-01 20:09:57 -07:00
Yuhong Sun
e229d27734
Unstructured UI (#2636)
* checkpoint

* k

* k

* need frontend

* add api key check + ui component

* add proper ports + icons + functions

* k

* k

* k

---------

Co-authored-by: pablodanswer <pablo@danswer.ai>
2024-10-01 04:50:03 +00:00
pablodanswer
45f67368a2
Add support for o1 (#2538)
* add o1 support + bump litellm/openai

* ports

* update exception message for testing
2024-09-22 23:16:28 +00:00
pablodanswer
9f179940f8
Asana connector (community originated) (#2485)
* initial Asana connector

* hint on how to get Asana workspace ID

* re-format with black

* re-order imports

* update asana connector for clarity

* minor robustification

* minor update to naming

* update for best practice

* update connector

---------

Co-authored-by: Daniel Naber <naber@danielnaber.de>
2024-09-19 23:54:18 +00:00
Yuhong Sun
f04ecbf87a Un-bump nltk due to llamaindex issue 2024-09-08 16:39:19 -07:00
dependabot[bot]
5e9c6d1499
Bump aiohttp from 3.9.4 to 3.10.2 in /backend/requirements (#2097)
Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.9.4 to 3.10.2.
- [Release notes](https://github.com/aio-libs/aiohttp/releases)
- [Changelog](https://github.com/aio-libs/aiohttp/blob/master/CHANGES.rst)
- [Commits](https://github.com/aio-libs/aiohttp/compare/v3.9.4...v3.10.2)

---
updated-dependencies:
- dependency-name: aiohttp
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-09-08 10:59:47 -07:00
dependabot[bot]
50211ec401
Bump nltk from 3.8.1 to 3.9 in /backend/requirements (#2174)
Bumps [nltk](https://github.com/nltk/nltk) from 3.8.1 to 3.9.
- [Changelog](https://github.com/nltk/nltk/blob/develop/ChangeLog)
- [Commits](https://github.com/nltk/nltk/compare/3.8.1...3.9)

---
updated-dependencies:
- dependency-name: nltk
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-09-08 10:50:36 -07:00
dependabot[bot]
1e4b27185d
Bump torch from 2.0.1 to 2.2.0 in /backend/requirements (#1933)
Bumps [torch](https://github.com/pytorch/pytorch) from 2.0.1 to 2.2.0.
- [Release notes](https://github.com/pytorch/pytorch/releases)
- [Changelog](https://github.com/pytorch/pytorch/blob/main/RELEASE.md)
- [Commits](https://github.com/pytorch/pytorch/compare/v2.0.1...v2.2.0)

---
updated-dependencies:
- dependency-name: torch
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-09-08 10:17:17 -07:00
Yuhong Sun
c8891a5829
Remove LangChain Community (#2362) 2024-09-08 09:41:20 -07:00
Art Matsak
51a13f5fc7
Implement indexing of simple tables in Word files (#1651) 2024-09-08 09:38:46 -07:00
dependabot[bot]
e2e04af7e2
Bump msal from 1.26.0 to 1.28.0 in /backend/requirements (#1626)
Bumps [msal](https://github.com/AzureAD/microsoft-authentication-library-for-python) from 1.26.0 to 1.28.0.
- [Release notes](https://github.com/AzureAD/microsoft-authentication-library-for-python/releases)
- [Commits](https://github.com/AzureAD/microsoft-authentication-library-for-python/compare/1.26.0...1.28.0)

---
updated-dependencies:
- dependency-name: msal
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-09-07 21:05:11 -07:00
rkuo-danswer
2933c3598b
first cut at redis (#2226)
* first cut at redis

* fix startup dependencies on redis

* kombu cleanup - fail silently

* mypy

* add redis_host environment override

* update REDIS_HOST env var in docker-compose.dev.yml

* update the rest of the docker files

* update contributing guide

* renaming cache to cache_volume

* add redis password to various deployments

* try setting up pr testing for helm

* fix indent

* hopefully this release version actually exists

* fix command line option to --chart-dirs

* fetch-depth 0

* edit values.yaml

* try setting ct working directory

* bypass testing only on change for now

* move files and lint them

* update helm testing

* some issues suggest using --config works

* add vespa repo

* add postgresql repo

* increase timeout

* try amd64 runner

* fix redis password reference

* add comment to helm chart testing workflow

* rename helm testing workflow to disable it

---------

Co-authored-by: Richard Kuo <rkuo@rkuo.com>
2024-09-06 19:21:29 +00:00
rkuo-danswer
abe01144ca
Update CONTRIBUTING.md (#2298) 2024-09-02 15:30:18 -07:00
josvdw
50c17438d5
Litellm bump (#2195)
* ran bump-pydantic

* replace root_validator with model_validator

* mostly working. some alternate assistant error. changed root_validator and typing_extensions

* working generation chat. changed type

* replacing .dict with .model_dump

* argument needed to bring model_dump up to parity with dict()

* fix a fewremaining issues -- working with llama and gpt

* updating requirements file

* more requirement updates

* more requirement updates

* fix to make search work

* return type fix:

* half way tpyes change

* fixes for mypy and pydantic:

* endpoint fix

* fix pydantic protected namespaces

* it works!

* removed unecessary None initializations

* better logging

* changed default values to empty lists

* mypy fixes

* fixed array defaulting

---------

Co-authored-by: hagen-danswer <hagen@danswer.ai>
2024-08-28 00:00:27 +00:00
Yuhong Sun
7dcc42aa95
Intent Model Update (#2069) 2024-08-08 20:45:53 -07:00
hagen-danswer
1be1959d80
Changed default local model to nomic (#1943) 2024-07-31 18:54:02 -07:00
Yuhong Sun
0e8ba111c8
Model Touchups (#1887) 2024-07-21 12:31:00 -07:00
hagen-danswer
eb3e7610fc
Added retries and multithreading for cloud embedding (#1879)
* added retries and multithreading for cloud embedding

* refactored a bit

* cleaned up code

* got the errors to bubble up to the ui correctly

* added exceptin printing

* added requirements

* touchups

---------

Co-authored-by: Yuhong Sun <yuhongsun96@gmail.com>
2024-07-20 22:10:18 -07:00
Zoltan Szabo
1b311d092e
Try to find the sitemap for a given site (#1538) 2024-07-14 13:24:10 -07:00
pablodanswer
e7f81d1688
add third party embedding models (#1818) 2024-07-14 10:19:53 -07:00
pablodanswer
09a11b5e1a
Fix citations + unit tests (#1760) 2024-07-10 10:05:20 -07:00
pablodanswer
de6d040349
add boto3 typing to default requirements (#1740) 2024-06-28 13:06:59 -07:00
pablodanswer
f03f97307f
Blob Storage (#1705)
S3 + OCI + Google Cloud Storage + R2
---------

Co-authored-by: Art Matsak <5328078+artmatsak@users.noreply.github.com>
2024-06-27 17:12:20 -07:00
Yuhong Sun
92de6acc6f Initial EE features (#3) 2024-06-25 15:07:56 -07:00
Weves
7746375bfd Custom tools 2024-06-17 15:12:50 -07:00
hagen-danswer
722a1dd919
Add salesforce connector (#1621) 2024-06-16 10:04:44 -07:00
dependabot[bot]
8c324f8f01
Bump requests from 2.31.0 to 2.32.2 in /backend/requirements (#1509)
Bumps [requests](https://github.com/psf/requests) from 2.31.0 to 2.32.2.
- [Release notes](https://github.com/psf/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
- [Commits](https://github.com/psf/requests/compare/v2.31.0...v2.32.2)

---
updated-dependencies:
- dependency-name: requests
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-11 18:20:08 -07:00
Bill Yang
1fb47d70b3
Add Dropbox connector (#956)
* start dropbox connector

* add wip ui

* polish ui

* Fix some ci

* ignore types

* addressed, fixed, and tested all comments

* ran prettier

* ran mypy fixes

---------

Co-authored-by: Bill Yang <bill@Bills-MacBook-Pro.local>
Co-authored-by: hagen-danswer <hagen@danswer.ai>
2024-06-04 17:58:01 -07:00
Matthieu Boret
fbdf882299 fix sharepoint connector missing objects 2024-05-29 10:13:41 +02:00
Andrew Sansom
94018e83b0
Add MediaWiki and Wikipedia Connectors (#1250)
* Add MediaWikiConnector first draft

* Add MediaWikiConnector first draft

* Add MediaWikiConnector first draft

* Add MediaWikiConnector sections for each document

* Add MediaWikiConnector to constants and factory

* Integrate MediaWikiConnector with connectors page

* Unit tests + bug fixes

* Allow adding multiple mediawikiconnectors

* add wikipedia connector

* add wikipedia connector to factory

* improve docstrings of mediawiki connector backend

* improve docstrings of mediawiki connector backend

* move wikipedia and mediawiki icon locations in admin page

* undo accidental commit of modified docker compose yaml
2024-05-24 08:51:20 -07:00
Weves
10232c7c54 Bump openai version 2024-05-14 17:28:50 -07:00
Weves
e573ba80b9 Revert black bump 2024-05-14 00:20:03 -07:00
Weves
5d1a81001e bump litellm 2024-05-13 18:29:33 -07:00
Weves
c2cdce4d49 Tool calling framework 2024-05-13 00:47:39 -07:00
Yuhong Sun
546815dc8c
Consolidate File Processing (#1449) 2024-05-11 23:11:22 -07:00
dependabot[bot]
cfd9159b27
Bump aiohttp from 3.9.2 to 3.9.4 in /backend/requirements (#1349) 2024-05-11 10:33:08 -07:00
dependabot[bot]
52fd18d3bd
Bump pydantic from 1.10.7 to 1.10.13 in /backend/requirements (#1377) 2024-05-11 10:32:31 -07:00
dependabot[bot]
94a6db51c8
Bump black from 23.3.0 to 24.3.0 in /backend/requirements (#1236) 2024-05-11 10:14:52 -07:00
Weves
f5b3333df3 Add UI-based LLM selection 2024-04-27 01:41:26 -07:00
Yuhong Sun
b59912884b
Fix Model Server (#1320) 2024-04-10 23:13:22 -07:00
Yuhong Sun
2db906b7a2
Always Use Model Server (#1306) 2024-04-07 21:25:06 -07:00
Weves
b06b95dc3a Bump litellm version to support latest Anthropic models 2024-04-02 19:32:19 -07:00
Yuhong Sun
15f7b42e2b
More Options for Search APIs (#1280) 2024-03-31 21:45:46 -07:00
Yuhong Sun
fd69203be8
More accurate input token count for LLM (#1267) 2024-03-28 11:11:37 -07:00
dependabot[bot]
5e7d740814 Bump aiohttp from 3.9.0rc0 to 3.9.2 in /backend/requirements
Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.9.0rc0 to 3.9.2.
- [Release notes](https://github.com/aio-libs/aiohttp/releases)
- [Changelog](https://github.com/aio-libs/aiohttp/blob/master/CHANGES.rst)
- [Commits](https://github.com/aio-libs/aiohttp/compare/v3.9.0rc0...v3.9.2)

---
updated-dependencies:
- dependency-name: aiohttp
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-02-29 16:52:31 -08:00
Weves
3941111685 Update mypy version 2024-02-27 16:02:24 -08:00
Weves
02d81c4be5 Bump up packages + add ddtrace 2024-02-27 12:18:11 -08:00