feat(document-search): async unstructured api #37

akonarski-ds · 2024-09-24T12:52:40Z

No description provided.

github-actions · 2024-09-24T12:54:00Z

🧪 Test coverage: 91.95%

Code Coverage Summary

Filename                                                                                                     Stmts    Miss  Cover    Missing
---------------------------------------------------------------------------------------------------------  -------  ------  -------  --------------------------------
packages/ragbits-core/src/ragbits/core/__init__.py                                                               0       0  100.00%
packages/ragbits-core/src/ragbits/core/embeddings/__init__.py                                                    0       0  100.00%
packages/ragbits-core/src/ragbits/core/embeddings/base.py                                                        4       0  100.00%
packages/ragbits-core/src/ragbits/core/llms/__init__.py                                                          4       0  100.00%
packages/ragbits-core/src/ragbits/core/llms/base.py                                                             33       4  87.88%   33, 52, 88, 97
packages/ragbits-core/src/ragbits/core/llms/litellm.py                                                          25       4  84.00%   8-9, 54, 85
packages/ragbits-core/src/ragbits/core/llms/local.py                                                            24      10  58.33%   8-9, 42-47, 57, 70-71
packages/ragbits-core/src/ragbits/core/llms/types.py                                                             8       2  75.00%   24, 28
packages/ragbits-core/src/ragbits/core/llms/clients/__init__.py                                                  4       0  100.00%
packages/ragbits-core/src/ragbits/core/llms/clients/base.py                                                     23       0  100.00%
packages/ragbits-core/src/ragbits/core/llms/clients/exceptions.py                                               14       6  57.14%   7-8, 17, 26-27, 36
packages/ragbits-core/src/ragbits/core/llms/clients/litellm.py                                                  50      10  80.00%   10-11, 70, 108, 122-127
packages/ragbits-core/src/ragbits/core/llms/clients/local.py                                                    37      12  67.57%   11-12, 62-70, 92-103
packages/ragbits-core/src/ragbits/core/prompt/__init__.py                                                        2       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/base.py                                                           18       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/parsers.py                                                        34       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/prompt.py                                                         99       2  97.98%   105, 109
packages/ragbits-core/src/ragbits/core/prompt/discovery/__init__.py                                              2       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/discovery/prompt_discovery.py                                     33       2  93.94%   56-57
packages/ragbits-core/src/ragbits/core/vector_store/__init__.py                                                  0       0  100.00%
packages/ragbits-core/src/ragbits/core/vector_store/base.py                                                     12       0  100.00%
packages/ragbits-core/src/ragbits/core/vector_store/chromadb_store.py                                           51       4  92.16%   9-10, 58, 100
packages/ragbits-core/src/ragbits/core/vector_store/in_memory.py                                                18       0  100.00%
packages/ragbits-core/tests/unit/__init__.py                                                                     0       0  100.00%
packages/ragbits-core/tests/unit/llms/__init__.py                                                                0       0  100.00%
packages/ragbits-core/tests/unit/llms/test_litellm.py                                                           63       0  100.00%
packages/ragbits-core/tests/unit/prompts/__init__.py                                                             0       0  100.00%
packages/ragbits-core/tests/unit/prompts/test_parsers.py                                                        65       0  100.00%
packages/ragbits-core/tests/unit/prompts/test_prompt.py                                                        112       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/__init__.py                                                   0       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/prompt_classes_for_tests.py                                  30       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/test_prompt_discovery.py                                     18       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/__init__.py                    2       1  50.00%   3
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/prompts/__init__.py            3       2  33.33%   2-4
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/prompts/temp_prompt1.py       14       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/prompts/temp_prompt2.py       14       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_chromadb_store.py                                           69       4  94.20%   31, 34, 39, 44
packages/ragbits-core/tests/unit/vector_stores/test_simple_vector_store.py                                      16       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/__init__.py                                         2       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/_main.py                                           45       1  97.78%   94
packages/ragbits-document-search/src/ragbits/document_search/documents/__init__.py                               0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/documents/document.py                              55       3  94.55%   52, 95, 124
packages/ragbits-document-search/src/ragbits/document_search/documents/element.py                               30       1  96.67%   34
packages/ragbits-document-search/src/ragbits/document_search/documents/sources.py                               46       9  80.43%   13-14, 58, 87, 106, 109, 120-122
packages/ragbits-document-search/src/ragbits/document_search/ingestion/__init__.py                               0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/document_processor.py                    20       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/__init__.py                     0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/base.py                        14       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/dummy.py                       11       1  90.91%   29
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured.py                43       7  83.72%   89, 92-93, 122-123, 127, 138
packages/ragbits-document-search/src/ragbits/document_search/retrieval/__init__.py                               0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/__init__.py                    0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/base.py                        5       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/noop.py                        5       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/__init__.py                     0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/base.py                         6       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/noop.py                         7       0  100.00%
packages/ragbits-document-search/tests/__init__.py                                                               0       0  100.00%
packages/ragbits-document-search/tests/helpers.py                                                                3       0  100.00%
packages/ragbits-document-search/tests/integration/__init__.py                                                   0       0  100.00%
packages/ragbits-document-search/tests/integration/test_unstructured.py                                         38      24  36.84%   22-29, 37-43, 51-57, 65-72
packages/ragbits-document-search/tests/unit/__init__.py                                                          0       0  100.00%
packages/ragbits-document-search/tests/unit/test_document_processor.py                                          17       0  100.00%
packages/ragbits-document-search/tests/unit/test_document_search.py                                             41       0  100.00%
packages/ragbits-document-search/tests/unit/test_documents.py                                                   13       0  100.00%
packages/ragbits-document-search/tests/unit/test_elements.py                                                    15       0  100.00%
packages/ragbits-document-search/tests/unit/test_gcs_source.py                                                  14       0  100.00%
packages/ragbits-document-search/tests/unit/test_providers.py                                                   23       0  100.00%
TOTAL                                                                                                         1354     109  91.95%

Diff against main

Filename                                                                                            Stmts    Miss  Cover
------------------------------------------------------------------------------------------------  -------  ------  --------
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured.py      +13      +2  +0.39%
packages/ragbits-document-search/tests/unit/test_providers.py                                          +3       0  +100.00%
TOTAL                                                                                                 +16      +2  -0.05%

Results for commit: 5a20236

Minimum allowed coverage is 60%

♻️ This comment has been updated with latest results

akonarski-ds · 2024-09-24T12:54:08Z

packages/ragbits-document-search/pyproject.toml

@@ -34,7 +34,8 @@ classifiers = [
 dependencies = [
    "numpy~=1.24.0",
    "ragbits",
-    "unstructured>=0.15.12",
+    "unstructured>=0.15.13",
+    "unstructured-client==0.26.0b2",


To be updated with 0.26 version after release

github-actions · 2024-09-24T12:56:36Z

Trivy scanning results.

.venv/lib/python3.10/site-packages/litellm/llms/huggingface_llms_metadata/hf_text_generation_models.txt (secrets)

Total: 1 (MEDIUM: 0, HIGH: 0, CRITICAL: 1)

CRITICAL: HuggingFace (hugging-face-access-token)
════════════════════════════════════════
Hugging Face Access Token
────────────────────────────────────────
.venv/lib/python3.10/site-packages/litellm/llms/huggingface_llms_metadata/hf_text_generation_models.txt:36162
────────────────────────────────────────
36160 mncai/Llama2-7B-Active_3rd-floor-LoRA-dim64_epoch4
36161 ajcdp/CM
36162 [ Nagharjun17/*************************************
36163 BigSalmon/InformalToFormalLincoln114Paraphrase
────────────────────────────────────────

.venv/lib/python3.10/site-packages/litellm/proxy/_types.py (secrets)

Total: 1 (MEDIUM: 1, HIGH: 0, CRITICAL: 0)

MEDIUM: Slack (slack-web-hook)
════════════════════════════════════════
Slack Webhook
────────────────────────────────────────
.venv/lib/python3.10/site-packages/litellm/proxy/_types.py:1288
────────────────────────────────────────
1286 alert_to_webhook_url: Optional[Dict] = Field(
1287 None,
1288 [ bhook_url: {'budget_alerts': '*****************************************************************************'}`",
1289 )
────────────────────────────────────────

.venv/lib/python3.10/site-packages/PyJWT-2.9.0.dist-info/METADATA (secrets)

Total: 1 (MEDIUM: 1, HIGH: 0, CRITICAL: 0)

MEDIUM: JWT (jwt-token)
════════════════════════════════════════
JWT token
────────────────────────────────────────
.venv/lib/python3.10/site-packages/PyJWT-2.9.0.dist-info/METADATA:80
────────────────────────────────────────
78 >>> encoded = jwt.encode({"some": "payload"}, "secret", algorithm="HS256")
79 >>> print(encoded)
80 [ *********************************************************************************************************
81 >>> jwt.decode(encoded, "secret", algorithms=["HS256"])
────────────────────────────────────────

mhordynski

lgtm 🥇

...ages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured.py

akonarski-ds added feature New feature or request document search Changes to the document search package labels Sep 24, 2024

akonarski-ds added this to the Ragbits 0.1 milestone Sep 24, 2024

akonarski-ds linked an issue Sep 24, 2024 that may be closed by this pull request

feat(document-search): async unstructured api #28

Closed

akonarski-ds commented Sep 24, 2024

View reviewed changes

akonarski-ds self-assigned this Sep 25, 2024

mhordynski approved these changes Sep 25, 2024

View reviewed changes

micpst reviewed Oct 4, 2024

View reviewed changes

...ages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured.py Outdated Show resolved Hide resolved

akonarski-ds force-pushed the 28-featdocument-search-async-unstructured-api branch 2 times, most recently from 09ff32e to cb436dc Compare October 7, 2024 09:10

akonarski-ds added 5 commits October 8, 2024 08:00

Swap sync with async Unstructured api call

567964d

Remove prints

8a57a19

Add api_key and api_server params to UnstructuredProvider constructor

788da3d

Update lock

010c5b9

Bump unstructured-client version

5a20236

akonarski-ds force-pushed the 28-featdocument-search-async-unstructured-api branch from cb436dc to 5a20236 Compare October 8, 2024 06:00

akonarski-ds merged commit 5398d20 into main Oct 8, 2024
3 checks passed

micpst deleted the 28-featdocument-search-async-unstructured-api branch October 8, 2024 09:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(document-search): async unstructured api #37

feat(document-search): async unstructured api #37

akonarski-ds commented Sep 24, 2024

github-actions bot commented Sep 24, 2024 •

edited

Loading

akonarski-ds Sep 24, 2024 •

edited

Loading

github-actions bot commented Sep 24, 2024 •

edited

Loading

mhordynski left a comment

feat(document-search): async unstructured api #37

feat(document-search): async unstructured api #37

Conversation

akonarski-ds commented Sep 24, 2024

github-actions bot commented Sep 24, 2024 • edited Loading

Code Coverage Summary

Diff against main

akonarski-ds Sep 24, 2024 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Sep 24, 2024 • edited Loading

.venv/lib/python3.10/site-packages/litellm/llms/huggingface_llms_metadata/hf_text_generation_models.txt (secrets)

.venv/lib/python3.10/site-packages/litellm/proxy/_types.py (secrets)

.venv/lib/python3.10/site-packages/PyJWT-2.9.0.dist-info/METADATA (secrets)

mhordynski left a comment

Choose a reason for hiding this comment

github-actions bot commented Sep 24, 2024 •

edited

Loading

akonarski-ds Sep 24, 2024 •

edited

Loading

github-actions bot commented Sep 24, 2024 •

edited

Loading