diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/developing-with-oci-data-science.md b/compliance-submission-oci-genai/developing-with-oci-data-science/developing-with-oci-data-science.md new file mode 100644 index 0000000..f95d70d --- /dev/null +++ b/compliance-submission-oci-genai/developing-with-oci-data-science/developing-with-oci-data-science.md @@ -0,0 +1,161 @@ +# OCI Data Science setup + +## Introduction + +In this lab, we will setup the OCI Data science environment necessary for developing code, API calls, customization and automation for generation of compliance documents + +Estimated Lab Time: -- 10 minutes + +### Oracle cloud Data science + +OCI Data science is a managed Python based Jupyter lab based notebook development environment for developing and deploying Machine learning and AI models including Generative AI services. The service supports both CPU and GPU infrastructure, has access to OCI lakehouse storage like Object store, Autonomous database, Data flow, Data catalog and other services. + +### Objectives + +In this lab, you will: + +* Deploy a pre-built langchain based conda environment +* Test connectivity to OCI Generative AI services +* Test connectivity with OCI Opensearch services +* Deploy OCI CLI connectivity with OCI Object store +* Download and install required pip libraries +* Install Compliance Document Generation Notebooks + +### Prerequisites + +This lab assumes you have: + +* An Oracle Cloud account with admin privileges in the Chicago region +* A running Data science notebook session environment +* A running OCI Opensearch service + +## Task 1: Deploy a pre-built langchain conda environment + +1. From the Launcher (File-->New Launcher if needed), click on the Environment explorer to view the list of conda environments + ![Install pre-built conda](images/lab3-ds-cnd-1.png) + +2. Filter the conda environment to view the ones containing the langchain libraries and select the one marked below + ![Install pre-built conda](images/lab3-ds-cnd-2.png) + +3. Copy the command command below to run in a terminal session + ![Install pre-built conda](images/lab3-ds-cnd-2-1.png) + +4. Open up a Terminal session as shown from the Launcher + ![Install pre-built conda](images/lab3-ds-cnd-3.png) + +5. Paste and run the *odsc conda install -s pytorch21_p39_gpu_v1* command as shown. It may take a few minutes to install the conda environment. Make sure it is successfully completed and installed as shown + ![Install pre-built conda](images/lab3-ds-cnd-4.png) + +## Task 2: Download and install required pip libraries + +1. Locate the notebooks in the /home/datascience/conda directory. This directory will be used for creating and running all notebooks for the workshop + ![Install pip libraries](images/lab3-ds-note-1.png) + +2. Create a new notebook + ![Install pip libraries](images/lab3-ds-note-2.png) + +3. Change the kernel to the installed conda environment + ![Install pip libraries](images/lab3-ds-note-3.png) + +4. Copy and execute to install the pip libraries as shown below in the notebook cell. Press *Shift+Enter* to execute the notebook cell + +```text + +!pip install langchain +!pip install langchain_community +!pip install opensearch-py +!pip install sentence-transformers +!pip install tabulate +!pip install pypdf +!pip install fillpdf + +``` + + ![Install pip libraries](images/lab3-ds-note-4.png) + +NOTE: It is possible that some of the libraries are pre-installed in the environment. Ignore if that is so. You may also have incompatibilities with other libraries in the pre-built conda. You may ignore them if that occurs. Comment them as shown below + + ![Install pip libraries](images/lab3-ds-note-5.png) + +## Task 3: Install Workshop Compliance Document Generation code + +1. Download [LAB-3 Conda zip](https://orasenatdpltintegration03.objectstorage.us-chicago-1.oci.customer-oci.com/p/SfhRh7OEvLj9yR0hAIM3BwT7bCpi3jALfP6NqoCODU7mFe51nl1PeBPWcJj2El9K/n/orasenatdpltintegration03/b/clinical-trials/o/conda.zip) and upload to the home directory */home/datascience* in the notebook session as shown below. You can also directly download in your environment using *wget * as well from a data science terminal session. + ![Install lab notebooks](images/lab3-ds-note-6.png) + +2. Open up a terminal session and run *unzip conda.zip* as shown below. + ![Install lab notebooks](images/lab3-ds-note-7.png) + +## Task 4: Test connectivity with OCI Opensearch services + +1. Copy the Opensearch API URL from the console + ![Test Opensearch Access](images/lab3-ds-os-1.png) + +2. Change to *cd /home/datascience/conda/scripts* directory in a data science terminal window and run. Sucecssful connection should display the json as shown below + +```text + +curl -k -u (os_userid):(os_password) (os_api_endpoint):9200 + +``` + + ![Test Opensearch Access](images/lab3-ds-os-2.png) + +## Task 5: Configure OCI CLI Connectivity to Object store and Generative AI + +1. Get your user OCID and your Tenancy ID from console as shown below + ![Test Opensearch Access](images/lab3-ds-cli-1.png) + ![Test Opensearch Access](images/lab3-ds-cli-2.png) + ![Test Opensearch Access](images/lab3-ds-cli-3.png) + +2. Open up a terminal window and enter *oci os ns get*. Enter values as follows + +```text + +Do you want to create a new config file ? Y +Create logging through a browser? n +Location of your config: Enter +Enter user OCID : +Enter Tenancy OCID : +Region by index or name : us-chicago-1 +Do you want to generate a new RSA key pair? Y +Enter directory for keys created : Enter +Enter name of your key : Enter +Enter passphrase: N/A +Re-enter passphrase : N/A + +``` + + ![Test Opensearch Access](images/lab3-ds-cli-4.png) + ![Test Opensearch Access](images/lab3-ds-cli-5.png) + +1. Move and download your generated public key pem file + ![Test Opensearch Access](images/lab3-ds-cli-7.png) + +2. Upload the downloaded public API key to OCI Console + ![Test Opensearch Access](images/lab3-ds-cli-8.png) + ![Test Opensearch Access](images/lab3-ds-cli-9.png) + ![Test Opensearch Access](images/lab3-ds-cli-10.png) + +3. Test out the OCI CLI access after from Data science notebook session. + ![Test Opensearch Access](images/lab3-ds-cli-11.png) + +## Task 6: Test connectivity to OCI Generative AI services + +1. Open up the Generative AI Generation Interface for API code testing. Please note that the *command R* chat interface may not work as of yet and is not required for this workshop. You may test with the cohere chat interface available in the generation interface + ![Test Opensearch Access](images/lab3-ds-gai-1.png) + +2. Generate a query and click on the *View Code* button and select *python* as the Language + ![Test Opensearch Access](images/lab3-ds-gai-2.png) + +3. Copy the generated code to a notebook cell. You should be able to generate output as shown below + ![Test Opensearch Access](images/lab3-ds-gai-3.png) + +## Learn More + +* [Generative AI made easy with OCI Datascience](https://www.oracle.com/artificial-intelligence/generative-ai/generative-ai-service/) +* [Data science github repository](https://github.com/oracle-samples/oci-data-science-ai-samples/tree/main/ai-quick-actions) + +## Acknowledgements + +* **Author** - Rajib Ghosh, Master Principal Cloud Architect, OCI AI and Gen AI Center of Excellence +* **Last Updated By/Date** - Aug 2024 diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/files/clean.sh b/compliance-submission-oci-genai/developing-with-oci-data-science/files/clean.sh new file mode 100644 index 0000000..938d697 --- /dev/null +++ b/compliance-submission-oci-genai/developing-with-oci-data-science/files/clean.sh @@ -0,0 +1,4 @@ +rm -Rf ~/conda/data +rm -Rf ~/conda/scripts +rm ~/conda/notebooks/demo*.ipynb +rm -f conda.zip diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/files/conda.zip b/compliance-submission-oci-genai/developing-with-oci-data-science/files/conda.zip new file mode 100644 index 0000000..d5ba9b4 Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/files/conda.zip differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-1.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-1.png new file mode 100644 index 0000000..c296c7c Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-1.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-10.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-10.png new file mode 100644 index 0000000..81289be Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-10.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-11.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-11.png new file mode 100644 index 0000000..ee0f450 Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-11.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-2.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-2.png new file mode 100644 index 0000000..1b15b11 Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-2.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-3.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-3.png new file mode 100644 index 0000000..f1f31b4 Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-3.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-4.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-4.png new file mode 100644 index 0000000..567f6a7 Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-4.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-5.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-5.png new file mode 100644 index 0000000..9c20988 Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-5.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-6.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-6.png new file mode 100644 index 0000000..9c20988 Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-6.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-7.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-7.png new file mode 100644 index 0000000..083f088 Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-7.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-8.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-8.png new file mode 100644 index 0000000..c4afe35 Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-8.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-9-1.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-9-1.png new file mode 100644 index 0000000..6a8e335 Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-9-1.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-9.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-9.png new file mode 100644 index 0000000..6a8e335 Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cli-9.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cnd-1.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cnd-1.png new file mode 100644 index 0000000..dd66f95 Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cnd-1.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cnd-2-1.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cnd-2-1.png new file mode 100644 index 0000000..e9fc546 Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cnd-2-1.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cnd-2.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cnd-2.png new file mode 100644 index 0000000..09f69d5 Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cnd-2.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cnd-3.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cnd-3.png new file mode 100644 index 0000000..90ba044 Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cnd-3.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cnd-4.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cnd-4.png new file mode 100644 index 0000000..56d45a2 Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-cnd-4.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-gai-1.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-gai-1.png new file mode 100644 index 0000000..0ceb4b7 Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-gai-1.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-gai-2.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-gai-2.png new file mode 100644 index 0000000..e27d788 Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-gai-2.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-gai-3.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-gai-3.png new file mode 100644 index 0000000..9c74e86 Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-gai-3.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-note-1.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-note-1.png new file mode 100644 index 0000000..87ef244 Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-note-1.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-note-2.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-note-2.png new file mode 100644 index 0000000..2f86263 Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-note-2.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-note-3.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-note-3.png new file mode 100644 index 0000000..13451ed Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-note-3.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-note-4.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-note-4.png new file mode 100644 index 0000000..06aaa35 Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-note-4.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-note-5.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-note-5.png new file mode 100644 index 0000000..40065af Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-note-5.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-note-6.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-note-6.png new file mode 100644 index 0000000..f8a34e7 Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-note-6.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-note-7.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-note-7.png new file mode 100644 index 0000000..cdbc4ce Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-note-7.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-os-1.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-os-1.png new file mode 100644 index 0000000..15e82c3 Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-os-1.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-os-2.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-os-2.png new file mode 100644 index 0000000..d2ec13a Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-os-2.png differ diff --git a/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-os-3.png b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-os-3.png new file mode 100644 index 0000000..1d8137d Binary files /dev/null and b/compliance-submission-oci-genai/developing-with-oci-data-science/images/lab3-ds-os-3.png differ diff --git a/compliance-submission-oci-genai/generate-clinical-trials-with-oci-gai/generate-clinical-trials-with-oci-gai.md b/compliance-submission-oci-genai/generate-clinical-trials-with-oci-gai/generate-clinical-trials-with-oci-gai.md new file mode 100644 index 0000000..a1c0545 --- /dev/null +++ b/compliance-submission-oci-genai/generate-clinical-trials-with-oci-gai/generate-clinical-trials-with-oci-gai.md @@ -0,0 +1,64 @@ +# OCI Generative AI Playground and Trial Generation + +## Introduction + +OCI Generative AI Playground allows you chat, generate and summarize content and also show numerical vector embeddings for textual data. It is REST API enabled for programmatic access and allows you to tune the outputs of your query based on certain parameters. It currently allows Cohere command R+ and Meta Llama model. In this workshop, all clinical trials data is generated with Cohere command R+ playground. Clinical trial datasets are generated for Cancer, diabetes and liver diseases. + +Estimated Lab Time: -- 5 minutes + +### Objectives + +In this lab, you will: + +* Learn how to use OCI Generative AI Playground interactively +* Prompt and generate a few clinical trial documents yourself +* Verify the generated document has no personal information +* Prompt to try out some summarization examples on your text +* View generated API code (Python) code + +### Prerequisites (Optional) + +This lab assumes you have: + +* An Oracle Cloud account in the Chicago region +* You have completed the required policy setup for this workshop +* You are part of the administrator group in the tenancy + +## Task 1: Accessing OCI Generative AI Playground + +In this section get familiar with the OCI Generative AI playground console + +1. Login to your Oracle cloud tenancy and change your tenancy to US Midwest (chicago) + ![Connect to US-Midwest Chicago Tenancy](images/lab-11.png) + +2. From Hamburger menu (top left corner), pull up Analytics & AI --> AI Services --> Generative AI + ![Connect to OCI Gen AI](images/lab-12.png) + +## Task 2: Generate a clinical trial in OCI Generative AI Playground + +1. Click on the Generative AI -> Overview -> Playground -> Chat and Run the example "Generate a job description" with the cohere-command-r-16k model + ![Test OCI Gen AI Example](images/lab-13.png) + +2. Copy the following text in the chat window *Generate a clinical trial report on drug evaluation on Advanced Non-Small Cell Lung Cancer* , change the *Maximum output settings* and the *Temperature* settings as shown. Press Submit to generate a clinical trial report. + ![Generate trial document](images/lab-14.png) + +3. Note how the Personal Identifiable Information (PII) is redacted and substituted + ![PII Redaction](images/lab-15.png) + +## Task 3: Generate a summary and view generated code + +1. Copy the generated clinical trial to Playground -> Summarization and generate summary. + ![Summary](images/lab-16.png) + +2. Click on the view code button to see the generated code + ![Generated python code](images/lab-17.png) + +## Learn More + +* [OCI Generative AI](https://www.oracle.com/artificial-intelligence/generative-ai/generative-ai-service/) +* [Realize business value by transforming data into action with Generative AI](https://blogs.oracle.com/ai-and-datascience/post/generative-ai-use-cases/) + +## Acknowledgements + +* **Author** - Rajib Ghosh, Master Principal Cloud Architect, OCI AI and GenAI Center of Excellence +* **Last Updated By/Date** - Aug, 2024 diff --git a/compliance-submission-oci-genai/generate-clinical-trials-with-oci-gai/images/lab-11.png b/compliance-submission-oci-genai/generate-clinical-trials-with-oci-gai/images/lab-11.png new file mode 100644 index 0000000..6232427 Binary files /dev/null and b/compliance-submission-oci-genai/generate-clinical-trials-with-oci-gai/images/lab-11.png differ diff --git a/compliance-submission-oci-genai/generate-clinical-trials-with-oci-gai/images/lab-12.png b/compliance-submission-oci-genai/generate-clinical-trials-with-oci-gai/images/lab-12.png new file mode 100644 index 0000000..151fc4f Binary files /dev/null and b/compliance-submission-oci-genai/generate-clinical-trials-with-oci-gai/images/lab-12.png differ diff --git a/compliance-submission-oci-genai/generate-clinical-trials-with-oci-gai/images/lab-13.png b/compliance-submission-oci-genai/generate-clinical-trials-with-oci-gai/images/lab-13.png new file mode 100644 index 0000000..4281220 Binary files /dev/null and b/compliance-submission-oci-genai/generate-clinical-trials-with-oci-gai/images/lab-13.png differ diff --git a/compliance-submission-oci-genai/generate-clinical-trials-with-oci-gai/images/lab-14.png b/compliance-submission-oci-genai/generate-clinical-trials-with-oci-gai/images/lab-14.png new file mode 100644 index 0000000..108e374 Binary files /dev/null and b/compliance-submission-oci-genai/generate-clinical-trials-with-oci-gai/images/lab-14.png differ diff --git a/compliance-submission-oci-genai/generate-clinical-trials-with-oci-gai/images/lab-15.png b/compliance-submission-oci-genai/generate-clinical-trials-with-oci-gai/images/lab-15.png new file mode 100644 index 0000000..9fe2c13 Binary files /dev/null and b/compliance-submission-oci-genai/generate-clinical-trials-with-oci-gai/images/lab-15.png differ diff --git a/compliance-submission-oci-genai/generate-clinical-trials-with-oci-gai/images/lab-16.png b/compliance-submission-oci-genai/generate-clinical-trials-with-oci-gai/images/lab-16.png new file mode 100644 index 0000000..768e307 Binary files /dev/null and b/compliance-submission-oci-genai/generate-clinical-trials-with-oci-gai/images/lab-16.png differ diff --git a/compliance-submission-oci-genai/generate-clinical-trials-with-oci-gai/images/lab-17.png b/compliance-submission-oci-genai/generate-clinical-trials-with-oci-gai/images/lab-17.png new file mode 100644 index 0000000..db8c0d6 Binary files /dev/null and b/compliance-submission-oci-genai/generate-clinical-trials-with-oci-gai/images/lab-17.png differ diff --git a/compliance-submission-oci-genai/generate-compliance-document/generate-compliance-document.md b/compliance-submission-oci-genai/generate-compliance-document/generate-compliance-document.md new file mode 100644 index 0000000..1da86ec --- /dev/null +++ b/compliance-submission-oci-genai/generate-compliance-document/generate-compliance-document.md @@ -0,0 +1,93 @@ +# Generate Compliance submission documents + +## Introduction + +In this lab, we will generate a sample compliance submission form summarizing the clinical trial data sections from a sample with a pre-canned PDF template. Prompting techniques and formatting has been used to hint the OCI Generative AI large language model to contain sectional summaries in a condensed manner. The process can be extended to summarize across multiple retrieval chunks of data with langchain and vector search. + +Estimated Lab Time: -- 10 minutes + +### Langchain output parsers and OCI Generative AI + +[LangChain](https://python.langchain.com/v0.2/docs/introduction/) is a Python library that simplifies the development, productionization, and deployment of applications powered by large language models (LLMs). It is a development framework that chains LLMs, agents and retrieval strategies to effectively make up an end-to-end application's cognitive architecture. Chains can be defined declaratively for convenience. Langchain output parsers offer various custom formatting options for sectional prompting to extract summarized textual data into a templated format. + +### Objectives + +In this lab, you will: + +* Load clinical-trial PDF documents and embeddings into an Opensearch index +* Use langchain output parser to produce sectional summaries for a document template +* Generate a sample compliance submission form from OCI Generative AI LLM + +### Prerequisites + +This lab assumes you have: + +* Working knowledge of Python and Notebooks +* Working knowledge of OCI Data science and conda packs +* Some knowledge of langchain framework but not required. + +## Task 1: Load clinical-trials documents and metadata + +1. Get the following information into a notepad or a script + + * Compartment OCID for *clinical-trials* compartment. (Search on OCI console for compartments, click your compartment and copy the OCID) + * Opensearch username - The username entered while provisioning Opensearch cluster (i,e *osmaster*) + * Opensearch password - The password entered while provisioning Opensearch cluster + * API end point Private IP from OCI Opensearch service console + +2. Double click to open up the notebook *demo-generate-document.ipynb* Run each of the cells one by one from top by using *Shift+Enter* or play button at the top + +3. Substitute the following definitions in the cell as shown below + ![Image alt text](images/lab5-note-os-1.png) + +4. Load all PDF documents using PyPDFDirectory loader to load all documents into a pandas data frame + +5. Generate page_content and document metadata embeddings using OCI Generative AI + +6. Check Opensearch client connectivity. It should show the *OpenSearch([{'host': 'hostname', 'port': 9200}])* as output + +7. Load both text and embeddings data into the *idx-oci-genai-clinical-trials* index + +8. Paste the title retrieved from the previous lab *demo-vector-search-ext.ipynb* to query based on page_content embeddings + +9. Report file metadata and the score. + +## Task 2: Generate Compliance document form + +1. Select the top retrieved document from the query search above and enter in your query. + +2. Run the rest of the cells to generate the compliance form for the trial. + +3. View the generated compliance form in */home/datascience/conda/data/outputs* directory + ![Image alt text](images/lab5-comp-doc.png) + +This involves + +1. Defining a pydantic Object base model class called *TrialInfo* to structure document sections and their description instructions. These are formatted instructions that are passed to the OCI Generative AI LLM at runtime. +2. This *TrialInfo* class is a superset representing sectional headers for all clinical-trial documents in this workshop. +3. Defining a langchain pydantic output parser object and passing the format instructions. +4. Defining a chat prompt template with specific instructions to use the format instructions +5. Using OCI Generative AI chat llm to perform sectional summarization based on the format instructions. +6. Mapping the summarized results to the form template fields +7. Generating the compliance document with a PDF filler + +## Task 3: Various other ways to customize this notebook + +Ways to extend and customize + +1. Using langchain chunking classes to split document, embed and load to an index +2. Perform embedding search on chunked documents index +3. Compare chunked retrievals vs full document retrievals and evaluate scores +4. Use a different template or use multiple clinical trial templates by disease +5. Use other prompting techniques with different format specifications +6. Use a better PDF form filler. + +## Learn More + +* [Deploy Langchain applications as OCI Model Deployments](https://blogs.oracle.com/ai-and-datascience/post/deploy-langchain-application-as-model-deployment) +* [OCI AI Quick actions](https://docs.oracle.com/en-us/iaas/data-science/using/ai-quick-actions.htm) + +## Acknowledgements + +* **Author** - Rajib Ghosh, Master Principal Cloud Architect, OCI AI and Gen AI Center of Excellence +* **Last Updated By/Date** - Aug 2024 diff --git a/compliance-submission-oci-genai/generate-compliance-document/images/lab5-comp-doc.png b/compliance-submission-oci-genai/generate-compliance-document/images/lab5-comp-doc.png new file mode 100644 index 0000000..76dcd63 Binary files /dev/null and b/compliance-submission-oci-genai/generate-compliance-document/images/lab5-comp-doc.png differ diff --git a/compliance-submission-oci-genai/generate-compliance-document/images/lab5-note-os-1.png b/compliance-submission-oci-genai/generate-compliance-document/images/lab5-note-os-1.png new file mode 100644 index 0000000..67f39f4 Binary files /dev/null and b/compliance-submission-oci-genai/generate-compliance-document/images/lab5-note-os-1.png differ diff --git a/compliance-submission-oci-genai/getting-started/getting-started.md b/compliance-submission-oci-genai/getting-started/getting-started.md new file mode 100644 index 0000000..3ebe28a --- /dev/null +++ b/compliance-submission-oci-genai/getting-started/getting-started.md @@ -0,0 +1,46 @@ +# Get started + +Generating templated compliance documents is a complex process that involves understanding of various components services, their interactions and the setup. +Though this workshop do not reflect an actual form, however the principles outlined here can be effectively used for this purpose.So, let's get started. + +## Objectives + +In this workshop, you will learn how to: + +* Use OCI Generative AI and OCI DataScience to vectorize clinical trial data +* Embedding and loading data into a vector store like OCI Opensearch +* Implement a Retrieval Augmented Generation (RAG) solution +* Effective prompting to generate query outputs with Cohere chat LLM model +* Structure LLM generated outputs with langchain +* Using a pre-canned template to generate a compliance form + +## Task 1: Prerequisites + +This lab assumes you have: + +* Basic familiarity with Generative AI concepts, RAG and Industry +* Some familiarity with OCI Generative AI Services and Tool sets +* Familiarity with Python programming language. +* Basic understanding of large language models (LLM) +* Some familiarity with OCI Opensearch service +* Some familiarity with open source langchain framework +* Familiarity with clinical trial and compliance submission process would be helpful but not required + +## Task 2: Downloads + +The zipped notebooks and scripts can be downloaded from [here](https://orasenatdpltintegration03.objectstorage.us-chicago-1.oci.customer-oci.com/p/SfhRh7OEvLj9yR0hAIM3BwT7bCpi3jALfP6NqoCODU7mFe51nl1PeBPWcJj2El9K/n/orasenatdpltintegration03/b/clinical-trials/o/conda.zip). Installation instructions are described in LAB-3 Developing with OCI Data science. + +## Task 3: Provision Oracle cloud tenancy and login + +Use the live-lab link to provision a cloud tenancy and testing your login + [provision new cloud tenancy account](https://github.com/oracle-livelabs/common/blob/main/labs/cloud-login/event-register-free-tier-account.md) + +## Learn More + +* [Oracle Generative AI Capabilities](https://www.oracle.com/artificial-intelligence/generative-ai/) +* [Oracle Clinical Digital Assistant](https://www.oracle.com/health/clinical-suite/clinical-digital-assistant/) + +## Acknowledgements + +* **Author** - Rajib Ghosh, Master Principal Cloud Architect, OCI GenAI Center of excellence +* **Last Updated By/Date** - Aug, 2024 diff --git a/compliance-submission-oci-genai/howtouse-deletewhenfinished.md b/compliance-submission-oci-genai/howtouse-deletewhenfinished.md new file mode 100644 index 0000000..27dec4b --- /dev/null +++ b/compliance-submission-oci-genai/howtouse-deletewhenfinished.md @@ -0,0 +1,73 @@ +# Workshop with a single set of labs + +## Instructions - Delete this file when finished + +1. Open the sample-workshop template in Atom or Visual Studio Code +2. We pre-created 5 folders. A workshop is created out of multiple labs. +3. Remove the comments like this one: *List objectives for this lab* +4. Make sure you use lower case folder and file name and dashes for spaces (setup-adb NOT Setup_ADB) +5. Your image names should have descriptive names. Not just adb1, adb2, adb3. For handicap accessibility, we need the image descriptions to explain what the image looks like. Remember all lowercase and dashes. +6. Download our QA doc from WMS. We find workshops get in production quicker when you know what's needed to move to production up front and you use the skeleton. + +PS: You do not need a Readme.md. Readme's exist only at the top library levels. We direct all traffic to LiveLabs since we can't track usage on GitHub. Do not create any direct links to GitHub, your workshop may be super popular but we can't track it so no one will know. + +## Absolute Path for Oracle Cloud menu Navigation + +**Lab 1: Provision an Instance -> Step 0: Use these Standardized Pictures for Oracle Cloud Navigation (Commonly for Provisioning)** - We've included a list of common screenshots for navigating the Oracle Cloud Menu. Please read this section and use the relevant absolute path images where appropriate. This will future proof your workshop in case of Oracle Cloud user interface updates. + +## Folder Structure + +In this example, the goal is to create several "children" workshops from one longer "parent" workshop. The children are made up of parts from the parent. + +sample-workshop/ + -- individual labs + + provision/ + setup/ + dataload/ + query/ + introduction/ + introduction.md -- description of the everything workshop, note that it is a "lab" since there is only one + + workshops/ + freetier/ -- freetier version of the workshop + index.html + manifest.json + livelabs/ -- livelabs version of the workshop + index.html + manifest.json + + +### FreeTier vs LiveLabs + +* "FreeTier" - includes Free Trials, Paid Accounts, and for some workshops, Always Free accounts (brown button) +* "LiveLabs" - these are workshops that use Oracle-provided tenancies (green button) +* "Desktop" - this is a new deployment where the workshop is encapsulated in a NoVNC environment running in a compute instance + +### About the Workshop + +The workshop includes all 6 of the individual labs in a single sequence. + +The folder structure includes an Introduction "lab" that describes the workshop as a complete set of 6 labs. Note: you may not need to have a different introduction for each of the parent and child versions of the workshops, this is illustrative only. + +Look at the product-name-workshop/freetier folder and look at the manifest.json file to see the structure. + +> **Note:** The use of "Lab n:" in the titles is optional + +The Prerequisite "lab" is the first lab in a common folder on the oracle-livelabs/common repo. Because this lab already exists, we can use a RAW/absolute URL instead: + + ``` + "filename": "https://oracle-livelabs.github.io/common/labs/cloud-login/cloud-login-livelabs2.md" }, + ``` + +The manifest.json file needs to know the location of each lab relative to where it exists in the hierarchy. In this structure, labs are located two levels up, for example: + + ``` + "filename": "../../provision/provision.md" + ``` + +### For example: + +This [APEX Workshop](https://oracle-livelabs.github.io/apex/spreadsheet/workshops/freetier/) is a good example of a workshop with a single set of labs: [https://github.com/oracle-livelabs/apex/tree/main/spreadsheet](https://github.com/oracle-livelabs/apex/tree/main/spreadsheet). + + diff --git a/compliance-submission-oci-genai/introduction/images/solution-architecture.png b/compliance-submission-oci-genai/introduction/images/solution-architecture.png new file mode 100644 index 0000000..6c818a0 Binary files /dev/null and b/compliance-submission-oci-genai/introduction/images/solution-architecture.png differ diff --git a/compliance-submission-oci-genai/introduction/introduction.md b/compliance-submission-oci-genai/introduction/introduction.md new file mode 100644 index 0000000..bec8997 --- /dev/null +++ b/compliance-submission-oci-genai/introduction/introduction.md @@ -0,0 +1,54 @@ +# Introduction + +Organizations today are challenged with analyzing large amounts of structured and unstructured data such as tables, invoices, reports for compliance reporting to regulatory authorities accurately.This requires analyzing, filtering and summarizing large volume of textual and tabular data that are typically resident in an enterprise lakehouse, on-premise servers and cloud. Producing form based templated outputs accurately and timely is a huge time consuming task and hence enterprises are looking at AI to resolve this issue + +## About this Workshop + +In this hands-on lab, we will show how you can auto-generate and fill compliance forms that summarizes information from a pool of clinical trial PDF documents for submission to regulatory authorities. This lab uses OCI Generative AI services to generate fillable form content from a healthcare organization's clinical trial dataset. + +In order to maintain generic nature of the lab and data compliance, all source clinical trial data are auto-generated by OCI Generative AI model such as Cohere Command R+. + +Learn more about the architecture, methodology and key automation steps to scale document generation process across your enterprise knowledge bases. + +## Task 1: Solution blueprint and Architecture + +Key points covering the solution are + +* Generic RAG framework with OCI only services +* Support for multiple data sources, embedding options, models and llms +* OCI Gen AI generated clinical trials +* Ability to score and compare various vector embedding functions +* Both open source and OCI gen AI embedding function examples +* OCI Generative AI Cohere chat llm based summarization +* Integration with langchain and output generation +* Support for external & in vector store embedding with OCI Opensearch +* Provision for using Opensearch Hybrid search +* Easy extensibility to OCI AI vector search or to other vector stores + +The diagram below shows a comprehensive architecture of the solution blueprint. + + ![Solution Architecture](images/solution-architecture.png) + +## Task 2: Labs + +| Lab | Module | Est. Time | + | --- | --- | --- | + | | Getting started | 5 minutes | + | 1 | Setup OCI Services | 15 minutes | + | 2 | Generate clinical trials with OCI Generative AI | 5 minutes | + | 3 | Developing with OCI Data science | 10 minutes | + | 4 | Vectorize, load and evaluate | 10 minutes | + | 5 | Generate Compliance document | 10 minutes | + | 6 | Using Neural and Hybrid search | 10 minutes | + +Total Estimated time: 65 minutes + +## Learn More + +* [Oracle Generative AI Capabilities](https://www.oracle.com/artificial-intelligence/generative-ai/) +* [Oracle Clinical Digital Assistant](https://www.oracle.com/health/clinical-suite/clinical-digital-assistant/) + +## Acknowledgements + +* **Author** - Rajib Ghosh, Master Principal Cloud Architect, OCI GenAI Center of excellence +* **Last Updated By/Date** - Aug, 2024 diff --git a/compliance-submission-oci-genai/other-livelabs/other-livelabs.md b/compliance-submission-oci-genai/other-livelabs/other-livelabs.md new file mode 100644 index 0000000..300a7dc --- /dev/null +++ b/compliance-submission-oci-genai/other-livelabs/other-livelabs.md @@ -0,0 +1,12 @@ +# Other LiveLabs you might like + +## Learn More + +* [Search Documents stored in Object storage with OpenSearch, Generative AI, Semantic search, RAG](https://livelabs.oracle.com/pls/apex/f?p=133:180:115467694451312::::wid:3762) +* [Build an AI chatbot engine with Oracle Database 23ai and OCI Generative AI](https://livelabs.oracle.com/pls/apex/r/dbpm/livelabs/view-workshop?wid=3939) +* [Build Innovative Q&A Interface powered by Generative AI with Oracle APEX](https://livelabs.oracle.com/pls/apex/r/dbpm/livelabs/view-workshop?wid=3947) + +## Acknowledgements + +* **Author** - Rajib Ghosh, Master Principal Cloud Architect, OCI AI and Gen AI Center of Excellence +* **Last Updated By/Date** - Aug 2024 diff --git a/compliance-submission-oci-genai/query/files/starter-file.sql b/compliance-submission-oci-genai/query/files/starter-file.sql new file mode 100644 index 0000000..9d7e762 --- /dev/null +++ b/compliance-submission-oci-genai/query/files/starter-file.sql @@ -0,0 +1,101 @@ +/* NOTE: Files cannot contain empty lines (line breaks) */ +/* Specify the base URL that you copied from your files in OCI Object Storage in the define base_URL line below*/ +/* change idthydc0kinr to your real namespace. The name is case-sensitive. */ +/* change ADWCLab to your real bucket name. The name is case-sensitive. */ +/* change us-phoenix-1 to your real region name. The name is case-sensitive. */ +/* you can find these values on the OCI Console .. Storage .. Object Storage screen */ +set define on +define base_URL='https://objectstorage.us-phoenix-1.oraclecloud.com/n/idthydc0kinr/b/ADWCLab/o' +/* copy Channels table */ +begin + dbms_cloud.copy_data( + table_name =>'CHANNELS', + credential_name =>'OBJ_STORE_CRED', + file_uri_list =>'&base_URL/chan_v3.dat', + format => json_object('ignoremissingcolumns' value 'true', 'removequotes' value 'true') + ); +end; +/ +/* copy Countries table */ +begin + dbms_cloud.copy_data( + table_name =>'COUNTRIES', + credential_name =>'OBJ_STORE_CRED', + file_uri_list =>'&base_URL/coun_v3.dat', + format => json_object('ignoremissingcolumns' value 'true', 'removequotes' value 'true') + ); +end; +/ +/* Copy customers */ +begin + dbms_cloud.copy_data( + table_name =>'CUSTOMERS', + credential_name =>'OBJ_STORE_CRED', + file_uri_list =>'&base_URL/cust1v3.dat', + format => json_object('ignoremissingcolumns' value 'true', 'removequotes' value 'true', 'dateformat' value 'YYYY-MM-DD-HH24-MI-SS') + ); +end; +/ +begin + dbms_cloud.copy_data( + table_name =>'SUPPLEMENTARY_DEMOGRAPHICS', + credential_name =>'OBJ_STORE_CRED', + file_uri_list =>'&base_URL/dem1v3.dat', + format => json_object('ignoremissingcolumns' value 'true', 'removequotes' value 'true') + ); +end; +/ +begin + dbms_cloud.copy_data( + table_name =>'SALES', + credential_name =>'OBJ_STORE_CRED', + file_uri_list =>'&base_URL/dmsal_v3.dat', + format => json_object('ignoremissingcolumns' value 'true', 'removequotes' value 'true', 'dateformat' value 'YYYY-MM-DD') + ); +end; +/ +begin + dbms_cloud.copy_data( + table_name =>'PRODUCTS', + credential_name =>'OBJ_STORE_CRED', + file_uri_list =>'&base_URL/prod1v3.dat', + format => json_object('delimiter' value '|', 'quote' value '^', 'ignoremissingcolumns' value 'true', 'dateformat' value 'YYYY-MM-DD-HH24-MI-SS', 'blankasnull' value 'true') + ); +end; +/ +begin + dbms_cloud.copy_data( + table_name =>'PROMOTIONS', + credential_name =>'OBJ_STORE_CRED', + file_uri_list =>'&base_URL/prom1v3.dat', + format => json_object('ignoremissingcolumns' value 'true', 'removequotes' value 'true', 'dateformat' value 'YYYY-MM-DD-HH24-MI-SS', 'blankasnull' value 'true') + ); +end; +/ +begin + dbms_cloud.copy_data( + table_name =>'SALES', + credential_name =>'OBJ_STORE_CRED', + file_uri_list =>'&base_URL/sale1v3.dat', + format => json_object('ignoremissingcolumns' value 'true', 'removequotes' value 'true', 'dateformat' value 'YYYY-MM-DD', 'blankasnull' value 'true') + ); +end; +/ +begin + dbms_cloud.copy_data( + table_name =>'TIMES', + credential_name =>'OBJ_STORE_CRED', + file_uri_list =>'&base_URL/time_v3.dat', + format => json_object('ignoremissingcolumns' value 'true', 'removequotes' value 'true', 'dateformat' value 'YYYY-MM-DD-HH24-MI-SS', 'blankasnull' value 'true') + ); +end; +/ +begin + dbms_cloud.copy_data( + table_name =>'COSTS', + credential_name =>'OBJ_STORE_CRED', + file_uri_list =>'&base_URL/costs.dat', + format => json_object('ignoremissingcolumns' value 'true', 'dateformat' value 'YYYY-MM-DD', 'blankasnull' value 'true') + ); +end; +/ diff --git a/compliance-submission-oci-genai/query/images/sample1.png b/compliance-submission-oci-genai/query/images/sample1.png new file mode 100644 index 0000000..2d9ad47 Binary files /dev/null and b/compliance-submission-oci-genai/query/images/sample1.png differ diff --git a/compliance-submission-oci-genai/query/images/sample2.png b/compliance-submission-oci-genai/query/images/sample2.png new file mode 100644 index 0000000..145c9d3 Binary files /dev/null and b/compliance-submission-oci-genai/query/images/sample2.png differ diff --git a/compliance-submission-oci-genai/query/query.md b/compliance-submission-oci-genai/query/query.md new file mode 100644 index 0000000..fe5d3a5 --- /dev/null +++ b/compliance-submission-oci-genai/query/query.md @@ -0,0 +1,92 @@ +# Title of the Lab + +## Introduction + +*Describe the lab in one or two sentences, for example:* This lab walks you through the steps to ... + +Estimated Lab Time: -- minutes + +### About (Optional) +Enter background information here about the technology/feature or product used in this lab - no need to repeat what you covered in the introduction. Keep this section fairly concise. If you find yourself needing more than two sections/paragraphs, please utilize the "Learn More" section. + +### Objectives + +*List objectives for this lab using the format below* + +In this lab, you will: +* Objective 1 +* Objective 2 +* Objective 3 + +### Prerequisites (Optional) + +*List the prerequisites for this lab using the format below. Fill in whatever knowledge, accounts, etc. is necessary to complete the lab. Do NOT list each previous lab as a prerequisite.* + +This lab assumes you have: +* An Oracle Cloud account +* All previous labs successfully completed + + +*This is the "fold" - below items are collapsed by default* + +## Task 1: Concise Task Description + +(optional) Task 1 opening paragraph. + +1. Step 1 + + ![Image alt text](images/sample1.png) + +2. Step 2 + + ![Image alt text](images/sample1.png) + +4. Example with inline navigation icon ![Image alt text](images/sample2.png) click **Navigation**. + +5. Example with bold **text**. + + If you add another paragraph, add 3 spaces before the line. + +## Task 2: Concise Task Description + +1. Step 1 - tables sample + + Use tables sparingly: + + | Column 1 | Column 2 | Column 3 | + | --- | --- | --- | + | 1 | Some text or a link | More text | + | 2 |Some text or a link | More text | + | 3 | Some text or a link | More text | + +2. You can also include bulleted lists - make sure to indent 4 spaces: + + - List item 1 + - List item 2 + +3. Code examples + + ``` + Adding code examples + Indentation is important for the code example to appear inside the step + Multiple lines of code + Enclose the text you want to copy in . + ``` + +4. Code examples that include variables + + ``` + ssh -i + ``` + +## Learn More + +*(optional - include links to docs, white papers, blogs, etc)* + +* [URL text 1](http://docs.oracle.com) +* [URL text 2](http://docs.oracle.com) + +## Acknowledgements +* **Author** - +* **Contributors** - -- optional +* **Last Updated By/Date** - diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-bucket-1.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-bucket-1.png new file mode 100644 index 0000000..2eb3865 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-bucket-1.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-bucket-2.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-bucket-2.png new file mode 100644 index 0000000..29adb85 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-bucket-2.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-bucket-3.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-bucket-3.png new file mode 100644 index 0000000..344d349 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-bucket-3.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-comp-1.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-comp-1.png new file mode 100644 index 0000000..9ead843 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-comp-1.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-comp-2.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-comp-2.png new file mode 100644 index 0000000..eaa38d5 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-comp-2.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-comp-3.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-comp-3.png new file mode 100644 index 0000000..e99c444 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-comp-3.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-compute-0.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-compute-0.png new file mode 100644 index 0000000..8a2852b Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-compute-0.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-compute-1.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-compute-1.png new file mode 100644 index 0000000..0bb56d3 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-compute-1.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-compute-2.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-compute-2.png new file mode 100644 index 0000000..ee0c1e0 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-compute-2.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-compute-3.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-compute-3.png new file mode 100644 index 0000000..f4d32f5 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-compute-3.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-1.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-1.png new file mode 100644 index 0000000..96db4b2 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-1.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-2.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-2.png new file mode 100644 index 0000000..a54c30f Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-2.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-3.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-3.png new file mode 100644 index 0000000..a5ae302 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-3.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-4.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-4.png new file mode 100644 index 0000000..1da37bd Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-4.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-5.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-5.png new file mode 100644 index 0000000..b205bc5 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-5.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-6.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-6.png new file mode 100644 index 0000000..1a552b8 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-6.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-7.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-7.png new file mode 100644 index 0000000..c34c28a Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-7.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-8.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-8.png new file mode 100644 index 0000000..2cf2394 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-8.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-9.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-9.png new file mode 100644 index 0000000..dd66f95 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-9.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-0.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-0.png new file mode 100644 index 0000000..0b38167 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-0.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-1.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-1.png new file mode 100644 index 0000000..d8fbf09 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-1.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-10.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-10.png new file mode 100644 index 0000000..0f77156 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-10.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-2.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-2.png new file mode 100644 index 0000000..a9c34a3 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-2.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-3.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-3.png new file mode 100644 index 0000000..98ad8a1 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-3.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-4.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-4.png new file mode 100644 index 0000000..aa4976b Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-4.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-5.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-5.png new file mode 100644 index 0000000..5ef0546 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-5.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-6.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-6.png new file mode 100644 index 0000000..0a0d354 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-6.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-7.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-7.png new file mode 100644 index 0000000..fc9a216 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-7.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-8.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-8.png new file mode 100644 index 0000000..454f308 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-8.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-9.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-9.png new file mode 100644 index 0000000..0759fb7 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-ds-pol-9.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-1.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-1.png new file mode 100644 index 0000000..7b85b32 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-1.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-10.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-10.png new file mode 100644 index 0000000..68e971a Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-10.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-11.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-11.png new file mode 100644 index 0000000..4c3f629 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-11.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-12.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-12.png new file mode 100644 index 0000000..8802fe1 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-12.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-2.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-2.png new file mode 100644 index 0000000..667c77c Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-2.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-3.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-3.png new file mode 100644 index 0000000..6f90942 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-3.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-4.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-4.png new file mode 100644 index 0000000..d7b509b Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-4.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-5.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-5.png new file mode 100644 index 0000000..66af65d Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-5.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-6.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-6.png new file mode 100644 index 0000000..cd2c8a4 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-6.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-7.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-7.png new file mode 100644 index 0000000..3fd5453 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-7.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-8.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-8.png new file mode 100644 index 0000000..db85ac7 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-8.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-9.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-9.png new file mode 100644 index 0000000..d11448c Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-os-9.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-pol-1.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-pol-1.png new file mode 100644 index 0000000..06cdfd2 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-pol-1.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-pol-2.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-pol-2.png new file mode 100644 index 0000000..295bbbc Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-pol-2.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-pol-3.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-pol-3.png new file mode 100644 index 0000000..c7b909d Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-pol-3.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-sec-1.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-sec-1.png new file mode 100644 index 0000000..8d1c897 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-sec-1.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-sec-2.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-sec-2.png new file mode 100644 index 0000000..37d7748 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-sec-2.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-sec-3.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-sec-3.png new file mode 100644 index 0000000..75eaf2a Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-sec-3.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-sec-4.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-sec-4.png new file mode 100644 index 0000000..17985cc Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-sec-4.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-sec-5.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-sec-5.png new file mode 100644 index 0000000..2741071 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-sec-5.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-sec-6.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-sec-6.png new file mode 100644 index 0000000..cc12b3d Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-sec-6.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-sec-7.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-sec-7.png new file mode 100644 index 0000000..d36e4f7 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-sec-7.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-sec-8.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-sec-8.png new file mode 100644 index 0000000..6afe51f Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-sec-8.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-tenancy-1.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-tenancy-1.png new file mode 100644 index 0000000..d117679 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-tenancy-1.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-tenancy-2-1.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-tenancy-2-1.png new file mode 100644 index 0000000..45defe0 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-tenancy-2-1.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-tenancy-2.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-tenancy-2.png new file mode 100644 index 0000000..6d8a5aa Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-tenancy-2.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-tenancy-3.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-tenancy-3.png new file mode 100644 index 0000000..9e21a40 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-tenancy-3.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-vcn-0.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-vcn-0.png new file mode 100644 index 0000000..69c7ca1 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-vcn-0.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-vcn-1.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-vcn-1.png new file mode 100644 index 0000000..61f6dd1 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-vcn-1.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-vcn-2.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-vcn-2.png new file mode 100644 index 0000000..eb474c7 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-vcn-2.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-vcn-3.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-vcn-3.png new file mode 100644 index 0000000..c6c7ad6 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-vcn-3.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-vcn-4.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-vcn-4.png new file mode 100644 index 0000000..d000033 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-vcn-4.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-vcn-5.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-vcn-5.png new file mode 100644 index 0000000..587664f Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-vcn-5.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/images/lab1-vcn-6.png b/compliance-submission-oci-genai/setup-oci-services/images/lab1-vcn-6.png new file mode 100644 index 0000000..849d085 Binary files /dev/null and b/compliance-submission-oci-genai/setup-oci-services/images/lab1-vcn-6.png differ diff --git a/compliance-submission-oci-genai/setup-oci-services/setup-oci-services.md b/compliance-submission-oci-genai/setup-oci-services/setup-oci-services.md new file mode 100644 index 0000000..8ee5fdb --- /dev/null +++ b/compliance-submission-oci-genai/setup-oci-services/setup-oci-services.md @@ -0,0 +1,205 @@ +# LAB-1: Setting up OCI Services + +## Introduction + +In this lab, we perform setting the Oracle cloud environment for running the workshop successfully. + +Estimated Lab Time: -- 15 minutes + +### Objectives + +In this lab, you will: + +* Set up the workshop compartment +* Set up OCI Networking components +* Set up OCI object storage buckets +* Set up OCI Compute instance +* Set up OCI Opensearch Service +* Set up OCI Data science service + +### Prerequisites + +This lab assumes you have: + +* An Oracle Cloud account with administrative access +* Working familiarity in the OCI cloud environment +* Familiarity with cloud services, access, networking + +### Recommendations + +This workshop uses the name **clinical-trials** for most of the objects in the workshop for ease of reference. Some recommendations - + +1. Use **clinical-trials** for naming all cloud resources +2. Use same password for accessing all cloud resources +3. Please note that this is a recommendation for easy working with the workshop + +### Prerequisites + +* You must have your email address registered with Oracle to provision a cloud tenancy +* You must have administrator privileges in your tenancy + +## Task 1: Setup compartment + +1. Pull down the Oracle cloud hamburger menu and search for compartments + ![Setup Compartment](images/lab1-comp-1.png) +2. Create the compartment called clinical-trials under the root compartment. Verify it is successfully created as shown below + ![Setup Compartment](images/lab1-comp-2.png) + ![Setup Compartment](images/lab1-comp-3.png) + +## Task 2: Setup OCI Networking components + +1.Setup a virtual cloud network (VCN) called "clinical-trials" in the clinical-trials compartment. Use VCN wizard option (shown below) + + ![setup VCN](images/lab1-vcn-0.png) + ![setup VCN](images/lab1-vcn-1.png) + ![setup VCN](images/lab1-vcn-2.png) + ![setup VCN](images/lab1-vcn-3.png) + ![setup VCN](images/lab1-vcn-4.png) + ![setup VCN](images/lab1-vcn-5.png) + ![setup VCN](images/lab1-vcn-6.png) + + 2.Add two Ingress rules entry to the private subnet security list for ports 9200 and 5601 for Opensearch access + + ![setup VCN](images/lab1-sec-1.png) + ![setup VCN](images/lab1-sec-2.png) + ![setup VCN](images/lab1-sec-3.png) + ![setup VCN](images/lab1-sec-4.png) + ![setup VCN](images/lab1-sec-5.png) + ![setup VCN](images/lab1-sec-6.png) + ![setup VCN](images/lab1-sec-7.png) + ![setup VCN](images/lab1-sec-8.png) + +## Task 3: Set up OCI Object storage buckets + +1. Setup an object storage bucket in the clinical-trials compartment as shown below + ![setup Object storage bucket](images/lab1-bucket-1.png) + ![setup Object storage bucket](images/lab1-bucket-2.png) + ![setup Object storage bucketN](images/lab1-bucket-3.png) + +## Task 4: Set up an OCI Compute instance + +1. Setup an OCI compute instance *clinical-trials* in the *clinical-trials* compartment. This is required for tunneling into the OCI Opensearch instance from your laptop. + ![setup OCI Compute](images/lab1-compute-0.png) + ![setup OCI Compute](images/lab1-compute-1.png) + +2. Generate a new SSH key pair and download both the private and public key pair to your laptop. You will need this to SSH login to the compute instance + ![setup OCI Compute](images/lab1-compute-2.png) + +3. Make sure that your compute instance is provisioned and is running. + ![setup OCI Compute](images/lab1-compute-3.png) + +## Task 5: Set up an OCI Opensearch service + +This involves setting up OCI Opensearch policies and creating an Opensearch instance. Please note that it can take up to 10 minutes to provision an Opensearch service instance + +1.Setup OCI Opensearch policy "clinical-trials" in the "clinical_trials" compartment as shown below + ![setup Opensearch policy](images/lab1-pol-1.png) + ![setup Opensearch policy](images/lab1-pol-2.png) + +2.Issue the following code in the policy builder manual editor below and hit create + +```text + +allow group SearchOpenSearchAdmins to manage vnics in compartment clinical-trials +allow group SearchOpenSearchAdmins to manage vcns in compartment clinical-trials +allow group SearchOpenSearchAdmins to manage subnets in compartment clinical-trials +allow group SearchOpenSearchAdmins to manage network-security-groups in compartment clinical-trials +allow group SearchOpenSearchAdmins to manage opensearch-family in compartment clinical-trials +allow group SearchOpenSearchAdmins to manage vnics in compartment clinical-trials + +``` + +3. Set up the opensearch policies as shown below + + ![setup Opensearch policy](images/lab1-pol-3.png) + +3.Create the Opensearch cluster clinical-trials in the clinical-trials compartment + ![setup Opensearch cluster](images/lab1-os-1.png) + ![setup Opensearch cluster](images/lab1-os-2.png) + ![setup Opensearch cluster](images/lab1-os-3.png) + ![setup Opensearch cluster](images/lab1-os-11.png) + +4.Enter **osmaster** as the user and your preferred common password. Create the cluster with all the default options as shown below + + ![setup Opensearch cluster](images/lab1-os-4.png) + ![setup Opensearch cluster](images/lab1-os-5.png) + ![setup Opensearch cluster](images/lab1-os-6.png) + ![setup Opensearch cluster](images/lab1-os-7.png) + ![setup Opensearch cluster](images/lab1-os-8.png) + ![setup Opensearch cluster](images/lab1-os-12.png) + +5.Note down the API endpoint , Private IP , Opensearch Dashboard API endpoint and Dashboard Private IP for later usage + +## Task 6: Set up the OCI Data science service policies + +Service policies are required for Data science environment setup. This workshop requires tenancy level policies to be setup at the root compartment level. + +1. Go to the OCI Data science service. + + ![setup OCI Data Science policy](images/lab1-ds-pol-0.png) + +2. Locate the Data science policies as shown to follow the steps + + ![setup OCI Data Science policy](images/lab1-ds-pol-1.png) + +3.Create a dynamic group for Data science. Dynamic group provides easy and secure authenticated run-time access to OCI resources for data science sessions + + ![setup OCI Data Science policy](images/lab1-ds-pol-2.png) + ![setup OCI Data Science policy](images/lab1-ds-pol-3.png) + ![setup OCI Data Science policy](images/lab1-ds-pol-4.png) + ![setup OCI Data Science policy](images/lab1-ds-pol-5.png) + ![setup OCI Data Science policy](images/lab1-ds-pol-6.png) + +4.Setup a policy called *clinical-trials* in the **root** compartment of the tenancy as shown below + + ![setup OCI Data Science policy](images/lab1-ds-pol-7.png) + ![setup OCI Data Science policy](images/lab1-ds-pol-8.png) + +5.Use the following policy statements and hit create + +```text + +allow service datascience to use virtual-network-family in tenancy +allow group Administrators to use virtual-network-family in tenancy +allow group Administrators to manage data-science-family in tenancy +allow dynamic-group clinical-trials to manage data-science-family in tenancy +allow group Administrators to manage generative-ai-family in tenancy + +``` + + ![setup OCI Data Science policy](images/lab1-ds-pol-10.png) + +## Task 7: Set up an OCI Data science project + +1. Setup an OCI Data science project + + ![setup OCI Data Science project](images/lab1-ds-1.png) + ![setup OCI Data Science project](images/lab1-ds-2.png) + ![setup OCI Data Science project](images/lab1-ds-3.png) + +## Task 8: Set up an OCI Data science Notebook session + +1.Create a notebook session called *clinical-trials* with the compute CPU and memory specifications as shown below. Select the *Custom Networking* option with a *private subnet* for the build. + + ![setup OCI Data Science notebook session](images/lab1-ds-4.png) + ![setup OCI Data Science notebook session](images/lab1-ds-5.png) + +2.Use a minimum compute shape of 4oCPU and 64GB of memory from a CPU based shape such as VM.Standard.E4.Flex + + ![setup OCI Data Science notebook session](images/lab1-ds-6.png) + ![setup OCI Data Science notebook session](images/lab1-ds-7.png) + ![setup OCI Data Science notebook session](images/lab1-ds-8.png) + +3.Verify that you are able to login to the data science environment and view the Jupyter Lab notebook session. + + ![setup OCI Data Science notebook session](images/lab1-ds-9.png) + +## Learn More + +* [OCI Opensearch setup](https://apexapps.oracle.com/pls/apex/r/dbpm/livelabs/view-workshop?wid=3427) +* [OCI Data science setup](https://docs.oracle.com/en-us/iaas/data-science/data-science-tutorial/get-started.htm) + +## Acknowledgements + +* **Author** - Rajib Ghosh, Master Principal Cloud Architect, OCI AI and Gen AI Center of Excellence +* **Last Updated By/Date** - Aug 2024 diff --git a/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-compute-3.png b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-compute-3.png new file mode 100644 index 0000000..f4d32f5 Binary files /dev/null and b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-compute-3.png differ diff --git a/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-note-os-1.png b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-note-os-1.png new file mode 100644 index 0000000..67f39f4 Binary files /dev/null and b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-note-os-1.png differ diff --git a/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-note-os-2.png b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-note-os-2.png new file mode 100644 index 0000000..2daee96 Binary files /dev/null and b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-note-os-2.png differ diff --git a/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-note-os-3.png b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-note-os-3.png new file mode 100644 index 0000000..4869ccd Binary files /dev/null and b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-note-os-3.png differ diff --git a/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-note-os-4.png b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-note-os-4.png new file mode 100644 index 0000000..61623ef Binary files /dev/null and b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-note-os-4.png differ diff --git a/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-note-os-5.png b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-note-os-5.png new file mode 100644 index 0000000..d58a1b4 Binary files /dev/null and b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-note-os-5.png differ diff --git a/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-10.png b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-10.png new file mode 100644 index 0000000..5b14131 Binary files /dev/null and b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-10.png differ diff --git a/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-11.png b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-11.png new file mode 100644 index 0000000..4bb3e45 Binary files /dev/null and b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-11.png differ diff --git a/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-12.png b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-12.png new file mode 100644 index 0000000..988b785 Binary files /dev/null and b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-12.png differ diff --git a/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-13.png b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-13.png new file mode 100644 index 0000000..8802fe1 Binary files /dev/null and b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-13.png differ diff --git a/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-2.png b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-2.png new file mode 100644 index 0000000..857d103 Binary files /dev/null and b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-2.png differ diff --git a/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-3.png b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-3.png new file mode 100644 index 0000000..1a77b03 Binary files /dev/null and b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-3.png differ diff --git a/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-4.png b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-4.png new file mode 100644 index 0000000..fd3fbb5 Binary files /dev/null and b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-4.png differ diff --git a/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-5.png b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-5.png new file mode 100644 index 0000000..9385b9a Binary files /dev/null and b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-5.png differ diff --git a/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-6.png b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-6.png new file mode 100644 index 0000000..8706e2f Binary files /dev/null and b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-6.png differ diff --git a/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-7.png b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-7.png new file mode 100644 index 0000000..b706b0d Binary files /dev/null and b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-7.png differ diff --git a/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-8.png b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-8.png new file mode 100644 index 0000000..91739cb Binary files /dev/null and b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-8.png differ diff --git a/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-9.png b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-9.png new file mode 100644 index 0000000..ab7bccf Binary files /dev/null and b/compliance-submission-oci-genai/using-neural-and-hybrid-search/images/lab6-os-9.png differ diff --git a/compliance-submission-oci-genai/using-neural-and-hybrid-search/using-neural-and-hybrid-search.md b/compliance-submission-oci-genai/using-neural-and-hybrid-search/using-neural-and-hybrid-search.md new file mode 100644 index 0000000..500711c --- /dev/null +++ b/compliance-submission-oci-genai/using-neural-and-hybrid-search/using-neural-and-hybrid-search.md @@ -0,0 +1,177 @@ +# Evaluating with Opensearch Neural and Hybrid search (Optional) + +## Introduction + +This is an optional lab, primarily geared for users who are familiar with OCI Opensearch. This lab shows you how to interact with OCI Opensearch console interface and run basic commands such as creating indexes, running queries to view results. It also shows how Machine learning based Neural search can be configured with OCI Opensearch and hybrid search can be run for this workshop + +Estimated Lab Time: -- 10 minutes + +### Oracle cloud Infrastructure Opensearch + +OCI Search with OpenSearch is a fully managed open source service that makes it easy to deploy, operate, and scale OpenSearch on customizable infrastructure. The cloud implementation of indexed search product is suitable for both lexical and neural searches based on AI/ML models. The product features creation of vector embeddings within the product as well as support for external embeddings only as a vector store. It also flexibility in finer model tuning and weight adjustments in a hybrid or combination search scenario. This flexibility along with its managed service capability makes it a good platform for Retrieval Augmented Generation (RAG) implementation for both small or high dimensional indexed searches. + +### Objectives + +In this lab, you will: + +* Access the Opensearch cluster dashboard from your laptop +* Use Opensearch UI tools such as dashboards and Index Management tools +* View clinical trial indexes and indexed data +* Test Opensearch access from OCI Data science environment +* Use Opensearch *dev tools* to create neural search models +* Use ingestion pipelines to create vector embeddings +* Use search pipelines and indexes to perform custom hybrid search (i,e lexical and neural together) + +### Prerequisites + +This lab assumes you have: + +* An Oracle Cloud account with US-Chicago region access +* You have successfully provisioned an OCI Opensearch cluster +* You have familiarity with Opensearch commands (GET/PUT/POST..) +* Some familiarity with using pre-trained ML models is helpful but not required + +## Task 1: Login into Opensearch instance and invoke Opensearch UI + +1. Create a windows batch or shell script substituting the following as shown below and execute in your laptop to login to SSH tunnel in opensearch instance + +```text + +ssh -C -v -t -L 127.0.0.1:5601:(Opensearch_dashboard_ip):5601 -L 127.0.0.1:9200:(Opensearch_api_private_ip):9200 opc@(compute instance_public_ip) -i (ssh_key_to_login_to_compute_instance) + +``` + +Refer to the following screenshots for pulling up the above relevant details + ![Install pip libraries](images/lab6-os-13.png) + ![Install pip libraries](images/lab6-compute-3.png) + +## Task 2: Create the Opensearch model group + +1.Search for an existing clinical-trial model group. + +```text + +#Search for the clinical trials model group +GET /_plugins/_ml/model_groups/_search +{ + "query": { + "match": {"name": "Clinical-trials"} + } +} + +``` + +2.If it does not exist then copy the code below to create it. It should display the model_group_id. Make a note of it + +```text + +# create a model group +POST /_plugins/_ml/model_groups/_register +{ + "name": "Clinical-trials", + "description": "A model group for clinical trials local models" +} + +``` + +3.To test that the model group is created, run code in step 1. The Id field in the output will contain *model_group_id* + +4.Use the code below if you need to delete the model group. Note that any models in the group must be deleted first. + +```text + +# Grab the model_group_id and delete the model group +DELETE _plugins/_ml/model_groups/(model_group_id) + +``` + +## Task 3: Create the Opensearch model + +1.Substitute the retrieved *model-group-id* in the code below to create the model. This currently pulls the model from a common object store location for OCI compliance that will expire. However, if you have issues pulling this model or running the code below, you may pull the model directly from hugging face or Opensearch pre-trained model documentation as well. + +```text + +# Create the model +POST /_plugins/_ml/models/_register +{ + "model_group_id": "(model-group-id)", + "name": "sentence-transformers/msmarco-distilbert-base-tas-b", + "version": "1.0.2", + "description": "This is a port of the DistilBert TAS-B Model to sentence-transformers model: It maps sentences & paragraphs to a 768 dimensional dense vector space and is optimized for the task of semantic search. This model version automatically truncates to a maximum of 512 tokens.", + "model_format": "TORCH_SCRIPT", + "model_task_type": "TEXT_EMBEDDING", + "model_config": { + "model_type": "distilbert", + "embedding_dimension": 768, + "framework_type": "sentence_transformers", + "pooling_mode": "CLS", + "normalize_result": false, + "all_config": "{\"_name_or_path\": \"/root/.cache/torch/sentence_transformers/sentence-transformers_msmarco-distilbert-base-tas-b/\", \"activation\": \"gelu\", \"architectures\": [\"DistilBertModel\"], \"attention_dropout\": 0.1, \"dim\": 768, \"dropout\": 0.1, \"hidden_dim\": 3072, \"initializer_range\": 0.02, \"max_position_embeddings\": 512, \"model_type\": \"distilbert\", \"n_heads\": 12, \"n_layers\": 6, \"pad_token_id\": 0, \"qa_dropout\": 0.1, \"seq_classif_dropout\": 0.2, \"sinusoidal_pos_embds\": false, \"tie_weights_\": true, \"torch_dtype\": \"float32\", \"transformers_version\": \"4.33.2\", \"vocab_size\": 30522}" + }, + "model_content_size_in_bytes": 266363799, + "model_content_hash_value": "54ee88869b39f0b7d7cede249409286333d152a49146259d271041fef5d39f03", + "url_connector": { + "protocol": "oci_sigv1", + "parameters": { + "auth_type": "resource_principal" + }, + "actions": [ + { + "method": "GET", + "action_type": "DOWNLOAD", + "url": "https://orasenatdpltintegration03.objectstorage.us-chicago-1.oci.customer-oci.com/p/HhDzaubWOjd86EHa5dzxBBDhGF6Wp2h42r97X60Dy8Z-tJRaFeWhslO0e-1n9a2E/n/orasenatdpltintegration03/b/clinical-trials/o/sentence-transformers_msmarco-distilbert-base-tas-b-1.0.2-torch_script.zip" + } + ] + } +} + +``` + +2.Record the *task_id*. Please note that Opensearch *model_id* can be retrieved from the registration *task_id* for the model register job + ![Install pip libraries](images/lab6-os-8.png) + +3.Retrieve the *model_id* + ![Install pip libraries](images/lab6-os-9.png) + +4.Deploy and verify the model + ![Install pip libraries](images/lab6-os-10.png) + ![Install pip libraries](images/lab6-os-11.png) + +5.Test the model embeddings created with a query + ![Install pip libraries](images/lab6-os-12.png) + +## Task 4: Using Hybrid search notebook + +1. Open up to execute the notebook *demo_hybrid_search.ipynb*. You can enter "Shift+enter* to execute the notebook cells one by one + +2. Enter the compartment id and the opensearch credentials + ![Enter OS credentials](images/lab6-note-os-1.png) + +3. Enter the recorded *model_id* retrieved from the Task + ![Enter OS model_id](images/lab6-note-os-2.png) + +4. Note the weight definitions of the search pipeline and how a combination of a match query and neural search query on description embeddings is used to control the retriever outputs + +5. Run the notebook cells and view the retrieval score + ![Enter OS model_id](images/lab6-note-os-3.png) + +6. Reverse the weights in the search pipeline and recreate the pipeline + ![Enter OS model_id](images/lab6-note-os-4.png) + +7. Run your query again to view the results + ![Enter OS model_id](images/lab6-note-os-5.png) + +## Task 5: Analyzing search results + +1. Validating and comparing hybrid search query results is complex and requires experimentation to find the right combination of weights that work for the data. As you see that top record returned is independent of the weights but the other records are affected and different titles with scores are reported. + +## Learn More + +* [Opensearch hybrid search](https://opensearch.org/docs/latest/search-plugins/hybrid-search/#step-4-configure-a-search-pipeline) +* [Search Documents stored in Object Storage using Opensearch, Generative AI, Semantic Search, RAG](https://apexapps.oracle.com/pls/apex/r/dbpm/livelabs/view-workshop?wid=3762) + +## Acknowledgements + +* **Author** - Rajib Ghosh, Master Principal Cloud Architect, OCI AI and Gen AI Center of Excellence +* **Last Updated By/Date** - Aug 2024 + diff --git a/compliance-submission-oci-genai/vectorize-load-and-evaluate/images/lab4-note-mod-1.png b/compliance-submission-oci-genai/vectorize-load-and-evaluate/images/lab4-note-mod-1.png new file mode 100644 index 0000000..6a893d4 Binary files /dev/null and b/compliance-submission-oci-genai/vectorize-load-and-evaluate/images/lab4-note-mod-1.png differ diff --git a/compliance-submission-oci-genai/vectorize-load-and-evaluate/images/lab4-note-oci-gai-1.png b/compliance-submission-oci-genai/vectorize-load-and-evaluate/images/lab4-note-oci-gai-1.png new file mode 100644 index 0000000..a521fc8 Binary files /dev/null and b/compliance-submission-oci-genai/vectorize-load-and-evaluate/images/lab4-note-oci-gai-1.png differ diff --git a/compliance-submission-oci-genai/vectorize-load-and-evaluate/images/lab4-note-os-1.png b/compliance-submission-oci-genai/vectorize-load-and-evaluate/images/lab4-note-os-1.png new file mode 100644 index 0000000..67f39f4 Binary files /dev/null and b/compliance-submission-oci-genai/vectorize-load-and-evaluate/images/lab4-note-os-1.png differ diff --git a/compliance-submission-oci-genai/vectorize-load-and-evaluate/vectorize-load-and-evaluate.md b/compliance-submission-oci-genai/vectorize-load-and-evaluate/vectorize-load-and-evaluate.md new file mode 100644 index 0000000..8ea4575 --- /dev/null +++ b/compliance-submission-oci-genai/vectorize-load-and-evaluate/vectorize-load-and-evaluate.md @@ -0,0 +1,87 @@ +# Vectorize, load and search and evaluate retrieval + +## Introduction + +This lab focuses on creating vector embeddings with the help of an open source pre-trained as well as OCI Generative AI embedding model. The purpose is to show a comparison of the retrieval search scores between two embedding models. This enables you use this as a test bed to test multiple embedding models and plugin the best possible embedding model best suited for your data in the Retrieval Augmented Generation (RAG) scenario + +Estimated Lab Time: -- 10 minutes + +### About Embedding models + +In this lab, we use a Hugging face pre-trained sentence transformer embedding model all-MiniLM-L6-v2 and Oracle cloud Generative AI Cohere command Embedding model to create vectors store in OCI Opensearch index. Embedding models vary by dimensions and algorithms and are also based of the volume of dataset searched. Retrieval scores can be better based on higher dimensionality of the model. In this lab, you have the option to check embedding dimension of the model you are using. + +### Objectives + +In this lab, you will: + +* Create vector embeddings on textual clinical data +* Create vector embeddings on short (title) and long (description) textual data +* Create vector embeddings with an open source pre-trained model and OCI Generative AI embedding model +* Compare query retrieval scores across various models and various short and long text segments. + +### Prerequisites + +This lab assumes you have: + +* Knowledge of Oracle cloud services +* Some knowledge of OCI Data science and OCI Opensearch +* Programmatic understanding of python +* Some understanding of vector embeddings, Generative AI and Large language models (LLMs) +* Some understanding of the langchain framework + +## Task 1: Vectorize, load and query data using open source pre-trained model + +1. Get the following information into a notepad or a script + + * Compartment OCID for *clinical-trials* compartment. (Search on OCI console for compartments, click your compartment and copy the OCID) + * Opensearch username - The username entered while provisioning Opensearch cluster (i,e *osmaster*) + * Opensearch password - The password entered while provisioning Opensearch cluster + * API end point Private IP from OCI Opensearch service console + +2. Double click to open up the notebook *demo-vector-search-ext.ipynb*. Run each of the cells one by one from top by using *Shift+Enter* or play button at the top + +3. Substitute the following definitions in the cell as shown below + ![Image alt text](images/lab4-note-os-1.png) + +4. Run the next cell to make sure the Opensearch client connectivity works. It should show the *OpenSearch([{'host': 'hostname', 'port': 9200}])* as outputs + +5. Create vector embeddings with the Open source pre-trained model *all-minilm-l6-v2* and load them with the corresponding texts into an index *idx-all-minilm-l6-v2* . Note that vector embeddings are created by the model on the clinical-trial metadata file *clinical-trial-titles.csv* that contains the Filename, Title and Descriptions only + +6. In the next few cells, you can execute your queries against Title and Descriptions embeddings in Opensearch index and retrieve the scores. + +## Task 2: Vectorize, load and query data using OCI Generative AI + +1. Run the remaining cells from the section highlighting **Vector embeddings and search with OCI Generative AI** + +2. Create the corresponding index and retrieve scores on vector searches based on OCI Generative AI embeddings + +## Task 3: Compare retrieval scores + +Retriever scores are an indication of how accurately the embedding model can pull up query results based on the query texts. + +1. The following shows a retriever score comparison between the *all-minilm-l6-v2* and OCI Generative AI *cohere.embed-english-v3.0* model + + * ![Open source model score](images/lab4-note-mod-1.png) + * ![OCI GAI model score](images/lab4-note-oci-gai-1.png) + +## Task 4: Testing to find the best retrievers (RAG) for your data + +1. Some notable points to observe + * How the selected model is used for query embedding in the notebook + * Search can be performed on short text embeddings (i,e Title) or long texts (i,e Descriptions) + * Different sentence transformer models can be chosen and their scores can be evaluated + * Change the *num_titles* parameter to configure the top *n* documents by score + * Increase Text Description length or context to see how scores change + * Increase volume of clinical-trials metadata in the *clinical-trial-titles.csv* data + +2. You can then test to find the embedding model that works best for your data + +## Learn More + +* [OpenSearch provided pre-trained models](https://opensearch.org/docs/latest/ml-commons-plugin/pretrained-models/) +* [Conversational Search with OCI Generative AI](https://docs.oracle.com/en-us/iaas/Content/search-opensearch/Concepts/ocigenaiconnector.htm) + +## Acknowledgements + +* **Author** - Rajib Ghosh, Master Principal Cloud Architect, OCI AI and Gen AI Center of Excellence +* **Last Updated By/Date** - Aug 2024 diff --git a/compliance-submission-oci-genai/workshops/tenancy/index.html b/compliance-submission-oci-genai/workshops/tenancy/index.html new file mode 100644 index 0000000..6acdb69 --- /dev/null +++ b/compliance-submission-oci-genai/workshops/tenancy/index.html @@ -0,0 +1,62 @@ + + + + + + + + + Oracle LiveLabs + + + + + + + + + + + + +
+
+
+
+
+
+
+
+ + + + + diff --git a/compliance-submission-oci-genai/workshops/tenancy/manifest.json b/compliance-submission-oci-genai/workshops/tenancy/manifest.json new file mode 100644 index 0000000..cb6ac60 --- /dev/null +++ b/compliance-submission-oci-genai/workshops/tenancy/manifest.json @@ -0,0 +1,52 @@ +{ + "workshoptitle": "Generate Draft regulatory documents for compliance with OCI Generative AI", + "include": {"conda.zip":"../../developing-with-oci-data-science/files/conda.zip"}, + "help": "livelabs-help-oci_us@oracle.com", + "tutorials": [ + { + "title": "Introduction", + "description": "The Introduction is always first. The title and contents menu title match for the Introduction.", + "filename": "../../introduction/introduction.md" + }, + { + "title": "Get Started", + "description": "This is the prerequisites for customers using Free Trial and Paid tenancies, and Always Free accounts (if applicable). The title of the lab and the Contents Menu title (the title above) match for Prerequisite lab. This lab is always first.", + "filename": "../../getting-started/getting-started.md" + }, + { + "title": "Lab 1: Setup OCI Services", + "description": "In this lab you will setup the required OCI services for running thr workshop", + "filename": "../../setup-oci-services/setup-oci-services.md" + }, + { + "title": "Lab 2: Generate Clinical trials with OCI Generative AI", + "filename": "../../generate-clinical-trials-with-oci-gai/generate-clinical-trials-with-oci-gai.md" + }, + { + "title": "Lab 3: Develop with OCI Data science", + "filename": "../../developing-with-oci-data-science/developing-with-oci-data-science.md" + }, + { + "title": "Lab 4: Vectorize, load and evaluate", + "filename": "../../vectorize-load-and-evaluate/vectorize-load-and-evaluate.md" + }, + { + "title": "Lab 5: Generate Compliance document", + "filename": "../../generate-compliance-document/generate-compliance-document.md" + }, + { + "title": "Lab 6: Use Neural and Hybrid search (Optional)", + "filename": "../../using-neural-and-hybrid-search/using-neural-and-hybrid-search.md" + }, + { + "title": "Need Help?", + "description": "Solutions to Common Problems and Directions for Receiving Live Help", + "filename":"https://oracle-livelabs.github.io/common/labs/need-help/need-help-freetier.md" + }, + { + "title": "Other LiveLabs you might like", + "description": "Other LiveLabs you might like", + "filename": "../../other-livelabs/other-livelabs.md" + } + ] +}