diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index d3de6258..47d8b109 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -1,2 +1,2 @@ # This should match the owning team set up in https://github.com/orgs/opensearch-project/teams -* @pjfitzgibbons @ps48 @kavithacm @derek-ho @joshuali925 @dai-chen @YANG-DB @rupal-bq @mengweieric @vamsi-amazon @swiddis @penghuo @seankao-az @anirudha \ No newline at end of file +* @lezzago @joshuali925 @anirudha @ruanyl @SuZhou-Joe @xluo-aws @gaobinlong @Hailong-am @wanglam @raintygao diff --git a/.whitesource b/.whitesource index db4b0fec..89842729 100644 --- a/.whitesource +++ b/.whitesource @@ -5,11 +5,28 @@ "projectToken": "", "baseBranches": [] }, + "scanSettingsSAST": { + "enableScan": false, + "scanPullRequests": false, + "incrementalScan": true, + "baseBranches": [], + "snippetSize": 10 + }, "checkRunSettings": { "vulnerableCheckRunConclusionLevel": "failure", - "displayMode": "diff" + "displayMode": "diff", + "useMendCheckNames": true + }, + "checkRunSettingsSAST": { + "checkRunConclusionLevel": "failure", + "severityThreshold": "high" }, "issueSettings": { - "minSeverityLevel": "LOW" + "minSeverityLevel": "LOW", + "issueType": "DEPENDENCY" + }, + "remediateSettings": { + "workflowRules": { + "enabled": true + } } -} \ No newline at end of file diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c970f9d3..c4b6a1c5 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,4 +1,59 @@ -## Contributing to this Project +# Contributing Guidelines -OpenSearch is a community project that is built and maintained by people just like **you**. -[This document](https://github.com/opensearch-project/.github/blob/main/CONTRIBUTING.md) explains how you can contribute to this and related projects. \ No newline at end of file +Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional +documentation, we greatly value feedback and contributions from our community. + +Please read through this document before submitting any issues or pull requests to ensure we have all the necessary +information to effectively respond to your bug report or contribution. + + +## Reporting Bugs/Feature Requests + +We welcome you to use the GitHub issue tracker to report bugs or suggest features. + +When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already +reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: + +* A reproducible test case or series of steps +* The version of our code being used +* Any modifications you've made relevant to the bug +* Anything unusual about your environment or deployment + + +## Contributing via Pull Requests +Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: + +1. You are working against the latest source on the *main* branch. +2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. +3. You open an issue to discuss any significant work - we would hate for your time to be wasted. + +To send us a pull request, please: + +1. Fork the repository. +2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. +3. Ensure local tests pass. +4. Commit to your fork using clear commit messages. +5. Send us a pull request, answering any default questions in the pull request interface. +6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. + +GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and +[creating a pull request](https://help.github.com/articles/creating-a-pull-request/). + + +## Finding contributions to work on +Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. + + +## Code of Conduct +This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). +For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact +opensource-codeofconduct@amazon.com with any additional questions or comments. + + +## Security issue notifications +If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. + + +## Licensing + +See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 9f4c0c74..e40f41d0 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -1,19 +1,36 @@ -## Developer Guide +- [Developer guide](#developer-guide) + - [Forking and Cloning](#forking-and-cloning) + - [Install Prerequisites](#install-prerequisites) + - [Environment Setup](#environment-setup) + - [Build](#build) + - [Run](#run) + - [Test](#test) + +## Developer guide So you want to contribute code to this project? Excellent! We're glad you're here. Here's what you need to do. -### Setup +### Forking and Cloning -1. Download OpenSearch for the version that matches the [OpenSearch Dashboards version specified in opensearch_dashboards.json](./opensearch_dashboards.json#L4) from [opensearch.org](https://opensearch.org/downloads.html). -1. Download the OpenSearch Dashboards source code for the [version specified in opensearch_dashboards.json](./opensearch_dashboards.json#L4) you want to set up. -1. Change your node version to the version specified in `.node-version` inside the OpenSearch Dashboards root directory. -1. cd into `OpenSearch-Dashboards` and remove the `plugins` directory. -1. Check out this package from version control as the `plugins/dashboards-assistant` directory. -```bash -git clone https://github.com/opensearch-project/dashboards-assistant plugins -git checkout main -``` -6. Run `yarn osd bootstrap` inside `OpenSearch-Dashboards`. +Fork this repository on GitHub, and clone locally with `git clone`. + +### Install Prerequisites + +You will need to install [node.js](https://nodejs.org/en/), [nvm](https://github.com/nvm-sh/nvm/blob/master/README.md), and [yarn](https://yarnpkg.com/) in your environment to properly pull down dependencies to build and bootstrap the plugin. + +### Environment Setup + +1. Download OpenSearch for the version that matches the [OpenSearch Dashboard version specified in package.json](./package.json#L9). +1. Download and install the appropriate [OpenSearch ML Commons plugin](https://github.com/opensearch-project/ml-commons). +1. Download the OpenSearch-Dashboards source code for the [version specified in package.json](./package.json#L9) you want to set up. + + See the [OpenSearch Dashboards developer guide](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/DEVELOPER_GUIDE.md) for more instructions on setting up your development environment. + +1. Change your node version to the version specified in `.node-version` inside the OpenSearch-Dashboards root directory. +1. cd into the `plugins` directory of the OpenSearch-Dashboards source code directory. +1. Check out this package from version control into the `plugins` directory. +1. Set `assistant.chat.enabled` to `true` in `opensearch_dashboards.yml` if you want to enable the chat feature. +1. Run `yarn osd bootstrap` inside `Opensearch-Dashboards/plugins/dashboards-assistant`. Ultimately, your directory structure should look like this: @@ -24,24 +41,40 @@ Ultimately, your directory structure should look like this: │ └── dashboards-assistant ``` -### Build -To build the plugin's distributable zip simply run `yarn build` in the plugin's directory. +## Build + +To build the plugin's distributable zip simply run `yarn build`. + +Example output: `./build/assistantDashboards-2.11.0.zip` -Example output: `./build/assistant*.zip` -### Run +## Run -cd back to `OpenSearch-Dashboards` directory and run `yarn start` to start OpenSearch Dashboards including this plugin. OpenSearch Dashboards will be available on `localhost:5601`. +- `yarn start` -### Submitting Changes + - Starts OpenSearch-Dashboards and includes this plugin. OpenSearch-Dashboards will be available on `localhost:5601`. + - Please run in the OpenSearch-Dashboards root directory + - You must have OpenSearch running with the ML Commons plugin + +## Test + +There are unit/stubbed integration tests. + +- `yarn test` + + - Runs the plugin unit tests. + +### Formatting + +This codebase uses Prettier as our code formatter. All new code that is added has to be reformatted using the Prettier version listed in `package.json`. In order to keep consistent formatting across the project developers should only use the prettier CLI to reformat their code using the following command: + +``` +yarn lint --fix +``` -See [CONTRIBUTING](CONTRIBUTING.md). +> NOTE: There also exists prettier plugins on several editors that allow for automatic reformatting on saving the file. However using this is discouraged as you must ensure that the plugin uses the correct version of prettier (listed in `package.json`) before using such a plugin. -### Backports +### Backport -The Github workflow in [`backport.yml`](.github/workflows/backport.yml) creates backport PRs automatically when the original PR -with an appropriate label `backport ` is merged to main with the backport workflow run successfully on the -PR. For example, if a PR on main needs to be backported to `1.x` branch, add a label `backport 1.x` to the PR and make sure the -backport workflow runs on the PR along with other checks. Once this PR is merged to main, the workflow will create a backport PR -to the `1.x` branch. \ No newline at end of file +- [Link to backport documentation](https://github.com/opensearch-project/opensearch-plugins/blob/main/BACKPORT.md) \ No newline at end of file diff --git a/GETTING_STARTED_GUIDE.md b/GETTING_STARTED_GUIDE.md new file mode 100644 index 00000000..a8feb955 --- /dev/null +++ b/GETTING_STARTED_GUIDE.md @@ -0,0 +1,153 @@ +## Getting started guide + +### How to run assistant on your own machine +Below are the set of steps to run OpenSearch and OpenSearch dashboards with the OpenSearch assistant and the query generation functionality in the Observability Log Explorer page correctly on the cluster. +**Note** that the `feature/langchain` is the branch used in this guide. + +1. Follow steps here to setup docker for OpenSearch: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/ + 1. Note: When running docker pull, use this command instead: `docker pull public.ecr.aws/w1m7p7g2/opensearch-reinvent2023:latest` + 2. Note: This docker image may get incrementally updated over time. + + +2. Follow steps here to setup docker to OpenSearch Dashboards: https://opensearch.org/docs/latest/install-and-configure/install-dashboards/docker/ + 1. Note: When running docker pull, use this command instead for OSD: `docker pull public.ecr.aws/w1m7p7g2/opensearch-dashboards-reinvent2023:latest` + 2. Note: This docker image may get incrementally updated over time. + 3. If you want to enable the chat assistant feature, set `assistant.chat.enabled` to `true` in the `opensearch_dashboards.yml` file. +3. After OpenSearch and OpenSearch Dashboards are running, we will setup ML Commons to connect to the LLM model +4. Run ML commons on Data node + ``` + PUT _cluster/settings + { + "persistent" : { + "plugins.ml_commons.only_run_on_ml_node":"false" + } + } + ``` +5. Add Trusted Endpoints ([reference doc](https://opensearch.org/docs/latest/ml-commons-plugin/remote-models/index/)) + ``` + PUT _cluster/settings + { + "persistent" : { + "plugins.ml_commons.trusted_connector_endpoints_regex": + [ "^https://runtime\\.sagemaker\\..*[a-z0-9-]\\.amazonaws\\.com/.*$", + "^https://api\\.openai\\.com/.*$", + "^https://api\\.cohere\\.ai/.*$", + "^https://bedrock-runtime\\.us-east-1\\.amazonaws\\.com/.*$" + ] + } + } + ``` +6. Create a connector ([reference doc](https://opensearch.org/docs/latest/ml-commons-plugin/remote-models/index/)). The below example is for connecting to the AWS Bedrock Claude model. Keep note of the connector id from the API response. (Ensure the credentials passed should have access to call the LLM model) + ``` + POST /_plugins/_ml/connectors/_create + { + "name": "BedRock test claude Connector", + "description": "The connector to BedRock service for claude model", + "version": 1, + "protocol": "aws_sigv4", + "parameters": { + "region": "us-east-1", + "service_name": "bedrock", + "anthropic_version": "bedrock-2023-05-31", + "endpoint": "bedrock.us-east-1.amazonaws.com", + "auth": "Sig_V4", + "content_type": "application/json", + "max_tokens_to_sample": 8000, + "temperature": 0.0001, + "response_filter": "$.completion" + }, + "credential": { + "access_key": "", + "secret_key": "", + "description": "test model", + "connector_id": "" + } + ``` +9. Create Embedding Model and note the model id from the get tasks API call + ``` + POST /_plugins/_ml/models/_register + { + "name": "huggingface/sentence-transformers/all-mpnet-base-v2", + "version": "1.0.1", + "model_group_id": "", + "model_format": "TORCH_SCRIPT" + } + GET /_plugins/_ml/tasks/ + ``` +10. Deploy the LLM and embedding models. Confirm the model has been deployed with the task id from the response with the get tasks API call + ``` + POST /_plugins/_ml/models//_deploy + POST /_plugins/_ml/models//_deploy + + GET /_plugins/_ml/tasks/ + ``` +11. Test connection with calling the Predict API + ``` + POST /_plugins/_ml/models//_predict + { + "parameters": { + "prompt": "\n\nHuman:hello\n\nnAssistant:" + } + } + ``` +12. Connect OS Assistant to the deployed models + ``` + POST /.chat-assistant-config/_doc/model-config + { + "model_type":"claude_bedrock", + "model_id":"", + "embeddings_model_id":"" + } + ``` +### How to create your own skill +1. To create your skill, you need to work backwards to see how that skill can be achieved by accessing different OpenSearch APIs/functions. For example, a skill to find the alerts related to a question would need to use the Alerting plugin APIs to get this info. +1. To power the skill to get alerts, we must build a tool to search alerts. +1. To create a tool, you must extend this [class](https://github.com/opensearch-project/ml-commons/blob/feature/agent_framework_dev/spi/src/main/java/org/opensearch/ml/common/spi/tools/Tool.java) and implement the specific tool. [This is an example tool](https://github.com/opensearch-project/ml-commons/pull/1629) that search alerts. diff --git a/LICENSE b/LICENSE index f49a4e16..261eeb9e 100644 --- a/LICENSE +++ b/LICENSE @@ -198,4 +198,4 @@ distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and - limitations under the License. \ No newline at end of file + limitations under the License. diff --git a/MAINTAINERS.md b/MAINTAINERS.md index 68e4e8ed..df7d91cf 100644 --- a/MAINTAINERS.md +++ b/MAINTAINERS.md @@ -4,31 +4,15 @@ This document contains a list of maintainers in this repo. See [opensearch-proje ## Current Maintainers -| Maintainer | GitHub ID | Affiliation | -| ----------------- | ------------------------------------------------- | ----------- | -| Eric Wei | [mengweieric](https://github.com/mengweieric) | Amazon | -| Joshua Li | [joshuali925](https://github.com/joshuali925) | Amazon | -| Shenoy Pratik | [ps48](https://github.com/ps48) | Amazon | -| Kavitha Mohan | [kavithacm](https://github.com/kavithacm) | Amazon | -| Rupal Mahajan | [rupal-bq](https://github.com/rupal-bq) | Amazon | -| Derek Ho | [derek-ho](https://github.com/derek-ho) | Amazon | -| Lior Perry | [YANG-DB](https://github.com/YANG-DB) | Amazon | -| Peter Fitzgibbons | [pjfitzgibbons](https://github.com/pjfitzgibbons) | Amazon | -| Simeon Widdis | [swiddis](https://github.com/swiddis) | Amazon | -| Chen Dai | [dai-chen](https://github.com/dai-chen) | Amazon | -| Vamsi Manohar | [vamsi-amazon](https://github.com/vamsi-amazon) | Amazon | -| Peng Huo | [penghuo](https://github.com/penghuo) | Amazon | -| Sean Kao | [seankao-az](https://github.com/seankao-az) | Amazon | -| Anirudha Jadhav | [anirudha](https://github.com/anirudha) | Amazon | - - -## Emeritus Maintainers - -| Maintainer | GitHub ID | Affiliation | -| ----------------- | ------------------------------------------------------- | ----------- | -| Charlotte Henkle | [CEHENKLE](https://github.com/CEHENKLE) | Amazon | -| Nick Knize | [nknize](https://github.com/nknize) | Amazon | -| David Cui | [davidcui1225](https://github.com/davidcui1225) | Amazon | -| Eugene Lee | [eugenesk24](https://github.com/eugenesk24) | Amazon | -| Zhongnan Su | [zhongnansu](https://github.com/zhongnansu) | Amazon | -| Sean Li | [sejli](https://github.com/sejli) | Amazon | \ No newline at end of file +| Maintainer | GitHub ID | Affiliation | +|-----------------|-----------------------------------------------| ----------- | +| Ashish Agrawal | [lezzago](https://github.com/lezzago) | Amazon | +| Joshua Li | [joshuali925](https://github.com/joshuali925) | Amazon | +| Anirudha Jadhav | [anirudha](https://github.com/anirudha) | Amazon | +| Yulong Ruan | [ruanyl](https://github.com/ruanyl) | Amazon | +| Su Zhou | [SuZhou-Joe](https://github.com/SuZhou-Joe) | Amazon | +| Xuesong Luo | [xluo-aws](https://github.com/xluo-aws) | Amazon | +| Binlong Gao | [gaobinlong](https://github.com/gaobinlong) | Amazon | +| Hailong Cui | [Hailong-am](https://github.com/Hailong-am) | Amazon | +| Lin Wang | [wanglam](https://github.com/wanglam) | Amazon | +| Tianyu Gao | [raintygao](https://github.com/raintygao) | Amazon | diff --git a/NOTICE b/NOTICE index 46ffe3ba..731cb600 100644 --- a/NOTICE +++ b/NOTICE @@ -1,2 +1,2 @@ OpenSearch (https://opensearch.org/) -Copyright OpenSearch Contributors \ No newline at end of file +Copyright OpenSearch Contributors diff --git a/README.md b/README.md index 900cc74d..7117031b 100644 --- a/README.md +++ b/README.md @@ -1,31 +1,49 @@ - +[![Forum](https://img.shields.io/badge/chat-on%20forums-blue)](https://forum.opensearch.org/t/feedback-opensearch-assistant/16741) +![PRs welcome!](https://img.shields.io/badge/PRs-welcome!-success) -- [Assistant](#assistant) + + +- [OpenSearch Assistant Dashboards](#opensearch-assistant-dashboards) +- [Highlights](#highlights) - [Documentation](#documentation) - [Contributing](#contributing) -- [Getting Help](#getting-help) - [Code of Conduct](#code-of-conduct) - [Security](#security) - [License](#license) - [Copyright](#copyright) -# Assistant +# OpenSearch Assistant Dashboards -Assistant. +The OpenSearch Assistant Dashboards plugin lets you have an opensearch assistant to help dashboards users use OpenSearch. -## Documentation +## How to get started -Please see our technical [documentation](https://opensearch.org/docs/latest/assistant/index/) to learn more about its features. +See [getting started guide](GETTING_STARTED_GUIDE.md) to how to deploy this Assistant on your machine with docker images. +**Note** that the `feature/langchain` is the branch used in the getting started guide. -## Contributing +### Supported tools +This requires setting `assistant.chat.enabled` to `true` in `opensearch_dashboards.yml` to use these skills: +1. Query OpenSearch: Use to generate and run a PPL Query to get results for a generic user question related to data stored in their OpenSearch cluster. +2. Get log info: Use to get information of logs if the question contains an OpenSearch log index. +3. Get log error info: Use to get information of logs with errors if the question contains an OpenSearch log index. +4. Search Alerting Monitors By Index: Use this tool to search alerting monitors by index name in the OpenSearch cluster. +5. Get All Alerts: Use this tool to search all alerts triggered in the OpenSearch cluster. +6. Get ticket information: Use this tool to find tickets in the system with incidents that are relevant to a question about error causes. +7. Get generic information: Use this tool to answer a generic question not related to OpenSearch cluster. This tool takes the question as input. +8. Get OpenSearch indices: Use this tool to get high-level information (e. health, status, index, docs.count) about indices in a cluster, including backing indices for data streams in the OpenSearch cluster. +9. Check OpenSearch index existence: Use this tool to check if a data stream, index, or alias exists in the OpenSearch cluster. +10. Find Visualizations: Use this tool to find user created visualizations. +11. Get trace groups: Use this to get information about each trace group. +12. Get traces: Use this to get information about each trace. +13. Get trace services: Use this to get information about each service in trace analytics. -See [developer guide](DEVELOPER_GUIDE.md) and [how to contribute to this project](CONTRIBUTING.md). +## Documentation -## Getting Help +In the works. -If you find a bug, or have a feature request, please don't hesitate to open an issue in this repository. +## Contributing -For more information, see [project website](https://opensearch.org/) and [documentation](https://opensearch.org/docs). If you need help and are unsure where to open an issue, try the [Forum](https://forum.opensearch.org/c/plugins). +See [developer guide](DEVELOPER_GUIDE.md) and [how to contribute to this project](CONTRIBUTING.md). ## Code of Conduct diff --git a/server/utils/message_parser_runner.test.ts b/server/utils/message_parser_runner.test.ts index ca4032a6..69f7f9d8 100644 --- a/server/utils/message_parser_runner.test.ts +++ b/server/utils/message_parser_runner.test.ts @@ -30,7 +30,7 @@ describe('MessageParserRunner', () => { interaction_id: '', create_time: '', additional_info: {}, - parent_interaction_id: '' + parent_interaction_id: '', }) ).toEqual([ { @@ -104,7 +104,7 @@ describe('MessageParserRunner', () => { interaction_id: '', create_time: '', additional_info: {}, - parent_interaction_id: '' + parent_interaction_id: '', }) ).toEqual([ { @@ -158,7 +158,7 @@ describe('MessageParserRunner', () => { interaction_id: '', create_time: '', additional_info: {}, - parent_interaction_id: '' + parent_interaction_id: '', }) ).toEqual([]); });