Merge branch 'main' into feature-plugin-remove-list

Signed-off-by: Yulong Ruan <[email protected]>

.github/CODEOWNERS

@@ -1,2 +1,2 @@
 # This should match the owning team set up in https://github.com/orgs/opensearch-project/teams
-*   @bbarani @TheAlgo @DandyDeveloper @peterzhuamazon @prudhvigodithi @gaiksaya
+*  @TheAlgo @DandyDeveloper @peterzhuamazon @prudhvigodithi @gaiksaya

.github/workflows/release.yaml

@@ -28,6 +28,6 @@ jobs:
       # See https://github.com/helm/chart-releaser
       # See https://github.com/helm/chart-releaser-action
       - name: Run chart-releaser
-        uses: helm/chart-releaser-action@v1.2.1
+        uses: helm/chart-releaser-action@v1.6.0
         env:
           CR_TOKEN: "${{ secrets.GITHUB_TOKEN }}"

MAINTAINERS.md

@@ -6,9 +6,14 @@ This document contains a list of maintainers in this repo. See [opensearch-proje
 
 | Maintainer        | GitHub ID                                           | Affiliation |
 | ----------------- | --------------------------------------------------- | ----------- |
-| Barani Bikshandi  | [bbarani](https://github.com/bbarani)               | Amazon      |
 | Peter Zhu         | [peterzhuamazon](https://github.com/peterzhuamazon) | Amazon      |
 | Sayali Gaikawad   | [gaiksaya](https://github.com/gaiksaya)             | Amazon      |
 | Prudhvi Godithi   | [prudhvigodithi](https://github.com/prudhvigodithi) | Amazon      |
 | Dhiraj Kumar Jain | [TheAlgo](https://github.com/TheAlgo)               | Amazon      |
 | Aaron Layfield    | [DandyDeveloper](https://github.com/DandyDeveloper) | Community   |
+
+## Emeritus
+
+| Maintainer            | GitHub ID                                 | Affiliation |
+| --------------------- | ----------------------------------------- | ----------- |
+| Barani Bikshandi  | [bbarani](https://github.com/bbarani)               | Amazon      |

README.md

@@ -62,7 +62,7 @@ Please see the `README.md` in the [OpenSearch](charts/opensearch) and [OpenSearc
 ### Breaking Change 
 For OpenSearch Chart version 2.18.0 and App Version OpenSearch version 2.12.0 and above require a custom strong password to be provided in order to setup demo admin user during the first time cluster installation, without this password the cluster would not spin up, unless demo config install is disabled or the cluster security settings are handled by the user.
 
-Note: This wont impact users who already installed the cluster and where the security index created, only impacts users starting the brand new cluster with OpenSearch version `2.12.0` and above.
+Note: This won’t impact users who have already installed the cluster and created the security index. It only affects users starting a brand new cluster with OpenSearch version 2.12.0 and above.
 
 The custom admin password can be supplied by adding the environment variable `OPENSEARCH_INITIAL_ADMIN_PASSWORD` in the `value.yml` as:
 ```
@@ -75,9 +75,9 @@ or during the installation example as `helm install opensearch opensearch/opense
 
 ### Notes About Default Installation
 
-By default, on startup, the `install_demo_configuration.sh` is runned via the `opensearch-docker-entrypoint.sh` script if `DISABLE_INSTALL_DEMO_CONFIG` is not `true`.
+By default, on startup, the `install_demo_configuration.sh` is run via the `opensearch-docker-entrypoint.sh` script if `DISABLE_INSTALL_DEMO_CONFIG` is not `true`.
 
-In case custom certificates are used and `allow_unsafe_democertificates` is set to `false` in the configuration, this can prevent pods to start with the following error: `Demo certificates found but plugins.security.allow_unsafe_democertificates is set to false.`
+If custom certificates are used and `allow_unsafe_democertificates` is set to `false` in the configuration, this can prevent pods from starting with the following error: `Demo certificates found but plugins.security.allow_unsafe_democertificates is set to false.`
 
 This can be solved by adding an environment variable in the `value.yml`:
 ```

charts/data-prepper/.helmignore

@@ -0,0 +1,23 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*.orig
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+.vscode/

charts/data-prepper/CHANGELOG.md

@@ -0,0 +1,10 @@
+# Changelog
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+### Added
+- Create initial version of data-prepper helm chart
+

charts/data-prepper/Chart.yaml

@@ -0,0 +1,50 @@
+apiVersion: v2
+name: data-prepper
+description: A Helm chart for Data Prepper
+
+# A chart can be either an 'application' or a 'library' chart.
+#
+# Application charts are a collection of templates that can be packaged into versioned archives
+# to be deployed.
+#
+# Library charts provide useful utilities or functions for the chart developer. They're included as
+# a dependency of application charts to inject those utilities and functions into the rendering
+# pipeline. Library charts do not define any templates and therefore cannot be deployed.
+type: application
+
+# This is the chart version. This version number should be incremented each time you make changes
+# to the chart and its templates, including the app version.
+# Versions are expected to follow Semantic Versioning (https://semver.org/)
+version: 0.1.0
+
+# This is the version number of the application being deployed. This version number should be
+# incremented each time you make changes to the application. Versions are not expected to
+# follow Semantic Versioning. They should reflect the version the application is using.
+# It is recommended to use it with quotes.
+appVersion: "2.8.0"
+
+maintainers:
+  - name: gaiksaya
+    url: https://github.com/gaiksaya
+  - name: peterzhuamazon
+    url: https://github.com/peterzhuamazon
+  - name: prudhvigodithi
+    url: https://github.com/prudhvigodithi
+  - name: sergk
+    url: https://github.com/sergk
+  - name: TheAlgo
+    url: https://github.com/TheAlgo
+
+home: https://opensearch.org/docs/latest/data-prepper/
+sources:
+  - https://github.com/opensearch-project/data-prepper
+  - https://github.com/opensearch-project/helm-charts
+
+annotations:
+  artifacthub.io/category: monitoring-logging
+  artifacthub.io/license: Apache-2.0
+  artifacthub.io/links: |
+    - name: Data Prepper Documentation
+      url: https://opensearch.org/docs/latest/data-prepper/
+    - name: OpenSearch Project
+      url: https://opensearch.org

charts/data-prepper/README.md

@@ -0,0 +1,125 @@
+# Data Prepper Helm Chart
+
+![Version: 0.1.0](https://img.shields.io/badge/Version-0.1.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 2.8.0](https://img.shields.io/badge/AppVersion-2.8.0-informational?style=flat-square)
+
+A Helm chart for Data Prepper
+
+**Homepage:** <https://opensearch.org/docs/latest/data-prepper/>
+
+Data Prepper is an essential component of the OpenSearch project, designed for high-volume data transformation and ingestion into OpenSearch. This Helm chart simplifies deploying Data Prepper on Kubernetes environments, ensuring you can easily set up your data processing pipelines.
+
+## Requirements
+
+Before installing the Data Prepper Helm chart, ensure your environment meets the following requirements:
+
+* Kubernetes >= 1.14
+* Helm >= 2.17.0
+* We recommend having at least 4 GiB of memory available for this deployment. A minimum of 2 GiB may suffice, but less than that could lead to deployment failures.
+
+## Installation
+
+To install the Data Prepper Helm chart, follow these steps:
+
+* Add the OpenSearch Helm repository if you haven't already:
+
+```bash
+helm repo add opensearch https://opensearch-project.github.io/helm-charts/
+helm repo update
+```
+
+* Install the Data Prepper chart with:
+
+```bash
+helm install my-data-prepper-release opensearch/data-prepper
+```
+
+Replace my-data-prepper-release with your desired release name.
+
+## Configuration
+
+The Data Prepper Helm chart comes with a variety of configuration options to tailor the deployment to your needs.
+The default values are specified in the [values.yaml](values.yaml) file. You can override these values by providing your own values.yaml file during installation or by specifying configuration options with --set flags.
+
+For a detailed list of configuration options, refer to the values.yaml file or the [Data Prepper documentation](https://opensearch.org/docs/latest/data-prepper/managing-data-prepper/configuring-data-prepper/).
+
+## Uninstalling the Chart
+
+To uninstall/delete the my-data-prepper deployment:
+
+```bash
+helm delete my-data-prepper
+```
+
+This command removes all the Kubernetes components associated with the chart and deletes the release.
+
+## Contributing
+
+We welcome contributions! Please read our [CONTRIBUTING.md](../../CONTRIBUTING.md) for details on how to submit contributions to this project.
+
+## Maintainers
+
+| Name | Email | Url |
+| ---- | ------ | --- |
+| gaiksaya |  | <https://github.com/gaiksaya> |
+| peterzhuamazon |  | <https://github.com/peterzhuamazon> |
+| prudhvigodithi |  | <https://github.com/prudhvigodithi> |
+| sergk |  | <https://github.com/sergk> |
+| TheAlgo |  | <https://github.com/TheAlgo> |
+
+## Source Code
+
+* <https://github.com/opensearch-project/data-prepper>
+* <https://github.com/opensearch-project/helm-charts>
+
+## Values
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| affinity | object | `{}` |  |
+| autoscaling.enabled | bool | `false` |  |
+| autoscaling.maxReplicas | int | `100` |  |
+| autoscaling.minReplicas | int | `1` |  |
+| autoscaling.targetCPUUtilizationPercentage | int | `80` |  |
+| config | object | `{"data-prepper-config.yaml":"ssl: false\n# circuit_breakers:\n#   heap:\n#     usage: 2gb\n#     reset: 30s\n#     check_interval: 5s\n","log4j2-rolling.properties":"#\n# Copyright OpenSearch Contributors\n# SPDX-License-Identifier: Apache-2.0\n#\n\nstatus = error\ndest = err\nname = PropertiesConfig\n\nproperty.filename = log/data-prepper/data-prepper.log\n\nappender.console.type = Console\nappender.console.name = STDOUT\nappender.console.layout.type = PatternLayout\nappender.console.layout.pattern = %d{ISO8601} [%t] %-5p %40C - %m%n\n\nappender.rolling.type = RollingFile\nappender.rolling.name = RollingFile\nappender.rolling.fileName = ${filename}\nappender.rolling.filePattern = logs/data-prepper.log.%d{MM-dd-yy-HH}-%i.gz\nappender.rolling.layout.type = PatternLayout\nappender.rolling.layout.pattern = %d{ISO8601} [%t] %-5p %40C - %m%n\nappender.rolling.policies.type = Policies\nappender.rolling.policies.time.type = TimeBasedTriggeringPolicy\nappender.rolling.policies.time.interval = 1\nappender.rolling.policies.time.modulate = true\nappender.rolling.policies.size.type = SizeBasedTriggeringPolicy\nappender.rolling.policies.size.size=100MB\nappender.rolling.strategy.type = DefaultRolloverStrategy\nappender.rolling.strategy.max = 168\n\nrootLogger.level = warn\nrootLogger.appenderRef.stdout.ref = STDOUT\nrootLogger.appenderRef.file.ref = RollingFile\n\nlogger.pipeline.name = org.opensearch.dataprepper.pipeline\nlogger.pipeline.level = info\n\nlogger.parser.name = org.opensearch.dataprepper.parser\nlogger.parser.level = info\n\nlogger.plugins.name = org.opensearch.dataprepper.plugins\nlogger.plugins.level = info\n"}` | Data Prepper configuration |
+| config."data-prepper-config.yaml" | string | `"ssl: false\n# circuit_breakers:\n#   heap:\n#     usage: 2gb\n#     reset: 30s\n#     check_interval: 5s\n"` | Main Data Prepper configuration file content |
+| config."log4j2-rolling.properties" | string | `"#\n# Copyright OpenSearch Contributors\n# SPDX-License-Identifier: Apache-2.0\n#\n\nstatus = error\ndest = err\nname = PropertiesConfig\n\nproperty.filename = log/data-prepper/data-prepper.log\n\nappender.console.type = Console\nappender.console.name = STDOUT\nappender.console.layout.type = PatternLayout\nappender.console.layout.pattern = %d{ISO8601} [%t] %-5p %40C - %m%n\n\nappender.rolling.type = RollingFile\nappender.rolling.name = RollingFile\nappender.rolling.fileName = ${filename}\nappender.rolling.filePattern = logs/data-prepper.log.%d{MM-dd-yy-HH}-%i.gz\nappender.rolling.layout.type = PatternLayout\nappender.rolling.layout.pattern = %d{ISO8601} [%t] %-5p %40C - %m%n\nappender.rolling.policies.type = Policies\nappender.rolling.policies.time.type = TimeBasedTriggeringPolicy\nappender.rolling.policies.time.interval = 1\nappender.rolling.policies.time.modulate = true\nappender.rolling.policies.size.type = SizeBasedTriggeringPolicy\nappender.rolling.policies.size.size=100MB\nappender.rolling.strategy.type = DefaultRolloverStrategy\nappender.rolling.strategy.max = 168\n\nrootLogger.level = warn\nrootLogger.appenderRef.stdout.ref = STDOUT\nrootLogger.appenderRef.file.ref = RollingFile\n\nlogger.pipeline.name = org.opensearch.dataprepper.pipeline\nlogger.pipeline.level = info\n\nlogger.parser.name = org.opensearch.dataprepper.parser\nlogger.parser.level = info\n\nlogger.plugins.name = org.opensearch.dataprepper.plugins\nlogger.plugins.level = info\n"` | Log4j2 configuration for Data Prepper logging |
+| extraEnvs | list | `[]` | Extra environment variables to pass to the Data Prepper container |
+| fullnameOverride | string | `""` | Override the default fullname for the deployment |
+| image.pullPolicy | string | `"IfNotPresent"` | The image tag to pull. Default: IfNotPresent |
+| image.repository | string | `"opensearchproject/data-prepper"` | The image repository from which to pull the Data Prepper image |
+| image.tag | string | `""` | Overrides the image tag whose default is the chart appVersion. |
+| imagePullSecrets | list | `[]` | List of imagePullSecrets to use if the Docker image is stored in a private registry |
+| ingress.annotations | object | `{}` |  |
+| ingress.className | string | `""` |  |
+| ingress.enabled | bool | `false` |  |
+| ingress.hosts[0].host | string | `"chart-example.local"` |  |
+| ingress.hosts[0].paths[0].path | string | `"/"` |  |
+| ingress.hosts[0].paths[0].pathType | string | `"ImplementationSpecific"` |  |
+| ingress.tls | list | `[]` |  |
+| nameOverride | string | `""` | Override the default name for the deployment |
+| nodeSelector | object | `{}` |  |
+| pipelineConfig | object | `{"config":{"simple-sample-pipeline":{"buffer":{"bounded_blocking":{"batch_size":256,"buffer_size":1024}},"delay":5000,"processor":[{"string_converter":{"upper_case":true}}],"sink":[{"stdout":null}],"source":{"random":null},"workers":2}},"enabled":true,"existingSecret":""}` | Pipeline configuration |
+| pipelineConfig.existingSecret | string | `""` | The name of the existing secret containing the pipeline configuration. If enabled is false existingSecret is used. The existingSecret must have a key named `pipelines.yaml`. |
+| podAnnotations | object | `{}` |  |
+| podLabels | object | `{}` |  |
+| podSecurityContext | object | `{}` |  |
+| ports | list | `[{"name":"http-source","port":2021},{"name":"otel-traces","port":21890},{"name":"otel-metrics","port":21891},{"name":"otel-logs","port":21892}]` | Data Prepper ports |
+| ports[0] | object | `{"name":"http-source","port":2021}` | The port that the source is running on. Default value is 2021. Valid options are between 0 and 65535. https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/http-source/ |
+| ports[1] | object | `{"name":"otel-traces","port":21890}` | The port that the otel_trace_source source runs on. Default value is 21890. https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/otel-trace-source/ |
+| ports[2] | object | `{"name":"otel-metrics","port":21891}` | The port that the OpenTelemtry metrics source runs on. Default value is 21891. https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/otel-metrics-source/ |
+| ports[3] | object | `{"name":"otel-logs","port":21892}` | Represents the port that the otel_logs_source source is running on. Default value is 21892. https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/otel-logs-source/ |
+| replicaCount | int | `1` |  |
+| resources | object | `{}` |  |
+| securityContext | object | `{}` |  |
+| service.type | string | `"ClusterIP"` |  |
+| serviceAccount.annotations | object | `{}` | Annotations to add to the service account |
+| serviceAccount.automount | bool | `true` | Automatically mount a ServiceAccount's API credentials? |
+| serviceAccount.create | bool | `true` | Specifies whether a service account should be created |
+| serviceAccount.name | string | `""` | The name of the service account to use. If not set and create is true, a name is generated using the fullname template |
+| tolerations | list | `[]` |  |
+| volumeMounts | list | `[]` |  |
+| volumes | list | `[]` |  |
+
+## License
+
+This project is licensed under the Apache License 2.0 - see the [LICENSE.txt](../../LICENSE.txt) file for details.

charts/data-prepper/README.md.gotmpl

@@ -0,0 +1,71 @@
+# Data Prepper Helm Chart
+
+{{ template "chart.deprecationWarning" . }}
+
+{{ template "chart.badgesSection" . }}
+
+{{ template "chart.description" . }}
+
+{{ template "chart.homepageLine" . }}
+
+Data Prepper is an essential component of the OpenSearch project, designed for high-volume data transformation and ingestion into OpenSearch. This Helm chart simplifies deploying Data Prepper on Kubernetes environments, ensuring you can easily set up your data processing pipelines.
+
+## Requirements
+
+Before installing the Data Prepper Helm chart, ensure your environment meets the following requirements:
+
+* Kubernetes >= 1.14
+* Helm >= 2.17.0
+* We recommend having at least 4 GiB of memory available for this deployment. A minimum of 2 GiB may suffice, but less than that could lead to deployment failures.
+
+## Installation
+
+To install the Data Prepper Helm chart, follow these steps:
+
+* Add the OpenSearch Helm repository if you haven't already:
+
+```bash
+helm repo add opensearch https://opensearch-project.github.io/helm-charts/
+helm repo update
+```
+
+* Install the Data Prepper chart with:
+
+```bash
+helm install my-data-prepper-release opensearch/data-prepper
+```
+
+Replace my-data-prepper-release with your desired release name.
+
+## Configuration
+
+The Data Prepper Helm chart comes with a variety of configuration options to tailor the deployment to your needs.
+The default values are specified in the [values.yaml](values.yaml) file. You can override these values by providing your own values.yaml file during installation or by specifying configuration options with --set flags.
+
+For a detailed list of configuration options, refer to the values.yaml file or the [Data Prepper documentation](https://opensearch.org/docs/latest/data-prepper/managing-data-prepper/configuring-data-prepper/).
+
+## Uninstalling the Chart
+
+To uninstall/delete the my-data-prepper deployment:
+
+```bash
+helm delete my-data-prepper
+```
+
+This command removes all the Kubernetes components associated with the chart and deletes the release.
+
+## Contributing
+
+We welcome contributions! Please read our [CONTRIBUTING.md](../../CONTRIBUTING.md) for details on how to submit contributions to this project.
+
+{{ template "chart.maintainersSection" . }}
+
+{{ template "chart.sourcesSection" . }}
+
+{{ template "chart.requirementsSection" . }}
+
+{{ template "chart.valuesSection" . }}
+
+## License
+
+This project is licensed under the Apache License 2.0 - see the [LICENSE.txt](../../LICENSE.txt) file for details.