Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: improving data streaming openapi to prevent health check errors #110

Closed
wants to merge 2 commits into from
Closed

refactor: improving data streaming openapi to prevent health check errors #110

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b694e5f
refactor: improving data streaming openapi to prevent health check er…
alisson-marcelo-azion Aug 17, 2023
a417e66
refactor: using endpoint definitions instead of string
alisson-marcelo-azion Aug 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
214 changes: 200 additions & 14 deletions data_streaming.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -418,8 +418,19 @@ components:
default: true

endpoint:
type: string
format: json
oneOf:
- $ref: '#/components/schemas/Endpoint_Standard'
- $ref: '#/components/schemas/Endpoint_Kafka'
- $ref: '#/components/schemas/Endpoint_S3'
- $ref: '#/components/schemas/Endpoint_Google_BigQuery'
- $ref: '#/components/schemas/Endpoint_Elasticsearch'
- $ref: '#/components/schemas/Endpoint_AWS_Kinesis_Firehose'
- $ref: '#/components/schemas/Endpoint_Datadog'
- $ref: '#/components/schemas/Endpoint_IBM_QRadar'
- $ref: '#/components/schemas/Endpoint_Azure_Monitor'
- $ref: '#/components/schemas/Endpoint_Azure_Blob_Storage'
- $ref: '#/components/schemas/Endpoint_Splunk'

description: >
Options' examples:
- `Standard HTTP/HTTPS POST` - { "endpoint_type": "standard", "url": "http://example.com", "log_line_separator": "\n", "payload_format": "$dataset", "max_size": 1000024 }
Expand Down Expand Up @@ -464,7 +475,14 @@ components:
Note:
* `Range` - From 0 to 100.
* `To use:` [Contact the sales team](https://www.azion.com/en/contact-sales/) to activate this feature in your account.


required:
- name
- template_id
- endpoint
- domains_ids


StandardDataStreamingPostBody:
type: object
properties:
Expand Down Expand Up @@ -503,7 +521,31 @@ components:
default: true

endpoint:
$ref: '#/components/schemas/DataStreamingEndpointTypeStandard'
oneOf:
- $ref: '#/components/schemas/Endpoint_Standard'
- $ref: '#/components/schemas/Endpoint_Kafka'
- $ref: '#/components/schemas/Endpoint_S3'
- $ref: '#/components/schemas/Endpoint_Google_BigQuery'
- $ref: '#/components/schemas/Endpoint_Elasticsearch'
- $ref: '#/components/schemas/Endpoint_AWS_Kinesis_Firehose'
- $ref: '#/components/schemas/Endpoint_Datadog'
- $ref: '#/components/schemas/Endpoint_IBM_QRadar'
- $ref: '#/components/schemas/Endpoint_Azure_Monitor'
- $ref: '#/components/schemas/Endpoint_Azure_Blob_Storage'
- $ref: '#/components/schemas/Endpoint_Splunk'
description: >
Options' examples:
- `Standard HTTP/HTTPS POST` - { "endpoint_type": "standard", "url": "http://example.com", "log_line_separator": "\n", "payload_format": "$dataset", "max_size": 1000024 }
- `Apache Kafka` - { "endpoint_type": "kafka", "kafka_topic": "example_topic", "bootstrap_servers": "kafka-server.com:9092,kafka-server-2.com:9092", "use_tls":true }
- `Simple Storage Service (S3)` - { "endpoint_type": "s3", "access_key": "MYACCESSKEY", "region": "us-east-1", "object_key_prefix": "my_prefix_", "bucket_name": "bucket_example", "content_type": "plain/text", "host_url": "http://aws-host.com", "secret_key": "MYSECRETKEY" }
- `Google BigQuery` - { "endpoint_type": "big_query", "dataset_id": "my_dataset", "project_id": "my_project", "table_id": "my_table", "service_account_key": "{ "service_account_key": "key_content" }" }
- `Elasticsearch` - { “endpoint_type”: "elasticsearch", “url”: “http://elasticsearch.com”, “api_key”: “XYZ_API_KEY” }
- `AWS Kinesis Data Firehose` - { "endpoint_type": "aws_kinesis_firehose", "access_key": "MYACCESSKEY", "stream_name": "my_stream_name", "region": "us-east-1", "secret_key": "MYSECRETKEY" }
- `Datadog` - { "endpoint_type": "datadog", "url": "https://http-intake.logs.datadoghq.com/v1/input", "api_key": "MYAPIKEY" }
- `IBM QRadar` - { "endpoint_type": "qradar", "url": "http://137.15.824.10:14440” }
- `Azure Monitor` - { "endpoint_type": "azure_monitor", "log_type": "myLogType", "shared_key": "mysharedkey", "time_generated_field": "timeGeneratedField", "workspace_id": "anfhw-123sd-466gcs"}
- `Azure Blob Storage` - { "endpoint_type": "azure_blob_storage", "storage_account": "mystorageaccount", "container_name": "log_container", "blob_sas_token": "fd56e23e1f12efe" }
- `Splunk` - { "endpoint_type": "splunk", "url": "https://inputs.splunk-client.splunkcloud.com:1337/services/collector", "api_key": "MYAPIKEY" }

domains_ids:
type: array
Expand All @@ -513,6 +555,8 @@ components:
description: >
Note:
* Field not used with the rtm_activity data source.
# OpenAPI 3.0 does not support parameter dependencies and mutually exclusive parameters.
# Link: https://stackoverflow.com/questions/63209596/how-to-indicate-that-a-parameter-is-conditionally-required-when-another-paramete/63213464#63213464

all_domains:
type: boolean
Expand All @@ -521,6 +565,24 @@ components:
description: >
Note:
* Field not used with the rtm_activity data source.
# OpenAPI 3.0 does not support parameter dependencies and mutually exclusive parameters.
# Link: https://stackoverflow.com/questions/63209596/how-to-indicate-that-a-parameter-is-conditionally-required-when-another-paramete/63213464#63213464

sampling_percentage:
type: integer
nullable: true
minimum: 0
maximum: 100
description: >
Note:
* `Range` - From 0 to 100.
* `To use:` [Contact the sales team](https://www.azion.com/en/contact-sales/) to activate this feature in your account.

required:
- name
- template_id
- endpoint
- domains_ids

CustomDataStreamingPostBody:
type: object
Expand Down Expand Up @@ -551,6 +613,69 @@ components:
nullable: true
default: true

endpoint:
oneOf:
- $ref: '#/components/schemas/Endpoint_Standard'
- $ref: '#/components/schemas/Endpoint_Kafka'
- $ref: '#/components/schemas/Endpoint_S3'
- $ref: '#/components/schemas/Endpoint_Google_BigQuery'
- $ref: '#/components/schemas/Endpoint_Elasticsearch'
- $ref: '#/components/schemas/Endpoint_AWS_Kinesis_Firehose'
- $ref: '#/components/schemas/Endpoint_Datadog'
- $ref: '#/components/schemas/Endpoint_IBM_QRadar'
- $ref: '#/components/schemas/Endpoint_Azure_Monitor'
- $ref: '#/components/schemas/Endpoint_Azure_Blob_Storage'
- $ref: '#/components/schemas/Endpoint_Splunk'
description: >
Options' examples:
- `Standard HTTP/HTTPS POST` - { "endpoint_type": "standard", "url": "http://example.com", "log_line_separator": "\n", "payload_format": "$dataset", "max_size": 1000024 }
- `Apache Kafka` - { "endpoint_type": "kafka", "kafka_topic": "example_topic", "bootstrap_servers": "kafka-server.com:9092,kafka-server-2.com:9092", "use_tls":true }
- `Simple Storage Service (S3)` - { "endpoint_type": "s3", "access_key": "MYACCESSKEY", "region": "us-east-1", "object_key_prefix": "my_prefix_", "bucket_name": "bucket_example", "content_type": "plain/text", "host_url": "http://aws-host.com", "secret_key": "MYSECRETKEY" }
- `Google BigQuery` - { "endpoint_type": "big_query", "dataset_id": "my_dataset", "project_id": "my_project", "table_id": "my_table", "service_account_key": "{ "service_account_key": "key_content" }" }
- `Elasticsearch` - { “endpoint_type”: "elasticsearch", “url”: “http://elasticsearch.com”, “api_key”: “XYZ_API_KEY” }
- `AWS Kinesis Data Firehose` - { "endpoint_type": "aws_kinesis_firehose", "access_key": "MYACCESSKEY", "stream_name": "my_stream_name", "region": "us-east-1", "secret_key": "MYSECRETKEY" }
- `Datadog` - { "endpoint_type": "datadog", "url": "https://http-intake.logs.datadoghq.com/v1/input", "api_key": "MYAPIKEY" }
- `IBM QRadar` - { "endpoint_type": "qradar", "url": "http://137.15.824.10:14440” }
- `Azure Monitor` - { "endpoint_type": "azure_monitor", "log_type": "myLogType", "shared_key": "mysharedkey", "time_generated_field": "timeGeneratedField", "workspace_id": "anfhw-123sd-466gcs"}
- `Azure Blob Storage` - { "endpoint_type": "azure_blob_storage", "storage_account": "mystorageaccount", "container_name": "log_container", "blob_sas_token": "fd56e23e1f12efe" }
- `Splunk` - { "endpoint_type": "splunk", "url": "https://inputs.splunk-client.splunkcloud.com:1337/services/collector", "api_key": "MYAPIKEY" }

domains_ids:
type: array
items:
type: integer
minItems: 1
description: >
Note:
* Field not used with the rtm_activity data source.
# OpenAPI 3.0 does not support parameter dependencies and mutually exclusive parameters.
# Link: https://stackoverflow.com/questions/63209596/how-to-indicate-that-a-parameter-is-conditionally-required-when-another-paramete/63213464#63213464

all_domains:
type: boolean
nullable: true
default: false
description: >
Note:
* Field not used with the rtm_activity data source.
# OpenAPI 3.0 does not support parameter dependencies and mutually exclusive parameters.
# Link: https://stackoverflow.com/questions/63209596/how-to-indicate-that-a-parameter-is-conditionally-required-when-another-paramete/63213464#63213464

sampling_percentage:
type: integer
nullable: true
minimum: 0
maximum: 100
description: >
Note:
* `Range` - From 0 to 100.
* `To use:` [Contact the sales team](https://www.azion.com/en/contact-sales/) to activate this feature in your account.

required:
- name
- template_id
- endpoint
- domains_ids


CreateDataStreamingResponse:
Expand Down Expand Up @@ -616,9 +741,9 @@ components:
type: array
items:
oneOf:
- $ref: '#/components/schemas/Endpoint_Default'
- $ref: '#/components/schemas/Endpoint_Standard'
- $ref: '#/components/schemas/Endpoint_Kafka'
- $ref: '#/components/schemas/Endpoinrt_S3'
- $ref: '#/components/schemas/Endpoint_S3'
- $ref: '#/components/schemas/Endpoint_Google_BigQuery'
- $ref: '#/components/schemas/Endpoint_Elasticsearch'
- $ref: '#/components/schemas/Endpoint_AWS_Kinesis_Firehose'
Expand Down Expand Up @@ -679,7 +804,7 @@ components:
Note:
* Field not used with the rtm_activity data source.

Endpoint_Default:
Endpoint_Standard:
type: object
properties:
endpoint_type:
Expand All @@ -703,7 +828,11 @@ components:
example:
additional_header: additional_header_value
maximum: 5

required:
- endpoint_type
- url
- payload_format
- log_line_separator

Endpoint_Kafka:
type: object
Expand All @@ -717,8 +846,12 @@ components:
type: string
use_tls:
type: boolean

Endpoinrt_S3:
required:
- endpoint_type
- kafka_topic
- bootstrap_servers

Endpoint_S3:
type: object
properties:
endpoint_type:
Expand All @@ -739,6 +872,14 @@ components:
format: uri
secret_key:
type: string
required:
- endpoint_type
- access_key
- secret_key
- content_type
- bucket_name
- host_url
- region

Endpoint_Google_BigQuery:
type: object
Expand All @@ -757,6 +898,12 @@ components:
properties:
service_account_key:
type: string
required:
- endpoint_type
- project_id
- table_id
- service_account_key
- dataset_id

Endpoint_Elasticsearch:
type: object
Expand All @@ -769,6 +916,10 @@ components:
format: uri
api_key:
format: string
required:
- endpoint_type
- url
- api_key

Endpoint_AWS_Kinesis_Firehose:
type: object
Expand All @@ -784,18 +935,28 @@ components:
type: string
secret_key:
type: string

required:
- endpoint_type
- access_key
- secret_key
- stream_name
- region

Endpoint_Datadog:
type: object
properties:
endpoint_type:
type: string
example: "datadog"
url:
datadog_url:
type: string
format: uri
api_key:
format: string
required:
- endpoint_type
- api_key
- datadog_url

Endpoint_IBM_QRadar:
type: object
Expand All @@ -806,6 +967,9 @@ components:
url:
type: string
format: uri
required:
- endpoint_type
- url

Endpoint_Azure_Monitor:
type: object
Expand All @@ -822,6 +986,11 @@ components:
workspace_id:
type: string
pattern: '[a-zA-Z0-9]{5}-[a-zA-Z0-9]{5}-[a-zA-Z0-9]{6}'
required:
- endpoint_type
- log_type
- shared_key
- workspace_id

Endpoint_Azure_Blob_Storage:
type: object
Expand All @@ -836,6 +1005,11 @@ components:
blob_sas_token:
type: string
pattern: '[a-zA-Z0-9]{15}'
required:
- endpoint_type
- storage_account
- container_name
- blob_sas_token

Endpoint_Splunk:
type: object
Expand All @@ -848,6 +1022,10 @@ components:
format: uri
api_key:
type: string
required:
- endpoint_type
- api_key
- url


DataStreamingResponseWithResults:
Expand Down Expand Up @@ -910,6 +1088,7 @@ components:
$ref: '#/components/schemas/DataStreamingEndpointTypeKafka'
all_domains:
type: boolean


DataStreamingEndpointTypeKafka:
type: object
Expand All @@ -922,7 +1101,10 @@ components:
type: string
bootstrap_servers:
type: string

required:
- kafka_topic
- bootstrap_servers

DataStreamingResponseGetResultTypeStandard:
type: object
properties:
Expand Down Expand Up @@ -957,6 +1139,10 @@ components:
type: integer
headers:
$ref: '#/components/schemas/DataStreamingEndpointTypeStandardHeadersExample'
required:
- url
- payload_format
- log_line_separator

DataStreamingEndpointTypeStandardHeadersExample:
type: object
Expand Down Expand Up @@ -1072,4 +1258,4 @@ components:
description: |
You must inform a token to auth.
Usage format: `Token <TOKEN>`


Loading