From d4c9b8831d04a31f56648338c8ff1b92af6f810c Mon Sep 17 00:00:00 2001 From: Sanju Date: Fri, 22 Nov 2024 13:55:06 +0100 Subject: [PATCH 01/17] notion: update doc --- docs/ingestion/notion.md | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/docs/ingestion/notion.md b/docs/ingestion/notion.md index 79abd9f7..e2d961bb 100644 --- a/docs/ingestion/notion.md +++ b/docs/ingestion/notion.md @@ -1,8 +1,15 @@ # Notion +[Notion ](https://www.notion.so/) is an all-in-one workspace for note-taking, project management, and database management. -In order to have set up a Notion connection, you need to add a configuration item to `connections` in the `.bruin.yml` file complying with the following schema. -For more information on how to get these credentials check the Notion Section in [Ingestr documentation](https://bruin-data.github.io/ingestr/supported-sources/notion.html). +Bruin supports Notion as a source for [Ingestr assets](/assets/ingestr), and you can use it to ingest data from Notion into your data warehouse. +In order to set up Notion connection, you need to add a configuration item in the `.bruin.yml` file and in `asset` file. You need `api_key`. For details on how to obtain these credentials, please refer [here](https://dlthub.com/docs/dlt-ecosystem/verified-sources/notion#setup-guide). + +Follow the steps below to correctly set up Notion as a data source and run ingestion. + +### Step 1: Add a connection to .bruin.yml file + +To connect to Notion, you need to add a configuration item to the connections section of the `.bruin.yml` file. This configuration must comply with the following schema: ```yaml connections: @@ -10,3 +17,20 @@ For more information on how to get these credentials check the Notion Section in - name: "connection_name" api_key: "XXXXXXXX" ``` +### Step 2: Create an asset file for data ingestion + +To ingest data from Notion, you need to create an [asset configuration](/assets/ingestr#asset-structure) file. This file defines the data flow from the source to the destination. Create a YAML file (e.g., notion_ingestion.yml) inside the assets folder and add the following content: + + +- `name`: The name of the asset. +- `type`: Specifies the type of the asset. Set this to ingestr to use the ingestr data pipeline. +- `connection`: This is the destination connection, which defines where the data should be stored. For example: `postgres` indicates that the ingested data will be stored in a Postgres database. +- `source_connection`: The name of the Notion connection defined in .bruin.yml. +- `source_table`: The name of the data table in Notion that you want to ingest. For example, `details` is the table of notion that you want to ingest. + +### Step 3: [Run](/commands/run) asset to ingest data +``` +bruin run assets/notion_ingestion.yml +``` + +As a result of this command, Bruin will ingest data from the given Notion table into your Postgres database. \ No newline at end of file From 74eacf134ca54d0451564335e115b6820f8d74ae Mon Sep 17 00:00:00 2001 From: Sanju Date: Fri, 22 Nov 2024 14:17:13 +0100 Subject: [PATCH 02/17] small changes --- docs/ingestion/notion.md | 17 ++++++++++++++--- .../simple-pipeline/assets/notion_asset.yml | 9 +++++++++ 2 files changed, 23 insertions(+), 3 deletions(-) create mode 100644 examples/simple-pipeline/assets/notion_asset.yml diff --git a/docs/ingestion/notion.md b/docs/ingestion/notion.md index e2d961bb..88afce91 100644 --- a/docs/ingestion/notion.md +++ b/docs/ingestion/notion.md @@ -14,19 +14,30 @@ To connect to Notion, you need to add a configuration item to the connections se ```yaml connections: notion: - - name: "connection_name" - api_key: "XXXXXXXX" + - name: "my-notion" + api_key: "" ``` ### Step 2: Create an asset file for data ingestion To ingest data from Notion, you need to create an [asset configuration](/assets/ingestr#asset-structure) file. This file defines the data flow from the source to the destination. Create a YAML file (e.g., notion_ingestion.yml) inside the assets folder and add the following content: +```yaml +name: public.notion +type: ingestr +connection: postgres + +parameters: + source_connection: my_notion + source_table: 'd8ee2d159ac34cfc85827ba5a0a8ae71' + + destination: postgress +``` - `name`: The name of the asset. - `type`: Specifies the type of the asset. Set this to ingestr to use the ingestr data pipeline. - `connection`: This is the destination connection, which defines where the data should be stored. For example: `postgres` indicates that the ingested data will be stored in a Postgres database. - `source_connection`: The name of the Notion connection defined in .bruin.yml. -- `source_table`: The name of the data table in Notion that you want to ingest. For example, `details` is the table of notion that you want to ingest. +- `source_table`: The name of the data table in Notion that you want to ingest. Use the `database ID` as the source_table. For example, if the Notion URL is: https://www.notion.so/d8ee2d159ac34cfc85827ba5a0a8ae71?v=c714dec3742440cc91a8c38914f83b6b, the database ID is the string immediately following notion.so/ and preceding any question marks. In this example, the `database ID` is `d8ee2d159ac34cfc85827ba5a0a8ae71`. ### Step 3: [Run](/commands/run) asset to ingest data ``` diff --git a/examples/simple-pipeline/assets/notion_asset.yml b/examples/simple-pipeline/assets/notion_asset.yml new file mode 100644 index 00000000..d3ffae72 --- /dev/null +++ b/examples/simple-pipeline/assets/notion_asset.yml @@ -0,0 +1,9 @@ +name: public.notion +type: ingestr +connection: postgres + +parameters: + source_connection: my_notion + source_table: '1361b9b5fabe807ca41ce24e1c5c7f5b' + + destination: postgres \ No newline at end of file From de821bb6ee2ebfbd3ba302f8cba3cfde55eaed21 Mon Sep 17 00:00:00 2001 From: Sanju Date: Fri, 22 Nov 2024 14:20:21 +0100 Subject: [PATCH 03/17] remove asset example file --- docs/ingestion/notion.md | 2 +- examples/simple-pipeline/assets/notion_asset.yml | 9 --------- 2 files changed, 1 insertion(+), 10 deletions(-) delete mode 100644 examples/simple-pipeline/assets/notion_asset.yml diff --git a/docs/ingestion/notion.md b/docs/ingestion/notion.md index 88afce91..c4ef3da9 100644 --- a/docs/ingestion/notion.md +++ b/docs/ingestion/notion.md @@ -15,7 +15,7 @@ To connect to Notion, you need to add a configuration item to the connections se connections: notion: - name: "my-notion" - api_key: "" + api_key: "YOUR_NOTION_API_KEY" ``` ### Step 2: Create an asset file for data ingestion diff --git a/examples/simple-pipeline/assets/notion_asset.yml b/examples/simple-pipeline/assets/notion_asset.yml deleted file mode 100644 index d3ffae72..00000000 --- a/examples/simple-pipeline/assets/notion_asset.yml +++ /dev/null @@ -1,9 +0,0 @@ -name: public.notion -type: ingestr -connection: postgres - -parameters: - source_connection: my_notion - source_table: '1361b9b5fabe807ca41ce24e1c5c7f5b' - - destination: postgres \ No newline at end of file From 60fd042136697455d376ff3692fe5c567306ada5 Mon Sep 17 00:00:00 2001 From: Sanju Shah <71315276+sanjushahgupta@users.noreply.github.com> Date: Fri, 22 Nov 2024 14:32:33 +0100 Subject: [PATCH 04/17] add image --- docs/ingestion/notion.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/ingestion/notion.md b/docs/ingestion/notion.md index c4ef3da9..477c1034 100644 --- a/docs/ingestion/notion.md +++ b/docs/ingestion/notion.md @@ -43,5 +43,7 @@ parameters: ``` bruin run assets/notion_ingestion.yml ``` +As a result of this command, Bruin will ingest data from the given Notion table into your Postgres database. + +notion -As a result of this command, Bruin will ingest data from the given Notion table into your Postgres database. \ No newline at end of file From 744942a42aab65cb735511ac8ab57cf3a3341ea9 Mon Sep 17 00:00:00 2001 From: Sanju Date: Fri, 22 Nov 2024 16:03:48 +0100 Subject: [PATCH 05/17] zendesk: update doc and change parameter to snake case --- docs/ingestion/zendesk.md | 55 +++++++++++++++++++++++++++++------- pkg/config/connections.go | 2 +- pkg/connection/connection.go | 1 - 3 files changed, 46 insertions(+), 12 deletions(-) diff --git a/docs/ingestion/zendesk.md b/docs/ingestion/zendesk.md index 2fe8c290..10ad742a 100644 --- a/docs/ingestion/zendesk.md +++ b/docs/ingestion/zendesk.md @@ -1,27 +1,62 @@ # Zendesk -[Zendesk](https://www.hubspot.com/) is a cloud-based customer service and support platform. It offers a range of features including ticket management, self-service options, knowledgebase management, live chat, customer analytics, and conversations. +[Zendesk](https://www.zendesk.com/) is a cloud-based customer service and support platform. It offers a range of features including ticket management, self-service options, knowledgebase management, live chat, customer analytics, and conversations. -ingestr supports Zendesk as a source for [ingestr assets](https://bruin-data.github.io/bruin/assets/ingestr.html), allowing you to ingest data from zendesk into your data warehouse. +Bruin supports Zendesk as a source for [Ingestr assets](/assets/ingestr), and you can use it to ingest data from Zendesk into your data warehouse. -In order to have set up Zendesk connection, you need to add a configuration item to `connections` in the `.bruin.yml` file complying with the following schema. -Depending on the data you are ingesting (source table), you will need to use either API Token authentication or OAuth Token authentication. Choose the appropriate method based on your source table. [Ingestr documentation](https://bruin-data.github.io/ingestr/supported-sources/zendesk.html) +In order to set up Zendesk connection, you need to add a configuration item to `connections` in the `.bruin.yml` file and in `asset` file. Depending on the data you are ingesting (source_table), you will need to use either `API Token authentication` or `OAuth Token authentication`. Choose the appropriate method based on your source table. [Ingestr documentation](https://bruin-data.github.io/ingestr/supported-sources/zendesk.html) +Follow the steps below to correctly set up zendesk as a data source and run ingestion. +### Step 1: Add a connection to .bruin.yml file + +To connect to Zendesk, you need to add a configuration item to the connections section of the `.bruin.yml` file. This configuration must comply with the following schema: API Token Authentication: ```yaml connections: zendesk: - - name: "connection_name", - api_key: "xyzKey", + - name: "my_zendesk", + api_token: "xyzKey", email: "example.zendesk@gmail.com", - subdomain: "myCompany", + sub_domain: "myCompany", ``` OAuth Token Authentication: ```yaml connections: zendesk: - - name: "connection_name", + - name: "my_zendesk", oauth_token: "abcToken", - subdomain: "myCompany", -``` \ No newline at end of file + sub_domain: "myCompany", +``` + +- `sub_domain`: the unique Zendesk subdomain that can be found in the account URL. For example, if your account URL is https://my_company.zendesk.com/, then `my_company` is your subdomain +- `email`: the email address of the user +- `api_token`: the API token used for authentication with Zendesk +- `oauth_token`: the OAuth token used for authentication with Zendesk + +### Step 2: Create an asset file for data ingestion +To ingest data from zendesk, you need to create an [asset configuration](/assets/ingestr#asset-structure) file. This file defines the data flow from the source to the destination. Create a YAML file (e.g., zendesk_ingestion.yml) inside the assets folder and add the following content: + +```yaml +name: public.zendesk +type: ingestr +connection: postgres + +parameters: + source_connection: my_zendesk + source_table: 'brands' + + destination: postgress +``` + +- `name`: The name of the asset. +- `type`: Specifies the type of the asset. Set this to ingestr to use the ingestr data pipeline. +- `connection`: This is the destination connection, which defines where the data should be stored. For example: `postgres` indicates that the ingested data will be stored in a Postgres database. +- `source_connection`: The name of the zendesk connection defined in .bruin.yml. +- `source_table`: The name of the data table in zendesk that you want to ingest. You can find the available source tables in Zendesk [here](https://bruin-data.github.io/ingestr/supported-sources/zendesk.html#tables) + +### Step 3: [Run](/commands/run) asset to ingest data +``` +bruin run assets/zendesk_ingestion.yml +``` +As a result of this command, Bruin will ingest data from the given zendesk table into your Postgres database. \ No newline at end of file diff --git a/pkg/config/connections.go b/pkg/config/connections.go index a5cc8419..93fc5d36 100644 --- a/pkg/config/connections.go +++ b/pkg/config/connections.go @@ -404,7 +404,7 @@ type ZendeskConnection struct { APIToken string `yaml:"api_token" json:"api_token" mapstructure:"api_token"` Email string `yaml:"email" json:"email" mapstructure:"email"` OAuthToken string `yaml:"oauth_token" json:"oauth_token" mapstructure:"oauth_token"` - Subdomain string `yaml:"subdomain" json:"subdomain" mapstructure:"subdomain"` + Subdomain string `yaml:"sub_domain" json:"sub_domain" mapstructure:"sub_domain"` } func (c ZendeskConnection) GetName() string { diff --git a/pkg/connection/connection.go b/pkg/connection/connection.go index 4c47f280..b27e2aa8 100644 --- a/pkg/connection/connection.go +++ b/pkg/connection/connection.go @@ -222,7 +222,6 @@ func (m *Manager) GetConnection(name string) (interface{}, error) { connZendesk, err := m.GetZendeskConnectionWithoutDefault(name) if err == nil { - fmt.Println("err", err) return connZendesk, nil } availableConnectionNames = append(availableConnectionNames, maps.Keys(m.Zendesk)...) From a9616a06bead1eeba659d073ad85b2663916f947 Mon Sep 17 00:00:00 2001 From: Sanju Date: Fri, 22 Nov 2024 16:35:15 +0100 Subject: [PATCH 06/17] update test name --- pkg/config/manager_test.go | 4 ++-- pkg/config/testdata/simple.yml | 18 +++++++++--------- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/pkg/config/manager_test.go b/pkg/config/manager_test.go index 95357711..6e10a03e 100644 --- a/pkg/config/manager_test.go +++ b/pkg/config/manager_test.go @@ -264,13 +264,13 @@ func TestLoadFromFile(t *testing.T) { }, Zendesk: []ZendeskConnection{ { - Name: "conn25", + Name: "conn27", APIToken: "zendeskKey", Email: "zendeskemail", Subdomain: "zendeskUrl", }, { - Name: "conn25-1", + Name: "conn27-1", OAuthToken: "zendeskToken", Subdomain: "zendeskUrl", }, diff --git a/pkg/config/testdata/simple.yml b/pkg/config/testdata/simple.yml index f92c8510..bfb887ab 100644 --- a/pkg/config/testdata/simple.yml +++ b/pkg/config/testdata/simple.yml @@ -151,15 +151,6 @@ environments: - name: conn23 base_id: "123" access_token: "accessKey" - zendesk: - - name: conn25 - api_token: "zendeskKey" - email: "zendeskemail" - sub_domain: "zendeskUrl" - - - name: conn25-1 - oauth_token: "zendeskToken" - sub_domain: "zendeskUrl" s3: - name: conn25 bucket_name: "my-bucket" @@ -174,6 +165,15 @@ environments: value: value1 - name: key2 value: value2 + zendesk: + - name: conn27 + api_token: "zendeskKey" + email: "zendeskemail" + sub_domain: "zendeskUrl" + + - name: conn27-1 + oauth_token: "zendeskToken" + sub_domain: "zendeskUrl" prod: connections: From 367d1c8c01775d2ff883e315cad25d680dc4db07 Mon Sep 17 00:00:00 2001 From: Sanju Date: Fri, 22 Nov 2024 16:35:34 +0100 Subject: [PATCH 07/17] update test name --- pkg/config/manager_test.go | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/pkg/config/manager_test.go b/pkg/config/manager_test.go index 6e10a03e..dbbaff1c 100644 --- a/pkg/config/manager_test.go +++ b/pkg/config/manager_test.go @@ -234,12 +234,6 @@ func TestLoadFromFile(t *testing.T) { CredentialsPath: "/path/to/service_account.json", }, }, - Chess: []ChessConnection{ - { - Name: "conn24", - Players: []string{"Max", "Peter"}, - }, - }, Airtable: []AirtableConnection{ { Name: "conn23", @@ -247,6 +241,12 @@ func TestLoadFromFile(t *testing.T) { AccessToken: "accessKey", }, }, + Chess: []ChessConnection{ + { + Name: "conn24", + Players: []string{"Max", "Peter"}, + }, + }, S3: []S3Connection{ { Name: "conn25", From cdb749ec97b958c6e0c7362c2181da3150941fb9 Mon Sep 17 00:00:00 2001 From: Sanju Date: Fri, 22 Nov 2024 17:47:36 +0100 Subject: [PATCH 08/17] fix schema issues --- integration-tests/expected_connections_schema.json | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/integration-tests/expected_connections_schema.json b/integration-tests/expected_connections_schema.json index c05ce212..ec9d7c18 100644 --- a/integration-tests/expected_connections_schema.json +++ b/integration-tests/expected_connections_schema.json @@ -951,7 +951,7 @@ "oauth_token": { "type": "string" }, - "subdomain": { + "sub_domain": { "type": "string" } }, @@ -962,7 +962,7 @@ "api_token", "email", "oauth_token", - "subdomain" + "sub_domain" ] } } From a810b5840117cac67c51aac7d20d7f45e6efd171 Mon Sep 17 00:00:00 2001 From: Sanju Date: Fri, 22 Nov 2024 18:14:37 +0100 Subject: [PATCH 09/17] fix schema issue --- pkg/config/manager_test.go | 16 ++++++++-------- pkg/config/testdata/simple.yml | 18 +++++++++--------- 2 files changed, 17 insertions(+), 17 deletions(-) diff --git a/pkg/config/manager_test.go b/pkg/config/manager_test.go index dbbaff1c..95357711 100644 --- a/pkg/config/manager_test.go +++ b/pkg/config/manager_test.go @@ -234,6 +234,12 @@ func TestLoadFromFile(t *testing.T) { CredentialsPath: "/path/to/service_account.json", }, }, + Chess: []ChessConnection{ + { + Name: "conn24", + Players: []string{"Max", "Peter"}, + }, + }, Airtable: []AirtableConnection{ { Name: "conn23", @@ -241,12 +247,6 @@ func TestLoadFromFile(t *testing.T) { AccessToken: "accessKey", }, }, - Chess: []ChessConnection{ - { - Name: "conn24", - Players: []string{"Max", "Peter"}, - }, - }, S3: []S3Connection{ { Name: "conn25", @@ -264,13 +264,13 @@ func TestLoadFromFile(t *testing.T) { }, Zendesk: []ZendeskConnection{ { - Name: "conn27", + Name: "conn25", APIToken: "zendeskKey", Email: "zendeskemail", Subdomain: "zendeskUrl", }, { - Name: "conn27-1", + Name: "conn25-1", OAuthToken: "zendeskToken", Subdomain: "zendeskUrl", }, diff --git a/pkg/config/testdata/simple.yml b/pkg/config/testdata/simple.yml index bfb887ab..f92c8510 100644 --- a/pkg/config/testdata/simple.yml +++ b/pkg/config/testdata/simple.yml @@ -151,6 +151,15 @@ environments: - name: conn23 base_id: "123" access_token: "accessKey" + zendesk: + - name: conn25 + api_token: "zendeskKey" + email: "zendeskemail" + sub_domain: "zendeskUrl" + + - name: conn25-1 + oauth_token: "zendeskToken" + sub_domain: "zendeskUrl" s3: - name: conn25 bucket_name: "my-bucket" @@ -165,15 +174,6 @@ environments: value: value1 - name: key2 value: value2 - zendesk: - - name: conn27 - api_token: "zendeskKey" - email: "zendeskemail" - sub_domain: "zendeskUrl" - - - name: conn27-1 - oauth_token: "zendeskToken" - sub_domain: "zendeskUrl" prod: connections: From 73f7872027be8e0dd46f20c27a3ef29224bc7fc8 Mon Sep 17 00:00:00 2001 From: Sanju Date: Fri, 22 Nov 2024 18:32:54 +0100 Subject: [PATCH 10/17] update param in test file --- pkg/config/testdata/simple_win.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pkg/config/testdata/simple_win.yml b/pkg/config/testdata/simple_win.yml index d3879ab6..6f1deee4 100644 --- a/pkg/config/testdata/simple_win.yml +++ b/pkg/config/testdata/simple_win.yml @@ -155,11 +155,11 @@ environments: - name: conn25 api_token: "zendeskKey" email: "zendeskemail" - subdomain: "zendeskUrl" + sub_domain: "zendeskUrl" - name: conn25-1 oauth_token: "zendeskToken" - subdomain: "zendeskUrl" + sub_domain: "zendeskUrl" s3: - name: conn25 bucket_name: "my-bucket" From 8a9b8d6d8333e7e87b87d3513828072db855e377 Mon Sep 17 00:00:00 2001 From: Sanju Date: Mon, 25 Nov 2024 14:50:55 +0100 Subject: [PATCH 11/17] add shopify docs --- docs/ingestion/shopify.md | 50 ++++++++++++++++++++++++++++++++------- 1 file changed, 41 insertions(+), 9 deletions(-) diff --git a/docs/ingestion/shopify.md b/docs/ingestion/shopify.md index 6b271fda..3b6bb13f 100644 --- a/docs/ingestion/shopify.md +++ b/docs/ingestion/shopify.md @@ -1,15 +1,47 @@ # Shopify +[Shopify](https://www.Shopify.com/) is a comprehensive e-commerce platform that enables individuals and businesses to create online stores. -In order to have set up a Shopify connection, you need to add a configuration item to connections in the .bruin.yml file complying with the following schema. +ingestr supports Shopify as a source for [ingestr assets](https://bruin-data.github.io/bruin/assets/ingestr.html), allowing you to ingest data from Shopify into your data warehouse. +To set up a Shopify connection, you need to have Shopify API key, shopify store URI and source table. For more information, read [here](https://bruin-data.github.io/ingestr/supported-sources/shopify.html) + +Follow the steps below to correctly set up shopify as a data source and run ingestion: + +**Step 1: Create an Asset File for Data Ingestion** + +To ingest data from Shopify, you need to create an [asset configuration file](https://bruin-data.github.io/bruin/assets/ingestr.html#template). This file defines the data flow from the source to the destination. +(For e.g., ingestr.shopify.asset.yml) and add the following content: + +***File: ingestr.shopify.asset.yml*** ```yaml -connections: - shopify: - - name: "shopify-default" - api_key: "********" - url: "" +name: public.shopify +type: ingestr +connection: postgres + +parameters: + source_connection: my_shopify + source_table: 'order' + destination: postgres ``` +- name: The name of the asset. +- type: Specifies the type of the asset. It will be always ingestr type for Shopify. +- connection: This is the destination connection. +**parameters:** +- source_connection: The name of the Shopify connection defined in .bruin.yml. +- source_table: The name of the data table in shopify you want to ingest. For example, "order" would ingest data related to order. + [Available source tables in Shopify](https://bruin-data.github.io/ingestr/supported-sources/shopify.html#available-tables) +Step 2: Add a Connection to [.bruin.yml](https://bruin-data.github.io/bruin/connections/overview.html) that stores connections and secrets to be used in pipelines. +You need to add a configuration item to `connections` in the `.bruin.yml` file complying with the following schema. -The following fields are required: -- `url`: Your Shopify store's URL -- `api_key`: A private app access token or admin API access token \ No newline at end of file +***File: .bruin.yml*** +```yaml + connections: + shopify: + - name: "my_Shopify" + api_key: "YOUR_Shopify_API_KEY" +``` +**Step 3: [Run](https://bruin-data.github.io/bruin/commands/run.html) Asset to Ingest Data** +``` +bruin run ingestr.shopify.asset.yml +``` +It will ingest shopify data to postgres. \ No newline at end of file From a3a7c3eea63dfcafb325e486239e96afbdbaf166 Mon Sep 17 00:00:00 2001 From: Sanju Shah <71315276+sanjushahgupta@users.noreply.github.com> Date: Mon, 25 Nov 2024 15:51:33 +0100 Subject: [PATCH 12/17] add images --- docs/ingestion/zendesk.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/ingestion/zendesk.md b/docs/ingestion/zendesk.md index 10ad742a..b5d731c3 100644 --- a/docs/ingestion/zendesk.md +++ b/docs/ingestion/zendesk.md @@ -59,4 +59,6 @@ parameters: ``` bruin run assets/zendesk_ingestion.yml ``` -As a result of this command, Bruin will ingest data from the given zendesk table into your Postgres database. \ No newline at end of file +As a result of this command, Bruin will ingest data from the given zendesk table into your Postgres database. + +zendesk From d266131798dcb42914a903b24ad8ca8982428495 Mon Sep 17 00:00:00 2001 From: Sanju Date: Mon, 25 Nov 2024 16:40:29 +0100 Subject: [PATCH 13/17] update docs --- docs/.vitepress/config.mjs | 2 +- docs/ingestion/notion.md | 2 ++ docs/ingestion/shopify.md | 58 +++++++++++++++++++------------------- docs/ingestion/zendesk.md | 2 +- 4 files changed, 33 insertions(+), 31 deletions(-) diff --git a/docs/.vitepress/config.mjs b/docs/.vitepress/config.mjs index 153ef522..5fcbbb93 100644 --- a/docs/.vitepress/config.mjs +++ b/docs/.vitepress/config.mjs @@ -119,9 +119,9 @@ export default defineConfig({ {text: "Notion", link: "/ingestion/notion"}, {text: "SAP HANA", link: "/ingestion/sap_hana"}, {text: "S3", link: "/ingestion/s3"}, + {text: "Shopify", link: "/ingestion/shopify"}, {text: "Slack", link: "/ingestion/slack.md"}, {text: "Zendesk", link: "/ingestion/zendesk.md"}, - {text: "Shopify", link: "/ingestion/shopify"}, ], }, ], diff --git a/docs/ingestion/notion.md b/docs/ingestion/notion.md index 477c1034..8f512500 100644 --- a/docs/ingestion/notion.md +++ b/docs/ingestion/notion.md @@ -17,6 +17,8 @@ To connect to Notion, you need to add a configuration item to the connections se - name: "my-notion" api_key: "YOUR_NOTION_API_KEY" ``` + +- `api_key`: the API key used for authentication with Notion ### Step 2: Create an asset file for data ingestion To ingest data from Notion, you need to create an [asset configuration](/assets/ingestr#asset-structure) file. This file defines the data flow from the source to the destination. Create a YAML file (e.g., notion_ingestion.yml) inside the assets folder and add the following content: diff --git a/docs/ingestion/shopify.md b/docs/ingestion/shopify.md index 3b6bb13f..88093405 100644 --- a/docs/ingestion/shopify.md +++ b/docs/ingestion/shopify.md @@ -1,47 +1,47 @@ # Shopify [Shopify](https://www.Shopify.com/) is a comprehensive e-commerce platform that enables individuals and businesses to create online stores. -ingestr supports Shopify as a source for [ingestr assets](https://bruin-data.github.io/bruin/assets/ingestr.html), allowing you to ingest data from Shopify into your data warehouse. +Bruin supports Shopify as a source for [Ingestr assets](/assets/ingestr), and you can use it to ingest data from Shopify into your data warehouse. + + +In order to set up Shopify connection, you need to add a configuration item in the `.bruin.yml` file and in `asset` file. You need `api_key`. For details on how to obtain these credentials, please refer [here](https://dlthub.com/docs/dlt-ecosystem/verified-sources/shopify#setup-guide). -To set up a Shopify connection, you need to have Shopify API key, shopify store URI and source table. For more information, read [here](https://bruin-data.github.io/ingestr/supported-sources/shopify.html) Follow the steps below to correctly set up shopify as a data source and run ingestion: -**Step 1: Create an Asset File for Data Ingestion** +### Step 1: Add a connection to .bruin.yml file +To ingest data from Shopify, you need to create an [asset configuration](/assets/ingestr#asset-structure) file. This file defines the data flow from the source to the destination. Create a YAML file (e.g., shopify_ingestion.yml) inside the assets folder and add the following content: + +```yaml + connections: + shopify: + - name: my-shopify + url: hello.myshopify.com + api_key: abckey +``` +- `api_key`: the API key used for authentication with Shopify -To ingest data from Shopify, you need to create an [asset configuration file](https://bruin-data.github.io/bruin/assets/ingestr.html#template). This file defines the data flow from the source to the destination. -(For e.g., ingestr.shopify.asset.yml) and add the following content: +### Step 2: Create an asset file for data ingestion -***File: ingestr.shopify.asset.yml*** ```yaml name: public.shopify type: ingestr connection: postgres parameters: - source_connection: my_shopify - source_table: 'order' + source_connection: my-shopify + source_table: 'orders' + destination: postgres ``` -- name: The name of the asset. -- type: Specifies the type of the asset. It will be always ingestr type for Shopify. -- connection: This is the destination connection. -**parameters:** -- source_connection: The name of the Shopify connection defined in .bruin.yml. -- source_table: The name of the data table in shopify you want to ingest. For example, "order" would ingest data related to order. - [Available source tables in Shopify](https://bruin-data.github.io/ingestr/supported-sources/shopify.html#available-tables) -Step 2: Add a Connection to [.bruin.yml](https://bruin-data.github.io/bruin/connections/overview.html) that stores connections and secrets to be used in pipelines. -You need to add a configuration item to `connections` in the `.bruin.yml` file complying with the following schema. - -***File: .bruin.yml*** -```yaml - connections: - shopify: - - name: "my_Shopify" - api_key: "YOUR_Shopify_API_KEY" -``` -**Step 3: [Run](https://bruin-data.github.io/bruin/commands/run.html) Asset to Ingest Data** -``` -bruin run ingestr.shopify.asset.yml +- `name`: The name of the asset. +- `type`: Specifies the type of the asset. It will be always ingestr type for Shopify. +- `connection`: This is the destination connection. +- `source_connection`: The name of the Shopify connection defined in .bruin.yml. +- `source_table`: The name of the data table in shopify you want to ingest. For example, "order" would ingest data related to order.You can find the available source tables in Shopify [here](https://bruin-data.github.io/ingestr/supported-sources/shopify.html#available-tables) + +### Step 3: [Run](/commands/run) asset to ingest data +``` +bruin run assets/shopify_ingestion.yml ``` -It will ingest shopify data to postgres. \ No newline at end of file +As a result of this command, Bruin will ingest data from the given Shopify table into your Postgres database. ` \ No newline at end of file diff --git a/docs/ingestion/zendesk.md b/docs/ingestion/zendesk.md index b5d731c3..d298cbe1 100644 --- a/docs/ingestion/zendesk.md +++ b/docs/ingestion/zendesk.md @@ -3,7 +3,7 @@ Bruin supports Zendesk as a source for [Ingestr assets](/assets/ingestr), and you can use it to ingest data from Zendesk into your data warehouse. -In order to set up Zendesk connection, you need to add a configuration item to `connections` in the `.bruin.yml` file and in `asset` file. Depending on the data you are ingesting (source_table), you will need to use either `API Token authentication` or `OAuth Token authentication`. Choose the appropriate method based on your source table. [Ingestr documentation](https://bruin-data.github.io/ingestr/supported-sources/zendesk.html) +In order to set up Zendesk connection, you need to add a configuration item to `connections` in the `.bruin.yml` file and in `asset` file. Depending on the data you are ingesting (source_table), you will need to use either `API Token authentication` or `OAuth Token authentication`. Choose the appropriate method based on your source table. For more details, please refer to the [Ingestr documentation](https://bruin-data.github.io/ingestr/supported-sources/zendesk.html) Follow the steps below to correctly set up zendesk as a data source and run ingestion. ### Step 1: Add a connection to .bruin.yml file From d4ef7d034918d1f5e2f954cb945a985519ea27cd Mon Sep 17 00:00:00 2001 From: Sanju Shah <71315276+sanjushahgupta@users.noreply.github.com> Date: Mon, 25 Nov 2024 16:54:32 +0100 Subject: [PATCH 14/17] add image --- docs/ingestion/shopify.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/ingestion/shopify.md b/docs/ingestion/shopify.md index 88093405..cba4c798 100644 --- a/docs/ingestion/shopify.md +++ b/docs/ingestion/shopify.md @@ -3,10 +3,8 @@ Bruin supports Shopify as a source for [Ingestr assets](/assets/ingestr), and you can use it to ingest data from Shopify into your data warehouse. - In order to set up Shopify connection, you need to add a configuration item in the `.bruin.yml` file and in `asset` file. You need `api_key`. For details on how to obtain these credentials, please refer [here](https://dlthub.com/docs/dlt-ecosystem/verified-sources/shopify#setup-guide). - Follow the steps below to correctly set up shopify as a data source and run ingestion: ### Step 1: Add a connection to .bruin.yml file @@ -16,7 +14,7 @@ To ingest data from Shopify, you need to create an [asset configuration](/assets connections: shopify: - name: my-shopify - url: hello.myshopify.com + url: test.myshopify.com api_key: abckey ``` - `api_key`: the API key used for authentication with Shopify @@ -44,4 +42,6 @@ parameters: ``` bruin run assets/shopify_ingestion.yml ``` -As a result of this command, Bruin will ingest data from the given Shopify table into your Postgres database. ` \ No newline at end of file +As a result of this command, Bruin will ingest data from the given Shopify table into your Postgres database. + +shopify From aff048fcee202afd8191e3a40a48694a68dd1fe7 Mon Sep 17 00:00:00 2001 From: Sanju Shah <71315276+sanjushahgupta@users.noreply.github.com> Date: Mon, 25 Nov 2024 17:04:49 +0100 Subject: [PATCH 15/17] fix typo --- docs/ingestion/zendesk.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ingestion/zendesk.md b/docs/ingestion/zendesk.md index d298cbe1..5846e26e 100644 --- a/docs/ingestion/zendesk.md +++ b/docs/ingestion/zendesk.md @@ -46,7 +46,7 @@ parameters: source_connection: my_zendesk source_table: 'brands' - destination: postgress + destination: postgres ``` - `name`: The name of the asset. From 19d616725167c0c65d95314d0ca8160c4e80686a Mon Sep 17 00:00:00 2001 From: Sanju Shah <71315276+sanjushahgupta@users.noreply.github.com> Date: Mon, 25 Nov 2024 17:13:43 +0100 Subject: [PATCH 16/17] fix typo --- docs/ingestion/shopify.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ingestion/shopify.md b/docs/ingestion/shopify.md index cba4c798..2dc41431 100644 --- a/docs/ingestion/shopify.md +++ b/docs/ingestion/shopify.md @@ -3,7 +3,7 @@ Bruin supports Shopify as a source for [Ingestr assets](/assets/ingestr), and you can use it to ingest data from Shopify into your data warehouse. -In order to set up Shopify connection, you need to add a configuration item in the `.bruin.yml` file and in `asset` file. You need `api_key`. For details on how to obtain these credentials, please refer [here](https://dlthub.com/docs/dlt-ecosystem/verified-sources/shopify#setup-guide). +In order to set up Shopify connection, you need to add a configuration item in the `.bruin.yml` file and in `asset` file. You need the `url` of your Shopify store` and the `api_key`. For details on how to obtain these credentials, please refer [here](https://dlthub.com/docs/dlt-ecosystem/verified-sources/shopify#setup-guide). Follow the steps below to correctly set up shopify as a data source and run ingestion: From d779533cd81e575eab0c1728106a2a8307e0e046 Mon Sep 17 00:00:00 2001 From: Sanju Shah <71315276+sanjushahgupta@users.noreply.github.com> Date: Mon, 25 Nov 2024 17:15:44 +0100 Subject: [PATCH 17/17] omit comma --- docs/ingestion/shopify.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ingestion/shopify.md b/docs/ingestion/shopify.md index 2dc41431..6a51bb75 100644 --- a/docs/ingestion/shopify.md +++ b/docs/ingestion/shopify.md @@ -3,7 +3,7 @@ Bruin supports Shopify as a source for [Ingestr assets](/assets/ingestr), and you can use it to ingest data from Shopify into your data warehouse. -In order to set up Shopify connection, you need to add a configuration item in the `.bruin.yml` file and in `asset` file. You need the `url` of your Shopify store` and the `api_key`. For details on how to obtain these credentials, please refer [here](https://dlthub.com/docs/dlt-ecosystem/verified-sources/shopify#setup-guide). +In order to set up Shopify connection, you need to add a configuration item in the `.bruin.yml` file and in `asset` file. You need the `url` of your Shopify store and the `api_key`. For details on how to obtain these credentials, please refer [here](https://dlthub.com/docs/dlt-ecosystem/verified-sources/shopify#setup-guide). Follow the steps below to correctly set up shopify as a data source and run ingestion: