Skip to content

Commit

Permalink
Deploy preview for PR 213 🛫
Browse files Browse the repository at this point in the history
  • Loading branch information
@sc250072
sc250072 committed Jul 25, 2024
1 parent 0daaca6 commit 56268c6
Show file tree
Hide file tree
Showing 4 changed files with 349 additions and 325 deletions.
110 changes: 61 additions & 49 deletions pr-preview/pr-213/airflow/airflow-azure-to-teradata-transfer-operator-doc.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2303,18 +2303,6 @@ <h2 id="_prerequisites"><a class="anchor" href="#_prerequisites"></a>Prerequisit
</div>
</div>
</li>
<li>
<p>An Azure account. You can start with a <a href="https://azure.microsoft.com/free/">free account</a>.</p>
</li>
<li>
<p>Create an <a href="https://docs.microsoft.com/en-us/azure/storage/common/storage-quickstart-create-account?tabs=azure-portal">Azure storage account</a> with name <code>azuretestquickstart</code></p>
</li>
<li>
<p>Create a <a href="https://learn.microsoft.com/en-us/azure/storage/blobs/blob-containers-portal">blob container</a> with name <code>quickstartdemo</code> under Azure storage account <code>azuretestquickstart</code></p>
</li>
<li>
<p><a href="https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal">Upload</a> CSV/JSON/Parquest format files to <code>quickstartdemo</code> blob container.</p>
</li>
</ul>
</div>
</div>
Expand Down Expand Up @@ -2358,40 +2346,6 @@ <h2 id="_install_apache_airflow"><a class="anchor" href="#_install_apache_airflo
</div>
</div>
<div class="sect1">
<h2 id="_create_teradata_authorization"><a class="anchor" href="#_create_teradata_authorization"></a>Create Teradata Authorization</h2>
<div class="sectionbody">
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
A database client connected to Vantage is needed to execute SQL statements. <a href="https://downloads.teradata.com/download/tools/teradata-studio-express">Teradata Studio Express</a> or <a href="https://quickstarts.teradata.com/other-integrations/configure-a-teradata-vantage-connection-in-dbeaver.html">dbeaver</a> can be used for this purpose.
</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlightjs highlight"><code class="language-teradata-sql hljs" data-lang="teradata-sql">CREATE AUTHORIZATION azure_authorization USER 'azuretestquickstart' PASSWORD 'AZURE_BLOB_ACCOUNT_SECRET_KEY'</code></pre>
</div>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
Replace <code>AZURE_BLOB_ACCOUNT_SECRET_KEY</code> with Azure storage account <code>azuretestquickstart</code> <a href="https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&amp;bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json&amp;tabs=azure-portal">access key</a>
</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_configure_apache_airflow"><a class="anchor" href="#_configure_apache_airflow"></a>Configure Apache Airflow</h2>
<div class="sectionbody">
<div class="olist arabic">
Expand Down Expand Up @@ -2490,7 +2444,7 @@ <h2 id="_define_apache_airflow_connection_to_vantage"><a class="anchor" href="#_
<h2 id="_define_dag_in_apache_airflow"><a class="anchor" href="#_define_dag_in_apache_airflow"></a>Define DAG in Apache Airflow</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Dags in airflow are defined as python files. The dag below transfer CSV/JSON/Parquet data from Microsoft Azure Blob Storage to Teradata Vantage instance.Copy the python code below and save it as <code>airflow-azure-to-teradata-transfer-operator-demo.py</code> under the directory $AIRFLOW_HOME/dags.</p>
<p>Dags in airflow are defined as python files. The dag below transfer CSV data from Teradata-supplied public blob containers to Teradata Vantage instance.Copy the python code below and save it as <code>airflow-azure-to-teradata-transfer-operator-demo.py</code> under the directory $AIRFLOW_HOME/dags.</p>
</div>
<div class="paragraph">
<p>Refer <a href="https://airflow.apache.org/docs/apache-airflow-providers-teradata/stable/_api/airflow/providers/teradata/transfers/azure_blob_to_teradata/index.html">AzureBlobStorageToTeradataOperator</a> for more information on <code>Azure Blob Storage to Teradata</code> Transfer Operator.</p>
Expand Down Expand Up @@ -2523,10 +2477,9 @@ <h2 id="_define_dag_in_apache_airflow"><a class="anchor" href="#_define_dag_in_a
# Transfer data from Azure blob storage to Teradata Vantage instance
transfer_data_csv = AzureBlobStorageToTeradataOperator(
task_id="transfer_data_blob_to_teradata_csv",
blob_source_key="/az/azuretestquickstart.blob.core.windows.net/quickstartdemo",
blob_source_key="/az/akiaxox5jikeotfww4ul.blob.core.windows.net/td-usgs/CSVDATA/",
teradata_table="example_blob_teradata_csv",
teradata_conn_id="teradata_default",
teradata_authorization_name="azure_authorization",
trigger_rule="always",
)
# Get the number of records transferred from azure blob to teradata table
Expand Down Expand Up @@ -2571,6 +2524,65 @@ <h2 id="_run_dag"><a class="anchor" href="#_run_dag"></a>Run DAG</h2>
</div>
</div>
<div class="sect1">
<h2 id="_transfer_data_from_private_blob_storage_container_to_teradata_instance"><a class="anchor" href="#_transfer_data_from_private_blob_storage_container_to_teradata_instance"></a>Transfer data from Private Blob Storage Container to Teradata instance</h2>
<div class="sectionbody">
<div class="paragraph">
<p>To successfully transfer data from a Private Blob Storage Container to a Teradata instance, the following prerequisites are necessary.</p>
</div>
<div class="ulist">
<ul>
<li>
<p>An Azure account. You can start with a <a href="https://azure.microsoft.com/free/">free account</a>.</p>
</li>
<li>
<p>Create an <a href="https://docs.microsoft.com/en-us/azure/storage/common/storage-quickstart-create-account?tabs=azure-portal">Azure storage account</a></p>
</li>
<li>
<p>Create a <a href="https://learn.microsoft.com/en-us/azure/storage/blobs/blob-containers-portal">blob container</a> under Azure storage account</p>
</li>
<li>
<p><a href="https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal">Upload</a> CSV/JSON/Parquest format files to blob container</p>
</li>
<li>
<p>Create Teradata Authorization object with Azure Blob Storage Account and Account Secret Key</p>
<div class="listingblock">
<div class="content">
<pre class="highlightjs highlight"><code class="language-teradata-sql hljs" data-lang="teradata-sql">CREATE AUTHORIZATION azure_authorization USER 'azuretestquickstart' PASSWORD 'AZURE_BLOB_ACCOUNT_SECRET_KEY'</code></pre>
</div>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
Replace <code>AZURE_BLOB_ACCOUNT_SECRET_KEY</code> with Azure storage account <code>azuretestquickstart</code> <a href="https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&amp;bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json&amp;tabs=azure-portal">access key</a>
</td>
</tr>
</table>
</div>
</li>
<li>
<p>Modify <code>blob_source_key</code> with YOUR-PRIVATE-OBJECT-STORE-URI in <code>transfer_data_csv</code> task and add <code>teradata_authorization_name</code> field with Teradata Authorization Object name</p>
<div class="listingblock">
<div class="content">
<pre class="highlightjs highlight"><code class="language-python hljs" data-lang="python">transfer_data_csv = AzureBlobStorageToTeradataOperator(
task_id="transfer_data_blob_to_teradata_csv",
blob_source_key="YOUR-OBJECT-STORE-URI",
teradata_table="example_blob_teradata_csv",
teradata_conn_id="teradata_default",
teradata_authorization_name="azure_authorization",
trigger_rule="always",
)</code></pre>
</div>
</div>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_summary"><a class="anchor" href="#_summary"></a>Summary</h2>
<div class="sectionbody">
<div class="paragraph">
Expand Down
110 changes: 61 additions & 49 deletions pr-preview/pr-213/airflow/airflow-s3-to-teradata-transfer-operator-doc.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2303,18 +2303,6 @@ <h2 id="_prerequisites"><a class="anchor" href="#_prerequisites"></a>Prerequisit
</div>
</div>
</li>
<li>
<p>Access to an <a href="https://aws.amazon.com">Amazon AWS account</a></p>
</li>
<li>
<p>Create a <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html">S3 bucket</a> with name <code>awstestquickstart</code></p>
</li>
<li>
<p><a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/uploading-an-object-bucket.html">Upload</a> CSV/JSON/Parquest format files to <code>awstestquickstart</code> S3 bucket.</p>
</li>
<li>
<p><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey">Access Keys</a> to access AWS account</p>
</li>
</ul>
</div>
</div>
Expand Down Expand Up @@ -2358,40 +2346,6 @@ <h2 id="_install_apache_airflow"><a class="anchor" href="#_install_apache_airflo
</div>
</div>
<div class="sect1">
<h2 id="_create_teradata_authorization"><a class="anchor" href="#_create_teradata_authorization"></a>Create Teradata Authorization</h2>
<div class="sectionbody">
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
A database client connected to Vantage is needed to execute SQL statements. <a href="https://downloads.teradata.com/download/tools/teradata-studio-express">Teradata Studio Express</a> or <a href="https://quickstarts.teradata.com/other-integrations/configure-a-teradata-vantage-connection-in-dbeaver.html">dbeaver</a> can be used for this purpose.
</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlightjs highlight"><code class="language-teradata-sql hljs" data-lang="teradata-sql">CREATE AUTHORIZATION aws_authorization USER 'AWSAccessKey' PASSWORD 'AWSSecretAccessKey'</code></pre>
</div>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
Replace <code>AWSAccessKey</code> and <code>AWSSecretAccessKey</code> with your AWS account <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html">access key</a>
</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_configure_apache_airflow"><a class="anchor" href="#_configure_apache_airflow"></a>Configure Apache Airflow</h2>
<div class="sectionbody">
<div class="olist arabic">
Expand Down Expand Up @@ -2490,7 +2444,7 @@ <h2 id="_define_apache_airflow_connection_to_vantage"><a class="anchor" href="#_
<h2 id="_define_dag_in_apache_airflow"><a class="anchor" href="#_define_dag_in_apache_airflow"></a>Define DAG in Apache Airflow</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Dags in airflow are defined as python files. The dag below transfer CSV/JSON/Parquet data from Amazon S3 to Teradata Vantage instance.Copy the python code below and save it as <code>airflow-aws-to-teradata-transfer-operator-demo.py</code> under the directory $AIRFLOW_HOME/dags.</p>
<p>Dags in airflow are defined as python files. The dag below transfer data from Teradata-supplied public buckets to Teradata Vantage instance.Copy the python code below and save it as <code>airflow-aws-to-teradata-transfer-operator-demo.py</code> under the directory $AIRFLOW_HOME/dags.</p>
</div>
<div class="paragraph">
<p>Refer <a href="https://airflow.apache.org/docs/apache-airflow-providers-teradata/stable/operators/s3_to_teradata.html">S3ToTeradataOperator</a> for more information on <code>Amazon S3 to Teradata</code> Transfer Operator.</p>
Expand Down Expand Up @@ -2523,10 +2477,9 @@ <h2 id="_define_dag_in_apache_airflow"><a class="anchor" href="#_define_dag_in_a
# Transfer data from S3 to Teradata Vantage instance
transfer_data_csv = S3ToTeradataOperator(
task_id="transfer_data_s3_to_teradata_csv",
s3_source_key="/s3/awstestquickstart.s3.amazonaws.com/",
s3_source_key="/s3/td-usgs-public.s3.amazonaws.com/CSVDATA/",
teradata_table="example_s3_teradata_csv",
teradata_conn_id="teradata_default",
teradata_authorization_name="aws_authorization",
trigger_rule="always",
)
# Get the number of records transferred from S3 to teradata table
Expand Down Expand Up @@ -2571,6 +2524,65 @@ <h2 id="_run_dag"><a class="anchor" href="#_run_dag"></a>Run DAG</h2>
</div>
</div>
<div class="sect1">
<h2 id="_transfer_data_from_private_amazon_s3_bucket_to_teradata_instance"><a class="anchor" href="#_transfer_data_from_private_amazon_s3_bucket_to_teradata_instance"></a>Transfer data from Private Amazon S3 bucket to Teradata instance</h2>
<div class="sectionbody">
<div class="paragraph">
<p>To successfully transfer data from Private Amazon S3 bucket to a Teradata instance, the following prerequisites are necessary.</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Access to an <a href="https://aws.amazon.com">Amazon AWS account</a></p>
</li>
<li>
<p>Create a <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html">S3 bucket</a></p>
</li>
<li>
<p><a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/uploading-an-object-bucket.html">Upload</a> CSV/JSON/Parquest format files to Private S3 bucket.</p>
</li>
<li>
<p><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey">Access Keys</a> to access AWS account</p>
</li>
<li>
<p>Create Teradata Authorization object with AWS Account Key and Account Secret Key</p>
<div class="listingblock">
<div class="content">
<pre class="highlightjs highlight"><code class="language-teradata-sql hljs" data-lang="teradata-sql">CREATE AUTHORIZATION aws_authorization USER 'AWSAccessKey' PASSWORD 'AWSSecretAccessKey'</code></pre>
</div>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
Replace <code>AWSAccessKey</code> and <code>AWSSecretAccessKey</code> with your AWS account <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html">access key</a>
</td>
</tr>
</table>
</div>
</li>
<li>
<p>Modify <code>blob_source_key</code> with YOUR-PRIVATE-OBJECT-STORE-URI in <code>transfer_data_csv</code> task and add <code>teradata_authorization_name</code> field with Teradata Authorization Object name</p>
<div class="listingblock">
<div class="content">
<pre class="highlightjs highlight"><code class="language-python hljs" data-lang="python">transfer_data_csv = S3ToTeradataOperator(
task_id="transfer_data_blob_to_teradata_csv",
s3_source_key="YOUR-OBJECT-STORE-URI",
teradata_table="example_blob_teradata_csv",
teradata_conn_id="teradata_default",
teradata_authorization_name="aws_authorization",
trigger_rule="always",
)</code></pre>
</div>
</div>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_summary"><a class="anchor" href="#_summary"></a>Summary</h2>
<div class="sectionbody">
<div class="paragraph">
Expand Down
2 changes: 1 addition & 1 deletion pr-preview/pr-213/search-index.js
View file

Large diffs are not rendered by default.

Loading

0 comments on commit 56268c6

Please sign in to comment.