The page you’re looking for does not exist. It may have been moved. You can follow one of the links in the navigation to the left.
+If you arrived on this page by clicking on a link, please notify the owner of the site that the link is broken. +If you typed the URL of this page manually, please double check that you entered the address correctly.
+Author: Daniel Herrera
+Last updated: May 22th, 2023
This project showcases the integration of dbt with Teradata Vantage from an advanced user perspective. +If you are new to data engineering with dbt we recommend that you start with our introductory project.
+The advanced use cases showcased in the demo are the following:
+Incremental materializations
+Utility macros
+Optimizing table/view creations with Teradata-specific modifiers
+The application of these concepts is illustrated through the ELT process of teddy_retailers, a fictional store.
Access to a Teradata Vantage instance.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
Python 3.7, 3.8, 3.9 or 3.10 installed.
+A database client for running database commands, an example of the configuration of one such client is presented in this tutorial..
+Clone the tutorial repository and cd into the project directory:
+git clone https://github.com/Teradata/teddy_retailers_dbt-dev teddy_retailers
+cd teddy_retailersCreate a new python environment to manage dbt and its dependencies. Confirm that the Python Version you are using to create the environment is within the supported versions listed above.
+python -m venv envActivate the python environment according to your operating system.
+source env/bin/activatefor Mac, Linux, or
+env\Scripts\activatefor Windows
+Install the dbt-teradata module. The core dbt module is included as a dependency so you don’t have to install it separately:
pip install dbt-teradataInstall the project’s dependencies dbt-utils and teradata-utils. This can be done through the following command:
dbt depsThe demo project assumes that the source data is already loaded into your data warehouse, this mimics the way that dbt is used in a production environment.
+To achieve this objective we provide public datasets available in Google Cload Platform (GCP), and scripts to load those datasets into your mock data warehouse.
Create or select a working database. The dbt profile in the project points to a database called teddy_retailers. You can change the schema value to point to an existing database in your Teradata Vantage instance or you can create the teddy_retailers database running the following script in your database client:
CREATE DATABASE teddy_retailers
+AS PERMANENT = 110e6,
+ SPOOL = 220e6;Load Initial data set.
+To load the initial data set into the data warehouse, the required scripts are available in the references/inserts/create_data.sql path of the project.
+You can execute these scripts by copying and pasting them into your database client. For guidance on running these scripts in your specific case please consult your database client’s documentation.
We will now configure dbt to connect to your Vantage database.
+Create the file $HOME/.dbt/profiles.yml with the following content. Adjust <host>, <user>, <password> to match your Teradata Vantage instance.
+If you have already used dbt before in your environment you only need to add a profile for the project in your home’s directory .dbt/profiles.yml file.
+If the directory .dbt doesn’t exist in your system yet you will need to create it and add the profiles.yml to manage your dbt profiles.
teddy_retailers:
+ outputs:
+ dev:
+ type: teradata
+ host: <host>
+ user: <user>
+ password: <password>
+ logmech: TD2
+ schema: teddy_retailers
+ tmode: ANSI
+ threads: 1
+ timeout_seconds: 300
+ priority: interactive
+ retries: 1
+ target: devNow, that we have the profile file in place, we can validate the setup:
+dbt debugIf the debug command returned errors, you likely have an issue with the content of profiles.yml.
As mentioned, teddy_retailers is a fictional store.
+Through dbt driven transformations we transform source data ingested from the`teddy_retailers` transactional database into a star schema ready for analytics.
The source data consists of the following tables customers, orders, products, and order_products, according to the following Entity Relations Diagram:
+
+Using dbt, we leverage the source data tables to construct the following dimensional model, which is optimized for analytics tools.
+
+For Teddy Retailers, the orders and order_products sources are periodically updated by the organization’s ELT (Extract, Load, Transform) process.
The updated data only includes the latest changes rather than the entire dataset due to its large volume.
+To address this challenge, it is necessary to capture these incremental updates while preserving the previously available data.
+The schema.yml file in the project’s models directory specifies the sources for our models. These sources align with the data we loaded from GCP using our SQL scripts.
The staging area models are merely ingesting the data from each of the sources and renaming each field, if appropiate. +In the schema.yml of this directory we define basic integrity checks for the primary keys.
+The following advanced dbt concepts are applied in the models at this stage:
+The schema.yml file in this directory specifies that the materializations of the two models we are building are incremental.
+We employ different strategies for these models:
For the all_orders model, we utilize the delete+insert strategy. This strategy is implemented because there may be changes in the status of an order that are included in the data updates.
For the all_order_products model, we employ the default append strategy. This approach is chosen because the same combination of order_id and product_id may appear multiple times in the sources.
+This indicates that a new quantity of the same product has been added or removed from a specific order.
Within the all_order_products model, we have included an assertion with the help of a macro to test and guarantee that the resulting model encompasses a unique combination of order_id and product_id. This combination denotes the latest quantity of products of a specific type per order.
For both the all_order and all_order_products models, we have incorporated Teradata Modifiers to enhance tracking of these two core models.
+To facilitate collecting statistics, we have added a post_hook that instructs the database connector accordingly. Additionally, we have created an index on the order_id column within the all_orders table.
By executing dbt, we generate the dimensional model using the baseline data.
+dbt runThis will create both our core and dimensional models using the baseline data.
+You can find sample business intelligence queries in the references/query path of the project. These queries allow you to analyze the factual data based on dimensions such as customers, orders, and products.
The scripts for loading updates into the source data set can be found in the references/inserts/update_data.sql path of the project.
After updating the data sources, you can proceed with the aforementioned steps: running dbt, testing the data, and executing sample queries. This will allow you to visualize the variations and incremental updates in the data.
+In this tutorial, we explored the utilization of advanced dbt concepts with Teradata Vantage. +The sample project showcased the transformation of source data into a dimensional data mart. +Throughout the project, we implemented several advanced dbt concepts, including incremental materializations, utility macros, and Teradata modifiers.
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Kevin Bogusch, Paul Ibberson
+Last updated: January 14th, 2022
| + + | ++This guide includes content from both Microsoft and Teradata product documentation. + | +
This article describes the process to connect your Power BI Desktop to Teradata Vantage for creating reports and dramatic visualizations of your data. Power BI supports Teradata Vantage as a data source and can use the underlying data just like any other data source in Power BI Desktop.
+Power BI is a collection of software services, applications, and connectors that work together to turn your unrelated sources of data into coherent, visually immersive, and interactive insights.
+A Windows desktop application, called Power BI Desktop
+An online SaaS (Software as a Service) service, called the Power BI service
+Power BI mobile apps for Windows, iOS, and Android devices
+
+These three elements—Power BI Desktop, the Power BI service, and the mobile apps—are designed to let people create, share, and consume business insights in the way that serves them, or their role, most effectively.
+
+A fourth element, Power BI Report Server, allows you to publish Power BI reports to an on-premises report server, after creating them in Power BI Desktop.
+Power BI Desktop supports Vantage as a 3rd party data source not as a ‘native’ data source. Instead, published reports on Power BI service will need to use the on-premises data gateway component to access Vantage.
+This getting started guide will show you how to connect to a Teradata Vantage. Power BI Desktop Teradata connector uses the .NET Data Provider for Teradata. You need to install the driver on computers that use the Power BI Desktop. The .NET Data Provider for Teradata single installation supports both 32-bit or 64-bit Power BI Desktop application.
+You are expected to be familiar with Azure services, Teradata Vantage, and Power BI Desktop.
+You will need the following accounts and system.
+The Power BI Desktop is a free application for Windows. (Power BI Desktop is not available for Macs. You could run it in a virtual machine, such as Parallels or VMware Fusion, or in Apple’s Boot Camp, but that is beyond the scope of this article.)
+A Teradata Vantage instance with a user and password. The user must have permission to data that can be used by Power BI Desktop. Vantage must be accessible from Power BI Desktop.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
You can install Power BI Desktop from the Microsoft Store or download the installer and run it directly.
+Download and install the latest version of the .NET Data Provider for Teradata.
+Note that there are multiple files available for download. You want the file that starts with “tdnetdp”.
+Run Power BI Desktop, which has a yellow icon.
+
+If the opening (splash) screen is showing, click on Get data.
+
+Otherwise, if you are in the main form of Power BI, ensure that you are on the Home ribbon and click on Get data. Click on More….
+
+Click on Database on the left.
+Scroll the list on the right until you see Teradata database. Click on Teradata database, and then click on the Connect button.
+(“Teradata database” and “Teradata Vantage” are synonymous in this article.)
+
+In the window that appears, enter the name or IP address of your Vantage system into the text box. You can choose to Import data directly into Power BI data model, or connect directly to the data source using DirectQuery and click OK.
+
+(Click Advanced options to submit hand-crafted SQL statement.)
+For credentials, you have the option of connecting with your Windows login or Database username defined in Vantage, which is more common. Select the appropriate authentication method and enter in your username and password. Click Connect.
+You also have the option of authenticating with an LDAP server. This option is hidden by default.
+If you set the environment variable, PBI_EnableTeradataLdap, to true, then the LDAP authentication method will become available.
+
+Do note that LDAP is not supported with the on-premises data gateway, which is used for reports that are published to the Power BI service. If you need LDAP authentication and are using the on-premises data gateway, you will need to submit an incident to Microsoft and request support.
+Alternatively, you can configure Kerberos-based SSO from Power BI service to on-premise data sources like Teradata.
+Once you have connected to the Vantage system, Power BI Desktop remembers the credentials for future connections to the system. You can modify these credentials by going to File > Options and settings > Data source settings.
+The Navigator window appears after a successful connection. It displays the data available on the Vantage system. You can select one or more elements to use in Power BI Desktop.
+
+You preview a table by clicking on its name. If you want to load it into Power BI Desktop, ensure that you click the checkbox next to the table name.
+You can Load the selected table, which brings it into Power BI Desktop. You can also Edit the query, which opens a query editor so you can filter and refine the set of data you want to load.
+Edit may be called Transform data, depending upon the version of Power BI Desktop that you have.
+For information on joining tables, see Create and Manage Relationships in Power BI Desktop feature.
+To publish your report, click Publish on Home ribbon in Power BI Desktop.
+
+Power BI Desktop will prompt you to save your report. Choose My workspace and click Select.
+
+Once report has been published, click Got it to close. You may also click the link, which has the report name in the link.
+
+This is an example of a report created in Power BI Desktop.
+
+You can combine data from many sources with Power BI Desktop. Look at the following links for more information.
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Rupal Shah
+Last updated: February 14th, 2022
This article describes the process to share an Azure Blob Storage dataset from one user to another using Azure Data Share service and then query it with Teradata Vantage leveraging Native Object Store (NOS) capability. We will create and use a storage account and data share account for both users.
+This is a diagram of the workflow.
+
Azure Data Share enables organizations to simply and securely share data with multiple customers and partners. Both the data provider and data consumer must have an Azure subscription to share and receive data. Azure Data Share currently offers snapshot-based sharing and in-place sharing. Today, Azure Data Share supported data stores include Azure Blob Storage, Azure Data Lake Storage Gen1 and Gen2, Azure SQL Database, Azure Synapse Analytics and Azure Data Explorer. Once a dataset share has been sent using Azure Data Share, the data consumer is able to receive that data in a data store of their choice like Azure Blob Storage and then use Teradata Vantage to explore and analyze the data.
+For more information see documentation.
+Vantage is the modern cloud platform that unifies data warehouses, data lakes, and analytics into a single connected ecosystem.
+Vantage combines descriptive, predictive, prescriptive analytics, autonomous decision-making, ML functions, and visualization tools into a unified, integrated platform that uncovers real-time business intelligence at scale, no matter where the data resides.
+Vantage enables companies to start small and elastically scale compute or storage, paying only for what they use, harnessing low-cost object stores and integrating their analytic workloads.
+Vantage supports R, Python, Teradata Studio, and any other SQL-based tools. You can deploy Vantage across public clouds, on-premises, on optimized or commodity infrastructure, or as-a-service.
+Teradata Vantage Native Object Store (NOS) can be used to explore data in external object stores, like Azure Blob Storage, using standard SQL. No special object storage-side compute infrastructure is required to use NOS. You can explore data located in an Blob Storage container by simply creating a NOS table definition that points to your container. With NOS, you can quickly import data from Blob Storage or even join it other tables in the database.
+Alternatively, the Teradata Parallel Transporter (TPT) utility can be used to import data from Blob Storage to Teradata Vantage in bulk fashion. Once loaded, data can be efficiently queried within Vantage.
+For more information see documentation.
+Access to a Teradata Vantage instance.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
An Azure account. You can start with a free account.
+An Azure Blob Storage account to store the dataset
+Once you have met the prerequisites, follow these steps:
+Create a Azure Blob Storage account and container
+Create a Data Share Account
+Create a share
+Accept and receive data using Data Share
+Configure NOS access to Blob Storage
+Query the dataset in Blob Storage
+Load data from Blob Storage into Vantage (optional)
+Open the Azure portal in a browser (Chrome, Firefox, and Safari work well) and follow the steps in create a storage account in a resource group called myProviderStorage_rg in this article.
+Enter a storage name and connectivity method. We will use myproviderstorage and public endpoint in this article.
+| + + | ++We suggest that you use the same location for all services you create. + | +
Select Review + create, then Create.
+Go to resource and click Containers to create container.
+Click the + Container button.
+Enter a container name. We will use providerdata in this article.
+
Click Create.
+We will create a Data Share account for the provider sharing the dataset.
+Follow the Create an Azure Data Share Account steps to create resource in a resource group called myDataShareProvider_rg in this article.
+In Basics tab, enter a data share account name. We will use mydatashareprovider in this article.
+
| + + | ++We suggest that you use the same location for all services you create. + | +
Select Review + create, then Create.
+When the deployment is complete, select Go to resource.
+Navigate to your Data Share Overview page and follow the steps in Create a share.
+Select Start sharing your data.
+Select + Create.
+In Details tab, enter a share name and share type. We will use WeatherData and Snapshot in this article.
+
| + + | +
+ Snapshot share
+
+
+Choose snapshot sharing to provide copy of the data to the recipient. +
+
+Supported data store: Azure Blob Storage, Azure Data Lake Storage Gen1, Azure Data Lake Storage Gen2, Azure SQL Database, Azure Synapse Analytics (formerly SQL DW) + |
+
| + + | +
+ In-place share
+
+
+Choose in-place sharing to provide access to data at its source. +
+
+Supported data store: Azure Data Explorer + |
+
Click Continue.
+In Datasets tab, click Add datasets
+Select Azure Blob Storage
+
Click Next.
+Enter Storage account providing the dataset. We will use myproviderstorage in this article.
+
Click Next.
+Double-click container to choose the dataset. We will use providerdata and onpoint_history_postal-code_hour.csv file in this article.
+
Figure 6 Select Storage container and dataset
+| + + | ++Azure Data Share can share at the folder and file level. Use Azure Blob Storage resource to upload a file. + | +
Click Next.
+Enter a Dataset name that the consumer will see for the folder and dataset. We will use the default names but delete the providerdata folder this article. Click Add datasets.
+
Click Add datasets.
+
Click Continue.
+In Recipients tab, click Add recipient email address to send share notification.
+Enter email address for consumer.
+
| + + | ++Set Share expiration for amount of time share is valid for consumer to accept. + | +
Click Continue.
+In Settings tab, set Snapshot schedule. We use default unchecked this article.
+
Click Continue.
+In Review + Create tab, click Create.
+
Your Azure Data Share has now been created and the recipient of your Data Share is now ready to accept your invitation.
+
In this article, the recipient/consumer is going to receive the data into their Azure Blob storage account.
+Similar to the Data Share Provider, ensure that all pre-requisites are complete for the Consumer before accepting a data share invitation.
+Azure Subscription: If you don’t have one, create a https://azure.microsoft.com/free/[free account] before you begin.
+Azure Blob Storage account and container: create resource group called myConsumerStorage_rg and create account name myconsumerstorage and container consumerdata.
+Azure Data Share account: create resource group called myDataShareConsumer_rg and create a data share account name called mydatashareconsumer to accept the data.
+Follow the steps in Accept and receive data using Azure Data Share.
+In your email, an invitation from Microsoft Azure with a subject titled "Azure Data Share invitation from yourdataprovider@domain.com. Click on the View invitation to see your invitation in Azure.
+
This action opens your browser to the list of Data Share invitations.
+
Select the share you would like to view. We will select WeatherData in this article.
+Under Target Data Share Account, select the Subscription and Resource Group that you would like to deployed your Data Share into or you can create a new Data Share here.
+| + + | ++f provider required a Terms of Use acceptance, a dialog box would appear and you’ll be required to check the box to indicate you agree to the terms of use. + | +
Enter the Resource group and Data share account. We will use myDataShareConsumer_rg and mydatashareconsumer account this article.
+
Select Accept and configure and a share subscription will be created.
+Select Datasets tab. Check the box next to the dataset you’d like to assign a destination to. Select + Map to target to choose a target data store.
+
Select a target data store type and path that you’d like the data to land in. We will use consumers Azure Blob Storage account myconsumerstorage and container consumerdata for our snapshot example in this article.
+| + + | ++Azure Data Share provides open and flexible data sharing, including the ability to share from and to different data stores. Check supported data sources that can accept snapshot and in place sharing. + | +
Click on Map to target.
+Once mapping is complete, for snapshot-based sharing click on Details tab and click Trigger snapshot for Full or Incremental. We will select full copy since this is your first time receiving data from your provider.
+
When the last run status is successful, go to target data store to view the received data. Select Datasets, and click on the link in the Target Path.
+
Native Object Store (NOS) can directly read data in Azure Blob Storage, which allows you to explore and analyze data in Blob Storage without explicitly loading the data.
+A foreign table definition allows data in Blob Storage to be easily referenced within the Advanced SQL Engine and makes the data available in a structured, relational format.
+| + + | ++NOS supports data in CSV, JSON, and Parquet formats. + | +
Login to your Vantage system with Teradata Studio.
+Create an AUTHORIZATION object to access your Blob Storage container with the following SQL command.
+CREATE AUTHORIZATION DefAuth_AZ
+AS DEFINER TRUSTED
+USER 'myconsumerstorage' /* Storage Account Name */
+PASSWORD '*****************' /* Storage Account Access Key or SAS Token */Replace the string for USER with your Storage Account Name.
+Replace the string for PASSWORD with your Storage Account Access Key or SAS Token.
+Create a foreign table definition for the CSV file on Blob Storage with the following SQL command.
+CREATE MULTISET FOREIGN TABLE WeatherData,
+EXTERNAL SECURITY DEFINER TRUSTED DefAuth_AZ (
+ Location VARCHAR(2048) CHARACTER SET UNICODE CASESPECIFIC,
+ Payload DATASET INLINE LENGTH 64000 STORAGE FORMAT CSV
+)
+USING (
+ LOCATION ('/AZ/myconsumerstorage.blob.core.windows.net/consumerdata/')
+)| + + | ++At a minimum, the foreign table definition must include a table name (WeatherData) and a location clause, which points to the object store data. + | +
The LOCATION requires a storage account name and container name. You will need to replace this with your own storage account and container name.
+If the object doesn’t have a standard extension (e.g. “.json”, “.csv”, “.parquet”), then the Location…Payload columns definition phrase is also needed, and the LOCATION phase need to include the file name. For example: LOCATION (AZ/<storage account name>.blob.core.windows.net/<container>/<filename>).
Foreign tables are always defined as No Primary Index (NoPI) tables.
+Run the following SQL command to query the dataset.
+SELECT * FROM WeatherData SAMPLE 10;The foreign table only contains two columns: Location and Payload. Location is the address in the object store system. The data itself is represented in the payload column, with the payload value within each record in the foreign table representing a single CSV row.
+
Run the following SQL command to focus on the data in the object.
+SELECT payload..* FROM WeatherData SAMPLE 10;
Views can simplify the names associated with the payload attributes, can make it easier to code SQL against the object data, and can hide the Location references in the foreign table.
+| + + | +
+Vantage foreign tables use the .. (double dot or double period) operator to separate the object name from the column name.
+ |
+
Run the following SQL command to create a view.
+REPLACE VIEW WeatherData_view AS (
+ SELECT
+ CAST(payload..postal_code AS VARCHAR(10)) Postal_code,
+ CAST(payload..country AS CHAR(2)) Country,
+ CAST(payload..time_valid_utc AS TIMESTAMP(0) FORMAT 'YYYY-MM-DDBHH:MI:SS') Time_Valid_UTC,
+ CAST(payload..doy_utc AS INTEGER) DOY_UTC,
+ CAST(payload..hour_utc AS INTEGER) Hour_UTC,
+ CAST(payload..time_valid_lcl AS TIMESTAMP(0) FORMAT 'YYYY-MM-DDBHH:MI:SS') Time_Valid_LCL,
+ CAST(payload..dst_offset_minutes AS INTEGER) DST_Offset_Minutes,
+ CAST(payload..temperature_air_2m_f AS DECIMAL(4,1)) Temperature_Air_2M_F,
+ CAST(payload..temperature_wetbulb_2m_f AS DECIMAL(3,1)) Temperature_Wetbulb_2M_F,
+ CAST(payload..temperature_dewpoint_2m_f AS DECIMAL(3,1)) Temperature_Dewpoint_2M_F,
+ CAST(payload..temperature_feelslike_2m_f AS DECIMAL(4,1)) Temperature_Feelslike_2M_F,
+ CAST(payload..temperature_windchill_2m_f AS DECIMAL(4,1)) Temperature_Windchill_2M_F,
+ CAST(payload..temperature_heatindex_2m_f AS DECIMAL(4,1)) Temperature_Heatindex_2M_F,
+ CAST(payload..humidity_relative_2m_pct AS DECIMAL(3,1)) Humidity_Relative_2M_Pct,
+ CAST(payload..humidity_specific_2m_gpkg AS DECIMAL(3,1)) Humdity_Specific_2M_GPKG,
+ CAST(payload..pressure_2m_mb AS DECIMAL(5,1)) Pressure_2M_Mb,
+ CAST(payload..pressure_tendency_2m_mb AS DECIMAL(2,1)) Pressure_Tendency_2M_Mb,
+ CAST(payload..pressure_mean_sea_level_mb AS DECIMAL(5,1)) Pressure_Mean_Sea_Level_Mb,
+ CAST(payload..wind_speed_10m_mph AS DECIMAL(3,1)) Wind_Speed_10M_MPH,
+ CAST(payload..wind_direction_10m_deg AS DECIMAL(4,1)) Wind_Direction_10M_Deg,
+ CAST(payload..wind_speed_80m_mph AS DECIMAL(3,1)) Wind_Speed_80M_MPH,
+ CAST(payload..wind_direction_80m_deg AS DECIMAL(4,1)) Wind_Direction_80M_Deg,
+ CAST(payload..wind_speed_100m_mph AS DECIMAL(3,1)) Wind_Speed_100M_MPH,
+ CAST(payload..wind_direction_100m_deg AS DECIMAL(4,1)) Wind_Direction_100M_Deg,
+ CAST(payload..precipitation_in AS DECIMAL(3,2)) Precipitation_in,
+ CAST(payload..snowfall_in AS DECIMAL(3,2)) Snowfall_in,
+ CAST(payload..cloud_cover_pct AS INTEGER) Cloud_Cover_Pct,
+ CAST(payload..radiation_solar_total_wpm2 AS DECIMAL(5,1)) Radiation_Solar_Total_WPM2
+ FROM WeatherData
+)Run the following SQL command to validate the view.
+SELECT * FROM WeatherData_view SAMPLE 10;
Now that you have created a view, you can easily reference the object store data in a query and combine it with other tables, both relational tables in Vantage as well as foreign tables in an object store. This allows you to leverage the full analytic capabilities of Vantage on 100% of the data, no matter where the data is located.
+Having a persistent copy of the Blob Storage data can be useful when repetitive access of the same data is expected. NOS does not automatically make a persistent copy of the Blob Storage data. Each time you reference a foreign table, Vantage will fetch the data from Blob Storage. (Some data may be cached, but this depends on the size of the data in Blob Storage and other active workloads in Vantage.)
+In addition, you may be charged network fees for data transferred from Blob Storage. If you will be referencing the data in Blob Storage multiple times, you may reduce your cost by loading it into Vantage, even temporarily.
+You can select among the approaches below to load the data into Vantage.
+You can use a single statement to both create the table and load the data. You can choose the desired attributes from the foreign table payload and what they will be called in the relational table.
+A CREATE TABLE AS … WITH DATA statement can be used with the foreign table definition as the source table.
+Run the following SQL command to create the relational table and load the data.
+CREATE MULTISET TABLE WeatherData_temp AS (
+ SELECT
+ CAST(payload..postal_code AS VARCHAR(10)) Postal_code,
+ CAST(payload..country AS CHAR(2)) Country,
+ CAST(payload..time_valid_utc AS TIMESTAMP(0) FORMAT 'YYYY-MM-DDBHH:MI:SS') Time_Valid_UTC,
+ CAST(payload..doy_utc AS INTEGER) DOY_UTC,
+ CAST(payload..hour_utc AS INTEGER) Hour_UTC,
+ CAST(payload..time_valid_lcl AS TIMESTAMP(0) FORMAT 'YYYY-MM-DDBHH:MI:SS') Time_Valid_LCL,
+ CAST(payload..dst_offset_minutes AS INTEGER) DST_Offset_Minutes,
+ CAST(payload..temperature_air_2m_f AS DECIMAL(4,1)) Temperature_Air_2M_F,
+ CAST(payload..temperature_wetbulb_2m_f AS DECIMAL(3,1)) Temperature_Wetbulb_2M_F,
+ CAST(payload..temperature_dewpoint_2m_f AS DECIMAL(3,1)) Temperature_Dewpoint_2M_F,
+ CAST(payload..temperature_feelslike_2m_f AS DECIMAL(4,1)) Temperature_Feelslike_2M_F,
+ CAST(payload..temperature_windchill_2m_f AS DECIMAL(4,1)) Temperature_Windchill_2M_F,
+ CAST(payload..temperature_heatindex_2m_f AS DECIMAL(4,1)) Temperature_Heatindex_2M_F,
+ CAST(payload..humidity_relative_2m_pct AS DECIMAL(3,1)) Humidity_Relative_2M_Pct,
+ CAST(payload..humidity_specific_2m_gpkg AS DECIMAL(3,1)) Humdity_Specific_2M_GPKG,
+ CAST(payload..pressure_2m_mb AS DECIMAL(5,1)) Pressure_2M_Mb,
+ CAST(payload..pressure_tendency_2m_mb AS DECIMAL(2,1)) Pressure_Tendency_2M_Mb,
+ CAST(payload..pressure_mean_sea_level_mb AS DECIMAL(5,1)) Pressure_Mean_Sea_Level_Mb,
+ CAST(payload..wind_speed_10m_mph AS DECIMAL(3,1)) Wind_Speed_10M_MPH,
+ CAST(payload..wind_direction_10m_deg AS DECIMAL(4,1)) Wind_Direction_10M_Deg,
+ CAST(payload..wind_speed_80m_mph AS DECIMAL(3,1)) Wind_Speed_80M_MPH,
+ CAST(payload..wind_direction_80m_deg AS DECIMAL(4,1)) Wind_Direction_80M_Deg,
+ CAST(payload..wind_speed_100m_mph AS DECIMAL(3,1)) Wind_Speed_100M_MPH,
+ CAST(payload..wind_direction_100m_deg AS DECIMAL(4,1)) Wind_Direction_100M_Deg,
+ CAST(payload..precipitation_in AS DECIMAL(3,2)) Precipitation_in,
+ CAST(payload..snowfall_in AS DECIMAL(3,2)) Snowfall_in,
+ CAST(payload..cloud_cover_pct AS INTEGER) Cloud_Cover_Pct,
+ CAST(payload..radiation_solar_total_wpm2 AS DECIMAL(5,1)) Radiation_Solar_Total_WPM2
+ FROM
+ WeatherData
+ WHERE
+ Postal_Code = '36101'
+)
+WITH DATA
+NO PRIMARY INDEXRun the following SQL command to validate the contents of the table.
+SELECT * FROM WeatherData_temp SAMPLE 10;
You can also use multiple statements to first create the relational table and then load the data. An advantage of this choice is that you can perform multiple loads, possibly selecting different data or loading in smaller increments if the object is very large.
+Run the following SQL command to create the relational table.
+CREATE MULTISET TABLE WeatherData_temp (
+ Postal_code VARCHAR(10),
+ Country CHAR(2),
+ Time_Valid_UTC TIMESTAMP(0) FORMAT 'YYYY-MM-DDBHH:MI:SS',
+ DOY_UTC INTEGER,
+ Hour_UTC INTEGER,
+ Time_Valid_LCL TIMESTAMP(0) FORMAT 'YYYY-MM-DDBHH:MI:SS',
+ DST_Offset_Minutes INTEGER,
+ Temperature_Air_2M_F DECIMAL(4,1),
+ Temperature_Wetbulb_2M_F DECIMAL(3,1),
+ Temperature_Dewpoint_2M_F DECIMAL(3,1),
+ Temperature_Feelslike_2M_F DECIMAL(4,1),
+ Temperature_Windchill_2M_F DECIMAL(4,1),
+ Temperature_Heatindex_2M_F DECIMAL(4,1),
+ Humidity_Relative_2M_Pct DECIMAL(3,1),
+ Humdity_Specific_2M_GPKG DECIMAL(3,1),
+ Pressure_2M_Mb DECIMAL(5,1),
+ Pressure_Tendency_2M_Mb DECIMAL(2,1),
+ Pressure_Mean_Sea_Level_Mb DECIMAL(5,1),
+ Wind_Speed_10M_MPH DECIMAL(3,1),
+ Wind_Direction_10M_Deg DECIMAL(4,1),
+ Wind_Speed_80M_MPH DECIMAL(3,1),
+ Wind_Direction_80M_Deg DECIMAL(4,1),
+ Wind_Speed_100M_MPH DECIMAL(3,1),
+ Wind_Direction_100M_Deg DECIMAL(4,1),
+ Precipitation_in DECIMAL(3,2),
+ Snowfall_in DECIMAL(3,2),
+ Cloud_Cover_Pct INTEGER,
+ Radiation_Solar_Total_WPM2 DECIMAL(5,1)
+)
+UNIQUE PRIMARY INDEX ( Postal_Code, Time_Valid_UTC )Run the following SQL to load the data into the table.
+INSERT INTO WeatherData_temp
+ SELECT
+ CAST(payload..postal_code AS VARCHAR(10)) Postal_code,
+ CAST(payload..country AS CHAR(2)) Country,
+ CAST(payload..time_valid_utc AS TIMESTAMP(0) FORMAT 'YYYY-MM-DDBHH:MI:SS') Time_Valid_UTC,
+ CAST(payload..doy_utc AS INTEGER) DOY_UTC,
+ CAST(payload..hour_utc AS INTEGER) Hour_UTC,
+ CAST(payload..time_valid_lcl AS TIMESTAMP(0) FORMAT 'YYYY-MM-DDBHH:MI:SS') Time_Valid_LCL,
+ CAST(payload..dst_offset_minutes AS INTEGER) DST_Offset_Minutes,
+ CAST(payload..temperature_air_2m_f AS DECIMAL (4,1)) Temperature_Air_2M_F,
+ CAST(payload..temperature_wetbulb_2m_f AS DECIMAL(3,1)) Temperature_Wetbulb_2M_F,
+ CAST(payload..temperature_dewpoint_2m_f AS DECIMAL(3,1)) Temperature_Dewpoint_2M_F,
+ CAST(payload..temperature_feelslike_2m_f AS DECIMAL(4,1)) Temperature_Feelslike_2M_F,
+ CAST(payload..temperature_windchill_2m_f AS DECIMAL(4,1)) Temperature_Windchill_2M_F,
+ CAST(payload..temperature_heatindex_2m_f AS DECIMAL(4,1)) Temperature_Heatindex_2M_F,
+ CAST(payload..humidity_relative_2m_pct AS DECIMAL(3,1)) Humidity_Relative_2M_Pct,
+ CAST(payload..humidity_specific_2m_gpkg AS DECIMAL(3,1)) Humdity_Specific_2M_GPKG,
+ CAST(payload..pressure_2m_mb AS DECIMAL(5,1)) Pressure_2M_Mb,
+ CAST(payload..pressure_tendency_2m_mb AS DECIMAL(2,1)) Pressure_Tendency_2M_Mb,
+ CAST(payload..pressure_mean_sea_level_mb AS DECIMAL(5,1)) Pressure_Mean_Sea_Level_Mb,
+ CAST(payload..wind_speed_10m_mph AS DECIMAL(3,1)) Wind_Speed_10M_MPH,
+ CAST(payload..wind_direction_10m_deg AS DECIMAL(4,1)) Wind_Direction_10M_Deg,
+ CAST(payload..wind_speed_80m_mph AS DECIMAL(3,1)) Wind_Speed_80M_MPH,
+ CAST(payload..wind_direction_80m_deg AS DECIMAL(4,1)) Wind_Direction_80M_Deg,
+ CAST(payload..wind_speed_100m_mph AS DECIMAL(3,1)) Wind_Speed_100M_MPH,
+ CAST(payload..wind_direction_100m_deg AS DECIMAL(4,1)) Wind_Direction_100M_Deg,
+ CAST(payload..precipitation_in AS DECIMAL(3,2)) Precipitation_in,
+ CAST(payload..snowfall_in AS DECIMAL(3,2)) Snowfall_in,
+ CAST(payload..cloud_cover_pct AS INTEGER) Cloud_Cover_Pct,
+ CAST(payload..radiation_solar_total_wpm2 AS DECIMAL(5,1)) Radiation_Solar_Total_WPM2
+ FROM
+ WeatherData
+ WHERE
+ Postal_Code = '30301'Run the following SQL command to validate the contents of the table.
+SELECT * FROM WeatherData_temp SAMPLE 10;
An alternative to defining a foreign table is to use the READ_NOS table operator. This table operator allows you to access data directly from an object store without first creating a foreign table, or to view a list of the keys associated with all the objects specified by a Location clause.
You can use the READ_NOS table operator to explore the data in an object.
Run the following command to explore the data in an object.
+ +The LOCATION requires a storage account name and container name. This is highlighted above in yellow. You will need to replace this with your own storage account and container name.
+Replace the string for ACCESS_ID with your Storage Account Name.
+Replace the string for ACCES_KEY with your Storage Account Access Key or SAS Token
+
You can also leverage the READ_NOS table operator to get the length (size) of the object.
+Run the following SQL command to view the size of the object.
+SELECT
+ location(CHAR(120)), ObjectLength
+FROM
+ READ_NOS (
+ ON (SELECT CAST( NULL AS DATASET STORAGE FORMAT CSV))
+ USING
+ LOCATION ('/AZ/myconsumerstorage.blob.core.windows.net/consumerdata')
+ ACCESS_ID('myconsumerstorage')
+ ACCESS_KEY('*****')
+ RETURNTYPE('NOSREAD_KEYS')
+ ) AS THE_TABLE
+ORDER BY 1Replace the values for LOCATION, ACCESS_ID, and ACCESS_KEY.
+
You can substitute the NOS_READ table operator for a foreign table definition in the above section for loading the data into a relational table.
+CREATE MULTISET TABLE WeatherData_temp AS (
+ SELECT
+ CAST(payload..postal_code AS VARCHAR(10)) Postal_code,
+ CAST(payload..country AS CHAR(2)) Country,
+ CAST(payload..time_valid_utc AS TIMESTAMP(0) FORMAT 'YYYY-MM-DDBHH:MI:SS') Time_Valid_UTC,
+ CAST(payload..doy_utc AS INTEGER) DOY_UTC,
+ CAST(payload..hour_utc AS INTEGER) Hour_UTC,
+ CAST(payload..time_valid_lcl AS TIMESTAMP(0) FORMAT 'YYYY-MM-DDBHH:MI:SS') Time_Valid_LCL,
+ CAST(payload..dst_offset_minutes AS INTEGER) DST_Offset_Minutes,
+ CAST(payload..temperature_air_2m_f AS DECIMAL (4,1)) Temperature_Air_2M_F,
+ CAST(payload..temperature_wetbulb_2m_f AS DECIMAL(3,1)) Temperature_Wetbulb_2M_F,
+ CAST(payload..temperature_dewpoint_2m_f AS DECIMAL(3,1)) Temperature_Dewpoint_2M_F,
+ CAST(payload..temperature_feelslike_2m_f AS DECIMAL(4,1)) Temperature_Feelslike_2M_F,
+ CAST(payload..temperature_windchill_2m_f AS DECIMAL(4,1)) Temperature_Windchill_2M_F,
+ CAST(payload..temperature_heatindex_2m_f AS DECIMAL(4,1)) Temperature_Heatindex_2M_F,
+ CAST(payload..humidity_relative_2m_pct AS DECIMAL(3,1)) Humidity_Relative_2M_Pct,
+ CAST(payload..humidity_specific_2m_gpkg AS DECIMAL(3,1)) Humdity_Specific_2M_GPKG,
+ CAST(payload..pressure_2m_mb AS DECIMAL(5,1)) Pressure_2M_Mb,
+ CAST(payload..pressure_tendency_2m_mb AS DECIMAL(2,1)) Pressure_Tendency_2M_Mb,
+ CAST(payload..pressure_mean_sea_level_mb AS DECIMAL(5,1)) Pressure_Mean_Sea_Level_Mb,
+ CAST(payload..wind_speed_10m_mph AS DECIMAL(3,1)) Wind_Speed_10M_MPH,
+ CAST(payload..wind_direction_10m_deg AS DECIMAL(4,1)) Wind_Direction_10M_Deg,
+ CAST(payload..wind_speed_80m_mph AS DECIMAL(3,1)) Wind_Speed_80M_MPH,
+ CAST(payload..wind_direction_80m_deg AS DECIMAL(4,1)) Wind_Direction_80M_Deg,
+ CAST(payload..wind_speed_100m_mph AS DECIMAL(3,1)) Wind_Speed_100M_MPH,
+ CAST(payload..wind_direction_100m_deg AS DECIMAL(4,1)) Wind_Direction_100M_Deg,
+ CAST(payload..precipitation_in AS DECIMAL(3,2)) Precipitation_in,
+ CAST(payload..snowfall_in AS DECIMAL(3,2)) Snowfall_in,
+ CAST(payload..cloud_cover_pct AS INTEGER) Cloud_Cover_Pct,
+ CAST(payload..radiation_solar_total_wpm2 AS DECIMAL(5,1)) Radiation_Solar_Total_WPM2
+ FROM
+ READ_NOS (
+ ON (SELECT CAST( NULL AS DATASET STORAGE FORMAT CSV))
+ USING
+ LOCATION ('/AZ/myconsumerstorage.blob.core.windows.net/consumerdata')
+ ACCESS_ID('myconsumerstorage')
+ ACCESS_KEY('*****')
+ ) AS THE_TABLE
+ WHERE
+ Postal_Code = '36101'
+)
+WITH DATA| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Hailing Jiang
+Last updated: June 28th, 2022
| + + | ++This how-to shows you how to add Teradata Extensions to a Jupyter Notebooks environment. A hosted version of Jupyter Notebooks integrated with Teradata Extensions and analytics tools is available for functional testing for free at https://clearscape.teradata.com. + | +
Teradata Jupyter extensions provide Teradata SQL kernel and several UI extensions to allow users to easily access and navigate Teradata database from Jupyter envioronment. Google Vertex AI is Google Cloud’s new unified ML platform. Vertex AI Workbench provides a Jupyter-base development environment for the entire data science workflow. This article describes how to integate our Jupyter extensions with Vertex AI Workbench so that Vertex AI users can take advantage of our Teradata extensions in their ML pipeline.
+Vertex AI workbench supports two types of notebooks: managed notebooks and user-managed notebooks. Here we will focus on user-managed notebooks. We will show two ways to integrate our Jupyter extensions with user-managed notebooks: use startup script to install our kernel and extensions or use custom container.
+Access to a Teradata Vantage instance
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
Google Cloud account with Vertex AI enabled
+Google cloud storage to store startup scripts and Teradata Jupyter extension package
+There are two ways to run Teradata Jupyter Extensions in Vertex AI:
+These two integration methods are described below.
+When we create a new notebook instance, we can specify a startup script. This script runs only once after the instance is created. Here are the steps:
+Download Teradata Jupyter extensions package
+Go to Vantage Modules for Jupyter page to download the Teradata Jupyter extensions package bundle Linux version.
+Upload the package to a Google Cloud storage bucket
+Write a startup script and upload it to cloud storage bucket
+Below is a sample script. It fetches Teradata Jupyter extension package from cloud storage bucket and installs Teradata SQL kernel and extensions.
+#! /bin/bash
+
+cd /home/jupyter
+mkdir teradata
+cd teradata
+gsutil cp gs://teradata-jupyter/* .
+unzip teradatasql*.zip
+
+# Install Teradata kernel
+cp teradatakernel /usr/local/bin
+
+jupyter kernelspec install ./teradatasql --prefix=/opt/conda
+
+# Install Teradata extensions
+pip install --find-links . teradata_preferences_prebuilt
+pip install --find-links . teradata_connection_manager_prebuilt
+pip install --find-links . teradata_sqlhighlighter_prebuilt
+pip install --find-links . teradata_resultset_renderer_prebuilt
+pip install --find-links . teradata_database_explorer_prebuilt
+
+# PIP install the Teradata Python library
+pip install teradataml
+
+# Install Teradata R library (optional, uncomment this line only if you use an environment that supports R)
+#Rscript -e "install.packages('tdplyr',repos=c('https://r-repo.teradata.com','https://cloud.r-project.org'))"Create a new notebook and add the startup script from cloud storage bucket
+
It may take a few minutes for the notebook creation process to complete. When it is done, click on Open notebook.
Another option is to provide a custom container when creating a notebook.
+Download Teradata Jupyter extensions package
+Go to Vantage Modules for Jupyter page to download the Teradata Jupyter extensions package bundle Linux version.
+Copy this package to your work directory and unzip it
+Build custom Docker image
+The custom container must expose a service on port 8080. It is recommended to create a container derived from a Google Deep Learning Containers image, because those images are already configured to be compatible with user-managed notebooks.
+Below is a sample Dockerfile you can use to build a Docker image with Teradata SQL kernel and extensions installed:
+# Use one of the deep learning images as base image
+# if you need both Python and R, use one of the R images
+FROM gcr.io/deeplearning-platform-release/r-cpu:latest
+
+USER root
+
+##############################################################
+# Install kernel and copy supporting files
+##############################################################
+
+# Copy the kernel
+COPY ./teradatakernel /usr/local/bin
+
+RUN chmod 755 /usr/local/bin/teradatakernel
+
+# Copy directory with kernel.json file into image
+COPY ./teradatasql teradatasql/
+
+# Copy notebooks and licenses
+COPY ./notebooks/ /home/jupyter
+COPY ./license.txt /home/jupyter
+COPY ./ThirdPartyLicenses/ /home/jupyter
+
+# Install the kernel file to /opt/conda jupyter lab instance
+RUN jupyter kernelspec install ./teradatasql --prefix=/opt/conda
+
+##############################################################
+# Install Teradata extensions
+##############################################################
+
+RUN pip install --find-links . teradata_preferences_prebuilt && \
+ pip install --find-links . teradata_connection_manager_prebuilt && \
+ pip install --find-links . teradata_sqlhighlighter_prebuilt && \
+ pip install --find-links . teradata_resultset_renderer_prebuilt && \
+ pip install --find-links . teradata_database_explorer_prebuilt
+
+# Give back ownership of /opt/conda to jovyan
+RUN chown -R jupyter:users /opt/conda
+
+# PIP install the Teradata Python libraries
+RUN pip install teradataml
+
+# Install Teradata R library (optional, include it only if you use a base image that supports R)
+RUN Rscript -e "install.packages('tdplyr',repos=c('https://r-repo.teradata.com','https://cloud.r-project.org'))"In your work directory (where you unzipped Teradata Jupyter extensions package), run docker build to build the image:
docker build -f Dockerfile imagename:imagetag .Push the docker image to Google container registry or artifact registry
+Please refer to the following documentations to push docker image to registry:
+Create a new notebook
+In Environment section, set custom container field to the location of your newly created custom container:
| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Hailing Jiang
+Last updated: September 27th, 2022
| + + | ++This how-to shows you how to add Teradata Extensions to a Jupyter Notebooks environment. A hosted version of Jupyter Notebooks integrated with Teradata Extensions and analytics tools is available for functional testing for free at https://clearscape.teradata.com. + | +
Teradata Jupyter extensions provide Teradata SQL kernel and several UI extensions to allow users to easily access and navigate Teradata database from Jupyter envioronment. This article describes how to integate our Jupyter extensions with SageMaker notebook instance.
+Access to a Teradata Vantage instance
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
AWS account
+AWS S3 bucket to store lifecycle configuration scripts and Teradata Jupyter extension package
+SageMaker supports customization of notebook instances using lifecycle configuration scripts. Below we will demo how to use lifecycle configuration scripts to install our Jupyter kernel and extensions in a notebook instance.
+Download Teradata Jupyter extensions package
+Download Linux version from https://downloads.teradata.com/download/tools/vantage-modules-for-jupyter and upload it to an S3 bucket. This zipped package contains Teradata Jupyter kernel and extensions. Each extension has 2 files, the one with "_prebuilt" in the name is prebuilt extension which can be installed using PIP, the other one is source extension that needs to be installed using "jupyter labextension". It is recommended to use prebuilt extensions.
+Create a lifecycle configuration for notebook instance
+
Here are sample scripts that fetches the Teradata package from S3 bucket and installs Jupyter kernel and extensions. Note that on-create.sh creates a custom conda env that persists on notebook instance’s EBS volume so that the installation will not get lost after notebook restarts. on-start.sh installs Teradata kernel and extensions to the custom conda env.
+on-create.sh
+#!/bin/bash
+
+set -e
+
+# This script installs a custom, persistent installation of conda on the Notebook Instance's EBS volume, and ensures
+# that these custom environments are available as kernels in Jupyter.
+
+
+sudo -u ec2-user -i <<'EOF'
+unset SUDO_UID
+# Install a separate conda installation via Miniconda
+WORKING_DIR=/home/ec2-user/SageMaker/custom-miniconda
+mkdir -p "$WORKING_DIR"
+wget https://repo.anaconda.com/miniconda/Miniconda3-4.6.14-Linux-x86_64.sh -O "$WORKING_DIR/miniconda.sh"
+bash "$WORKING_DIR/miniconda.sh" -b -u -p "$WORKING_DIR/miniconda"
+rm -rf "$WORKING_DIR/miniconda.sh"
+# Create a custom conda environment
+source "$WORKING_DIR/miniconda/bin/activate"
+KERNEL_NAME="teradatasql"
+
+PYTHON="3.8"
+conda create --yes --name "$KERNEL_NAME" python="$PYTHON"
+conda activate "$KERNEL_NAME"
+pip install --quiet ipykernel
+
+EOFon-start.sh
+#!/bin/bash
+
+set -e
+
+# This script installs Teradata Jupyter kernel and extensions.
+
+
+sudo -u ec2-user -i <<'EOF'
+unset SUDO_UID
+
+WORKING_DIR=/home/ec2-user/SageMaker/custom-miniconda
+
+source "$WORKING_DIR/miniconda/bin/activate" teradatasql
+
+# fetch Teradata Jupyter extensions package from S3 and unzip it
+mkdir -p "$WORKING_DIR/teradata"
+aws s3 cp s3://sagemaker-teradata-bucket/teradatasqllinux_3.3.0-ec06172022.zip "$WORKING_DIR/teradata"
+cd "$WORKING_DIR/teradata"
+
+unzip -o teradatasqllinux_3.3.0-ec06172022.zip
+
+# install Teradata kernel
+cp teradatakernel /home/ec2-user/anaconda3/condabin
+jupyter kernelspec install --user ./teradatasql
+
+# install Teradata Jupyter extensions
+source /home/ec2-user/anaconda3/bin/activate JupyterSystemEnv
+
+pip install teradata_connection_manager_prebuilt-3.3.0.tar.gz
+pip install teradata_database_explorer_prebuilt-3.3.0.tar.gz
+pip install teradata_preferences_prebuilt-3.3.0.tar.gz
+pip install teradata_resultset_renderer_prebuilt-3.3.0.tar.gz
+pip install teradata_sqlhighlighter_prebuilt-3.3.0.tar.gz
+
+conda deactivate
+EOFCreate a notebook instance. Please select 'Amazon Linux 2, Jupyter Lab3' for Platform identifier and select the lifecycle configuration created in step 2 for Lifecycle configuration.
+
You might also need to add vpc, subnet and security group in 'Network' section to gain access to Teradata databases.
+Wait until notebook instance Status turns 'InService', click 'Open JupyterLab' to open the notebook.
+
Access the demo notebooks to get usage tips
++
+
| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Wenjie Tehan
+Last updated: February 14th, 2022
This how-to describes the process to migrate data between Salesforce and Teradata Vantage. It contains two use cases:
+Retrieve customer information from Salesforce, and combine it with order and shipping information from Vantage to derive analytical insights.
+Update newleads table on Vantage with the Salesforce data, then add the new lead(s) back to Salesforce using AppFlow.
Amazon AppFlow transfers the customer account data from Salesforce to Amazon S3. Vantage then uses Native Object Store (NOS) read functionality to join the data in Amazon S3 with data in Vantage with a single query.
+The account information is used to update the newleads table on Vantage. Once the table is updated, Vantage writes it back to the Amazon S3 bucket with NOS Write. A Lambda function is triggered upon arrival of the new lead data file to convert the data file from Parquet format to CSV format, and AppFlow then inserts the new lead(s) back into Salesforce.
Amazon AppFlow is a fully managed integration service that enables users to securely transfer data between Software-as-a-Service (SaaS) applications like Salesforce, Marketo, Slack, and ServiceNow, and AWS services like Amazon S3 and Amazon Redshift. AppFlow automatically encrypts data in motion, and allows users to restrict data from flowing over the public internet for SaaS applications that are integrated with AWS PrivateLink, reducing exposure to security threats.
+As of today, Amazon AppFlow has 16 sources to choose from, and can send the data to four destinations.
+Teradata Vantage is the connected multi-cloud data platform for enterprise analytics, solving data challenges from start to scale.
+Vantage enables companies to start small and elastically scale compute or storage, paying only for what they use, harnessing low-cost object stores and integrating their analytic workloads. Vantage supports R, Python, Teradata Studio, and any other SQL-based tools.
+Vantage combines descriptive, predictive, prescriptive analytics, autonomous decision-making, ML functions, and visualization tools into a unified, integrated platform that uncovers real-time business intelligence at scale, no matter where the data resides.
+Teradata Vantage Native Object Store (NOS) can be used to explore data in external object stores, like Amazon S3, using standard SQL. No special object storage-side compute infrastructure is required to use NOS. Users can explore data located in an Amazon S3 bucket by simply creating a NOS table definition that points to your bucket. With NOS, you can quickly import data from Amazon S3 or even join it with other tables in the Vantage database.
+You are expected to be familiar with Amazon AppFlow service and Teradata Vantage.
+You will need the following accounts, and systems:
+Access to a Teradata Vantage instance.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
An AWS account with the role that can create and run flows.
+An Amazon S3 bucket to store Salesforce data (i.e., ptctsoutput)
+An Amazon S3 bucket to store raw Vantage data (Parquet file) (i.e., vantageparquet). This bucket needs to have policy to allow Amazon AppFlow access
+An Amazon S3 bucket to store converted Vantage data (CSV file) (i.e., vantagecsv)
+A Salesforce account that satisfies the following requirements:
+Your Salesforce account must be enabled for API access. API access is enabled by default for Enterprise, Unlimited, Developer, and Performance editions.
+Your Salesforce account must allow you to install connected apps. If this is disabled, contact your Salesforce administrator. After you create a Salesforce connection in Amazon AppFlow, verify that the connected app named "Amazon AppFlow Embedded Login App" is installed in your Salesforce account.
+The refresh token policy for the "Amazon AppFlow Embedded Login App" must be set to "Refresh token is valid until revoked". Otherwise, your flows will fail when your refresh token expires.
+You must enable Change Data Capture in Salesforce to use event-driven flow triggers. From Setup, enter "Change Data Capture" in Quick Find.
+If your Salesforce app enforces IP address restrictions, you must whitelist the addresses used by Amazon AppFlow. For more information, see https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html[AWS IP address ranges] in the Amazon Web Services General Reference.
+If you are transferring over 1 million Salesforce records, you cannot choose any Salesforce compound field. Amazon AppFlow uses Salesforce Bulk APIs for the transfer, which does not allow transfer of compound fields.
+To create private connections using AWS PrivateLink, you must enable both "Manager Metadata" and "Manage External Connections" user permissions in your Salesforce account. Private connections are currently available in the us-east-1 and us-west-2 AWS Regions.
+Some Salesforce objects can’t be updated, such as history objects. For these objects, Amazon AppFlow does not support incremental export (the "Transfer new data only" option) for schedule-triggered flows. Instead, you can choose the "Transfer all data" option and then select the appropriate filter to limit the records you transfer.
+Once you have met the prerequisites, follow these steps:
+Create a Salesforce to Amazon S3 Flow
+Exploring Data using NOS
+Export Vantage Data to Amazon S3 using NOS
+Create an Amazon S3 to Salesforce Flow
+This step creates a flow using Amazon AppFlow. For this example, we’re using a Salesforce developer account to connect to Salesforce.
+Go to AppFlow console, sign in with your AWS login credentials and click Create flow. Make sure you are in the right region, and the bucket is created to store Salesforce data.
+
This step provides basic information for your flow.
+Fill in Flow name (i.e. salesforce) and Flow description (optional), leave Customize encryption settings (advanced) unchecked. Click Next.
+This step provides information about the source and destination for your flow. For this example, we will be using Salesforce as the source, and Amazon S3 as the destination.
+For Source name, choose Salesforce, then Create new connection for Choose Salesforce connection.
+
Use default for Salesforce environment and Data encryption. Give your connection a name (i.e. salesforce) and click Continue.
+
At the salesforce login window, enter your Username and Password. Click Log In
+
Click Allow to allow AppFlow to access your salesforce data and information.
+
Back at the AppFlow Configure flow window, use Salesforce objects, and choose Account to be the Salesforce object.
+
Use Amazon S3 as Destination name. Pick the bucket you created earlier where you want the data to be stored (i.e., ptctsoutput).
+
Flow trigger is Run on demand. Click Next.
+
This step determines how data is transferred from the source to the destination.
+Use Manually map fields as Mapping method
+For simplicity, choose Map all fields directly for Source to destination filed mapping.
+
Once you click on "Map all fields directly", all the fields will show under Mapped fields. Click on the checkbox for the field(s) you want to Add formula (concatenates), Modify values (mask or truncate field values), or Remove selected mappings.
+For this example, no checkbox will be ticked.
+For Validations, add in a condition to ignore the record that contains no "Billing Address" (optional). Click Next.
+
You can specify a filter to determine which records to transfer. For this example, add a condition to filter out the records that are deleted (optional). Click Next.
+
Review all the information you just entered. Modify if necessary. Click Create flow.
+A message of successful flow creation will be displayed with the flow information once the flow is created,
+
Click Run flow on the upper right corner.
+Upon completion of the flow run, message will be displayed to indicate a successful run.
+Message example:
+
Click the link to the bucket to view data. Salesforce data will be in JSON format.
+By default, Salesforce data is encrypted. We need to remove the encryption for NOS to access it.
+Click on the data file in your Amazon S3 bucket, then click the Properties tab.
+
Click on the AWS-KMS from Encryption and change it from AWS-KMS encryption to None. Click Save.
+
Native Object Store has built in functionalities to explore and analyze data in Amazon S3. This section lists a few commonly used functions of NOS.
+Foreign table allows the external data to be easily referenced within the Vantage Advanced SQL Engine and makes the data available in a structured relational format.
+To create a foreign table, first login to Teradata Vantage system with your credentials. Create AUTHORIZATION object with access keys for Amazon S3 bucket access. Authorization object enhances security by establishing control over who is allowed to use a foreign table to access Amazon S3 data.
+CREATE AUTHORIZATION DefAuth_S3
+AS DEFINER TRUSTED
+USER 'A*****************' /* AccessKeyId */
+PASSWORD '********'; /* SecretAccessKey */"USER" is the AccessKeyId for your AWS account, and "PASSWORD" is the SecretAccessKey.
+Create a foreign table against the JSON file on Amazon S3 using following command.
+CREATE MULTISET FOREIGN TABLE salesforce,
+EXTERNAL SECURITY DEFINER TRUSTED DefAuth_S3
+(
+ Location VARCHAR(2048) CHARACTER SET UNICODE CASESPECIFIC,
+ Payload JSON(8388096) INLINE LENGTH 32000 CHARACTER SET UNICODE
+)
+USING
+(
+ LOCATION ('/S3/s3.amazonaws.com/ptctstoutput/salesforce/1ce190bc-25a9-4493-99ad-7497b497a0d0/903790813-2020-08-21T21:02:25')
+);At a minimum, the foreign table definition must include a table name and location clause (highlighted in yellow) which points to the object store data. The Location requires a top-level single name, referred to as a "bucket" in Amazon.
+If the file name doesn’t have standard extension (.json, .csv, .parquet) at the end, the Location and Payload columns definition is also required (highlighted in turquoise) to indicate the type of the data file.
+Foreign tables are always defined as No Primary Index (NoPI) tables.
+Once foreign table’s created, you can query the content of the Amazon S3 data set by doing "Select" on the foreign table.
+SELECT * FROM salesforce;
+SELECT payload.* FROM salesforce;The foreign table only contains two columns: Location and Payload. Location is the address in the object store system. The data itself is represented in the payload column, with the payload value within each record in the foreign table representing a single JSON object and all its name-value pairs.
+Sample output from "SELECT * FROM salesforce;".
+
Sample output form "SELECT payload.* FROM salesforce;".
+
JSON data may contain different attributes in different records. To determine the full list of possible attributes in a data store, use JSON_KEYS:
+|SELECT DISTINCT * FROM JSON_KEYS (ON (SELECT payload FROM salesforce)) AS j;Partial Output:
+
Views can simplify the names associated with the payload attributes, make it easier to code executable SQL against object store data, and hide the Location references in the foreign table to make it look like normal columns.
+Following is a sample create view statement with the attributes discovered from the JSON_KEYS table operator above.
+REPLACE VIEW salesforceView AS (
+ SELECT
+ CAST(payload.Id AS VARCHAR(20)) Customer_ID,
+ CAST(payload."Name" AS VARCHAR(100)) Customer_Name,
+ CAST(payload.AccountNumber AS VARCHAR(10)) Acct_Number,
+ CAST(payload.BillingStreet AS VARCHAR(20)) Billing_Street,
+ CAST(payload.BillingCity AS VARCHAR(20)) Billing_City,
+ CAST(payload.BillingState AS VARCHAR(10)) Billing_State,
+ CAST(payload.BillingPostalCode AS VARCHAR(5)) Billing_Post_Code,
+ CAST(payload.BillingCountry AS VARCHAR(20)) Billing_Country,
+ CAST(payload.Phone AS VARCHAR(15)) Phone,
+ CAST(payload.Fax AS VARCHAR(15)) Fax,
+ CAST(payload.ShippingStreet AS VARCHAR(20)) Shipping_Street,
+ CAST(payload.ShippingCity AS VARCHAR(20)) Shipping_City,
+ CAST(payload.ShippingState AS VARCHAR(10)) Shipping_State,
+ CAST(payload.ShippingPostalCode AS VARCHAR(5)) Shipping_Post_Code,
+ CAST(payload.ShippingCountry AS VARCHAR(20)) Shipping_Country,
+ CAST(payload.Industry AS VARCHAR(50)) Industry,
+ CAST(payload.Description AS VARCHAR(200)) Description,
+ CAST(payload.NumberOfEmployees AS VARCHAR(10)) Num_Of_Employee,
+ CAST(payload.CustomerPriority__c AS VARCHAR(10)) Priority,
+ CAST(payload.Rating AS VARCHAR(10)) Rating,
+ CAST(payload.SLA__c AS VARCHAR(10)) SLA,
+ CAST(payload.AnnualRevenue AS VARCHAR(10)) Annual_Revenue,
+ CAST(payload."Type" AS VARCHAR(20)) Customer_Type,
+ CAST(payload.Website AS VARCHAR(100)) Customer_Website,
+ CAST(payload.LastActivityDate AS VARCHAR(50)) Last_Activity_Date
+ FROM salesforce
+);SELECT * FROM salesforceView;Partial output:
+
READ_NOS table operator can be used to sample and explore a percent of the data without having first defined a foreign table, or to view a list of the keys associated with all the objects specified by a Location clause.
+SELECT top 5 payload.*
+FROM READ_NOS (
+ ON (SELECT CAST(NULL AS JSON CHARACTER SET Unicode))
+USING
+LOCATION ('/S3/s3.amazonaws.com/ptctstoutput/salesforce/1ce190bc-25a9-4493-99ad-7497b497a0d0/903790813-2020-08-21T21:02:25')
+ ACCESS_ID ('A**********') /* AccessKeyId */
+ ACCESS_KEY ('***********') /* SecretAccessKey */
+ ) AS D
+GROUP BY 1;Output:
+
Foreign table can be joined with a table(s) in Vantage for further analysis. For example, ordering and shipping information are in Vantage in these three tables – Orders, Order_Items and Shipping_Address.
+DDL for Orders:
+CREATE TABLE Orders (
+ Order_ID INT NOT NULL,
+ Customer_ID VARCHAR(20) CHARACTER SET LATIN CASESPECIFIC,
+ Order_Status INT,
+ -- Order status: 1 = Pending; 2 = Processing; 3 = Rejected; 4 = Completed
+ Order_Date DATE NOT NULL,
+ Required_Date DATE NOT NULL,
+ Shipped_Date DATE,
+ Store_ID INT NOT NULL,
+ Staff_ID INT NOT NULL
+) Primary Index (Order_ID);DDL for Order_Items:
+CREATE TABLE Order_Items(
+ Order_ID INT NOT NULL,
+ Item_ID INT,
+ Product_ID INT NOT NULL,
+ Quantity INT NOT NULL,
+ List_Price DECIMAL (10, 2) NOT NULL,
+ Discount DECIMAL (4, 2) NOT NULL DEFAULT 0
+) Primary Index (Order_ID, Item_ID);DDL for Shipping_Address:
+CREATE TABLE Shipping_Address (
+ Customer_ID VARCHAR(20) CHARACTER SET LATIN CASESPECIFIC NOT NULL,
+ Street VARCHAR(100) CHARACTER SET LATIN CASESPECIFIC,
+ City VARCHAR(20) CHARACTER SET LATIN CASESPECIFIC,
+ State VARCHAR(15) CHARACTER SET LATIN CASESPECIFIC,
+ Postal_Code VARCHAR(10) CHARACTER SET LATIN CASESPECIFIC,
+ Country VARCHAR(20) CHARACTER SET LATIN CASESPECIFIC
+) Primary Index (Customer_ID);And the tables have following data:
+Orders:
+
Order_Items:
+
Shipping_Address:
+
By joining the salesforce foreign table to the established database table Orders, Order_Items and Shipping_Address, we can retrieve customer’s order information with customer’s shipping information.
+SELECT
+ s.payload.Id as Customer_ID,
+ s.payload."Name" as Customer_Name,
+ s.payload.AccountNumber as Acct_Number,
+ o.Order_ID as Order_ID,
+ o.Order_Status as Order_Status,
+ o.Order_Date as Order_Date,
+ oi.Item_ID as Item_ID,
+ oi.Product_ID as Product_ID,
+ sa.Street as Shipping_Street,
+ sa.City as Shipping_City,
+ sa.State as Shipping_State,
+ sa.Postal_Code as Shipping_Postal_Code,
+ sa.Country as Shipping_Country
+FROM
+ salesforce s, Orders o, Order_Items oi, Shipping_Address sa
+WHERE
+ s.payload.Id = o.Customer_ID
+ AND o.Customer_ID = sa.Customer_ID
+ AND o.Order_ID = oi.Order_ID
+ORDER BY 1;Results:
+
Having a persistent copy of the Amazon S3 data can be useful when repetitive access of the same data is expected. NOS foreign table does not automatically make a persistent copy of the Amazon S3 data. A few approaches to capture the data in the database are described below:
+A "CREATE TABLE AS … WITH DATA" statement can be used with the foreign table definition acting as the source table. Use this approach you can selectively choose which attributes within the foreign table payload that you want to include in the target table, and what the relational table columns will be named.
+CREATE TABLE salesforceVantage AS (
+ SELECT
+ CAST(payload.Id AS VARCHAR(20)) Customer_ID,
+ CAST(payload."Name" AS VARCHAR(100)) Customer_Name,
+ CAST(payload.AccountNumber AS VARCHAR(10)) Acct_Number,
+ CAST(payload.BillingStreet AS VARCHAR(20)) Billing_Street,
+ CAST(payload.BillingCity AS VARCHAR(20)) Billing_City,
+ CAST(payload.BillingState AS VARCHAR(10)) Billing_State,
+ CAST(payload.BillingPostalCode AS VARCHAR(5)) Billing_Post_Code,
+ CAST(payload.BillingCountry AS VARCHAR(20)) Billing_Country,
+ CAST(payload.Phone AS VARCHAR(15)) Phone,
+ CAST(payload.Fax AS VARCHAR(15)) Fax,
+ CAST(payload.ShippingStreet AS VARCHAR(20)) Shipping_Street,
+ CAST(payload.ShippingCity AS VARCHAR(20)) Shipping_City,
+ CAST(payload.ShippingState AS VARCHAR(10)) Shipping_State,
+ CAST(payload.ShippingPostalCode AS VARCHAR(5)) Shipping_Post_Code,
+ CAST(payload.ShippingCountry AS VARCHAR(20)) Shipping_Country,
+ CAST(payload.Industry AS VARCHAR(50)) Industry,
+ CAST(payload.Description AS VARCHAR(200)) Description,
+ CAST(payload.NumberOfEmployees AS INT) Num_Of_Employee,
+ CAST(payload.CustomerPriority__c AS VARCHAR(10)) Priority,
+ CAST(payload.Rating AS VARCHAR(10)) Rating,
+ CAST(payload.SLA__c AS VARCHAR(10)) SLA,
+ CAST(payload."Type" AS VARCHAR(20)) Customer_Type,
+ CAST(payload.Website AS VARCHAR(100)) Customer_Website,
+ CAST(payload.AnnualRevenue AS VARCHAR(10)) Annual_Revenue,
+ CAST(payload.LastActivityDate AS DATE) Last_Activity_Date
+ FROM salesforce)
+WITH DATA
+NO PRIMARY INDEX;SELECT* * FROM salesforceVantage; partial results:
An alternative to using foreign table is to use the READ_NOS table operator. This table operator allows you to access data directly from an object store without first building a foreign table. Combining READ_NOS with a CREATE TABLE AS clause to build a persistent version of the data in the database.
+CREATE TABLE salesforceReadNOS AS (
+ SELECT
+ CAST(payload.Id AS VARCHAR(20)) Customer_ID,
+ CAST(payload."Name" AS VARCHAR(100)) Customer_Name,
+ CAST(payload.AccountNumber AS VARCHAR(10)) Acct_Number,
+ CAST(payload.BillingStreet AS VARCHAR(20)) Billing_Street,
+ CAST(payload.BillingCity AS VARCHAR(20)) Billing_City,
+ CAST(payload.BillingState AS VARCHAR(10)) Billing_State,
+ CAST(payload.BillingPostalCode AS VARCHAR(5)) Billing_Post_Code,
+ CAST(payload.BillingCountry AS VARCHAR(20)) Billing_Country,
+ CAST(payload.Phone AS VARCHAR(15)) Phone,
+ CAST(payload.Fax AS VARCHAR(15)) Fax,
+ CAST(payload.ShippingStreet AS VARCHAR(20)) Shipping_Street,
+ CAST(payload.ShippingCity AS VARCHAR(20)) Shipping_City,
+ CAST(payload.ShippingState AS VARCHAR(10)) Shipping_State,
+ CAST(payload.ShippingPostalCode AS VARCHAR(5)) Shipping_Post_Code,
+ CAST(payload.ShippingCountry AS VARCHAR(20)) Shipping_Country,
+ CAST(payload.Industry AS VARCHAR(50)) Industry,
+ CAST(payload.Description AS VARCHAR(200)) Description,
+ CAST(payload.NumberOfEmployees AS INT) Num_Of_Employee,
+ CAST(payload.CustomerPriority__c AS VARCHAR(10)) Priority,
+ CAST(payload.Rating AS VARCHAR(10)) Rating,
+ CAST(payload.SLA__c AS VARCHAR(10)) SLA,
+ CAST(payload."Type" AS VARCHAR(20)) Customer_Type,
+ CAST(payload.Website AS VARCHAR(100)) Customer_Website,
+ CAST(payload.AnnualRevenue AS VARCHAR(10)) Annual_Revenue,
+ CAST(payload.LastActivityDate AS DATE) Last_Activity_Date
+ FROM READ_NOS (
+ ON (SELECT CAST(NULL AS JSON CHARACTER SET Unicode))
+ USING
+ LOCATION ('/S3/s3.amazonaws.com/ptctstoutput/salesforce/1ce190bc-25a9-4493-99ad-7497b497a0d0/903790813-2020-08-21T21:02:25')
+ ACCESS_ID ('A**********') /* AccessKeyId */
+ ACCESS_KEY ('***********') /* SecretAccessKey */
+ ) AS D
+) WITH DATA;Results from the salesforceReadNOS table:
SELECT * FROM salesforceReadNOS;
Another way of placing Amazon S3 data into a relational table is by "INSERT SELECT". Using this approach, the foreign table is the source table, while a newly created permanent table is the table to be inserted into. Contrary to the READ_NOS example above, this approach does require the permanent table be created beforehand.
+One advantage of the INSERT SELECT method is that you can change the target table’s attributes. For example, you can specify that the target table be MULTISET or not, or you can choose a different primary index.
CREATE TABLE salesforcePerm, FALLBACK ,
+NO BEFORE JOURNAL,
+NO AFTER JOURNAL,
+CHECKSUM = DEFAULT,
+DEFAULT MERGEBLOCKRATIO,
+MAP = TD_MAP1
+(
+ Customer_Id VARCHAR(20) CHARACTER SET LATIN NOT CASESPECIFIC,
+ Customer_Name VARCHAR(100) CHARACTER SET LATIN NOT CASESPECIFIC,
+ Acct_Number VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC,
+ Billing_Street VARCHAR(20) CHARACTER SET LATIN NOT CASESPECIFIC,
+ Billing_City VARCHAR(20) CHARACTER SET LATIN NOT CASESPECIFIC,
+ Billing_State VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC,
+ Billing_Post_Code VARCHAR(5) CHARACTER SET LATIN NOT CASESPECIFIC,
+ Billing_Country VARCHAR(20) CHARACTER SET LATIN NOT CASESPECIFIC,
+ Phone VARCHAR(15) CHARACTER SET LATIN NOT CASESPECIFIC,
+ Fax VARCHAR(15) CHARACTER SET LATIN NOT CASESPECIFIC,
+ Shipping_Street VARCHAR(20) CHARACTER SET LATIN NOT CASESPECIFIC,
+ Shipping_City VARCHAR(20) CHARACTER SET LATIN NOT CASESPECIFIC,
+ Shipping_State VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC,
+ Shipping_Post_Code VARCHAR(5) CHARACTER SET LATIN NOT CASESPECIFIC,
+ Shipping_Country VARCHAR(20) CHARACTER SET LATIN NOT CASESPECIFIC,
+ Industry VARCHAR(50) CHARACTER SET LATIN NOT CASESPECIFIC,
+ Description VARCHAR(200) CHARACTER SET LATIN NOT CASESPECIFIC,
+ Num_Of_Employee INT,
+ Priority VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC,
+ Rating VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC,
+ SLA VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC,
+ Customer_Type VARCHAR(20) CHARACTER SET LATIN NOT CASESPECIFIC,
+ Customer_Website VARCHAR(100) CHARACTER SET LATIN NOT CASESPECIFIC,
+ Annual_Revenue VARCHAR(10) CHARACTER SET LATIN NOT CASESPECIFIC,
+ Last_Activity_Date DATE
+) PRIMARY INDEX (Customer_ID);INSERT INTO salesforcePerm
+ SELECT
+ CAST(payload.Id AS VARCHAR(20)) Customer_ID,
+ CAST(payload."Name" AS VARCHAR(100)) Customer_Name,
+ CAST(payload.AccountNumber AS VARCHAR(10)) Acct_Number,
+ CAST(payload.BillingStreet AS VARCHAR(20)) Billing_Street,
+ CAST(payload.BillingCity AS VARCHAR(20)) Billing_City,
+ CAST(payload.BillingState AS VARCHAR(10)) Billing_State,
+ CAST(payload.BillingPostalCode AS VARCHAR(5)) Billing_Post_Code,
+ CAST(payload.BillingCountry AS VARCHAR(20)) Billing_Country,
+ CAST(payload.Phone AS VARCHAR(15)) Phone,
+ CAST(payload.Fax AS VARCHAR(15)) Fax,
+ CAST(payload.ShippingStreet AS VARCHAR(20)) Shipping_Street,
+ CAST(payload.ShippingCity AS VARCHAR(20)) Shipping_City,
+ CAST(payload.ShippingState AS VARCHAR(10)) Shipping_State,
+ CAST(payload.ShippingPostalCode AS VARCHAR(5)) Shipping_Post_Code,
+ CAST(payload.ShippingCountry AS VARCHAR(20)) Shipping_Country,
+ CAST(payload.Industry AS VARCHAR(50)) Industry,
+ CAST(payload.Description AS VARCHAR(200)) Description,
+ CAST(payload.NumberOfEmployees AS INT) Num_Of_Employee,
+ CAST(payload.CustomerPriority__c AS VARCHAR(10)) Priority,
+ CAST(payload.Rating AS VARCHAR(10)) Rating,
+ CAST(payload.SLA__c AS VARCHAR(10)) SLA,
+ CAST(payload."Type" AS VARCHAR(20)) Customer_Type,
+ CAST(payload.Website AS VARCHAR(100)) Customer_Website,
+ CAST(payload.AnnualRevenue AS VARCHAR(10)) Annual_Revenue,
+ CAST(payload.LastActivityDate AS DATE) Last_Activity_Date
+ FROM salesforce;SELECT * FROM salesforcePerm;Sample results:
+
I have a newleads table with 1 row in it on Vantage system.
Note there’s no address information for this lead. Let’s use the account information retrieved from Salesforce to update newleads table
UPDATE nl
+FROM
+ newleads AS nl,
+ salesforceReadNOS AS srn
+SET
+ Street = srn.Billing_Street,
+ City = srn.Billing_City,
+ State = srn.Billing_State,
+ Post_Code = srn.Billing_Post_Code,
+ Country = srn.Billing_Country
+ WHERE Account_ID = srn.Acct_Number;Now the new lead has address information.
+
Write the new lead information into S3 bucket using WRITE_NOS.
+SELECT * FROM WRITE_NOS (
+ON (
+ SELECT
+ Account_ID,
+ Last_Name,
+ First_Name,
+ Company,
+ Cust_Title,
+ Email,
+ Status,
+ Owner_ID,
+ Street,
+ City,
+ State,
+ Post_Code,
+ Country
+ FROM newleads
+)
+USING
+ LOCATION ('/s3/vantageparquet.s3.amazonaws.com/')
+ AUTHORIZATION ('{"Access_ID":"A*****","Access_Key":"*****"}')
+ COMPRESSION ('SNAPPY')
+ NAMING ('DISCRETE')
+ INCLUDE_ORDERING ('FALSE')
+ STOREDAS ('CSV')
+) AS d;Where Access_ID is the AccessKeyID, and Access_Key is the SecretAccessKey to the bucket.
+Repeat Step 1 to create a flow using Amazon S3 as source and Salesforce as destination.
+This step provides basic information for your flow.
+Fill in Flow name (i.e., vantage2sf) and Flow description (optional), leave Customize encryption settings (advanced) unchecked. Click Next.
+This step provides information about the source and destination for your flow. For this example, we will be using Amazon S3 as the source, and Salesforce as the destination.
+For Source details, choose Amazon S3, then choose the bucket where you wrote your CSV file to (i.e. vantagecsv)
+For Destination details, choose Salesforce, use the connection you created in Step 1 from the drop-down list for Choose Salesforce connection, and Lead as Choose Salesforce object.
+For Error handling, use the default Stop the current flow run.
+Flow trigger is Run on demand. Click Next.
+This step determines how data is transferred from the source to the destination.
+Use Manually map fields as Mapping method
+Use Insert new records (default) as Destination record preference
+For Source to destination filed mapping, use the following mapping
+
Click Next.
+You can specify a filter to determine which records to transfer. For this example, no filter is added. Click Next.
+Once you are done with the Salesforce data, to avoid incurring charges to your AWS account (i.e., AppFlow, Amazon S3, Vantage and VM) for the resources used, follow these steps:
+AppFlow:
+Delete the "Connections" you created for the flow
+Delete the flows
+Amazon S3 bucket and file:
+Go to the Amazon S3 buckets where the Vantage data file is stored, and delete the file(s)
+If there are no need to keep the buckets, delete the buckets
+Teradata Vantage Instance
+Stop/Terminate the instance if no longer needed
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Wenjie Tehan
+Last updated: February 14th, 2022
This article describes the process to connect Teradata Vantage with Google Cloud Data Catalog using the Data Catalog Teradata Connector on GitHub, and then explore the metadata of the Vantage tables via Data Catalog.
+
Scrape: Connect to Teradata Vantage and retrieve all the available metadata
+Prepare: Transform metadata in Data Catalog entities and create Tags
+Ingest: Send the Data Catalog entities to the Google Cloud project
+Google Cloud Data Catalog is a fully managed data discovery and metadata management service. Data Catalog can catalog the native metadata on data assets. Data Catalog is serverless, and provides a central catalog to capture both technical metadata as well as business metadata in a structured format.
+Vantage is the modern cloud platform that unifies data warehouses, data lakes, and analytics into a single connected ecosystem.
+Vantage combines descriptive, predictive, prescriptive analytics, autonomous decision-making, ML functions, and visualization tools into a unified, integrated platform that uncovers real-time business intelligence at scale, no matter where the data resides.
+Vantage enables companies to start small and elastically scale compute or storage, paying only for what they use, harnessing low-cost object stores and integrating their analytic workloads.
+Vantage supports R, Python, Teradata Studio, and any other SQL-based tools. You can deploy Vantage across public clouds, on-premises, on optimized or commodity infrastructure, or as-a-service.
+See the documentation for more information on Teradata Vantage.
+Access to a Teradata Vantage instance.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
A Google Service Account with Data Catalog Admin role
+A Cloud Console Project created for your account (i.e. partner-integration-lab)
+Billing enabled
+Google Cloud SDK installed and initialized
+Python installed
+Pip installed
+Enable Data Catalog APIs
+Install Teradata Data Catalog Connector
+Run
+Explore Teradata Vantage metadata with Data Catalog
+Logon to Google console, choose APIs & Services from the Navigation menu, then click on Library. Make sure your project is selected on the top menu bar.
+
Put Data Catalog in the search box and click on Google Cloud Data Catalog API, click ENABLE
+
A Teradata Data Catalog connector is available on GitHub. This connector is written in Python.
+Run following command to authorize gcloud to access the Cloud Platform with Google user credentials.
+
gcloud auth loginChoose your Google account when the Google login page opens up and click Allow on the next page.
+Next, set up default project if you haven’t already done so
+gcloud config set project <project id>We recommend you install the Teradata Data Catalog Connector in an isolated Python environment. To do so, install virtualenv first:
+Run in Powershell as Administrator:
+pip install virtualenv
+virtualenv --python python3.6 <your-env>
+<your-env>\Scripts\activatepip install virtualenv
+virtualenv --python python3.6 <your-env>
+source <your-env>/bin/activatepip install virtualenv
+virtualenv --python python3.6 <your-env>
+source <your-env>/bin/activateexport GOOGLE_APPLICATION_CREDENTIALS=<google_credentials_file>
+export TERADATA2DC_DATACATALOG_PROJECT_ID=<google_cloud_project_id>
+export TERADATA2DC_DATACATALOG_LOCATION_ID=<google_cloud_location_id>
+export TERADATA2DC_TERADATA_SERVER=<teradata_server>
+export TERADATA2DC_TERADATA_USERNAME=<teradata_username>
+export TERADATA2DC_TERADATA_PASSWORD=<teradata_password>Where <google_credential_file> is the key for your service account (json file).
Execute google-datacatalog-teradata-connector command to establish entry point to Vantage database.
google-datacatalog-teradata-connector \
+ --datacatalog-project-id=$TERADATA2DC_DATACATALOG_PROJECT_ID \
+ --datacatalog-location-id=$TERADATA2DC_DATACATALOG_LOCATION_ID \
+ --teradata-host=$TERADATA2DC_TERADATA_SERVER \
+ --teradata-user=$TERADATA2DC_TERADATA_USERNAME \
+ --teradata-pass=$TERADATA2DC_TERADATA_PASSWORDSample output from the google-datacatalog-teradata-connector command:
+INFO:root: +==============Starting CLI=============== +INFO:root:This SQL connector does not implement the user defined datacatalog-entry-resource-url-prefix +INFO:root:This SQL connector uses the default entry resoure URL + +============Start teradata-to-datacatalog=========== + +==============Scrape metadata=============== +INFO:root:Scrapping metadata from connection_args + +1 table containers ready to be ingested... + +==============Prepare metadata=============== + +--> database: Gcpuser +37 tables ready to be ingested... + +==============Ingest metadata=============== + +DEBUG:google.auth._default:Checking /Users/Teradata/Apps/Cloud/GCP/teradata2dc-credentials.json for explicit credentials as part of auth process... +INFO:root:Starting to clean up the catalog... +DEBUG:google.auth.transport.requests:Making request: POST https://oauth2.googleapis.com/token +DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): oauth2.googleapis.com:443 +DEBUG:urllib3.connectionpool:https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None +INFO:root:0 entries that match the search query exist in Data Catalog! +INFO:root:Looking for entries to be deleted... +INFO:root:0 entries will be deleted. + +Starting to ingest custom metadata... + +DEBUG:google.auth._default:Checking /Users/Teradata/Apps/Cloud/GCP/teradata2dc-credentials.json for explicit credentials as part of auth process... +INFO:root:Starting the ingestion flow... +DEBUG:google.auth.transport.requests:Making request: POST https://oauth2.googleapis.com/token +DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): oauth2.googleapis.com:443 +DEBUG:urllib3.connectionpool:https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None +INFO:root:Tag Template created: projects/partner-integration-lab/locations/us-west1/tagTemplates/teradata_database_metadata +INFO:root:Tag Template created: projects/partner-integration-lab/locations/us-west1/tagTemplates/teradata_table_metadata +INFO:root:Tag Template created: projects/partner-integration-lab/locations/us-west1/tagTemplates/teradata_column_metadata +INFO:root:Entry Group created: projects/partner-integration-lab/locations/us-west1/entryGroups/teradata +INFO:root:1/38 +INFO:root:Entry does not exist: projects/partner-integration-lab/locations/us-west1/entryGroups/teradata/entries/gcpuser +INFO:root:Entry created: projects/partner-integration-lab/locations/us-west1/entryGroups/teradata/entries/gcpuser +INFO:root: ^ [database] 34.105.107.155/gcpuser +INFO:root:Starting the upsert tags step +INFO:root:Processing Tag from Template: projects/partner-integration-lab/locations/us-west1/tagTemplates/teradata_database_metadata ... +INFO:root:Tag created: projects/partner-integration-lab/locations/us-west1/entryGroups/teradata/entries/gcpuser/tags/CWHNiGQeQmPT +INFO:root:2/38 +INFO:root:Entry does not exist: projects/partner-integration-lab/locations/us-west1/entryGroups/teradata/entries/gcpuser_Categories +INFO:root:Entry created: projects/partner-integration-lab/locations/us-west1/entryGroups/teradata/entries/gcpuser_Categories +INFO:root: ^ [table] 34.105.107.155/gcpuser/Categories +INFO:root:Starting the upsert tags step +INFO:root:Processing Tag from Template: projects/partner-integration-lab/locations/us-west1/tagTemplates/teradata_table_metadata ... +INFO:root:Tag created: projects/partner-integration-lab/locations/us-west1/entryGroups/teradata/entries/gcpuser_Categories/tags/Ceij5G9t915o +INFO:root:38/38 +INFO:root:Entry does not exist: projects/partner-integration-lab/locations/us-west1/entryGroups/teradata/entries/gcpuser_tablesv_instantiated_latest +INFO:root:Entry created: projects/partner-integration-lab/locations/us-west1/entryGroups/teradata/entries/gcpuser_tablesv_instantiated_latest +INFO:root: ^ [table] 34.105.107.155/gcpuser/tablesv_instantiated_latest +INFO:root:Starting the upsert tags step +INFO:root:Processing Tag from Template: projects/partner-integration-lab/locations/us-west1/tagTemplates/teradata_table_metadata ... +INFO:root:Tag created: projects/partner-integration-lab/locations/us-west1/entryGroups/teradata/entries/gcpuser_tablesv_instantiated_latest/tags/Ceij5G9t915o +INFO:root: +============End teradata-to-datacatalog============+
Go to Data Catalog console, click on the project (i.e. partner-integration-lab) under Projects. The Teradata tables are showing on the right panel.
+
Click on the table to your interest (i.e. CITY_LEVEL_TRANS), and you’ll see the metadata about this table:
+
Clean up metadata from Data Catalog. To do that, copy https://github.com/GoogleCloudPlatform/datacatalog-connectors-rdbms/blob/master/google-datacatalog-teradata-connector/tools/cleanup_datacatalog.py to local directory.
+Change directory to where the file is and then run following command:
+python cleanup_datacatalog.py --datacatalog-project-ids=$TERADATA2DC_DATACATALOG_PROJECT_ID| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Wenjie Tehan
+Last updated: February 8th, 2022
This how-to will help you to integrate Amazon SageMaker with Teradata Vantage. The approach this guide explains is one of many potential approaches to integrate with the service.
+Amazon SageMaker provides a fully managed Machine Learning Platform. There are two use cases for Amazon SageMaker and Teradata:
+Data resides on Teradata Vantage and Amazon SageMaker will be used for both the Model definition and subsequent scoring. Under this use case Teradata will provide data into the Amazon S3 environment so that Amazon SageMaker can consume training and test data sets for the purpose of model development. Teradata would further make data available via Amazon S3 for subsequent scoring by Amazon SageMaker. Under this model Teradata is a data repository only.
+Data resides on Teradata Vantage and Amazon SageMaker will be used for the Model definition, and Teradata for the subsequent scoring. Under this use case Teradata will provide data into the Amazon S3 environment so that Amazon SageMaker can consume training and test data sets for the purpose of model development. Teradata will need to import the Amazon SageMaker model into a Teradata table for subsequent scoring by Teradata Vantage. Under this model Teradata is a data repository and a scoring engine.
+The first use case is discussed in this document.
+Amazon SageMaker consumes training and test data from an Amazon S3 bucket. This article describes how you can load Teradata analytics data sets into an Amazon S3 bucket. The data can then available to Amazon SageMaker to build and train machine learning models and deploy them into a production environment.
+Access to a Teradata Vantage instance.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
IAM permission to access Amazon S3 bucket, and to use Amazon SageMaker service.
+An Amazon S3 bucket to store training data.
+Amazon SageMaker trains data from an Amazon S3 bucket. Following are the steps to load training data from Vantage to an Amazon S3 bucket:
+Go to Amazon SageMaker console and create a notebook instance. See Amazon SageMaker Developer Guide for instructions on how to create a notebook instance:
+
+Open your notebook instance:
+
+Start a new file by clicking on New → conda_python3:
+Install Teradata Python library:
+!pip install teradatamlIn a new cell and import additional libraries:
+import teradataml as tdml
+from teradataml import create_context, get_context, remove_context
+from teradataml.dataframe.dataframe import DataFrame
+import pandas as pd
+import boto3, osIn a new cell, connect to Teradata Vantage. Replace <hostname>, <database user name>, <database password> to match your Vantage environment:
create_context(host = '<hostname>', username = '<database user name>', password = '<database password>')Retrieve data rom the table where the training dataset resides using TeradataML DataFrame API:
+train_data = tdml.DataFrame('table_with_training_data')
+trainDF = train_data.to_pandas()Write data to a local file:
+trainFileName = 'train.csv'
+trainDF.to_csv(trainFileName, header=None, index=False)Upload the file to Amazon S3:
+bucket = 'sagedemo'
+prefix = 'sagemaker/train'
+
+trainFile = open(trainFileName, 'rb')
+boto3.Session().resource('s3').Bucket(bucket).Object(os.path.join(prefix, localFile)).upload_fileobj(trainFile)Select Training jobs on the left menu under Training, then click on Create training job:
+At the Create training job window, fill in the Job name (e.g. xgboost-bank) and Create a new role for the IAM role. Choose Any S3 bucket for the Amazon S3 buckets and Create role:
+Back in the Create training job window, use XGBoost as the algorithm:
+Use the default ml.m4.xlarge instance type, and 30GB of additional storage volume per instance. This is a short training job, shouldn’t take more than 10 minutes.
+Fill in following hyperparameters and leave everything else as default:
+num_round=100 +silent=0 +eta=0.2 +gamma=4 +max_depth=5 +min_child_weight=6 +subsample=0.8 +objective='binary:logistic'+
For Input data configuration, enter the Amazon S3 bucket where you stored your training data. Input mode is File. Content type is csv. S3 location is where the file uploaded to:
+For Output data configuration, enter path where the output data will be stored:
+Leave everything else as default, and click on “Create training job”. Detail instructions on how to configure the training job can be found in Amazon SageMaker Developer Guide.
+Once the training job’s created, Amazon SageMaker launches the ML instances to train the model, and stores the resulting model artifacts and other output in the Output data configuration (path/<training job name>/output by default).
After you train your model, deploy it using a persistent endpoint
+Select Models under Inference from the left panel, then Create model. Fill in the model name (e.g. xgboost-bank), and choose the IAM role you created from the previous step.
For Container definition 1, use 433757028032.dkr.ecr.us-west-2.amazonaws.com/xgboost:latest as Location of inference code image. Location of model artifacts is the output path of your training job
+Leave everything else as default, then Create model.
Select the model you just created, then click on Create endpoint configuration:
+Fill in the name (e.g. xgboost-bank) and use default for everything else. The model name and training job should be automatically populated for you. Click on Create endpoint configuration.
Select Inference → Models from the left panel, select the model again, and click on Create endpoint this time:
+Fill in the name (e.g. xgboost-bank), and select Use an existing endpoint configuration:
+image::sagemaker-with-teradata-vantage/attach.endpoint.configuration.png[Attach endpoint configuration]
Select the endpoint configuration created from last step, and click on Select endpoint configuration:
+Leave everything else as default and click on Create endpoint.
Now the model is deployed to the endpoint and can be used by client applications.
+This how-to demonstrated how to extract training data from Vantage and use it to train a model in Amazon SageMaker. The solution used a Jupyter notebook to extract data from Vantage and write it to an S3 bucket. A SageMaker training job read data from the S3 bucket and produced a model. The model was deployed to AWS as a service endpoint.
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Rupal Shah
+Last updated: February 14th, 2022
Azure Machine Learning (ML) Studio is a collaborative, drag-and-drop tool you can use to build, test, and deploy predictive analytics solutions on your data. ML Studio can consume data from Azure Blob Storage. This getting started guide will show how you can copy Teradata Vantage data sets to a Blob Storage using ML Studio 'built-in' Jupter Notebook feature. The data can then be used by ML Studio to build and train machine learning models and deploy them into a production environment.
+
Access to a Teradata Vantage instance.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
Azure subscription or create free account
+(Optional) Download AdventureWorks DW 2016 database (i.e. 'Training the Model' section)
+Restore and copy 'vTargetMail' table from SQL Server to Teradata Vantage
+During ML Studio workspace creation, you may need to create 'new' storage account unless you have one in current availability locations and choose DEVTEST Standard for Web service plan for this getting started guide. Logon to Azure portal, open your storage account and create a container if one does not exist already.
+
Copy your storage account name and key to notepad which we will use for Python3 Notebook to access your Azure Blob Storage account.
+
Finally, open Configuration property and set 'Secure transfer required' to Disabled to allow ML Studio Import Data module to access blob storage account.
+
To get the data to ML Studio, we first need to load data from Teradata Vantage to a Azure Blob Storage. We will create a ML Jupyter Notebook, install Python packages to connect to Teradata and save data to Azure Blob Storage,
+Logon to Azure portal, go to to your ML Studio workspace and Launch Machine Learning Studio and Sign In.
+You should see the following screen and click on Notebooks, ensure you are in the right region/ workspace and click on Notebook New
+
Choose Python3 and name your notebook instance
+
In your jupyter notebook instance, install Teradata Vantage Python package for Advanced Analytics:
+pip install teradataml| + + | ++There is no validation between Microsoft Azure ML Studio and Teradata Vantage Python package. + | +
Install Microsoft Azure Storage Blob Client Library for Python:
+!pip install azure-storage-blobImport the following libraries:
+import teradataml as tdml
+from teradataml import create_context, get_context, remove_context
+from teradataml.dataframe.dataframe import DataFrame
+import pandas as pd
+from azure.storage.blob import (BlockBlobService)Connect to Teradata using command:
+create_context(host = '<hostname>', username = '<database user name>', password = '<password>')Retrieve Data using Teradata Python DataFrame module:
+train_data = DataFrame.from_table("<table_name>")Convert Teradata DataFrame to Panda DataFrame:
+trainDF = train_data.to_pandas()Convert data to CSV:
+trainDF = trainDF.to_csv(head=True,index=False)Assign variables for Azue Blob Storage account name, key and container name:
+accountName="<account_name>"
+accountKey="<account_key>"
+containerName="mldata"Upload file to Azure Blob Storage:
+blobService = BlockBlobService(account_name=accountName, account_key=accountKey)
+blobService.create_blob_from_text(containerNAme, 'vTargetMail.csv', trainDF)Logon to Azure portal, open blob storage account to view uploaded file:
+
We will use the existing Analyze data with Azure Machine Learning article to build a predictive machine learning model based on data from Azure Blob Storage. We will build a targeted marketing campaign for Adventure Works, the bike shop, by predicting if a customer is likely to buy a bike or not.
+The data is on Azure Blob Storage file called vTargetMail.csv which we copied in the section above.
1.. Sign into Azure Machine Learning studio and click on Experiments. +2.. Click +NEW on the bottom left of the screen and select Blank Experiment. +3.. Enter a name for your experiment: Targeted Marketing. +4.. Drag Import data module under Data Input and output from the modules pane into the canvas. +5.. Specify the details of your Azure Blob Storage (account name, key and container name) in the Properties pane.
+Run the experiment by clicking Run under the experiment canvas.
+
After the experiment finishes running successfully, click the output port at the bottom of the Import Data module and select Visualize to see the imported data.
+
To clean the data, drop some columns that are not relevant for the model. To do this:
+Drag Select Columns in Dataset module under Data Transformation < Manipulation into the canvas. Connect this module to the Import Data module.
+Click Launch column selector in Properties pane to specify which columns you wish to drop.
+
Exclude two columns: CustomerAlternateKey and GeographyKey.
+
We will split the data 80-20: 80% to train a machine learning model and 20% to test the model. We will make use of the "Two-Class" algorithms for this binary classification problem.
+Drag SplitData module into the canvas and connect with 'Select Columns in DataSet'.
+In the properties pane, enter 0.8 for Fraction of rows in the first output dataset.
+
Search and drag Two-Class Boosted Decision Tree module into the canvas.
+Search and drag Train Model module into the canvas and specify inputs by connecting it to the Two-Class Boosted Decision Tree (ML algorithm) and Split Data (data to train the algorithm on) modules.
+
Then, click Launch column selector in the Properties pane. Select the BikeBuyer column as the column to predict.
+
Now, we will test how the model performs on test data. We will compare the algorithm of our choice with a different algorithm to see which performs better.
+Drag Score Model module into the canvas and connect it to Train Model and Split Data modules.
+
Search and drag Two-Class Bayes Point Machine into the experiment canvas. We will compare how this algorithm performs in comparison to the Two-Class Boosted Decision Tree.
+Copy and Paste the modules Train Model and Score Model in the canvas.
+Search and drag Evaluate Model module into the canvas to compare the two algorithms.
+Run the experiment.
+
Click the output port at the bottom of the Evaluate Model module and click Visualize.
+
The metrics provided are the ROC curve, precision-recall diagram and lift curve. Looking at these metrics, we can see that the first model performed better than the second one. To look at the what the first model predicted, click on output port of the Score Model and click Visualize.
+
You will see two more columns added to your test dataset. +1. Scored Probabilities: the likelihood that a customer is a bike buyer. +2. Scored Labels: the classification done by the model - bike buyer (1) or not (0). This probability threshold for labeling is set to 50% and can be adjusted.
+Comparing the column BikeBuyer (actual) with the Scored Labels (prediction), you can see how well the model has performed. As next steps, you can use this model to make predictions for new customers and publish this model as a web service or write results back to SQL Data Warehouse.
+To learn more about building predictive machine learning models, refer to Introduction to Machine Learning on Azure.
+For large data set copies, consider using the Teradata Access Module for Azure that interfaces between the Teradata Parallel Transporter load/unload operators and Azure Blob Storage.
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
The page you requested has been relocated to https://quickstarts.teradata.com/mule-teradata-connector/examples-configuration.html.
diff --git a/pr-preview/pr-115/connectors/database/reference.html b/pr-preview/pr-115/connectors/database/reference.html new file mode 100644 index 000000000..f081ea1b5 --- /dev/null +++ b/pr-preview/pr-115/connectors/database/reference.html @@ -0,0 +1,9 @@ + + + + + + +The page you requested has been relocated to https://quickstarts.teradata.com/mule-teradata-connector/reference.html.
diff --git a/pr-preview/pr-115/connectors/database/release-notes.html b/pr-preview/pr-115/connectors/database/release-notes.html new file mode 100644 index 000000000..cec2a502b --- /dev/null +++ b/pr-preview/pr-115/connectors/database/release-notes.html @@ -0,0 +1,9 @@ + + + + + + +The page you requested has been relocated to https://quickstarts.teradata.com/mule-teradata-connector/release-notes.html.
diff --git a/pr-preview/pr-115/connectors/db/example-configuration.html b/pr-preview/pr-115/connectors/db/example-configuration.html new file mode 100644 index 000000000..971287125 --- /dev/null +++ b/pr-preview/pr-115/connectors/db/example-configuration.html @@ -0,0 +1,9 @@ + + + + + + +The page you requested has been relocated to https://quickstarts.teradata.com/mule-teradata-connector/examples-configuration.html.
diff --git a/pr-preview/pr-115/connectors/db/reference.html b/pr-preview/pr-115/connectors/db/reference.html new file mode 100644 index 000000000..f081ea1b5 --- /dev/null +++ b/pr-preview/pr-115/connectors/db/reference.html @@ -0,0 +1,9 @@ + + + + + + +The page you requested has been relocated to https://quickstarts.teradata.com/mule-teradata-connector/reference.html.
diff --git a/pr-preview/pr-115/connectors/db/release-notes.html b/pr-preview/pr-115/connectors/db/release-notes.html new file mode 100644 index 000000000..cec2a502b --- /dev/null +++ b/pr-preview/pr-115/connectors/db/release-notes.html @@ -0,0 +1,9 @@ + + + + + + +The page you requested has been relocated to https://quickstarts.teradata.com/mule-teradata-connector/release-notes.html.
diff --git a/pr-preview/pr-115/create-parquet-files-in-object-storage.html b/pr-preview/pr-115/create-parquet-files-in-object-storage.html new file mode 100644 index 000000000..393b06755 --- /dev/null +++ b/pr-preview/pr-115/create-parquet-files-in-object-storage.html @@ -0,0 +1,2725 @@ + + + + + +Author: Obed Vega
+Last updated: August 2nd, 2022
Native Object Storage (NOS) is a Vantage feature that allows you to query data stored in files such as CSV, JSON, and Parquet format datasets. +These datasets are located on external S3-compatible object storage such as AWS S3, Google GCS, Azure Blob or on-prem implementations. +It’s useful in scenarios where you want to explore data without building a data pipeline to bring it into Vantage. This tutorial demonstrates how to export data from Vantage to object storage using the Parquet file format.
+You need access to a Teradata Vantage instance. NOS is enabled in all Vantage editions from Vantage Express through Developer, DYI to Vantage as a Service starting from version 17.10.
+| + + | ++This tutorial is based on s3 aws object storage. You will need your own s3 bucket with write permissions to complete the tutorial. + | +
| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
WRITE_NOS allows you to extract selected or all columns from a database table or from derived results and write to external object storage, such as Amazon S3, Azure Blob storage, Azure Data Lake Storage Gen2, and Google Cloud Storage. This functionality stores data in Parquet format.
You can find more documentation about WRITE_NOS functionality in the NOS documentation.
You will need access to a database where you can execute WRITE_NOS function. If you don’t have such a database, run the following commands:
CREATE USER db AS PERM=10e7, PASSWORD=db;
+
+-- Don't forget to give the proper access rights
+GRANT EXECUTE FUNCTION on TD_SYSFNLIB.READ_NOS to db;
+GRANT EXECUTE FUNCTION on TD_SYSFNLIB.WRITE_NOS to db;| + + | ++If you would like to learn more about setting up users and their privileges, checkout the NOS documentation. + | +
Let’s first create a table on your Teradata Vantage instance:
+CREATE SET TABLE db.parquet_table ,FALLBACK ,
+ NO BEFORE JOURNAL,
+ NO AFTER JOURNAL,
+ CHECKSUM = DEFAULT,
+ DEFAULT MERGEBLOCKRATIO,
+ MAP = TD_MAP1
+ (
+ column1 SMALLINT NOT NULL,
+ column2 DATE FORMAT 'YY/MM/DD' NOT NULL,
+ column3 DECIMAL(10,2))
+PRIMARY INDEX ( column1 );Populate your table with example data:
+INSERT INTO db.parquet_table (1,'2022/01/01',1.1);
+INSERT INTO db.parquet_table (2,'2022/01/02',2.2);
+INSERT INTO db.parquet_table (3,'2022/01/03',3.3);Your table should now look like this:
+column1 column2 column3 +------- -------- ------------ + 1 22/01/01 1.10 + 2 22/01/02 2.20 + 3 22/01/03 3.30+
Create the parquet file with WRITE_NOS. Don’t forget to replace <BUCKET_NAME> with the name of your s3 bucket. Also,replace <YOUR-ACCESS-KEY-ID> and <YOUR-SECRET-ACCESS-KEY> with your access key and secret.
| + + | ++Check your cloud provider docs how to create credentials to access object storage. For example, for AWS check out How do I create an AWS access key? + | +
SELECT * FROM WRITE_NOS (
+ON ( SELECT * FROM db.parquet_table)
+USING
+LOCATION('/s3/<BUCKET_NAME>.s3.amazonaws.com/parquet_file_on_NOS.parquet')
+AUTHORIZATION('{"ACCESS_ID":"<YOUR-ACCESS-KEY-ID>",
+"ACCESS_KEY":"<YOUR-SECRET-ACCESS-KEY>"}')
+STOREDAS('PARQUET')
+MAXOBJECTSIZE('16MB')
+COMPRESSION('SNAPPY')
+INCLUDE_ORDERING('TRUE')
+INCLUDE_HASHBY('TRUE')
+) as d;Now you have created a parquet file in your object storage bucket. Now to easily query your file you need to follow step number 4.
+Create a NOS-backed foreign table. Don’t forget to replace <BUCKET_NAME> with the name of your s3 bucket. Also,replace <YOUR-ACCESS-KEY-ID> and <YOUR-SECRET-ACCESS-KEY> with your access key and secret:
CREATE MULTISET FOREIGN TABLE db.parquet_table_to_read_file_on_NOS
+, EXTERNAL SECURITY DEFINER TRUSTED CEPH_AUTH,
+MAP = TD_MAP1
+(
+ Location VARCHAR(2048) CHARACTER SET UNICODE CASESPECIFIC
+ , col1 SMALLINT
+ , col2 DATE
+ , col3 DECIMAL(10,2)
+
+)
+USING (
+ LOCATION ('/s3/<BUCKET_NAME>.s3.amazonaws.com/parquet_file_on_NOS.parquet')
+ AUTHORIZATION('{"ACCESS_ID":"<YOUR-ACCESS-KEY-ID>",
+ "ACCESS_KEY":"<YOUR-SECRET-ACCESS-KEY>"}')
+ STOREDAS ('PARQUET')
+)NO PRIMARY INDEX;Now you are ready to Query your parquet file on NOS, let’s try the following query:
+SELECT col1, col2, col3 FROM db.parquet_table_to_read_file_on_NOS;The data returned from the query should look something like this:
+col1 col2 col3 +------ -------- ------------ + 1 22/01/01 1.10 + 2 22/01/02 2.20 + 3 22/01/03 3.30+
In this tutorial we have learned how to export data from Vantage to a parquet file on object storage using Native Object Storage (NOS). NOS supports reading and importing data stored in CSV, JSON and Parquet formats. NOS can also export data from Vantage to object storage.
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Adam Tworkiewicz
+Last updated: July 12th, 2023
This tutorial demonstrates how to use dbt (Data Build Tool) with Teradata Vantage. It’s based on the original dbt Jaffle Shop tutorial. A couple of models have been adjusted to the SQL dialect supported by Vantage.
+Access to a Teradata Vantage instance.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
Python 3.7, 3.8, 3.9, 3.10 or 3.11 installed.
+Clone the tutorial repository and cd into the project directory:
+git clone https://github.com/Teradata/jaffle_shop-dev.git jaffle_shop
+cd jaffle_shopCreate a new python environment to manage dbt and its dependencies. Activate the environment:
+python3 -m venv env
+source env/bin/activateInstall dbt-teradata module and its dependencies. The core dbt module is included as a dependency so you don’t have to install it separately:
pip install dbt-teradataWe will now configure dbt to connect to your Vantage database. Create file $HOME/.dbt/profiles.yml with the following content. Adjust <host>, <user>, <password> to match your Teradata instance.
| + + | +
+ Database setup
+
+
+The following dbt profile points to a database called
+
+
+
+ |
+
jaffle_shop:
+ outputs:
+ dev:
+ type: teradata
+ host: <host>
+ user: <user>
+ password: <password>
+ logmech: TD2
+ schema: jaffle_shop
+ tmode: ANSI
+ threads: 1
+ timeout_seconds: 300
+ priority: interactive
+ retries: 1
+ target: devNow, that we have the profile file in place, we can validate the setup:
+dbt debugIf the debug command returned errors, you likely have an issue with the content of profiles.yml.
jaffle_shop is a fictional e-commerce store. This dbt project transforms raw data from an app database into a dimensional model with customer and order data ready for analytics.
The raw data from the app consists of customers, orders, and payments, with the following entity-relationship diagram:
+
+dbt takes these raw data table and builds the following dimensional model, which is more suitable for analytics tools:
+
+In real life, we will be getting raw data from platforms like Segment, Stitch, Fivetran or another ETL tool. In our case, we will use dbt’s seed functionality to create tables from csv files. The csv files are located in ./data directory. Each csv file will produce one table. dbt will inspect the files and do type inference to decide what data types to use for columns.
Let’s create the raw data tables:
+dbt seedYou should now see 3 tables in your jaffle_shop database: raw_customers, raw_orders, raw_payments. The tables should be populated with data from the csv files.
Now that we have the raw tables, we can instruct dbt to create the dimensional model:
+dbt runSo what exactly happened here? dbt created additional tables using CREATE TABLE/VIEW FROM SELECT SQL. In the first transformation, dbt took raw tables and built denormalized join tables called customer_orders, order_payments, customer_payments. You will find the definitions of these tables in ./marts/core/intermediate.
+In the second step, dbt created dim_customers and fct_orders tables. These are the dimensional model tables that we want to expose to our BI tool.
dbt applied multiple transformations to our data. How can we ensure that the data in the dimensional model is correct? dbt allows us to define and execute tests against the data. The tests are defined in ./marts/core/schema.yml. The file describes each column in all relationships. Each column can have multiple tests configured under tests key. For example, we expect that fct_orders.order_id column will contain unique, non-null values. To validate that the data in the produced tables satisfies the test conditions run:
dbt testOur model consists of just a few tables. Imagine a scenario where where we have many more sources of data and a much more complex dimensional model. We could also have an intermediate zone between the raw data and the dimensional model that follows the Data Vault 2.0 principles. Would it not be useful, if we had the inputs, transformations and outputs documented somehow? dbt allows us to generate documentation from its configuration files:
+dbt docs generateThis will produce html files in ./target directory.
You can start your own server to browse the documentation. The following command will start a server and open up a browser tab with the docs' landing page:
+dbt docs serveThis tutorial demonstrated how to use dbt with Teradata Vantage. The sample project takes raw data and produces a dimensional data mart. We used multiple dbt commands to populate tables from csv files (dbt seed), create models (dbt run), test the data (dbt test), and generate and serve model documentation (dbt docs generate, dbt docs serve).
| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Krutik Pathak
+Last updated: July 27, 2023
This tutorial demonstrates how to use dbt (Data Build Tool) to transform external data load through Airbyte (an Open-Source Extract Load tool) in Teradata Vantage.
+This tutorial is based on the original dbt Jaffle Shop tutorial with a small change, instead of using the dbt seed command, the Jaffle Shop dataset is loaded from Google Sheets into Teradata Vantage using Airbyte. Data loaded through airbyte is contained in JSON columns as can be seen in the picture below:
+Access to a Teradata Vantage Instance.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
Sample data: The sample data Jaffle Shop Dataset can be found in Google Sheets.
+Reference dbt project repository: Jaffle Project with Airbyte.
+Python 3.7, 3.8, 3.9, 3.10 or 3.11 installed.
+Follow the steps in the Airbyte tutorial. Make sure you load data from the Jaffle Shop spreadsheet and not the default dataset referenced by the Airbyte tutorial. Also, set the Default Schema in the Teradata destination to airbyte_jaffle_shop.
| + + | +
+
+
+When you configure a Teradata destination in Airbyte, it will ask for a |
+
Clone the tutorial repository and change the directory to the project directory:
++
+git clone https://github.com/Teradata/airbyte-dbt-jaffle
+cd airbyte-dbt-jaffleCreate a new python environment to manage dbt and its dependencies. Activate the environment:
+python3 -m venv env
+source env/bin/activate| + + | +
+
+
+You can activate the virtual environment in Windows by executing the corresponding batch file |
+
Install dbt-teradata module and its dependencies. The core dbt module is included as a dependency so you don’t have to install it separately:
pip install dbt-teradataInitialize a dbt project.
+dbt initThe dbt project wizard will ask you for a project name and database management system to use in the project. In this demo, we define the project name as dbt_airbyte_demo. Since we are using the dbt-teradata connector, the only database management system available is Teradata.
+
+Configure the profiles.yml file located in the $HOME/.dbt directory. If the profiles.yml file is not present, you can create a new one.
Adjust server, username, password to match your Teradata instance’s HOST, Username, Password respectively.
In this configuration, schema stands for the database that contains the sample data, in our case that is the default schema that we defined in Airbyte airbyte_jaffle_shop.
dbt_airbyte_demo:
+ target: dev
+ outputs:
+ dev:
+ type: teradata
+ server: <host>
+ schema: airbyte_jaffle_shop
+ username: <user>
+ password: <password>
+ tmode: ANSIOnce the profiles.yml file is ready, we can validate the setup. Go to the dbt project folder and run the command:
dbt debugIf the debug command returned errors, you likely have an issue with the content of profiles.yml. If the setup is correct, you will get message All checks passed!
+jaffle_shop is a fictional restaurant that takes orders online. The data of this business consists of tables for customers, orders and `payments`that follow the entity relations diagram below:
+The data in the source system is normalized. A dimensional model based on the same data, more suitable for analytics tools, is presented below:
+
+| + + | +
+
+
+The complete dbt project encompassing the transformations detailed below is located at Jaffle Project with Airbyte. + |
+
The reference dbt project performs two types of transformations.
+First, it transforms the raw data (in JSON format), loaded from Google Sheets via Airbyte, into staging views. At this stage the data is normalized.
+Next, it transforms the normalized views into a dimensional model ready for analytics.
+The following diagram shows the transformation steps in Teradata Vantage using dbt:
+
+As in all dbt projects, the folder models contains the data models that the project materializes as tables, or views, according to the corresponding configurations at the project, or individual model level.
The models can be organized into different folders according to their purpose in the organization of the data warehouse/lake. Common folder layouts include a folder for staging, a folder for core, and a folder for marts. This structure can be simplified without affecting the workings of dbt.
In the original dbt Jaffle Shop tutorial the project’s data is loaded from csv files located in the ./data folder through dbt’s seed command. The seed command is commonly used to load data from tables, however, this command is not designed to perform data loading.
In this demo we are assuming a more typical setup in which a tool designed for data loading, Airbyte, was used to load data into the datawarehouse/lake. +Data loaded through Airbyte though is represented as raw JSON strings. From these raw data we are creating normalized staging views. We perform this task through the following staging models.
+The stg_customers model creates the normalized staging view for customers from the _airbyte_raw_customers table.
The stg_orders model creates the normalized view for orders from the _airbyte_raw_orders table
The stg_payments model creates the normalized view for payments from the _airbyte_raw_payments table.
| + + | +
+
+
+As the method of extracting JSON strings remains consistent across all staging models, we will provide a detailed explanation for the transformations using just one of these models as an example. + |
+
Below an example of transforming raw JSON data into a view through the stg_orders.sql model :
WITH source AS (
+ SELECT * FROM {{ source('airbyte_jaffle_shop', '_airbyte_raw_orders')}}
+),
+
+flattened_json_data AS (
+ SELECT
+ _airbyte_data.JSONExtractValue('$.id') AS order_id,
+ _airbyte_data.JSONExtractValue('$.user_id') AS customer_id,
+ _airbyte_data.JSONExtractValue('$.order_date') AS order_date,
+ _airbyte_data.JSONExtractValue('$.status') AS status
+ FROM source
+)
+
+
+SELECT * FROM flattened_json_dataIn this model the source is defined as the raw table _airbyte_raw_orders.
This raw table columns contains both metadata, and the actual ingested data. The data column is called _airbyte_data.
This column is of Teradata JSON type. This type supports the method JSONExtractValue for retrieving scalar values from the JSON object.
+In this model we are retrieving each of the attributes of interest and adding meaningful aliases in order to materialize a view.
+Building a Dimensional Model is a two-step process:
+First, we take the normalized views in stg_orders, stg_customers, stg_payments and build denormalized intermediate join tables customer_orders, order_payments, customer_payments. You will find the definitions of these tables in ./models/marts/core/intermediate.
In the second step, we create the dim_customers and fct_orders models. These constitute the dimensional model tables that we want to expose to our BI tool. You will find the definitions of these tables in ./models/marts/core.
For executing the transformations defined in the dbt project we run:
+dbt runYou will get the status of each model as given below:
+
+To ensure that the data in the dimensional model is correct, dbt allows us to define and execute tests against the data.
+The tests are defined in ./models/marts/core/schema.yml and ./models/staging/schema.yml. Each column can have multiple tests configured under the tests key.
For example, we expect that fct_orders.order_id column will contain unique, non-null values.
To validate that the data in the produced tables satisfies the test conditions run:
+dbt testIf the data in the models satisfies all the test cases, the result of this command will be as below:
+
+Our model consists of just a few tables. In a scenario with more sources of data, and a more complex dimensional model, documenting the data lineage and what is the purpose of each of the intermediate models is very important.
+Generating this type of documentation with dbt is very straight forward.
+dbt docs generateThis will produce html files in the ./target directory.
You can start your own server to browse the documentation. The following command will start a server and open up a browser tab with the docs' landing page:
+dbt docs serveThis tutorial demonstrated how to use dbt to transform raw JSON data loaded through Airbyte into dimensional model in Teradata Vantage. The sample project takes raw JSON data loaded in Teradata Vantage, creates normalized views and finally produces a dimensional data mart. We used dbt to transform JSON into Normalized views and multiple dbt commands to create models (dbt run), test the data (dbt test), and generate and serve model documentation (dbt docs generate, dbt docs serve).
| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Krutik Pathak
+Last updated: June 9th, 2023
This tutorial showcases how to use Airbyte (an open-source Extract Load Transform tool) with Teradata Vantage. We work with a very simple end-to-end setup to load data from Google Sheets to Teradata Vantage using Airbyte.
+
+Source: Google Sheets
+Destination: Teradata Vantage
+Access to a Teradata Vantage Instance. This will be defined as the destination of the Airbyte connection. You will need a database Host, Username, and Password for Airbyte’s configuration.
| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
Docker Compose to run Airbyte Open Source locally. Docker Compose comes with Docker Desktop. Please refer to docker docs for more details.
+Data from the source system. In this case, we use a sample spreadsheet from google sheets. The sample data is a breakdown of payrate by employee type.
+Google Cloud Platform API enabled for your personal or organizational account. You’ll need to authenticate your Google account via OAuth or via Service Account Key Authenticator. In this example, we use Service Account Key Authenticator.
+Clone the Airbyte Open Source repository and go to the airbyte directory.
+
+git clone --depth 1 https://github.com/airbytehq/airbyte.git
+cd airbyteMake Sure to have Docker Desktop running before running the shell script run-ab-platform.
Run the shell script run-ab-platform as
./run-ab-platform.sh| + + | +
+
+
+You can run the above commands with |
+
Log in to the web app http://localhost:8000/ by entering the default credentials found in the .env file included in the repository.
BASIC_AUTH_USERNAME=airbyte
+BASIC_AUTH_PASSWORD=passwordWhen logging in for the first time, Airbyte will prompt you to provide your email address and specify your preferences for product improvements. Enter your preferences and click on "Get started."
+
+Once Airbyte Open Source is launched you will see a connections dashboard. If you launched Airbyte Open Source for the first time, it would not show any connections.
+You can either click "Create your first connection" or click on the top right corner to initiate the new connection workflow on Airbyte’s Connections dashboard.
+
+Airbyte will ask you for the Source, you can select from an existing source (if you have set it up already) or you can set up a new source, in this case we select Google Sheets.
For authentication we are using Service Account Key Authentication which uses a service account key in JSON format. Toggle from the default OAuth to Service Account Key Authentication. To authenticate your Google account via Service Account Key Authentication, enter your Google Cloud service account key in JSON format.
+Make sure the Service Account has the Project Viewer permission. If your spreadsheet is viewable by anyone with its link, no further action is needed. If not, give your Service account access to your spreadsheet.
Add the link to the source spreadsheet as Spreadsheet Link.
+| + + | +
+
+
+For more details, please refer Setting Google Sheets as Source Connector in Airbyte Open Source + |
+
Click Set up source, if the configuration is correct, you will get the message All connection tests passed!
Assuming you want to create a fresh new connection with Teradata Vantage, Select Teradata Vantage as the destination type under the "Set up the destination" section.
Add the Host, User, and Password. These are the same as the Host, Username, and Password respectively, used by your Clearscape Analytics Environment.
Provide a default schema name appropriate to your specific context. Here we have provided gsheet_airbyte_td.
| + + | +
+
+
+If you do not provide a |
+
+Click Set up destination, if the configuration is correct, you will get the message All connection tests passed!
| + + | +
+
+
+You might get a configuration check failed error. Make sure your Teradata Vantage instance is running properly before making a connection through Airbyte. + |
+
A namespace is a group of streams (tables) in a source or destination. A schema in a relational database system is an example of a namespace. In a source, the namespace is the location from where the data is replicated to the destination. In a destination, the namespace is the location where the replicated data is stored in the destination. +For more details please refer to Airbyte Namespace.
+
+In our example the destination is a database, so the namespace is the default schema gsheet_airbyte_td we defined when we configured the destination. The stream name is a table that is mirroring the name of the spreadsheet in the source, which is sample_employee_payrate in this case. Since we are using the single spreadsheet connector, it only supports one stream (the active spreadsheet).
Other type of sources and destinations might have a different layout. In this example, Google sheets, as source, does not support a namespace.
+In our example, we have used <destination schema> as the Namespace of the destination, this is the default namespace assigned by Airbyte based on the Default Schema we declared in the destination settings. The database gsheet_airbyte_td will be created in our Teradata Vantage Instance.
| + + | +
+
+
+We use the term "schema", as it is the term used by Airbyte. In a Teradata context the term "database" is the equivalent. + |
+
It shows how often data should sync to destination. You can select every hour, 2 hours, 3 hours etc. In our case we used every 24 hours.
+
+You can also use a Cron expression to specify the time when the sync should run. In the example below, we set the Cron expression to run the sync on every Wednesday at 12:43 PM (US/Pacific) time.
+
+Airbyte tracks synchronization attempts in the "Sync History" section of the Status tab.
+Next, you can go to the ClearScape Analytics Experience and run a Jupyter notebook, notebooks in ClearScape Analytics Experience are configured to run Teradata SQL queries, to verify if the database gsheet_airbyte_td, streams (tables) and complete data is present.
+%connect localSELECT DatabaseName, TableName, CreateTimeStamp, LastAlterTimeStamp
+FROM DBC.TablesV
+WHERE DatabaseName = 'gsheet_airbyte_td'
+ORDER BY TableName;DATABASE gsheet_airbyte_td;SELECT * FROM _airbyte_raw_sample_employee_payrate;The stream (table) name in destination is prefixed with _airbyte_raw_ because Normalization and Transformation are not supported for this connection, and we only have the raw table. Each stream (table) contains 3 columns:
_airbyte_ab_id: a uuid assigned by Airbyte to each event that is processed. The column type in Teradata is VARCHAR(256).
_airbyte_emitted_at: a timestamp representing when the event was pulled from the data source. The column type in Teradata is TIMESTAMP(6).
_airbyte_data: a json blob representing the event data. The column type in Teradata is JSON.
Here in the _airbyte_data column, we see 9 rows, the same as we have in the source Google sheet, and the data is in JSON format which can be transformed further as needed.
You can close the connection in Airbyte by disabling the connection. This will stop the data sync process.
+
+You can also delete the connection.
+
+This tutorial demonstrated how to extract data from a source system like Google sheets and use the Airbyte ELT tool to load the data into the Teradata Vantage Instance. We saw the end-to-end data flow and complete configuration steps for running Airbyte Open Source locally, and configuring the source and destination connections. We also discussed about the available data sync configurations based on replication frequency. We validated the results in the destination using Cloudscape Analytics Experience and finally we saw the methods to pause and delete the Airbyte connection.
+Author: Obed Vega
+Last updated: May 1st, 2023
The explain plan is a feature of the sql engine it can help you understand how the Teradata Optimizer plans to execute a specific SQL query. +It provides a detailed explanation of how a query is going to be executed. It shows the steps that the optimizer will take to retrieve the requested data +from the database. +It also shows the indexes that will be used, if any, as well as any sorting or aggregation that will take place.
+You need access to a Teradata Vantage instance. The explain plan feature is enabled in all Vantage editions from Vantage Express through Developer, DYI to Vantage as a Service starting from version 17.10.
+| + + | ++If you need a new instance of Vantage, you can install a free version called Vantage Express in the cloud on Google Cloud, Azure, and AWS. You can also run Vantage Express on your local machine using VMware, VirtualBox, or UTM. + | +
WRITE_NOS allows you to extract selected or all columns from a database table or from derived results and write to external object storage, such as Amazon S3, Azure Blob storage, Azure Data Lake Storage Gen2, and Google Cloud Storage. This functionality stores data in Parquet format.
You can find more documentation about Explain Plam functionality in the Teradata documentation.
You will need access to a database where you can execute WRITE_NOS function. If you don’t have such a database, run the following commands:
CREATE USER db AS PERM=10e7, PASSWORD=db;
+
+-- Don't forget to give the proper access rights
+GRANT EXECUTE FUNCTION on TD_SYSFNLIB.READ_NOS to db;
+GRANT EXECUTE FUNCTION on TD_SYSFNLIB.WRITE_NOS to db;| + + | ++If you would like to learn more about setting up users and their privileges, checkout the NOS documentation. + | +
Let’s first create a table on your Teradata Vantage instance:
+CREATE SET TABLE db.parquet_table ,FALLBACK ,
+ NO BEFORE JOURNAL,
+ NO AFTER JOURNAL,
+ CHECKSUM = DEFAULT,
+ DEFAULT MERGEBLOCKRATIO,
+ MAP = TD_MAP1
+ (
+ column1 SMALLINT NOT NULL,
+ column2 DATE FORMAT 'YY/MM/DD' NOT NULL,
+ column3 DECIMAL(10,2))
+PRIMARY INDEX ( column1 );Populate your table with example data:
+INSERT INTO db.parquet_table (1,'2022/01/01',1.1);
+INSERT INTO db.parquet_table (2,'2022/01/02',2.2);
+INSERT INTO db.parquet_table (3,'2022/01/03',3.3);Your table should now look like this:
+column1 column2 column3 +------- -------- ------------ + 1 22/01/01 1.10 + 2 22/01/02 2.20 + 3 22/01/03 3.30+
Create the parquet file with WRITE_NOS. Don’t forget to replace <BUCKET_NAME> with the name of your s3 bucket. Also,replace <YOUR-ACCESS-KEY-ID> and <YOUR-SECRET-ACCESS-KEY> with your access key and secret.
| + + | ++Check your cloud provider docs how to create credentials to access object storage. For example, for AWS check out How do I create an AWS access key? + | +
SELECT * FROM WRITE_NOS (
+ON ( SELECT * FROM db.parquet_table)
+USING
+LOCATION('/s3/<BUCKET_NAME>.s3.amazonaws.com/parquet_file_on_NOS.parquet')
+AUTHORIZATION('{"ACCESS_ID":"<YOUR-ACCESS-KEY-ID>",
+"ACCESS_KEY":"<YOUR-SECRET-ACCESS-KEY>"}')
+STOREDAS('PARQUET')
+MAXOBJECTSIZE('16MB')
+COMPRESSION('SNAPPY')
+INCLUDE_ORDERING('TRUE')
+INCLUDE_HASHBY('TRUE')
+) as d;Now you have created a parquet file in your object storage bucket. Now to easily query your file you need to follow step number 4.
+Create a NOS-backed foreign table. Don’t forget to replace <BUCKET_NAME> with the name of your s3 bucket. Also,replace <YOUR-ACCESS-KEY-ID> and <YOUR-SECRET-ACCESS-KEY> with your access key and secret:
CREATE MULTISET FOREIGN TABLE db.parquet_table_to_read_file_on_NOS
+, EXTERNAL SECURITY DEFINER TRUSTED CEPH_AUTH,
+MAP = TD_MAP1
+(
+ Location VARCHAR(2048) CHARACTER SET UNICODE CASESPECIFIC
+ , col1 SMALLINT
+ , col2 DATE
+ , col3 DECIMAL(10,2)
+
+)
+USING (
+ LOCATION ('/s3/<BUCKET_NAME>.s3.amazonaws.com/parquet_file_on_NOS.parquet')
+ AUTHORIZATION('{"ACCESS_ID":"<YOUR-ACCESS-KEY-ID>",
+ "ACCESS_KEY":"<YOUR-SECRET-ACCESS-KEY>"}')
+ STOREDAS ('PARQUET')
+)NO PRIMARY INDEX;Now you are ready to Query your parquet file on NOS, let’s try the following query:
+SELECT col1, col2, col3 FROM db.parquet_table_to_read_file_on_NOS;The data returned from the query should look something like this:
+col1 col2 col3 +------ -------- ------------ + 1 22/01/01 1.10 + 2 22/01/02 2.20 + 3 22/01/03 3.30+
In this tutorial we have learned how to export data from Vantage to a parquet file on object storage using Native Object Storage (NOS). NOS supports reading and importing data stored in CSV, JSON and Parquet formats. NOS can also export data from Vantage to object storage.
+Author: Adam Tworkiewicz
+Last updated: April 6th, 2022
| + + | +
+ Deprecation notice
+
+
+This how-to describes |
+
We often have a need to move large volumes of data into Vantage. Teradata offers Fastload utility that can efficiently load large amounts of data into Teradata Vantage. This how-to demonstrates how to use Fastload. In this scenario, we will load over 300k records, over 40MB of data, in a couple of seconds.
Access to a Teradata Vantage instance.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
Download Teradata Tools and Utilities (TTU) - supported platforms: Windows, MacOS, Linux (requires registration).
+We will be working with the US tax fillings for nonprofit organizations. Nonprofit tax filings are public data. The US Internal Revenue Service publishes them in S3 bucket. Let’s grab a summary of filings for 2020: https://s3.amazonaws.com/irs-form-990/index_2020.csv. You can use your browser, wget or curl to save the file locally.
Let’s create a database in Vantage. Use your favorite SQL tool to run the following query:
+CREATE DATABASE irs
+AS PERMANENT = 120e6, -- 120MB
+ SPOOL = 120e6; -- 120MBWe will now run Fastload. Fastload is a command-line tool that is very efficient in uploading large amounts of data into Vantage. Fastload, in order to be fast, has several restrictions in place. It can only populate empty tables, no inserts to already populated tables are supported. It doesn’t support tables with secondary indices. Also, it won’t insert duplicate records, even if a table is a MULTISET table. For the full list of restrictions check out Teradata® Fastload Reference.
Fastload has its own scripting language. The language allows you to prepare the database with arbitrary SQL commands, declare the input source and define how the data should be inserted into Vantage. The tool supports both interactive and batch mode. In this section, we are going to use the interactive mode.
+Let’s start Fastload in the interactive mode:
fastloadFirst, let’s log in to a Vantage database. I’ve a Vantage Express running locally, so I’ll use localhost as the hostname and dbc for username and password:
LOGON localhost/dbc,dbc;Now, that we are logged in, I’m going to prepare the database. I’m switching to irs database and making sure that the target table irs_returns and error tables (more about error tables later) do not exist:
DATABASE irs;
+DROP TABLE irs_returns;
+DROP TABLE irs_returns_err1;
+DROP TABLE irs_returns_err2;I’ll now create an empty table that can hold the data elements from the csv file.
+CREATE MULTISET TABLE irs_returns (
+ return_id INT,
+ filing_type VARCHAR(5) CHARACTER SET LATIN NOT CASESPECIFIC,
+ ein INT,
+ tax_period INT,
+ sub_date VARCHAR(100) CHARACTER SET LATIN NOT CASESPECIFIC,
+ taxpayer_name VARCHAR(100) CHARACTER SET LATIN NOT CASESPECIFIC,
+ return_type VARCHAR(5) CHARACTER SET LATIN NOT CASESPECIFIC,
+ dln BIGINT,
+ object_id BIGINT
+)
+PRIMARY INDEX ( return_id );Now, that the target table has been prepared, we can start loading the data. ERRORFILES directive defines error files. The error files are really tables that Fastload creates. The first table contains information about data conversion and other issues. The second table keeps track of data uniqueness issues, e.g. primary key violations.
BEGIN LOADING irs_returns
+ ERRORFILES irs_returns_err1, irs_returns_err2;We instruct Fastload to save a checkpoint every 10k rows. It’s useful in case we have to restart our job. It will be able to resume from the last checkpoint.
CHECKPOINT 10000;We also need to tell Fastload to skip the first row in the CSV file as start at record 2. That’s because the first row contains column headers.
RECORD 2;Fastload also needs to know that it’s a comma-separated file:
SET RECORD VARTEXT ",";DEFINE block specifies what columns we should expect:
DEFINE in_return_id (VARCHAR(19)),
+ in_filing_type (VARCHAR(5)),
+ in_ein (VARCHAR(19)),
+ in_tax_period (VARCHAR(19)),
+ in_sub_date (VARCHAR(22)),
+ in_taxpayer_name (VARCHAR(100)),
+ in_return_type (VARCHAR(5)),
+ in_dln (VARCHAR(19)),
+ in_object_id (VARCHAR(19)),DEFINE block also has FILE attribute that points to the file with the data. Replace FILE = /tmp/index_2020.csv; with your location of index_2020.csv file:
FILE = /tmp/index_2020.csv;Finally, we define the INSERT statement that will put data into the database and we close off LOADING block:
INSERT INTO irs_returns (
+ return_id,
+ filing_type,
+ ein,
+ tax_period,
+ sub_date,
+ taxpayer_name,
+ return_type,
+ dln,
+ object_id
+ ) VALUES (
+ :in_return_id,
+ :in_filing_type,
+ :in_ein,
+ :in_tax_period,
+ :in_sub_date,
+ :in_taxpayer_name,
+ :in_return_type,
+ :in_dln,
+ :in_object_id
+ );
+END LOADING;Once the job has finished, we are logging off from the database to clean things up.
+LOGOFF;Here is what the entire script looks like:
+LOGON localhost/dbc,dbc;
+
+DATABASE irs;
+DROP TABLE irs_returns;
+DROP TABLE irs_returns_err1;
+DROP TABLE irs_returns_err2;
+
+CREATE MULTISET TABLE irs_returns (
+ return_id INT,
+ filing_type VARCHAR(5) CHARACTER SET LATIN NOT CASESPECIFIC,
+ ein INT,
+ tax_period INT,
+ sub_date VARCHAR(100) CHARACTER SET LATIN NOT CASESPECIFIC,
+ taxpayer_name VARCHAR(100) CHARACTER SET LATIN NOT CASESPECIFIC,
+ return_type VARCHAR(5) CHARACTER SET LATIN NOT CASESPECIFIC,
+ dln BIGINT,
+ object_id BIGINT
+)
+PRIMARY INDEX ( return_id );
+
+BEGIN LOADING irs_returns
+ ERRORFILES irs_returns_err1, irs_returns_err2;
+ CHECKPOINT 10000;
+ RECORD 2;
+ SET RECORD VARTEXT ",";
+
+ DEFINE in_return_id (VARCHAR(19)),
+ in_filing_type (VARCHAR(5)),
+ in_ein (VARCHAR(19)),
+ in_tax_period (VARCHAR(19)),
+ in_sub_date (VARCHAR(22)),
+ in_taxpayer_name (VARCHAR(100)),
+ in_return_type (VARCHAR(5)),
+ in_dln (VARCHAR(19)),
+ in_object_id (VARCHAR(19)),
+ FILE = /tmp/index_2020.csv;
+
+ INSERT INTO irs_returns (
+ return_id,
+ filing_type,
+ ein,
+ tax_period,
+ sub_date,
+ taxpayer_name,
+ return_type,
+ dln,
+ object_id
+ ) VALUES (
+ :in_return_id,
+ :in_filing_type,
+ :in_ein,
+ :in_tax_period,
+ :in_sub_date,
+ :in_taxpayer_name,
+ :in_return_type,
+ :in_dln,
+ :in_object_id
+ );
+END LOADING;
+
+LOGOFF;To run our example in batch mode, simply save all instructions in a single file and run:
+fastload < file_with_instruction.fastloadFastload vs. NOSIn our case, the file is in an S3 bucket. That means, that we can use Native Object Storage (NOS) to ingest the data:
+-- create an S3-backed foreign table
+CREATE FOREIGN TABLE irs_returns_nos
+ USING ( LOCATION('/s3/s3.amazonaws.com/irs-form-990/index_2020.csv') );
+
+-- load the data into a native table
+CREATE MULTISET TABLE irs_returns_nos_native
+ (RETURN_ID, FILING_TYPE, EIN, TAX_PERIOD, SUB_DATE, TAXPAYER_NAME)
+AS (
+ SELECT RETURN_ID, FILING_TYPE, EIN, TAX_PERIOD, SUB_DATE, TAXPAYER_NAME FROM irs_returns_nos
+) WITH DATA
+NO PRIMARY INDEX;The NOS solution is convenient as it doesn’t depend on additional tools. It can be implemented using only SQL. It performs well, especially for Vantage deployments with a high number of AMPs as NOS tasks are delegated to AMPs and run in parallel. Also, splitting the data in object storage into multiple files may further improve performance.
+This how-to demonstrated how to ingest large amounts of data into Vantage. We loaded hundreds of thousands or records into Vantage in a couple of seconds using Fastload.
| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Rémi Turpaud
+Last updated: Feb ember 14th, 2022
This post demonstrates how you can leverage any geographic dataset in GeoJson format and use it for geospatial analytics in Teradata Vantage, with just a few lines of code.
+Today we be gathering reference geographical data (official maps, points of interest, etc…) form public sources and use it to support our day to day analytics.
+You will learn two methods to get your GeoJson data into Teradata Vantage:
+Load it as a single document and use native ClearScape analytics functions to parse it into a table usable for analytics.
+Lightly transform it in native Python as we load it into Vantage to produce an analytics ready dataset.
+The first method is a straig forward ELT pattern for semi-structured format processing in Vantage with a single SQL statement, the second one involves some lightweight preparation in (pure) Python and may allow more flexibility (for example to add early quality checks or optimize the load of large documents).
+You will need:
+Access to a Teradata Vantage instance.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
A Python 3 interpreter
+A SQL Client
+Here we will load a GeoJson document as a single Character Large OBject (CLOB) into the Vantage Data Store and use a single SQL statement, backed by ClearScape Analytics native functions, to parse this document into a usable structure for geospatial analytics.
+The http://geojson.xyz/ website is a fantastic source for open geographical data in GeoJson format. We will load the "Populated Places" dataset that provides with a list of over 1000 significant world cities.
+Open you favourite Python 3 interpreter and make sure you have the following packages installed:
+wget
+teradatasql
+getpass
+Download and read the cities dataset:
+import wget
+world_cities=wget.download('https://d2ad6b4ur7yvpq.cloudfront.net/naturalearth-3.3.0/ne_50m_populated_places.geojson')
+with open(world_cities) as geo_json:
+ jmap = jmap = geo_json.read()Modify this code as needed with your Vantage host name, user name and specify an advanced login mechanism if any (eg. LDAP, Kerberos…) with the logmech parameter if you need to. +All the connection parameters are documented on the teradatasql PyPi page there: https://pypi.org/project/teradatasql/
+The code below simply creates a Vantage connection, and opens a cursor creating a table and loading it with our file.
+import teradatasql
+import getpass
+tdhost='<Your-Vantage-System-HostName-Here>'
+tdUser='<Your-Vantage-User-Name-Here>'
+
+# Create a connection to Teradata Vantage
+con = teradatasql.connect(None, host=tdhost, user=tdUser, password=getpass.getpass())
+
+# Create a table named geojson_src and load the JSON map into it as a single CLOB
+with con.cursor () as cur:
+ cur.execute ("create table geojson_src (geojson_nm VARCHAR(32), geojson_clob CLOB CHARACTER SET UNICODE);")
+ r=cur.execute ("insert into geojson_src (?, ?)", ['cities',jmap])Now open your favourite SQL client and connect to your Vantage system.
+We will use ClearScape analytics JSON functions to parse our GeoJson document and extract the most relevant properties and the geometry itself (the coordinates of the city) for each feature (each feature representing a city in this example). +We then use the GeomFromGeoJSON function to cast our geometry as a native Vantage geometry data type (ST_GEOMETRY).
+For user convenience, will wrap all this SQL code in a view:
+REPLACE VIEW cities_geo AS
+SEL city_name, country_name, region_name, code_country_isoa3, GeomFromGeoJSON(geom, 4326) city_coord
+FROM JSON_Table
+(ON (
+ SEL
+ geojson_nm id
+ ,cast(geojson_clob as JSON) jsonCol
+ FROM geojson_src where geojson_nm='cities'
+)
+USING rowexpr('$.features[*]')
+ colexpr('[ {"jsonpath" : "$.geometry",
+ "type" : "VARCHAR(32000)"},
+ {"jsonpath" : "$.properties.NAME",
+ "type" : "VARCHAR(50)"},
+ {"jsonpath" : "$.properties.SOV0NAME",
+ "type" : "VARCHAR(50)"},
+ {"jsonpath" : "$.properties.ADM1NAME",
+ "type" : "VARCHAR(50)"},
+ {"jsonpath" : "$.properties.SOV_A3",
+ "type" : "VARCHAR(50)"}]')
+) AS JT(id, geom, city_name, country_name, region_name, code_country_isoa3);That’s all, you can now view the prepared geometry data as a table, ready to enrich your analytics:
+SEL TOP 5 * FROM cities_geo;Result:
+| city_name | +country_name | +region_name | +code_country_isoa3 | +city_coord | +
|---|---|---|---|---|
Potenza |
+Italy |
+Basilicata |
+ITA |
+POINT (15.798996495640267 40.642002130098206) |
+
Mariehamn |
+Finland |
+Finström |
+ALD |
+POINT (19.949004471869102 60.096996184895431) |
+
Ramallah |
+Indeterminate |
++ | PSE |
+POINT (35.206209378189556 31.902944751424059) |
+
Poitier |
+French Republic |
+Poitou-Charentes |
+FRA |
+POINT (0.333276528534554 46.583292255736581) |
+
Clermont-Ferrand |
+French Republic |
+Auvergne |
+FRA |
+POINT (3.080008095928406 45.779982115759424) |
+
Calculate the distance between two cities:
+SEL b.city_coord.ST_SphericalDistance(l.city_coord)
+FROM
+(SEL city_coord FROM cities_geo WHERE city_name='Bordeaux') b
+CROSS JOIN (SEL city_coord FROM cities_geo WHERE city_name='Lvov') lResult:
+city_coord.ST_SPHERICALDISTANCE(city_coord) |
+
1.9265006861079421e+06 |
+
The previous example demonstrated how to load a complete document as a large object into Teradata Vantage and use built in analytic functions to parse it into a usable dataset.
+This is convenient but limited: we need to parse this document every time we need to use it, as the original document is not directly usable for analytics, JSON documents are currently limited to 16MB in Vantage and it may be inconvenient to fix data quality or formatting issues within the document stored as a CLOB.
+In this example, we will parse our JSON document using the Python json package and load it as a table that can be used directly and efficiently for analysis.
+Python json and list manipulation functions, along with the Teradata SQL driver for Python make this process really simple and efficient.
+For this example, we will use the boundaries of the world countries available on https://datahub.io.
+Let’s get into it.
+Open you favourite Python 3 interpreter and make sure you have the following packages installed:
+wget
+teradatasql
+getpass
+import wget
+countries_geojson=wget.download('https://datahub.io/core/geo-countries/r/countries.geojson')import json +with open(countries_geojson) as geo_json: + countries_json = json.load(geo_json)
+The good thing about loading this JSON in memory, if you are using an interactive Python terminal, is that you can now explore the document to understand its structure. For example
+print(countries_json.keys())
+print(countries_json['type'])
+print(countries_json['features'][0]['properties'].keys())
+print(countries_json['features'][0]['geometry']['coordinates'])What we have here is a collection of GeoFeatures (as earlier).
+We will now lightly model this data in a Vantage table, for that:
+We will load each feature as a raw.
+We will extract the properties that look interesting for immediate analysis (in our example, the country name and ISO code).
+We will extract the geometry itself and load it as a separate column.
+To load a set of rows with a teradatasql cursor, we need to represent each row as an array (or tuples) of values, and the complete dataset as an array of all the row-arrays. +This is fairly easy to do as a list comprehension
+For example:
+[(f['properties']['ADMIN'], f['properties']['ISO_A3'], f['geometry']) for f in countries_json['features'][:1]]NB: Not featured here, but recommended for richer datasets, consider loading the entire and original feature payload as a separate column (this is a JSON document). This will allow you to go back to the original record and extract new properties that you may have missed during your first analysis but have become relevant, directly in SQL and without having to reload the file entirely.
+Modify this code as needed with your Vantage host name, user name and specify an advanced login mechanism if any (eg. LDAP, Kerberos…) with the logmech parameter if you need to. +All the connection parameters are documented on the teradatasql PyPi page there: https://pypi.org/project/teradatasql/
+The code below simply creates a Vantage connection, and opens a cursor creating a table and loading it with our list.
+import teradatasql
+import getpass
+tdhost='<Your-Vantage-System-HostName-Here>'
+tdUser='<Your-Vantage-User-Name-Here>'
+
+# Create a connection to Teradata Vantage
+con = teradatasql.connect(None, host=tdhost, user=tdUser, password=tdPassword)
+
+# Create a table and load our country names, codes, and geometries.
+with con.cursor () as cur:
+ cur.execute ("create table stg_countries_map (country_nm VARCHAR(32), ISO_A3_cd VARCHAR(32), boundaries_geo CLOB CHARACTER SET UNICODE);")
+ r=cur.execute ("insert into stg_countries_map (?, ?, ?)", [(f['properties']['ADMIN'], f['properties']['ISO_A3'], str(f['geometry'])) for f in countries_json['features']])The code below performs the table creation from the Python interpreter, you can also run the sql statement defined below in your prefered SQL client you might as well simply define this logic as a SQL view to avoid having to refresh this table.
+We will use ClearScape analytics the GeomFromGeoJSON function to cast our geometry as a native Vantage geometry data type (ST_GEOMETRY).
+# Now create our final reference table, casting the geometry CLOB as a ST_GEOMETRY object
+sql='''
+CREATE TABLE ref_countries_map AS
+(
+SEL
+ISO_A3_cd
+,country_nm
+,GeomFromGeoJSON(boundaries_geo, 4326) boundaries_geo
+FROM stg_countries_map
+) WITH DATA
+'''
+
+WITH con.cursor () AS cur:
+ cur.execute (sql)That’s all, you may now query your tables using your favourite SQL client and Teradata’s excellent Geospatial data types and analytic functions.
+For example, using the two datasets we have loaded during this tutorial, check in what countries are
+SEL cty.city_name, cty.city_coord, ctry.country_nm
+FROM cities_geo cty
+LEFT JOIN ref_countries_map ctry
+ ON ctry.boundaries_geo.ST_Contains(cty.city_coord)=1
+WHERE cty.city_name LIKE 'a%'city_name |
+city_coord |
+country_nm |
+
Acapulco |
+POINT (-99.915979046410712 16.849990864016206) |
+Mexico +Aosta |
+
POINT (7.315002595706176 45.737001067072299) |
+Italy +Ancona |
+POINT (13.499940550397127 43.600373554552903) |
+
Italy +Albany |
+POINT (117.891604776075155 -35.016946595501224) |
+Australia |
+
Note that none of the above code does not implement any control procedure or checks to, for example, manage the state of the target tables, manage locking, control error codes, etc… This is meant to be a demonstrations of the available features to acquire and use geospatial reference data.
+Consider using SQLAlchemy ORM if you are defining your pipeline in Python, dbt, or your favorite ELT and orchestration toolset to create your products you can operationalize.
+You now can know how to get any open geographic dataset and use it to augment your analytics with Teradata Vantage!
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Adam Tworkiewicz
+Last updated: January 9th, 2023
This how-to shows how to gain access to a Teradata database by running it on your local machine. There are many ways to install Teradata. This document optimizes for the lowest time to first query without spending money on cloud resources. Once you finish the steps you will have a working Teradata Vantage Express database on your computer.
+| + + | ++Starting with version 17.20, Vantage Express includes the following analytics packages: Vantage Analytics Library, Bring Your Own Model (BYOM), API Integration with AWS SageMaker. + | +
A Mac computer. Both Intel and M1/2 chips are supported.
+| + + | ++Vantage Express runs on x86 architecture. When you run the VM on M1/2 chips, UTM has to emulate x86. This is significantly slower then virtualization. If you determine that Vantage Express on M1/M2 is too slow for your needs, consider running Vantage Express in the cloud: AWS, Azure, Google Cloud. + | +
30GB of disk space and enough CPU and RAM to be able to dedicate at least one core and 4GB RAM to the virtual machine.
+Admin rights to be able to install and run the software.
+| + + | ++No admin rights on your local machine? Have a look at how to run Vantage Express in AWS, Azure, Google Cloud. + | +
The latest version of Vantage Express. If you have not used the Teradata downloads website before, you will need to register.
+The latest version of UTM.
+Install UTM by running the installer and accepting the default values.
+Go to the directory where you downloaded Vantage Express and unzip the downloaded file.
+Start UTM, click on the + sign and select Virtualize (for Intel Macs) or Emulate (for M1 Macs).
On Operating System screen select Other.
On Other screen select Skip ISO Boot.
On Hardware screen allocate at least 4GB of memory and at least 1 CPU core. We recommend 10GB RAM and 2 CPUs.
+On Storage screen accept the defaults by clicking Next.
On Shared Direct screen click Next.
On Summary screen check Open VM Settings and click Save.
Go through the setup wizard. You only need to adjust the following tabs:
+QEMU - disable UEFI Boot option
Network - expose ssh (22) and Vantage (1025) ports on the host computer:
+
+Map drives:
+Delete the default IDE Drive.
Map the 3 Vantage Express drives by importing the disk files from the downloaded VM zip file. Make sure you map them in the right order, -disk1, -disk2, -disk3 . The first disk is bootable and contains the database itself. Disks 2 and 3 are so called pdisks and contain data. As you import the files UTM will automatically convert them fro vmdk into qcow2 format. Make sure that each disk is configured using the IDE interface:
+Once you are done mapping all 3 drives, your configuration should look like this:
+
+Save the configuration and start the VM.
+Press ENTER to select the highlighted LINUX boot partition.
+On the next screen, press ENTER again to select the default SUSE Linux kernel.
+
+After completing the bootup sequence a terminal login prompt as shown in the screenshot below will appear. Don’t enter anything in the terminal. Wait till the system starts the GUI.
+
+After a while the following prompt will appear - assuming that you did not enter anything after the command login prompt above. Press okay button in the screen below.
+Once the VM is up, you will see its desktop environment. When prompted for username/password enter root for both.
+The database is configured to autostart with the VM. To confirm that the database has started go to the virtual desktop and start Gnome Terminal.
+In the terminal execute pdestate command that will inform you if Vantage has already started:
| + + | ++To paste into Gnome Terminal press SHIFT+CTRL+V. + | +
watch pdestate -aYou want to wait till you see the following message:
+PDE state is RUN/STARTED. +DBS state is 5: Logons are enabled - The system is quiescent+
pdestate returns when the database is still initializing.PDE state is DOWN/HARDSTOP. + +PDE state is START/NETCONFIG. + +PDE state is START/GDOSYNC. + +PDE state is START/TVSASTART. + +PDE state is START/READY. +PDE state is RUN/STARTED. + +DBS state is 1/1: DBS Startup - Initializing DBS Vprocs +PDE state is RUN/STARTED. + +DBS state is 1/5: DBS Startup - Voting for Transaction Recovery +PDE state is RUN/STARTED. + +DBS state is 1/4: DBS Startup - Starting PE Partitions +PDE state is RUN/STARTED.+
Now that the database is up, go back to the virtual desktop and launch Teradata Studio Express.
+When you first start it you will be offered a tour. Once you close the tour, you will see a wizard window to add a new connection. Select Teradata:
+On the next screen, connect to the database on your localhost using dbc for the username and password:
+We will now run some queries in the VM. To avoid copy/paste issues between the host and the VM, we will open this quick start in the VM. Go to the virtual desktop, start Firefox and point it to this quick start.
+Once in Teradata Studio Express, go to Query Development perspective (go to the top menu and select Window → Query Development).
Connect using the previously created connection profile by double-clicking on Database Connections → New Teradata.
Using dbc user, we will create a new database called HR. Copy/paste this query and run it by hitting the run query button (
) or pressing F5 key:
CREATE DATABASE HR
+AS PERMANENT = 60e6, -- 60MB
+ SPOOL = 120e6; -- 120MBLet’s create a sample table and insert some data and query it. We will first create a table to hold employee information:
+CREATE SET TABLE HR.Employees (
+ GlobalID INTEGER,
+ FirstName VARCHAR(30),
+ LastName VARCHAR(30),
+ DateOfBirth DATE FORMAT 'YYYY-MM-DD',
+ JoinedDate DATE FORMAT 'YYYY-MM-DD',
+ DepartmentCode BYTEINT
+)
+UNIQUE PRIMARY INDEX ( GlobalID );Now, let’s insert a record:
+INSERT INTO HR.Employees (
+ GlobalID,
+ FirstName,
+ LastName,
+ DateOfBirth,
+ JoinedDate,
+ DepartmentCode
+)
+VALUES (
+ 101,
+ 'Adam',
+ 'Tworkowski',
+ '1980-01-05',
+ '2004-08-01',
+ 01
+);Finally, let’s see if we can retrieve the data:
+SELECT * FROM HR.Employees;You should get the following results:
+GlobalID FirstName LastName DateOfBirth JoinedDate DepartmentCode +-------- --------- ---------- ----------- ---------- -------------- + 101 Adam Tworkowski 1980-01-05 2004-08-01 1+
In this guide we have covered how to quickly create a working Teradata environment. We used Teradata Vantage Express in a VM running on VMware. In the same VM, we ran Teradata Studio Express to issue queries. We installed all software locally and didn’t have to pay for cloud resources.
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Adam Tworkiewicz
+Last updated: January 9th, 2023
This how-to shows how to gain access to a Teradata database by running it on your local machine. There are many ways to install Teradata. This document optimizes for the lowest time to first query without spending money on cloud resources. Once you finish the steps you will have a working Teradata Vantage Express database on your computer.
+| + + | ++Starting with version 17.20, Vantage Express includes the following analytics packages: Vantage Analytics Library, Bring Your Own Model (BYOM), API Integration with AWS SageMaker. + | +
A computer using one of the following operating systems: Windows 10, Linux or Intel-based MacOS.
+| + + | ++For M1/M2 MacOS systems, see Run Vantage Express on UTM. + | +
30GB of disk space and enough CPU and RAM to be able to dedicate at least one core and 6GB RAM to the virtual machine.
+Admin rights to be able to install and run the software.
+The latest version of Vantage Express VirtualBox Open Virtual Appliance (OVA).
+| + + | ++If you have not used the Teradata Downloads website before, you will need to register first. + | +
VirtualBox, version 6.1.
+| + + | +
+You can also install VirtualBox using brew and other package managers.
+ |
+
Install VirtualBox by running the installer and accepting the default values.
+| + + | ++VirtualBox includes functionality that requires elevated privileges. When you start VirtualBox for the first time, you will be asked to confirm this elevated access. You may also need to reboot your machine to activate the VirtualBox kernel plugin. + | +
Start VirtualBox.
+Go to File → Import Appliance… menu.
In File field, select the downloaded OVA file.
On the next screen, accept the defaults and click on Import.
Back in the main VirtualBox panel, start the Vantage Express appliance double clicking on VM Vantage 17.20.
+Press ENTER to select the highlighted LINUX boot partition.
+On the next screen, press ENTER again to select the default SUSE Linux kernel.
+
+After completing the bootup sequence a terminal login prompt as shown in the screenshot below will appear. Don’t enter anything in the terminal. Wait till the system starts the GUI.
+
+After a while the following prompt will appear - assuming that you did not enter anything after the command login prompt above. Press okay button in the screen below.
+Once the VM is up, you will see its desktop environment. When prompted for username/password enter root for both.
+The database is configured to autostart with the VM. To confirm that the database has started go to the virtual desktop and start Gnome Terminal.
+In the terminal execute pdestate command that will inform you if Vantage has already started:
| + + | ++To paste into Gnome Terminal press SHIFT+CTRL+V. + | +
watch pdestate -aYou want to wait till you see the following message:
+PDE state is RUN/STARTED. +DBS state is 5: Logons are enabled - The system is quiescent+
pdestate returns when the database is still initializing.PDE state is DOWN/HARDSTOP. + +PDE state is START/NETCONFIG. + +PDE state is START/GDOSYNC. + +PDE state is START/TVSASTART. + +PDE state is START/READY. +PDE state is RUN/STARTED. + +DBS state is 1/1: DBS Startup - Initializing DBS Vprocs +PDE state is RUN/STARTED. + +DBS state is 1/5: DBS Startup - Voting for Transaction Recovery +PDE state is RUN/STARTED. + +DBS state is 1/4: DBS Startup - Starting PE Partitions +PDE state is RUN/STARTED.+
Now that the database is up, go back to the virtual desktop and launch Teradata Studio Express.
+When you first start it you will be offered a tour. Once you close the tour, you will see a wizard window to add a new connection. Select Teradata:
+On the next screen, connect to the database on your localhost using dbc for the username and password:
+Once in Teradata Studio Express, go to Query Development perspective (go to the top menu and select Window → Query Development).
Connect using the previously created connection profile by double-clicking on Database Connections → New Teradata.
Using dbc user, we will create a new database called HR. Copy/paste this query and run it by hitting the run query button () or pressing F5 key:
CREATE DATABASE HR
+AS PERMANENT = 60e6, -- 60MB
+ SPOOL = 120e6; -- 120MBLet’s create a sample table and insert some data and query it. We will first create a table to hold employee information:
+CREATE SET TABLE HR.Employees (
+ GlobalID INTEGER,
+ FirstName VARCHAR(30),
+ LastName VARCHAR(30),
+ DateOfBirth DATE FORMAT 'YYYY-MM-DD',
+ JoinedDate DATE FORMAT 'YYYY-MM-DD',
+ DepartmentCode BYTEINT
+)
+UNIQUE PRIMARY INDEX ( GlobalID );Now, let’s insert a record:
+INSERT INTO HR.Employees (
+ GlobalID,
+ FirstName,
+ LastName,
+ DateOfBirth,
+ JoinedDate,
+ DepartmentCode
+)
+VALUES (
+ 101,
+ 'Adam',
+ 'Tworkowski',
+ '1980-01-05',
+ '2004-08-01',
+ 01
+);Finally, let’s see if we can retrieve the data:
+SELECT * FROM HR.Employees;You should get the following results:
+GlobalID FirstName LastName DateOfBirth JoinedDate DepartmentCode +-------- --------- ---------- ----------- ---------- -------------- + 101 Adam Tworkowski 1980-01-05 2004-08-01 1+
VirtualBox Guest Extensions is a piece of software that runs in a VM. It makes the VM run faster on VirtualBox. It also improves the resolution of the VM screen and its responsiveness to resizing. It implements two-way clipboard, and drag and drop between the host and the guest. VirtualBox Guest Extensions in the VM needs to match the version of your VirtualBox install. You will likely have to update VirtualBox Guest Extensions for optimal performance.
+To update VirtualBox Guest Extensions:
+Insert the VirtualBox Guest Extensions DVD by clicking on SATA Port 3: [Optical Drive] in Storage section:
+Back in the VM window, start the Gnome Terminal application.
Run the following command in the terminal:
+mount /dev/cdrom /media/dvd; /media/dvd/VBoxLinuxAdditions.runIn this guide we have covered how to quickly create a working Teradata environment. We used Teradata Vantage Express in a VM running on VMware. In the same VM, we ran Teradata Studio Express to issue queries. We installed all software locally and didn’t have to pay for cloud resources.
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Adam Tworkiewicz
+Last updated: January 9th, 2023
This how-to shows how to gain access to a Teradata database by running it on your local machine. There are many ways to install Teradata. This document optimizes for the lowest time to first query without spending money on cloud resources. Once you finish the steps you will have a working Teradata Vantage Express database on your computer.
+| + + | ++Starting with version 17.20, Vantage Express includes the following analytics packages: Vantage Analytics Library, Bring Your Own Model (BYOM), API Integration with AWS SageMaker. + | +
A computer using one of the following operating systems: Windows, Linux or Intel-based MacOS.
+| + + | ++For M1/M2 MacOS systems, see Run Vantage Express on UTM. + | +
30GB of disk space and enough CPU and RAM to be able to dedicate at least one core and 6GB RAM to the virtual machine.
+Admin rights to be able to install and run the software.
+The latest version of Vantage Express. If you have not used the Teradata downloads website before, you will need to register.
+| + + | ++Commercial organizations require commercial licenses to use VMware Workstation Player. If you don’t want to acquire VMware licenses you can run Vantage Express on VirtualBox. + | +
| + + | ++VMware doesn’t offer VMware Workstation Player for MacOS. If you are on a Mac, you will need to install VMware Fusion instead. It’s a paid product but VMware offers a free 30-day trial. Alternatively, you can run Vantage Express on VirtualBox or UTM. + | +
On Windows, you will also need 7zip to unzip Vantage Express.
+Install VMware Player or VMware Fusion by running the installer and accepting the default values.
+If on Windows, install 7zip.
Go to the directory where you downloaded Vantage Express and unzip the downloaded file.
+Double-click on the .vmx file. This will start the VM image in VMware Player/Fusion.
Press ENTER to select the highlighted LINUX boot partition.
+On the next screen, press ENTER again to select the default SUSE Linux kernel.
+
+After completing the bootup sequence a terminal login prompt as shown in the screenshot below will appear. Don’t enter anything in the terminal. Wait till the system starts the GUI.
+
+After a while the following prompt will appear - assuming that you did not enter anything after the command login prompt above. Press okay button in the screen below.
+Once the VM is up, you will see its desktop environment. When prompted for username/password enter root for both.
+The database is configured to autostart with the VM. To confirm that the database has started go to the virtual desktop and start Gnome Terminal.
+In the terminal execute pdestate command that will inform you if Vantage has already started:
| + + | ++To paste into Gnome Terminal press SHIFT+CTRL+V. + | +
watch pdestate -aYou want to wait till you see the following message:
+PDE state is RUN/STARTED. +DBS state is 5: Logons are enabled - The system is quiescent+
pdestate returns when the database is still initializing.PDE state is DOWN/HARDSTOP. + +PDE state is START/NETCONFIG. + +PDE state is START/GDOSYNC. + +PDE state is START/TVSASTART. + +PDE state is START/READY. +PDE state is RUN/STARTED. + +DBS state is 1/1: DBS Startup - Initializing DBS Vprocs +PDE state is RUN/STARTED. + +DBS state is 1/5: DBS Startup - Voting for Transaction Recovery +PDE state is RUN/STARTED. + +DBS state is 1/4: DBS Startup - Starting PE Partitions +PDE state is RUN/STARTED.+
Now that the database is up, go back to the virtual desktop and launch Teradata Studio Express.
+When you first start it you will be offered a tour. Once you close the tour, you will see a wizard window to add a new connection. Select Teradata:
+On the next screen, connect to the database on your localhost using dbc for the username and password:
+We will now run some queries in the VM. To avoid copy/paste issues between the host and the VM, we will open this quick start in the VM. Go to the virtual desktop, start Firefox and point it to this quick start.
+Once in Teradata Studio Express, go to Query Development perspective (go to the top menu and select Window → Query Development).
Connect using the previously created connection profile by double-clicking on Database Connections → New Teradata.
Using dbc user, we will create a new database called HR. Copy/paste this query and run it by hitting the run query button () or pressing F5 key:
CREATE DATABASE HR
+AS PERMANENT = 60e6, -- 60MB
+ SPOOL = 120e6; -- 120MBLet’s create a sample table and insert some data and query it. We will first create a table to hold employee information:
+CREATE SET TABLE HR.Employees (
+ GlobalID INTEGER,
+ FirstName VARCHAR(30),
+ LastName VARCHAR(30),
+ DateOfBirth DATE FORMAT 'YYYY-MM-DD',
+ JoinedDate DATE FORMAT 'YYYY-MM-DD',
+ DepartmentCode BYTEINT
+)
+UNIQUE PRIMARY INDEX ( GlobalID );Now, let’s insert a record:
+INSERT INTO HR.Employees (
+ GlobalID,
+ FirstName,
+ LastName,
+ DateOfBirth,
+ JoinedDate,
+ DepartmentCode
+)
+VALUES (
+ 101,
+ 'Adam',
+ 'Tworkowski',
+ '1980-01-05',
+ '2004-08-01',
+ 01
+);Finally, let’s see if we can retrieve the data:
+SELECT * FROM HR.Employees;You should get the following results:
+GlobalID FirstName LastName DateOfBirth JoinedDate DepartmentCode +-------- --------- ---------- ----------- ---------- -------------- + 101 Adam Tworkowski 1980-01-05 2004-08-01 1+
In this guide we have covered how to quickly create a working Teradata environment. We used Teradata Vantage Express in a VM running on VMware. In the same VM, we ran Teradata Studio Express to issue queries. We installed all software locally and didn’t have to pay for cloud resources.
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
| + Didn’t find what you were looking for? + | ++ Contribute or request a topic + | ++ request + contribute + | +
Author: Satish Chinthanippu
+Last updated: August 14th, 2023
This how-to goes through the installation of Teradata Studio and Teradata Studio Express on Apple Mac M1/M2 machines.
+Install and enable Rosetta binary translator. Follow the Apple Mac Rosetta Installation Guide.
+Download and Install a x86 64-bit based JDK 11 from your preferred vendor. For example, you can download x86 64-bit JDK 11 from Azul
+Download the latest Teradata Studio or Teradata Studio Express release from the Teradata Downloads page:
+Install the Teradata Studio/Teradata Studio Express. Refer to Teradata Studio and Teradata Studio Express Installation Guide for details.
+Apple has introduced ARM-based processors in Apple MAC M1/M2 machines. Intel x64-based applications won’t work by default on ARM-based processors. Teradata Studio or Teradata Studio Express also doesn’t work by default as the current Studio macOS build is an intel x64-based application. This how-to demonstrates how to install Intel x64-based JDK and Teradata Studio or Teradata Studio Express on Apple Mac M1/M2.
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Adam Tworkiewicz
+Last updated: November 14th, 2022
This how-to demonstrates how to connect to Teradata Vantage using JDBC using a sample Java application: https://github.com/Teradata/jdbc-sample-app.
+Access to a Teradata Vantage instance.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
JDK
+Maven
+Add the Teradata JDBC driver as a dependency to your Maven POM XML file:
+| + + | +
+This step assumes that your Vantage database is available on localhost on port 1025. If you are running Vantage Express on your laptop, you need to expose the port from the VM to the host machine. Refer to your virtualization software documentation how to forward ports.
+ |
+
The project is set up. All that is left, is to load the driver, pass connection and authentication parameters and run a query:
+This how-to demonstrated how to connect to Teradata Vantage using JDBC. It described a sample Java application with Maven as the build tool that uses the Teradata JDBC driver to send SQL queries to Teradata Vantage.
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Get this notebook in .ipynb format.
+Vertex AI is Google's single environment for data scientists to develop and deploy ML models, from experimentation, to deployment, to managing and monitoring models. In this tutorial, we will show how to integrate Vantage Analytics capabilites in Vertex AI ML Pipelines. We'll create two pipelines:
+Training - the first will be a three step pipeline to train and deploy a model; the first step transforms data in Vantage and then exports a file for training, the second step trains a model using scikit-learn, and the final step deploys the model to Vantage using Bring Your Own Model (BYOM) feature of Teradata Vantage.
+
Scoring - the second pipeline will use the model created by the the first pipeline to score new data stored in a table on Vantage.
+
Both pipelines are very simple, but the first pipeline could be triggered to retrain a model with new data when the production model has drifted. The second pipeline could be run on a regular schedule when new data for scoring was available.
+ +
+ 
+ 
+ 
+ 
+ 
+ import sys
+!{sys.executable} -m pip install --upgrade --force-reinstall ipython-sql
+!{sys.executable} -m pip install teradatasqlalchemy teradataml kaggle ipython-sql kfp
+Follow the Run Vantage Express on Google Cloud how-to to get Vantage setup. Make sure to follow the instructions to open the VM up to the Internet.
+You will need a GCS bucket to store artifacts managed by KubeFlow.
+Define the bucket name:
+ +BUCKET_NAME = "<your-bucket-name>"
+If the bucket doesn't exist, go ahead and create it:
+ +!gsutil ls -b gs://$BUCKET_NAME || gsutil mb gs://$BUCKET_NAME
+Go to IAM tab in GCS console and assign Storage Admin role to your default Compute Engine. The principal of the default Compute Engine account looks like this: <project-id>-compute@developer.gserviceaccount.com.
We'll use the Boston Housing dataset which can be obtained from Kaggle.
+Login to your Kaggle account. In the top right corner click on your user icon and select Account. Find API section and click on Create New API Token. This will produce kaggle.json file. Open kaggle.json and copy the username and key. Substitute the values in the cell and run it:
%env KAGGLE_USERNAME=<your-kaggle-username>
+%env KAGGLE_KEY=<your-kaggle-key>
+!kaggle datasets download -f housing.csv vikrishnan/boston-house-prices
+Let's setup DATABASE_URL environment variable that will point to your instance of Vantage. Make sure that you default to mldb database where BYOM package is installed, e.g.:
DATABASE_URL='teradatasql://dbc:dbc@34.121.78.209/mldb'
+%env DATABASE_URL=$DATABASE_URL
+import pandas
+import os
+
+df=pandas.read_fwf('housing.csv', names=['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'MEDV'])
+df.to_sql('housing', con=DATABASE_URL, index=False)
+For this tutiorial we need a table to store the trained model and another table with some new data that we want to score with our model. Use teradatasql to execute the following SQL on your Vantage instance.
+ +%%sql
+CREATE SET TABLE demo_models (model_id VARCHAR (30), model BLOB) PRIMARY INDEX (model_id);
+CREATE SET TABLE test_housing (ID INTEGER, CRIM FLOAT, ZN FLOAT,INDUS FLOAT,CHAS INTEGER,NOX FLOAT,RM FLOAT,
+ AGE FLOAT,DIS FLOAT, RAD INTEGER,TAX INTEGER,PTRATIO FLOAT,B FLOAT,LSTAT FLOAT) PRIMARY INDEX (CRIM);
+INSERT INTO test_housing (ID, CRIM, ZN, INDUS, CHAS, NOX, RM, AGE, DIS, RAD, TAX, PTRATIO, B, LSTAT)
+ VALUES (1,.02,0.0,7.07,0,.46,6.4,78.9,4.9,2,242,17.8,396.9,9.14);
+Now we are ready to create the components in the pipeline. Vertex AI Pipelines can run pipelines built using the Kubeflow Pipelines SDK or TensorFlow Extended. We'll be using the Kubeflow Pipelines SDK for this simple example using scikit-learn.
+In this example we will create the following three components:
+read_data_from_vantagetrain_modeldeploy_modelFirst, import the Kubeflow Pipeline component and dsl packages.
+ +import kfp.v2.dsl as dsl
+from kfp.v2.dsl import (
+ component,
+ Input,
+ Output,
+ Dataset,
+ Model,
+ Metrics,
+)
+The first component reads data from a Vantage warehouse (see above and make sure you have set up Vantage Express in Google Cloud including opening up a firewall to the VM so you can access Vantage from the Internet.)
+ +The component connects to Vantage using the connection string passed as an input parameter, reads the rows from the table mldb.housing in Vantage and then outputs the data to an Output[Dataset]. The Output is a temporary file used to pass data between components (see more about passing data between components here).
The component uses sqlalchemy to talk to Teradata. Each component is run in a separate container on Kubernetes so all import statements need to be done within the component. We have created a base image with teradatasql already installed. When you pass base_image='python' the component will use that image to create a container. packages_to_install parameter defines what other packages the component needs.
@component(base_image='python', packages_to_install=['teradatasqlalchemy'])
+def read_data_from_vantage(
+ connection_string: str,
+ output_file: Output[Dataset]
+):
+ import sqlalchemy
+
+ file_name = output_file.path
+ engine = sqlalchemy.create_engine(connection_string)
+
+ with engine.connect() as con:
+ rs = con.execute('SELECT * FROM housing')
+ with open(output_file.path, 'w') as output_file:
+ output_file.write('CRIM,ZN,INDUS,CHAS,NOX,RM,AGE,DIS,RAD,TAX,PTRATIO,B,LSTAT,MEDV\n')
+ for row in rs:
+ output_file.write(','.join([str(i) for i in row]) + '\n')
+Next we'll create a component to train a model with the training data.
+The input into this component is the file from the previous component. The output is the file with the trained model using joblib.dump and a file with the test data.
+The component will use scikit-learn and pandas so we need to pass packages_to_install=['pandas==1.3.5','scikit-learn'] - this will tell Kubeflow to install the packages when the container is created.
@component(base_image='teradata/python-sklearn2pmml', packages_to_install=['pandas==1.3.5','scikit-learn','sklearn-pandas==1.5.0'])
+def train_model(
+ input_file : Input[Dataset],
+ output_model: Output[Model],
+ output_metrics: Output[Metrics]
+):
+ import pandas as pd
+ from sklearn.model_selection import train_test_split
+ from sklearn.ensemble import RandomForestRegressor
+ from sklearn.preprocessing import StandardScaler
+ from sklearn import metrics
+ from sklearn_pandas import DataFrameMapper
+ import joblib
+ from sklearn2pmml.pipeline import PMMLPipeline
+ from sklearn2pmml import sklearn2pmml
+
+ df = pd.read_csv(input_file.path)
+
+ train, test = train_test_split(df, test_size = .33)
+ train = train.apply(pd.to_numeric, errors='ignore')
+ test = test.apply(pd.to_numeric, errors='ignore')
+
+ target = 'MEDV'
+ features = train.columns.drop(target)
+
+
+ pipeline = PMMLPipeline([
+ ("mapping", DataFrameMapper([
+ (['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT'], StandardScaler())
+ ])),
+ ("rfc", RandomForestRegressor(n_estimators = 100, random_state = 0))
+ ])
+
+ pipeline.fit(train[features], train[target])
+ y_pred = pipeline.predict(test[features])
+
+ metric_accuracy = metrics.mean_squared_error(y_pred,test[target])
+ output_metrics.log_metric('accuracy', metric_accuracy)
+ output_model.metadata['accuracy'] = metric_accuracy
+
+ joblib.dump(pipeline, output_model.path)
+
+The last component loads the model and tests it on the test data. The Output[Metrics] creates an artifact with the models performance and can be visualize in the Runtime Graph.
@component(base_image='teradata/python-sklearn2pmml')
+def deploy_model(
+ connection_string: str,
+ input_model : Input[Model],
+):
+ import sqlalchemy
+ import teradataml as tdml
+ import joblib
+ from sklearn2pmml.pipeline import PMMLPipeline
+ from sklearn2pmml import sklearn2pmml
+
+ engine = sqlalchemy.create_engine(connection_string)
+ tdml.create_context(tdsqlengine = engine)
+
+ pipeline = joblib.load(input_model.path)
+
+ sklearn2pmml(pipeline, "test_local.pmml", with_repr = True)
+
+ model_id = 'housing_rf'
+ model_file = 'test_local.pmml'
+ table_name = 'demo_models'
+
+ tdml.configure.byom_install_location = "mldb"
+
+ try:
+ res = tdml.save_byom(model_id = model_id, model_file = model_file, table_name = table_name)
+ except Exception as e:
+ # if our model exists, delete and rewrite
+ if str(e.args).find('TDML_2200') >= 1:
+ res = tdml.delete_byom(model_id = model_id, table_name = table_name)
+ res = tdml.save_byom(model_id = model_id, model_file = model_file, table_name = table_name)
+ pass
+ else:
+ raise
+
+Now we'll create a function to execute each component in the pipeline.
+ +@dsl.pipeline(
+ name='run-vantage-pipeline',
+ description='An example pipeline that connects to Vantage.',
+)
+def run_vantage_pipeline_vertex(
+ connection_string: str
+):
+ data_file = read_data_from_vantage(connection_string).output
+ test_model_data = train_model(data_file)
+ deploy_model(connection_string,test_model_data.outputs['output_model'])
+Compile the pipeline. The pipline will be saved in a json file with the name provided as the package_path.
from kfp.v2 import compiler
+compiler.Compiler().compile(pipeline_func=run_vantage_pipeline_vertex,
+ package_path='train_housing_pipeline.json')
+Now use the Vertex AI client to execute the pipeline. Import the google.cloud.aiplatform package. Vertex AI needs a Cloud Storage bucket to for temporary files. Create a new job using the json file above and pass the ipaddr as the parameter. Then submit the job.
When the job starts a link will appear that will take you to the Runtime Graph.
+ +import google.cloud.aiplatform as aip
+
+pipeline_root_path = 'gs://' + BUCKET_NAME
+
+job = aip.PipelineJob(
+ display_name="housing_training_deploy",
+ template_path="train_housing_pipeline.json",
+ pipeline_root=pipeline_root_path,
+ parameter_values={
+ 'connection_string': DATABASE_URL
+ }
+)
+
+job.submit()
+When the pipeline has completed running (each component in the graph should have a green check mark). You can click on each component to see details of the execution and the logs created. If you click on the output_metrics artifact, in the Pipeline run analysis window the Node Info will show the accuracy of the model. Yyou can learn more about other metrics you can pass and visulation using the Metrics artifict here.)
Let's test the model we have just deployed by scoring some new data. We'll use the teradataml driver to retrieve the saved model and score the rows in a table with new data.
import teradataml as tdml
+import sqlalchemy
+import os
+
+engine = sqlalchemy.create_engine(DATABASE_URL)
+eng = tdml.create_context(tdsqlengine = engine)
+
+#indicate the database that BYOM is using
+tdml.configure.byom_install_location = "mldb"
+
+tdf_test = tdml.DataFrame('test_housing')
+
+modeldata = tdml.retrieve_byom("housing_rf", table_name="demo_models")
+
+predictions = tdml.PMMLPredict(
+ modeldata = modeldata,
+ newdata = tdf_test,
+ accumulate = ['ID']
+ )
+predictions.result.to_pandas()
+This pipeline will have only one component that uses the teradatasql driver to execute a SQL query that retrieves the model from the demo_model table and scores the rows in the test_housing table.
@component(base_image='teradata/python-sklearn2pmml', packages_to_install=['pandas==1.3.5','scikit-learn'])
+def score_new_data(
+ connection_string: str,
+ model_name: str,
+ model_table: str,
+ data_table: str,
+ prediction_table: str
+):
+ import teradataml as tdml
+ import sqlalchemy
+
+ engine = sqlalchemy.create_engine(connection_string)
+
+ with engine.connect() as con:
+ con.execute(f'CREATE TABLE {prediction_table} AS (SELECT * FROM mldb.PMMLPredict ( ON {data_table} ON (SELECT * FROM {model_table} where model_id=\'{model_name}\') DIMENSION USING Accumulate (\'ID\')) AS td ) WITH DATA')
+
+The run_new_data_score pipeline takes the following parameters:
model_name: ID of the modelmodel_table: the name of the table storing the modeldata_table: the name of the table with new data to scoreprediction_table: the name of the table to store the scoring resultsWhen the pipeline is executed the dashboard will provide fields to enter the values you want to use.
+ +@dsl.pipeline(
+ name='new-data-pipeline',
+ description='An example of a component that scores new data with a saved model.',
+)
+def run_new_data_score(
+ connection_string: str,
+ model_name: str,
+ model_table: str,
+ data_table: str,
+ prediction_table: str
+):
+ score_new_data(DATABASE_URL,model_name,model_table,data_table,prediction_table)
+To compile the pipeline run the following code. The pipeline will be saved in score_new_data_pipeline_sql.json file.
compiler.Compiler().compile(pipeline_func=run_new_data_score,
+ package_path='score_new_data_pipeline_sql.json')
+We will now execute the pipeline in Vertex AI Pipelines.
+ +import google.cloud.aiplatform as aip
+
+pipeline_root_path = 'gs://' + BUCKET_NAME
+
+job = aip.PipelineJob(
+ display_name="new_data_housing",
+ template_path="score_new_data_pipeline_sql.json",
+ pipeline_root=pipeline_root_path,
+ parameter_values={
+ 'connection_string': DATABASE_URL,
+ 'model_name': 'housing_rf',
+ 'model_table': 'demo_models',
+ 'data_table': 'test_housing',
+ 'prediction_table': 'housing_predictions'
+ }
+)
+
+job.submit()
+Once the job completes, you can view the batch predictions with:
+ +%%sql
+SELECT * FROM housing_predictions;
+To stop incurring charges you need to clean up the following resources:
+Delete.| + Didn’t find a demo you were looking for? + | ++ Contribute or request a demo + | ++ request + contribute + | +
Author: Adam Tworkiewicz
+Last updated: November 10th, 2022
| + + | ++This how-to shows you how to add Teradata Extensions to a Jupyter Notebooks environment. A hosted version of Jupyter Notebooks integrated with Teradata Extensions and analytics tools is available for functional testing for free at https://clearscape.teradata.com. + | +
In this how-to we will go through the steps for connecting to Teradata Vantage from a Jupyter notebook.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
There are a couple of ways to connect to Vantage from a Jupyter Notebook:
+Use python or R libraries in a regular Python/R kernel notebook - this option works well when you are in a restricted environment that doesn’t allow you to spawn your own Docker images. Also, it’s useful in traditional datascience scenarios when you have to mix SQL and Python/R in a notebook. If you are proficient with Jupyter and have your own set of preferred libraries and extensions, start with this option.
+Use the Teradata Jupyter Docker image - the Teradata Jupyter Docker image bundles the Teradata SQL kernel (more on this later), teradataml and tdplyr libraries, python and R drivers. It also contains Jupyter extensions that allow you to manage Teradata connections, explore objects in Vantage database. It’s convenient when you work a lot with SQL or would find a visual Navigator helpful. If you are new to Jupyter or if you prefer to get a currated assembly of libraries and extensions, start with this option.
This option uses a regular Jupyter Lab notebook. We will see how to load the Teradata Python driver and use it from Python code. We will also examine ipython-sql extension that adds support for SQL-only cells.
We start with a plain Jupyter Lab notebook. Here, I’m using docker but any method of starting a notebook, including Jupyter Hub, Google Cloud AI Platform Notebooks, AWS SageMaker Notebooks, Azure ML Notebooks will do.
+docker run --rm -p 8888:8888 -e JUPYTER_ENABLE_LAB=yes \
+ -v "${PWD}":/home/jovyan/work jupyter/datascience-notebookDocker logs will display the url that you need to go to:
+Entered start.sh with args: jupyter lab +Executing the command: jupyter lab +.... +To access the server, open this file in a browser: + file:///home/jovyan/.local/share/jupyter/runtime/jpserver-7-open.html +Or copy and paste one of these URLs: + http://d5c2323ae5db:8888/lab?token=5fb43e674367c6895e8c2404188aa550b5c7bdf96f5b4a3a + or http://127.0.0.1:8888/lab?token=5fb43e674367c6895e8c2404188aa550b5c7bdf96f5b4a3a+
We will open a new notebook and create a cell to install the required libraries:
+| + + | ++I’ve published a notebook with all the cells described below on GitHub: https://github.com/Teradata/quickstarts/blob/main/modules/ROOT/attachments/vantage-with-python-libraries.ipynb + | +
import sys
+!{sys.executable} -m pip install teradatasqlalchemyNow, we will import Pandas and define the connection string to connect to Teradata. Since I’m running my notebook in Docker on my local machine and I want to connect to a local Vantage Express VM, I’m using host.docker.internal DNS name provided by Docker to reference the IP of my machine.
import pandas as pd
+# Define the db connection string. Pandas uses SQLAlchemy connection strings.
+# For Teradata Vantage, it's teradatasql://username:password@host/database_name .
+# See https://pypi.org/project/teradatasqlalchemy/ for details.
+db_connection_string = "teradatasql://dbc:dbc@host.docker.internal/dbc"I can now call Pandas to query Vantage and move the result to a Pandas dataframe:
+pd.read_sql("SELECT * FROM dbc.dbcinfo", con = db_connection_string)The syntax above is concise but it can get tedious if all you need is to explore data in Vantage. We will use ipython-sql and its %%sql magic to create SQL-only cells. We start with importing the required libraries.
import sys
+!{sys.executable} -m pip install ipython-sql teradatasqlalchemyWe load ipython-sql and define the db connection string:
%load_ext sql
+# Define the db connection string. The sql magic uses SQLAlchemy connection strings.
+# For Teradata Vantage, it's teradatasql://username:password@host/database_name .
+# See https://pypi.org/project/teradatasqlalchemy/ for details.
+%sql teradatasql://dbc:dbc@host.docker.internal/dbcWe can now use %sql and %%sql magic. Let’s say we want to explore data in a table. We can create a cell that says:
%%sql
+SELECT * FROM dbc.dbcinfoIf we want to move the data to a Pandas frame, we can say:
+result = %sql SELECT * FROM dbc.dbcinfo
+result.DataFrame()There are many other features that ipython-sql provides, including variable substitution, plotting with matplotlib, writting results to a local csv file or back to the database. See the demo notebook for examples and ipython-sql github repo for a complete reference.
The Teradata Jupyter Docker image builds on jupyter/datascience-notebook Docker image. It adds the Teradata SQL kernel, Teradata Python and R libraries, Jupyter extensions to make you productive while interacting with Teradata Vantage. The image also contains sample notebooks that demonstrate how to use the SQL kernel and Teradata libraries.
The SQL kernel and Teradata Jupyter extensions are useful for people that spend a lot of time with the SQL interface. Think about it as a notebook experience that, in many cases, is more convenient than using Teradata Studio. The Teradata Jupyter Docker image doesn’t try to replace Teradata Studio. It doesn’t have all the features. It’s designed for people who need a lightweight, web-based interface and enjoy the notebook UI.
+The Teradata Jupyter Docker image can be used when you want to run Jupyter locally or you have a place where you can run custom Jupyter docker images. The steps below demonstrate how to use the image locally.
+Run the image:
+| + + | +
+By passing -e "accept_license=Y you accept the license agreement for Teradata Jupyter Extensions.
+ |
+
docker volume create notebooks
+docker run -e "accept_license=Y" -p :8888:8888 \
+ -v notebooks:/home/jovyan/JupyterLabRoot \
+ teradata/jupyterlab-extensionsDocker logs will display the url that you need to go to. For example, this is what I’ve got:
+Starting JupyterLab ... +Docker Build ID = 3.2.0-ec02012022 +Using unencrypted HTTP + +Enter this URL in your browser: http://localhost:8888?token=96a3ab874a03779c400966bf492fe270c2221cdcc74b61ed + +* Or enter this token when prompted by Jupyter: 96a3ab874a03779c400966bf492fe270c2221cdcc74b61ed +* If you used a different port to run your Docker, replace 8888 with your port number+
Open up the URL and use the file explorer to open the following notebook: jupyterextensions → notebooks → sql → GettingStartedDemo.ipynb.
Go through the demo of the Teradata SQL Kernel:
+
+This quick start covered different options to connect to Teradata Vantage from a Jupyter Notebook. We learned about the Teradata Jupyter Docker image that bundles multiple Teradata Python and R libraries. It also provides an SQL kernel, database object explorer and connection management. These features are useful when you spend a lot of time with the SQL interface. For more traditional data science scenarios, we explored the standalone Teradata Python driver and integration through the ipython sql extension.
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Hailing Jiang
+Last updated: November 17th, 2021
For customers who have their own JupyterHub clusters, there are two options to integrate Teradata Jupyter extensions into the existing clusters:
+Use Teradata Jupyter Docker image.
+Customize an existing Docker image to include Teradata extensions.
+This page contains detailed instructions on the two options. Instructions are based on the assumption that the customer JupyterHub deployment is based on Zero to JupyterHub with Kubernetes.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
Teradata provides a ready-to-run Docker image that builds on the jupyter/datascience-notebook image. It bundles the Teradata SQL kernel, Teradata Python and R libraries and drivers and Teradata extensions for Jupyter to make you productive while interacting with Teradata database. The image also contains sample notebooks that demonstrate how to use the SQL kernel, extensions and Teradata libraries.
+You can use this image in the following ways:
+Start a personal Jupyter Notebook server in a local Docker container
+Run JupyterLab servers for a team using JupyterHub
+For instructions to start a personal JupyterLab server in a local Docker container, please see installation guide. This section will focus on how to use the Teradata Jupyter Docker image in a customer’s existing JupyterHub environment.
+Go to Vantage Modules for Jupyter page and download the Docker image. It is a tarball with name in this format teradatajupyterlabext_VERSION.tar.gz.
Load the image:
+docker load -i teradatajupyterlabext_VERSION.tar.gzPush the image to your Docker registry:
+docker push| + + | +
+
+
+You may want to consider changing the name of the loaded image for simplicity: +
+
+
+
+ |
+
To use the Teradata Jupyter Docker image directly in your JupyterHub cluster, modify the override file as described in herein the JupyterHub documentation. Replace REGISTRY_URL and VERSION with appropriate values from the step above:
singleuser:
+ image:
+ name: REGISTRY_URL/teradatajupyterlabext_VERSION
+ tag: latestApply the changes to the cluster as described in JupyterHub documentation.
+| + + | ++You can use multiple profiles to allow users to select which image they want to use when they log in to JupyterHub. For detailed instructions and examples on configuring multiple profiles, please see JupyterHub documentation. + | +
If your users need some packages or notebooks that are not bundled in the Teradata Jupyter Docker image, we recommend that you use Teradata image as a base image and build a new one on top of it.
+Here is an example Dockerfile that builds on top of Teradata image and adds additional packages and notebooks. Use the Dockerfile to build a new Docker image, push the image to a designated registry, modify override file as shown above to use the new image as singleuser image, apply the changes to the cluster as described above. Replace REGISTRY_URL and VERSION with appropriate values:
FROM REGISTRY_URL/teradatajupyterlabext_VERSION:latest
+
+# install additional packages
+RUN pip install --no-cache-dir astropy
+
+# copy notebooks
+COPY notebooks/. /tmp/JupyterLabRoot/DemoNotebooks/If you prefer, you can include the Teradata SQL kernel and extensions into into an existing image you are currently using.
+Go to Vantage Modules for Jupyter page to download the zipped Teradata Jupyter extensions package bundle. Assuming your existing +docker image is Linux based, you will want to use the Linux version of the download. Otherwise, download for the platform you are using. The .zip file contains the Teradata SQL Kernel, extensions and sample +notebooks.
+Unzip the bundle file to your working directory.
+Below is an example Dockerfile to add Teradata Jupyter extensions to your existing Docker image. Use the Dockerfile to build a new Docker image, push the image to a designated registry, modify override file as shown above to use the new image as singleuser image, apply the changes to the cluster:
+FROM REGISTRY_URL/your-existing-image:tag
+ENV NB_USER=jovyan \
+ HOME=/home/jovyan \
+ EXT_DIR=/opt/teradata/jupyterext/packages
+
+USER root
+
+##############################################################
+# Install kernel and copy supporting files
+##############################################################
+
+# Copy the kernel
+COPY ./teradatakernel /usr/local/bin
+RUN chmod 755 /usr/local/bin/teradatakernel
+
+# Copy directory with kernel.json file into image
+COPY ./teradatasql teradatasql/
+
+##############################################################
+# Switch to user jovyan to copy the notebooks and license files.
+##############################################################
+
+USER $NB_USER
+
+# Copy notebooks
+COPY ./notebooks/ /tmp/JupyterLabRoot/TeradataSampleNotebooks/
+
+# Copy license files
+COPY ./ThirdPartyLicenses /tmp/JupyterLabRoot/ThirdPartyLicenses/
+
+USER root
+
+# Install the kernel file to /opt/conda jupyter lab instance
+RUN jupyter kernelspec install ./teradatasql --prefix=/opt/conda
+
+##############################################################
+# Install Teradata extensions
+##############################################################
+
+COPY ./teradata_*.tgz $EXT_DIR
+
+WORKDIR $EXT_DIR
+
+RUN jupyter labextension install --no-build teradata_database* && \
+ jupyter labextension install --no-build teradata_resultset* && \
+ jupyter labextension install --no-build teradata_sqlhighlighter* && \
+ jupyter labextension install --no-build teradata_connection_manager* && \
+ jupyter labextension install --no-build teradata_preferences* && \
+ jupyter lab build --dev-build=False --minimize=False && \
+ rm -rf *
+
+WORKDIR $HOME
+
+# Give back ownership of /opt/conda to jovyan
+RUN chown -R jovyan:users /opt/conda
+
+# Jupyter will create .local directory
+RUN rm -rf $HOME/.localYou can optionally install Teradata package for Python and Teradata package for R. See the following pages for details:
+ +| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Adam Tworkiewicz
+Last updated: September 12th, 2021
There are situations when you want to quickly validate a machine learning model idea. You have a model type in mind. You don’t want to operationalize with an ML pipeline just yet. You just want to test out if the relationship you had in mind exists. Also, sometimes even your production deployment doesn’t require constant relearning with MLops. In such cases, you can use Vantage Analytics Library (VAL) and multiple ML model types it supports.
+You need access to a Teradata Vantage instance.
+| + + | ++If you need a new instance of Vantage, you can install a free version called Vantage Express in the cloud on Google Cloud, Azure, and AWS. You can also run Vantage Express on your local machine using VMware, VirtualBox, or UTM. + | +
Support for ML in Vantage requires Vantage Analytics Library (VAL). In this section, we will install VAL and load some sample data.
+VAL is distributed as an rpm file. Go to Teradata Downloads and download the VAL rpm to your local machine.
+Upload the file to your Vantage install. If you are running Vantage Express locally, you have many ways to do it:
+If you installed Vantage Express on VirtualBox, you should be able to drag & drop the file to the VM’s desktop. You can also use scp by connecting to port 4422, e.g.:
scp -P 4422 ~/Downloads/VAL-2.0.0.3-1.x86_64.rpm root@localhost:/root/DesktopIf you use VMware and you have enabled drag & drop, you should be able to drag and drop the file to the VM’s desktop.
+If you have SSH access to your Vantage nodes, you can use scp to upload the binary, e.g.:
scp ~/Downloads/VAL-2.0.0.3-1.x86_64.rpm root@vantage.server.name:/tmp/We will now create a new database where VAL functions and procedures will be installed. You could install VAL in a global location such as SYSLIB, but installing VAL in a specific database will make it easier to start over if things go wrong.
+Let’s create a database called val and grant appropriate permissions to our user. Please edit to match your database name and user id:
CREATE DATABASE val
+AS PERMANENT = 60e6, -- 60MB
+ SPOOL = 120e6; -- 120MB
+
+GRANT CREATE FUNCTION ON val to dbc;
+GRANT ALTER FUNCTION ON val to dbc;
+GRANT EXECUTE PROCEDURE on SQLJ.REMOVE_JAR to dbc;
+GRANT EXECUTE PROCEDURE on SQLJ.INSTALL_JAR to dbc;
+GRANT EXECUTE PROCEDURE on SQLJ.REPLACE_JAR to dbc;
+GRANT CREATE EXTERNAL PROCEDURE ON val to dbc;Open Gnome Terminal in the VM and start the installation process. Adjust the rpm path as necessary:
+rpm -Uvh --nodeps ~/Desktop/VAL-2.0.0.3-1.x86_64.rpmThe install wizard will ask you for the hostname, user id, and password. If you are running the install on your Vantage Express VM, the values are:
+Hostname: localhost
Userid: dbc
Password: dbc
Account string: leave empty, press ENTER
+BTEQ or FASTLOAD command: leave empty, press ENTER
+The wizard will ask you to choose which part of VAL you want to install.
+We want to start with installing td_analyze procedure, i.e. option 1. Once you select option 1, the script will ask for the database name where td_analyze will be installed. Enter val and press ENTER.
While still in the wizard, install option 5, i.e. Tutorial Tables. These are sample tables with data that we are going to use to build a sample model.
Now, that we have VAL and sample tables loaded, let’s explore the data. It’s a simplistic, fictitious dataset of banking customers (1K-ish rows), Accounts (10K-ish rows) and Transactions (100K-ish rows). They are related to each other in the following ways:
+
+In later parts of this how-to we are going to explore if we can build a model that predicts average monthly balance that a banking customer has on their credit card based on all non-credit card related variables in the tables.
+Let’s start by creating a wide table (Analytic Data Set, or ADS) that joins the three tables above.
+| + + | ++You must have CREATE TABLE permissions on the Database where the Vantage Analytic Library is installed. + | +
-- Switch to val database.
+DATABASE val;
+
+-- Create the ADS.
+CREATE TABLE VAL_ADS AS (
+ SELECT
+ T1.cust_id AS cust_id
+ ,MIN(T1.income) AS tot_income
+ ,MIN(T1.age) AS tot_age
+ ,MIN(T1.years_with_bank) AS tot_cust_years
+ ,MIN(T1.nbr_children) AS tot_children
+ ,CASE WHEN MIN(T1.marital_status) = 1 THEN 1 ELSE 0 END AS single_ind
+ ,CASE WHEN MIN(T1.gender) = 'F' THEN 1 ELSE 0 END AS female_ind
+ ,CASE WHEN MIN(T1.marital_status) = 2 THEN 1 ELSE 0 END AS married_ind
+ ,CASE WHEN MIN(T1.marital_status) = 3 THEN 1 ELSE 0 END AS separated_ind
+ ,MAX(CASE WHEN T1.state_code = 'CA' THEN 1 ELSE 0 END) AS ca_resident_ind
+ ,MAX(CASE WHEN T1.state_code = 'NY' THEN 1 ELSE 0 END) AS ny_resident_ind
+ ,MAX(CASE WHEN T1.state_code = 'TX' THEN 1 ELSE 0 END) AS tx_resident_ind
+ ,MAX(CASE WHEN T1.state_code = 'IL' THEN 1 ELSE 0 END) AS il_resident_ind
+ ,MAX(CASE WHEN T1.state_code = 'AZ' THEN 1 ELSE 0 END) AS az_resident_ind
+ ,MAX(CASE WHEN T1.state_code = 'OH' THEN 1 ELSE 0 END) AS oh_resident_ind
+ ,MAX(CASE WHEN T2.acct_type = 'CK' THEN 1 ELSE 0 END) AS ck_acct_ind
+ ,MAX(CASE WHEN T2.acct_type = 'SV' THEN 1 ELSE 0 END) AS sv_acct_ind
+ ,MAX(CASE WHEN T2.acct_type = 'CC' THEN 1 ELSE 0 END) AS cc_acct_ind
+ ,AVG(CASE WHEN T2.acct_type = 'CK' THEN T2.starting_balance+T2.ending_balance ELSE 0 END) AS ck_avg_bal
+ ,AVG(CASE WHEN T2.acct_type = 'SV' THEN T2.starting_balance+T2.ending_balance ELSE 0 END) AS sv_avg_bal
+ ,AVG(CASE WHEN T2.acct_type = 'CC' THEN T2.starting_balance+T2.ending_balance ELSE 0 END) AS cc_avg_bal
+ ,AVG(CASE WHEN T2.acct_type = 'CK' THEN T3.principal_amt+T3.interest_amt ELSE 0 END) AS ck_avg_tran_amt
+ ,AVG(CASE WHEN T2.acct_type = 'SV' THEN T3.principal_amt+T3.interest_amt ELSE 0 END) AS sv_avg_tran_amt
+ ,AVG(CASE WHEN T2.acct_type = 'CC' THEN T3.principal_amt+T3.interest_amt ELSE 0 END) AS cc_avg_tran_amt
+ ,COUNT(CASE WHEN ((EXTRACT(MONTH FROM T3.tran_date) + 2) / 3) = 1 THEN T3.tran_id ELSE NULL END) AS q1_trans_cnt
+ ,COUNT(CASE WHEN ((EXTRACT(MONTH FROM T3.tran_date) + 2) / 3) = 2 THEN T3.tran_id ELSE NULL END) AS q2_trans_cnt
+ ,COUNT(CASE WHEN ((EXTRACT(MONTH FROM T3.tran_date) + 2) / 3) = 3 THEN T3.tran_id ELSE NULL END) AS q3_trans_cnt
+ ,COUNT(CASE WHEN ((EXTRACT(MONTH FROM T3.tran_date) + 2) / 3) = 4 THEN T3.tran_id ELSE NULL END) AS q4_trans_cnt
+ FROM Customer AS T1
+ LEFT OUTER JOIN Accounts AS T2
+ ON T1.cust_id = T2.cust_id
+ LEFT OUTER JOIN Transactions AS T3
+ ON T2.acct_nbr = T3.acct_nbr
+GROUP BY T1.cust_id) WITH DATA UNIQUE PRIMARY INDEX (cust_id);We will now build a linear regression model that takes parameters from the dataset and tries to predict the monthly credit card balance.
+We call td_analyze and tell it we want a linear model. The input is in table VAL_ADS and consists of multiple columns. The dependent variable is cc_avg_bal. We want the model to be written to val database in table called LINEAR_REGRESSION_DEMO:
call td_analyze('linear',
+ 'database=val;
+ tablename=VAL_ADS;
+ columns=tot_age,tot_income,tot_cust_years,tot_children,single_ind,female_ind,married_ind,separated_ind,ck_acct_ind,sv_acct_ind,sv_avg_bal,ck_avg_bal,ca_resident_ind,ny_resident_ind,tx_resident_ind,il_resident_ind,az_resident_ind,oh_resident_ind;
+ dependent=cc_avg_bal;
+ outputdatabase=val;
+ outputtablename=linear_regression_demo');The procedure creates several output tables. For now, we don’t have to analyze what is in the tables. Let’s see how we can use the newly created model to perform scoring.
+Let’s use the model to perform predictions and evaluate the scores. To do this, we call td_analyze with linearscore parameter. We point to the input table (VAL_ADS), the model tables (prefix linear_regression_demo) and define the target table (linear_regression_score) in val database:
call td_analyze('linearscore',
+ 'database=val;
+ tablename=VAL_ADS;
+ modeldatabase=val;
+ modeltablename=linear_regression_demo;
+ outputdatabase=val;
+ outputtablename=linear_regression_score;
+ predicted=estimate;
+ retain=cc_avg_bal;
+ scoringmethod=scoreandevaluate;');As a result, we get linear_regression_score table that contains the real balance, the predicted balance and the difference between these two. Let’s have a look at a sample:
SELECT * FROM linear_regression_score SAMPLE 10;You will see results similar to:
+cust_id|cc_avg_bal |estimate |Residual | +-------+------------------+------------------+-------------------+ +1362498| 0.0| 284.7057772484358| -284.7057772484358| +1362828| 1184.35|463.74177458594215| 720.6082254140578| +1362839| 2933.135802469136| 982.9240031182255| 1950.2117993509103| +1362986| 500.9148148148148| 881.4116539412856| -380.4968391264708| +1362511|235.85941489361701|294.35369563202846|-58.494280738411426| +1363134| 0.0|430.27950420065997|-430.27950420065997| +1363481| 0.0| 411.2359958542745| -411.2359958542745| +1362644| 209.3304347826087|279.75770904482033| -70.42727426221163| +1363141| 0.0| 550.1681921045503| -550.1681921045503| +1363290| 0.0|120.35348558871233|-120.35348558871233|+
In this quick start we have learned how to create ML models in SQL. The method used Vantage Analytics Library (VAL). We were able to build a linear regression model and run predictions using the model. We have done that using SQL without any coding.
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
| \n", + " | model_version | \n", + "model_id | \n", + "model_type | \n", + "project_id | \n", + "deployed_at | \n", + "model | \n", + "
|---|---|---|---|---|---|---|
| 0 | \n", + "9de00f0d-060c-4737-b0a3-531768363ced | \n", + "2354a903-601b-5f72-b014-8983306005b4 | \n", + "PMML | \n", + "414bec4e-c677-4f2e-a370-0076e57918ea | \n", + "2022-07-20 11:02:35.470 | \n", + "b'<?xml version=\"1.0\" encoding=\"UTF-8\"?>\\n<PMML xmlns=\"http://www.dmg.org/PMML-4_4\" version=\"4.4\">\\n <Header copyright=\"Copyright (c) 2018 Software AG\" description=\"Default Description\">\\n <Application name=\"Nyoka\" version=\"4.3.0\"/>\\n ... | \n", + "
| 1 | \n", + "5761d5c1-bf57-456b-8076-c3062be0b544 | \n", + "2354a903-601b-5f72-b014-8983306005b4 | \n", + "PMML | \n", + "414bec4e-c677-4f2e-a370-0076e57918ea | \n", + "2022-07-18 07:04:19.430 | \n", + "b'<?xml version=\"1.0\" encoding=\"UTF-8\"?>\\n<PMML xmlns=\"http://www.dmg.org/PMML-4_4\" version=\"4.4\">\\n <Header copyright=\"Copyright (c) 2018 Software AG\" description=\"Default Description\">\\n <Application name=\"Nyoka\" version=\"4.3.0\"/>\\n ... | \n", + "
| \n", + " | PatientId | \n", + "prediction | \n", + "json_report | \n", + "
|---|---|---|---|
| 0 | \n", + "545 | \n", + "1 | \n", + "{\"probability_0\":0.015597303259093032,\"probability_1\":0.984402696740907,\"predicted_HasDiabetes\":1} | \n", + "
| 1 | \n", + "265 | \n", + "0 | \n", + "{\"probability_0\":0.8621357683534957,\"probability_1\":0.1378642316465043,\"predicted_HasDiabetes\":0} | \n", + "
| 2 | \n", + "40 | \n", + "0 | \n", + "{\"probability_0\":0.574482742227689,\"probability_1\":0.425517257772311,\"predicted_HasDiabetes\":0} | \n", + "
| 3 | \n", + "385 | \n", + "0 | \n", + "{\"probability_0\":0.9902272802038437,\"probability_1\":0.009772719796156322,\"predicted_HasDiabetes\":0} | \n", + "
| 4 | \n", + "0 | \n", + "1 | \n", + "{\"probability_0\":0.0950635688096706,\"probability_1\":0.9049364311903294,\"predicted_HasDiabetes\":1} | \n", + "
| 5 | \n", + "600 | \n", + "0 | \n", + "{\"probability_0\":0.9499693357624215,\"probability_1\":0.050030664237578494,\"predicted_HasDiabetes\":0} | \n", + "
| 6 | \n", + "530 | \n", + "0 | \n", + "{\"probability_0\":0.9729698349887085,\"probability_1\":0.02703016501129154,\"predicted_HasDiabetes\":0} | \n", + "
| 7 | \n", + "120 | \n", + "1 | \n", + "{\"probability_0\":0.11393098981414929,\"probability_1\":0.8860690101858507,\"predicted_HasDiabetes\":1} | \n", + "
| 8 | \n", + "650 | \n", + "0 | \n", + "{\"probability_0\":0.9820752256818963,\"probability_1\":0.0179247743181037,\"predicted_HasDiabetes\":0} | \n", + "
| 9 | \n", + "80 | \n", + "0 | \n", + "{\"probability_0\":0.9819309856605802,\"probability_1\":0.01806901433941982,\"predicted_HasDiabetes\":0} | \n", + "
| \n", + " | PatientId | \n", + "prediction | \n", + "json_report | \n", + "
|---|---|---|---|
| 0 | \n", + "1 | \n", + "0 | \n", + "{\"probability_0\":0.9850747504768098,\"probability_1\":0.014925249523190227,\"predicted_HasDiabetes\":0} | \n", + "
| 1 | \n", + "3 | \n", + "0 | \n", + "{\"probability_0\":0.9945255942085638,\"probability_1\":0.005474405791436156,\"predicted_HasDiabetes\":0} | \n", + "
| 2 | \n", + "8 | \n", + "1 | \n", + "{\"probability_0\":0.05020155260184678,\"probability_1\":0.9497984473981532,\"predicted_HasDiabetes\":1} | \n", + "
| 3 | \n", + "9 | \n", + "1 | \n", + "{\"probability_0\":0.28263442350828416,\"probability_1\":0.7173655764917158,\"predicted_HasDiabetes\":1} | \n", + "
| 4 | \n", + "13 | \n", + "1 | \n", + "{\"probability_0\":0.08177880479006427,\"probability_1\":0.9182211952099357,\"predicted_HasDiabetes\":1} | \n", + "
| 5 | \n", + "17 | \n", + "1 | \n", + "{\"probability_0\":0.23832592730256774,\"probability_1\":0.7616740726974323,\"predicted_HasDiabetes\":1} | \n", + "
| 6 | \n", + "19 | \n", + "1 | \n", + "{\"probability_0\":0.37077762620144294,\"probability_1\":0.629222373798557,\"predicted_HasDiabetes\":1} | \n", + "
| 7 | \n", + "26 | \n", + "1 | \n", + "{\"probability_0\":0.18449399225529128,\"probability_1\":0.8155060077447087,\"predicted_HasDiabetes\":1} | \n", + "
| 8 | \n", + "43 | \n", + "1 | \n", + "{\"probability_0\":0.006277949339332567,\"probability_1\":0.9937220506606674,\"predicted_HasDiabetes\":1} | \n", + "
| 9 | \n", + "44 | \n", + "0 | \n", + "{\"probability_0\":0.7477334297352949,\"probability_1\":0.25226657026470506,\"predicted_HasDiabetes\":0} | \n", + "
| \n", + " | PatientId | \n", + "json_report | \n", + "
|---|---|---|
| 0 | \n", + "545 | \n", + "{\"output_probability\":[{\"0\":0.013714135,\"1\":0.98628587}],\"output_label\":[1]} | \n", + "
| 1 | \n", + "265 | \n", + "{\"output_probability\":[{\"0\":0.78800213,\"1\":0.21199787}],\"output_label\":[0]} | \n", + "
| 2 | \n", + "600 | \n", + "{\"output_probability\":[{\"0\":0.9766014,\"1\":0.023398578}],\"output_label\":[0]} | \n", + "
| 3 | \n", + "530 | \n", + "{\"output_probability\":[{\"0\":0.97246957,\"1\":0.027530432}],\"output_label\":[0]} | \n", + "
| 4 | \n", + "5 | \n", + "{\"output_probability\":[{\"0\":0.9863973,\"1\":0.013602674}],\"output_label\":[0]} | \n", + "
| 5 | \n", + "305 | \n", + "{\"output_probability\":[{\"0\":0.8996898,\"1\":0.10031021}],\"output_label\":[0]} | \n", + "
| 6 | \n", + "20 | \n", + "{\"output_probability\":[{\"0\":0.94164395,\"1\":0.058356047}],\"output_label\":[0]} | \n", + "
| 7 | \n", + "570 | \n", + "{\"output_probability\":[{\"0\":0.96782416,\"1\":0.03217584}],\"output_label\":[0]} | \n", + "
| 8 | \n", + "60 | \n", + "{\"output_probability\":[{\"0\":0.994561,\"1\":0.0054389834}],\"output_label\":[0]} | \n", + "
| 9 | \n", + "610 | \n", + "{\"output_probability\":[{\"0\":0.9871934,\"1\":0.012806594}],\"output_label\":[0]} | \n", + "
| PatientId | \n", + "\t\tHasDiabetes | \n", + "\t\tjson_report | \n", + "\t
|---|---|---|
| 390 | \n", + "\t\t0 | \n", + "\t\t{\"probability_0\":0.8534884407791314,\"probability_1\":0.1465115592208686,\"predicted_HasDiabetes\":0} | \n", + "\t
| 575 | \n", + "\t\t0 | \n", + "\t\t{\"probability_0\":0.5411704893844536,\"probability_1\":0.4588295106155465,\"predicted_HasDiabetes\":0} | \n", + "\t
| 740 | \n", + "\t\t1 | \n", + "\t\t{\"probability_0\":0.18309459019075702,\"probability_1\":0.816905409809243,\"predicted_HasDiabetes\":1} | \n", + "\t
| 290 | \n", + "\t\t0 | \n", + "\t\t{\"probability_0\":0.9275644263007603,\"probability_1\":0.0724355736992397,\"predicted_HasDiabetes\":0} | \n", + "\t
| 430 | \n", + "\t\t0 | \n", + "\t\t{\"probability_0\":0.9993898309624312,\"probability_1\":6.101690375688237E-4,\"predicted_HasDiabetes\":0} | \n", + "\t
| 410 | \n", + "\t\t0 | \n", + "\t\t{\"probability_0\":0.7306026985110992,\"probability_1\":0.26939730148890084,\"predicted_HasDiabetes\":0} | \n", + "\t
| 460 | \n", + "\t\t0 | \n", + "\t\t{\"probability_0\":0.7245463351662919,\"probability_1\":0.27545366483370815,\"predicted_HasDiabetes\":0} | \n", + "\t
| 325 | \n", + "\t\t0 | \n", + "\t\t{\"probability_0\":0.6100536262346685,\"probability_1\":0.3899463737653314,\"predicted_HasDiabetes\":0} | \n", + "\t
| 360 | \n", + "\t\t1 | \n", + "\t\t{\"probability_0\":0.014436294801631777,\"probability_1\":0.9855637051983682,\"predicted_HasDiabetes\":1} | \n", + "\t
| 560 | \n", + "\t\t1 | \n", + "\t\t{\"probability_0\":0.681609701643378,\"probability_1\":0.3183902983566221,\"predicted_HasDiabetes\":0} | \n", + "\t
| \n", + " | model_version | \n", + "model_id | \n", + "model_type | \n", + "project_id | \n", + "deployed_at | \n", + "model | \n", + "
|---|---|---|---|---|---|---|
| 0 | \n", + "9de00f0d-060c-4737-b0a3-531768363ced | \n", + "2354a903-601b-5f72-b014-8983306005b4 | \n", + "PMML | \n", + "414bec4e-c677-4f2e-a370-0076e57918ea | \n", + "2022-07-20 11:02:35.470 | \n", + "b'<?xml version=\"1.0\" encoding=\"UTF-8\"?>\\n<PMML xmlns=\"http://www.dmg.org/PMML-4_4\" version=\"4.4\">\\n <Header copyright=\"Copyright (c) 2018 Software AG\" description=\"Default Description\">\\n <Application name=\"Nyoka\" version=\"4.3.0\"/>\\n ... | \n", + "
| 1 | \n", + "5761d5c1-bf57-456b-8076-c3062be0b544 | \n", + "2354a903-601b-5f72-b014-8983306005b4 | \n", + "PMML | \n", + "414bec4e-c677-4f2e-a370-0076e57918ea | \n", + "2022-07-18 07:04:19.430 | \n", + "b'<?xml version=\"1.0\" encoding=\"UTF-8\"?>\\n<PMML xmlns=\"http://www.dmg.org/PMML-4_4\" version=\"4.4\">\\n <Header copyright=\"Copyright (c) 2018 Software AG\" description=\"Default Description\">\\n <Application name=\"Nyoka\" version=\"4.3.0\"/>\\n ... | \n", + "
| \n", + " | PatientId | \n", + "prediction | \n", + "json_report | \n", + "
|---|---|---|---|
| 0 | \n", + "545 | \n", + "1 | \n", + "{\"probability_0\":0.015597303259093032,\"probability_1\":0.984402696740907,\"predicted_HasDiabetes\":1} | \n", + "
| 1 | \n", + "265 | \n", + "0 | \n", + "{\"probability_0\":0.8621357683534957,\"probability_1\":0.1378642316465043,\"predicted_HasDiabetes\":0} | \n", + "
| 2 | \n", + "40 | \n", + "0 | \n", + "{\"probability_0\":0.574482742227689,\"probability_1\":0.425517257772311,\"predicted_HasDiabetes\":0} | \n", + "
| 3 | \n", + "385 | \n", + "0 | \n", + "{\"probability_0\":0.9902272802038437,\"probability_1\":0.009772719796156322,\"predicted_HasDiabetes\":0} | \n", + "
| 4 | \n", + "0 | \n", + "1 | \n", + "{\"probability_0\":0.0950635688096706,\"probability_1\":0.9049364311903294,\"predicted_HasDiabetes\":1} | \n", + "
| 5 | \n", + "600 | \n", + "0 | \n", + "{\"probability_0\":0.9499693357624215,\"probability_1\":0.050030664237578494,\"predicted_HasDiabetes\":0} | \n", + "
| 6 | \n", + "530 | \n", + "0 | \n", + "{\"probability_0\":0.9729698349887085,\"probability_1\":0.02703016501129154,\"predicted_HasDiabetes\":0} | \n", + "
| 7 | \n", + "120 | \n", + "1 | \n", + "{\"probability_0\":0.11393098981414929,\"probability_1\":0.8860690101858507,\"predicted_HasDiabetes\":1} | \n", + "
| 8 | \n", + "650 | \n", + "0 | \n", + "{\"probability_0\":0.9820752256818963,\"probability_1\":0.0179247743181037,\"predicted_HasDiabetes\":0} | \n", + "
| 9 | \n", + "80 | \n", + "0 | \n", + "{\"probability_0\":0.9819309856605802,\"probability_1\":0.01806901433941982,\"predicted_HasDiabetes\":0} | \n", + "
| \n", + " | PatientId | \n", + "prediction | \n", + "json_report | \n", + "
|---|---|---|---|
| 0 | \n", + "1 | \n", + "0 | \n", + "{\"probability_0\":0.9850747504768098,\"probability_1\":0.014925249523190227,\"predicted_HasDiabetes\":0} | \n", + "
| 1 | \n", + "3 | \n", + "0 | \n", + "{\"probability_0\":0.9945255942085638,\"probability_1\":0.005474405791436156,\"predicted_HasDiabetes\":0} | \n", + "
| 2 | \n", + "8 | \n", + "1 | \n", + "{\"probability_0\":0.05020155260184678,\"probability_1\":0.9497984473981532,\"predicted_HasDiabetes\":1} | \n", + "
| 3 | \n", + "9 | \n", + "1 | \n", + "{\"probability_0\":0.28263442350828416,\"probability_1\":0.7173655764917158,\"predicted_HasDiabetes\":1} | \n", + "
| 4 | \n", + "13 | \n", + "1 | \n", + "{\"probability_0\":0.08177880479006427,\"probability_1\":0.9182211952099357,\"predicted_HasDiabetes\":1} | \n", + "
| 5 | \n", + "17 | \n", + "1 | \n", + "{\"probability_0\":0.23832592730256774,\"probability_1\":0.7616740726974323,\"predicted_HasDiabetes\":1} | \n", + "
| 6 | \n", + "19 | \n", + "1 | \n", + "{\"probability_0\":0.37077762620144294,\"probability_1\":0.629222373798557,\"predicted_HasDiabetes\":1} | \n", + "
| 7 | \n", + "26 | \n", + "1 | \n", + "{\"probability_0\":0.18449399225529128,\"probability_1\":0.8155060077447087,\"predicted_HasDiabetes\":1} | \n", + "
| 8 | \n", + "43 | \n", + "1 | \n", + "{\"probability_0\":0.006277949339332567,\"probability_1\":0.9937220506606674,\"predicted_HasDiabetes\":1} | \n", + "
| 9 | \n", + "44 | \n", + "0 | \n", + "{\"probability_0\":0.7477334297352949,\"probability_1\":0.25226657026470506,\"predicted_HasDiabetes\":0} | \n", + "
| \n", + " | PatientId | \n", + "json_report | \n", + "
|---|---|---|
| 0 | \n", + "545 | \n", + "{\"output_probability\":[{\"0\":0.013714135,\"1\":0.98628587}],\"output_label\":[1]} | \n", + "
| 1 | \n", + "265 | \n", + "{\"output_probability\":[{\"0\":0.78800213,\"1\":0.21199787}],\"output_label\":[0]} | \n", + "
| 2 | \n", + "600 | \n", + "{\"output_probability\":[{\"0\":0.9766014,\"1\":0.023398578}],\"output_label\":[0]} | \n", + "
| 3 | \n", + "530 | \n", + "{\"output_probability\":[{\"0\":0.97246957,\"1\":0.027530432}],\"output_label\":[0]} | \n", + "
| 4 | \n", + "5 | \n", + "{\"output_probability\":[{\"0\":0.9863973,\"1\":0.013602674}],\"output_label\":[0]} | \n", + "
| 5 | \n", + "305 | \n", + "{\"output_probability\":[{\"0\":0.8996898,\"1\":0.10031021}],\"output_label\":[0]} | \n", + "
| 6 | \n", + "20 | \n", + "{\"output_probability\":[{\"0\":0.94164395,\"1\":0.058356047}],\"output_label\":[0]} | \n", + "
| 7 | \n", + "570 | \n", + "{\"output_probability\":[{\"0\":0.96782416,\"1\":0.03217584}],\"output_label\":[0]} | \n", + "
| 8 | \n", + "60 | \n", + "{\"output_probability\":[{\"0\":0.994561,\"1\":0.0054389834}],\"output_label\":[0]} | \n", + "
| 9 | \n", + "610 | \n", + "{\"output_probability\":[{\"0\":0.9871934,\"1\":0.012806594}],\"output_label\":[0]} | \n", + "
| PatientId | \n", + "\t\tHasDiabetes | \n", + "\t\tjson_report | \n", + "\t
|---|---|---|
| 390 | \n", + "\t\t0 | \n", + "\t\t{\"probability_0\":0.8534884407791314,\"probability_1\":0.1465115592208686,\"predicted_HasDiabetes\":0} | \n", + "\t
| 575 | \n", + "\t\t0 | \n", + "\t\t{\"probability_0\":0.5411704893844536,\"probability_1\":0.4588295106155465,\"predicted_HasDiabetes\":0} | \n", + "\t
| 740 | \n", + "\t\t1 | \n", + "\t\t{\"probability_0\":0.18309459019075702,\"probability_1\":0.816905409809243,\"predicted_HasDiabetes\":1} | \n", + "\t
| 290 | \n", + "\t\t0 | \n", + "\t\t{\"probability_0\":0.9275644263007603,\"probability_1\":0.0724355736992397,\"predicted_HasDiabetes\":0} | \n", + "\t
| 430 | \n", + "\t\t0 | \n", + "\t\t{\"probability_0\":0.9993898309624312,\"probability_1\":6.101690375688237E-4,\"predicted_HasDiabetes\":0} | \n", + "\t
| 410 | \n", + "\t\t0 | \n", + "\t\t{\"probability_0\":0.7306026985110992,\"probability_1\":0.26939730148890084,\"predicted_HasDiabetes\":0} | \n", + "\t
| 460 | \n", + "\t\t0 | \n", + "\t\t{\"probability_0\":0.7245463351662919,\"probability_1\":0.27545366483370815,\"predicted_HasDiabetes\":0} | \n", + "\t
| 325 | \n", + "\t\t0 | \n", + "\t\t{\"probability_0\":0.6100536262346685,\"probability_1\":0.3899463737653314,\"predicted_HasDiabetes\":0} | \n", + "\t
| 360 | \n", + "\t\t1 | \n", + "\t\t{\"probability_0\":0.014436294801631777,\"probability_1\":0.9855637051983682,\"predicted_HasDiabetes\":1} | \n", + "\t
| 560 | \n", + "\t\t1 | \n", + "\t\t{\"probability_0\":0.681609701643378,\"probability_1\":0.3183902983566221,\"predicted_HasDiabetes\":0} | \n", + "\t
| job_id | \n", + "\t\tPatientId | \n", + "\t\tHasDiabetes | \n", + "\t\tjson_report | \n", + "\t
|---|---|---|---|
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t360 | \n", + "\t\t1 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t45 | \n", + "\t\t1 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t490 | \n", + "\t\t0 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t660 | \n", + "\t\t1 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t30 | \n", + "\t\t1 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t220 | \n", + "\t\t1 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t355 | \n", + "\t\t1 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t560 | \n", + "\t\t0 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t460 | \n", + "\t\t0 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t325 | \n", + "\t\t0 | \n", + "\t\t\n", + "\t |
| job_id | \n", + "\t\tPatientId | \n", + "\t\tHasDiabetes | \n", + "\t\tjson_report | \n", + "\t
|---|---|---|---|
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t360 | \n", + "\t\t1 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t45 | \n", + "\t\t1 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t490 | \n", + "\t\t0 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t660 | \n", + "\t\t1 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t30 | \n", + "\t\t1 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t220 | \n", + "\t\t1 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t355 | \n", + "\t\t1 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t560 | \n", + "\t\t0 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t460 | \n", + "\t\t0 | \n", + "\t\t\n", + "\t |
| 2d16fcf4-78e3-4801-a052-a9c224814b85 | \n", + "\t\t325 | \n", + "\t\t0 | \n", + "\t\t\n", + "\t |
| \n", + " | job_id | \n", + "PatientId | \n", + "HasDiabetes | \n", + "json_report | \n", + "
|---|
Author: Pablo Escobar de la Oliva
+Last updated: May 29th, 2023
This is a how-to for people who are new to ClearScape Analytics ModelOps. In the tutorial, you will be able to create a new project in ModelOps, upload the required data to Vantage, and track the full lifecycle of an imported Diabetes demo model using BYOM mechanisms.
+Access to a Teradata Vantage instance with ClearScape Analytics (includes ModelOps)
+Ability to run Jupyter notebooks
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
Files needed
+Let’s start by downloading the needed files for this tutorial. Download these 4 attachments and upload them in your Notebook filesystem. Select the files depending on your version of ModelOps:
+ModelOps version 6 (October 2022):
+Alternatively you can git clone following repos
+git clone https://github.com/willfleury/modelops-getting-started
+git clone https://github.com/Teradata/modelops-demo-models/ModelOps version 7 (April 2023):
+git clone -b v7 https://github.com/willfleury/modelops-getting-started.git +git clone https://github.com/Teradata/modelops-demo-models/+
Setting up the Database and Jupyter environment
+Follow the ModelOps_Training Jupyter Notebook to setup the database, tables and libraries needed for the demo.
+Add a new Project
+create project
+Details
+Name: Demo: your-name
+Description: ModelOps Demo
+Group: your-name
+Credentials: No Credentials
+Branch: master
+Here you can test the git connection. If is green then save and continue. Skip the service connection settings for now.
+When creating a new project, ModelOps will ask you for a new connection.
+Personal connection
+Name: Vantage personal your-name
+Description: Vantage demo env
+Host: tdprd.td.teradata.com (internal for teradata transcend only)
+Database: your-db
+VAL Database: TRNG_XSP (internal for teradata transcend only)
+BYOM Database: TRNG_BYOM (internal for teradata transcend only)
+Login Mech: TDNEGO
+Username/Password
+You can check the permissions with the new healthcheck panel in the connections panel
+
+Let’s create a new dataset template, then 1 dataset for training and 2 datasets for evaluation so we can monitor model quality metrics with 2 different datasets
+Add datasets
+create dataset template
+Catalog
+Name: PIMA
+Description: PIMA Diabetes
+Feature Catalog: Vantage
+Database: your-db
+Table: aoa_feature_metadata
+Features +Query:
+
+SELECT * FROM {your-db}.pima_patient_featuresEntity Key: PatientId +Features: NumTimesPrg, PlGlcConc, BloodP, SkinThick, TwoHourSerIns, BMI, DiPedFunc, Age
+Entity & Target +Query:
+SELECT * FROM {your-db}.pima_patient_diagnosesEntity Key: PatientId +Target: HasDiabetes
+Predictions
+Database: your-db
+Table: pima_patient_predictions
+Entity selection:
+Query:
+SELECT * FROM pima_patient_features WHERE patientid MOD 5 = 0Only for v6 (in v7 you will define this in the BYOM no code screen): BYOM Target Column: CAST(CAST(json_report AS JSON).JSONExtractValue('$.predicted_HasDiabetes') AS INT)
+Basic
+Name: Train
+Description: Training dataset
+Scope: Training
+Entity & Target
+Query:
+SELECT * FROM {your-db}.pima_patient_diagnoses WHERE patientid MOD 5 = 1Basic
+Name: Evaluate
+Description: Evaluation dataset
+Scope: Evaluation
+Entity & Target
+Query:
+SELECT * FROM {your-db}.pima_patient_diagnoses WHERE patientid MOD 5 = 2Basic
+Name: Evaluate
+Description: Evaluation dataset
+Scope: Evaluation
+Entity & Target
+Query:
+SELECT * FROM {your-db}.pima_patient_diagnoses WHERE patientid MOD 5 = 3Download and unzip the files needed, links are at the top of the tutorial. For PMML file you can also download a PMML generated in the training of a GIT model.
+BYOM.ipynb
+model.pmml
+requirements.txt
+evaluation.py
+data_stats.json
+init.py
+Define BYOM Model with Evaluation and Monitoring
+Import Version
+for v7 - BYOM no code is available - You can enable automated evaluation and data drift monitoring. +In Monitoring page use BYOM Target Column: CAST(CAST(json_report AS JSON).JSONExtractValue('$.predicted_HasDiabetes') AS INT)
+Evaluate
+Review evaluation report, including dataset statistics
+Approve
+Deploy in Vantage - Engine, Publish, Schedule. Scoring dataset is required +Use your connection and select a database. e.g "aoa_byom_models"
+Deployments/executions
+Evaluate again with dataset2 - to monitor model metrics behavior
+Monitor Model Drift - Data and Metrics
+for v7 - Review your predictions directly from Deployments → Job page
+Open BYOM notebook to execute the PMML predict from SQL code
+Retire
+In this quick start we have learned how to follow a full lifecycle of BYOM models into ModelOps and how to deploy it into Vantage. Then how we can schedule a batch scoring or test restful or on-demand scorings and start monitoring on Data Drift and Model Quality metrics.
+link:https://docs.teradata.com/search/documents?query=ModelOps&sort=last_update&virtual-field=title_only&content-lang=
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Pablo Escobar de la Oliva
+Last updated: May 29th, 2022
This is a how-to for people who are new to ClearScape Analytics ModelOps. In the tutorial, you will be able to create a new project in ModelOps, upload the required data to Vantage, and track the full lifecycle of a demo model using code templates and following the methodology for GIT models in ModelOps.
+Access to a Teradata Vantage instance with ClearScape Analytics (includes ModelOps)
+Ability to run Jupyter notebooks
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
Files needed
+Let’s start by downloading the needed files for this tutorial. Download these 4 attachments and upload them in your Notebook filesystem. Select the files depending on your version of ModelOps:
+ModelOps version 6 (October 2022):
+Alternatively you can git clone following repos
+git clone https://github.com/willfleury/modelops-getting-started
+git clone https://github.com/Teradata/modelops-demo-models/ModelOps version 7 (April 2023):
+git clone -b v7 https://github.com/willfleury/modelops-getting-started.git +git clone https://github.com/Teradata/modelops-demo-models/+
Setting up the database and Jupyter environment
+Follow the ModelOps_Training Jupyter Notebook to setup the database, tables and libraries needed for the demo.
+Add a new Project
+create project
+Details
+Name: Demo: your-name
+Description: ModelOps Demo
+Group: your-name
+Credentials: No Credentials
+Branch: master
+Here you can test the git connection. If is green then save and continue. Skip the service connection settings for now.
+When creating a new project, ModelOps will ask you for a new connection.
+Personal connection
+Name: Vantage personal your-name
+Description: Vantage demo env
+Host: tdprd.td.teradata.com (internal for teradata transcend only)
+Database: your-db
+VAL Database: TRNG_XSP (internal for teradata transcend only)
+BYOM Database: TRNG_BYOM (internal for teradata transcend only)
+Login Mech: TDNEGO
+Username/Password
+You can check the permissions with the new healthcheck panel in the connections panel
+
+Let’s create a new dataset template, then 1 dataset for training and 2 datasets for evaluation so we can monitor model quality metrics with 2 different datasets
+Add datasets
+create dataset template
+Catalog
+Name: PIMA
+Description: PIMA Diabetes
+Feature Catalog: Vantage
+Database: your-db
+Table: aoa_feature_metadata
+Features +Query:
+
+SELECT * FROM {your-db}.pima_patient_featuresEntity Key: PatientId +Features: NumTimesPrg, PlGlcConc, BloodP, SkinThick, TwoHourSerIns, BMI, DiPedFunc, Age
+Entity & Target +Query:
+SELECT * FROM {your-db}.pima_patient_diagnosesEntity Key: PatientId +Target: HasDiabetes
+Predictions
+Database: your-db
+Table: pima_patient_predictions
+Entity selection:
+Query:
+SELECT * FROM pima_patient_features WHERE patientid MOD 5 = 0Only for v6 (in v7 you will define this in the BYOM no code screen): BYOM Target Column: CAST(CAST(json_report AS JSON).JSONExtractValue('$.predicted_HasDiabetes') AS INT)
+Basic
+Name: Train
+Description: Training dataset
+Scope: Training
+Entity & Target
+Query:
+SELECT * FROM {your-db}.pima_patient_diagnoses WHERE patientid MOD 5 = 1Basic
+Name: Evaluate
+Description: Evaluation dataset
+Scope: Evaluation
+Entity & Target
+Query:
+SELECT * FROM {your-db}.pima_patient_diagnoses WHERE patientid MOD 5 = 2Basic
+Name: Evaluate
+Description: Evaluation dataset
+Scope: Evaluation
+Entity & Target
+Query:
+SELECT * FROM {your-db}.pima_patient_diagnoses WHERE patientid MOD 5 = 3For Git Models we need to fill the code templates available when adding a new model.
+These code scripts will be stored in the git repository under: model_definitions/your-model/model_modules/
+init.py : this an empty file required for python modules
+training.py: this script contains train function
+def train(context: ModelContext, **kwargs):
+ aoa_create_context()
+
+ # your training code
+
+ # save your model
+ joblib.dump(model, f"{context.artifact_output_path}/model.joblib")
+
+ record_training_stats(...)Review the Operationalize notebook to see how you can execute this from CLI or from notebook as an alternative to ModelOps UI.
+evaluation.py: this script contains evaluate function
+def evaluate(context: ModelContext, **kwargs):
+ aoa_create_context()
+
+ # read your model
+ model = joblib.load(f"{context.artifact_input_path}/model.joblib")
+
+ # your evaluation logic
+
+ record_evaluation_stats(...)Review the Operationalize notebook to see how you can execute this from CLI or from notebook as an alternative to ModelOps UI.
+scoring.py: this script contains score function
+def score(context: ModelContext, **kwargs):
+ aoa_create_context()
+
+ # read your model
+ model = joblib.load(f"{context.artifact_input_path}/model.joblib")
+
+ # your evaluation logic
+
+ record_scoring_stats(...)Review the Operationalize notebook to see how you can execute this from CLI or from notebook as an alternative to ModelOps UI.
+requirements.txt: this file contains the library names and versions required for your code scripts. Example:
+%%writefile ../model_modules/requirements.txt
+xgboost==0.90
+scikit-learn==0.24.2
+shap==0.36.0
+matplotlib==3.3.1
+teradataml==17.0.0.4
+nyoka==4.3.0
+aoa==6.0.0config.json: this file located in the parent folder (your-model folder) contains default hyper-parameters
+%%writefile ../config.json
+{
+ "hyperParameters": {
+ "eta": 0.2,
+ "max_depth": 6
+ }
+}Go and review the code scripts for the demo model in the repository: https://github.com/Teradata/modelops-demo-models/
+Go into model_definitions→python-diabetes→model_modules
+Open Project to see models available from GIT
+Train a new model version
+see how CommitID from code repository is tracked
+Evaluate
+Review evaluation report, including dataset statistics and model metrics
+Compare with other model versions
+Approve
+Deploy in Vantage - Engine, Publish, Schedule. Scoring dataset is required +Use your connection and select a database. e.g "aoa_byom_models"
+Deploy in Docker Batch - Engine, Publish, Schedule. Scoring dataset is required +Use your connection and select a database. e.g "aoa_byom_models"
+Deploy in Restful Batch - Engine, Publish, Schedule. Scoring dataset is required +Use your connection and select a database. e.g "aoa_byom_models"
+Deployments/executions
+Evaluate again with dataset2 - to monitor model metrics behavior
+Monitor Model Drift - data and metrics
+Open BYOM notebook to execute the PMML predict from SQL code when deployed in Vantage
+Test Restful from ModelOps UI or from curl command
+Retire deployments
+In this quick start we have learned how to follow a full lifecycle of GIT models into ModelOps and how to deploy it into Vantage or into Docker containers for Edge deployments. Then how we can schedule a batch scoring or test restful or on-demand scorings and start monitoring on Data Drift and Model Quality metrics.
+link:https://docs.teradata.com/search/documents?query=ModelOps&sort=last_update&virtual-field=title_only&content-lang=
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Mohammmad Taha Wahab, Mohammad Harris Mansur and Will Fleury
+Last updated: January 5th, 2023
Feast’s connector for Teradata is a complete implementation with support for all features and uses Teradata Vantage as an online and offline store.
+Access to a Teradata Vantage instance.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
This how-to assumes you know Feast terminology. If you need a refresher check out the official FEAST documentation
+This document demonstrates how developers can integrate Teradata’s offline and online store with Feast. Teradata’s offline stores allow users to use any underlying data store as their offline feature store. Features can be retrieved from the offline store for model training and can be materialized into the online feature store for use during model inference.
On the other hand, online stores are used to serve features at low latency. The materialize command can be used to load feature values from the data sources (or offline stores) into the online store
The feast-teradata library adds support for Teradata as
OfflineStore
OnlineStore
Additionally, using Teradata as the registry (catalog) is already supported via the registry_type: sql and included in our examples. This means that everything is located in Teradata. However, depending on the requirements, installation, etc, this can be mixed and matched with other systems as appropriate.
To get started, install the feast-teradata library
pip install feast-teradataLet’s create a simple feast setup with Teradata using the standard drivers' dataset. Note that you cannot use feast init as this command only works for templates that are part of the core feast library. We intend on getting this library merged into feast core eventually but for now, you will need to use the following cli command for this specific task. All other feast cli commands work as expected.
feast-td init-repoThis will then prompt you for the required information for the Teradata system and upload the example dataset. Let’s assume you used the repo name demo when running the above command. You can find the repository files along with a file called test_workflow.py. Running this test_workflow.py will execute a complete workflow for the feast with Teradata as the Registry, OfflineStore, and OnlineStore.
demo/
+ feature_repo/
+ driver_repo.py
+ feature_store.yml
+ test_workflow.pyFrom within the demo/feature_repo directory, execute the following feast command to apply (import/update) the repo definition into the registry. You will be able to see the registry metadata tables in the teradata database after running this command.
feast applyTo see the registry information in the feast UI, run the following command. Note the --registry_ttl_sec is important as by default it polls every 5 seconds.
+feast ui --registry_ttl_sec=120project: <name of project>
+registry: <registry>
+provider: local
+offline_store:
+ type: feast_teradata.offline.teradata.TeradataOfflineStore
+ host: <db host>
+ database: <db name>
+ user: <username>
+ password: <password>
+ log_mech: <connection mechanism>Below is an example of definition.py which elaborates how +to set the entity, source connector, and feature view.
+Now to explain the different components:
+TeradataSource: Data Source for features stored in Teradata (Enterprise or Lake) or accessible via a Foreign Table from Teradata (NOS, QueryGrid)
Entity: A collection of semantically related features
Feature View: A feature view is a group of feature data from a specific data source. Feature views allow you to consistently define features and their data sources, enabling the reuse of feature groups across a project
driver = Entity(name="driver", join_keys=["driver_id"])
+project_name = yaml.safe_load(open("feature_store.yaml"))["project"]
+
+driver_stats_source = TeradataSource(
+ database=yaml.safe_load(open("feature_store.yaml"))["offline_store"]["database"],
+ table=f"{project_name}_feast_driver_hourly_stats",
+ timestamp_field="event_timestamp",
+ created_timestamp_column="created",
+)
+
+driver_stats_fv = FeatureView(
+ name="driver_hourly_stats",
+ entities=[driver],
+ ttl=timedelta(weeks=52 * 10),
+ schema=[
+ Field(name="driver_id", dtype=Int64),
+ Field(name="conv_rate", dtype=Float32),
+ Field(name="acc_rate", dtype=Float32),
+ Field(name="avg_daily_trips", dtype=Int64),
+ ],
+ source=driver_stats_source,
+ tags={"team": "driver_performance"},
+)There are two different ways to test your offline store as explained below. But first, there are a few mandatory steps to follow:
+Now, let’s batch-read some features for training, using only entities (population) for which we have seen an event in the last 60 days. The predicates (filter) used can be on anything relevant for the entity (population) selection for the given training dataset. The event_timestamp is only for example purposes.
from feast import FeatureStore
+store = FeatureStore(repo_path="feature_repo")
+training_df = store.get_historical_features(
+ entity_df=f"""
+ SELECT
+ driver_id,
+ event_timestamp
+ FROM demo_feast_driver_hourly_stats
+ WHERE event_timestamp BETWEEN (CURRENT_TIMESTAMP - INTERVAL '60' DAY) AND CURRENT_TIMESTAMP
+ """,
+ features=[
+ "driver_hourly_stats:conv_rate",
+ "driver_hourly_stats:acc_rate",
+ "driver_hourly_stats:avg_daily_trips"
+ ],
+).to_df()
+print(training_df.head())The feast-teradata library allows you to use the complete set of feast APIs and functionality. Please refer to the official feast quickstart for more details on the various things you can do.
Feast materializes data to online stores for low-latency lookup at model inference time. Typically, key-value stores are used for online stores, however, relational databases can be used for this purpose as well.
+Users can develop their own online stores by creating a class that implements the contract in the OnlineStore class.
+project: <name of project>
+registry: <registry>
+provider: local
+offline_store:
+ type: feast_teradata.offline.teradata.TeradataOfflineStore
+ host: <db host>
+ database: <db name>
+ user: <username>
+ password: <password>
+ log_mech: <connection mechanism>There are a few mandatory steps to follow before we can test the online store:
+The command materialize_incremental is used to incrementally materialize features in the online store. If there are no new features to be added, this command will essentially not be doing anything. With feast materialize_incremental, the start time is either now — ttl (the ttl that we defined in our feature views) or the time of the most recent materialization. If you’ve materialized features at least once, then subsequent materializations will only fetch features that weren’t present in the store at the time of the previous materializations.
CURRENT_TIME=$(date +'%Y-%m-%dT%H:%M:%S')
+feast materialize-incremental $CURRENT_TIMENext, while fetching the online features, we have two parameters features and entity_rows. The features parameter is a list and can take any number of features that are present in the df_feature_view. The example above shows all 4 features present but these can be less than 4 as well. Secondly, the entity_rows parameter is also a list and takes a dictionary of the form {feature_identifier_column: value_to_be_fetched}. In our case, the column driver_id is used to uniquely identify the different rows of the entity driver. We are currently fetching values of the features where driver_id is equal to 5. We can also fetch multiple such rows using the format: [{driver_id: val_1}, {driver_id: val_2}, .., {driver_id: val_n}] [{driver_id: val_1}, {driver_id: val_2}, .., {driver_id: val_n}]
entity_rows = [
+ {
+ "driver_id": 1001,
+ },
+ {
+ "driver_id": 1002,
+ },
+ ]
+features_to_fetch = [
+ "driver_hourly_stats:acc_rate",
+ "driver_hourly_stats:conv_rate",
+ "driver_hourly_stats:avg_daily_trips"
+ ]
+returned_features = store.get_online_features(
+ features=features_to_fetch,
+ entity_rows=entity_rows,
+).to_dict()
+for key, value in sorted(returned_features.items()):
+ print(key, " : ", value)Another important thing is the SQL Registry. We first make a path variable that uses the username, password, database name, etc. to make a connection string which it then uses to establish a connection to Teradata’s Database.
+path = 'teradatasql://'+ teradata_user +':' + teradata_password + '@'+host + '/?database=' + teradata_database + '&LOGMECH=' + teradata_log_mechIt will create the following table in your database:
+Entities (entity_name,project_id,last_updated_timestamp,entity_proto)
+Data_sources (data_source_name,project_id,last_updated_timestamp,data_source_proto)
+Feature_views (feature_view_name,project_id,last_updated_timestamp,materialized_intervals,feature_view_proto,user_metadata)
+Request_feature_views (feature_view_name,project_id,last_updated_timestamp,feature_view_proto,user_metadata)
+Stream_feature_views (feature_view_name,project_id,last_updated_timestamp,feature_view_proto,user_metadata)
+managed_infra (infra_name, project_id, last_updated_timestamp, infra_proto)
+validation_references (validation_reference_name, project_id, last_updated_timestamp, validation_reference_proto)
+saved_datasets (saved_dataset_name, project_id, last_updated_timestamp, saved_dataset_proto)
+feature_services (feature_service_name, project_id, last_updated_timestamp, feature_service_proto)
+on_demand_feature_views (feature_view_name, project_id, last_updated_timestamp, feature_view_proto, user_metadata)
+Additionally, if you want to see a complete (but not real-world), end-to-end example workflow example, see the demo/test_workflow.py script. This is used for testing the complete feast functionality.
An Enterprise Feature Store accelerates the value-gaining process in crucial stages of data analysis. It enhances productivity and reduces the time taken to introduce products in the market. By integrating Teradata with Feast, it enables the use of Teradata’s highly efficient parallel processing within a Feature Store, thereby enhancing performance.
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Tan Nguyen
+Last updated: February 13th, 2023
Anypoint Studio (Studio) editors help you design and update your Mule applications, properties, and configuration files.
+To add and configure a connector in Studio:
+When you run the connector, you can view the app log to check for problems, as described in View the App Log.
+If you are new to configuring connectors in Studio, see Using Anypoint Studio to Configure a Connector. If, after reading this topic, you need additional information about the connector fields, see the Teradata Connector Reference.
+In Studio, create a new Mule project in which to add and configure the connector:
+In Studio, select File > New > Mule Project.
+Enter a name for your Mule project and click Finish.
+Add Teradata Connector to your Mule project to automatically populate the XML code with the connector’s namespace and schema location and to add the required dependencies to the project’s pom.xml file:
In the Mule Palette view, click (X) Search in Exchange.
+In the Add Dependencies to Project window, type teradata in the search field.
Click Teradata Connector in Available modules.
+Click Add.
+Click Finish.
+Adding a connector to a Mule project in Studio does not make that connector available to other projects in your Studio workspace.
+A source initiates a flow when a specified condition is met. +You can configure one of these input sources to use with Teradata Connector:
+Teradata > On Table Row
+Initiates a flow by selecting from a table at a regular interval and generates one message per obtained row
HTTP > Listener
+Initiates a flow each time it receives a request on the configured host and port
Scheduler
+Initiates a flow when a time-based condition is met
For example, to configure an On Table Row source, follow these steps:
+In the Mule Palette view, select Teradata > On Table Row.
+Drag On Table Row to the Studio canvas.
+In the On Table Row configuration screen, optionally change the value of the Display Name field.
+Click the plus sign (+) next to the Connector configuration field to configure a global element that can be used by all instances of the source in the app.
+In the Teradata Config window, on the General tab, specify the database connection information for the connector.
+On the Transactions tab, optionally specify the transaction isolation, or XA transactions, when connecting to the database.
+On the Advanced tab, optionally specify connection pooling and reconnection information, including a reconnection strategy.
+Click Test Connection to confirm that Mule can connect with the specified database.
+Click OK to close the window.
+In the On Table Row configuration screen, in Table, specify the name of the table to select from.
+When you add a connector operation to your flow, you immediately define a specific operation for that connector to perform.
+To add an operation for Teradata Connector, follow these steps:
+In the Mule Palette view, select Teradata Connector and then select the desired operation.
+Drag the operation onto the Studio canvas and to the right of the input source.
+The following screenshot shows the Teradata Connector operations in the Mule Palette view of Anypoint Studio:
+
+When you configure a connector, it’s best to configure a global element that all instances of that connector in the app can use.
+To configure the global element for Teradata Connector, follow these steps:
+Select the operation in the Studio canvas.
+In the configuration screen for the operation, click the plus sign (+) next to the Connector configuration field to access the global element configuration fields.
+In the Teradata Config window, on the General tab, specify the database connection information for the connector.
+On the Transactions tab, optionally specify the transaction isolation, or XA transactions, when connecting to the database.
+On the Advanced tab, optionally specify connection pooling and reconnection information, including a reconnection strategy.
+Click Test Connection to confirm that Mule can connect with the specified database.
+Click OK.
+The following screenshot shows the Teradata Connector Global Element Configuration window in Anypoint Studio:
+
+To check for problems, you can view the app log as follows:
+If you’re running the app from Anypoint Platform, the output is visible in the Anypoint Studio console window.
+If you’re running the app using Mule from the command line, the app log is visible in your OS console.
+Unless the log file path is customized in the app’s log file (log4j2.xml), you can also view the app log in the default location MULE_HOME/logs/<app-name>.log.
| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Tan Nguyen
+Last updated: February 10th, 2023
Anypoint Connector for Teradata (Teradata Connector) establishes communication between your Mule app and a Teradata Vantage database, enabling you to connect with your Teradata Vantage instance to load data and run SQL queries in Teradata Vantage tables.
+Reference: Teradata Connector Reference
+Release Notes: Teradata Connector Release Notes
+To use this connector, you must be familiar with:
+Teradata Vantage SQL
+Anypoint Connectors
+Mule runtime engine (Mule)
+Elements and global elements in a Mule flow
+Anypoint Studio (Studio)
+Before creating an app, you must have:
+Anypoint Studio version 7.5 or later
+Credentials to access the Teradata Vantage target resource
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
Teradata Connector enables you to:
+Perform predefined queries, dynamically constructed queries, and template queries that are self-sufficient and customizable.
+Use a source listener operation to read from a database in the data source section of a flow.
+Execute other operations to read and write to a database anywhere in the process section.
+Run a single bulk update to perform multiple SQL requests.
+Make Data Definition Language (DDL) requests.
+Execute stored procedures and SQL scripts.
+The Teradata Connector supports:
+Connection pooling
+Auto reconnects after timeouts
+After you complete the prerequisites, you can try the examples and configure the connector using Anypoint Studio.
+Author: Tan Nguyen
+Last updated: February 10th, 2023
Anypoint Connector for Teradata (Teradata Connector) establishes communication between your Mule app and a Teradata Vantage database, enabling you to connect with your Teradata Vantage instance to load data and run SQL queries in Teradata Vantage tables.
+Release Notes: Teradata Connector Release Notes
+Use these parameters to configure the default configuration.
+| Name | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Name |
+
+ String + |
+
+ The name for this configuration. Connectors reference the configuration with this name. + |
++ |
+ x + |
+
Connection |
+
+
|
+
+ The connection types to provide to this configuration. + |
++ |
+ x + |
+
Expiration Policy |
++ |
+ Configures the minimum amount of time that a dynamic configuration instance can remain idle before Mule considers it eligible for expiration. This does not mean that the platform expires the instance at the exact moment that it becomes eligible. Mule purges the instances as appropriate. + |
++ | + |
Configure the connection provider implementation that creates database connections from a referenced data source.
When you use a provider’s custom type in a Data Source Reference Connection, define the type inside the Column Types form of the Advanced section in the Database config.
| Name | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Pooling Profile |
++ |
+ Provides a way to configure database connection pooling + |
++ | + |
Column Types |
+
+ Array of Column Type + |
+
+ Specifies non-standard column types + |
++ | + |
Reconnection |
+
+
+ |
+
+ When the application is deployed, a connectivity test is performed on all connectors. If set to |
++ | + |
| Name | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Pooling Profile |
++ |
+ Provides a way to configure database connection pooling + |
++ | + |
Column Types |
+
+ Array of Column Type + |
+
+ Specifies non-standard column types + |
++ | + |
Transaction Isolation |
+
+
+Enumeration, one of: +
+
|
+
+ The transaction isolation level to set on the driver when connecting the database + |
+
+
|
++ |
Use XA Transactions |
+
+ Boolean + |
+
+ Indicates whether or not the created datasource must support XA transactions + |
+
+
|
++ |
URL |
+
+ String + |
+
+ JDBC URL to use to connect to the database + |
++ |
+ x + |
+
User |
+
+ String + |
+
+ Database username + |
++ | + |
Password |
+
+ String + |
+
+ Database password + |
++ | + |
Reconnection |
+
+
+ |
+
+ When the application is deployed, a connectivity test is performed on all connectors. If set to |
++ | + |
| + + | +
+To specify an SQL function in an SQL query in an operation, specify the SQL function in the {fn function()} format. For example, the SQL function CURRENT_TIMESTAMP is specified as {fn CURRENT_TIMESTAMP()}.
+ |
+
<db:bulk-delete>
This operation allows delete operations to execute at various times using different parameter bindings and a single database statement. This improves performance compared to executing a single delete operation at various times.
+| Name | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Configuration |
+
+ String + |
+
+ The name of the configuration to use + |
++ |
+ x + |
+
Input Parameters |
+
+ Array of Object + |
+
+ Specifies a list of maps, which contain the parameter names as keys and the value the parameter is bound to, and in which every list item represents a row to insert. + |
+
+
|
++ |
Transactional Action |
+
+
+Enumeration, one of: +
+
|
+
+ The type of joining action that operations can take regarding transactions + |
+
+
|
++ |
Query Timeout |
+
+ Number + |
+
+ Indicates the minimum amount of time before the JDBC driver attempts to cancel a running statement. By default, no timeout is used. + |
+
+
|
++ |
Query Timeout Unit |
+
+
+Enumeration, one of: +
+
|
+
+ A time unit that qualifies the |
+
+
|
++ |
Fetch Size |
+
+ Number + |
+
+ Indicates how many rows to fetch from the database when rows are read from a ResultSet. This property is required when streaming is true, in which case a default value of |
++ | + |
Max Rows |
+
+ Number + |
+
+ Sets the limit for the maximum number of rows that any ResultSet object generated by this message processor can contain for the given number. If the limit is exceeded, the excess rows are silently dropped. + |
++ | + |
SQL Query Text |
+
+ String + |
+
+ The text of the SQL query to execute + |
++ |
+ x + |
+
Parameter Types |
+
+ Array of Parameter Type + |
+
+ This parameter allows you to optionally specify the type of one or more of the parameters in the query. If a value is provided, you’re not required to reference all of the parameters; but you cannot reference a parameter that is not present in the input values. + |
++ | + |
Target Variable |
+
+ String + |
+
+ The name of a variable to store the operation’s output + |
++ | + |
Target Value |
+
+ String + |
+
+ An expression to evaluate against the operation’s output and store the expression outcome in the target variable + |
+
+
|
++ |
Reconnection Strategy |
+
+
|
+
+ A retry strategy in case of connectivity errors + |
++ | + |
<db:bulk-insert>
This operation allows inserts to execute at various times using different parameter bindings and a single database statement. This improves performance compared to executing a single insert operation at various times.
+| Name | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Configuration |
+
+ String + |
+
+ The name of the configuration to use + |
++ |
+ x + |
+
Input Parameters |
+
+ Array of Object + |
+
+ A list of maps in which every list item represents a row to be inserted, and the map contains the parameter names as keys and the value the parameter is bound to. + |
+
+
|
++ |
Transactional Action |
+
+
+Enumeration, one of: +
+
|
+
+ The type of joining action that operations can take regarding transactions. + |
+
+ JOIN_IF_POSSIBLE + |
++ |
Query Timeout |
+
+ Number + |
+
+ Indicates the minimum amount of time before the JDBC driver attempts to cancel a running statement. No timeout is used by default. + |
+
+ 0 + |
++ |
Query Timeout Unit |
+
+
+Enumeration, one of: +
+
|
+
+ A TimeUnit that qualifies the |
+
+
|
++ |
Fetch Size |
+
+ Number + |
+
+ Indicates how many rows to fetch from the database when rows are read from a resultSet. This property is required when streaming is true; in that case a default value (10) is used. + |
++ | + |
Max Rows |
+
+ Number + |
+
+ Sets the limit for the maximum number of rows that any ResultSet object generated by this message processor can contain for the given number. If the limit is exceeded, the excess rows are silently dropped. + |
++ | + |
SQL Query Text |
+
+ String + |
+
+ The text of the SQL query to execute + |
++ |
+ x + |
+
Parameter Types |
+
+ Array of Parameter Type + |
+
+ Allows you to optionally specify the type of one or more of the parameters in the query. If provided, you’re not required to reference all of the parameters, but you cannot reference a parameter not present in the input values + |
++ | + |
Target Variable |
+
+ String + |
+
+ The name of a variable to store the operation’s output. + |
++ | + |
Target Value |
+
+ String + |
+
+ An expression to evaluate against the operation’s output and store the expression outcome in the target variable + |
+
+
|
++ |
Reconnection Strategy |
+
+
|
+
+ A retry strategy in case of connectivity errors. + |
++ | + |
<db:bulk-update>
This operation allows updates to execute at various times using different parameter bindings and a single database statement. This improves performance compared to executing one single update operation at various times.
+| Name | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Configuration |
+
+ String + |
+
+ The name of the configuration to use + |
++ |
+ x + |
+
Input Parameters |
+
+ Array of Object + |
+
+ Specifies a list of maps, which contain the parameter names as keys and the value the parameter is bound to, and in which every list item represents a row to insert. + |
+
+
|
++ |
Transactional Action |
+
+
+Enumeration, one of: +
+
|
+
+ The type of joining action that operations can take regarding transactions. + |
+
+ JOIN_IF_POSSIBLE + |
++ |
Query Timeout |
+
+ Number + |
+
+ Indicates the minimum amount of time before the JDBC driver attempts to cancel a running statement. By default, no timeout is used. + |
+
+
|
++ |
Query Timeout Unit |
+
+
+Enumeration, one of: +
+
|
+
+ A time unit that qualifies the |
+
+
|
++ |
Fetch Size |
+
+ Number + |
+
+ Indicates how many rows to fetch from the database when rows are read from a ResultSet. + |
+
+ 10 + |
++ |
Max Rows |
+
+ Number + |
+
+ Sets the limit for the maximum number of rows that any ResultSet object generated by this message processor can contain for the given number. If the limit is exceeded, the excess rows are silently dropped. + |
++ | + |
SQL Query Text |
+
+ String + |
+
+ The text of the SQL query to execute + |
++ |
+ x + |
+
Parameter Types |
+
+ Array of Parameter Type + |
+
+ Allows you to optionally specify the type of one or more of the parameters in the query. If provided, you’re not required to reference all of the parameters; but you cannot reference a parameter that is not present in the input values. + |
++ | + |
Target Variable |
+
+ String + |
+
+ The name of a variable to store the operation’s output + |
++ | + |
Target Value |
+
+ String + |
+
+ An expression to evaluate against the operation’s output and store the expression outcome in the target variable + |
+
+
|
++ |
Reconnection Strategy |
+
+
|
+
+ A retry strategy in case of connectivity errors + |
++ | + |
<db:delete>
This operation deletes data in a database.
+| Name | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Configuration |
+
+ String + |
+
+ The name of the configuration to use + |
++ |
+ x + |
+
Transactional Action |
+
+
+Enumeration, one of: +
+
|
+
+ The type of joining action that operations can take regarding transactions + |
+
+
|
++ |
Query Timeout |
+
+ Number + |
+
+ Indicates the minimum amount of time before the JDBC driver attempts to cancel a running statement. By default, no timeout is used. + |
+
+
|
++ |
Query Timeout Unit |
+
+
+Enumeration, one of: +
+
|
+
+ A time unit that qualifies the |
+
+
|
++ |
Fetch Size |
+
+ Number + |
+
+ Indicates how many rows to fetch from the database when rows are read from a ResultSet. + |
+
+ 10 + |
++ |
Max Rows |
+
+ Number + |
+
+ Sets the limit for the maximum number of rows that any ResultSet object generated by this message processor can contain for the given number. If the limit is exceeded, the excess rows are silently dropped. + |
++ | + |
SQL Query Text |
+
+ String + |
+
+ The text of the SQL query to execute + |
++ |
+ x + |
+
Parameter Types |
+
+ Array of Parameter Type + |
+
+ Allows you to optionally specify the type of one or more of the parameters in the query. If a value is provided, you’re not required to reference all of the parameters; but you cannot reference a parameter that is not present in the input values. + |
++ | + |
Input Parameters |
+
+ Object + |
+
+ A map in which keys are the name of an input parameter to set on the JDBC prepared statement. Each parameter should be referenced in the SQL text using a colon prefix (for example, |
++ | + |
Target Variable |
+
+ String + |
+
+ The name of a variable to store the operation’s output + |
++ | + |
Target Value |
+
+ String + |
+
+ An expression to evaluate against the operation’s output and store the expression outcome in the target variable + |
+
+
|
++ |
Reconnection Strategy |
+
+
|
+
+ A retry strategy in case of connectivity errors + |
++ | + |
<db:execute-ddl>
This operation allows execution of DDL queries against a database.
+| Name | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Configuration |
+
+ String + |
+
+ The name of the configuration to use + |
++ |
+ x + |
+
SQL Query Text |
+
+ String + |
+
+ The text of the SQL query to execute + |
++ |
+ x + |
+
Transactional Action |
+
+
+Enumeration, one of: +
+
|
+
+ The type of joining action that operations can take regarding transactions + |
+
+
|
++ |
Query Timeout |
+
+ Number + |
+
+ Indicates the minimum amount of time before the JDBC driver attempts to cancel a running statement. By default, no timeout is used. + |
+
+
|
++ |
Query Timeout Unit |
+
+
+Enumeration, one of: +
+
|
+
+ A time unit that qualifies the |
+
+
|
++ |
Fetch Size |
+
+ Number + |
+
+ Indicates how many rows to fetch from the database when rows are read from a ResultSet. + |
+
+ 10 + |
++ |
Max Rows |
+
+ Number + |
+
+ Sets the limit for the maximum number of rows that any ResultSet object generated by this message processor can contain for the given number. If the limit is exceeded, the excess rows are silently dropped. + |
++ | + |
Target Variable |
+
+ String + |
+
+ The name of a variable to store the operation’s output + |
++ | + |
Target Value |
+
+ String + |
+
+ An expression to evaluate against the operation’s output and store the expression outcome in the target variable + |
+
+
|
++ |
Reconnection Strategy |
+
+
|
+
+ A retry strategy in case of connectivity errors + |
++ | + |
<db:execute-script>
This operation executes an SQL script in a single database statement. The script is executed as provided by the user, without any parameter binding.
+| Name | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Configuration |
+
+ String + |
+
+ The name of the configuration to use + |
++ |
+ x + |
+
Transactional Action |
+
+
+Enumeration, one of: +
+
|
+
+ The type of joining action that operations can take for transactions. + |
+
+
|
++ |
SQL Query Text |
+
+ String + |
+
+ The text of the SQL query to execute + |
++ | + |
Script Path |
+
+ String + |
+
+ Specifies the location of a file to load. The file can point to a resource on the classpath, or on a disk. + |
++ | + |
Query Timeout |
+
+ Number + |
+
+ Indicates the minimum amount of time before the JDBC driver attempts to cancel a running statement. By default, no timeout is used. + |
+
+
|
++ |
Query Timeout Unit |
+
+
+Enumeration, one of: +
+
|
+
+ A time unit that qualifies the |
+
+
|
++ |
Fetch Size |
+
+ Number + |
+
+ Indicates how many rows to fetch from the database when rows are read from a ResultSet. + |
+
+ 10 + |
++ |
Max Rows |
+
+ Number + |
+
+ Sets the limit for the maximum number of rows that any ResultSet object generated by this message processor can contain for the given number. If the limit is exceeded, the excess rows are silently dropped. + |
++ | + |
Target Variable |
+
+ String + |
+
+ The name of a variable to store the operation’s output + |
++ | + |
Target Value |
+
+ String + |
+
+ An expression to evaluate against the operation’s output and store the expression outcome in the target variable + |
+
+
|
++ |
Reconnection Strategy |
+
+
|
+
+ A retry strategy in case of connectivity errors + |
++ | + |
<db:insert>
This operation inserts data into a database.
+| Name | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Configuration |
+
+ String + |
+
+ The name of the configuration to use + |
++ |
+ x + |
+
Transactional Action |
+
+
+Enumeration, one of: +
+
|
+
+ The type of joining action that operations can take regarding transactions + |
+
+
|
++ |
Query Timeout |
+
+ Number + |
+
+ Indicates the minimum amount of time before the JDBC driver attempts to cancel a running statement. By default, no timeout is used. + |
+
+
|
++ |
Query Timeout Unit |
+
+
+Enumeration, one of: +
+
|
+
+ A time unit that qualifies the |
+
+
|
++ |
Fetch Size |
+
+ Number + |
+
+ Indicates how many rows to fetch from the database when rows are read from a ResultSet. + |
+
+ 10 + |
++ |
Max Rows |
+
+ Number + |
+
+ Sets the limit for the maximum number of rows that any ResultSet object generated by this message processor can contain for the given number. If the limit is exceeded, the excess rows are silently dropped. + |
++ | + |
SQL Query Text |
+
+ String + |
+
+ The text of the SQL query to execute + |
++ |
+ x + |
+
Parameter Types |
+
+ Array of Parameter Type + |
+
+ Allows you to optionally specify the type of one or more of the parameters in the query. If provided, you’re not required to reference all of the parameters; but you cannot reference a parameter that is not present in the input values. + |
++ | + |
Input Parameters |
+
+ Object + |
+
+ A map in which keys are the name of an input parameter to be set on the JDBC prepared statement. Each parameter should be referenced in the SQL text using a colon prefix (E.g: where id = :myParamName)). The map’s values contain the actual assignation for each parameter. + |
++ | + |
Auto Generate Keys |
+
+ Boolean + |
+
+ Indicates when to make auto-generated keys available for retrieval. + |
+
+
|
++ |
Auto Generated Keys Column Indexes |
+
+ Array of Number + |
+
+ List of column indexes that indicates which auto-generated keys to make available for retrieval + |
++ | + |
Auto Generated Keys Column Names |
+
+ Array of String + |
+
+ List of column names that indicates which auto-generated keys to make available for retrieval + |
++ | + |
Target Variable |
+
+ String + |
+
+ The name of a variable to store the operation’s output + |
++ | + |
Target Value |
+
+ String + |
+
+ An expression to evaluate against the operation’s output and store the expression outcome in the target variable + |
+
+
|
++ |
Reconnection Strategy |
+
+
|
+
+ A retry strategy in case of connectivity errors + |
++ | + |
<db:select>
This operation queries data from a database. To prevent loading all the results at once, which can lead to performance and memory issues, results are automatically streamed. This means that pages of fetchSize rows are loaded when needed. If this operation is performed inside a transaction (that is, within a Try scope component) and that transaction is closed before consuming the data, accessing the results that haven’t been loaded will fail.
+| Name | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Configuration |
+
+ String + |
+
+ The name of the configuration to use + |
++ |
+ x + |
+
Transactional Action |
+
+
+Enumeration, one of: +
+
|
+
+ The type of joining action that operations can take regarding transactions + |
+
+
|
++ |
Streaming Strategy |
+
+
|
+
+ Configure to use repeatable streams + |
++ | + |
Query Timeout |
+
+ Number + |
+
+ Indicates the minimum amount of time before the JDBC driver attempts to cancel a running statement. By default, no timeout is used. + |
+
+
|
++ |
Query Timeout Unit |
+
+
+Enumeration, one of: +
+
|
+
+ A time unit that qualifies the |
+
+
|
++ |
Fetch Size |
+
+ Number + |
+
+ Indicates how many rows to fetch from the database when rows are read from a ResultSet. + |
+
+ 10 + |
++ |
Max Rows |
+
+ Number + |
+
+ Sets the limit for the maximum number of rows that any ResultSet object generated by this message processor can contain for the given number. If the limit is exceeded, the excess rows are silently dropped. + |
++ | + |
SQL Query Text |
+
+ String + |
+
+ The text of the SQL query to execute + |
++ |
+ x + |
+
Parameter Types |
+
+ Array of Parameter Type + |
+
+ Allows you to optionally specify the type of one or more of the parameters in the query. If provided, you’re not required to reference all of the parameters; but you cannot reference a parameter that is not present in the input values. + |
++ | + |
Input Parameters |
+
+ Object + |
+
+ A map in which keys are the name of an input parameter to be set on the JDBC prepared statement. Each parameter should be referenced in the SQL text using a colon prefix (for example: where id = :myParamName)). The map’s values will contain the actual assignation for each parameter. + |
++ | + |
Target Variable |
+
+ String + |
+
+ The name of a variable to store the operation’s output. + |
++ | + |
Target Value |
+
+ String + |
+
+ An expression to evaluate against the operation’s output and store the expression outcome in the target variable + |
+
+
|
++ |
Reconnection Strategy |
+
+
|
+
+ A retry strategy in case of connectivity errors. + |
++ | + |
When working with pooling profiles and the Select operation, the connection remains open until one of the following occurs:
+The flow execution ends
+The content of the streams are consumed completely
+The connection is the transaction key.
+| + + | ++Because LOBs are treated as streams, the connection remains open until the flow execution ends, or until the content is consumed before the flow completes, in which case the best approach is taken to close the related connection. + | +
This behavior occurs because the result set the operation generates can have a stream or be part of an ongoing transaction.
+<db:query-single>
This operation selects a single data record from a database. If you provide an SQL query that returns more than one row, then only the first record is processed and returned. This operation does not use streaming, which means that immediately after performing the Query Single operation, the complete content of the selected record is loaded into memory.
+| Name | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Configuration |
+
+ String + |
+
+ The name of the configuration to use + |
++ |
+ x + |
+
Transactional Action |
+
+
+Enumeration, one of: +
+
|
+
+ The type of join action that operations can take regarding transactions + |
+
+
|
++ |
Streaming Strategy |
+
+
|
+
+ Configure to use repeatable streams + |
++ | + |
Query Timeout |
+
+ Number + |
+
+ Indicates the minimum amount of time before the JDBC driver attempts to cancel a running statement. By default, no timeout is used. + |
+
+
|
++ |
Query Timeout Unit |
+
+
+Enumeration, one of: +
+
|
+
+ A time unit that qualifies the |
+
+
|
++ |
Fetch Size |
+
+ Number + |
+
+ Indicates how many rows to fetch from the database when rows are read from a ResultSet. + |
+
+ 10 + |
++ |
Max Rows |
+
+ Number + |
+
+ The maximum number of rows that any ResultSet object generated by this message processor can contain. If the limit is exceeded, the excess rows are silently dropped. + |
++ | + |
SQL Query Text |
+
+ String + |
+
+ The text of the SQL query to execute + |
++ |
+ x + |
+
Parameter Types |
+
+ Array of Parameter Type + |
+
+ Enables you to optionally specify the type of one or more of the parameters in the query. If provided, you’re not required to reference all of the parameters; but you cannot reference a parameter that is not present in the input values. + |
++ | + |
Input Parameters |
+
+ Object + |
+
+ A map in which keys are the name of an input parameter to be set on the JDBC prepared statement. Each parameter should be referenced in the SQL text using a colon prefix (for example: where id = :myParamName)). The map’s values will contain the actual assignation for each parameter. + |
++ | + |
Target Variable |
+
+ String + |
+
+ Name of the variable in which to store the operation’s output + |
++ | + |
Target Value |
+
+ String + |
+
+ Expression that evaluates the operation’s output. The expression outcome is stored in the target variable. + |
+
+
|
++ |
Reconnection Strategy |
+
+
|
+
+ A retry strategy in case of connectivity errors. + |
++ | + |
When working with pooling profiles and the Query Single operation, the connection returns to the pool immediately after the operation is performed.
+<db:stored-procedure>
Invokes a stored procedure on the database. When the stored procedure returns one or more ResultSet instances, results are not read all at once. Instead, results are automatically streamed to prevent performance and memory issues. This behavior means that pages of fetchSize rows are loaded lazily when needed. If the Stored procedure operation is performed inside a transaction (for example, in a Try scope component), and that transaction is closed before consuming the data, accessing the results that haven’t been loaded will fail.
| Name | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Configuration |
+
+ String + |
+
+ The name of the configuration to use. + |
++ |
+ x + |
+
Transactional Action |
+
+
+Enumeration, one of: +
+
|
+
+ The type of joining action that operations can take regarding transactions. + |
+
+ JOIN_IF_POSSIBLE + |
++ |
Streaming Strategy |
+
+
|
+
+ Configure to use repeatable streams + |
++ | + |
Query Timeout |
+
+ Number + |
+
+ Indicates the minimum amount of time before the JDBC driver attempts to cancel a running statement. No timeout is used by default. + |
+
+ 0 + |
++ |
Query Timeout Unit |
+
+
+Enumeration, one of: +
+
|
+
+ A time unit that qualifies the |
+
+
|
++ |
Fetch Size |
+
+ Number + |
+
+ Indicates how many rows to fetch from the database when rows are read from a resultSet. This property is required when streaming is true; in that case a default value (10) is used. + |
++ | + |
Max Rows |
+
+ Number + |
+
+ Sets the limit for the maximum number of rows that any ResultSet object generated by this message processor can contain for the given number. If the limit is exceeded, the excess rows are silently dropped. + |
++ | + |
SQL Query Text |
+
+ String + |
+
+ The text of the SQL query to execute + |
++ |
+ x + |
+
Parameter Types |
+
+ Array of Parameter Type + |
+
+ Allows to optionally specify the type of one or more of the parameters in the query. If provided, you’re not even required to reference all of the parameters, but you cannot reference a parameter not present in the input values + |
++ | + |
Input Parameters |
+
+ Object + |
+
+ A map in which keys are the name of an input parameter to be set on the JDBC prepared statement. Each parameter should be referenced in the SQL text using a colon prefix (for example: where id = :myParamName)). The map’s values will contain the actual assignation for each parameter. + |
++ | + |
Input - Output Parameters |
+
+ Object + |
+
+ A map in which keys are the name of a parameter to be set on the JDBC prepared statement which is both input and output. Each parameter should be referenced in the SQL text using a colon prefix (for example: where id = :myParamName)). The map’s values will contain the actual assignation for each parameter. + |
++ | + |
Output Parameters |
+
+ Array of Output Parameter + |
+
+ A list of output parameters to be set on the JDBC prepared statement. Each parameter should be referenced in the SQL text using a colon prefix (for example: call multiply(:value, :result)) + |
++ | + |
Auto Generate Keys |
+
+ Boolean + |
+
+ Indicates when to make auto-generated keys available for retrieval. + |
+
+
|
++ |
Auto Generated Keys Column Indexes |
+
+ Array of Number + |
+
+ List of column indexes that indicates which auto-generated keys to make available for retrieval. + |
++ | + |
Auto Generated Keys Column Names |
+
+ Array of String + |
+
+ List of column names that indicates which auto-generated keys should be made available for retrieval. + |
++ | + |
Target Variable |
+
+ String + |
+
+ The name of a variable to store the operation’s output. + |
++ | + |
Target Value |
+
+ String + |
+
+ An expression to evaluate against the operation’s output and store the expression outcome in the target variable + |
+
+
|
++ |
Reconnection Strategy |
+
+
|
+
+ A retry strategy in case of connectivity errors + |
++ | + |
When working with pooling profiles and the Stored procedure operation, the connection remains open until the flow execution ends or the content of the streams are consumed completely, or if the connection is the transaction key. This behavior occurs because the resultset the operation generates can have a stream or be part of an ongoing transaction.
+Starting with Database Connector 1.8.3, the connections on the Stored procedure operation are released if they are not part of a stream or transaction.
+<db:update>
Updates data in a database.
+| Name | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Configuration |
+
+ String + |
+
+ The name of the configuration to use + |
++ |
+ x + |
+
Transactional Action |
+
+
+Enumeration, one of: +
+
|
+
+ The type of joining action that operations can take for transactions + |
+
+
|
++ |
Query Timeout |
+
+ Number + |
+
+ Indicates the minimum amount of time before the JDBC driver attempts to cancel a running statement. By default, no timeout is used. + |
+
+
|
++ |
Query Timeout Unit |
+
+
+Enumeration, one of: +
+
|
+
+ A time unit that qualifies the |
+
+
|
++ |
Fetch Size |
+
+ Number + |
+
+ Indicates how many rows to fetch from the database when rows are read from a ResultSet. + |
+
+ 10 + |
++ |
Max Rows |
+
+ Number + |
+
+ Sets the limit for the maximum number of rows that any ResultSet object generated by this message processor can contain for the given number. If the limit is exceeded, the excess rows are silently dropped. + |
++ | + |
SQL Query Text |
+
+ String + |
+
+ The text of the SQL query to execute + |
++ |
+ x + |
+
Parameter Types |
+
+ Array of Parameter Type + |
+
+ Allows you to optionally specify the type of one or more of the parameters in the query. If provided, you’re not required to reference all of the parameters; but you cannot reference a parameter that is not present in the input values. + |
++ | + |
Input Parameters |
+
+ Object + |
+
+ A map in which keys are the name of an input parameter to set on the JDBC prepared statement. Each parameter should be referenced in the SQL text using a colon prefix (for example: where id = :myParamName)). The map’s values contain the actual assignation for each parameter. + |
++ | + |
Auto Generate Keys |
+
+ Boolean + |
+
+ Indicates when to make auto-generated keys available for retrieval + |
+
+
|
++ |
Auto Generated Keys Column Indexes |
+
+ Array of Number + |
+
+ List of column indexes that indicates which auto-generated keys to make available for retrieval + |
++ | + |
Auto Generated Keys Column Names |
+
+ Array of String + |
+
+ List of column names that indicates which auto-generated keys should be made available for retrieval + |
++ | + |
Target Variable |
+
+ String + |
+
+ The name of a variable to store the operation’s output + |
++ | + |
Target Value |
+
+ String + |
+
+ An expression to evaluate against the operation’s output and store the expression outcome in the target variable + |
+
+
|
++ |
Reconnection Strategy |
+
+
|
+
+ A retry strategy in case of connectivity errors + |
++ | + |
<db:listener>
This operation selects from a table at a regular interval and generates one message per obtained row. Optionally, you can provide watermark and ID columns. If a watermark column is provided, the values taken from that column are used to filter the contents of the next poll, so that only rows with a greater watermark value are returned. If an ID column is provided, this component automatically verifies that the same row is not picked twice by concurrent polls.
+This operation does not support streaming, meaning that there is no need to perform additional transformations to the payload in order to access the operation results. This behavior is identical to the Query Single operation released in version 1.9.
+| Name | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Configuration |
+
+ String + |
+
+ The name of the configuration to use + |
++ |
+ x + |
+
Table |
+
+ String + |
+
+ The name of the table to select from + |
++ |
+ x + |
+
Watermark Column |
+
+ String + |
+
+ The name of the column to use for a watermark. Values taken from this column are used to filter the contents of the next poll, so that only rows with a greater watermark value are processed. + |
++ | + |
Id Column |
+
+ String + |
+
+ The name of the column to consider as the row ID. If provided, this component makes sure that the same row is not processed twice by concurrent polls. + |
++ | + |
Transactional Action |
+
+
+Enumeration, one of: +
+
|
+
+ The type of beginning action that sources can take regarding transactions + |
+
+
|
++ |
Transaction Type |
+
+
+Enumeration, one of: +
+
|
+
+ The type of transaction to create. Availability depends on the runtime version. + |
+
+
|
++ |
Primary Node Only |
+
+ Boolean + |
+
+ Whether this source should be executed only on the primary node when running in a cluster + |
++ | + |
Scheduling Strategy |
+
+ scheduling-strategy + |
+
+ Configures the scheduler that triggers the polling + |
++ |
+ x + |
+
Redelivery Policy |
++ |
+ Defines a policy for processing the redelivery of the same message + |
++ | + |
Query Timeout |
+
+ Number + |
+
+ Indicates the minimum amount of time before the JDBC driver attempts to cancel a running statement. By default, no timeout is used. + |
+
+
|
++ |
Query Timeout Unit |
+
+
+Enumeration, one of: +
+
|
+
+ A time unit that qualifies the |
+
+
|
++ |
Fetch Size |
+
+ Number + |
+
+ Indicates how many rows to fetch from the database when rows are read from a ResultSet. + |
+
+ 10 + |
++ |
Max Rows |
+
+ Number + |
+
+ Sets the limit for the maximum number of rows that any ResultSet object generated by this message processor can contain for the given number. If the limit is exceeded, the excess rows are silently dropped. + |
++ | + |
Reconnection Strategy |
+
+
|
+
+ A retry strategy in case of connectivity errors. + |
++ | + |
| Field | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Max Pool Size |
+
+ Number + |
+
+ Maximum number of connections a pool maintains at any given time + |
+
+
|
++ |
Min Pool Size |
+
+ Number + |
+
+ Minimum number of connections a pool maintains at any given time + |
+
+
|
++ |
Acquire Increment |
+
+ Number + |
+
+ Determines how many connections at a time to try to acquire when the pool is exhausted + |
+
+
|
++ |
Prepared Statement Cache Size |
+
+ Number + |
+
+ Determines how many statements are cached per pooled connection. Setting this to zero disables statement caching. + |
+
+
|
++ |
Max Wait |
+
+ Number + |
+
+ The amount of time a client trying to obtain a connection waits for it to be acquired when the pool is exhausted. Setting this value to zero (default) means wait indefinitely. This is equivalent to |
+
+
|
++ |
Max Wait Unit |
+
+
+Enumeration, one of: +
+
|
+
+ A #maxWait. + |
+
+ SECONDS + |
++ |
Max Idle Time |
+
+ Number + |
+
+ Determines how many seconds a connection can remain pooled but unused before being discarded. Setting this value to zero (default) means idle connections never expire. + |
+
+
|
++ |
Additional Properties |
+
+ Object + |
+
+ A map in which keys are the name of a pooling profile configuration property. Does not support the use of expressions. These properties cannot be used to override any of the previously specified properties (like Max Pool Size or Min Pool Size), the main property prevails if an attempt is made to override it. The map’s values contain the actual assignation for each parameter. + |
++ | + |
Max Statement |
+
+ Number + |
+
+ Defines the total number PreparedStatements a DataSource will cache. The pool destroys the least-recently-used PreparedStatement when it reaches the specified limit. When set to |
++ | + |
Test connection on checkout |
+
+ Boolean + |
+
+ Disables connection testing on checkout to improve performance. If set to |
+
+
|
++ |
| Field | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Id |
+
+ Number + |
+
+ Type identifier used by the JDBC driver + |
++ |
+ x + |
+
Type Name |
+
+ String + |
+
+ Name of the data type used by the JDBC driver + |
++ |
+ x + |
+
Class Name |
+
+ String + |
+
+ Indicates which Java class must be used to map the database type + |
++ | + |
| Field | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Fails Deployment |
+
+ Boolean + |
+
+ When the application is deployed, a connectivity test is performed on all connectors. If set to |
++ | + |
Reconnection Strategy |
+
+
|
+
+ The reconnection strategy to use + |
++ | + |
| Field | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Frequency |
+
+ Number + |
+
+ How often to reconnect (in milliseconds) + |
++ | + |
Count |
+
+ Number + |
+
+ The number of reconnection attempts to make + |
++ | + |
blocking |
+
+ Boolean + |
+
+ If set to |
+
+
|
++ |
| Field | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Frequency |
+
+ Number + |
+
+ How often in milliseconds to reconnect + |
++ | + |
blocking |
+
+ Boolean + |
+
+ If set to |
+
+
|
++ |
| Field | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Enabled Protocols |
+
+ String + |
+
+ A comma-separated list of protocols enabled for this context. + |
++ | + |
Enabled Cipher Suites |
+
+ String + |
+
+ A comma-separated list of cipher suites enabled for this context. + |
++ | + |
Trust Store |
+
+
+ |
++ | + | + |
Key Store |
+
+
+ |
++ | + | + |
Revocation Check |
+
+
|
++ | + | + |
| Field | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Path |
+
+ String + |
+
+ The location (which will be resolved relative to the current classpath and file system, if possible) of the trust store. + |
++ | + |
Password |
+
+ String + |
+
+ The password used to protect the trust store. + |
++ | + |
Type |
+
+ String + |
+
+ The type of store used. + |
++ | + |
Algorithm |
+
+ String + |
+
+ The algorithm used by the trust store. + |
++ | + |
Insecure |
+
+ Boolean + |
+
+ If true, no certificate validations will be performed, rendering connections vulnerable to attacks. Use at your own risk. + |
++ | + |
| Field | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Path |
+
+ String + |
+
+ The location (which will be resolved relative to the current classpath and file system, if possible) of the key store. + |
++ | + |
Type |
+
+ String + |
+
+ The type of store used. + |
++ | + |
Alias |
+
+ String + |
+
+ When the key store contains many private keys, this attribute indicates the alias of the key that should be used. If not defined, the first key in the file will be used by default. + |
++ | + |
Key Password |
+
+ String + |
+
+ The password used to protect the private key. + |
++ | + |
Password |
+
+ String + |
+
+ The password used to protect the key store. + |
++ | + |
Algorithm |
+
+ String + |
+
+ The algorithm used by the key store. + |
++ | + |
| Field | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Only End Entities |
+
+ Boolean + |
+
+ Only verify the last element of the certificate chain. + |
++ | + |
Prefer Crls |
+
+ Boolean + |
+
+ Try CRL instead of OCSP first. + |
++ | + |
No Fallback |
+
+ Boolean + |
+
+ Do not use the secondary checking method (the one not selected before). + |
++ | + |
Soft Fail |
+
+ Boolean + |
+
+ Avoid verification failure when the revocation server can not be reached or is busy. + |
++ | + |
| Field | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Url |
+
+ String + |
+
+ The URL of the OCSP responder. + |
++ | + |
Cert Alias |
+
+ String + |
+
+ Alias of the signing certificate for the OCSP response (must be in the trust store), if present. + |
++ | + |
| Field | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Path |
+
+ String + |
+
+ The path to the CRL file. + |
++ | + |
| Field | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Max Idle Time |
+
+ Number + |
+
+ A scalar time value for the maximum amount of time a dynamic configuration instance should be allowed to be idle before it’s considered eligible for expiration + |
++ | + |
Time Unit |
+
+
+Enumeration, one of: +
+
|
+
+ A time unit that qualifies the |
++ | + |
| Field | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Max Redelivery Count |
+
+ Number + |
+
+ The maximum number of times a message can be redelivered and processed unsuccessfully before triggering a process-failed-message + |
++ | + |
Use Secure Hash |
+
+ Boolean + |
+
+ Whether to use a secure hash algorithm to identify a redelivered message. + |
++ | + |
Message Digest Algorithm |
+
+ String + |
+
+ The secure hashing algorithm to use. If this is not set, the default is SHA-256. + |
+
+
|
++ |
Id Expression |
+
+ String + |
+
+ Defines one or more expressions to use to determine when a message has been redelivered. This property can be set only if Use secure hash is set to |
++ | + |
Object Store |
+
+ Object Store + |
+
+ The object store where the redelivery counter for each message is stored + |
++ | + |
| Field | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Key |
+
+ String + |
+
+ The name of the input parameter + |
++ |
+ x + |
+
Type Classifier |
++ | + | + |
+ x + |
+
| Field | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Type |
+
+
+Enumeration, one of: +
+
|
++ | + | + |
Custom Type |
+
+ String + |
++ | + | + |
| Field | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Affected Rows |
+
+ Number + |
++ | + | + |
Generated Keys |
+
+ Object + |
++ | + | + |
| Field | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Initial Buffer Size |
+
+ Number + |
+
+ The number of instances that are initially allowed to be kept in memory to consume the stream and provide random access to it. If the stream contains more data than can fit into this buffer, then the buffer expands according to the Buffer size increment attribute, with an upper limit of Max in memory size. The default value is 100 instances. + |
+
+
|
++ |
Buffer Size Increment |
+
+ Number + |
+
+ Specifies by how much the buffer size expands if it exceeds its initial size. Setting a value of zero or lower means that the buffer should not expand, in which case a |
+
+
|
++ |
Max Buffer Size |
+
+ Number + |
+
+ The maximum amount of memory to use. If more than the specified maximum amount of memory is used, then a `STREAM_MAXIMUM_SIZE_EXCEEDE`D error is raised. A value lower than, or equal to, zero means no limit. + |
++ | + |
| Field | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
In Memory Objects |
+
+ Number + |
+
+ The maximum number of instances to keep in memory. If more than the specified maximum is required, then content starts to buffer on disk. + |
++ | + |
Buffer Unit |
+
+
+Enumeration, one of: +
+
|
+
+ The unit in which maxInMemorySize is expressed + |
++ | + |
| Field | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
Initial Buffer Size |
+
+ Number + |
+
+ The number of instances that are initially allowed to be kept in memory to consume the stream and provide random access to it. If the stream contains more data than can fit into this buffer, then the buffer expands according to the Buffer size increment attribute, with an upper limit of Max in memory size + |
++ | + |
Buffer Size Increment |
+
+ Number + |
+
+ Specifies by how much the buffer size expands if it exceeds its initial size. Setting a value of zero or lower means that the buffer should not expand, in which case a |
++ | + |
Max Buffer Size |
+
+ Number + |
+
+ The maximum amount of memory to use. If more than the specified maximum amount of memory is used, then a |
++ | + |
Buffer Unit |
+
+ Enumeration, one of: + BYTE + KB + MB + GB + |
+
+ The unit in which all these attributes are expressed + |
++ | + |
| Field | +Type | +Description | +Default Value | +Required | +
|---|---|---|---|---|
In Memory Size |
+
+ Number + |
+
+ Defines the maximum memory that the stream should use to keep data in memory. If more than that is consumed content on the disk is buffered. + |
++ | + |
Buffer Unit |
+
+
+Enumeration, one of: +
+
|
+
+ The unit in which Max in memory size is expressed + |
++ | + |
Author: Tan Nguyen
+Last updated: February 13th, 2023
Anypoint Connector for Teradata (Teradata Connector) establishes communication between your Mule app and a Teradata Vantage database, enabling you to connect with your Teradata Vantage instance to load data and run SQL queries in Teradata Vantage tables.
+Date: February 8, 2023
+The initial version is based and extended on MuleSoft’s Database Connector - Mule 4. This version supports the list of features:
+Perform predefined queries, dynamically constructed queries, and template queries that are self-sufficient and customizable.
+Use a source listener operation to read from a database in the data source section of a flow.
+Execute other operations to read and write to a database anywhere in the process section.
+Run a single bulk update to perform multiple SQL requests.
+Make Data Definition Language (DDL) requests.
+Execute stored procedures and SQL scripts.
+Support pooling profile configuration for database connection
+Support auto reconnect to database
+Author: Adam Tworkiewicz
+Last updated: August 30, 2023
This example is a clone of the Mulesoft MySQL sample project. +It demonstrates how to query a Teradata database and expose results over REST API.
+Mulesoft Anypoint Studio. You can download a 30-day trial from https://www.mulesoft.com/platform/studio.
+Access to a Teradata Vantage instance.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
This example Mule service takes an HTTP request, queries the Teradata Vantage database and returns results in JSON format.
+
+The Mule HTTP connector listens for HTTP GET requests with the form: http://<host>:8081/?lastname=<parameter>;.
+The HTTP connector passes the value of <parameter> as one of the message properties to a database connector.
+The database connector is configured to extract this value and use it in this SQL query:
SELECT * FROM hr.employees WHERE LastName = :lastNameAs you can see, we are using parameterized query with reference to the value of the parameter passed to the HTTP connector. +So if the HTTP connector receives http://localhost:8081/?lastname=Smith, the SQL query will be:
+SELECT * FROM employees WHERE last_name = SmithThe database connector instructs the database server to run the SQL query, retrieves the result of the query, and passes it to the Transform message processor which converts the result to JSON. +Since the HTTP connector is configured as request-response, the result is returned to the originating HTTP client.
+Clone Teradata/mule-jdbc-example repository:
git clone https://github.com/Teradata/mule-jdbc-exampleEdit src/main/mule/querying-a-teradata-database.xml, find the Teradata connection string jdbc:teradata://<HOST>/user=<username>,password=<password> and replace Teradata connection parameters to match your environment.
| + + | +
+
+
+Should your Vantage instance be accessible via ClearScape Analytics Experience, you must replace <HOST> with the host URL of your ClearScape Analytics Experience environment. Additionally, the 'user' and 'password' should be updated to reflect your ClearScape Analytics Environment’s username and password. + |
+
Create a sample database in your Vantage instance. +Populate it with sample data.
+ -- create database
+ CREATE DATABASE HR
+ AS PERMANENT = 60e6, SPOOL = 120e6;
+
+ -- create table
+ CREATE SET TABLE HR.Employees (
+ GlobalID INTEGER,
+ FirstName VARCHAR(30),
+ LastName VARCHAR(30),
+ DateOfBirth DATE FORMAT 'YYYY-MM-DD',
+ JoinedDate DATE FORMAT 'YYYY-MM-DD',
+ DepartmentCode BYTEINT
+ )
+ UNIQUE PRIMARY INDEX ( GlobalID );
+
+ -- insert a record
+ INSERT INTO HR.Employees (
+ GlobalID,
+ FirstName,
+ LastName,
+ DateOfBirth,
+ JoinedDate,
+ DepartmentCode
+ ) VALUES (
+ 101,
+ 'Test',
+ 'Testowsky',
+ '1980-01-05',
+ '2004-08-01',
+ 01
+ );Open the project in Anypoint Studio.
+Once in Anypoint Studio, click on Import projects..:
Select Anypoint Studio project from File System:
Use the directory where you cloned the git repository as the Project Root. Leave all other settings at their default values.
Run the example application in Anypoint Studio using the Run menu.
+The project will now build and run. It will take a minute.
Go to your web browser and send the following request: http://localhost:8081/?lastname=Testowsky.
+You should get the following JSON response:
+[
+ {
+ "JoinedDate": "2004-08-01T00:00:00",
+ "DateOfBirth": "1980-01-05T00:00:00",
+ "FirstName": "Test",
+ "GlobalID": 101,
+ "DepartmentCode": 1,
+ "LastName": "Testowsky"
+ }
+]View this document for more information on how to configure a database connector on your machine.
+Access plain Reference material for the Database Connector.
+Learn more about DataSense.
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Adam Tworkiewicz
+Last updated: September 7th, 2021
Native Object Storage (NOS) is a Vantage feature that allows you to query data stored in files in object storage such as AWS S3, Google GCS, Azure Blob or on-prem implementations. It’s useful in scenarios where you want to explore data without building a data pipeline to bring it into Vantage.
+You need access to a Teradata Vantage instance. NOS is enabled in all Vantage editions from Vantage Express through Developer, DYI to Vantage as a Service starting from version 17.10.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
| + + | ++Currently, NOS supports CSV, JSON (as array or new-line delimited), and Parquet data formats. + | +
Let’s say you have a dataset stored as CSV files in an S3 bucket. You want to explore the dataset before you decide if you want to bring it into Vantage. For this scenario, we are going to use a public dataset published by Teradata that contains river flow data collected by the +U.S. Geological Survey. The bucket is at https://td-usgs-public.s3.amazonaws.com/.
+Let’s first have a look at sample CSV data. We take the first 10 rows that Vantage will fetch from the bucket:
+SELECT
+ TOP 10 *
+FROM (
+ LOCATION='/s3/td-usgs-public.s3.amazonaws.com/CSVDATA/'
+) AS d;Here is what I’ve got:
+GageHeight2 Flow site_no datetime Precipitation GageHeight +----------- ----- -------- ---------------- ------------- ----------- +10.9 15300 09380000 2018-06-28 00:30 671 9.80 +10.8 14500 09380000 2018-06-28 01:00 673 9.64 +10.7 14100 09380000 2018-06-28 01:15 672 9.56 +11.0 16200 09380000 2018-06-27 00:00 669 9.97 +10.9 15700 09380000 2018-06-27 00:30 668 9.88 +10.8 15400 09380000 2018-06-27 00:45 672 9.82 +10.8 15100 09380000 2018-06-27 01:00 672 9.77 +10.8 14700 09380000 2018-06-27 01:15 672 9.68 +10.9 16000 09380000 2018-06-27 00:15 668 9.93 +10.8 14900 09380000 2018-06-28 00:45 672 9.72+
We have got plenty of numbers, but what do they mean? To answer this question, we will ask Vantage to detect the schema of the CSV files:
+SELECT
+ *
+FROM (
+ LOCATION='/s3/td-usgs-public.s3.amazonaws.com/CSVDATA/'
+ RETURNTYPE='NOSREAD_SCHEMA'
+) AS d;Vantage will now fetch a data sample to analyze the schema and return results:
+Name Datatype FileType Location +--------------- ----------------------------------- --------- ------------------------------------------------------------------- +GageHeight2 decimal(3,2) csv /S3/s3.amazonaws.com/td-usgs-public/CSVDATA/09513780/2018/06/27.csv +Flow decimal(3,2) csv /S3/s3.amazonaws.com/td-usgs-public/CSVDATA/09513780/2018/06/27.csv +site_no int csv /S3/s3.amazonaws.com/td-usgs-public/CSVDATA/09513780/2018/06/27.csv +datetime TIMESTAMP(0) FORMAT'Y4-MM-DDBHH:MI' csv /S3/s3.amazonaws.com/td-usgs-public/CSVDATA/09513780/2018/06/27.csv +Precipitation decimal(3,2) csv /S3/s3.amazonaws.com/td-usgs-public/CSVDATA/09513780/2018/06/27.csv +GageHeight decimal(3,2) csv /S3/s3.amazonaws.com/td-usgs-public/CSVDATA/09513780/2018/06/27.csv+
We see that the CSV files have 6 columns. For each column, we get the name, the datatype and the file coordinates that were used to infer the schema.
+Now that we know the schema, we can work with the dataset as if it was a regular SQL table. To prove the point, let’s try to do some data aggregation. Let’s get an average temperature per site for sites that collect temperatures.
+SELECT
+ site_no Site_no, AVG(Flow) Avg_Flow
+FROM (
+ LOCATION='/s3/td-usgs-public.s3.amazonaws.com/CSVDATA/'
+) AS d
+GROUP BY
+ site_no
+HAVING
+ Avg_Flow IS NOT NULL;Result:
+Site_no Avg_Flow +-------- --------- +09380000 11 +09423560 73 +09424900 93 +09429070 81+
To register your ad hoc exploratory activity as a permanent source, create it as a foreign table:
+-- If you are running this sample as dbc user you will not have permissions
+-- to create a table in dbc database. Instead, create a new database and use
+-- the newly create database to create a foreign table.
+
+CREATE DATABASE Riverflow
+ AS PERMANENT = 60e6, -- 60MB
+ SPOOL = 120e6; -- 120MB
+
+-- change current database to Riverflow
+DATABASE Riverflow;
+
+CREATE FOREIGN TABLE riverflow
+ USING ( LOCATION('/s3/td-usgs-public.s3.amazonaws.com/CSVDATA/') );
+
+SELECT top 10 * FROM riverflow;Result:
+Location GageHeight2 Flow site_no datetime Precipitation GageHeight +------------------------------------------------------------------- ----------- ---- ------- ------------------- ------------- ---------- +/S3/s3.amazonaws.com/td-usgs-public/CSVDATA/09429070/2018/07/02.csv null null 9429070 2018-07-02 14:40:00 1.21 null +/S3/s3.amazonaws.com/td-usgs-public/CSVDATA/09400815/2018/07/10.csv null 0.00 9400815 2018-07-10 00:30:00 0.00 -0.01 +/S3/s3.amazonaws.com/td-usgs-public/CSVDATA/09400815/2018/07/10.csv null 0.00 9400815 2018-07-10 00:45:00 0.00 -0.01 +/S3/s3.amazonaws.com/td-usgs-public/CSVDATA/09400815/2018/07/10.csv null 0.00 9400815 2018-07-10 01:00:00 0.00 -0.01 +/S3/s3.amazonaws.com/td-usgs-public/CSVDATA/09400815/2018/07/10.csv null 0.00 9400815 2018-07-10 00:15:00 0.00 -0.01 +/S3/s3.amazonaws.com/td-usgs-public/CSVDATA/09429070/2018/07/02.csv null null 9429070 2018-07-02 14:38:00 1.06 null+
This time, the SELECT statement looks like a regular select against an in-database table. If you require subsecond response time when querying the data, there is an easy way to bring the CSV data into Vantage to speed things up. Read on to find out how.
Querying object storage takes time. What if you decided that the data looks interesting and you want to do some more analysis with a solution that will you quicker answers? The good news is that data returned with NOS can be used as a source for CREATE TABLE statements. Assuming you have CREATE TABLE privilege, you will be able to run:
| + + | +
+This query assumes you created database Riverflow and a foreign table called riverflow in the previous step.
+ |
+
-- This query assumes you created database `Riverflow`
+-- and a foreign table called `riverflow` in the previous step.
+
+CREATE MULTISET TABLE riverflow_native (site_no, Flow, GageHeight, datetime)
+AS (
+ SELECT site_no, Flow, GageHeight, datetime FROM riverflow
+) WITH DATA
+NO PRIMARY INDEX;
+
+SELECT TOP 10 * FROM riverflow_native;Result:
+site_no Flow GageHeight datetime +------- ----- ---------- ------------------- +9400815 .00 -.01 2018-07-10 00:30:00 +9400815 .00 -.01 2018-07-10 01:00:00 +9400815 .00 -.01 2018-07-10 01:15:00 +9400815 .00 -.01 2018-07-10 01:30:00 +9400815 .00 -.01 2018-07-10 02:00:00 +9400815 .00 -.01 2018-07-10 02:15:00 +9400815 .00 -.01 2018-07-10 01:45:00 +9400815 .00 -.01 2018-07-10 00:45:00 +9400815 .00 -.01 2018-07-10 00:15:00 +9400815 .00 -.01 2018-07-10 00:00:00+
This time, the SELECT query returned in less than a second. Vantage didn’t have to fetch the data from NOS. Instead, it answered using data that was already on its nodes.
So far, we have used a public bucket. What if you have a private bucket? How do you tell Vantage what credentials it should use?
+It is possible to inline your credentials directly into your query:
+SELECT
+ TOP 10 *
+FROM (
+ LOCATION='/s3/td-usgs-public.s3.amazonaws.com/CSVDATA/'
+ AUTHORIZATION='{"ACCESS_ID":"","ACCESS_KEY":""}'
+) AS d;Entering these credentials all the time can be tedious and less secure. In Vantage, you can create an authorization object that will serve as a container for your credentials:
+CREATE AUTHORIZATION aws_authorization
+ USER 'YOUR-ACCESS-KEY-ID'
+ PASSWORD 'YOUR-SECRET-ACCESS-KEY';You can then reference your authorization object when you create a foreign table:
+CREATE FOREIGN TABLE riverflow
+, EXTERNAL SECURITY aws_authorization
+USING ( LOCATION('/s3/td-usgs-public.s3.amazonaws.com/CSVDATA/') );So far, we have talked about reading and importing data from object storage. Wouldn’t it be nice if we had a way to use SQL to export data from Vantage to object storage? This is exactly what WRITE_NOS function is for. Let’s say we want to export data from riverflow_native table to object storage. You can do so with the following query:
SELECT * FROM WRITE_NOS (
+ ON ( SELECT * FROM riverflow_native )
+ PARTITION BY site_no ORDER BY site_no
+ USING
+ LOCATION('YOUR-OBJECT-STORE-URI')
+ AUTHORIZATION(aws_authorization)
+ STOREDAS('PARQUET')
+ COMPRESSION('SNAPPY')
+ NAMING('RANGE')
+ INCLUDE_ORDERING('TRUE')
+) AS d;Here, we instruct Vantage to take data from riverflow_native and save it in YOUR-OBJECT-STORE-URI bucket using parquet format. The data will be split into files by site_no attribute. The files will be compressed.
In this quick start we have learned how to read data from object storage using Native Object Storage (NOS) functionality in Vantage. NOS supports reading and importing data stored in CSV, JSON and Parquet formats. NOS can also export data from Vantage to object storage.
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Adam Tworkiewicz
+Last updated: January 5th, 2022
This how-to demonstrates how to use the ODBC driver with Teradata Vantage on Ubuntu.
+Access to a Teradata Vantage instance.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
Root access to a Ubuntu machine.
+Install dependencies:
+apt update && DEBIAN_FRONTEND=noninteractive apt install -y wget unixodbc unixodbc-dev iodbc python3-pipInstall Teradata ODBC driver for Ubuntu:
+wget https://downloads.teradata.com/download/cdn/connectivity/odbc/17.10.x.x/tdodbc1710__ubuntu_x8664.17.10.00.14-1.tar.gz \
+ && tar -xzf tdodbc1710__ubuntu_x8664.17.10.00.14-1.tar.gz \
+ && dpkg -i tdodbc1710/tdodbc1710-17.10.00.14-1.x86_64.debConfigure ODBC, by creating file /etc/odbcinst.ini with the following content:
[ODBC Drivers]
+Teradata Database ODBC Driver 17.10=Installed
+
+[Teradata Database ODBC Driver 17.10]
+Description=Teradata Database ODBC Driver 17.10
+Driver=/opt/teradata/client/17.10/odbc_64/lib/tdataodbc_sb64.soWe will validate the installation with a sample Python application. Create test.py file with the following content.
+Replace DBCName=192.168.86.33;UID=dbc;PWD=dbc with the IP address of your Teradata Vantage instance, username and password:
import pyodbc
+
+print(pyodbc.drivers())
+
+cnxn = pyodbc.connect('DRIVER={Teradata Database ODBC Driver 17.10};DBCName=192.168.86.33;UID=dbc;PWD=dbc;')
+cursor = cnxn.cursor()
+
+cursor.execute("SELECT CURRENT_DATE")
+for row in cursor.fetchall():
+ print(row)
+EOFRun the test application:
+python3 test.pyYou should get output similar to:
+['ODBC Drivers', 'Teradata Database ODBC Driver 17.10'] +(datetime.date(2022, 1, 5), )+
This how-to demonstrated how to use ODBC with Teradata Vantage on Ubuntu. The how-to shows how to install the ODBC Teradata driver and the dependencies. It then shows how to configure ODBC and validate connectivity with a simple Python application.
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Adam Tworkiewicz
+Last updated: September 12th, 2022
This how-to demonstrates how to create a connection to Teradata Vantage with DBeaver.
+Access to a Teradata Vantage instance.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
DBeaver installed. See DBeaver Community or DBeaver PRO for installation options.
+Start the new connection wizard by clicking on the plug icon (
) in the upper left corner of the application window or go to Database → New Database Connection.
On Select your database screen, start typing teradata and select the Teradata icon.
+On the main tab, you need to set all primary connection settings. The required ones include Host, Port, Database, Username, and Password.
| + + | +
+With DBeaver PRO, you can not only use the standard ordering of tables but also hierarchically link tables to a specific database or user. Expanding and collapsing the databases or users will help you navigate from one area to another without swamping the Database Navigator window. Check the Show databases and users hierarchically box to enable this setting.
+ |
+
| + + | +
+In many environments Teradata Vantage can only be accessed using the TLS protocol. When in DBeaver PRO, check Use TLS protocol option to enable TLS.
+ |
+
+Click on Finish.
If your database cannot be accessed directly, you can use an SSH tunnel. All settings are available on the SSH tab. DBeaver supports the following authentication methods: user/password, public key, SSH agent authentication.
+This how-to demonstrated how to create a connection to Teradata Vantage with DBeaver.
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Igor Machin, Ambrose Inman
+Last updated: November 18th, 2022
This tutorial demonstrates how to install Airflow on an AWS EC2 VM, configure the workflow to use dbt, and run it against a Teradata Vantage database. Airflow is a task scheduling tool that is typically used to build data pipelines to process and load data. In this example, we go through the Airflow installation process, which creates a Docker-based Airflow environment. Once Airflow is installed, we run several Airflow DAG (Direct Acyclic Graph, or simply workflow) examples that load data into a Teradata Vantage database.
+Access to AWS (Amazon Web Services) with permissions to create a VM.
+| + + | ++This tutorial can be adjusted to other compute platforms or even on a bare metal machine as long as it has a computing and storage capacity comparable to the machine mentioned in this document (t2.2xlarge EC2 on AWS with approximately 100GB of storage) and is connected to the internet. If you decide to use a different compute platform, some steps in the tutorial will have to be altered. + | +
An SSH client.
+ +Access to a Teradata Vantage database. If you don’t have access to Teradata Vantage, explore Vantage Express - a free edition for developers.
+Go to the AWS EC2 console and click on Launch instance.
Select Red Hat for OS image.
Select t2.2xlarge for instance type.
Create a new key pair or use an existing one.
+Apply network settings that will allow you ssh to the server and the server will have outbound connectivity to the Internet. Usually, applying the default settings will do.
+Assign 100GB of storage.
+ssh to the machine using ec2-user user.
Check if python is installed (should be Python 3.7 or higher). Type python or python3 on the command line.
If python is not installed (you are getting command not found message) run the commands below to install it. The commands may require you to confirm the installation by typing y and enter.
sudo yum install python3
+# create a virtual environment for the project
+sudo yum install python3-pip
+sudo pip3 install virtualenvCreate the Airflow directory structure (from the ec2-user home directory /home/ec2-user)
+mkdir airflow
+cd airflow
+mkdir -p ./dags ./logs ./plugins ./data ./config ./data
+echo -e "AIRFLOW_UID=$(id -u)" > .envUse your preferred file transfer tool (scp, PuTTY, MobaXterm, or similar) to upload airflow.cfg file to airflow/config directory.
Docker is a containerization tool that allows us to install Airflow in a containerized environment.
+| + + | +
+The steps must be executed in airflow directory.
+ |
+
Uninstall podman (RHEL containerization tool)
+sudo yum remove docker \
+docker-client \
+docker-client-latest \
+docker-common \
+docker-latest \
+docker-latest-logrotate \
+docker-logrotate \
+docker-engine \
+podman \
+runcInstall yum utilities
+sudo yum install -y yum-utilsAdd docker to yum repository.
+sudo yum-config-manager \
+--add-repo \
+https://download.docker.com/linux/centos/docker-ce.repoInstall docker.
+sudo yum install docker-ce docker-ce-cli containerd.ioStart docker as a service. The first command runs the docker service automatically when the system starts up next time. The second command starts Docker now.
+sudo systemctl enable docker
+sudo systemctl start dockerCheck if Docker is installed correctly. This command should return an empty list of containers (since we have not started any container yet):
+sudo docker psdocker-compose and docker environment configuration filesUpload docker-compose.yaml and Dockerfile files to the VM and save them in airflow directory.
| + + | +
+ What
+docker-compose.yaml and Dockerfile do
+
+
+
+If you would like to learn more what |
+
Install docker-compose (necessary to run the yaml file).
+| + + | ++The instructions are based on version 1.29.2. Check out https://github.com/docker/compose/releases site for the latest release and update the command below as needed. + | +
sudo curl -L https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
+sudo chmod +x /usr/local/bin/docker-compose
+sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-composeTest your docker-compose installation. The command should return the docker-compose version, for example docker-compose version 1.29.2, build 5becea4c:
docker-compose --version| + + | +
+These steps set up a sample dbt project. dbt tool itself will be installed on the containers later by docker-compose.
+ |
+
Install git:
+sudo yum install gitGet the sample jaffle shop dbt project:
+| + + | +
+The dbt directories will be created under the home directory (not under airflow). The home directory in our example is /home/ec2-user.
+ |
+
# move to home dir
+cd
+mkdir dbt
+cd dbt
+git clone https://github.com/Teradata/jaffle_shop-dev.git jaffle_shop
+cd jaffle_shop
+mkdir target
+chmod 777 target
+echo '' > target/index.html
+chmod o+w target/index.htmlCreate the airflowtest and jaffle_shop users/databases on your Teradata database by using your preferred database tool (Teradata Studio Express, bteq or similar). Log into the database as dbc, then execute the commands (change the passwords if needed):
CREATE USER "airflowtest" FROM "dbc" AS PERM=5000000000 PASSWORD="abcd";
+CREATE USER "jaffle_shop" FROM "dbc" AS PERM=5000000000 PASSWORD="abcd";Create the dbt configuration directory:
+cd
+mkdir .dbtCopy profiles.yml into the .dbt directory.
Edit the file so it corresponds to your Teradata database setup. At a minium, you will need to change the host, user and password. Use jaffle_shop user credentials you set up in step 3.
Run the docker environment creation script in the airflow directory where Dockerfile and docker-compose.yaml:
cd ~/airflow
+sudo docker-compose up --buildThis can take 5-10 minutes, when the installation is complete you should see on the screen a message similar to this:
+airflow-webserver_1 | 127.0.0.1 - - [13/Sep/2022:00:20:48 +0000] "GET /health HTTP/1.1" 200 187 "-" "curl/7.74.0"This means the Airflow webserver is ready to accept calls.
+Now Airflow should be up. The terminal session that we were using during the installation will be used to display log messages, so it is recommended +to open another terminal session for subsequent steps. To check the Airflow installation type:
+sudo docker psThe result should be something like:
+CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
+60d50d9f43f5 apache/airflow:2.2.4 "/usr/bin/dumb-init …" 18 minutes ago Up 18 minutes (healthy) 8080/tcp airflow_airflow-scheduler_1
+e2b46ec98274 apache/airflow:2.2.4 "/usr/bin/dumb-init …" 18 minutes ago Up 18 minutes (healthy) 8080/tcp airflow_airflow-worker_3_1
+7b44004c7277 apache/airflow:2.2.4 "/usr/bin/dumb-init …" 18 minutes ago Up 18 minutes (healthy) 8080/tcp airflow_airflow-worker_1_1
+4017b8ce9235 apache/airflow:2.2.4 "/usr/bin/dumb-init …" 18 minutes ago Up 18 minutes (healthy) 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp airflow_airflow-webserver_1
+3cc407e2d565 apache/airflow:2.2.4 "/usr/bin/dumb-init …" 18 minutes ago Up 18 minutes (healthy) 0.0.0.0:5555->5555/tcp, :::5555->5555/tcp, 8080/tcp airflow_flower_1
+340a83b202e3 apache/airflow:2.2.4 "/usr/bin/dumb-init …" 18 minutes ago Up 18 minutes (healthy) 8080/tcp airflow_airflow-triggerer_1
+82198f0d8b84 apache/airflow:2.2.4 "/usr/bin/dumb-init …" 18 minutes ago Up 18 minutes (healthy) 8080/tcp airflow_airflow-worker_2_1
+382c3077c1e5 redis:latest "docker-entrypoint.s…" 18 minutes ago Up 18 minutes (healthy) 6379/tcp airflow_redis_1
+8a3be8d8a7f4 nginx "/docker-entrypoint.…" 18 minutes ago Up 18 minutes (healthy) 0.0.0.0:4000->80/tcp, :::4000->80/tcp airflow_nginx_1
+9ca888e9e8df postgres:13 "docker-entrypoint.s…" 18 minutes ago Up 18 minutes (healthy) 5432/tcp airflow_postgres_1OPTIONAL: If you want to delete the docker installation (for example to update the docker-compose.yaml and the Dockerfile files and recreate a different environment), the command is (from the airflow directory where these files are located):
+sudo docker-compose down --volumes --rmi allOnce the stack is down, update the configuration files and restart by running the command in step 1.
+To test if the Airflow web UI works, type the following urls on your browser. Replace <VM_IP_ADDRESS> with the external IP address of the VM:
DAG UI: http://<YOUR_IP_ADDRESS>:8080/home - username: airflow / password: airflow
Flower Airflow UI (worker control): http://<YOUR_IP_ADDRESS>:5555/
Copy airflow_dbt_integration.py, db_test_example_dag.py, discover_dag.txt, variables.json files to /home/ec2-user/airflow/dags.
Examine the files:
+airflow_dbt_integration.py - a simple Teradata sql example that creates a few tables and runs queries.
db_test_example_dag.py - runs a dbt example (i.e. integration of dbt and airflow with a Teradata database). In this example a fictitious jaffle_shop data model is created, loaded and the documentation for this project is produced (you can view it by pointing your browser to http://<VM_IP_ADDRESS>:4000/)
| + + | +
+ Adjust
+db_test_example_dag.py
+
+
|
+
discover_dag.py - an example on how to load various types of data files (CSV, Parquet, JSON). The source code file contains comments that explain what the program does and how to use it. This example relies on variables.json file. The file needs to be imported into Airflow. It will happen in subsequent steps.
Wait for a few minutes until these dag files are picked up by the airflow tool. Once they are picked up they will appear on the list of dags on the Airflow home page.
+Import variables.json file as a variable file into Airflow:
Click on Admin → Variables menu item to go to the Variables page
+Click on Choose File, then select variable.json in your file explorer and click on Import Variables
+Edit the variables to match your environment
+Run the dags from the UI and check the logs.
+This tutorial aimed at providing a hands on exercise on how to install an Airflow environment on a Linux server and how to use Airflow to interact with a Teradata Vantage database. An additional example is provided on how to integrate Airflow and the data modelling and maintenance tool dbt to create and load a Teradata Vantage database.
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Ravi Chillanki
+Last updated: August 4th, 2023
This tutorial shows an approach to creating a dbt pipeline that takes raw data and turns it FEAST features. The pipeline leverages 'ClearScape Analytics functions' for data transformations. The output of the transformations is loaded into FEAST to materialize features that can be used in ML models.
+dbt (Data Build Tool) is a data transformation tool that is the cornerstone of the Modern Data Stack. It takes care of the T in ELT (Extract Load Transform). The assumption is that some other process brings raw data into your data warehouse or lake. This data then needs to be transformed.
+Feast (Feature Store) is a flexible data system that utilizes existing technology to manage and provide machine learning features to real-time models. It allows for customization to meet specific needs. It also allows us to make features consistently available for training and serving, avoid data leakage and decouple ML from data infrastructure.
+Access to a Teradata Vantage database instance. +NOTE: If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com.
+Feast-Teradata installed locally. See Feast-Teradata installation instructions
+dbt installed locally. See dbt installation instructions
+The goal is to create a data pipeline with Teradata Vantage as a source, and perform data transformation on some variables in dbt. The principle transformation of data we do in dbt is the one-hot encoding of several columns like gender, marital status, state code, etc. On top of that, the account type column data will be transformed by performing aggregation operations on a couple of columns. All of this together generates the desired dataset with transformed data. The transformed dataset is used as an input into FEAST to store features. Features are then used to generate a training dataset for models.
+Create a new python environment to manage dbt, feast and their dependencies. Activate the environment:
+python3 -m venv env
+source env/bin/activateClone the tutorial repository and change the directory to the project directory:
+git clone https://github.com/Teradata/tdata-pipeline.gitThe directory structure of the project cloned looks like this:
+tdata-pipeline/ + feature_repo/ + feature_views.py + feature_store.yml + dbt_transformation/ + ... + macros + models + ... + generate_training_data.py + CreateDB.sql + dbt_project.yml+
teddy_bank is a fictitious dataset of banking customers, consisting of mainly 3 tables customers, accounts, and +transactions, with the following entity-relationship diagram:
+
+dbt takes this raw data and builds the following model, which is more suitable for ML modeling and analytics tools:
+
+Create file $HOME/.dbt/profiles.yml with the following content. Adjust <host>, <user>, <password> to match your Teradata instance.
| + + | +
+ Database setup
+
+
+The following dbt profile points to a database called |
+
dbt_transformation:
+ target: dev
+ outputs:
+ dev:
+ type: teradata
+ host: <host>
+ user: <user>
+ password: <password>
+ schema: teddy_bank
+ tmode: ANSIValidate the setup:
+dbt debugIf the debug command returned errors, you likely have an issue with the content of profiles.yml.
Feast configuration addresses connection to your Vantage database. The yaml file created while initializing the feast
+project, $HOME/.feast/feature_repo/feature_store.yml can hold the details of offline storage, online storage, provider
+and registry. Adjust <host>, <user>, <password> to match your Teradata instance.
| + + | +
+ Database setup
+
+
+The following dbt profile points to a database called |
+
project: td_pipeline
+registry:
+ registry_type: sql
+ path: teradatasql://<user>:<password>@<hostIP>/?database=teddy_bank&LOGMECH=TDNEGO
+provider: local
+offline_store:
+ type: feast_teradata.offline.teradata.TeradataOfflineStore
+ host: <host>
+ database: teddy_bank
+ user: <user>
+ password: <password>
+ log_mech: TDNEGO
+entity_key_serialization_version: 2TeradataSource: Data Source for features stored in Teradata (Enterprise or Lake) or accessible via a Foreign Table from Teradata (NOS, QueryGrid)
Entity: A collection of semantically related features
Feature View: A feature view is a group of feature data from a specific data source. Feature views allow you to consistently define features and their data sources, enabling the reuse of feature groups across a project
DBT_source = TeradataSource( database=dbload, table=f"Analytic_Dataset", timestamp_field="event_timestamp")
+
+customer = Entity(name = "customer", join_keys = ['cust_id'])
+
+ads_fv = FeatureView(name="ads_fv",entities=[customer],source=DBT_source, schema=[
+ Field(name="age", dtype=Float32),
+ Field(name="income", dtype=Float32),
+ Field(name="q1_trans_cnt", dtype=Int64),
+ Field(name="q2_trans_cnt", dtype=Int64),
+ Field(name="q3_trans_cnt", dtype=Int64),
+ Field(name="q4_trans_cnt", dtype=Int64),
+ ],)The approach to generating training data can vary. Depending upon the requirements, 'entitydf' may be joined with the source data tables using the feature views mapping. Here is a sample function that generates a training dataset.
+def get_Training_Data():
+ # Initialize a FeatureStore with our current repository's configurations
+ store = FeatureStore(repo_path="feature_repo")
+ con = create_context(host = os.environ["latest_vm"], username = os.environ["dbc_pwd"],
+ password = os.environ["dbc_pwd"], database = "EFS")
+ entitydf = DataFrame('Analytic_Dataset').to_pandas()
+ entitydf.reset_index(inplace=True)
+ print(entitydf)
+ entitydf = entitydf[['cust_id','event_timestamp']]
+ training_data = store.get_historical_features(
+ entity_df=entitydf,
+ features=[
+ "ads_fv:age"
+ ,"ads_fv:income"
+ ,"ads_fv:q1_trans_cnt"
+ ,"ads_fv:q2_trans_cnt"
+ ,"ads_fv:q3_trans_cnt"
+ ,"ads_fv:q4_trans_cnt"
+ ],
+ full_feature_names=True
+ ).to_df()
+
+ return training_dataThis tutorial demonstrated how to use dbt and FEAST with Teradata Vantage. The sample project takes raw data from Teradata Vantage and produces features with dbt. Metadata of features that form the base to generate a training dataset for a model was then created with FEAST; all its corresponding tables that create the feature store, are also generated at runtime within the same database.
+Enabling highly scalable feature store with Teradata Vantage and FEAST
+Git repository for this project.
+Author: Jeremy Yu
+Last updated: May 18th, 2022
Access to a Teradata Vantage instance, version 17.10 or higher.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
KNIME installed locally. See KNIME installation instructions for details.
+Go to https://downloads.teradata.com/download/connectivity/jdbc-driver (first time users will need to register) and download the latest version of the JDBC driver.
+Unzip the downloaded file. You will find terajdbc4.jar file.
In KNIME, click on File → Preference. Under Databases, click Add:
+Register a new database driver. Provide values for ID, Name and Description like below. Click on Add file and point to the .jar file you downloaded earlier. Click on the Find driver classes and the Driver class: should populate with the jdbc.TeraDriver:
+Click Apply and Close:
+To test the connection, create a new KNIME workflow and add a Database Reader (legacy) node by dragging it to the workspace to the right:
+
+Right-click on the Database Reader (legacy) to configure settings. Select com.teradata.jdbc.Teradriver from the drop-down:
+Enter the name of the Vantage server and login mechanism, e.g.:
+
+To test connection, enter SQL statement in box in lower right. For example, enter SELECT * FROM DBC.DBCInfoV and click Apply to close the dialog:
+Execute the node to test the connection:
+
+The node will show a green light when run successfully. Right-click and select Data from Database to view the results:
+
+This how-to demonstrats how to connect from KNIME Analytics Platform to Teradata Vantage.
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Remi Turpaud
+Last updated: April 15th, 2022
Time series is series of data points indexed in time order. It is data continuously produced and collected by a wide range of applications and devices including but not limited to Internet of Things. Teradata Vantage offers various functionalities to simplify time series data analysis.
+You need access to a Teradata Vantage instance. Times series functionalities and NOS are enabled in all Vantage editions from Vantage Express through Developer, DYI to Vantage as a Service starting from version 17.10.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
Our sample data sets are available on S3 bucket and can be accessed from Vantage directly using Vantage NOS. Data is in CSV format and let’s ingest them into Vantage for our time series analysis.
+Let’s have a look at the data first. Below query will fetch 10 rows from S3 bucket.
+SELECT TOP 10 * FROM (
+ LOCATION='/s3/nos-demo-apj.s3.amazonaws.com/taxi/2014/11/data_2014-11-25.csv'
+) AS d;Here is what we’ve got:
+Location vendor_id pickup_datetime dropoff_datetime passenger_count trip_distance pickup_longitude pickup_latitude rate_code store_and_fwd_flag dropoff_longitude dropoff_latitude payment_type fare_amount surcharge mta_tax tip_amount tolls_amount total_amount +------------------------------------------------------------------ --------- ----------------- ----------------- ---------------- -------------- ----------------- ---------------- ---------- ------------------- ------------------ ----------------- ------------- ------------ ---------- -------- ---------- ------------ ------------ +/S3/s3.amazonaws.com/nyc-tlc/csv_backup/yellow_tripdata_2013-11.csv CMT 25/11/2013 15:18 25/11/2013 15:33 1 1 -73.992423 40.749517 1 N -73.98816 40.746557 CRD 10 0 0.5 2.22 0 12.72 +/S3/s3.amazonaws.com/nyc-tlc/csv_backup/yellow_tripdata_2013-11.csv CMT 25/11/2013 5:34 25/11/2013 5:48 1 3.6 -73.971555 40.794548 1 N -73.975399 40.755404 CRD 14.5 0.5 0.5 1 0 16.5 +/S3/s3.amazonaws.com/nyc-tlc/csv_backup/yellow_tripdata_2013-11.csv CMT 25/11/2013 8:31 25/11/2013 8:55 1 5.9 -73.94764 40.830465 1 N -73.972323 40.76332 CRD 21 0 0.5 3 0 24.5 +/S3/s3.amazonaws.com/nyc-tlc/csv_backup/yellow_tripdata_2013-11.csv CMT 25/11/2013 7:00 25/11/2013 7:04 1 1.2 -73.983357 40.767193 1 N -73.978394 40.75558 CRD 5.5 0 0.5 1 0 7 +/S3/s3.amazonaws.com/nyc-tlc/csv_backup/yellow_tripdata_2013-11.csv CMT 25/11/2013 15:24 25/11/2013 15:30 1 0.5 -73.982313 40.764827 1 N -73.982129 40.758889 CRD 5.5 0 0.5 3 0 9 +/S3/s3.amazonaws.com/nyc-tlc/csv_backup/yellow_tripdata_2013-11.csv CMT 25/11/2013 15:53 25/11/2013 16:00 1 0.6 -73.978104 40.752966 1 N -73.985756 40.762685 CRD 6 1 0.5 1 0 8.5 +/S3/s3.amazonaws.com/nyc-tlc/csv_backup/yellow_tripdata_2013-11.csv CMT 25/11/2013 6:49 25/11/2013 7:04 1 3.8 -73.976005 40.744481 1 N -74.016063 40.717298 CRD 14 0 0.5 2.9 0 17.4 +/S3/s3.amazonaws.com/nyc-tlc/csv_backup/yellow_tripdata_2013-11.csv CMT 25/11/2013 21:20 25/11/2013 21:26 1 1.1 -73.946371 40.775369 1 N -73.95309 40.785103 CRD 6.5 0.5 0.5 1.5 0 9 +/S3/s3.amazonaws.com/nyc-tlc/csv_backup/yellow_tripdata_2013-11.csv CMT 25/11/2013 10:02 25/11/2013 10:17 1 2.2 -73.952625 40.780962 1 N -73.98163 40.777978 CRD 12 0 0.5 2 0 14.5 +/S3/s3.amazonaws.com/nyc-tlc/csv_backup/yellow_tripdata_2013-11.csv CMT 25/11/2013 9:43 25/11/2013 10:02 1 3.3 -73.982013 40.762507 1 N -74.006854 40.719582 CRD 15 0 0.5 2 0 17.5+
Let’s extract the complete data and bring it into Vantage for further analysis.
+CREATE TABLE trip
+(
+ vendor_id varchar(10) character set latin NOT casespecific,
+ rate_code integer,
+ pickup_datetime timestamp(6),
+ dropoff_datetime timestamp(6),
+ passenger_count smallint,
+ trip_distance float,
+ pickup_longitude float,
+ pickup_latitude float,
+ dropoff_longitude float,
+ dropoff_latitude float
+)
+NO PRIMARY INDEX ;
+
+
+
+INSERT INTO trip
+SELECT TOP 200000 vendor_id ,
+ rate_code,
+ pickup_datetime,
+ dropoff_datetime ,
+ passenger_count,
+ trip_distance ,
+ pickup_longitude,
+ pickup_latitude ,
+ dropoff_longitude ,
+ dropoff_latitude FROM (
+ LOCATION='/s3/nos-demo-apj.s3.amazonaws.com/taxi/2014/11/data_2014-11-25.csv'
+) AS d;Result:
+200000 rows affected.+
Vantage will now fetch the data from S3 and insert into trip table we just created.
+Now that we are familiar with the data set, we can use Vantage capabilities to quickly analyse the data set. First, let’s identify how many passengers are being picked up by hour in the month of November.
+SELECT TOP 10
+ $TD_TIMECODE_RANGE
+ ,begin($TD_TIMECODE_RANGE) time_bucket_start
+ ,sum(passenger_count) passenger_count
+FROM trip
+WHERE extract(month from pickup_datetime)=11
+GROUP BY TIME(HOURS(1))
+USING TIMECODE(pickup_datetime)
+ORDER BY 1;For further reading on GROUP BY TIME.
+Result:
+TIMECODE_RANGE time_bucket_start passenger_count +--------------------------------------------------------- --------------------------------- ---------------- +(2013-11-04 11:00:00.000000, 2013-11-04 12:00:00.000000) 2013-11-04 11:00:00.000000-05:00 4 +(2013-11-04 12:00:00.000000, 2013-11-04 13:00:00.000000) 2013-11-04 12:00:00.000000-05:00 2 +(2013-11-04 14:00:00.000000, 2013-11-04 15:00:00.000000) 2013-11-04 14:00:00.000000-05:00 5 +(2013-11-04 15:00:00.000000, 2013-11-04 16:00:00.000000) 2013-11-04 15:00:00.000000-05:00 2 +(2013-11-04 16:00:00.000000, 2013-11-04 17:00:00.000000) 2013-11-04 16:00:00.000000-05:00 9 +(2013-11-04 17:00:00.000000, 2013-11-04 18:00:00.000000) 2013-11-04 17:00:00.000000-05:00 11 +(2013-11-04 18:00:00.000000, 2013-11-04 19:00:00.000000) 2013-11-04 18:00:00.000000-05:00 41 +(2013-11-04 19:00:00.000000, 2013-11-04 20:00:00.000000) 2013-11-04 19:00:00.000000-05:00 2791 +(2013-11-04 20:00:00.000000, 2013-11-04 21:00:00.000000) 2013-11-04 20:00:00.000000-05:00 15185 +(2013-11-04 21:00:00.000000, 2013-11-04 22:00:00.000000) 2013-11-04 21:00:00.000000-05:00 27500+
Yes, this can also be achieved by extracting the hour from time and then aggregating - it’s additional code/work, but can be done without timeseries specific functionality.
+But, now let’s go a step further to identify how many passengers are being picked up and what is the average trip duration by vendor every 15 minutes in November.
+SELECT TOP 10
+ $TD_TIMECODE_RANGE,
+ vendor_id,
+ SUM(passenger_count),
+ AVG((dropoff_datetime - pickup_datetime ) MINUTE (4)) AS avg_trip_time_in_mins
+FROM trip
+GROUP BY TIME (MINUTES(15) AND vendor_id)
+USING TIMECODE(pickup_datetime)
+WHERE EXTRACT(MONTH FROM pickup_datetime)=11
+ORDER BY 1,2;Result:
+TIMECODE_RANGE vendor_id passenger_count avg_trip_time_in_mins +-------------------------------------------------------- ---------- ---------------- ---------------------- +(2013-11-04 11:00:00.000000, 2013-11-04 11:15:00.000000) VTS 1 16 +(2013-11-04 11:15:00.000000, 2013-11-04 11:30:00.000000) VTS 1 10 +(2013-11-04 11:45:00.000000, 2013-11-04 12:00:00.000000) VTS 2 6 +(2013-11-04 12:00:00.000000, 2013-11-04 12:15:00.000000) VTS 1 11 +(2013-11-04 12:15:00.000000, 2013-11-04 12:30:00.000000) VTS 1 57 +(2013-11-04 14:15:00.000000, 2013-11-04 14:30:00.000000) VTS 1 3 +(2013-11-04 14:30:00.000000, 2013-11-04 14:45:00.000000) VTS 2 19 +(2013-11-04 14:45:00.000000, 2013-11-04 15:00:00.000000) VTS 2 9 +(2013-11-04 15:15:00.000000, 2013-11-04 15:30:00.000000) VTS 1 11 +(2013-11-04 15:30:00.000000, 2013-11-04 15:45:00.000000) VTS 1 31+
This is the power of Vantage time series functionality. Without needing complicated, cumbersome logic we are able to find average trip duration by vendor every 15 minutes just by modifying the group by time clause. Let’s now look at how simple it is to build moving averages based on this. First, let’s start by creating a view as below.
+REPLACE VIEW NYC_taxi_trip_ts as
+SELECT
+ $TD_TIMECODE_RANGE time_bucket_per
+ ,vendor_id
+ ,sum(passenger_count) passenger_cnt
+ ,avg(CAST((dropoff_datetime - pickup_datetime MINUTE(4) ) AS INTEGER)) avg_trip_time_in_mins
+FROM trip
+GROUP BY TIME (MINUTES(15) and vendor_id)
+USING TIMECODE(pickup_datetime)
+WHERE extract(month from pickup_datetime)=11Let’s calculate a 2 hours moving average on our 15-minutes time series. 2 hour is 8 * 15 minutes periods.
+SELECT * FROM MovingAverage (
+ ON NYC_taxi_trip_ts PARTITION BY vendor_id ORDER BY time_bucket_per
+ USING
+ MAvgType ('S')
+ WindowSize (8)
+ TargetColumns ('passenger_cnt')
+) AS dt
+WHERE begin(time_bucket_per)(date) = '2014-11-25'
+ORDER BY vendor_id, time_bucket_per;Result:
+time_bucket_per vendor_id passenger_cnt avg_trip_time_in_mins passenger_cnt_smavg +--------------------------------------------------------- -------------- ---------------------- -------------------- -------------------- +(2013-11-04 14:45:00.000000, 2013-11-04 15:00:00.000000) VTS 2 9 1.375 +(2013-11-04 15:15:00.000000, 2013-11-04 15:30:00.000000) VTS 1 11 1.375 +(2013-11-04 15:30:00.000000, 2013-11-04 15:45:00.000000) VTS 1 31 1.375 +(2013-11-04 16:15:00.000000, 2013-11-04 16:30:00.000000) VTS 2 16 1.375 +(2013-11-04 16:30:00.000000, 2013-11-04 16:45:00.000000) VTS 1 3 1.375 +(2013-11-04 16:45:00.000000, 2013-11-04 17:00:00.000000) VTS 6 38 2 +(2013-11-04 17:15:00.000000, 2013-11-04 17:30:00.000000) VTS 2 29.5 2.125 +(2013-11-04 17:45:00.000000, 2013-11-04 18:00:00.000000) VTS 9 20.33333333 3 +(2013-11-04 18:00:00.000000, 2013-11-04 18:15:00.000000) VTS 6 23.4 3.5 +(2013-11-04 18:15:00.000000, 2013-11-04 18:30:00.000000) VTS 4 15.66666667 3.875 +(2013-11-04 18:30:00.000000, 2013-11-04 18:45:00.000000) VTS 8 24.5 4.75 +(2013-11-04 18:45:00.000000, 2013-11-04 19:00:00.000000) VTS 23 38.33333333 7.375 +(2013-11-04 19:00:00.000000, 2013-11-04 19:15:00.000000) VTS 195 26.61538462 31.625 +(2013-11-04 19:15:00.000000, 2013-11-04 19:30:00.000000) VTS 774 13.70083102 127.625 +(2013-11-04 19:30:00.000000, 2013-11-04 19:45:00.000000) VTS 586 12.38095238 200.625 +(2013-11-04 19:45:00.000000, 2013-11-04 20:00:00.000000) VTS 1236 15.54742097 354 +(2013-11-04 20:00:00.000000, 2013-11-04 20:15:00.000000) VTS 3339 11.78947368 770.625 +(2013-11-04 20:15:00.000000, 2013-11-04 20:30:00.000000) VTS 3474 10.5603396 1204.375 +(2013-11-04 20:30:00.000000, 2013-11-04 20:45:00.000000) VTS 3260 12.26484323 1610.875 +(2013-11-04 20:45:00.000000, 2013-11-04 21:00:00.000000) VTS 5112 12.05590062 2247+
| + + | ++In addition to above time series operations, Vantage also provides a special time series tables with Primary Time Index (PTI). These are regular Vantage tables with PTI defined rather than a Primary Index (PI). Though tables with PTI are not mandatory for time series functionality/operations, PTI optimizes how the time series data is stored physically and hence improves performance considerably compared to regular tables. + | +
In this quick start we have learnt how easy it is to analyse time series datasets using Vantage’s time series capabilities.
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Sudha Vedula
+Last updated: May 29th, 2023
Teradata Query Service is a REST API for Vantage that you can use to run standard SQL statements without managing client-side drivers. Use Query Service if you are looking to query and access the Analytics Database through a REST API.
+This how-to provides examples of common use cases to help you get started with Query Service API.
+Before starting, make sure you have:
+Access to a VantageCloud system where Query Service is provisioned, or a VantageCore with Query Service enabled connectivity. If you are an admin and need to install Query Service, see Query Service Installation, Configuration, and Usage Guide.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
Query Service hostname and system name
+Authorization credentials to connect to the database
+Having trouble with the prerequisites? Contact Teradata for setup information.
+When using the examples, please keep in mind that:
+The examples in this document use Python, and you can use these to create examples in your language of choice.
+The examples provided here are complete and ready for you to use, although most require a little customization.
+The examples in this document use the URL https://<QS_HOSTNAME>:1443/.
Replace the following variables with your own value:
+<QS_HOSTNAME>: Server where Query Service is installed
<SYSTEM_NAME>: Preconfigured alias of the system
| + + | +
+
+
+If your Vantage instance is provided through ClearScape Analytics Experience,<QS_HOSTNAME>, is the host URL of your ClearScape Analytics Experience environment, <SYSTEM_NAME> is 'local'. + |
+
Provide valid credentials to access the target Analytics Database using HTTP Basic or JWT authentication.
+The database username and password are combined into a string ("username : password") which is then encoded using Base64. The API response contains the authorization method and encoded credentials.
Request
+import requests
+import json
+import base64
+requests.packages.urllib3.disable_warnings()
+
+# run it from local.
+
+db_user, db_password = 'dbc','dbc'
+auth_encoded = db_user + ':' + db_password
+auth_encoded = base64.b64encode(bytes(auth_encoded, 'utf-8'))
+auth_str = 'Basic ' + auth_encoded.decode('utf-8')
+
+print(auth_str)
+
+headers = {
+ 'Content-Type': 'application/json',
+ 'Authorization': auth_str # base 64 encoded username:password
+}
+
+print(headers)Response
+Basic ZGJjOmRiYw==
+{
+ 'Content-Type': 'application/json',
+ 'Authorization': 'Basic ZGJjOmRiYw=='
+}
+Prerequisites:
+The user must already exist in the database.
+The database must be JWT enabled.
+Request
+import requests
+import json
+requests.packages.urllib3.disable_warnings()
+
+# run it from local.
+
+auth_encoded_jwt = "<YOUR_JWT_HERE>"
+auth_str = "Bearer " + auth_encoded_jwt
+
+headers = {
+ 'Content-Type': 'application/json',
+ 'Authorization': auth_str
+}
+
+print(headers)Response
+{'Content-Type': 'application/json', 'Authorization': 'Bearer <YOUR_JWT_HERE>'}
+In the following example, the request includes:
+SELECT * FROM DBC.DBCInfo: The query to the system with the alias <SYSTEM_NAME>.
'format': 'OBJECT': The format for response. The formats supported are: JSON object, JSON array, and CSV.
| + + | ++The JSON object format creates one JSON object per row where the column name is the field name, and the column value is the field value. + | +
'includeColumns': true: The request to include column metadata, such as column names and types, in the response.
'rowLimit': 4: The number of rows to be returned from a query.
Request
+url = 'https://<QS_HOSTNAME>:1443/systems/<SYSTEM_NAME>/queries'
+
+payload = {
+ 'query': example_query, # 'SELECT * FROM DBC.DBCInfo;',
+ 'format': 'OBJECT',
+ 'includeColumns': True,
+ 'rowLimit': 4
+}
+
+payload_json = json.dumps(payload)
+
+response = requests.request('POST', url, headers=headers, data=payload_json, verify=False)
+
+num_rows = response.json().get('results')[0].get('rowCount')
+print('NUMBER of ROWS', num_rows)
+print('==========================================================')
+
+print(response.json())Response
+NUMBER of ROWS 4
+==========================================================
+{
+ "queueDuration":7,
+ "queryDuration":227,
+ "results":[
+ {
+ "resultSet":True,
+ "columns":[
+ {
+ "name":"DatabaseName",
+ "type":"CHAR"
+ },
+ {
+ "name":"USEDSPACE_IN_GB",
+ "type":"FLOAT"
+ },
+ {
+ "name":"MAXSPACE_IN_GB",
+ "type":"FLOAT"
+ },
+ {
+ "name":"Percentage_Used",
+ "type":"FLOAT"
+ },
+ {
+ "name":"REMAININGSPACE_IN_GB",
+ "type":"FLOAT"
+ }
+ ],
+ "data":[
+ {
+ "DatabaseName":"DBC",
+ "USEDSPACE_IN_GB":317.76382541656494,
+ "MAXSPACE_IN_GB":1510.521079641879,
+ "Percentage_Used":21.03670247964377,
+ "REMAININGSPACE_IN_GB":1192.757254225314
+ },
+ {
+ "DatabaseName":"EM",
+ "USEDSPACE_IN_GB":0.0007491111755371094,
+ "MAXSPACE_IN_GB":11.546071618795395,
+ "Percentage_Used":0.006488017745513208,
+ "REMAININGSPACE_IN_GB":11.545322507619858
+ },
+ {
+ "DatabaseName":"user10",
+ "USEDSPACE_IN_GB":0.019153594970703125,
+ "MAXSPACE_IN_GB":9.313225746154785,
+ "Percentage_Used":0.20566016,
+ "REMAININGSPACE_IN_GB":9.294072151184082
+ },
+ {
+ "DatabaseName":"EMEM",
+ "USEDSPACE_IN_GB":0.006140708923339844,
+ "MAXSPACE_IN_GB":4.656612873077393,
+ "Percentage_Used":0.13187072,
+ "REMAININGSPACE_IN_GB":4.650472164154053
+ },
+ {
+ "DatabaseName":"EMWork",
+ "USEDSPACE_IN_GB":0.0,
+ "MAXSPACE_IN_GB":4.656612873077393,
+ "Percentage_Used":0.0,
+ "REMAININGSPACE_IN_GB":4.656612873077393
+ }
+ ],
+ "rowCount":4,
+ "rowLimitExceeded":True
+ }
+ ]
+}
+For response parameters, see Query Service Installation, Configuration, and Usage Guide.
+To return an API response in CSV format, set the format field in the request with the value CSV.
The CSV format contains only the query results and not response metadata. The response contains a line for each row, where each line contains the row columns separated by a comma. The following example returns the data as comma-separated values.
+Request
+# CSV with all rows included
+
+url = 'https://<QS_HOSTNAME>:1443/systems/<SYSTEM_NAME>/queries'
+
+payload = {
+ 'query': example_query, # 'SELECT * FROM DBC.DBCInfo;',
+ 'format': 'CSV',
+ 'includeColumns': True
+}
+
+payload_json = json.dumps(payload)
+
+response = requests.request('POST', url, headers=headers, data=payload_json, verify=False)
+
+print(response.text)Response
+DatabaseName,USEDSPACE_IN_GB,MAXSPACE_IN_GB,Percentage_Used,REMAININGSPACE_IN_GB +DBC ,317.7634754180908,1510.521079641879,21.036679308932754,1192.7576042237881 +EM ,7.491111755371094E-4,11.546071618795395,0.006488017745513208,11.545322507619858 +user10 ,0.019153594970703125,9.313225746154785,0.20566016,9.294072151184082 +EMEM ,0.006140708923339844,4.656612873077393,0.13187072,4.650472164154053 +EMWork ,0.0,4.656612873077393,0.0,4.656612873077393 +EMJI ,0.0,2.3283064365386963,0.0,2.3283064365386963 +USER_NAME ,0.0,2.0,0.0,2.0 +readonly ,0.0,0.9313225746154785,0.0,0.9313225746154785 +aug12_db ,7.200241088867188E-5,0.9313225746154785,0.0077312,0.9312505722045898 +SystemFe ,1.8024444580078125E-4,0.7450580596923828,0.024192,0.744877815246582 +dbcmngr ,3.814697265625E-6,0.09313225746154785,0.004096,0.09312844276428223 +EMViews ,0.027594566345214844,0.09313225746154785,29.62944,0.06553769111633301 +tdwm ,6.732940673828125E-4,0.09313225746154785,0.722944,0.09245896339416504 +Crashdumps ,0.0,0.06984921544790268,0.0,0.06984921544790268 +SYSLIB ,0.006252288818359375,0.03725290298461914,16.78336,0.031000614166259766 +SYSBAR ,4.76837158203125E-6,0.03725290298461914,0.0128,0.03724813461303711 +SYSUDTLIB ,3.5381317138671875E-4,0.029802322387695312,1.1872,0.029448509216308594 +External_AP ,0.0,0.01862645149230957,0.0,0.01862645149230957 +SysAdmin ,0.002307891845703125,0.01862645149230957,12.3904,0.016318559646606445 +KZXaDtQp ,0.0,0.009313225746154785,0.0,0.009313225746154785 +s476QJ6O ,0.0,0.009313225746154785,0.0,0.009313225746154785 +hTzz03i7 ,0.0,0.009313225746154785,0.0,0.009313225746154785 +Y5WYUUXj ,0.0,0.009313225746154785,0.0,0.009313225746154785+
Use explicit sessions when a transaction needs to span multiple requests or when using volatile tables. These sessions are only reused if you reference the sessions in a query request. The request is queued if a request references an explicit session already in use.
+Create a session
+Send a POST request to the /system/<SYSTEM_NAME>/sessions endpoint. The request creates a new database session and returns the session details as the response.
In the following example, the request includes 'auto_commit': True - the request to commit the query upon completion.
Request
+# first create a session
+url = 'https://<QS_HOSTNAME>:1443/systems/<SYSTEM_NAME>/sessions'
+
+payload = {
+ 'auto_commit': True
+}
+
+payload_json = json.dumps(payload)
+
+response = requests.request('POST', url, headers=headers, data=payload_json, verify=False)
+
+print(response.text)Response
+{
+ 'sessionId': 1366010,
+ 'system': 'testsystem',
+ 'user': 'dbc',
+ 'tdSessionNo': 1626922,
+ 'createMode': 'EXPLICIT',
+ 'state': 'LOGGINGON',
+ 'autoCommit': true
+}
+Use the session created in Step 1 to submit queries
+Send a POST request to the /system/<SYSTEM_NAME>/queries endpoint.
The request submits queries to the target system and returns the release and version number of the target system.
+In the following example, the request includes:
+SELECT * FROM DBC.DBCInfo: The query to the system with the alias <SYSTEM_NAME>.
'format': 'OBJECT': The format for response.
'Session' : <Session ID>: The session ID returned in Step 1 to create an explicit session.
Request
+# use this session to submit queries afterwards
+
+url = 'https://<QS_HOSTNAME>:1443/systems/<SYSTEM_NAME>/queries'
+
+payload = {
+ 'query': 'SELECT * FROM DBC.DBCInfo;',
+ 'format': 'OBJECT',
+ 'session': 1366010 # <-- sessionId
+}
+payload_json = json.dumps(payload)
+
+response = requests.request('POST', url, headers=headers, data=payload_json, verify=False)
+
+print(response.text)Response
+{
+ "queueDuration":6,
+ "queryDuration":41,
+ "results":[
+ {
+ "resultSet":true,
+ "data":[
+ {
+ "InfoKey":"LANGUAGE SUPPORT MODE",
+ "InfoData":"Standard"
+ },
+ {
+ "InfoKey":"RELEASE",
+ "InfoData":"15.10.07.02"
+ },
+ {
+ "InfoKey":"VERSION",
+ "InfoData":"15.10.07.02"
+ }
+ ],
+ "rowCount":3,
+ "rowLimitExceeded":false
+ }
+ ]
+}
+Use asynchronous queries when a system or network performance is affected by querying a large group of data or long running queries.
+Submit asynchronous queries to the target system and retrieve a Query ID
+Send a POST request to the /system/<SYSTEM_NAME>/queries endpoint.
In the following example, the request includes:
+SELECT * FROM DBC.DBCInfo: The query to the system with the alias <SYSTEM_NAME>.
'format': 'OBJECT': The format for response.
'spooled_result_set': True: The indication that the request is asynchronous.
Request
+## Run async query .
+
+url = 'https://<QS_HOSTNAME>:1443/systems/<SYSTEM_NAME>/queries'
+
+payload = {
+ 'query': 'SELECT * FROM DBC.DBCInfo;',
+ 'format': 'OBJECT',
+ 'spooled_result_set': True
+}
+
+payload_json = json.dumps(payload)
+response = requests.request('POST', url, headers=headers, data=payload_json, verify=False)
+
+print(response.text)Response
+{"id":1366025}
+Get query details using the ID retrieved from Step 1
+Send a GET request to the /system/<SYSTEM_NAME>/queries/<queryID> endpoint, replacing <queryID> with the ID retrieved from Step 1.
The request returns the details of the specific query, including queryState, queueOrder, queueDuration, and so on. For a complete list of the response fields and their descriptions, see Query Service Installation, Configuration, and Usage Guide.
Request
+## response for async query .
+
+url = 'https://<QS_HOSTNAME>:1443/systems/<SYSTEM_NAME>/queries/1366025'
+
+payload_json = json.dumps(payload)
+response = requests.request('GET', url, headers=headers, verify=False)
+
+print(response.text)Response
+{
+ "queryId":1366025,
+ "query":"SELECT * FROM DBC.DBCInfo;",
+ "batch":false,
+ "system":"testsystem",
+ "user":"dbc",
+ "session":1366015,
+ "queryState":"RESULT_SET_READY",
+ "queueOrder":0,
+ "queueDuration":6,
+ "queryDuration":9,
+ "statusCode":200,
+ "resultSets":{
+
+ },
+ "counts":{
+
+ },
+ "exceptions":{
+
+ },
+ "outParams":{
+
+ }
+}
+View resultset for asynchronous query
+Send a GET request to the /system/<SYSTEM_NAME>/queries/<queryID>/results endpoint, replacing <queryID> with the ID retrieved from Step 1.
+The request returns an array of the result sets and update counts produced by the submitted query.
Request
+url = 'https://<QS_HOSTNAME>:1443/systems/<SYSTEM_NAME>/queries/1366025/results'
+
+payload_json = json.dumps(payload)
+response = requests.request('GET', url, headers=headers, verify=False)
+
+print(response.text)Response
+{
+ "queueDuration":6,
+ "queryDuration":9,
+ "results":[
+ {
+ "resultSet":true,
+ "data":[
+ {
+ "InfoKey":"LANGUAGE SUPPORT MODE",
+ "InfoData":"Standard"
+ },
+ {
+ "InfoKey":"RELEASE",
+ "InfoData":"15.10.07.02"
+ },
+ {
+ "InfoKey":"VERSION",
+ "InfoData":"15.10.07.02"
+ }
+ ],
+ "rowCount":3,
+ "rowLimitExceeded":false
+ }
+ ]
+}
+Send a GET request to the /system/<SYSTEM_NAME>/queries endpoint. The request returns the IDs of active queries.
Request
+url = 'https://<QS_HOSTNAME>:1443/systems/<SYSTEM_NAME>/queries'
+
+payload={}
+
+response = requests.request('GET', url, headers=headers, data=payload, verify=False)
+
+print(response.json())Response
+[
+ {
+ "queryId": 12516087,
+ "query": "SELECt * from dbcmgr.AlertRequest;",
+ "batch": false,
+ "system": "BasicTestSys",
+ "user": "dbc",
+ "session": 12516011,
+ "queryState": "REST_SET_READY",
+ "queueOrder": 0,
+ "queueDurayion": 3,
+ "queryDuration": 3,
+ "statusCode": 200,
+ "resultSets": {},
+ "counts": {},
+ "exceptions": {},
+ "outparams": {}
+ },
+ {
+ "queryId": 12516088,
+ "query": "SELECt * from dbc.DBQLAmpDataTbl;",
+ "batch": false,
+ "system": "BasicTestSys",
+ "user": "dbc",
+ "session": 12516011,
+ "queryState": "REST_SET_READY",
+ "queueOrder": 0,
+ "queueDurayion": 3,
+ "queryDuration": 3,
+ "statusCode": 200,
+ "resultSets": {},
+ "counts": {},
+ "exceptions": {},
+ "outparams": {}
+ }
+]
+Features, examples, and references: Query Service Installation, Configuration, and Usage Guide
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Thripti Aravind
+Last updated: May 16th, 2023
| + + | ++This product is in preview and is subject to change. To get early access to Regulus, sign up on the Regulus Home page. + | +
This document walks you through a simple workflow where you can use JupyterLab to:
+Deploy on-demand, scalable compute
+Connect to your external data source
+Run the workload
+Suspend the compute
+Install and configure Regulus. See Install and Configure Regulus Using Docker.
+Copy and retain the following:
+AWS environment variables, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN from your AWS Console. See Environment Variables.
API Key from Workspaces.
+Run %help or %help <command> for details on any magic command. See Regulus JupyterLab Magic Command Reference for more details.
Connect to JupyterLab using the URL: http://localhost:8888 and enter the token when prompted.
+Connect to Workspaces using the API Key.
+%workspaces_config host=<ip_or_hostname>, apikey=<API_Key>, withtls=FCreate a new project.
+| + + | ++Currently, Regulus supports only AWS. + | +
%project_create project=<Project_Name>, env=aws[Optional] Create an authorization object to store the CSP credentials.
+Replace AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_REGION with your values.
%project_auth_create name=<Auth_Name>, project=<Project_Name>, key=<AWS_ACCESS_KEY_ID>, secret=<AWS_SECRET_ACCESS_KEy>, region=<AWS_REGION>Deploy a query engine for the project.
+Replace the <Project_Name> to a name of your choice. The size parameter value can be small, medium, large, or extralarge. The default size is small.
+%project_engine_deploy name=<Project_Name>, size=<Size_of_Engine>The deployment process will take a few minutes to complete. On successful deployment, a password is generated.
+Establish a connection to your project.
+%connect <Project_Name>When a connection is established, the interface prompts you for a password. Enter the password generated in the previous step.
+Run the sample workload.
+| + + | ++Make sure that you do not have tables named SalesCenter or SalesDemo in the selected database. + | +
Create a table to store the sales center data.
+First, drop the table if it already exists. The command fails if the table does not exist.
+DROP TABLE SalesCenter;
+CREATE MULTISET TABLE SalesCenter ,NO FALLBACK ,
+ NO BEFORE JOURNAL,
+ NO AFTER JOURNAL,
+ CHECKSUM = DEFAULT,
+ DEFAULT MERGEBLOCKRATIO
+ (
+ Sales_Center_id INTEGER NOT NULL,
+ Sales_Center_Name VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC)
+NO PRIMARY INDEX ;Load data into the SalesCenter table using the %dataload magic command.
%dataload DATABASE=<Project_Name>, TABLE=SalesCenter, FILEPATH=notebooks/sql/data/salescenter.csv| + + | ++Unable to locate the salescenter.csv file? Download the file from GitHub Demo: Charting and Visualization Data + | +
Verify that the data was inserted.
+SELECT * FROM SalesCenter ORDER BY 1Create a table with the sales demo data.
+DROP TABLE SalesDemo;
+CREATE MULTISET TABLE SalesDemo ,NO FALLBACK ,
+ NO BEFORE JOURNAL,
+ NO AFTER JOURNAL,
+ CHECKSUM = DEFAULT,
+ DEFAULT MERGEBLOCKRATIO
+ (
+ Sales_Center_ID INTEGER NOT NULL,
+ UNITS DECIMAL(15,4),
+ SALES DECIMAL(15,2),
+ COST DECIMAL(15,2))
+NO PRIMARY INDEX ;Load data into the SalesDemo table using the %dataload magic command.
%dataload DATABASE=<Project_Name>, TABLE=SalesDemo, FILEPATH=notebooks/sql/data/salesdemo.csv| + + | ++Unable to locate the salesdemo.csv file? Download the file from GitHub Demo: Charting and Visualization Data + | +
Verify that the sales demo data was inserted successfully.
+SELECT * FROM SalesDemo ORDER BY salesOpen the Navigator for your connection and verify that the tables were created. Run a row count on the tables to verify that the data was loaded.
+Use charting magic to visualize the result.
+Provide X and Y axes for your chart.
+%chart sales_center_name, sales, title=Sales DataDrop the tables.
+DROP TABLE SalesCenter;
+DROP TABLE SalesDemo;Back up your project metadata and object definitions in your GitHub repository.
+%project_backup project=<Project_Name>Suspend the query engine.
+%project_engine_suspend project=<Project_Name>Congrats! You’ve successfully run your first use case in JupyterLab.
+Interested in exploring advanced use cases? Coming soon! Keep watching this space for the GitHub link.
+Learn about the magic commands available in JupyterLab. See Regulus JupyterLab Magic Command Reference.
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Thripti Aravind
+Last updated: May 16th, 2023
| + + | ++This product is in preview and is subject to change. To get early access to Regulus, sign up on the Regulus Home page. + | +
Regulus is a self-service platform that enables you to deploy and connect an SQL query engine to your data lake. You can then run your workloads on the on-demand, scalable query engine deployed on your preferred Cloud Service Provider (CSP). Using the query engine, you can leverage the capabilities of a highly parallel database while eliminating the need for data management.
+Regulus contains the following components:
+Workspaces: An orchestration service that controls and manages Regulus automation and deployments. It also controls the integration elements that provide a seamless user experience when running data-related projects. Workspaces includes a web-based UI that you can use to authorize the user and define your choice of CSP integrations.
+Interface: An environment to write and run data projects, connect to the Teradata system, and visualize data. You can use either JupyterLab or Workspaces CLI.
+Query Engine: A fully managed computational resource that you can use to run your data science and analytical workloads.
+This document outlines the steps for installing and configuring Regulus using Docker. To use Regulus with Workspaces CLI, see Use Regulus With Workspaces CLI.
+Make sure you have the following:
+GitHub account: If you don’t already have a GitHub account, create one at https://github.com/.
+AWS account: If you don’t have an AWS account, sign up for an AWS Free Tier account at https://aws.amazon.com/free/.
+| + + | ++For the developer preview, you can host the query engine only on AWS. + | +
Docker: To download and install Docker, See https://docs.docker.com/docker-for-windows/install/.
+The Workspaces Docker images are monolithic images of Workspaces running the necessary services in a single container.
+Pull the docker image from Docker Hub.
+docker pull teradata/regulus-workspacesBefore proceeding, make sure to:
+Copy and retain the environment variables, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN, from your AWS Console. See Environment Variables.
Set the environment variable, WORKSPACES_HOME, to the directory where the configuration and data files are located. Make sure that the directory exists and appropriate permission is granted.
| Local Location | +Container Location | +Usage | +
|---|---|---|
$WORKSPACES_HOME |
+/etc/td |
+Stores data and configuration |
+
$WORKSPACES_HOME/tls |
+/etc/td/tls |
+Stores cert files |
+
You can install Workspaces using one of the following methods:
+Run the Docker image once you’ve set the WORKSPACES_HOME variable.
| + + | ++Modify the directories based on your requirements. + | +
docker run -detach \
+ --env accept_license="Y" \
+ --env AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" \
+ --env AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" \
+ --env AWS_SESSION_TOKEN="${AWS_SESSION_TOKEN}" \
+ --publish 3000:3000 \
+ --publish 3282:3282 \
+ --volume ${WORKSPACES_HOME}:/etc/td \
+ teradata/regulus-workspaces:latestThe command downloads and starts a Workspaces container and publishes the ports needed to access it. Once the Workspaces server is initialized and started, you can access Workspaces using the URL: http://<ip_or_hostname>:3000/.
+With Docker Compose, you can easily configure, install, and upgrade your Docker-based Workspaces installation.
+Install Docker Compose. See https://docs.docker.com/compose/install/.
+Create a docker-compose.yml file.
+version: "3.9"
+
+services:
+ workspaces:
+ deploy:
+ replicas: 1
+ container_name: workspaces
+ image: ${WORKSPACES_IMAGE_NAME:-teradata/regulus-workspaces}:${WORKSPACES_IMAGE_TAG:-latest}
+ command: workspaces serve -v
+ restart: unless-stopped
+ ports:
+ - "443:443/tcp"
+ - "3000:3000/tcp"
+ - "3282:3282/tcp"
+ environment:
+ accept_license: "Y"
+ TZ: ${WS_TZ:-UTC}
+ AWS_ACCESS_KEY_ID: "${AWS_ACCESS_KEY_ID}"
+ AWS_SECRET_ACCESS_KEY: "${AWS_SECRET_ACCESS_KEY}"
+ AWS_SESSION_TOKEN: "${AWS_SESSION_TOKEN}"
+ volumes:
+ - ${WORKSPACES_HOME:-./volumes/workspaces}:/etc/td
+ - ${WORKSPACES_AWS_CONFIG:-~/.aws}:/root/.awsGo to the directory where the docker-compose.yml file is located and start Workspaces.
+Docker compose up -dOnce the Workspaces server is initialized and started, you can access Workspaces using the URL: http://<ip_or_hostname>:3000/.
+Workspaces uses the GitHub OAuth App to authorize users and manage the project state. To authorize Workspaces to save your project instance configuration, use the Client ID and Client secret key generated during the GitHub OAuth App registration. The project instance configuration values are maintained in your GitHub repositories.
+First-time users must perform the following steps before proceeding:
+Log on to your GitHub account and create an OAuth App. See GitHub Documentation.
+While registering the OAuth App, type the following Workspaces URLs in the URL fields:
+Homepage URL: http://<ip_or_hostname>:3000/
+Authorization callback URL: http://<ip_or_hostname>:3000/auth/github/callback
+Copy and retain the Client ID and Client secret key.
+To set up Workspaces, do the following:
+Access Workspaces using the URL: http://<ip_or_hostname>:3000/.
+
+Apply the following general service configuration under Setup.
+| Setting | +Description | +Required? | +
|---|---|---|
Service Base URL |
+[Non-Editable] The root URL of the service. |
+Yes |
+
Git Provider |
+The provider for Git integration. Currently, Regulus supports only GitHub. |
+Yes |
+
Service Log Lev |
+The level of logging. |
+Yes |
+
Engine IP Network Type |
+The type of network assigned to a query engine instance, which can be either public or private. |
+Yes |
+
Use TLS |
+Indicates if TLS support is enabled. Teradata recommends enabling this option. |
+Yes |
+
Service TLS Certification |
+The server certificate to authenticate the server identity. |
+No |
+
Service TLS Certificate Key |
+The server certificate key. |
+No |
+
To use a self-signed certificate for Service Base URL, select GENERATE TLS. A certificate and private key are generated and displayed in the respective fields.
+Select Next.
+Apply the following settings under Cloud Integrations: AWS.
+| Setting | +Description | +Required? | +
|---|---|---|
Default Region |
+The AWS region you want to deploy the workload in. Teradata recommends choosing the region closest to your primary work location. |
+Yes |
+
Default Subnet |
+The default location to launch the query engine. |
+Yes |
+
Default CIDRs |
+The list of Classless Inter-Domain Routing (CIDR) addresses used for the query engine. Use CIDR to allocate IP addresses flexibly and efficiently in your network. If you don’t specify a CIDR, the query engine is automatically associated with the default CIDR. |
+No |
+
Default Security Groups |
+The list of security groups for the VPC in each region. If you don’t specify a security group, the query engine is automatically associated with the default security group for the VPC. |
+No |
+
Select Next.
+Apply the following settings under Configure GitHub.
+| Setting | +Description | +Required? | +
|---|---|---|
GitHub Application URL |
+[Non-Editable] The URL where the Workspaces server is hosted. |
+Yes |
+
GitHub Callback URL |
+[Non-Editable] The URL you are redirected to after you authorize. |
+Yes |
+
GitHub Client ID |
+The Client ID you received from GitHub on creating your OAuth App. |
+Yes |
+
Use TLS |
+Enable TLS support. |
+Yes |
+
GitHub Client Secret |
+The Client secret ID you received from GitHub on creating your OAuth App. |
+Yes |
+
GitHub Organization |
+The name of the GitHub organization account that you use to collaborate with your team. |
+No |
+
GitHub Base URL |
+The base URL of your GitHub account. The URL may vary based on your account type. For example, https://github.company.com/ for GitHub Enterprise account. |
+No |
+
Select Save and then select Login. You are redirected to GitHub.
+Log on with your GitHub credentials to authorize Workspaces.
+After authentication, you are redirected to the Workspaces Profile page, and an API Key is generated. You can use the API Key to make requests to the Workspaces service.
+| + + | ++A new API Key is generated each time you connect to Workspaces. + | +
You can use either JupyterLab or Workspaces CLI as your Regulus interface.
+JupyterLab: Install using one of the following methods:
+Workspaces CLI: See Use Regulus With Workspaces CLI.
+Pull the Docker image from the DockerHub at https://hub.docker.com/r/teradata/regulus-jupyter.
+Run the Docker image once you’ve set the JUPYTER_HOME variable.
| + + | ++Modify the directories based on your requirements. + | +
docker run -detach \
+ --env “accept_license=Y” \
+ --publish 8888:8888 \
+ --volume ${JUPYTER_HOME}: /home/jovyan/JupyterLabRoot \
+ teradata/regulus-jupyter:latestThe command downloads and starts a JupyterLab container and publishes the ports needed to access it. +Connect to JupyterLab using the URL: http://localhost:8888 and enter the token when prompted. For detailed instructions, see Teradata Vantage™ Modules for Jupyter Installation Guide or Use Vantage from a Jupyter Notebook.
+With Docker Compose, you can easily configure, install, and upgrade your Docker-based JupyterLab installation.
+Install Docker Compose. See https://docs.docker.com/compose/install/.
+Create a docker-compose.yml file.
+version: "3.9"
+
+services:
+ jupyter:
+ deploy:
+ replicas: 1
+ image: teradata/regulus-jupyter:latest
+ environment:
+ - "accept_license=Y"
+ ports:
+ - 8888:8888
+ volumes:
+ - ${JUPYTER_HOME:-./volumes/jupyter}:/home/jovyan/JupyterLabRoot/userdata
+ - ${WORKSPACES_AWS_CONFIG:-~/.aws}:/root/.awsGo to the directory where the docker-compose.yml file is located and start JupyterLab.
+Docker compose up -dOnce the JupyterLab server is initialized and started, you can connect to JupyterLab using the URL: http://localhost:8888 and enter the token when prompted. For detailed instructions, see Teradata Vantage™ Modules for Jupyter Installation Guide or Use Vantage from a Jupyter Notebook.
+Congrats! You’re all set up to use Regulus.
+Get started with Regulus by running a simple workflow. See Run a Sample Workload in JupyterLab.
+Interested in learning how Regulus can help you with real-life use cases? Coming soon! Keep watching this space for the GitHub link.
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Thripti Aravind
+Last updated: May 16th, 2023
| + + | ++This product is in preview and is subject to change. To get early access to Regulus, sign up on the Regulus Home page. + | +
Regulus JupyterLab supports the following magic commands in addition to the existing Teradata SQL Kernel magic commands. See Teradata JupyterLab Getting Started Guide.
+Description: One-time configuration to bind with the Workspaces service.
+Usage:
+%workspaces_config host=<RPC_Service_URL>, apikey=<Workspace_API_Key>, withtls=FWhere:
+host: Name or IP address of the query engine service.
+apikey: API Key value from the Workspaces Profile page.
+[Optional] withTLS: If False (F), the default client-server communication does not use TLS.
+Output:
+Workspace configured for host=<RPC_Service_URL>+
Description: Create a new project. This command also creates a new repository with the project name in your GitHub account. The configurations are stored in the engine.yml file.
+Usage:
+%project_create project=<Project_Name>, env=<CSP>, team=<Project_Team>Where:
+project: Name of the project to be created.
+env: Cloud environment where the project is hosted. The value can be aws, azure, gcp, or vsphere. For the current release, only AWS is supported.
+[Optional] team: Name of the team collaborating on the project.
+Output:
+Project <Project_Name> created+
Description: Delete a project.
+| + + | ++Running this command removes the GitHub repository containing the objects created using Regulus. + | +
Usage:
+%project_delete project=<Project_Name>, team=<Project_Team>Where:
+project: Name of the project to be deleted.
+[Optional] team: Name of the team collaborating on the project.
+Output:
+Project <Project_Name> deleted+
Description: List the details of the projects.
+Use the project parameter to get the details of a specific project. All the projects are listed if you run the command without any parameters.
+Usage:
+%project_list project=<Project_Name>Where:
+project: Name of the project to be listed.
+Output:
+
+Description: Create an authorization object to store object store credentials.
+You must create the authorization object before deploying the query engine. The authorization details are retained and are included while redeploying the project. Optionally, you can create authorizations manually using the CREATE AUTHORIZATION SQL command after deploying the query engine. In this case, the authorization details are not retained.
Usage:
+%project_auth_create project=<Project_Name>, name=<Auth_Name>, key=<Auth_Key>, secret=<Auth_Secret>, region=<ObjectStore_Region>Where:
+project: Name of the project.
+name: Authorization name for the object store.
+key: Authorization key of the object store.
+secret: Authorization secret access ID of the object store.
+region: Region of the object store; local for the local object store.
+Output:
+Authorization 'name' created+
Description: Remove an object store authorization.
+Usage:
+%project_auth_delete project=<Project_Name>, name=<Auth_Name>Where:
+project: Name of the project.
+name: Authorization name for the object store.
+Output:
+Authorization 'name' deleted+
Description: List object store authorizations that are created for a project.
+Usage:
+%project_auth_list project=<Project_Name>Where:
+project: Name of the project.
+Output:
+
+Description: Deploy a query engine for the project. The deployment process will take a few minutes to complete. On successful deployment, a password is generated.
+Usage:
+%project_engine_deploy project=<Project_Name>, size=<Size_of_Engine>, node=<Number_of_Nodes>, subnet=<Subnet_id>, region=<Region>, secgroups=<Security_Group>, cidrs=<CIDR>Where:
+project: Name of the project.
+size: Size of the query engine. The value can be:
+small
+medium
+large
+extralarge
+[Optional] node: Number of query engine nodes to be deployed. The default value is 1.
+[Optional] subnet: Subnet used for the query engine if there are no default values from the service.
+[Optional] region: Region used for the query engine if there are no default values from service.
+[Optional] secgroups: List of security groups for the VPC in each region. If you don’t specify a security group, the query engine is automatically associated with the default security group for the VPC.
+[Optional] cidr: List of CIDR addresses used for the query engine.
+Output:
+Started deploying. +Success: Compute Engine setup, look at the connection manager+
+Description: Stop the query engine after you’re done with your work.
+Usage:
+%project_engine_suspend <Project_Name>Where:
+project: Name of the project.
+Output:
+Started suspend. Success: connection removed +Success: Suspending Compute Engine+
Description: View the list of query engines deployed for your project.
+Usage:
+%project_engine_list project=<Project_Name>Where:
+project: Name of the project.
+Output:
+
+Description: View the list of collaborators assigned to the project.
+Usage:
+%project_user_list project=<Project_Name>Where:
+[Optional] project: Name of the project.
+Output:
+
+Description: Back up your project metadata and object definition inside the query engine.
+Usage:
+%project_backup project=<Project_Name>Where:
+project: Name of the project.
+Output:
+Backup of the object definitions created+
Description: Restore your project metadata and object definition from your GitHub repository.
+Usage:
+%project_restore project=<Project_Name>, gitref=<Git_Reference>Where:
+project: Name of the project.
+[Optional] gitref: Git reference.
+Output:
+Restore of the object definitions done+
Description: View the list of magics provided with Regulus-Teradata SQL CE Kernel.
+Usage:
+%helpAdditionally, you can see detailed help messages per command.
+Usage:
+%help <command>| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Thripti Aravind
+Last updated: May 16th, 2023
| + + | ++This product is in preview and is subject to change. To get early access to Regulus, contact Support. + | +
Workspaces CLI is a command line interface (CLI) for Regulus. This document provides step-by-step instructions to install Workspaces CLI. In this document, you can find all the necessary information and guidance on the CLI commands, allowing you to navigate the command line quickly and efficiently.
+You can also use JupyterLab as your Regulus interface. See Run a Sample Workload in JupyterLab.
+Make sure you have:
+Installed, configured, and set up Workspaces using the steps outlined in Install and Configure Regulus Using Docker.
+Copied and retained the AWS environment variables and API Key.
+Download the Workspaces CLI executable file from https://downloads.teradata.com/download/tools/regulus-ctl.
+| + + | ++Workspaces CLI supports all major operating systems. + | +
Open the terminal window and run the workspacesctl file.
+ +
+Configure Workspaces using the API Key.
+workspacesctl workspaces configCreate a new project.
+workspacesctl project create <Project_Name> -e <CSP> --no-tlsDeploy a query engine for the project.
+workspacesctl project engine deploy <Project_Name> -t <Size_of_Engine> --no-tlsRun a sample workload.
+Manage your project and query engine.
+Backup your project.
+workspacesctl project backup <Project_Name> --no-tlsSuspend the query engine.
+workspacesctl project engine suspend <Project_Name> --no-tlsFor a supported list of commands, see Workspaces CLI Reference.
+Description: One-time configuration to bind CLI with the Workspaces service. Go to the Workspaces Profile page and copy the API Key.
+Usage:
+workspacesctl workspaces configFlags:
+-h, --help: List the details of the command.
Output:
+
+Follow the prompts to choose the Workspaces endpoint and API Key.
+Description: View the list of users set up for Regulus on GitHub.
+Usage:
+workspacesctl workspaces user list --no-tls| + + | +
+If your setup includes TLS configuration, you need not add the -no-tls parameter.
+ |
+
Flags:
+-h, --help: List the details of the command.
Output:
+
+Description: Create a project in Regulus. The command also creates a corresponding GitHub repository for the project.
+Usage:
+workspacesctl project create <Project_Name> -e <CSP> --no-tls| + + | +
+If your setup includes TLS configuration, you need not add the -no-tls parameter.
+ |
+
Flags:
+| Flag | +Type | +Description | +Required? | +
|---|---|---|---|
|
+String |
+Environment where the project query engine is hosted. Values: aws, azure, or gcloud. Currently, Regulus supports only aws. |
+Yes |
+
|
+String |
+Path to manifest the yaml file to be used for the input. |
+No |
+
|
+String |
+Team assigned to the project. |
+No |
+
|
++ | List the details of the command. |
+No |
+
Output:
+
+Description: View the list of all projects set up in Regulus.
+Usage:
+workspacesctl project list --no-tlsor
+workspacesctl project list <Project_Name> --no-tls| + + | +
+If your setup includes TLS configuration, you need not add the -no-tls parameter.
+ |
+
Flags:
+-h, --help: List the details of the command.
Output:
+
+Description: Delete a project in Regulus.
+Usage:
+ workspacesctl project delete <Project_Name> --no-tls| + + | +
+If your setup includes TLS configuration, you need not add the -no-tls parameter.
+ |
+
Flags:
+-h, --help: List the details of the command.
Output:
+| + + | ++The output is in YAML format. + | +
+Description: View the list of collaborators assigned to the project in GitHub.
+Usage:
+workspacesctl project user list <Project_Name> --no-tls| + + | +
+If your setup includes TLS configuration, you need not add the -no-tls parameter.
+ |
+
Flags:
+-h, --help: List the details of the command.
Output:
+
+Description: Back up the query engine object definitions to the GitHub repository assigned for the project.
+Usage:
+workspacesctl project backup <Project_Name> --no-tls| + + | +
+If your setup includes TLS configuration, you need not add the -no-tls parameter.
+ |
+
Flags:
+-h, --help: List the details of the command.
Output:
+| + + | ++The output is in YAML format. + | +
+Description: Restore all query engine object definitions from the project GitHub repository.
+Usage:
+workspacesctl project restore <Project_Name> --no-tlsor
+workspacesctl project restore <Project_Name> --gitref <git_reference> --no-tls| + + | +
+If your setup includes TLS configuration, you need not add the -no-tls parameter.
+ |
+
Flags:
+| Flag | +Type | +Description | +Required? | +
|---|---|---|---|
|
+String |
+Tag, SHA, or branch name. |
+No |
+
|
++ | List the details of the command. |
+No |
+
Output:
+| + + | ++The output is in YAML format. + | +
+Description: Deploy a query engine for the project.
+Usage:
+workspacesctl project engine deploy <Project_Name> -t small --no-tls| + + | +
+If your setup includes TLS configuration, you need not add the -no-tls parameter.
+ |
+
Flags:
+| Flag | +Type | +Description | +Required? | +
|---|---|---|---|
|
+Int |
+Number of query engine nodes. The default value is 1. |
+No |
+
|
+String |
+Instance size of the query engine. |
+No |
+
|
+String |
+Path to manifest the yaml file to use for the input. |
+No |
+
|
+String |
+Region for the deployment. |
+No |
+
|
+String |
+Subnet ID for the deployment. |
+No |
+
|
++ | List the details of the command. |
+No |
+
Description: Destroy the deployed query engine and back up the object definitions created during the session.
+Usage:
+workspacesctl project engine suspend <Project_Name> --no-tls| + + | +
+If your setup includes TLS configuration, you need not add the -no-tls parameter.
+ |
+
Flags:
+-h, --help: List the details of the command.
Output:
+| + + | ++The output is in YAML format. + | +
+Description: View the detailed information about the query engine for a project. The command displays the last state of the query engine.
+Usage:
+workspacesctl project engine list <Project_Name> --no-tls| + + | +
+If your setup includes TLS configuration, you need not add the -no-tls parameter.
+ |
+
Flags:
+-h, --help: List the details of the command.
Output:
+| + + | ++The output is in YAML format. + | +
+Description: Create authorization for object store.
+Usage:
+workspacesctl project auth create <Project_Name> -n <Auth_Name> -a <Auth_Key> -s <Auth_Secret> -r <ObjectStore_Region> --no-tls| + + | +
+If your setup includes TLS configuration, you need not add the -no-tls parameter.
+ |
+
Flags:
+| Flag | +Type | +Description | +Required? | +
|---|---|---|---|
|
+String |
+Authorization access key or ID. |
+Yes, if you’re not using the |
+
|
+String |
+Authorization name for the object store. |
+Yes, if you’re not using the |
+
|
+String |
+Path to manifest the yaml file to use for the input. |
+No |
+
|
+String |
+Region of the object store. |
+Yes |
+
-s, --secret string |
+String |
+Authorization secret access key of the object store. |
+Yes, if you’re not using the |
+
|
++ | List the details of the command. |
+No |
+
Output:
+| + + | ++The output is in YAML format. + | +
+Description: List object store authorizations that are created for a project.
+Usage:
+workspacesctl project auth list <Project_Name> --no-tls| + + | +
+If your setup includes TLS configuration, you need not add the -no-tls parameter.
+ |
+
Flags:
+-h, --help: List the details of the command.
Output:
+| + + | ++The output is in YAML format. + | +
+Description: Delete object store authorizations that are created for a project.
+Usage:
+workspacesctl project auth delete <Project_Name> -n <Auth_Name> --no-tls| + + | +
+If your setup includes TLS configuration, you need not add the -no-tls parameter.
+ |
+
Flags:
+| Flag | +Type | +Description | +Required? | +
|---|---|---|---|
|
+String |
+Name of the object store authorization to delete. |
+Yes |
+
|
++ | List the details of the command. |
+No |
+
Output:
+| + + | ++The output is in YAML format. + | +
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Adam Tworkiewicz
+Last updated: December 12th, 2022
This how-to demonstrates how to run Vantage Express on AWS. Vantage Express is a small footprint configuration that contains a fully functional Teradata SQL Engine.
+| + + | +
+ Cloud charges
+
+
+Vantage Express is distributed as a virtual machine image. This how-to uses the EC2
+
+If you want a cheaper option, try Google Cloud and Azure which support nested virtualization and can run Vantage Express on cheap VM’s. +
+
+If you do not wish to pay for cloud usage at all, install Vantage Express locally using VMware, VirtualBox, or UTM. + |
+
An AWS account. If you need to create a new account follow the official AWS instructions.
+awscli command line utility installed and configured on your machine. You can find installation instructions here: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html.
You will need a VPC with an Internet-facing subnet. If you don’t have one available, here is how you can create it:
+# Copied from https://cloudaffaire.com/how-to-create-a-custom-vpc-using-aws-cli/
+
+# Create VPC
+AWS_VPC_ID=$(aws ec2 create-vpc \
+ --cidr-block 10.0.0.0/16 \
+ --query 'Vpc.{VpcId:VpcId}' \
+ --output text)
+
+# Enable DNS hostname for your VPC
+aws ec2 modify-vpc-attribute \
+ --vpc-id $AWS_VPC_ID \
+ --enable-dns-hostnames "{\"Value\":true}"
+
+# Create a public subnet
+AWS_SUBNET_PUBLIC_ID=$(aws ec2 create-subnet \
+ --vpc-id $AWS_VPC_ID --cidr-block 10.0.1.0/24 \
+ --query 'Subnet.{SubnetId:SubnetId}' \
+ --output text)
+
+# Enable Auto-assign Public IP on Public Subnet
+aws ec2 modify-subnet-attribute \
+ --subnet-id $AWS_SUBNET_PUBLIC_ID \
+ --map-public-ip-on-launch
+
+# Create an Internet Gateway
+AWS_INTERNET_GATEWAY_ID=$(aws ec2 create-internet-gateway \
+ --query 'InternetGateway.{InternetGatewayId:InternetGatewayId}' \
+ --output text)
+
+# Attach Internet gateway to your VPC
+aws ec2 attach-internet-gateway \
+ --vpc-id $AWS_VPC_ID \
+ --internet-gateway-id $AWS_INTERNET_GATEWAY_ID
+
+# Create a route table
+AWS_CUSTOM_ROUTE_TABLE_ID=$(aws ec2 create-route-table \
+ --vpc-id $AWS_VPC_ID \
+ --query 'RouteTable.{RouteTableId:RouteTableId}' \
+ --output text )
+
+# Create route to Internet Gateway
+aws ec2 create-route \
+ --route-table-id $AWS_CUSTOM_ROUTE_TABLE_ID \
+ --destination-cidr-block 0.0.0.0/0 \
+ --gateway-id $AWS_INTERNET_GATEWAY_ID \
+ --output text
+
+# Associate the public subnet with route table
+AWS_ROUTE_TABLE_ASSOID=$(aws ec2 associate-route-table \
+ --subnet-id $AWS_SUBNET_PUBLIC_ID \
+ --route-table-id $AWS_CUSTOM_ROUTE_TABLE_ID \
+ --output text | head -1)
+
+# Create a security group
+aws ec2 create-security-group \
+ --vpc-id $AWS_VPC_ID \
+ --group-name myvpc-security-group \
+ --description 'My VPC non default security group' \
+ --output text
+
+# Get security group ID's
+AWS_DEFAULT_SECURITY_GROUP_ID=$(aws ec2 describe-security-groups \
+ --filters "Name=vpc-id,Values=$AWS_VPC_ID" \
+ --query 'SecurityGroups[?GroupName == `default`].GroupId' \
+ --output text) &&
+ AWS_CUSTOM_SECURITY_GROUP_ID=$(aws ec2 describe-security-groups \
+ --filters "Name=vpc-id,Values=$AWS_VPC_ID" \
+ --query 'SecurityGroups[?GroupName == `myvpc-security-group`].GroupId' \
+ --output text)
+
+# Create security group ingress rules
+aws ec2 authorize-security-group-ingress \
+ --group-id $AWS_CUSTOM_SECURITY_GROUP_ID \
+ --ip-permissions '[{"IpProtocol": "tcp", "FromPort": 22, "ToPort": 22, "IpRanges": [{"CidrIp": "0.0.0.0/0", "Description": "Allow SSH"}]}]' \
+ --output text
+
+# Add a tag to the VPC
+aws ec2 create-tags \
+ --resources $AWS_VPC_ID \
+ --tags "Key=Name,Value=vantage-express-vpc"
+
+# Add a tag to public subnet
+aws ec2 create-tags \
+ --resources $AWS_SUBNET_PUBLIC_ID \
+ --tags "Key=Name,Value=vantage-express-vpc-public-subnet"
+
+# Add a tag to the Internet-Gateway
+aws ec2 create-tags \
+ --resources $AWS_INTERNET_GATEWAY_ID \
+ --tags "Key=Name,Value=vantage-express-vpc-internet-gateway"
+
+# Add a tag to the default route table
+AWS_DEFAULT_ROUTE_TABLE_ID=$(aws ec2 describe-route-tables \
+ --filters "Name=vpc-id,Values=$AWS_VPC_ID" \
+ --query 'RouteTables[?Associations[0].Main != `false`].RouteTableId' \
+ --output text) &&
+ aws ec2 create-tags \
+ --resources $AWS_DEFAULT_ROUTE_TABLE_ID \
+ --tags "Key=Name,Value=vantage-express-vpc-default-route-table"
+
+# Add a tag to the public route table
+aws ec2 create-tags \
+ --resources $AWS_CUSTOM_ROUTE_TABLE_ID \
+ --tags "Key=Name,Value=vantage-express-vpc-public-route-table"
+
+# Add a tags to security groups
+aws ec2 create-tags \
+ --resources $AWS_CUSTOM_SECURITY_GROUP_ID \
+ --tags "Key=Name,Value=vantage-express-vpc-security-group" &&
+ aws ec2 create-tags \
+ --resources $AWS_DEFAULT_SECURITY_GROUP_ID \
+ --tags "Key=Name,Value=vantage-express-vpc-default-security-group"To create a VM you will need an ssh key pair. If you don’t have it already, create one:
+aws ec2 create-key-pair --key-name vantage-key --query 'KeyMaterial' --output text > vantage-key.pemRestrict access to the private key. Replace <path_to_private_key_file> with the private key path returned by the previous command:
chmod 600 vantage-key.pemGet the AMI id of the latest Ubuntu image in your region:
+AWS_AMI_ID=$(aws ec2 describe-images \
+ --filters 'Name=name,Values=ubuntu/images/hvm-ssd/ubuntu-*amd64*' \
+ --query 'Images[*].[Name,ImageId,CreationDate]' --output text \
+ | sort -k3 -r | head -n1 | cut -f 2)Create a Ubuntu VM with 4 CPU’s and 8GB of RAM, and a 70GB disk.
+AWS_INSTANCE_ID=$(aws ec2 run-instances \
+ --image-id $AWS_AMI_ID \
+ --count 1 \
+ --instance-type c5n.metal \
+ --block-device-mapping DeviceName=/dev/sda1,Ebs={VolumeSize=70} \
+ --key-name vantage-key \
+ --security-group-ids $AWS_CUSTOM_SECURITY_GROUP_ID \
+ --subnet-id $AWS_SUBNET_PUBLIC_ID \
+ --query 'Instances[0].InstanceId' \
+ --output text)ssh to your VM:
+AWS_INSTANCE_PUBLIC_IP=$(aws ec2 describe-instances \
+ --query "Reservations[*].Instances[*].PublicIpAddress" \
+ --output=text --instance-ids $AWS_INSTANCE_ID)
+ssh -i vantage-key.pem ubuntu@$AWS_INSTANCE_PUBLIC_IPOnce in the VM, switch to root user:
sudo -iPrepare the download directory for Vantage Express:
+mkdir /opt/downloads
+cd /opt/downloadsInstall VirtualBox and 7zip:
+apt update && apt-get install p7zip-full p7zip-rar virtualbox -yRetrieve the curl command to download Vantage Express.
+Go to Vantage Expess download page (registration required).
+Click on the latest download link, e.g. "Vantage Express 17.20". You will see a license agreement popup. Don’t accept the license yet.
+Open the network view in your browser. For example, in Chrome press F12 and navigate to Network tab:
+Accept the license by clicking on I Agree button and cancel the download.
In the network view, find the last request that starts with VantageExpress. Right click on it and select Copy → Copy as cURL:
+Head back to the ssh session and download Vantage Express by pasting the curl command. Add -o ve.7z to the command to save the download to file named ve.7z. You can remove all the HTTP headers, e.g.:
curl -o ve.7z 'http://d289lrf5tw1zls.cloudfront.net/database/teradata-express/VantageExpress17.20_Sles12_202108300444.7z?Expires=1638719978&Signature=GKBkNvery_long_signature__&Key-Pair-Id=********************'Unzip the downloaded file. It will take several minutes:
+7z x ve.7zStart the VM in VirtualBox. The command will return immediately but the VM init process will take several minutes:
+export VM_IMAGE_DIR="/opt/downloads/VantageExpress17.20_Sles12"
+DEFAULT_VM_NAME="vantage-express"
+VM_NAME="${VM_NAME:-$DEFAULT_VM_NAME}"
+vboxmanage createvm --name "$VM_NAME" --register --ostype openSUSE_64
+vboxmanage modifyvm "$VM_NAME" --ioapic on --memory 6000 --vram 128 --nic1 nat --cpus 4
+vboxmanage storagectl "$VM_NAME" --name "SATA Controller" --add sata --controller IntelAhci
+vboxmanage storageattach "$VM_NAME" --storagectl "SATA Controller" --port 0 --device 0 --type hdd --medium "$(find $VM_IMAGE_DIR -name '*disk1*')"
+vboxmanage storageattach "$VM_NAME" --storagectl "SATA Controller" --port 1 --device 0 --type hdd --medium "$(find $VM_IMAGE_DIR -name '*disk2*')"
+vboxmanage storageattach "$VM_NAME" --storagectl "SATA Controller" --port 2 --device 0 --type hdd --medium "$(find $VM_IMAGE_DIR -name '*disk3*')"
+vboxmanage modifyvm "$VM_NAME" --natpf1 "tdssh,tcp,,4422,,22"
+vboxmanage modifyvm "$VM_NAME" --natpf1 "tddb,tcp,,1025,,1025"
+vboxmanage startvm "$VM_NAME" --type headless
+vboxmanage controlvm "$VM_NAME" keyboardputscancode 1c 1cssh to Vantage Express VM. Use root as password:
ssh -p 4422 root@localhostValidate that the DB is up:
+pdestate -aIf the command returns PDE state is RUN/STARTED. DBS state is 5: Logons are enabled - The system is quiescent, it means that Vantage Express has started.
+If the status is different, repeat pdestate -a till you get the correct status.
Once Vantage Express is up and running, start bteq client command line client. BTEQ (pronounced “bee-teek”) is a general-purpose, command-based client tool used to submit SQL queries to a Teradata Database.
bteqOnce in bteq, connect to your Vantage Express instance. When asked for the password, enter dbc:
.logon localhost/dbcUsing dbc user, we will create a new database called HR. Copy/paste this query and run press Enter:
CREATE DATABASE HR
+AS PERMANENT = 60e6, -- 60MB
+ SPOOL = 120e6; -- 120MBLet’s create a sample table and insert some data and query it. We will first create a table to hold employee information:
+CREATE SET TABLE HR.Employees (
+ GlobalID INTEGER,
+ FirstName VARCHAR(30),
+ LastName VARCHAR(30),
+ DateOfBirth DATE FORMAT 'YYYY-MM-DD',
+ JoinedDate DATE FORMAT 'YYYY-MM-DD',
+ DepartmentCode BYTEINT
+)
+UNIQUE PRIMARY INDEX ( GlobalID );Now, let’s insert a record:
+INSERT INTO HR.Employees (
+ GlobalID,
+ FirstName,
+ LastName,
+ DateOfBirth,
+ JoinedDate,
+ DepartmentCode
+)
+VALUES (
+ 101,
+ 'Adam',
+ 'Tworkowski',
+ '1980-01-05',
+ '2004-08-01',
+ 01
+);Finally, let’s see if we can retrieve the data:
+SELECT * FROM HR.Employees;You should get the following results:
+GlobalID FirstName LastName DateOfBirth JoinedDate DepartmentCode +-------- --------- ---------- ----------- ---------- -------------- + 101 Adam Tworkowski 1980-01-05 2004-08-01 1+
If you intend to stop and start the VM, you may want to add Vantage Express to autostart. ssh to your VM and run the following commands:
+sudo -i
+
+cat <<EOF >> /etc/default/virtualbox
+VBOXAUTOSTART_DB=/etc/vbox
+VBOXAUTOSTART_CONFIG=/etc/vbox/autostart.cfg
+EOF
+
+cat <<EOF > /etc/systemd/system/vantage-express.service
+[Unit]
+Description=vm1
+After=network.target virtualbox.service
+Before=runlevel2.target shutdown.target
+[Service]
+User=root
+Group=root
+Type=forking
+Restart=no
+TimeoutSec=5min
+IgnoreSIGPIPE=no
+KillMode=process
+GuessMainPID=no
+RemainAfterExit=yes
+ExecStart=/usr/bin/VBoxManage startvm vantage-express --type headless
+ExecStop=/usr/bin/VBoxManage controlvm vantage-express savestate
+[Install]
+WantedBy=multi-user.target
+EOF
+
+systemctl daemon-reload
+systemctl enable vantage-express
+systemctl start vantage-expressIf you would like to connect to Vantage Express from the Internet, you will need to open up firewall holes to your VM. You should also change the default password to dbc user:
To change the password for dbc user go to your VM and start bteq:
bteqLogin to your database using dbc as username and password:
.logon localhost/dbcChange the password for dbc user:
MODIFY USER dbc AS PASSWORD = new_password;You can now open up port 1025 to the internet:
+aws ec2 authorize-security-group-ingress \
+ --group-id $AWS_CUSTOM_SECURITY_GROUP_ID \
+ --ip-permissions '[{"IpProtocol": "tcp", "FromPort": 1025, "ToPort": 1025, "IpRanges": [{"CidrIp": "0.0.0.0/0", "Description": "Allow Teradata port"}]}]'To stop incurring charges, delete all the resources:
+# Delete the VM
+aws ec2 terminate-instances --instance-ids $AWS_INSTANCE_ID --output text
+
+# Wait for the VM to terminate
+
+# Delete custom security group
+aws ec2 delete-security-group \
+ --group-id $AWS_CUSTOM_SECURITY_GROUP_ID
+
+# Delete internet gateway
+aws ec2 detach-internet-gateway \
+ --internet-gateway-id $AWS_INTERNET_GATEWAY_ID \
+ --vpc-id $AWS_VPC_ID &&
+ aws ec2 delete-internet-gateway \
+ --internet-gateway-id $AWS_INTERNET_GATEWAY_ID
+
+# Delete the custom route table
+aws ec2 disassociate-route-table \
+ --association-id $AWS_ROUTE_TABLE_ASSOID &&
+ aws ec2 delete-route-table \
+ --route-table-id $AWS_CUSTOM_ROUTE_TABLE_ID
+
+# Delete the public subnet
+aws ec2 delete-subnet \
+ --subnet-id $AWS_SUBNET_PUBLIC_ID
+
+# Delete the vpc
+aws ec2 delete-vpc \
+ --vpc-id $AWS_VPC_ID| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Adam Tworkiewicz
+Last updated: August 23rd, 2022
This how-to demonstrates how to run Vantage Express in Microsoft Azure. Vantage Express contains a fully functional Teradata SQL Engine.
+| + + | ++If do not wish to pay for cloud usage you can install Vantage Express locally using VMware, VirtualBox, or UTM. + | +
An Azure account. You can create one here: https://azure.microsoft.com/en-us/free/
+az command line utility installed on your machine. You can find installation instructions here: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli.
Setup the default region to the closest region to you (to list locations run az account list-locations -o table):
az config set defaults.location=<location>Create a new resource group called tdve-resource-group and add it to defaults:
az group create -n tdve-resource-group
+az config set defaults.group=tdve-resource-groupTo create a VM you will need an ssh key pair. If you don’t have it already, create one:
+az sshkey create --name vantage-ssh-keyRestrict access to the private key. Replace <path_to_private_key_file> with the private key path returned by the previous command:
chmod 600 <path_to_private_key_file>Create a Ubuntu VM with 4 CPU’s and 8GB of RAM, a 30GB os disk and a 60GB data disk.
+az disk create -n teradata-vantage-express --size-gb 60
+az vm create `
+ --name teradata-vantage-express `
+ --image UbuntuLTS `
+ --admin-username azureuser `
+ --ssh-key-name vantage-ssh-key `
+ --size Standard_F4s_v2 `
+ --public-ip-sku Standard
+
+$diskId = (az disk show -n teradata-vantage-express --query 'id' -o tsv) | Out-String
+az vm disk attach --vm-name teradata-vantage-express --name $diskIdaz disk create -n teradata-vantage-express --size-gb 60
+az vm create \
+ --name teradata-vantage-express \
+ --image UbuntuLTS \
+ --admin-username azureuser \
+ --ssh-key-name vantage-ssh-key \
+ --size Standard_F4s_v2 \
+ --public-ip-sku Standard
+
+DISK_ID=$(az disk show -n teradata-vantage-express --query 'id' -o tsv)
+az vm disk attach --vm-name teradata-vantage-express --name $DISK_IDaz disk create -n teradata-vantage-express --size-gb 60
+az vm create \
+ --name teradata-vantage-express \
+ --image UbuntuLTS \
+ --admin-username azureuser \
+ --ssh-key-name vantage-ssh-key \
+ --size Standard_F4s_v2 \
+ --public-ip-sku Standard
+
+DISK_ID=$(az disk show -n teradata-vantage-express --query 'id' -o tsv)
+az vm disk attach --vm-name teradata-vantage-express --name $DISK_IDssh to your VM. Replace <path_to_private_key_file> and <vm_ip> with values that match your environment:
ssh -i <path_to_private_key_file> azureuser@<vm_ip>Once in the VM, switch to root user:
sudo -iPrepare the download directory for Vantage Express:
+mkdir /opt/downloads
+cd /opt/downloadsMount the data disk:
+parted /dev/sdc --script mklabel gpt mkpart xfspart xfs 0% 100%
+mkfs.xfs /dev/sdc1
+partprobe /dev/sdc1
+export DISK_UUID=$(blkid | grep sdc1 | cut -d"\"" -f2)
+echo "UUID=$DISK_UUID /opt/downloads xfs defaults,nofail 1 2" >> /etc/fstabInstall VirtualBox and 7zip:
+apt update && apt-get install p7zip-full p7zip-rar virtualbox -yRetrieve the curl command to download Vantage Express.
+Go to Vantage Expess download page (registration required).
+Click on the latest download link, e.g. "Vantage Express 17.20". You will see a license agreement popup. Don’t accept the license yet.
+Open the network view in your browser. For example, in Chrome press F12 and navigate to Network tab:
+Accept the license by clicking on I Agree button and cancel the download.
In the network view, find the last request that starts with VantageExpress. Right click on it and select Copy → Copy as cURL:
+Head back to the ssh session and download Vantage Express by pasting the curl command. Add -o ve.7z to the command to save the download to file named ve.7z. You can remove all the HTTP headers, e.g.:
curl -o ve.7z 'http://d289lrf5tw1zls.cloudfront.net/database/teradata-express/VantageExpress17.20_Sles12_202108300444.7z?Expires=1638719978&Signature=GKBkNvery_long_signature__&Key-Pair-Id=********************'Unzip the downloaded file. It will take several minutes:
+7z x ve.7zStart the VM in VirtualBox. The command will return immediately but the VM init process will take several minutes:
+export VM_IMAGE_DIR="/opt/downloads/VantageExpress17.20_Sles12"
+DEFAULT_VM_NAME="vantage-express"
+VM_NAME="${VM_NAME:-$DEFAULT_VM_NAME}"
+vboxmanage createvm --name "$VM_NAME" --register --ostype openSUSE_64
+vboxmanage modifyvm "$VM_NAME" --ioapic on --memory 6000 --vram 128 --nic1 nat --cpus 4
+vboxmanage storagectl "$VM_NAME" --name "SATA Controller" --add sata --controller IntelAhci
+vboxmanage storageattach "$VM_NAME" --storagectl "SATA Controller" --port 0 --device 0 --type hdd --medium "$(find $VM_IMAGE_DIR -name '*disk1*')"
+vboxmanage storageattach "$VM_NAME" --storagectl "SATA Controller" --port 1 --device 0 --type hdd --medium "$(find $VM_IMAGE_DIR -name '*disk2*')"
+vboxmanage storageattach "$VM_NAME" --storagectl "SATA Controller" --port 2 --device 0 --type hdd --medium "$(find $VM_IMAGE_DIR -name '*disk3*')"
+vboxmanage modifyvm "$VM_NAME" --natpf1 "tdssh,tcp,,4422,,22"
+vboxmanage modifyvm "$VM_NAME" --natpf1 "tddb,tcp,,1025,,1025"
+vboxmanage startvm "$VM_NAME" --type headless
+vboxmanage controlvm "$VM_NAME" keyboardputscancode 1c 1cssh to Vantage Express VM. Use root as password:
ssh -p 4422 root@localhostValidate that the DB is up:
+pdestate -aIf the command returns PDE state is RUN/STARTED. DBS state is 5: Logons are enabled - The system is quiescent, it means that Vantage Express has started.
+If the status is different, repeat pdestate -a till you get the correct status.
Once Vantage Express is up and running, start bteq client command line client. BTEQ (pronounced “bee-teek”) is a general-purpose, command-based client tool used to submit SQL queries to a Teradata Database.
bteqOnce in bteq, connect to your Vantage Express instance. When asked for the password, enter dbc:
.logon localhost/dbcUsing dbc user, we will create a new database called HR. Copy/paste this query and run press Enter:
CREATE DATABASE HR
+AS PERMANENT = 60e6, -- 60MB
+ SPOOL = 120e6; -- 120MBLet’s create a sample table and insert some data and query it. We will first create a table to hold employee information:
+CREATE SET TABLE HR.Employees (
+ GlobalID INTEGER,
+ FirstName VARCHAR(30),
+ LastName VARCHAR(30),
+ DateOfBirth DATE FORMAT 'YYYY-MM-DD',
+ JoinedDate DATE FORMAT 'YYYY-MM-DD',
+ DepartmentCode BYTEINT
+)
+UNIQUE PRIMARY INDEX ( GlobalID );Now, let’s insert a record:
+INSERT INTO HR.Employees (
+ GlobalID,
+ FirstName,
+ LastName,
+ DateOfBirth,
+ JoinedDate,
+ DepartmentCode
+)
+VALUES (
+ 101,
+ 'Adam',
+ 'Tworkowski',
+ '1980-01-05',
+ '2004-08-01',
+ 01
+);Finally, let’s see if we can retrieve the data:
+SELECT * FROM HR.Employees;You should get the following results:
+GlobalID FirstName LastName DateOfBirth JoinedDate DepartmentCode +-------- --------- ---------- ----------- ---------- -------------- + 101 Adam Tworkowski 1980-01-05 2004-08-01 1+
If you intend to stop and start the VM, you may want to add Vantage Express to autostart. ssh to your VM and run the following commands:
+sudo -i
+
+cat <<EOF >> /etc/default/virtualbox
+VBOXAUTOSTART_DB=/etc/vbox
+VBOXAUTOSTART_CONFIG=/etc/vbox/autostart.cfg
+EOF
+
+cat <<EOF > /etc/systemd/system/vantage-express.service
+[Unit]
+Description=vm1
+After=network.target virtualbox.service
+Before=runlevel2.target shutdown.target
+[Service]
+User=root
+Group=root
+Type=forking
+Restart=no
+TimeoutSec=5min
+IgnoreSIGPIPE=no
+KillMode=process
+GuessMainPID=no
+RemainAfterExit=yes
+ExecStart=/usr/bin/VBoxManage startvm vantage-express --type headless
+ExecStop=/usr/bin/VBoxManage controlvm vantage-express savestate
+[Install]
+WantedBy=multi-user.target
+EOF
+
+systemctl daemon-reload
+systemctl enable vantage-express
+systemctl start vantage-expressIf you would like to connect to Vantage Express from the Internet, you will need to open up firewall holes to your VM. You should also change the default password to dbc user:
To change the password for dbc user go to your VM and start bteq:
bteqLogin to your database using dbc as username and password:
.logon localhost/dbcChange the password for dbc user:
MODIFY USER dbc AS PASSWORD = new_password;You can now open up port 1025 to the internet using gcloud command:
+az vm open-port --name teradata-vantage-express --port 1025To stop incurring charges, delete all the resources associated with the resource group:
+az group delete --no-wait -n tdve-resource-group| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Adam Tworkiewicz
+Last updated: January 18th, 2022
This solution listens to events from Twilio Segment and writes data to a Teradata Vantage instance. The example uses Google Cloud but it can be translated into any cloud platform.
+In this solution, Twilio Segment writes raw event data to Google Cloud Pub/Sub. Pub/Sub forwards events to a Cloud Run application. The Cloud Run app writes data to a Teradata Vantage database. It’s a serverless solution that doesn’t require allocation or management of any VM’s.
+
+A Google Cloud account. If you don’t have an account, you can create one at https://console.cloud.google.com/.
+gcloud installed. See https://cloud.google.com/sdk/docs/install.
A Teradata Vantage instance that Google Cloud Run can talk to.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
Clone the sample repository:
+git clone git@github.com:Teradata/segment-integration-tutorial.gitThe repo contains segment.sql file that sets up the database. the script on your Vantage db using your favorite SQL IDE, [Teradata Studio](https://downloads.teradata.com/download/tools/teradata-studio) or command line tool called bteq (download for Windows, Linux, macOS).
+The SQL script will create a new database called Segment and a set of tables to store Segment events.
Set the default project and region:
+gcloud config set project <PROJECT_ID>
+gcloud config set compute/region <REGION>Retrieve the project id and the number. We will need it in subsequent steps:
+export PROJECT_ID=$(gcloud config get-value project)
+
+export PROJECT_NUMBER=$(gcloud projects list \
+ --filter="$(gcloud config get-value project)" \
+ --format="value(PROJECT_NUMBER)")Enable required Google Cloud services:
+gcloud services enable cloudbuild.googleapis.com containerregistry.googleapis.com run.googleapis.com secretmanager.googleapis.com pubsub.googleapis.comBuild the application:
+gcloud builds submit --tag gcr.io/$PROJECT_ID/segment-listenerDefine an API key that you will share with Segment. Store the API key in Google Cloud Secret Manager:
+gcloud secrets create VANTAGE_USER_SECRET
+echo -n 'dbc' > /tmp/vantage_user.txt
+gcloud secrets versions add VANTAGE_USER_SECRET --data-file=/tmp/vantage_user.txt
+
+gcloud secrets create VANTAGE_PASSWORD_SECRET
+echo -n 'dbc' > /tmp/vantage_password.txt
+gcloud secrets versions add VANTAGE_PASSWORD_SECRET --data-file=/tmp/vantage_password.txtThe application that write Segment data to Vantage will use Cloud Run. We first need to allow Cloud Run to access secrets:
+gcloud projects add-iam-policy-binding $PROJECT_ID \
+ --member=serviceAccount:$PROJECT_NUMBER-compute@developer.gserviceaccount.com \
+ --role=roles/secretmanager.secretAccessorDeploy the app to Cloud Run (replace <VANTAGE_HOST> with the hostname or IP of your Teradata Vantage database). The second export statement saves the service url as we need it for subsequent commands:
gcloud run deploy --image gcr.io/$PROJECT_ID/segment-listener segment-listener \
+ --region $(gcloud config get-value compute/region) \
+ --update-env-vars VANTAGE_HOST=35.239.251.1 \
+ --update-secrets 'VANTAGE_USER=VANTAGE_USER_SECRET:1, VANTAGE_PASSWORD=VANTAGE_PASSWORD_SECRET:1' \
+ --no-allow-unauthenticated
+
+export SERVICE_URL=$(gcloud run services describe segment-listener --platform managed --region $(gcloud config get-value compute/region) --format 'value(status.url)')Create a Pub/Sub topic that will receive events from Segment:
+gcloud pubsub topics create segment-eventsCreate a service account that will be used by Pub/Sub to invoke the Cloud Run app:
+gcloud iam service-accounts create cloud-run-pubsub-invoker \
+ --display-name "Cloud Run Pub/Sub Invoker"Give the service account permission to invoke Cloud Run:
+gcloud run services add-iam-policy-binding segment-listener \
+ --region $(gcloud config get-value compute/region) \
+ --member=serviceAccount:cloud-run-pubsub-invoker@$PROJECT_ID.iam.gserviceaccount.com \
+ --role=roles/run.invokerAllow Pub/Sub to create authentication tokens in your project:
+gcloud projects add-iam-policy-binding $PROJECT_ID \
+ --member=serviceAccount:service-$PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com \
+ --role=roles/iam.serviceAccountTokenCreatorCreate a Pub/Sub subscription with the service account:
+gcloud pubsub subscriptions create segment-events-cloudrun-subscription --topic projects/$PROJECT_ID/topics/segment-events \
+ --push-endpoint=$SERVICE_URL \
+ --push-auth-service-account=cloud-run-pubsub-invoker@$PROJECT_ID.iam.gserviceaccount.com \
+ --max-retry-delay 600 \
+ --min-retry-delay 30Allow Segment to publish to your topic. To do that, assign pubsub@segment-integrations.iam.gserviceaccount.com role Pub/Sub Publisher in your project at https://console.cloud.google.com/cloudpubsub/topic/list. See Segment manual for details.
Configure your Google Cloud Pub/Sub a destination in Segment. Use the full topic projects/<PROJECT_ID>/topics/segment-events and map all Segment event types (using * character) to the topic.
Use Segment’s Event Tester functionality to send a sample payload to the topic. Verify that the sample data has been stored in Vantage.
+The example shows how to deploy the app in a single region. In many cases, this setup doesn’t guarantee enough uptime. The Cloud Run app should be deployed in more than one region behind a Global Load Balancer.
+This how-to demonstrates how to send Segment events to Teradata Vantage. The configuration forwards events from Segment to Google Cloud Pub/Sub and then on to a Cloud Run application. The application writes data to Teradata Vantage.
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Krutik Pathak
+Last updated: August 9th, 2023
This article outlines different use cases involving data ingestion, lists available tools, and recommends the optimal tool for each use case.
+Teradata NOS: NOS is the recommended option to ingest data from files in object storage. This situation is very common in the cloud and on-prem data fabric systems.
+| + + | +
+
+
+Teradata Parallel Transporter (TPT) could be used to load data from external object storage into Teradata Vantage, however the recommended tool is Teradata NOS. + |
+
Teradata Parallel Transporter (TPT): TPT is the recommended option to load data from local files. TPT is optimized for scalability and parallelism, thus it has the best throughput from all available options.
+BTEQ: BTEQ has full scripting capabilities and the ability to read files. It is a good option if it is already the primary ingestion tool used by a customer.
+Teradata QueryGrid: QueryGrid is the recommended option to move limited quantities of data between different systems/platforms. This includes movement within Vantage instances, Apache Spark, Oracle, Presto, etc. It is especially suited to situations when what needs to be synced is described by complex conditions that can be expressed in SQL.
+Airbyte: Airbyte is an ELT tool that has more than 350 connectors and is Open Source. It’s a favored option for conducting lightweight ingestions from SaaS applications into Teradata Vantage.
+In this article, we explored various data ingestion use cases, provided a list of available tools for each use case, and identified the recommended option for different scenarios.
+Author: Adam Tworkiewicz
+Last updated: September 7th, 2021
Sometimes, you need to apply complex logic to your data that can’t be easily expressed in SQL. One option is to wrap your logic in a User Defined Function (UDF). What if you already have this logic coded in a language that is not supported by UDF? Script Table Operator is a Vantage feature that allows you to bring your logic to the data and run it on Vantage. The advantage of this approach is that you don’t have to retrieve data from Vantage to operate on it. Also, by running your data applications on Vantage, you leverage its parallel nature. You don’t have to think how your applications will scale. You can let Vantage take care of it.
+You need access to a Teradata Vantage instance.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
Let’s start with something simple. What if you wanted the database to print "Hello World"?
+SELECT *
+FROM
+ SCRIPT(
+ SCRIPT_COMMAND('echo Hello World!')
+ RETURNS ('Message varchar(512)'));Here is what I’ve got:
+Message +------------ +Hello World! +Hello World!+
Let’s analyze what just happened here. The SQL includes echo Hello World!. This is a Bash command. OK, so now we know how to run Bash commands. But why did we get 2 rows and not one? That’s because our simple script was run once on each AMP and I happen to have 2 AMPs:
-- Teradata magic that returns the number of AMPs in a system
+SELECT hashamp()+1 AS number_of_amps;Returns:
+number_of_amps +-------------- + 2+
This simple script demonstrates the idea behind the Script Table Operator (STO). You provide your script and the database runs it in parallel, once for each AMP. This is an attractive model in case you have transformation logic in a script and a lot of data to process. Normally, you would need to build concurrency into your application. By letting STO do it, you let Teradata select the right concurrency level for your data.
+OK, so we did echo in Bash but Bash is hardly a productive environment to express complex logic. What other languages are supported then? The good news is that any binary that can run on Vantage nodes can be used in STO. Remember, that the binary and all its dependencies need to be installed on all your Vantage nodes. In practice, it means that your options will be limited to what your administrator is willing and able to maintain on your servers. Python is a very popular choice.
Ok, Hello World is super exciting, but what if we have existing logic in a large file. Surely, you don’t want to paste your entire script and escape quotes in an SQL query. We solve the script upload issue with the User Installed Files (UIF) feature.
+Say you have helloworld.py script with the following content:
print("Hello World!")Let’s assume the script is on your local machine at /tmp/helloworld.py.
First, we need to setup permissions in Vantage. We are going to do this using a new database to keep it clean.
+-- Create a new database called sto
+CREATE DATABASE STO
+AS PERMANENT = 60e6, -- 60MB
+ SPOOL = 120e6; -- 120MB
+
+-- Allow dbc user to create scripts in database STO
+GRANT CREATE EXTERNAL PROCEDURE ON STO to dbc;You can upload the script to Vantage using the following procedure call:
+call SYSUIF.install_file('helloworld',
+ 'helloworld.py', 'cz!/tmp/helloworld.py');Now that the script has been uploaded, you can call it like this:
+-- We switch to STO database
+DATABASE STO
+
+-- We tell Vantage where to look for the script. This can be
+-- any string and it will create a symbolic link to the directory
+-- where our script got uploaded. By convention, we use the
+-- database name.
+SET SESSION SEARCHUIFDBPATH = sto;
+
+-- We now call the script. Note, how we use a relative path that
+-- starts with `./sto/`, which is where SEARCHUIFDBPATH
+-- is pointing.
+SELECT *
+FROM SCRIPT(
+ SCRIPT_COMMAND('python3 ./sto/helloworld.py')
+ RETURNS ('Message varchar(512)'));The last call should return:
+Message +------------ +Hello World! +Hello World!+
That was a lot of work and we are still at Hello World. Let’s try to pass some data into SCRIPT.
So far, we have been using SCRIPT operator to run standalone scripts. But the main purpose to run scripts on Vantage is to process data that is in Vantage. Let’s see how we can retrieve data from Vantage and pass it to SCRIPT.
We will start with creating a table with a few rows.
+-- Switch to STO database.
+DATABASE STO
+
+-- Create a table with a few urls
+CREATE TABLE urls(url varchar(10000));
+INS urls('https://www.google.com/finance?q=NYSE:TDC');
+INS urls('http://www.ebay.com/sch/i.html?_trksid=p2050601.m570.l1313.TR0.TRC0.H0.Xteradata+merchandise&_nkw=teradata+merchandise&_sacat=0&_from=R40');
+INS urls('https://www.youtube.com/results?search_query=teradata%20commercial&sm=3');
+INS urls('https://www.contrivedexample.com/example?mylist=1&mylist=2&mylist=...testing');We will use the following script to parse out query parameters:
+from urllib.parse import urlparse
+from urllib.parse import parse_qsl
+import sys
+
+for line in sys.stdin:
+ # remove leading and trailing whitespace
+ url = line.strip()
+ parsed_url = urlparse(url)
+ query_params = parse_qsl(parsed_url.query)
+
+ for element in query_params:
+ print("\t".join(element))Note, how the scripts assumes that urls will be fed into stdin one by one, line by line. Also, note how it prints results line by line, using the tab character as a delimiter between values.
Let’s install the script. Here, we assume that the script file is at /tmp/urlparser.py on our local machine:
CALL SYSUIF.install_file('urlparser',
+ 'urlparser.py', 'cz!/tmp/urlparser.py');With the script installed, we will now retrieve data from urls table and feed it into the script to retrieve query parameters:
-- We inform Vantage to create a symbolic link from the UIF directory to ./sto/
+SET SESSION SEARCHUIFDBPATH = sto ;
+
+SELECT *
+ FROM SCRIPT(
+ ON(SELECT url FROM urls)
+ SCRIPT_COMMAND('python3 ./sto/urlparser.py')
+ RETURNS ('param_key varchar(512)', 'param_value varchar(512)'));As a result, we get query params and their values. There are as many rows as key/value pairs. Also, since we inserted a tab between the key and the value output in the script, we get 2 columns from STO.
+param_key |param_value +------------+----------------------------------------------------- +q |NYSE:TDC +_trksid |p2050601.m570.l1313.TR0.TRC0.H0.Xteradata merchandise +search_query|teradata commercial +_nkw |teradata merchandise +sm |3 +_sacat |0 +mylist |1 +_from |R40 +mylist |2 +mylist |...testing+
We have learned how to take data from Vantage, pass it to a script and get output. Is there an easy way to store this output in a table? Sure, there is. We can combine the select above with CREATE TABLE statement:
-- We inform Vantage to create a symbolic link from the UIF directory to ./sto/
+SET SESSION SEARCHUIFDBPATH = sto ;
+
+CREATE MULTISET TABLE
+ url_params(param_key, param_value)
+AS (
+ SELECT *
+ FROM SCRIPT(
+ ON(SELECT url FROM urls)
+ SCRIPT_COMMAND('python3 ./sto/urlparser.py')
+ RETURNS ('param_key varchar(512)', 'param_value varchar(512)'))
+) WITH DATA
+NO PRIMARY INDEX;Now, let’s inspect the contents of url_params table:
SELECT * FROM url_params;You should see the following output:
+param_key |param_value +------------+----------------------------------------------------- +q |NYSE:TDC +_trksid |p2050601.m570.l1313.TR0.TRC0.H0.Xteradata merchandise +search_query|teradata commercial +_nkw |teradata merchandise +sm |3 +_sacat |0 +mylist |1 +_from |R40 +mylist |2 +mylist |...testing+
In this quick start we have learned how to run scripts against data in Vantage. We ran scripts using Script Table Operator (STO). The operator allows us to bring logic to the data. It offloads concurrency considerations to the database by running our scripts in parallel, one per AMP. All you need to do is provide a script and the database will execute it in parallel.
+| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Krutik Pathak
+Last updated: August 7th, 2023
This article explains the underlying concepts of Teradata Vantage engine architecture. All editions of Vantage, including the Primary Cluster in VantageCloud Lake utilize the same engine.
+Teradata’s architecture is designed around a Massively Parallel Processing (MPP), shared-nothing architecture, which enables high-performance data processing and analytics. The MPP architecture distributes the workload into multiple vprocs or virtual processors. The virtual processor where query processing takes place is commonly referred to as an Access Module Processor (AMP). Each AMP is isolated from other AMPs, and processes the queries in parallel allowing Teradata to process large volumes of data rapidly.
+The major architectural components of the Teradata Vantage engine include the Parsing Engines (PEs), BYNET, Access Module Processors (AMPs), and Virtual Disks (Vdisks). Vdisks are assigned to AMPs in enterprise platforms, and to the Primary Cluster in the case of VantageCloud Lake environments.
+
+The Teradata Vantage engine consists of the components below:
+When a SQL query is run in Teradata, it first reaches the Parsing Engine. The functions of the Parsing Engine are:
+Manage individual user sessions (up to 120).
+Check if the objects used in the SQL query exist.
+Check if the user has required privileges against the objects used in the SQL query.
+Parse and optimize the SQL queries.
+Prepare the execution plan to execute the SQL query and passes it to the corresponding AMPs.
+Receive the response from the AMPs and send it back to the requesting client.
+BYNET is a system that enables component communication. The BYNET system provides high-speed bi-directional broadcast, multicast, and point-to-point communication and merge functions. It performs three key functions: coordinating multi-AMP queries, reading data from multiple AMPs, regulating message flow to prevent congestion, and processing platform throughput. These functions of BYNET make Vantage highly scalable and enable Massively Parallel Processing (MPP) capabilities.
+Parallel Database Extension (PDE) is an intermediary software layer positioned between the operating system and the Teradata Vantage database. PDE enables MPP systems to use features such as BYNET and shared disks. It facilitates the parallelism that is responsible for the speed and linear scalability of the Teradata Vantage database.
+AMPs are responsible for data storage and retrieval. Each AMP is associated with its own set of Virtual Disks (Vdisks) where the data is stored, and no other AMP can access that content in line with the shared-nothing architecture. The functions of AMP are:
+Access storage using Vantage’s Block File System Software
+Lock management
+Sorting rows
+Aggregating columns
+Join processing
+Output conversion
+Disk space management
+Accounting
+Recovery processing
+| + + | +
+
+
+AMPs in VantageCore IntelliFlex, VantageCore VMware, VantageCloud Enterprise, and the Primary Cluster in the case of VantageCloud Lake, store data in a Block File System (BFS) format on Vdisks. AMPs in Compute Clusters and Compute Worker Nodes on VantageCloud Lake do not have BFS, they can only access data in object storage using the Object File System (OFS). + |
+
These are units of storage space owned by an AMP. Virtual Disks are used to hold user data (rows within tables). Virtual Disks map to physical space on a disk.
+A node, in the context of Teradata systems, represents an individual server that functions as a hardware platform for the database software. It serves as a processing unit where database operations are executed under the control of a single operating system. When Teradata is deployed in a cloud, it follows the same MPP, shared-nothing architecture but the physical nodes are replaced with virtual machines (VMs).
+The concepts below are applicable to Teradata Vantage.
+Teradata is a linearly expandable RDBMS. As the workload and data volume increase, adding more hardware resources such as servers or nodes results in a proportional increase in performance and capacity. Linear Scalability allows for increased workload without decreased throughput.
+Teradata parallelism refers to the inherent ability of the Teradata Database to perform parallel processing of data and queries across multiple nodes or components simultaneously.
+Each Parsing Engine (PE) in Teradata has the capability to handle up to 120 sessions concurrently.
+The BYNET in Teradata enables parallel handling of all message activity, including data redistribution for subsequent tasks.
+All Access Module Processors (AMPs) in Teradata can collaborate in parallel to serve any incoming request.
+Each AMP can work on multiple requests concurrently, allowing for efficient parallel processing.
+
+The key steps involved in Teradata Retrieval Architecture are:
+The Parsing Engine sends a request to retrieve one or more rows.
+The BYNET activates the relevant AMP(s) for processing.
+The AMP(s) concurrently locate and retrieve the desired row(s) through parallel access.
+The BYNET returns the retrieved row(s) to the Parsing Engine.
+The Parsing Engine then delivers the row(s) back to the requesting client application.
+
+Teradata’s MPP architecture requires an efficient means of distributing and retrieving data and does so using hash partitioning. Most tables in Vantage use hashing to distribute data for the tables based on the value of the row’s Primary Index (PI) to disk storage in Block File System (BFS) and may scan the entire table or use indexes to access the data. This approach ensures scalable performance and efficient data access.
+If the Primary Index is unique then the rows in the tables are automatically distributed evenly by hash partitioning.
+The designated Primary Index column(s) are hashed to generate consistent hash codes for the same values.
+No reorganization, repartitioning, or space management is required.
+Each AMP typically contains rows from all tables, ensuring efficient data access and processing.
+
+In this article, we covered the major architectural components of Teradata Vantage, such as the Parsing Engines (PEs), BYNET, Access Module Processors (AMPs), Virtual Disk (Vdisk), other architectural components such as Parallel Database Extension (PDE), Node and the essential concepts of Teradata Vantage such as Linear Growth and Expandability, Parallelism, Data Retrieval, and Data Distribution.
+Author: Krutik Pathak
+Last updated: August 2nd, 2023
This how-to demonstrates how to connect to Vantage using teradatasql Python database driver for Teradata Vantage.
+64-bit Python 3.4 or later.
+teradatasql driver installed in your system:
pip install teradatasql| + + | +
+
+
+
|
+
Access to a Teradata Vantage instance. Currently driver is supported for use with Teradata Database 16.10 and later releases.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
This is a simple Python code to connect to Teradata Vantage using teradatasql. All that is left, is to pass connection and authentication parameters and run a query:
This how-to demonstrated how to connect to Teradata Vantage using teradatasql Python database driver. It described a sample Python code to send SQL queries to Teradata Vantage using teradatasql.
| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Adam Tworkiewicz
+Last updated: April 6th, 2022
We often have a need to move large volumes of data into Vantage. Teradata offers Teradata Parallel Transporter (TPT) utility that can efficiently load large amounts of data into Teradata Vantage. This how-to demonstrates how to use TPT. In this scenario, we will load over 300k records, over 40MB of data, in a couple of seconds.
Access to a Teradata Vantage instance.
+| + + | ++If you need a test instance of Vantage, you can provision one for free at https://clearscape.teradata.com. + | +
Download Teradata Tools and Utilities (TTU) - supported platforms: Windows, MacOS, Linux (requires registration).
+We will be working with the US tax fillings for nonprofit organizations. Nonprofit tax filings are public data. The US Internal Revenue Service publishes them in S3 bucket. Let’s grab a summary of filings for 2020: https://downloads.teradata.com/sites/default/files/2022-11/index_2020.csv. You can use your browser, wget or curl to save the file locally.
Let’s create a database in Vantage. Use your favorite SQL tool to run the following query:
+CREATE DATABASE irs
+AS PERMANENT = 120e6, -- 120MB
+ SPOOL = 120e6; -- 120MBWe will now run TPT. TPT is a command-line tool that can be used to load, extract and update data in Teradata Vantage. These various functions are implemented in so called operators. For example, loading data into Vantage is handled by the Load operator. The Load operator is very efficient in uploading large amounts of data into Vantage. The Load operator, in order to be fast, has several restrictions in place. It can only populate empty tables. Inserts to already populated tables are not supported. It doesn’t support tables with secondary indices. Also, it won’t insert duplicate records, even if a table is a MULTISET table. For the full list of restrictions check out Teradata® TPT Reference - Load Operator - Restrictions and Limitations.
TPT has its own scripting language. The language allows you to prepare the database with arbitrary SQL commands, declare the input source and define how the data should be inserted into Vantage.
+To load the csv data to Vantage, we will define and run a job. The job will prepare the database. It will remove old log and error tables and create the target table. It will then read the file and insert the data into the database.
+Create a job variable file that will tell TPT how to connect to our Vantage database. Create file jobvars.txt and insert the following content. Replace host with the host name of your database. For example, if you are using a local Vantage Express instance, use 127.0.0.1. username with the database user name, and password with the database password. Note that the preparation step (DDL) and the load step have their own configuration values and that the config values need to be entered twice to configure both the DDL and the load step.
TargetTdpId = 'host'
+TargetUserName = 'username'
+TargetUserPassword = 'password'
+
+FileReaderDirectoryPath = ''
+FileReaderFileName = 'index_2020.csv'
+FileReaderFormat = 'Delimited'
+FileReaderOpenMode = 'Read'
+FileReaderTextDelimiter = ','
+FileReaderSkipRows = 1
+
+DDLErrorList = '3807'
+
+LoadLogTable = 'irs.irs_returns_lg'
+LoadErrorTable1 = 'irs.irs_returns_et'
+LoadErrorTable2 = 'irs.irs_returns_uv'
+LoadTargetTable = 'irs.irs_returns'Create a file with the following content and save it as load.txt. See comments within the job file to understand its structure.
DEFINE JOB file_load
+DESCRIPTION 'Load a Teradata table from a file'
+(
+ /*
+ Define the schema of the data in the csv file
+ */
+ DEFINE SCHEMA SCHEMA_IRS
+ (
+ in_return_id VARCHAR(19),
+ in_filing_type VARCHAR(5),
+ in_ein VARCHAR(19),
+ in_tax_period VARCHAR(19),
+ in_sub_date VARCHAR(22),
+ in_taxpayer_name VARCHAR(100),
+ in_return_type VARCHAR(5),
+ in_dln VARCHAR(19),
+ in_object_id VARCHAR(19)
+ );
+
+ /*
+ In the first step, we are sending statements to remove old tables
+ and create a new one.
+ This step replies on configuration stored in `od_IRS` operator
+ */
+ STEP st_Setup_Tables
+ (
+ APPLY
+ ('DROP TABLE ' || @LoadLogTable || ';'),
+ ('DROP TABLE ' || @LoadErrorTable1 || ';'),
+ ('DROP TABLE ' || @LoadErrorTable2 || ';'),
+ ('DROP TABLE ' || @LoadTargetTable || ';'),
+ ('CREATE TABLE ' || @LoadTargetTable || ' (
+ return_id INT,
+ filing_type VARCHAR(5) CHARACTER SET LATIN NOT CASESPECIFIC,
+ ein INT,
+ tax_period INT,
+ sub_date VARCHAR(100) CHARACTER SET LATIN NOT CASESPECIFIC,
+ taxpayer_name VARCHAR(100) CHARACTER SET LATIN NOT CASESPECIFIC,
+ return_type VARCHAR(5) CHARACTER SET LATIN NOT CASESPECIFIC,
+ dln BIGINT,
+ object_id BIGINT
+ )
+ PRIMARY INDEX ( return_id );')
+ TO OPERATOR ($DDL);
+ );
+
+ /*
+ Finally, in this step we read the data from the file operator
+ and send it to the load operator.
+ */
+ STEP st_Load_File
+ (
+ APPLY
+ ('INSERT INTO ' || @LoadTargetTable || ' (
+ return_id,
+ filing_type,
+ ein,
+ tax_period,
+ sub_date,
+ taxpayer_name,
+ return_type,
+ dln,
+ object_id
+ ) VALUES (
+ :in_return_id,
+ :in_filing_type,
+ :in_ein,
+ :in_tax_period,
+ :in_sub_date,
+ :in_taxpayer_name,
+ :in_return_type,
+ :in_dln,
+ :in_object_id
+ );')
+ TO OPERATOR ($LOAD)
+ SELECT * FROM OPERATOR($FILE_READER(SCHEMA_IRS));
+ );
+);Run the job:
+tbuild -f load.txt -v jobvars.txt -j file_loadA successful run will return logs that look like this:
+Teradata Parallel Transporter Version 17.10.00.10 64-Bit +The global configuration file '/opt/teradata/client/17.10/tbuild/twbcfg.ini' is used. + Log Directory: /opt/teradata/client/17.10/tbuild/logs + Checkpoint Directory: /opt/teradata/client/17.10/tbuild/checkpoint + +Job log: /opt/teradata/client/17.10/tbuild/logs/file_load-4.out +Job id is file_load-4, running on osboxes +Teradata Parallel Transporter SQL DDL Operator Version 17.10.00.10 +od_IRS: private log not specified +od_IRS: connecting sessions +od_IRS: sending SQL requests +od_IRS: TPT10508: RDBMS error 3807: Object 'irs_returns_lg' does not exist. +od_IRS: TPT18046: Error is ignored as requested in ErrorList +od_IRS: TPT10508: RDBMS error 3807: Object 'irs_returns_et' does not exist. +od_IRS: TPT18046: Error is ignored as requested in ErrorList +od_IRS: TPT10508: RDBMS error 3807: Object 'irs_returns_uv' does not exist. +od_IRS: TPT18046: Error is ignored as requested in ErrorList +od_IRS: disconnecting sessions +od_IRS: Total processor time used = '0.013471 Second(s)' +od_IRS: Start : Thu Apr 7 20:56:32 2022 +od_IRS: End : Thu Apr 7 20:56:32 2022 +Job step st_Setup_Tables completed successfully +Teradata Parallel Transporter Load Operator Version 17.10.00.10 +ol_IRS: private log not specified +Teradata Parallel Transporter DataConnector Operator Version 17.10.00.10 +op_IRS[1]: Instance 1 directing private log report to 'dtacop-root-368731-1'. +op_IRS[1]: DataConnector Producer operator Instances: 1 +op_IRS[1]: ECI operator ID: 'op_IRS-368731' +op_IRS[1]: Operator instance 1 processing file 'index_2020.csv'. +ol_IRS: connecting sessions +ol_IRS: preparing target table +ol_IRS: entering Acquisition Phase +ol_IRS: entering Application Phase +ol_IRS: Statistics for Target Table: 'irs.irs_returns' +ol_IRS: Total Rows Sent To RDBMS: 333722 +ol_IRS: Total Rows Applied: 333722 +ol_IRS: Total Rows in Error Table 1: 0 +ol_IRS: Total Rows in Error Table 2: 0 +ol_IRS: Total Duplicate Rows: 0 +op_IRS[1]: Total files processed: 1. +ol_IRS: disconnecting sessions +Job step st_Load_File completed successfully +Job file_load completed successfully +ol_IRS: Performance metrics: +ol_IRS: MB/sec in Acquisition phase: 9.225 +ol_IRS: Elapsed time from start to Acquisition phase: 2 second(s) +ol_IRS: Elapsed time in Acquisition phase: 5 second(s) +ol_IRS: Elapsed time in Application phase: 3 second(s) +ol_IRS: Elapsed time from Application phase to end: < 1 second +ol_IRS: Total processor time used = '0.254337 Second(s)' +ol_IRS: Start : Thu Apr 7 20:56:32 2022 +ol_IRS: End : Thu Apr 7 20:56:42 2022 +Job start: Thu Apr 7 20:56:32 2022 +Job end: Thu Apr 7 20:56:42 2022+
TPT vs. NOSIn our case, the file is in an S3 bucket. That means, that we can use Native Object Storage (NOS) to ingest the data:
+-- create an S3-backed foreign table
+CREATE FOREIGN TABLE irs_returns_nos
+ USING ( LOCATION('/s3/s3.amazonaws.com/irs-form-990/index_2020.csv') );
+
+-- load the data into a native table
+CREATE MULTISET TABLE irs_returns_nos_native
+ (RETURN_ID, FILING_TYPE, EIN, TAX_PERIOD, SUB_DATE, TAXPAYER_NAME)
+AS (
+ SELECT RETURN_ID, FILING_TYPE, EIN, TAX_PERIOD, SUB_DATE, TAXPAYER_NAME FROM irs_returns_nos
+) WITH DATA
+NO PRIMARY INDEX;The NOS solution is convenient as it doesn’t depend on additional tools. It can be implemented using only SQL. It performs well, especially for Vantage deployments with a high number of AMPs as NOS tasks are delegated to AMPs and run in parallel. Also, splitting the data in object storage into multiple files may further improve performance.
+This how-to demonstrated how to ingest large amounts of data into Vantage. We loaded hundreds of thousands or records into Vantage in a couple of seconds using TPT.
| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +
Author: Adam Tworkiewicz
+Last updated: August 23rd, 2022
This how-to demonstrates how to run Vantage Express in Google Cloud Platform. Vantage Express contains a fully functional Teradata SQL Engine.
+| + + | ++If do not wish to pay for cloud usage you can install Vantage Express locally using VMware, VirtualBox, UTM. + | +
A Google Cloud account.
+gcloud command line utility installed on your machine. You can find installation instructions here: https://cloud.google.com/sdk/docs/install.
Create a Ubuntu VM with 4 CPU’s and 8GB of RAM, a 70GB balanced disk. The following command creates a VM in us-central1 region. For best performance, replace the region with one that is the closest to you. For the list of supported regions see Google Cloud regions documentation.
Run in Powershell:
+gcloud compute instances create teradata-vantage-express `
+ --zone=us-central1-a `
+ --machine-type=n2-custom-4-8192 `
+ --create-disk=boot=yes,device-name=ve-disk,image-project=ubuntu-os-cloud,image-family=ubuntu-2004-lts,size=70,type=pd-balanced `
+ --enable-nested-virtualization `
+ --tags=vegcloud compute instances create teradata-vantage-express \
+ --zone=us-central1-a \
+ --machine-type=n2-custom-4-8192 \
+ --create-disk=boot=yes,device-name=ve-disk,image-project=ubuntu-os-cloud,image-family=ubuntu-2004-lts,size=70,type=pd-balanced \
+ --enable-nested-virtualization \
+ --tags=vegcloud compute instances create teradata-vantage-express \
+ --zone=us-central1-a \
+ --machine-type=n2-custom-4-8192 \
+ --create-disk=boot=yes,device-name=ve-disk,image-project=ubuntu-os-cloud,image-family=ubuntu-2004-lts,size=70,type=pd-balanced \
+ --enable-nested-virtualization \
+ --tags=vessh to your VM:
+gcloud compute ssh teradata-vantage-express --zone=us-central1-aSwitch to root user:
sudo -iPrepare the download directory for Vantage Express:
+mkdir /opt/downloads
+cd /opt/downloadsInstall VirtualBox and 7zip:
+apt update && apt-get install p7zip-full p7zip-rar virtualbox -yRetrieve the curl command to download Vantage Express.
+Go to Vantage Expess download page (registration required).
+Click on the latest download link, e.g. "Vantage Express 17.20". You will see a license agreement popup. Don’t accept the license yet.
+Open the network view in your browser. For example, in Chrome press F12 and navigate to Network tab:
+Accept the license by clicking on I Agree button and cancel the download.
In the network view, find the last request that starts with VantageExpress. Right click on it and select Copy → Copy as cURL:
+Head back to the ssh session and download Vantage Express by pasting the curl command. Add -o ve.7z to the command to save the download to file named ve.7z. You can remove all the HTTP headers, e.g.:
curl -o ve.7z 'http://d289lrf5tw1zls.cloudfront.net/database/teradata-express/VantageExpress17.20_Sles12_202108300444.7z?Expires=1638719978&Signature=GKBkNvery_long_signature__&Key-Pair-Id=********************'Unzip the downloaded file. It will take several minutes:
+7z x ve.7zStart the VM in VirtualBox. The command will return immediately but the VM init process will take several minutes:
+export VM_IMAGE_DIR="/opt/downloads/VantageExpress17.20_Sles12"
+DEFAULT_VM_NAME="vantage-express"
+VM_NAME="${VM_NAME:-$DEFAULT_VM_NAME}"
+vboxmanage createvm --name "$VM_NAME" --register --ostype openSUSE_64
+vboxmanage modifyvm "$VM_NAME" --ioapic on --memory 6000 --vram 128 --nic1 nat --cpus 4
+vboxmanage storagectl "$VM_NAME" --name "SATA Controller" --add sata --controller IntelAhci
+vboxmanage storageattach "$VM_NAME" --storagectl "SATA Controller" --port 0 --device 0 --type hdd --medium "$(find $VM_IMAGE_DIR -name '*disk1*')"
+vboxmanage storageattach "$VM_NAME" --storagectl "SATA Controller" --port 1 --device 0 --type hdd --medium "$(find $VM_IMAGE_DIR -name '*disk2*')"
+vboxmanage storageattach "$VM_NAME" --storagectl "SATA Controller" --port 2 --device 0 --type hdd --medium "$(find $VM_IMAGE_DIR -name '*disk3*')"
+vboxmanage modifyvm "$VM_NAME" --natpf1 "tdssh,tcp,,4422,,22"
+vboxmanage modifyvm "$VM_NAME" --natpf1 "tddb,tcp,,1025,,1025"
+vboxmanage startvm "$VM_NAME" --type headless
+vboxmanage controlvm "$VM_NAME" keyboardputscancode 1c 1cssh to Vantage Express VM. Use root as password:
ssh -p 4422 root@localhostValidate that the DB is up:
+pdestate -aIf the command returns PDE state is RUN/STARTED. DBS state is 5: Logons are enabled - The system is quiescent, it means that Vantage Express has started.
+If the status is different, repeat pdestate -a till you get the correct status.
Once Vantage Express is up and running, start bteq client command line client. BTEQ (pronounced “bee-teek”) is a general-purpose, command-based client tool used to submit SQL queries to a Teradata Database.
bteqOnce in bteq, connect to your Vantage Express instance. When asked for the password, enter dbc:
.logon localhost/dbcUsing dbc user, we will create a new database called HR. Copy/paste this query and run press Enter:
CREATE DATABASE HR
+AS PERMANENT = 60e6, -- 60MB
+ SPOOL = 120e6; -- 120MBLet’s create a sample table and insert some data and query it. We will first create a table to hold employee information:
+CREATE SET TABLE HR.Employees (
+ GlobalID INTEGER,
+ FirstName VARCHAR(30),
+ LastName VARCHAR(30),
+ DateOfBirth DATE FORMAT 'YYYY-MM-DD',
+ JoinedDate DATE FORMAT 'YYYY-MM-DD',
+ DepartmentCode BYTEINT
+)
+UNIQUE PRIMARY INDEX ( GlobalID );Now, let’s insert a record:
+INSERT INTO HR.Employees (
+ GlobalID,
+ FirstName,
+ LastName,
+ DateOfBirth,
+ JoinedDate,
+ DepartmentCode
+)
+VALUES (
+ 101,
+ 'Adam',
+ 'Tworkowski',
+ '1980-01-05',
+ '2004-08-01',
+ 01
+);Finally, let’s see if we can retrieve the data:
+SELECT * FROM HR.Employees;You should get the following results:
+GlobalID FirstName LastName DateOfBirth JoinedDate DepartmentCode +-------- --------- ---------- ----------- ---------- -------------- + 101 Adam Tworkowski 1980-01-05 2004-08-01 1+
If you intend to stop and start the VM, you may want to add Vantage Express to autostart. ssh to your VM and run the following commands:
+sudo -i
+
+cat <<EOF >> /etc/default/virtualbox
+VBOXAUTOSTART_DB=/etc/vbox
+VBOXAUTOSTART_CONFIG=/etc/vbox/autostart.cfg
+EOF
+
+cat <<EOF > /etc/systemd/system/vantage-express.service
+[Unit]
+Description=vm1
+After=network.target virtualbox.service
+Before=runlevel2.target shutdown.target
+[Service]
+User=root
+Group=root
+Type=forking
+Restart=no
+TimeoutSec=5min
+IgnoreSIGPIPE=no
+KillMode=process
+GuessMainPID=no
+RemainAfterExit=yes
+ExecStart=/usr/bin/VBoxManage startvm vantage-express --type headless
+ExecStop=/usr/bin/VBoxManage controlvm vantage-express savestate
+[Install]
+WantedBy=multi-user.target
+EOF
+
+systemctl daemon-reload
+systemctl enable vantage-express
+systemctl start vantage-expressIf you would like to connect to Vantage Express from the Internet, you will need to open up firewall holes to your VM. You should also change the default password to dbc user:
To change the password for dbc user go to your VM and start bteq:
bteqLogin to your database using dbc as username and password:
.logon localhost/dbcChange the password for dbc user:
MODIFY USER dbc AS PASSWORD = new_password;You can now open up port 1025 to the internet using gcloud command:
+gcloud compute firewall-rules create vantage-express --allow=tcp:1025 --direction=IN --target-tags=veTo stop incurring charges, delete the VM:
+gcloud compute instances delete teradata-vantage-express --zone=us-central1-aAlso, remember to remove any firewall rules that you have added, e.g.:
+gcloud compute firewall-rules delete vantage-express| + + | ++If you have any questions or need further assistance, please visit our community forum where you can get support and interact with other community members. + | +