From d152009c15490e57dde11ba292c25f59b4cc9f63 Mon Sep 17 00:00:00 2001 From: Talla Date: Mon, 16 Oct 2023 10:32:06 +0530 Subject: [PATCH 01/15] updated readme --- .../connect-data-platform/teradata-setup.md | 425 +++++++++++++++++- 1 file changed, 422 insertions(+), 3 deletions(-) diff --git a/website/docs/docs/core/connect-data-platform/teradata-setup.md b/website/docs/docs/core/connect-data-platform/teradata-setup.md index 1fe33ff8929..8d29e3747e0 100644 --- a/website/docs/docs/core/connect-data-platform/teradata-setup.md +++ b/website/docs/docs/core/connect-data-platform/teradata-setup.md @@ -41,6 +41,28 @@ pip is the easiest way to install the adapter:

Installing {frontMatter.meta.pypi_package} will also install dbt-core and any other dependencies.

+

Python compatibility

+ +| Plugin version | Python 3.6 | Python 3.7 | Python 3.8 | Python 3.9 | Python 3.10 | Python 3.11 | +| -------------- | ----------- | ----------- | ----------- | ----------- | ----------- | ------------ | +| 0.19.0.x | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ +| 0.20.0.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ +| 0.21.1.x | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ +| 1.0.0.x | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ +|1.1.x.x | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ +|1.2.x.x | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ +|1.3.x.x | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ +|1.4.x.x | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ +|1.5.x | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ +|1.6.x | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ + +

Python compatibility

+ +| dbt-teradta | dbt-core | dbt-teradata-util | dbt-util | +|-------------|------------|-------------------|----------------| +| 1.2.x | 1.2.x | 0.1.0 | 0.9.x or below | + +

Configuring {frontMatter.meta.pypi_package}

For {frontMatter.meta.platform_name}-specifc configuration please refer to {frontMatter.meta.platform_name} Configuration

@@ -88,11 +110,15 @@ The plugin also supports the following optional connection parameters: Parameter | Default | Type | Description ----------------------- | ----------- | -------------- | --- `account` | | string | Specifies the database account. Equivalent to the Teradata JDBC Driver `ACCOUNT` connection parameter. +`browser` | | string | Specifies the command to open the browser for Browser Authentication, when logmech is BROWSER. Browser Authentication is supported for Windows and macOS. Equivalent to the Teradata JDBC Driver BROWSER connection parameter. +`browser_tab_timeout` | `"5"` | quoted integer | Specifies the number of seconds to wait before closing the browser tab after Browser Authentication is completed. The default is 5 seconds. The behavior is under the browser's control, and not all browsers support automatic closing of browser tabs. +`browser_timeout` | `"180"` | quoted integer | Specifies the number of seconds that the driver will wait for Browser Authentication to complete. The default is 180 seconds (3 minutes). `column_name` | `"false"` | quoted boolean | Controls the behavior of cursor `.description` sequence `name` items. Equivalent to the Teradata JDBC Driver `COLUMN_NAME` connection parameter. False specifies that a cursor `.description` sequence `name` item provides the AS-clause name if available, or the column name if available, or the column title. True specifies that a cursor `.description` sequence `name` item provides the column name if available, but has no effect when StatementInfo parcel support is unavailable. `connect_failure_ttl` | `"0"` | quoted integer | Specifies the time-to-live in seconds to remember the most recent connection failure for each IP address/port combination. The driver subsequently skips connection attempts to that IP address/port for the duration of the time-to-live. The default value of zero disables this feature. The recommended value is half the database restart time. Equivalent to the Teradata JDBC Driver `CONNECT_FAILURE_TTL` connection parameter. +`connect_timeout` | `"10000"` | quoted integer | Specifies the timeout in milliseconds for establishing a TCP socket connection. Specify 0 for no timeout. The default is 10 seconds (10000 milliseconds). `cop` | `"true"` | quoted boolean | Specifies whether COP Discovery is performed. Equivalent to the Teradata JDBC Driver `COP` connection parameter. `coplast` | `"false"` | quoted boolean | Specifies how COP Discovery determines the last COP hostname. Equivalent to the Teradata JDBC Driver `COPLAST` connection parameter. When `coplast` is `false` or omitted, or COP Discovery is turned off, then no DNS lookup occurs for the coplast hostname. When `coplast` is `true`, and COP Discovery is turned on, then a DNS lookup occurs for a coplast hostname. -`dbs_port` | `"1025"` | quoted integer | Specifies the database port number. Equivalent to the Teradata JDBC Driver `DBS_PORT` connection parameter. +`port` | `"1025"` | quoted integer | Specifies the database port number. Equivalent to the Teradata JDBC Driver `DBS_PORT` connection parameter. `encryptdata` | `"false"` | quoted boolean | Controls encryption of data exchanged between the driver and the database. Equivalent to the Teradata JDBC Driver `ENCRYPTDATA` connection parameter. `fake_result_sets` | `"false"` | quoted boolean | Controls whether a fake result set containing statement metadata precedes each real result set. `field_quote` | `"\""` | string | Specifies a single character string used to quote fields in a CSV file. @@ -101,12 +127,28 @@ Parameter | Default | Type | Description `https_port` | `"443"` | quoted integer | Specifies the database port number for HTTPS/TLS connections. Equivalent to the Teradata JDBC Driver `HTTPS_PORT` connection parameter. `lob_support` | `"true"` | quoted boolean | Controls LOB support. Equivalent to the Teradata JDBC Driver `LOB_SUPPORT` connection parameter. `log` | `"0"` | quoted integer | Controls debug logging. Somewhat equivalent to the Teradata JDBC Driver `LOG` connection parameter. This parameter's behavior is subject to change in the future. This parameter's value is currently defined as an integer in which the 1-bit governs function and method tracing, the 2-bit governs debug logging, the 4-bit governs transmit and receive message hex dumps, and the 8-bit governs timing. Compose the value by adding together 1, 2, 4, and/or 8. -`logdata` | | string | Specifies extra data for the chosen logon authentication method. Equivalent to the Teradata JDBC Driver `LOGDATA` connection parameter. +`logdata` | | string | Specifies extra data for the chosen logon authentication method. Equivalent to the Teradata JDBC Driver +`LOGDATA` connection parameter. +`logon_timeout` | `"0"` | quoted integer | Specifies the logon timeout in seconds. Zero means no timeout. + `logmech` | `"TD2"` | string | Specifies the logon authentication method. Equivalent to the Teradata JDBC Driver `LOGMECH` connection parameter. Possible values are `TD2` (the default), `JWT`, `LDAP`, `KRB5` for Kerberos, or `TDNEGO`. `max_message_body` | `"2097000"` | quoted integer | Specifies the maximum Response Message size in bytes. Equivalent to the Teradata JDBC Driver `MAX_MESSAGE_BODY` connection parameter. `partition` | `"DBC/SQL"` | string | Specifies the database partition. Equivalent to the Teradata JDBC Driver `PARTITION` connection parameter. +`request_timeout` | `"0"` | quoted integer | Specifies the timeout for executing each SQL request. Zero means no timeout. +`retries` | `0` | integer | Allows an adapter to automatically try again when the attempt to open a new connection on the database has a transient, infrequent error. This option can be set using the retries configuration. Default value is 0. The default wait period between connection attempts is one second. retry_timeout (seconds) option allows us to adjust this waiting period. +`runstartup` | "false" | quoted boolean | Controls whether the user's STARTUP SQL request is executed after logon. For more information, refer to User STARTUP SQL Request. Equivalent to the Teradata JDBC Driver RUNSTARTUP connection parameter. + +If retries is set to 3, the adapter will try to establish a new connection three times if an error occurs. +`sessions` | | quoted integer | Specifies the number of data transfer connections for FastLoad or FastExport. The default (recommended) lets the database choose the appropriate number of connections. Equivalent to the Teradata JDBC Driver SESSIONS connection parameter. + `sip_support` | `"true"` | quoted boolean | Controls whether StatementInfo parcel is used. Equivalent to the Teradata JDBC Driver `SIP_SUPPORT` connection parameter. -`sslca` | | string | Specifies the file name of a PEM file that contains Certificate Authority (CA) certificates for use with `sslmode` values `VERIFY-CA` or `VERIFY-FULL`. Equivalent to the Teradata JDBC Driver `SSLCA` connection parameter. +`sp_spl` | `"true"` | quoted boolean | Controls whether stored procedure source code is saved in the database when a SQL stored procedure is created. Equivalent to the Teradata JDBC Driver SP_SPL connection parameter. + +`sslca` | | string | Specifies the file name of a PEM file that contains Certificate Authority (CA) certificates for use with +`sslmode` values `VERIFY-CA` or `VERIFY-FULL`. Equivalent to the Teradata JDBC Driver `SSLCA` connection parameter. +`sslcrc` | `"ALLOW"` | string | Equivalent to the Teradata JDBC Driver SSLCRC connection parameter. Values are case-insensitive. +• ALLOW provides "soft fail" behavior such that communication failures are ignored during certificate revocation checking. +• REQUIRE mandates that certificate revocation checking must succeed. `sslcapath` | | string | Specifies a directory of PEM files that contain Certificate Authority (CA) certificates for use with `sslmode` values `VERIFY-CA` or `VERIFY-FULL`. Only files with an extension of `.pem` are used. Other files in the specified directory are not used. Equivalent to the Teradata JDBC Driver `SSLCAPATH` connection parameter. `sslcipher` | | string | Specifies the TLS cipher for HTTPS/TLS connections. Equivalent to the Teradata JDBC Driver `SSLCIPHER` connection parameter. `sslmode` | `"PREFER"` | string | Specifies the mode for connections to the database. Equivalent to the Teradata JDBC Driver `SSLMODE` connection parameter.
• `DISABLE` disables HTTPS/TLS connections and uses only non-TLS connections.
• `ALLOW` uses non-TLS connections unless the database requires HTTPS/TLS connections.
• `PREFER` uses HTTPS/TLS connections unless the database does not offer HTTPS/TLS connections.
• `REQUIRE` uses only HTTPS/TLS connections.
• `VERIFY-CA` uses only HTTPS/TLS connections and verifies that the server certificate is valid and trusted.
• `VERIFY-FULL` uses only HTTPS/TLS connections, verifies that the server certificate is valid and trusted, and verifies that the server certificate matches the database hostname. @@ -124,6 +166,383 @@ For the full description of the connection parameters see https://github.com/Ter * `ephemeral` * `incremental` +#### Incremental Materialization +The following incremental materialization strategies are supported: +* `append` (default) +* `delete+insert` +* `merge` + +To learn more about dbt incremental strategies please check [the dbt incremental strategy documentation](https://docs.getdbt.com/docs/build/incremental-models#about-incremental_strategy). + ### Commands All dbt commands are supported. + +### Custom configurations + +#### General + +* *Enable view column types in docs* - Teradata Vantage has a dbscontrol configuration flag called `DisableQVCI` (QVCI - Queryable View Column Index). This flag instructs the database to build `DBC.ColumnsJQV` with view column type definitions. + + > :information_source: Existing customers, please see [KB0022230](https://support.teradata.com/knowledge?id=kb_article_view&sys_kb_id=d066248b1b0000187361c8415b4bcb48) for more information about enabling QVCI. + + To enable this functionality you need to: + 1. Enable QVCI mode in Vantage. Use `dbscontrol` utility and then restart Teradata. Run these commands as a privileged user on a Teradata node: + ```bash + # option 551 is DisableQVCI. Setting it to false enables QVCI. + dbscontrol << EOF + M internal 551=false + W + EOF + + # restart Teradata + tpareset -y Enable QVCI + ``` + 2. Instruct `dbt` to use `QVCI` mode. Include the following variable in your `dbt_project.yml`: + ```yaml + vars: + use_qvci: true + ``` + For example configuration, see `test/catalog/with_qvci/dbt_project.yml`. + +#### Models + +##### Table + +The following options apply to table, snapshots and seed materializations. + +* `table_kind` - define the table kind. Legal values are `MULTISET` (default for ANSI transaction mode required by `dbt-teradata`) and `SET`, e.g.: + * in sql materialization definition file: + ```yaml + {{ + config( + materialized="table", + table_kind="SET" + ) + }} + ``` + * in seed configuration: + ```yaml + seeds: + : + table_kind: "SET" + ``` + For details, see [CREATE TABLE documentation](https://docs.teradata.com/r/76g1CuvvQlYBjb2WPIuk3g/B6Js16DRQVwPDjgJ8rz7hg). +* `table_option` - define table options. Legal values are: + ```ebnf + { MAP = map_name [COLOCATE USING colocation_name] | + [NO] FALLBACK [PROTECTION] | + WITH JOURNAL TABLE = table_specification | + [NO] LOG | + [ NO | DUAL ] [BEFORE] JOURNAL | + [ NO | DUAL | LOCAL | NOT LOCAL ] AFTER JOURNAL | + CHECKSUM = { DEFAULT | ON | OFF } | + FREESPACE = integer [PERCENT] | + mergeblockratio | + datablocksize | + blockcompression | + isolated_loading + } + ``` + where: + * mergeblockratio: + ```ebnf + { DEFAULT MERGEBLOCKRATIO | + MERGEBLOCKRATIO = integer [PERCENT] | + NO MERGEBLOCKRATIO + } + ``` + * datablocksize: + ```ebnf + DATABLOCKSIZE = { + data_block_size [ BYTES | KBYTES | KILOBYTES ] | + { MINIMUM | MAXIMUM | DEFAULT } DATABLOCKSIZE + } + ``` + * blockcompression: + ```ebnf + BLOCKCOMPRESSION = { AUTOTEMP | MANUAL | ALWAYS | NEVER | DEFAULT } + [, BLOCKCOMPRESSIONALGORITHM = { ZLIB | ELZS_H | DEFAULT } ] + [, BLOCKCOMPRESSIONLEVEL = { value | DEFAULT } ] + ``` + * isolated_loading: + ```ebnf + WITH [NO] [CONCURRENT] ISOLATED LOADING [ FOR { ALL | INSERT | NONE } ] + ``` + + Examples: + * in sql materialization definition file: + ```yaml + {{ + config( + materialized="table", + table_option="NO FALLBACK" + ) + }} + ``` + ```yaml + {{ + config( + materialized="table", + table_option="NO FALLBACK, NO JOURNAL" + ) + }} + ``` + ```yaml + {{ + config( + materialized="table", + table_option="NO FALLBACK, NO JOURNAL, CHECKSUM = ON, + NO MERGEBLOCKRATIO, + WITH CONCURRENT ISOLATED LOADING FOR ALL" + ) + }} + ``` + * in seed configuration: + ```yaml + seeds: + : + table_option:"NO FALLBACK" + ``` + ```yaml + seeds: + : + table_option:"NO FALLBACK, NO JOURNAL" + ``` + ```yaml + seeds: + : + table_option: "NO FALLBACK, NO JOURNAL, CHECKSUM = ON, + NO MERGEBLOCKRATIO, + WITH CONCURRENT ISOLATED LOADING FOR ALL" + ``` + + For details, see [CREATE TABLE documentation](https://docs.teradata.com/r/76g1CuvvQlYBjb2WPIuk3g/B6Js16DRQVwPDjgJ8rz7hg). +* `with_statistics` - should statistics be copied from the base table, e.g.: + ```yaml + {{ + config( + materialized="table", + with_statistics="true" + ) + }} + ``` + This option is not available for seeds as seeds do not use `CREATE TABLE ... AS` syntax. + + For details, see [CREATE TABLE documentation](https://docs.teradata.com/r/76g1CuvvQlYBjb2WPIuk3g/B6Js16DRQVwPDjgJ8rz7hg). + +* `index` - defines table indices: + ```ebnf + [UNIQUE] PRIMARY INDEX [index_name] ( index_column_name [,...] ) | + NO PRIMARY INDEX | + PRIMARY AMP [INDEX] [index_name] ( index_column_name [,...] ) | + PARTITION BY { partitioning_level | ( partitioning_level [,...] ) } | + UNIQUE INDEX [ index_name ] [ ( index_column_name [,...] ) ] [loading] | + INDEX [index_name] [ALL] ( index_column_name [,...] ) [ordering] [loading] + [,...] + ``` + where: + * partitioning_level: + ```ebnf + { partitioning_expression | + COLUMN [ [NO] AUTO COMPRESS | + COLUMN [ [NO] AUTO COMPRESS ] [ ALL BUT ] column_partition ] + } [ ADD constant ] + ``` + * ordering: + ```ebnf + ORDER BY [ VALUES | HASH ] [ ( order_column_name ) ] + ``` + * loading: + ```ebnf + WITH [NO] LOAD IDENTITY + ``` + e.g.: + * in sql materialization definition file: + ```yaml + {{ + config( + materialized="table", + index="UNIQUE PRIMARY INDEX ( GlobalID )" + ) + }} + ``` + > :information_source: Note, unlike in `table_option`, there are no commas between index statements! + ```yaml + {{ + config( + materialized="table", + index="PRIMARY INDEX(id) + PARTITION BY RANGE_N(create_date + BETWEEN DATE '2020-01-01' + AND DATE '2021-01-01' + EACH INTERVAL '1' MONTH)" + ) + }} + ``` + ```yaml + {{ + config( + materialized="table", + index="PRIMARY INDEX(id) + PARTITION BY RANGE_N(create_date + BETWEEN DATE '2020-01-01' + AND DATE '2021-01-01' + EACH INTERVAL '1' MONTH) + INDEX index_attrA (attrA) WITH LOAD IDENTITY" + ) + }} + ``` + * in seed configuration: + ```yaml + seeds: + : + index: "UNIQUE PRIMARY INDEX ( GlobalID )" + ``` + > :information_source: Note, unlike in `table_option`, there are no commas between index statements! + ```yaml + seeds: + : + index: "PRIMARY INDEX(id) + PARTITION BY RANGE_N(create_date + BETWEEN DATE '2020-01-01' + AND DATE '2021-01-01' + EACH INTERVAL '1' MONTH)" + ``` + ```yaml + seeds: + : + index: "PRIMARY INDEX(id) + PARTITION BY RANGE_N(create_date + BETWEEN DATE '2020-01-01' + AND DATE '2021-01-01' + EACH INTERVAL '1' MONTH) + INDEX index_attrA (attrA) WITH LOAD IDENTITY" + ``` + +#### Seeds + +Seeds, in addition to the above materialization modifiers, have the following options: +* `use_fastload` - use [fastload](https://github.com/Teradata/python-driver#FastLoad) when handling `dbt seed` command. The option will likely speed up loading when your seed files have hundreds of thousands of rows. You can set this seed configuration option in your `project.yml` file, e.g.: + ```yaml + seeds: + : + +use_fastload: true + ``` + +#### Grants + +Grants are supported in dbt-teradata adapter with release version 1.2.0 and above. You can use grants to manage access to the datasets you're producing with dbt. To implement these permissions, define grants as resource configs on each model, seed, or snapshot. Define the default grants that apply to the entire project in your `dbt_project.yml`, and define model-specific grants within each model's SQL or YAML file. + +for e.g. : + models/schema.yml + ```yaml + models: + - name: model_name + config: + grants: + select: ['user_a', 'user_b'] + ``` + +Another e.g. for adding multiple grants: + + ```yaml + models: + - name: model_name + config: + materialized: table + grants: + select: ["user_b"] + insert: ["user_c"] + ``` +> :information_source: `copy_grants` is not supported in Teradata. + +More on Grants can be found at https://docs.getdbt.com/reference/resource-configs/grants + +### Cross DB macros +Starting with release 1.3, some macros were migrated from [teradata-dbt-utils](https://github.com/Teradata/dbt-teradata-utils) dbt package to the connector. See the table below for the macros supported from the connector. + +For using cross DB macros, teradata-utils as a macro namespace will not be used, as cross DB macros have been migrated from teradata-utils to Dbt-Teradata. + + +#### Compatibility + +| Macro Group | Macro Name | Status | Comment | +|:---------------------:|:-----------------------------:|:---------------------:|:----------------------------------------------------------------------:| +| Cross-database macros | current_timestamp | :white_check_mark: | custom macro provided | +| Cross-database macros | dateadd | :white_check_mark: | custom macro provided | +| Cross-database macros | datediff | :white_check_mark: | custom macro provided, see [compatibility note](#datediff) | +| Cross-database macros | split_part | :white_check_mark: | custom macro provided | +| Cross-database macros | date_trunc | :white_check_mark: | custom macro provided | +| Cross-database macros | hash | :white_check_mark: | custom macro provided, see [compatibility note](#hash) | +| Cross-database macros | replace | :white_check_mark: | custom macro provided | +| Cross-database macros | type_string | :white_check_mark: | custom macro provided | +| Cross-database macros | last_day | :white_check_mark: | no customization needed, see [compatibility note](#last_day) | +| Cross-database macros | width_bucket | :white_check_mark: | no customization + + +#### examples for cross DB macros +Replace: +{{ dbt.replace("string_text_column", "old_chars", "new_chars") }} +{{ replace('abcgef', 'g', 'd') }} + +Date truncate: +{{ dbt.date_trunc("date_part", "date") }} +{{ dbt.date_trunc("DD", "'2018-01-05 12:00:00'") }} + +#### datediff +`datediff` macro in teradata supports difference between dates. Differece between timestamps is not supported. + +#### hash + +`Hash` macro needs an `md5` function implementation. Teradata doesn't support `md5` natively. You need to install a User Defined Function (UDF): +1. Download the md5 UDF implementation from Teradata (registration required): https://downloads.teradata.com/download/extensibility/md5-message-digest-udf. +1. Unzip the package and go to `src` directory. +1. Start up `bteq` and connect to your database. +1. Create database `GLOBAL_FUNCTIONS` that will host the UDF. You can't change the database name as it's hardcoded in the macro: + ```sql + CREATE DATABASE GLOBAL_FUNCTIONS AS PERMANENT = 60e6, SPOOL = 120e6; + ``` +1. Create the UDF. Replace `` with your current database user: + ```sql + GRANT CREATE FUNCTION ON GLOBAL_FUNCTIONS TO ; + DATABASE GLOBAL_FUNCTIONS; + .run file = hash_md5.btq + ``` +1. Grant permissions to run the UDF with grant option. + ```sql + GRANT EXECUTE FUNCTION ON GLOBAL_FUNCTIONS TO PUBLIC WITH GRANT OPTION; + ``` +#### last_day + +`last_day` in `teradata_utils`, unlike the corresponding macro in `dbt_utils`, doesn't support `quarter` datepart. + +## Common Teradata-specific tasks +* *collect statistics* - when a table is created or modified significantly, there might be a need to tell Teradata to collect statistics for the optimizer. It can be done using `COLLECT STATISTICS` command. You can perform this step using dbt's `post-hooks`, e.g.: + ```yaml + {{ config( + post_hook=[ + "COLLECT STATISTICS ON {{ this }} COLUMN (column_1, column_2 ...);" + ] + )}} + ``` + See [Collecting Statistics documentation](https://docs.teradata.com/r/76g1CuvvQlYBjb2WPIuk3g/RAyUdGfvREwbO9J0DMNpLw) for more information. + +## Support for model contracts +Model constracts are not yet supported with dbt-teradata. + +## Support for `dbt-utils` package +`dbt-utils` package is supported through `teradata/teradata_utils` dbt package. The package provides a compatibility layer between `dbt_utils` and `dbt-teradata`. See [teradata_utils](https://hub.getdbt.com/teradata/teradata_utils/latest/) package for install instructions. + +## Limitations + +### Transaction mode +Only ANSI transaction mode is supported. + +## Credits + +The adapter was originally created by [Doug Beatty](https://github.com/dbeatty10). Teradata took over the adapter in January 2022. We are grateful to Doug for founding the project and accelerating the integration of dbt + Teradata. + +## License + +The adapter is published using Apache-2.0 License. Please see [the license](LICENSE) for terms and conditions, such as creating derivative work and the support model. From e2ccdb410195bb87c40ca53c142cb3adbf3aac02 Mon Sep 17 00:00:00 2001 From: Talla Date: Mon, 16 Oct 2023 10:41:00 +0530 Subject: [PATCH 02/15] Update teradata-setup.md --- .../docs/core/connect-data-platform/teradata-setup.md | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/website/docs/docs/core/connect-data-platform/teradata-setup.md b/website/docs/docs/core/connect-data-platform/teradata-setup.md index 8d29e3747e0..54486aa9001 100644 --- a/website/docs/docs/core/connect-data-platform/teradata-setup.md +++ b/website/docs/docs/core/connect-data-platform/teradata-setup.md @@ -127,23 +127,17 @@ Parameter | Default | Type | Description `https_port` | `"443"` | quoted integer | Specifies the database port number for HTTPS/TLS connections. Equivalent to the Teradata JDBC Driver `HTTPS_PORT` connection parameter. `lob_support` | `"true"` | quoted boolean | Controls LOB support. Equivalent to the Teradata JDBC Driver `LOB_SUPPORT` connection parameter. `log` | `"0"` | quoted integer | Controls debug logging. Somewhat equivalent to the Teradata JDBC Driver `LOG` connection parameter. This parameter's behavior is subject to change in the future. This parameter's value is currently defined as an integer in which the 1-bit governs function and method tracing, the 2-bit governs debug logging, the 4-bit governs transmit and receive message hex dumps, and the 8-bit governs timing. Compose the value by adding together 1, 2, 4, and/or 8. -`logdata` | | string | Specifies extra data for the chosen logon authentication method. Equivalent to the Teradata JDBC Driver -`LOGDATA` connection parameter. +`logdata` | | string | Specifies extra data for the chosen logon authentication method. Equivalent to the Teradata JDBC Driver `LOGDATA` connection parameter. `logon_timeout` | `"0"` | quoted integer | Specifies the logon timeout in seconds. Zero means no timeout. - `logmech` | `"TD2"` | string | Specifies the logon authentication method. Equivalent to the Teradata JDBC Driver `LOGMECH` connection parameter. Possible values are `TD2` (the default), `JWT`, `LDAP`, `KRB5` for Kerberos, or `TDNEGO`. `max_message_body` | `"2097000"` | quoted integer | Specifies the maximum Response Message size in bytes. Equivalent to the Teradata JDBC Driver `MAX_MESSAGE_BODY` connection parameter. `partition` | `"DBC/SQL"` | string | Specifies the database partition. Equivalent to the Teradata JDBC Driver `PARTITION` connection parameter. `request_timeout` | `"0"` | quoted integer | Specifies the timeout for executing each SQL request. Zero means no timeout. `retries` | `0` | integer | Allows an adapter to automatically try again when the attempt to open a new connection on the database has a transient, infrequent error. This option can be set using the retries configuration. Default value is 0. The default wait period between connection attempts is one second. retry_timeout (seconds) option allows us to adjust this waiting period. -`runstartup` | "false" | quoted boolean | Controls whether the user's STARTUP SQL request is executed after logon. For more information, refer to User STARTUP SQL Request. Equivalent to the Teradata JDBC Driver RUNSTARTUP connection parameter. - -If retries is set to 3, the adapter will try to establish a new connection three times if an error occurs. +`runstartup` | "false" | quoted boolean | Controls whether the user's STARTUP SQL request is executed after logon. For more information, refer to User STARTUP SQL Request. Equivalent to the Teradata JDBC Driver RUNSTARTUP connection parameter. If retries is set to 3, the adapter will try to establish a new connection three times if an error occurs. `sessions` | | quoted integer | Specifies the number of data transfer connections for FastLoad or FastExport. The default (recommended) lets the database choose the appropriate number of connections. Equivalent to the Teradata JDBC Driver SESSIONS connection parameter. - `sip_support` | `"true"` | quoted boolean | Controls whether StatementInfo parcel is used. Equivalent to the Teradata JDBC Driver `SIP_SUPPORT` connection parameter. `sp_spl` | `"true"` | quoted boolean | Controls whether stored procedure source code is saved in the database when a SQL stored procedure is created. Equivalent to the Teradata JDBC Driver SP_SPL connection parameter. - `sslca` | | string | Specifies the file name of a PEM file that contains Certificate Authority (CA) certificates for use with `sslmode` values `VERIFY-CA` or `VERIFY-FULL`. Equivalent to the Teradata JDBC Driver `SSLCA` connection parameter. `sslcrc` | `"ALLOW"` | string | Equivalent to the Teradata JDBC Driver SSLCRC connection parameter. Values are case-insensitive. From 1ee13b74ee0c8ef2e1a6e9d63f22d4ec4e145536 Mon Sep 17 00:00:00 2001 From: Talla Date: Mon, 16 Oct 2023 10:47:17 +0530 Subject: [PATCH 03/15] Update teradata-setup.md --- .../docs/docs/core/connect-data-platform/teradata-setup.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/website/docs/docs/core/connect-data-platform/teradata-setup.md b/website/docs/docs/core/connect-data-platform/teradata-setup.md index 54486aa9001..4dd65de0ef4 100644 --- a/website/docs/docs/core/connect-data-platform/teradata-setup.md +++ b/website/docs/docs/core/connect-data-platform/teradata-setup.md @@ -140,9 +140,7 @@ Parameter | Default | Type | Description `sp_spl` | `"true"` | quoted boolean | Controls whether stored procedure source code is saved in the database when a SQL stored procedure is created. Equivalent to the Teradata JDBC Driver SP_SPL connection parameter. `sslca` | | string | Specifies the file name of a PEM file that contains Certificate Authority (CA) certificates for use with `sslmode` values `VERIFY-CA` or `VERIFY-FULL`. Equivalent to the Teradata JDBC Driver `SSLCA` connection parameter. -`sslcrc` | `"ALLOW"` | string | Equivalent to the Teradata JDBC Driver SSLCRC connection parameter. Values are case-insensitive. -• ALLOW provides "soft fail" behavior such that communication failures are ignored during certificate revocation checking. -• REQUIRE mandates that certificate revocation checking must succeed. +`sslcrc` | `"ALLOW"` | string | Equivalent to the Teradata JDBC Driver SSLCRC connection parameter. Values are case-insensitive.
• ALLOW provides "soft fail" behavior such that communication failures are ignored during certificate revocation checking.
• REQUIRE mandates that certificate revocation checking must succeed. `sslcapath` | | string | Specifies a directory of PEM files that contain Certificate Authority (CA) certificates for use with `sslmode` values `VERIFY-CA` or `VERIFY-FULL`. Only files with an extension of `.pem` are used. Other files in the specified directory are not used. Equivalent to the Teradata JDBC Driver `SSLCAPATH` connection parameter. `sslcipher` | | string | Specifies the TLS cipher for HTTPS/TLS connections. Equivalent to the Teradata JDBC Driver `SSLCIPHER` connection parameter. `sslmode` | `"PREFER"` | string | Specifies the mode for connections to the database. Equivalent to the Teradata JDBC Driver `SSLMODE` connection parameter.
• `DISABLE` disables HTTPS/TLS connections and uses only non-TLS connections.
• `ALLOW` uses non-TLS connections unless the database requires HTTPS/TLS connections.
• `PREFER` uses HTTPS/TLS connections unless the database does not offer HTTPS/TLS connections.
• `REQUIRE` uses only HTTPS/TLS connections.
• `VERIFY-CA` uses only HTTPS/TLS connections and verifies that the server certificate is valid and trusted.
• `VERIFY-FULL` uses only HTTPS/TLS connections, verifies that the server certificate is valid and trusted, and verifies that the server certificate matches the database hostname. From 3d54ddaacd7ff192bbca126fb380def2f33ccf50 Mon Sep 17 00:00:00 2001 From: Talla Date: Mon, 16 Oct 2023 10:58:16 +0530 Subject: [PATCH 04/15] Update teradata-setup.md --- website/docs/docs/core/connect-data-platform/teradata-setup.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/website/docs/docs/core/connect-data-platform/teradata-setup.md b/website/docs/docs/core/connect-data-platform/teradata-setup.md index 4dd65de0ef4..f6cb8b08197 100644 --- a/website/docs/docs/core/connect-data-platform/teradata-setup.md +++ b/website/docs/docs/core/connect-data-platform/teradata-setup.md @@ -138,8 +138,7 @@ Parameter | Default | Type | Description `sessions` | | quoted integer | Specifies the number of data transfer connections for FastLoad or FastExport. The default (recommended) lets the database choose the appropriate number of connections. Equivalent to the Teradata JDBC Driver SESSIONS connection parameter. `sip_support` | `"true"` | quoted boolean | Controls whether StatementInfo parcel is used. Equivalent to the Teradata JDBC Driver `SIP_SUPPORT` connection parameter. `sp_spl` | `"true"` | quoted boolean | Controls whether stored procedure source code is saved in the database when a SQL stored procedure is created. Equivalent to the Teradata JDBC Driver SP_SPL connection parameter. -`sslca` | | string | Specifies the file name of a PEM file that contains Certificate Authority (CA) certificates for use with -`sslmode` values `VERIFY-CA` or `VERIFY-FULL`. Equivalent to the Teradata JDBC Driver `SSLCA` connection parameter. +`sslca` | | string | Specifies the file name of a PEM file that contains Certificate Authority (CA) certificates for use with `sslmode` values `VERIFY-CA` or `VERIFY-FULL`. Equivalent to the Teradata JDBC Driver `SSLCA` connection parameter. `sslcrc` | `"ALLOW"` | string | Equivalent to the Teradata JDBC Driver SSLCRC connection parameter. Values are case-insensitive.
• ALLOW provides "soft fail" behavior such that communication failures are ignored during certificate revocation checking.
• REQUIRE mandates that certificate revocation checking must succeed. `sslcapath` | | string | Specifies a directory of PEM files that contain Certificate Authority (CA) certificates for use with `sslmode` values `VERIFY-CA` or `VERIFY-FULL`. Only files with an extension of `.pem` are used. Other files in the specified directory are not used. Equivalent to the Teradata JDBC Driver `SSLCAPATH` connection parameter. `sslcipher` | | string | Specifies the TLS cipher for HTTPS/TLS connections. Equivalent to the Teradata JDBC Driver `SSLCIPHER` connection parameter. From 912dcf2e07810769980cd6569a6f957b185d4bed Mon Sep 17 00:00:00 2001 From: Talla Date: Mon, 23 Oct 2023 06:00:46 +0530 Subject: [PATCH 05/15] Updated setup and config docs --- .../connect-data-platform/teradata-setup.md | 302 +----------------- .../resource-configs/teradata-configs.md | 229 ++++++++----- 2 files changed, 155 insertions(+), 376 deletions(-) diff --git a/website/docs/docs/core/connect-data-platform/teradata-setup.md b/website/docs/docs/core/connect-data-platform/teradata-setup.md index f6cb8b08197..5de1846b6e8 100644 --- a/website/docs/docs/core/connect-data-platform/teradata-setup.md +++ b/website/docs/docs/core/connect-data-platform/teradata-setup.md @@ -4,7 +4,7 @@ description: "Read this guide to learn about the Teradata warehouse setup in dbt id: "teradata-setup" meta: maintained_by: Teradata - authors: Doug Beatty and Adam Tworkiewicz + authors: Teradata github_repo: 'Teradata/dbt-teradata' pypi_package: 'dbt-teradata' min_core_version: 'v0.21.0' @@ -169,286 +169,11 @@ To learn more about dbt incremental strategies please check [the dbt incremental All dbt commands are supported. -### Custom configurations - -#### General - -* *Enable view column types in docs* - Teradata Vantage has a dbscontrol configuration flag called `DisableQVCI` (QVCI - Queryable View Column Index). This flag instructs the database to build `DBC.ColumnsJQV` with view column type definitions. - - > :information_source: Existing customers, please see [KB0022230](https://support.teradata.com/knowledge?id=kb_article_view&sys_kb_id=d066248b1b0000187361c8415b4bcb48) for more information about enabling QVCI. - - To enable this functionality you need to: - 1. Enable QVCI mode in Vantage. Use `dbscontrol` utility and then restart Teradata. Run these commands as a privileged user on a Teradata node: - ```bash - # option 551 is DisableQVCI. Setting it to false enables QVCI. - dbscontrol << EOF - M internal 551=false - W - EOF - - # restart Teradata - tpareset -y Enable QVCI - ``` - 2. Instruct `dbt` to use `QVCI` mode. Include the following variable in your `dbt_project.yml`: - ```yaml - vars: - use_qvci: true - ``` - For example configuration, see `test/catalog/with_qvci/dbt_project.yml`. - -#### Models - -##### Table - -The following options apply to table, snapshots and seed materializations. - -* `table_kind` - define the table kind. Legal values are `MULTISET` (default for ANSI transaction mode required by `dbt-teradata`) and `SET`, e.g.: - * in sql materialization definition file: - ```yaml - {{ - config( - materialized="table", - table_kind="SET" - ) - }} - ``` - * in seed configuration: - ```yaml - seeds: - : - table_kind: "SET" - ``` - For details, see [CREATE TABLE documentation](https://docs.teradata.com/r/76g1CuvvQlYBjb2WPIuk3g/B6Js16DRQVwPDjgJ8rz7hg). -* `table_option` - define table options. Legal values are: - ```ebnf - { MAP = map_name [COLOCATE USING colocation_name] | - [NO] FALLBACK [PROTECTION] | - WITH JOURNAL TABLE = table_specification | - [NO] LOG | - [ NO | DUAL ] [BEFORE] JOURNAL | - [ NO | DUAL | LOCAL | NOT LOCAL ] AFTER JOURNAL | - CHECKSUM = { DEFAULT | ON | OFF } | - FREESPACE = integer [PERCENT] | - mergeblockratio | - datablocksize | - blockcompression | - isolated_loading - } - ``` - where: - * mergeblockratio: - ```ebnf - { DEFAULT MERGEBLOCKRATIO | - MERGEBLOCKRATIO = integer [PERCENT] | - NO MERGEBLOCKRATIO - } - ``` - * datablocksize: - ```ebnf - DATABLOCKSIZE = { - data_block_size [ BYTES | KBYTES | KILOBYTES ] | - { MINIMUM | MAXIMUM | DEFAULT } DATABLOCKSIZE - } - ``` - * blockcompression: - ```ebnf - BLOCKCOMPRESSION = { AUTOTEMP | MANUAL | ALWAYS | NEVER | DEFAULT } - [, BLOCKCOMPRESSIONALGORITHM = { ZLIB | ELZS_H | DEFAULT } ] - [, BLOCKCOMPRESSIONLEVEL = { value | DEFAULT } ] - ``` - * isolated_loading: - ```ebnf - WITH [NO] [CONCURRENT] ISOLATED LOADING [ FOR { ALL | INSERT | NONE } ] - ``` - - Examples: - * in sql materialization definition file: - ```yaml - {{ - config( - materialized="table", - table_option="NO FALLBACK" - ) - }} - ``` - ```yaml - {{ - config( - materialized="table", - table_option="NO FALLBACK, NO JOURNAL" - ) - }} - ``` - ```yaml - {{ - config( - materialized="table", - table_option="NO FALLBACK, NO JOURNAL, CHECKSUM = ON, - NO MERGEBLOCKRATIO, - WITH CONCURRENT ISOLATED LOADING FOR ALL" - ) - }} - ``` - * in seed configuration: - ```yaml - seeds: - : - table_option:"NO FALLBACK" - ``` - ```yaml - seeds: - : - table_option:"NO FALLBACK, NO JOURNAL" - ``` - ```yaml - seeds: - : - table_option: "NO FALLBACK, NO JOURNAL, CHECKSUM = ON, - NO MERGEBLOCKRATIO, - WITH CONCURRENT ISOLATED LOADING FOR ALL" - ``` - - For details, see [CREATE TABLE documentation](https://docs.teradata.com/r/76g1CuvvQlYBjb2WPIuk3g/B6Js16DRQVwPDjgJ8rz7hg). -* `with_statistics` - should statistics be copied from the base table, e.g.: - ```yaml - {{ - config( - materialized="table", - with_statistics="true" - ) - }} - ``` - This option is not available for seeds as seeds do not use `CREATE TABLE ... AS` syntax. - - For details, see [CREATE TABLE documentation](https://docs.teradata.com/r/76g1CuvvQlYBjb2WPIuk3g/B6Js16DRQVwPDjgJ8rz7hg). - -* `index` - defines table indices: - ```ebnf - [UNIQUE] PRIMARY INDEX [index_name] ( index_column_name [,...] ) | - NO PRIMARY INDEX | - PRIMARY AMP [INDEX] [index_name] ( index_column_name [,...] ) | - PARTITION BY { partitioning_level | ( partitioning_level [,...] ) } | - UNIQUE INDEX [ index_name ] [ ( index_column_name [,...] ) ] [loading] | - INDEX [index_name] [ALL] ( index_column_name [,...] ) [ordering] [loading] - [,...] - ``` - where: - * partitioning_level: - ```ebnf - { partitioning_expression | - COLUMN [ [NO] AUTO COMPRESS | - COLUMN [ [NO] AUTO COMPRESS ] [ ALL BUT ] column_partition ] - } [ ADD constant ] - ``` - * ordering: - ```ebnf - ORDER BY [ VALUES | HASH ] [ ( order_column_name ) ] - ``` - * loading: - ```ebnf - WITH [NO] LOAD IDENTITY - ``` - e.g.: - * in sql materialization definition file: - ```yaml - {{ - config( - materialized="table", - index="UNIQUE PRIMARY INDEX ( GlobalID )" - ) - }} - ``` - > :information_source: Note, unlike in `table_option`, there are no commas between index statements! - ```yaml - {{ - config( - materialized="table", - index="PRIMARY INDEX(id) - PARTITION BY RANGE_N(create_date - BETWEEN DATE '2020-01-01' - AND DATE '2021-01-01' - EACH INTERVAL '1' MONTH)" - ) - }} - ``` - ```yaml - {{ - config( - materialized="table", - index="PRIMARY INDEX(id) - PARTITION BY RANGE_N(create_date - BETWEEN DATE '2020-01-01' - AND DATE '2021-01-01' - EACH INTERVAL '1' MONTH) - INDEX index_attrA (attrA) WITH LOAD IDENTITY" - ) - }} - ``` - * in seed configuration: - ```yaml - seeds: - : - index: "UNIQUE PRIMARY INDEX ( GlobalID )" - ``` - > :information_source: Note, unlike in `table_option`, there are no commas between index statements! - ```yaml - seeds: - : - index: "PRIMARY INDEX(id) - PARTITION BY RANGE_N(create_date - BETWEEN DATE '2020-01-01' - AND DATE '2021-01-01' - EACH INTERVAL '1' MONTH)" - ``` - ```yaml - seeds: - : - index: "PRIMARY INDEX(id) - PARTITION BY RANGE_N(create_date - BETWEEN DATE '2020-01-01' - AND DATE '2021-01-01' - EACH INTERVAL '1' MONTH) - INDEX index_attrA (attrA) WITH LOAD IDENTITY" - ``` - -#### Seeds - -Seeds, in addition to the above materialization modifiers, have the following options: -* `use_fastload` - use [fastload](https://github.com/Teradata/python-driver#FastLoad) when handling `dbt seed` command. The option will likely speed up loading when your seed files have hundreds of thousands of rows. You can set this seed configuration option in your `project.yml` file, e.g.: - ```yaml - seeds: - : - +use_fastload: true - ``` - -#### Grants - -Grants are supported in dbt-teradata adapter with release version 1.2.0 and above. You can use grants to manage access to the datasets you're producing with dbt. To implement these permissions, define grants as resource configs on each model, seed, or snapshot. Define the default grants that apply to the entire project in your `dbt_project.yml`, and define model-specific grants within each model's SQL or YAML file. - -for e.g. : - models/schema.yml - ```yaml - models: - - name: model_name - config: - grants: - select: ['user_a', 'user_b'] - ``` - -Another e.g. for adding multiple grants: - - ```yaml - models: - - name: model_name - config: - materialized: table - grants: - select: ["user_b"] - insert: ["user_c"] - ``` -> :information_source: `copy_grants` is not supported in Teradata. +## Support for model contracts +Model constracts are not yet supported with dbt-teradata. -More on Grants can be found at https://docs.getdbt.com/reference/resource-configs/grants +## Support for `dbt-utils` package +`dbt-utils` package is supported through `teradata/teradata_utils` dbt package. The package provides a compatibility layer between `dbt_utils` and `dbt-teradata`. See [teradata_utils](https://hub.getdbt.com/teradata/teradata_utils/latest/) package for install instructions. ### Cross DB macros Starting with release 1.3, some macros were migrated from [teradata-dbt-utils](https://github.com/Teradata/dbt-teradata-utils) dbt package to the connector. See the table below for the macros supported from the connector. @@ -508,23 +233,6 @@ Date truncate: `last_day` in `teradata_utils`, unlike the corresponding macro in `dbt_utils`, doesn't support `quarter` datepart. -## Common Teradata-specific tasks -* *collect statistics* - when a table is created or modified significantly, there might be a need to tell Teradata to collect statistics for the optimizer. It can be done using `COLLECT STATISTICS` command. You can perform this step using dbt's `post-hooks`, e.g.: - ```yaml - {{ config( - post_hook=[ - "COLLECT STATISTICS ON {{ this }} COLUMN (column_1, column_2 ...);" - ] - )}} - ``` - See [Collecting Statistics documentation](https://docs.teradata.com/r/76g1CuvvQlYBjb2WPIuk3g/RAyUdGfvREwbO9J0DMNpLw) for more information. - -## Support for model contracts -Model constracts are not yet supported with dbt-teradata. - -## Support for `dbt-utils` package -`dbt-utils` package is supported through `teradata/teradata_utils` dbt package. The package provides a compatibility layer between `dbt_utils` and `dbt-teradata`. See [teradata_utils](https://hub.getdbt.com/teradata/teradata_utils/latest/) package for install instructions. - ## Limitations ### Transaction mode diff --git a/website/docs/reference/resource-configs/teradata-configs.md b/website/docs/reference/resource-configs/teradata-configs.md index f0f4f1a6f3e..12a8929429d 100644 --- a/website/docs/reference/resource-configs/teradata-configs.md +++ b/website/docs/reference/resource-configs/teradata-configs.md @@ -35,14 +35,21 @@ id: "teradata-configs" ### * `table_kind` - define the table kind. Legal values are `MULTISET` (default for ANSI transaction mode required by `dbt-teradata`) and `SET`, e.g.: - ```yaml - {{ - config( - materialized="table", - table_kind="SET" - ) - }} - ``` + * in sql materialization definition file: + ```yaml + {{ + config( + materialized="table", + table_kind="SET" + ) + }} + ``` + * in seed configuration: + ```yaml + seeds: + : + table_kind: "SET" + ``` For details, see [CREATE TABLE documentation](https://docs.teradata.com/r/76g1CuvvQlYBjb2WPIuk3g/B6Js16DRQVwPDjgJ8rz7hg). * `table_option` - defines table options. The config supports multiple statements. The definition below uses the Teradata syntax definition to explain what statements are allowed. Square brackets `[]` denote optional parameters. The pipe symbol `|` separates statements. Use commas to combine multiple statements as shown in the examples below: ``` @@ -87,37 +94,51 @@ id: "teradata-configs" ``` Examples: - - :::info Separators between statements - Note the commas that separate statements in `table_option` config. - ::: - - ```yaml - {{ - config( - materialized="table", - table_option="NO FALLBACK" - ) - }} - ``` - ```yaml - {{ - config( - materialized="table", - table_option="NO FALLBACK, NO JOURNAL" - ) - }} - ``` - ```yaml - {{ - config( - materialized="table", - table_option="NO FALLBACK, NO JOURNAL, CHECKSUM = ON, - NO MERGEBLOCKRATIO, - WITH CONCURRENT ISOLATED LOADING FOR ALL" - ) - }} - ``` + * in sql materialization definition file: + ```yaml + {{ + config( + materialized="table", + table_option="NO FALLBACK" + ) + }} + ``` + ```yaml + {{ + config( + materialized="table", + table_option="NO FALLBACK, NO JOURNAL" + ) + }} + ``` + ```yaml + {{ + config( + materialized="table", + table_option="NO FALLBACK, NO JOURNAL, CHECKSUM = ON, + NO MERGEBLOCKRATIO, + WITH CONCURRENT ISOLATED LOADING FOR ALL" + ) + }} + ``` + * in seed configuration: + ```yaml + seeds: + : + table_option:"NO FALLBACK" + ``` + ```yaml + seeds: + : + table_option:"NO FALLBACK, NO JOURNAL" + ``` + ```yaml + seeds: + : + table_option: "NO FALLBACK, NO JOURNAL, CHECKSUM = ON, + NO MERGEBLOCKRATIO, + WITH CONCURRENT ISOLATED LOADING FOR ALL" + ``` For details, see [CREATE TABLE documentation](https://docs.teradata.com/r/76g1CuvvQlYBjb2WPIuk3g/B6Js16DRQVwPDjgJ8rz7hg). @@ -160,46 +181,67 @@ id: "teradata-configs" ``` Examples: - - :::info Separators between statements - Note, unlike with `table_option` statements, there are no commas between statements in `index` config. - ::: - - ```yaml - {{ - config( - materialized="table", - index="UNIQUE PRIMARY INDEX ( GlobalID )" - ) - }} - ``` - - ```yaml - {{ - config( - materialized="table", - index="PRIMARY INDEX(id) - PARTITION BY RANGE_N(create_date - BETWEEN DATE '2020-01-01' - AND DATE '2021-01-01' - EACH INTERVAL '1' MONTH)" - ) - }} - ``` - - ```yaml - {{ - config( - materialized="table", - index="PRIMARY INDEX(id) - PARTITION BY RANGE_N(create_date - BETWEEN DATE '2020-01-01' - AND DATE '2021-01-01' - EACH INTERVAL '1' MONTH) - INDEX index_attrA (attrA) WITH LOAD IDENTITY" - ) - }} - ``` + * in sql materialization definition file: + ```yaml + {{ + config( + materialized="table", + index="UNIQUE PRIMARY INDEX ( GlobalID )" + ) + }} + ``` + > :information_source: Note, unlike in `table_option`, there are no commas between index statements! + ```yaml + {{ + config( + materialized="table", + index="PRIMARY INDEX(id) + PARTITION BY RANGE_N(create_date + BETWEEN DATE '2020-01-01' + AND DATE '2021-01-01' + EACH INTERVAL '1' MONTH)" + ) + }} + ``` + ```yaml + {{ + config( + materialized="table", + index="PRIMARY INDEX(id) + PARTITION BY RANGE_N(create_date + BETWEEN DATE '2020-01-01' + AND DATE '2021-01-01' + EACH INTERVAL '1' MONTH) + INDEX index_attrA (attrA) WITH LOAD IDENTITY" + ) + }} + ``` + * in seed configuration: + ```yaml + seeds: + : + index: "UNIQUE PRIMARY INDEX ( GlobalID )" + ``` + > :information_source: Note, unlike in `table_option`, there are no commas between index statements! + ```yaml + seeds: + : + index: "PRIMARY INDEX(id) + PARTITION BY RANGE_N(create_date + BETWEEN DATE '2020-01-01' + AND DATE '2021-01-01' + EACH INTERVAL '1' MONTH)" + ``` + ```yaml + seeds: + : + index: "PRIMARY INDEX(id) + PARTITION BY RANGE_N(create_date + BETWEEN DATE '2020-01-01' + AND DATE '2021-01-01' + EACH INTERVAL '1' MONTH) + INDEX index_attrA (attrA) WITH LOAD IDENTITY" + ``` ## Seeds :::info Using seeds to load raw data @@ -220,6 +262,35 @@ Loading CSVs using dbt's seed functionality is not performant for large files. C +use_fastload: true ``` +#### Grants + +Grants are supported in dbt-teradata adapter with release version 1.2.0 and above. You can use grants to manage access to the datasets you're producing with dbt. To implement these permissions, define grants as resource configs on each model, seed, or snapshot. Define the default grants that apply to the entire project in your `dbt_project.yml`, and define model-specific grants within each model's SQL or YAML file. + +for e.g. : + models/schema.yml + ```yaml + models: + - name: model_name + config: + grants: + select: ['user_a', 'user_b'] + ``` + +Another e.g. for adding multiple grants: + + ```yaml + models: + - name: model_name + config: + materialized: table + grants: + select: ["user_b"] + insert: ["user_c"] + ``` +> :information_source: `copy_grants` is not supported in Teradata. + +More on Grants can be found at https://docs.getdbt.com/reference/resource-configs/grants + ## Common Teradata-specific tasks * *collect statistics* - when a table is created or modified significantly, there might be a need to tell Teradata to collect statistics for the optimizer. It can be done using `COLLECT STATISTICS` command. You can perform this step using dbt's `post-hooks`, e.g.: From a2f9d82258da4ea3400cc07159c6621949280899 Mon Sep 17 00:00:00 2001 From: Talla Date: Mon, 23 Oct 2023 06:09:57 +0530 Subject: [PATCH 06/15] Update teradata-setup.md --- .../connect-data-platform/teradata-setup.md | 68 +++++++++---------- 1 file changed, 34 insertions(+), 34 deletions(-) diff --git a/website/docs/docs/core/connect-data-platform/teradata-setup.md b/website/docs/docs/core/connect-data-platform/teradata-setup.md index 5de1846b6e8..67cf847b64d 100644 --- a/website/docs/docs/core/connect-data-platform/teradata-setup.md +++ b/website/docs/docs/core/connect-data-platform/teradata-setup.md @@ -198,40 +198,40 @@ For using cross DB macros, teradata-utils as a macro namespace will not be used, #### examples for cross DB macros -Replace: -{{ dbt.replace("string_text_column", "old_chars", "new_chars") }} -{{ replace('abcgef', 'g', 'd') }} - -Date truncate: -{{ dbt.date_trunc("date_part", "date") }} -{{ dbt.date_trunc("DD", "'2018-01-05 12:00:00'") }} - -#### datediff -`datediff` macro in teradata supports difference between dates. Differece between timestamps is not supported. - -#### hash - -`Hash` macro needs an `md5` function implementation. Teradata doesn't support `md5` natively. You need to install a User Defined Function (UDF): -1. Download the md5 UDF implementation from Teradata (registration required): https://downloads.teradata.com/download/extensibility/md5-message-digest-udf. -1. Unzip the package and go to `src` directory. -1. Start up `bteq` and connect to your database. -1. Create database `GLOBAL_FUNCTIONS` that will host the UDF. You can't change the database name as it's hardcoded in the macro: - ```sql - CREATE DATABASE GLOBAL_FUNCTIONS AS PERMANENT = 60e6, SPOOL = 120e6; - ``` -1. Create the UDF. Replace `` with your current database user: - ```sql - GRANT CREATE FUNCTION ON GLOBAL_FUNCTIONS TO ; - DATABASE GLOBAL_FUNCTIONS; - .run file = hash_md5.btq - ``` -1. Grant permissions to run the UDF with grant option. - ```sql - GRANT EXECUTE FUNCTION ON GLOBAL_FUNCTIONS TO PUBLIC WITH GRANT OPTION; - ``` -#### last_day - -`last_day` in `teradata_utils`, unlike the corresponding macro in `dbt_utils`, doesn't support `quarter` datepart. + ##### replace + {{ dbt.replace("string_text_column", "old_chars", "new_chars") }} + {{ replace('abcgef', 'g', 'd') }} + + ##### date_trunc + {{ dbt.date_trunc("date_part", "date") }} + {{ dbt.date_trunc("DD", "'2018-01-05 12:00:00'") }} + + ##### datediff + `datediff` macro in teradata supports difference between dates. Differece between timestamps is not supported. + + ##### hash + + `Hash` macro needs an `md5` function implementation. Teradata doesn't support `md5` natively. You need to install a User Defined Function (UDF): + 1. Download the md5 UDF implementation from Teradata (registration required): https://downloads.teradata.com/download/extensibility/md5-message-digest-udf. + 1. Unzip the package and go to `src` directory. + 1. Start up `bteq` and connect to your database. + 1. Create database `GLOBAL_FUNCTIONS` that will host the UDF. You can't change the database name as it's hardcoded in the macro: + ```sql + CREATE DATABASE GLOBAL_FUNCTIONS AS PERMANENT = 60e6, SPOOL = 120e6; + ``` + 1. Create the UDF. Replace `` with your current database user: + ```sql + GRANT CREATE FUNCTION ON GLOBAL_FUNCTIONS TO ; + DATABASE GLOBAL_FUNCTIONS; + .run file = hash_md5.btq + ``` + 1. Grant permissions to run the UDF with grant option. + ```sql + GRANT EXECUTE FUNCTION ON GLOBAL_FUNCTIONS TO PUBLIC WITH GRANT OPTION; + ``` + ##### last_day + + `last_day` in `teradata_utils`, unlike the corresponding macro in `dbt_utils`, doesn't support `quarter` datepart. ## Limitations From a100fa1dd151c0cadb2a2775d12882553d7555ee Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Thu, 26 Oct 2023 11:02:53 +0100 Subject: [PATCH 07/15] add callout to explain the difference --- ...22-05-03-making-dbt-cloud-api-calls-using-dbt-cloud-cli.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/website/blog/2022-05-03-making-dbt-cloud-api-calls-using-dbt-cloud-cli.md b/website/blog/2022-05-03-making-dbt-cloud-api-calls-using-dbt-cloud-cli.md index 91ad1080ce6..b5f4049885f 100644 --- a/website/blog/2022-05-03-making-dbt-cloud-api-calls-using-dbt-cloud-cli.md +++ b/website/blog/2022-05-03-making-dbt-cloud-api-calls-using-dbt-cloud-cli.md @@ -12,6 +12,10 @@ date: 2022-05-03 is_featured: true --- +:::info Different from dbt Cloud CLI +This is blog explains how to use the Python library, `dbt-cloud-cli` and create a data catalog app with dbt Cloud artifacts. This is different from the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation), a dbt Cloud-powered tool that allows you to run dbt commands against your dbt Cloud development environment from your local command line. +::: + dbt Cloud is a hosted service that many organizations use for their dbt deployments. Among other things, it provides an interface for creating and managing deployment jobs. When triggered (e.g., cron schedule, API trigger), the jobs generate various artifacts that contain valuable metadata related to the dbt project and the run results. dbt Cloud provides a REST API for managing jobs, run artifacts and other dbt Cloud resources. Data/analytics engineers would often write custom scripts for issuing automated calls to the API using tools [cURL](https://curl.se/) or [Python Requests](https://requests.readthedocs.io/en/latest/). In some cases, the engineers would go on and copy/rewrite them between projects that need to interact with the API. Now, they have a bunch of scripts on their hands that they need to maintain and develop further if business requirements change. If only there was a dedicated tool for interacting with the dbt Cloud API that abstracts away the complexities of the API calls behind an easy-to-use interface… Oh wait, there is: [the dbt-cloud-cli](https://github.com/data-mie/dbt-cloud-cli)! From 4e1b523f0ac8656bedb40d5891a42c9d32948477 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Thu, 26 Oct 2023 14:07:09 +0100 Subject: [PATCH 08/15] adds command per slack feedback --- website/docs/docs/cloud/cloud-cli-installation.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/cloud-cli-installation.md b/website/docs/docs/cloud/cloud-cli-installation.md index 179af745685..20af8da314b 100644 --- a/website/docs/docs/cloud/cloud-cli-installation.md +++ b/website/docs/docs/cloud/cloud-cli-installation.md @@ -270,4 +270,7 @@ If you have dbt Core installed locally, either: You can always uninstall the dbt Cloud CLI to return to using dbt Core. - +
+Why am I receiving a Session occupied error? +If you've ran a dbt command and receive a Session occupied error, you can reattach to your existing session with dbt reattach and then press Control-C and choose to cancel the invocation. +
From ff863e67b324d9ab8cc42985b0cd56c8baf24e9e Mon Sep 17 00:00:00 2001 From: Talla Date: Mon, 30 Oct 2023 17:16:34 +0530 Subject: [PATCH 09/15] Update teradata-setup.md --- website/docs/docs/core/connect-data-platform/teradata-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/core/connect-data-platform/teradata-setup.md b/website/docs/docs/core/connect-data-platform/teradata-setup.md index 67cf847b64d..9f6c6b6f0b6 100644 --- a/website/docs/docs/core/connect-data-platform/teradata-setup.md +++ b/website/docs/docs/core/connect-data-platform/teradata-setup.md @@ -170,7 +170,7 @@ To learn more about dbt incremental strategies please check [the dbt incremental All dbt commands are supported. ## Support for model contracts -Model constracts are not yet supported with dbt-teradata. +Model contracts are not yet supported with dbt-teradata. ## Support for `dbt-utils` package `dbt-utils` package is supported through `teradata/teradata_utils` dbt package. The package provides a compatibility layer between `dbt_utils` and `dbt-teradata`. See [teradata_utils](https://hub.getdbt.com/teradata/teradata_utils/latest/) package for install instructions. From 49232ca832d4cc902eafb38de48e15fcce04aae7 Mon Sep 17 00:00:00 2001 From: mirnawong1 <89008547+mirnawong1@users.noreply.github.com> Date: Wed, 1 Nov 2023 13:34:18 +0000 Subject: [PATCH 10/15] Update website/docs/docs/core/connect-data-platform/teradata-setup.md --- website/docs/docs/core/connect-data-platform/teradata-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/core/connect-data-platform/teradata-setup.md b/website/docs/docs/core/connect-data-platform/teradata-setup.md index 9f6c6b6f0b6..bcda41c8fe9 100644 --- a/website/docs/docs/core/connect-data-platform/teradata-setup.md +++ b/website/docs/docs/core/connect-data-platform/teradata-setup.md @@ -244,4 +244,4 @@ The adapter was originally created by [Doug Beatty](https://github.com/dbeatty10 ## License -The adapter is published using Apache-2.0 License. Please see [the license](LICENSE) for terms and conditions, such as creating derivative work and the support model. +The adapter is published using Apache-2.0 License. Please see the link (ADD LINK HERE) for terms and conditions, such as creating derivative work and the support model. From 4a874a765ba30c0c00c099bab307966ee180961d Mon Sep 17 00:00:00 2001 From: mirnawong1 <89008547+mirnawong1@users.noreply.github.com> Date: Wed, 1 Nov 2023 14:42:33 +0000 Subject: [PATCH 11/15] Update website/blog/2022-05-03-making-dbt-cloud-api-calls-using-dbt-cloud-cli.md --- ...2022-05-03-making-dbt-cloud-api-calls-using-dbt-cloud-cli.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/blog/2022-05-03-making-dbt-cloud-api-calls-using-dbt-cloud-cli.md b/website/blog/2022-05-03-making-dbt-cloud-api-calls-using-dbt-cloud-cli.md index b5f4049885f..2ec00735161 100644 --- a/website/blog/2022-05-03-making-dbt-cloud-api-calls-using-dbt-cloud-cli.md +++ b/website/blog/2022-05-03-making-dbt-cloud-api-calls-using-dbt-cloud-cli.md @@ -13,7 +13,7 @@ is_featured: true --- :::info Different from dbt Cloud CLI -This is blog explains how to use the Python library, `dbt-cloud-cli` and create a data catalog app with dbt Cloud artifacts. This is different from the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation), a dbt Cloud-powered tool that allows you to run dbt commands against your dbt Cloud development environment from your local command line. +This is blog explains how to use the Python library, `dbt-cloud-cli`, and create a data catalog app with dbt Cloud artifacts. This is different from the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation), a dbt Cloud-powered tool that allows you to run dbt commands against your dbt Cloud development environment from your local command line. ::: dbt Cloud is a hosted service that many organizations use for their dbt deployments. Among other things, it provides an interface for creating and managing deployment jobs. When triggered (e.g., cron schedule, API trigger), the jobs generate various artifacts that contain valuable metadata related to the dbt project and the run results. From 97f5594a0d0b66d266d4a06a824f296b53187a4a Mon Sep 17 00:00:00 2001 From: mirnawong1 <89008547+mirnawong1@users.noreply.github.com> Date: Wed, 1 Nov 2023 14:44:04 +0000 Subject: [PATCH 12/15] Update website/blog/2022-05-03-making-dbt-cloud-api-calls-using-dbt-cloud-cli.md --- ...2022-05-03-making-dbt-cloud-api-calls-using-dbt-cloud-cli.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/blog/2022-05-03-making-dbt-cloud-api-calls-using-dbt-cloud-cli.md b/website/blog/2022-05-03-making-dbt-cloud-api-calls-using-dbt-cloud-cli.md index 2ec00735161..3bb04801579 100644 --- a/website/blog/2022-05-03-making-dbt-cloud-api-calls-using-dbt-cloud-cli.md +++ b/website/blog/2022-05-03-making-dbt-cloud-api-calls-using-dbt-cloud-cli.md @@ -13,7 +13,7 @@ is_featured: true --- :::info Different from dbt Cloud CLI -This is blog explains how to use the Python library, `dbt-cloud-cli`, and create a data catalog app with dbt Cloud artifacts. This is different from the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation), a dbt Cloud-powered tool that allows you to run dbt commands against your dbt Cloud development environment from your local command line. +This blog explains how to use the `dbt-cloud-cli` Python library to create a data catalog app with dbt Cloud artifacts. This is different from the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation), a tool that allows you to run dbt commands against your dbt Cloud development environment from your local command line ::: dbt Cloud is a hosted service that many organizations use for their dbt deployments. Among other things, it provides an interface for creating and managing deployment jobs. When triggered (e.g., cron schedule, API trigger), the jobs generate various artifacts that contain valuable metadata related to the dbt project and the run results. From 2de4815e79fe12118418924588aa999251362d09 Mon Sep 17 00:00:00 2001 From: mirnawong1 <89008547+mirnawong1@users.noreply.github.com> Date: Wed, 1 Nov 2023 14:44:16 +0000 Subject: [PATCH 13/15] Update website/blog/2022-05-03-making-dbt-cloud-api-calls-using-dbt-cloud-cli.md --- ...2022-05-03-making-dbt-cloud-api-calls-using-dbt-cloud-cli.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/blog/2022-05-03-making-dbt-cloud-api-calls-using-dbt-cloud-cli.md b/website/blog/2022-05-03-making-dbt-cloud-api-calls-using-dbt-cloud-cli.md index 3bb04801579..2ee774d4f1d 100644 --- a/website/blog/2022-05-03-making-dbt-cloud-api-calls-using-dbt-cloud-cli.md +++ b/website/blog/2022-05-03-making-dbt-cloud-api-calls-using-dbt-cloud-cli.md @@ -13,7 +13,7 @@ is_featured: true --- :::info Different from dbt Cloud CLI -This blog explains how to use the `dbt-cloud-cli` Python library to create a data catalog app with dbt Cloud artifacts. This is different from the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation), a tool that allows you to run dbt commands against your dbt Cloud development environment from your local command line +This blog explains how to use the `dbt-cloud-cli` Python library to create a data catalog app with dbt Cloud artifacts. This is different from the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation), a tool that allows you to run dbt commands against your dbt Cloud development environment from your local command line. ::: dbt Cloud is a hosted service that many organizations use for their dbt deployments. Among other things, it provides an interface for creating and managing deployment jobs. When triggered (e.g., cron schedule, API trigger), the jobs generate various artifacts that contain valuable metadata related to the dbt project and the run results. From c241eaf55a8fe4c1bc470f7da93215f068a17db8 Mon Sep 17 00:00:00 2001 From: mirnawong1 <89008547+mirnawong1@users.noreply.github.com> Date: Wed, 1 Nov 2023 16:09:26 +0000 Subject: [PATCH 14/15] Update dbt-commands.md update order --- website/docs/reference/dbt-commands.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/website/docs/reference/dbt-commands.md b/website/docs/reference/dbt-commands.md index 1448d9849d3..4bc3ddc24d7 100644 --- a/website/docs/reference/dbt-commands.md +++ b/website/docs/reference/dbt-commands.md @@ -28,6 +28,7 @@ You can run dbt commands in your specific tool by prefixing them with `dbt`. Fo | [deps](/reference/commands/deps) | Downloads dependencies for a project | All | All [supported versions](/docs/dbt-versions/core) | | [docs](/reference/commands/cmd-docs) | Generates documentation for a project | All | All [supported versions](/docs/dbt-versions/core) | | help | Displays help information for any command | dbt Core
dbt Cloud CLI | All [supported versions](/docs/dbt-versions/core) | +| [init](/reference/commands/init) | Initializes a new dbt project | dbt Core | All [supported versions](/docs/dbt-versions/core) | | [list](/reference/commands/list) | Lists resources defined in a dbt project | All | All [supported versions](/docs/dbt-versions/core) | | [parse](/reference/commands/parse) | Parses a project and writes detailed timing info | All | All [supported versions](/docs/dbt-versions/core) | | reattach | Reattaches to the most recent invocation to retrieve logs and artifacts. | dbt Cloud CLI | Requires [dbt v1.6 or higher](/docs/dbt-versions/core) | @@ -39,7 +40,7 @@ You can run dbt commands in your specific tool by prefixing them with `dbt`. Fo | [snapshot](/reference/commands/snapshot) | Executes "snapshot" jobs defined in a project | All | All [supported versions](/docs/dbt-versions/core) | | [source](/reference/commands/source) | Provides tools for working with source data (including validating that sources are "fresh") | All | All [supported versions](/docs/dbt-versions/core) | | [test](/reference/commands/test) | Executes tests defined in a project | All | All [supported versions](/docs/dbt-versions/core) | -| [init](/reference/commands/init) | Initializes a new dbt project | dbt Core | All [supported versions](/docs/dbt-versions/core) | + From f9e18c9d355348815e33b625917ef0f44b86ec78 Mon Sep 17 00:00:00 2001 From: mirnawong1 <89008547+mirnawong1@users.noreply.github.com> Date: Wed, 1 Nov 2023 17:21:31 +0000 Subject: [PATCH 15/15] Update teradata-setup.md add terms and conditions link --- website/docs/docs/core/connect-data-platform/teradata-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/core/connect-data-platform/teradata-setup.md b/website/docs/docs/core/connect-data-platform/teradata-setup.md index bcda41c8fe9..db544021fe7 100644 --- a/website/docs/docs/core/connect-data-platform/teradata-setup.md +++ b/website/docs/docs/core/connect-data-platform/teradata-setup.md @@ -244,4 +244,4 @@ The adapter was originally created by [Doug Beatty](https://github.com/dbeatty10 ## License -The adapter is published using Apache-2.0 License. Please see the link (ADD LINK HERE) for terms and conditions, such as creating derivative work and the support model. +The adapter is published using Apache-2.0 License. Refer to the [terms and conditions](https://github.com/dbt-labs/dbt-core/blob/main/License.md) to understand items such as creating derivative work and the support model.