v0.8.21.0

Release Highlights

• Provide BigQuery credentials via recipe config: Previously BigQuery credentials were provided via environment variable. Going forward they can be provided directly inside the Recipe config.

Full Changelog: v0.8.20.0...v0.8.21.0

v0.8.20.0

Release Highlights

• Disable expensive profilers by default
• Improve snowflake connector to support allow / deny patterns for lineage, usage
• cleanup deprecated datahub.integrations.airflow.* imports
• Remove unecessary isalpha check for data platform urns
• Enable dbt metadata files to be loaded from URI
• Mode retry wait logic to avoid hitting Mode API rate limit
• Fix version compatibility issue with Google Cloud Composer

Full Changelog: v0.8.19.1...v0.8.20.0

Acryl DataHub v0.8.19.1

Release Highlights

• fixes for Snowflake connector. Improves how usage stats are calculated + adds support for allow/deny patterns in snowflake usage & lineage.
• adds Metabase connector

Changelog

• datahub-project#3745 @hsheth2 test(ingest): fix pytest warning for class starting with Test
• datahub-project#3747 @jjoyce0510 refactor(auth): Remove base64 decoding of token service signing key
• datahub-project#3748 @varunbharill fix(ingest): snowflake honor allow/deny pattern for lineage and usage.
• datahub-project#3744 @gabe-lyons feat(ingest): snowflake-usage add knob for direct objects accessed vs base objects accessed
• datahub-project#3742 @jjoyce0510 fix(ingest): remove data platform isalpha check as it complains about s3
• datahub-project#3743 @varunbharill fix(ingest): changing datahub-graph to use underlying session connection.
• datahub-project#3741 @jjoyce0510 fix(CI): regenerate Docker Quickstart
• datahub-project#3740 @gabe-lyons Revert "feat(graph): Make Dgraph a proper Neo4j alternative (datahub-project#3578)"
• datahub-project#3733 @nsbala-tw fix(elastic): Fix for log4j CVE-2021-44228 vulnerability
• datahub-project#3672 @gabe-lyons ci(cypress): adding the foundation for cypress integration tests & some starter coverage for login, search & updates
• datahub-project#3728 @anshbansal feat(perf-test): changes for perf testing
• datahub-project#3731 @hsheth2 fix(ingest): count profiled tables separately in report
• datahub-project#3602 @jawadqu feat(ingestion) : Add Metabase Source Connector
• datahub-project#3732 @hsheth2 feat(ingest): cleanup deprecated datahub.integrations.airflow.* imports
• datahub-project#3715 @dexter-mh-lee feat(retention): Add retention to Local DB
• datahub-project#3578 @EnricoMi feat(graph): Make Dgraph a proper Neo4j alternative
• datahub-project#3655 @anshbansal feat(cli): allow to nuke without deleting data in quickstart
• datahub-project#3664 @MugdhaHardikar-GSLab feat(spark-lineage): add ability to push data lineage from spark to d…

Acryl DataHub v0.8.19.0

Release Highlights

• Add support for definitions in Mode connector
• Handle long queries in usage connector
• Fix issue where top users would sometimes have malformed email addresses

What's Changed

• datahub-project#3714 @grumbler fix(superset): handle dashboards without charts (datahub-project#3713)
• datahub-project#3729 @varunbharill feat(ingest): adding utilities methods to DataHubGraph class.
• datahub-project#3723 @dexter-mh-lee fix(docker): fix issues with base64 not working on some platforms
• datahub-project#3702 @lvicentesanchez fix(ingest): add source.config.connection.schema_registry_config to SchemaRegistryClient creation
• datahub-project#3706 @sgomezvillamor fix(ingestion): adds missing port to the connection bootstrap
• datahub-project#3725 @treff7es feat(ingest): trim long sql queries in usage connector
• datahub-project#3712 @mayurinehate fix(ingest): update trino source get_table_comment to handle not found error
• datahub-project#3726 @treff7es fix(ingest): get mysql geotypes properly
• datahub-project#3727 @treff7es fix(profiler): fix division by zero in pct_unique calculation
• datahub-project#3708 @treff7es docs(redshift): adding svv_table privilege requirement to redshift source
• datahub-project#3717 @varunbharill fix(docs): datahub cli docs have the wrong case for corpuser
• datahub-project#3721 @gabe-lyons fix(mode): support definitions in mode query
• datahub-project#3720 @gabe-lyons fix 0 artifact when there are no search recommendations

Acryl DataHub v0.8.18.1

DataHub Release 0.8.18 is here! See the full DataHub release notes here.

Release Highlights

1. Redshift Lineage: Out-of-the-box support for ingesting Dataset->Dataset lineage from Redshift system tables. Includes Tables, Views, and COPY from S3

   1. Video Demo

2. Apache Nifi Connector (Beta): Integration with Apache Nifi to extract DataJobs and DataFlows! Read the source docs here. This source is currently incubating in beta.

3. Mode Connector (Beta): Integration with Mode Analytics to extract reports, charts, and more! Read the source docs here. This source is currently incubating in beta.

4. Glossary Term Transformer: Allows users to add tags or glossary terms to entities based on a regex match filter (Shoutout to community Member ecooklin!)

5. Bug Fixes:

   1. [bigquery] Fix handling of partitioned & snapshotted tables for lineage usage, and basic table indexing.
   2. Add config to specify ca certificate path for datahub-rest sink
   3. [snowflake] Handling for special characters in snowflake databases and schemas.
   4. [cli] Fix deletes by search cannot accept auth token

What's Changed

• fix(bigquery-usage): Fix filters and event joining logic. by @varunbharill in datahub-project#3610
• feat(cli): adding a put command and docs by @swaroopjagadish in datahub-project#3614
• feat(profiler): dynamically combine queries by @hsheth2 in datahub-project#3572
• fix(ingest): rectify platform urn in kafka connect source by @mayurinehate in datahub-project#3624
• fix(ci): pin great expectations version by @swaroopjagadish in datahub-project#3629
• fix(delete): Remove logic that adds an invalid filter for platform field by @dexter-mh-lee in datahub-project#3619
• feat(cfg): Add configurable GCP log page size by @jjoyce0510 in datahub-project#3556
• fix(ingestion): Add config to specify ca certificate path for datahub-rest sink by @dexter-mh-lee in datahub-project#3632
• fix(ingest): workaround great-expectations compatibility issue by @hsheth2 in datahub-project#3634
• fix(ingestion): Handling for special characters in snowflake databases and schemas. by @rslanka in datahub-project#3635
• fix(ingestion): Enhance supported and unsupported base_objects_accessed for Snowflake Usage by @rslanka in datahub-project#3608
• fix(mongodb): enable version check for document size filter. by @varunbharill in datahub-project#3644
• fix(cli): fix for deletion cli by @anshbansal in datahub-project#3653
• feat(ingest): add --strict-warnings option by @hsheth2 in datahub-project#3665
• fix(ingestion): black formatting by @hsheth2 in datahub-project#3676
• feat(ingestion): anonymous usage stats by @kevinhu in datahub-project#3668
• feat(delete): support deleting by search w/ tokens by @gabe-lyons in datahub-project#3684
• refactor(profiling): clean up SQL query analysis by @hsheth2 in datahub-project#3674
• fix(snowflake): support geo types by @gabe-lyons in datahub-project#3686
• feat(profiler): add query combiner report statistics by @hsheth2 in datahub-project#3678
• feat(transformer) Adds glossary terms transformer by @ecooklin in datahub-project#3657
• feat(ingest): add nifi source in metadata-ingestion by @mayurinehate in datahub-project#3681
• feat(bigquery): support snapshot and partition tables in bigquery ingest & lineage by @gabe-lyons in datahub-project#3695
• fix(ingest): refactor urn deletion by @kevinhu in datahub-project#3694
• feat(ingestion): Add lineage support for Redshift source by @gabe-lyons in datahub-project#3697
• fix(ingestion): Disable query parser failure reporting to Datahub in redshift lineage by default by @treff7es in datahub-project#3699
• fix(nifi): add env in nifi config, add unit tests, fix nifi doc by @mayurinehate in datahub-project#3703
• feat(mode): add mode analytics ingestion source by @gabe-lyons in datahub-project#3710

Acryl DataHub v0.8.17.7

Release Highlights

• Mode analytics ingestion source

Changelog

• datahub-project#3710 @gabe-lyons feat(mode): add mode analytics ingestion source
• datahub-project#3703 @mayurinehate fix(nifi): add env in nifi config, add unit tests, fix nifi doc
• datahub-project#3707 @treff7es docs(redshift): Adding requirements for redshift permissions
• datahub-project#3687 @anshbansal docs(airflow): add some troubleshooting for error

Acryl DataHub v0.8.17.6

Release Highlights

• Add support for lineage in Redshift source
• (new) Apache Nifi source
• Fixes for BigQuery source related to partitioned tables

ChangeLog

• datahub-project#3699 @treff7es fix(ingest): disable query parser failure reporting to datahub in redshift lineage by default
• datahub-project#3697 @gabe-lyons feat(ingestion): Add lineage support for Redshift source
• datahub-project#3698 @gabe-lyons fix(ingest): revert accidental change to example recipe file_to_datahub_rest.yml
• datahub-project#3689 @anshbansal fix(docker): fix for local docker compose for M1
• datahub-project#3694 @kevinhu refactor(ingest): cli deletion function
• datahub-project#3695 @gabe-lyons feat(ingest): bigquery - support snapshot and partition tables during ingest & lineage
• datahub-project#3681 @mayurinehate feat(ingest): add nifi source
• datahub-project#3693 @jjoyce0510 fix(deletes): Fixing system metadata index deletes

Full Changelog: v0.8.17.5...v0.8.17.6

Acryl DataHub v0.8.17.5

Highlights

• Better reporting of performance of data profiling
• Adds glossary terms transformer
• Fixes for snowflake
• Fixes for delete cli using tokens
• cli usage stats

ChangeLog

• datahub-project#3657 @ecooklin feat(ingest): adds glossary terms transformer
• datahub-project#3678 @hsheth2 feat(profiler): add query combiner report statistics
• datahub-project#3686 @gabe-lyons fix(snowflake): support geo types
• datahub-project#3596 @dexter-mh-lee fix(elasticsearch): Use auth tokens to authorize curl requests in dockerize
• datahub-project#3683 @jjoyce0510 fix(auth): Tag Create Privileges were incorrectly using Dataset Create privileges
• datahub-project#3674 @hsheth2 refactor(profiling): clean up SQL query analysis
• datahub-project#3679 @anshbansal docs(business glossary): fix specification of the file
• datahub-project#3685 @jeffmerrick docs: change roadmap link in docs
• datahub-project#3684 @gabe-lyons feat(delete): support deleting by search w/ tokens
• datahub-project#3669 @anshbansal docs(scheduling): re-arrange docs related to scheduling, lineage, CLI
• datahub-project#3668 @kevinhu feat(ingestion): anonymous usage stats
• datahub-project#3675 @dexter-mh-lee fix(kafka): Add back changes to centralize kafka config
• datahub-project#3677 @gabe-lyons fix(build): correcting m1 preflight check
• datahub-project#3676 @hsheth2 fix(ingestion): black formatting
• datahub-project#3667 @swaroopjagadish feat(metadata-model): adding metadata model doc generation and upload…
• datahub-project#3660 @dexter-mh-lee fix(search): Improve search and recs performance
• datahub-project#3665 @hsheth2 feat(ingest): add --strict-warnings option
• datahub-project#3662 @anshbansal fix(docs): fix build on m1
• datahub-project#3659 @dexter-mh-lee refactor: remove unused PDL files
• datahub-project#3654 @pedro93 fix(test): Adds docker engine configuration checks before running docker-based tests for metadata-io

Full Changelog: v0.8.17.4...v0.8.17.5

Acryl DataHub v0.8.17.4

Highlights

• Fixes mongodb compatibility with versions <4.4
• Fixes snowflake lineage for accounts with and without access history
• Fixes bug in delete command in cli

ChangeLog

• datahub-project#3653 @anshbansal fix(cli): fix response handling for deletion cli
• datahub-project#3650 @bartlomiejolma docs: correct title of postgres gms
• datahub-project#3649 @jjoyce0510 docs(graphql): adding additional GraphQL docs
• datahub-project#3652 @treff7es feat(build): Preflight script for metadata ingestion on m1
• datahub-project#3643 @gabe-lyons fix(ui): fix bug where key/value toggle would appear on schema tabs with no fields
• datahub-project#3646 @gabe-lyons fix(ui): fix issue where markdown links are unclickable
• datahub-project#3647 @swaroopjagadish feat(build): add incremental builds for python
• datahub-project#3648 @maggiehays docs: update to DataHub Adopter logos & Townhall details
• datahub-project#3644 @varunbharill fix(ingest): mongodb enable version check for document size filter.
• datahub-project#3640 @swaroopjagadish fix(docs): small fixes to docs and docker images for custom metadata …
• datahub-project#3434 @EnricoMi feat(quickstart): simplify docker generate and compare script
• datahub-project#3608 @rslanka fix(ingest): enhance supported and unsupported base_objects_accessed for snowflake
• datahub-project#3639 @gabe-lyons docs(business glossary): document the business glossary file format
• datahub-project#3597 @gabe-lyons feat(autorender): Auto render aspects that don't have frontend components in the UI

Acryl DataHub v0.8.17.3

Highlights

• Adds support for CA certificates for rest sink
• Fixes snowflake ingestion with special characters in databases and schemas
• Works around issues with latest great-expectations versions

Changelog

• datahub-project#3638 @jjoyce0510 fix(ui): fixing ownership bug
• datahub-project#3635 @rslanka fix(ingestion): Handling for special characters in snowflake databases and schemas.
• datahub-project#3634 @hsheth2 fix(ingest): workaround great-expectations compatibility issue
• datahub-project#3632 @dexter-mh-lee feat(ingest): add ca_certificate_path to rest sink

Full Changelog: v0.8.17.2...v0.8.17.3