Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(ingestion/snowflake):adds streams as a new dataset with lineage and properties. #12318

Open
wants to merge 40 commits into
base: master
Choose a base branch
from
Open

feat(ingestion/snowflake):adds streams as a new dataset with lineage and properties. #12318

wants to merge 40 commits into from
+3,783 −1,811

Conversation

brock-acryl
Copy link

@brock-acryl brock-acryl commented Jan 10, 2025 •
edited
Loading

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub
    Properly picks up:
  • Streams and columns in a stream (including metadata columns)
  • Stream upstream lineage from show streams statement
  • Stream downstream lineage by parsing queries (tested inserts, CTAS, inserts with unions)
  • Stream properties
Screenshot 2025-01-10 at 11 11 23 AM

@github-actions github-actions bot added ingestion PR or Issue related to the ingestion of metadata community-contribution PR or Issue raised by member(s) of DataHub Community labels Jan 10, 2025
@datahub-cyborg datahub-cyborg bot added the pending-submitter-response Issue/request has been reviewed but requires a response from the submitter label Jan 10, 2025
@datahub-cyborg datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Jan 11, 2025
@brock-acryl brock-acryl reopened this Jan 13, 2025
@codecov Codecov
Copy link

codecov bot commented Jan 13, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 90.17341% with 17 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...ingestion/source/snowflake/snowflake_schema_gen.py 90.72% 9 Missing ⚠️
...ub/ingestion/source/snowflake/snowflake_queries.py 0.00% 5 Missing ⚠️
...hub/ingestion/source/snowflake/snowflake_schema.py 95.83% 2 Missing ⚠️
...ahub/ingestion/source/snowflake/snowflake_utils.py 80.00% 1 Missing ⚠️
Files with missing lines Coverage Δ
...on/src/datahub/ingestion/source/common/subtypes.py 100.00% <100.00%> (ø)
...rc/datahub/ingestion/source/snowflake/constants.py 100.00% <100.00%> (ø)
...hub/ingestion/source/snowflake/snowflake_config.py 98.01% <100.00%> (+0.02%) ⬆️
...ahub/ingestion/source/snowflake/snowflake_query.py 95.05% <100.00%> (+1.30%) ⬆️
...hub/ingestion/source/snowflake/snowflake_report.py 99.09% <100.00%> (+0.04%) ⬆️
...datahub/ingestion/source/snowflake/snowflake_v2.py 89.12% <100.00%> (+0.03%) ⬆️
...ahub/ingestion/source/snowflake/snowflake_utils.py 89.20% <80.00%> (-0.58%) ⬇️
...hub/ingestion/source/snowflake/snowflake_schema.py 88.92% <95.83%> (+1.14%) ⬆️
...ub/ingestion/source/snowflake/snowflake_queries.py 43.39% <0.00%> (-0.67%) ⬇️
...ingestion/source/snowflake/snowflake_schema_gen.py 86.11% <90.72%> (+0.75%) ⬆️

... and 4 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d8ac6cd...217c295. Read the comment docs.

mayurinehate
mayurinehate reviewed Jan 14, 2025
View reviewed changes
metadata-ingestion/src/datahub/ingestion/source/sql/sql_config.py Outdated Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_v2.py Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_schema_gen.py Outdated Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_v2.py Outdated Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_query.py Outdated Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_schema_gen.py Outdated Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_v2.py Outdated Show resolved Hide resolved
@datahub-cyborg datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Jan 14, 2025
@brock-acryl
- moved stream_pattern from sql_config.py to snowflake_config.py
b7dfa7c 
- moved stream lineage from snowflake_v2.py & snowflake_lineage_v2.py to snowflake_schema_gen.py
- updated snowflake_schema_gen.py to use snowflake_utils.py
@datahub-cyborg datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Jan 14, 2025
@brock-acryl
- removed unused method
e0b3c3b 
- changed from clone table logic to manually mapping columns since metadata columns were attempting to be mapped from the stream source
@datahub-cyborg datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Jan 24, 2025
mayurinehate
mayurinehate reviewed Jan 27, 2025
View reviewed changes
Copy link
Collaborator

@mayurinehate mayurinehate left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks good.
Only one request to use add_known_lineage_mapping as opposed to add_known_query_lineage.

@datahub-cyborg datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Jan 27, 2025
@datahub-cyborg datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Jan 28, 2025
mayurinehate
mayurinehate requested changes Jan 29, 2025
View reviewed changes
metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_schema_gen.py Outdated Show resolved Hide resolved
metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_schema_gen.py
"Failed to get columns for stream:", stream.name, exc=e
)

yield from self.gen_dataset_workunits(stream, schema_name, db_name)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you forget to remove this outer yield , as its moved inside ?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An additional yield was added to populate the columns so column lineage would populate correctly.

@datahub-cyborg datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Jan 29, 2025
@datahub-cyborg datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Jan 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gabe-lyons gabe-lyons gabe-lyons left review comments

@hsheth2 hsheth2 Awaiting requested review from hsheth2

@mayurinehate mayurinehate Awaiting requested review from mayurinehate

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
community-contribution PR or Issue raised by member(s) of DataHub Community ingestion PR or Issue related to the ingestion of metadata needs-review Label for PRs that need review from a maintainer.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@brock-acryl @gabe-lyons @hsheth2 @mayurinehate