Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

debug/ developer tooling accomodations #157

Open
MACKAT05 opened this issue Feb 18, 2023 · 6 comments
Open

debug/ developer tooling accomodations #157

MACKAT05 opened this issue Feb 18, 2023 · 6 comments
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@MACKAT05
Copy link
Contributor

MACKAT05 commented Feb 18, 2023

Developer tooling.
I wrote a simple PowerShell script to run the render command over all the files and redirect them into a subfolder that is listed in the .gitignore
Can we add Block quotes to the outputs like so?:
image

This would allow Vs code users to execute and debug the rendered code interactively as captured from the output with the new snowflake extension.

@MACKAT05
Copy link
Contributor Author

MACKAT05 commented Feb 18, 2023

Optionally this loop over command could be incorporated (and refined to match the deployment gathering Regular expressions) into an expanded render command syntax... as per #156

@sfc-gh-tmathew
Copy link
Collaborator

Interesting @MACKAT05. Using the Render to verify the sql that will be executed appears to be useful application of the render functionality. It is as if we need a --dry-run option for the render command without incrementing the version of the SQL files to during the development lifecycle.

How do you think this pattern will fare for CI/CD based deployment workflows in Development environment? Are you envisioning the iterative development is done locally, say on a clone and then pushed for actual deployment in Development and higher environments?

@sfc-gh-tmathew sfc-gh-tmathew self-assigned this Sep 26, 2023
@sfc-gh-tmathew sfc-gh-tmathew added enhancement New feature or request help wanted Extra attention is needed labels Sep 26, 2023
@sfc-gh-tmathew sfc-gh-tmathew removed their assignment Sep 26, 2023
@MACKAT05
Copy link
Contributor Author

MACKAT05 commented Feb 7, 2024

@sfc-gh-tmathew i started working on a fork to deal with this and other issues...

  • parallel execution for the jinja interpreter, full debug reporting
  • new macro from_csv --> create a jinja python table to iterate over for use in higher level pipeline macro development.
  • state detection of deployment --> eg what is deployed where among multiple targets
  • strict option --> fail deployment if a skipped v script is detected or a regression of an R script is detected.
  • rework snowflake deployment to use the underlying file iterator(https://github.com/snowflakedb/snowflake-connector-python/blob/ca87fd8412efebfbdeb34ff9fabe20b929da9f27/src/snowflake/connector/util_text.py#L35) so comments and empty statement can be ignored... refine activity detection so state normalization only occurs when needed. which kicks open the door for more improvements.

Getting to the point of this query
while I was working on it I split cli.py into individual class files fore each level of functionality. I needed to do this to help isolate the levels of responsibility.

I recall seeing somewhere that there was a technical reason for keeping it as one file... does that still need to happen?
or would a split of files like this ( with some naming oversight be a maintainable option)
image

@sfc-gh-tmathew
Copy link
Collaborator

This appears to be a major refactor. Appears to be a 4.0 release candidate. Will get back to you @MACKAT05 once I have a chance to sync up with @jeremiahhansen.

@zanebclark
Copy link

@MACKAT05 , in #275 I've split the cli.py file out into multiple files / directories to make it easier to write unit tests and such. I poked around your fork and couldn't find a branch with the folder structure above. Is that work in a local branch that you haven't committed? Do you think you could build your feature off of #275?

@MACKAT05
Copy link
Contributor Author

MACKAT05 commented Aug 7, 2024

yeah sorry about that it is still on my local... i'll try to get that visible and up later tonight.
edit added experimental branch and published the current state https://github.com/MACKAT05/schemachange/tree/experimental/schemachange

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

3 participants