feat: code reorganization

.github/CONTRIBUTING.md

@@ -67,14 +67,14 @@ Thanks for your interest in contributing code to schemachange!
     2. Activate your virtual environment. The following table is a replication
        of [this](https://docs.python.org/3/library/venv.html#how-venvs-work) table:
 
-       | Platform 	| Shell      	| Command                               	|
-                                   |----------	|------------	|---------------------------------------	|
-       | POSIX    	| bash/zsh   	| `$ source <venv>/bin/activate`        	|
-       | POSIX    	| fish       	| `$ source <venv>/bin/activate.fish`   	|
-       | POSIX    	| csh/tcsh   	| `$ source <venv>/bin/activate.csh`    	|
-       | POSIX    	| PowerShell 	| `$ <venv>/bin/Activate.ps1`           	|
-       | Windows  	| cmd.exe    	| `C:\> <venv>\Scripts\activate.bat`    	|
-       | Windows  	| PowerShell 	| `PS C:\> <venv>\Scripts\Activate.ps1` 	|
+       | Platform | Shell      | Command                               |
+                                   |----------|------------|---------------------------------------|
+       | POSIX    | bash/zsh   | `$ source <venv>/bin/activate`        |
+       | POSIX    | fish       | `$ source <venv>/bin/activate.fish`   |
+       | POSIX    | csh/tcsh   | `$ source <venv>/bin/activate.csh`    |
+       | POSIX    | PowerShell | `$ <venv>/bin/Activate.ps1`           |
+       | Windows  | cmd.exe    | `C:\> <venv>\Scripts\activate.bat`    |
+       | Windows  | PowerShell | `PS C:\> <venv>\Scripts\Activate.ps1` |
 
     3. With your virtual environment activated, upgrade pip
        ```bash
@@ -86,7 +86,6 @@ Thanks for your interest in contributing code to schemachange!
        pip install -e .[dev]
        ```
 
-
 3. Develop your contribution
     + Create a branch for the features you want to work on. Since the branch name will appear in the merge message, use
       a sensible name such as 'update-build-library-dependencies':
@@ -114,5 +113,5 @@ Thanks for your interest in contributing code to schemachange!
 
     + Go to GitHub. The new branch will show up with a
       green [Pull Request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests#initiating-the-pull-request)
-      button. Make sure the title and message are clear, concise and self explanatory. Then click the button to submit
+      button. Make sure the title and message are clear, concise and self-explanatory. Then click the button to submit
       it.

.pre-commit-config.yaml

@@ -31,12 +31,6 @@ repos:
         args: [ --fix ]
       # Run the formatter.
       - id: ruff-format
-  - repo: [email protected]:GitGuardian/ggshield.git
-    rev: v1.28.0
-    hooks:
-      - id: ggshield
-        language_version: python3
-        stages: [ commit ]
   - repo: https://github.com/PyCQA/flake8
     rev: '6.0.0'
     hooks:

demo/README.MD

@@ -1,37 +1,59 @@
 # Demo
 
-The contents of this Demo serves two audiences - Consumers and Contributors. For the Consumer, the demo provides a basis to see how schemachange works with the main feature set. For the contributor, who forks the repo and submits PRs to the codebase, this will serve as a basis to test the PR against your own snowflake account to ensure your code change does not break any existing functionality.
+The contents of this Demo serves two audiences - Consumers and Contributors. For the Consumer, the demo provides a basis
+to see how schemachange works with the main feature set. For the contributor, who forks the repo and submits PRs to the
+codebase, this will serve as a basis to test the PR against your own snowflake account to ensure your code change does
+not break any existing functionality.
 
-## Prequisite
+## Prerequisite
 
 - You will need your own snowflake Account to test the Demo - Both as a contributor and consumer.
-- You will need to review and run statements in the provision folder or setup your own database and schema.
-  - [initialize.sql](provision/initialize.sql): Contains the SQL variables to initialize your environment.
-  - [setup_schemachange_schema.sql](provision/setup_schemachange_schema.sql): Contains the SQL variables to track the individual demo scenarios in its own change history table.
+- You will need to review and run statements in the provision folder or set up your own database and schema.
+    - [initialize.sql](provision/initialize.sql): Contains the SQL variables to initialize your environment.
+    - [setup_schemachange_schema.sql](provision/setup_schemachange_schema.sql): Contains the SQL variables to track the
+      individual demo scenarios in its own change history table.
 
 ### Contributors
-As a contributor, you will have to set up schemachange demo database and schemachange schema (See Initialize and Setup scripts below). Along with that you will also set up the following Secrets in your forked repository so that the github actions can setup, test and teardown the temporary schema it creates to test the changes to your code in the master and dev branches respectively.
+
+As a contributor, you will have to set up schemachange demo database and schemachange schema (See Initialize and Setup
+scripts below). Along with that you will also set up the following Secrets in your forked repository so that the GitHub
+actions can set up, test and teardown the temporary schema it creates to test the changes to your code in the master and
+dev branches respectively.
+
 - SCHEMACHANGE_SNOWFLAKE_PASSWORD
 - SCHEMACHANGE_SNOWFLAKE_USER
 - SCHEMACHANGE_SNOWFLAKE_ACCOUNT
 
 ### Consumers
-- If you are consumer who is installing schemachange and wants to test-run the demo, then you will have to set the following environment variables.
+
+- If you are consumer who is installing schemachange and wants to test-run the demo, then you will have to set the
+  following environment variables.
     - SNOWFLAKE_ACCOUNT: This will be the account identifier for your snowflake account.
     - SNOWFLAKE_USER: This will be the user that will connect to you snowflake account.
     - SNOWFLAKE_PASSWORD: This is the password for the user (SNOWFLAKE_USER) that will connect to the snowflake account.
-    - SCENARIO_NAME: This will be demo folder you intend to experiment with. For starters, `basics_demo`, `citibike_demo` or `citibike_demo_jinja` are included with the repo that will set the root folder value in the respective schemachange-config.yml file.
-    - SNOWFLAKE_WAREHOUSE: This will be the warehouse you setup for the demo. Default setup is SCHEMACHANGE_DEMO_WH
+    - SCENARIO_NAME: This will be demo folder you intend to experiment with. For
+      starters, `basics_demo`, `citibike_demo` or `citibike_demo_jinja` are included with the repo that will set the
+      root folder value in the respective schemachange-config.yml file.
+    - SNOWFLAKE_WAREHOUSE: This will be the warehouse you set up for the demo. Default setup is SCHEMACHANGE_DEMO_WH
     - SNOWFLAKE_DATABASE Keyed to SCHEMACHANGE_DEMO
     - SNOWFLAKE_ROLE Keyed to SCHEMACHANGE_DEMO-DEPLOY
 
-The scripts in the `provision` folder can be used to setup up your demo database along with a schema in that database that will house the change tracking tables needed to setup and teardown the schemas used to test a working version of the demo DDL scripts.
+The scripts in the `provision` folder can be used to set up up your demo database along with a schema in that database
+that will house the change tracking tables needed to set up and teardown the schemas used to test a working version of
+the demo DDL scripts.
 
-- The [initialize](provision/initialize.sql) script setups the database, warehouse and account level access roles that will be used on the
-- The [setup](provision/setup_schemachange_schema.sql) script creates the `SCHEMACHANGE` schema in the database that you created in the initialize step.
+- The [initialize](provision/initialize.sql) script setups the database, warehouse and account level access roles that
+  will be used on the
+- The [setup](provision/setup_schemachange_schema.sql) script creates the `SCHEMACHANGE` schema in the database that you
+  created in the initialize step.
 
 # Setup
-The setup scripts are included to build the schema needed by the GitHub Actions Workflow to avoid conflict across jobs when tested in parallel. The Setup script will create a new schema to run the schemachange script for the corresponding scenario.
+
+The setup scripts are included to build the schema needed by the GitHub Actions Workflow to avoid conflict across jobs
+when tested in parallel. The Setup script will create a new schema to run the schemachange script for the corresponding
+scenario.
 
 # Teardown
-The Teardown scripts are the bookend pairing of the Setup script for each scenario so that when the build process is done using github actions, you will have a log of the testing done to ensure that schemachange is working as expected.
+
+The Teardown scripts are the bookend pairing of the Setup script for each scenario so that when the build process is
+done using GitHub actions, you will have a log of the testing done to ensure that schemachange is working as expected.

requirements.txt

@@ -1,3 +1,4 @@
 Jinja2~=3.0
 PyYAML~=6.0
 snowflake-connector-python>=2.8,<4.0
+structlog~=24.1.0

schemachange/JinjaEnvVar.py

@@ -0,0 +1,33 @@
+from __future__ import annotations
+
+import os
+
+import jinja2.ext
+
+
+class JinjaEnvVar(jinja2.ext.Extension):
+    """
+    Extends Jinja Templates with access to environmental variables
+    """
+
+    def __init__(self, environment: jinja2.Environment):
+        super().__init__(environment)
+
+        # add globals
+        environment.globals["env_var"] = JinjaEnvVar.env_var
+
+    @staticmethod
+    def env_var(env_var: str, default: str | None = None) -> str:
+        """
+        Returns the value of the environmental variable or the default.
+        """
+        result = default
+        if env_var in os.environ:
+            result = os.environ[env_var]
+
+        if result is None:
+            raise ValueError(
+                f"Could not find environmental variable {env_var} and no default value was provided"
+            )
+
+        return result

schemachange/JinjaTemplateProcessor.py

@@ -0,0 +1,57 @@
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+import jinja2
+import jinja2.ext
+import structlog
+from jinja2.loaders import BaseLoader
+
+from schemachange.JinjaEnvVar import JinjaEnvVar
+
+logger = structlog.getLogger(__name__)
+
+
+class JinjaTemplateProcessor:
+    _env_args = {
+        "undefined": jinja2.StrictUndefined,
+        "autoescape": False,
+        "extensions": [JinjaEnvVar],
+    }
+
+    def __init__(self, project_root: Path, modules_folder: Path = None):
+        loader: BaseLoader
+        if modules_folder:
+            loader = jinja2.ChoiceLoader(
+                [
+                    jinja2.FileSystemLoader(project_root),
+                    jinja2.PrefixLoader(
+                        {"modules": jinja2.FileSystemLoader(modules_folder)}
+                    ),
+                ]
+            )
+        else:
+            loader = jinja2.FileSystemLoader(project_root)
+        self.__environment = jinja2.Environment(loader=loader, **self._env_args)
+        self.__project_root = project_root
+
+    def list(self):
+        return self.__environment.list_templates()
+
+    def override_loader(self, loader: jinja2.BaseLoader):
+        # to make unit testing easier
+        self.__environment = jinja2.Environment(loader=loader, **self._env_args)
+
+    def render(self, script: str, variables: dict[str, Any] | None) -> str:
+        if not variables:
+            variables = {}
+        # jinja needs posix path
+        posix_path = Path(script).as_posix()
+        template = self.__environment.get_template(posix_path)
+        content = template.render(**variables).strip()
+        content = content[:-1] if content.endswith(";") else content
+        return content
+
+    def relpath(self, file_path: Path):
+        return file_path.relative_to(self.__project_root)

schemachange/__init__.py

@@ -0,0 +1,15 @@
+import logging
+
+import structlog
+
+structlog.configure(
+    wrapper_class=structlog.make_filtering_bound_logger(logging.INFO),
+    processors=[
+        structlog.contextvars.merge_contextvars,
+        structlog.processors.add_log_level,
+        structlog.processors.TimeStamper(fmt="iso"),
+        structlog.stdlib.PositionalArgumentsFormatter(),
+        structlog.processors.dict_tracebacks,
+        structlog.dev.ConsoleRenderer(),
+    ],
+)