generated from liquibase/liquibase-extension-example
-
Notifications
You must be signed in to change notification settings - Fork 7
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Dev - Foundational Testing Completion (#17)
- Foundational Test Harness is now passing - Updated build logic - Add Optimize, Analyze, and Vacuum custom Databricks change types. - Add Terraform to create a custom runtime cluster, main catalog, and lb_test_harness schema - Add CreateTableDatabricks change type to enable: LOCATION, USING DELTA Co-authored-by: Nathan Voxland <[email protected]>
- Loading branch information
1 parent
1d79a29
commit ea60e3e
Showing
43 changed files
with
1,106 additions
and
194 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| name: Liquibase Test Harness | ||
|
|
||
| on: | ||
| pull_request: | ||
| push: | ||
| branches: | ||
| - main | ||
|
|
||
| jobs: | ||
| liquibase-test-harness: | ||
| name: Liquibase Test Harness | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| checks: write | ||
|
|
||
| env: | ||
| TF_VAR_DBX_HOST: ${{ secrets.TH_DATABRICKS_WORKSPACE_HOST }} | ||
| TF_VAR_DBX_TOKEN: ${{ secrets.TH_DATABRICKS_WORKSPACE_TOKEN }} | ||
| TF_VAR_TEST_CATALOG: main | ||
| TF_VAR_TEST_SCHEMA: lb_test_harness | ||
| WORKSPACE_ID: ${{ secrets.TH_DATABRICKS_WORKSPACE_ID }} | ||
|
|
||
| strategy: | ||
| matrix: | ||
| liquibase-support-level: [Foundational] # Define the different test levels to run | ||
| fail-fast: false # Set fail-fast to false to run all test levels even if some of them fail | ||
|
|
||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v3 | ||
|
|
||
| - run: terraform init | ||
| working-directory: src/test/terraform | ||
|
|
||
| - run: terraform plan | ||
| working-directory: src/test/terraform | ||
|
|
||
| - run: terraform apply -auto-approve | ||
| working-directory: src/test/terraform | ||
|
|
||
| - name: Collect Databricks Config | ||
| working-directory: src/test/terraform | ||
| run: | | ||
| CLUSTER_ID=$(terraform output -raw cluster_url) | ||
| DATABRICKS_HOST=${TF_VAR_DBX_HOST#https://} | ||
| echo "DATABRICKS_URL=jdbc:databricks://$DATABRICKS_HOST:443/default;transportMode=http;ssl=1;httpPath=sql/protocolv1/o/$WORKSPACE_ID/$CLUSTER_ID;AuthMech=3;ConnCatalog=$TF_VAR_TEST_CATALOG;ConnSchema=$TF_VAR_TEST_SCHEMA;EnableArrow=0" >> "$GITHUB_ENV" | ||
| - name: Setup Temurin Java 17 | ||
| uses: actions/setup-java@v3 | ||
| with: | ||
| java-version: 17 | ||
| distribution: temurin | ||
| cache: 'maven' | ||
|
|
||
| - name: Build with Maven # Build the code with Maven (skip tests) | ||
| run: mvn -B -ntp -Dmaven.test.skip package | ||
|
|
||
| - name: Run ${{ matrix.liquibase-support-level }} Liquibase Test Harness # Run the Liquibase test harness at each test level | ||
| continue-on-error: true # Continue to run the action even if the previous steps fail | ||
| run: mvn -B -ntp -DdbPassword=${{env.TF_VAR_DBX_TOKEN}} -DdbUrl='${{env.DATABRICKS_URL}}' -Dtest=liquibase.ext.databricks.${{ matrix.liquibase-support-level }}ExtensionHarnessTestSuite test # Run the Liquibase test harness at each test level | ||
|
|
||
| - name: Test Reporter # Generate a test report using the Test Reporter action | ||
| uses: dorny/[email protected] | ||
| if: always() # Run the action even if the previous steps fail | ||
| with: | ||
| name: Liquibase Test Harness - ${{ matrix.liquibase-support-level }} Reports # Set the name of the test report | ||
| path: target/surefire-reports/TEST-*.xml # Set the path to the test report files | ||
| reporter: java-junit # Set the reporter to use | ||
| fail-on-error: false # Set fail-on-error to false to show report even if it has failed tests | ||
|
|
||
| - name: Stop test database | ||
| if: always() # Always destroy, even if the previous steps fail | ||
| working-directory: src/test/terraform | ||
| run: terraform destroy -auto-approve |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,15 +1,63 @@ | ||
| # liquibase-databricks | ||
|
|
||
|
|
||
| ## Current Summary | ||
| Base and Foundational Change types should be supported at this stage. Change types such as procedures, triggers, merge column, indexes are not supported. | ||
| Databricks specific change types that are added are listed below along with their completion status. | ||
|
|
||
|
|
||
| ## To Do: | ||
|
|
||
| 1. Add unit tests with liquibase test harness | ||
| 1. Add unit tests with liquibase test harness - Cody Davis - Done | ||
| 2. Pass Foundational Test Harness - Cody Davis - Done | ||
| 3. Pass Base Test Harness - Cody Davis - In Progress - ETA May 15, 2023 | ||
| 4. Pass Advanced Test Harness - Unassigned - Not Started | ||
|
|
||
| ## Change Types to Add: | ||
|
|
||
| 1. COPY INTO | ||
| 2. MERGE | ||
| 3. RESTORE VERSION AS OF | ||
| 4. ANALYZE TABLE | ||
| 5. SET TBL PROPERTIES | ||
| 4. ANALYZE TABLE - Code Complete - Cody Davis | ||
| 5. SET TBL PROPERTIES - In Progress - Cody Davis | ||
| 6. CLONE | ||
| 7. BLOOM FILTERS | ||
| 7. BLOOM FILTERS | ||
| 8. OPTIMIZE / ZORDER - Code Complete - No Test Yet - Cody Davis | ||
| 9. VACUUM - Code Complete - Cody Davis | ||
|
|
||
|
|
||
|
|
||
| ## How to use the Liquibase-Databricks Extension | ||
|
|
||
| ### Steps: | ||
|
|
||
| 1. Download and install liquibase from [here](https://docs.liquibase.com/start/install/home.html) | ||
|
|
||
| 2. Download the Databricks Driver from [here](https://www.databricks.com/spark/jdbc-drivers-download). | ||
| Then put this driver jar under the liquibase/lib directory. | ||
|
|
||
| 3. Build this project or retrieve the jar from the latest release. | ||
| Then put this extension jar under the liquibase/lib directory. | ||
|
|
||
| 4. Edit the connection parameters to your Databricks catlaog/database under the liquibase.properties file. The format will look like this: | ||
|
|
||
| ``` | ||
| url: jdbc:databricks://<workspace_url>:443/default;transportMode=http;ssl=1;httpPath=<http_path>;AuthMech=3;ConnCatalog=<catalog>;ConnSchema=<database>; | ||
| username: token | ||
| password: <dbx_token> | ||
| ``` | ||
|
|
||
| Where the following parameters are: | ||
| <li> <b>workspace_url</b>: The url of the host name you are connecting to</li> | ||
| <li> <b>dbx_token</b>: The token of your user or application service principal authorized for running any needed Liquibase operations.</li> | ||
| <li> <b>http_path</b>: This is the http_path of a Databricks SQL Warehouse or a Databricks Cluster (DBR). Either cluster type can be used. Best Results are with Serverless SQL Warehouses. </li> | ||
| <li> <b>catalog</b>: The catalog name you want to connect to (default is main). </li> | ||
| <li> <b>database</b>: The database / schema name you want to connect to. </li> | ||
|
|
||
|
|
||
|
|
||
| 5. Add changes and run your change logs like so: | ||
| ``` | ||
| liquibase --changeLogFile=changelog.sql update | ||
| ``` | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,3 @@ | ||
| driver: com.databricks.jdbc.Driver | ||
| url: jdbc:databricks://<host>:443/default;transportMode=http;ssl=1;AuthMech=3;httpPath=/sql/1.0/warehouses/<warehouseId>;ConnCatalog=main;ConnSchema=default | ||
| user: token | ||
| password: <user_token> | ||
| defaultCatalogName: main | ||
| defaultSchemaName: default | ||
| liquibaseCatalogName: liquibase | ||
| url: jdbc:databricks://<workspace_url>:443/default;transportMode=http;ssl=1;httpPath=sql/<warehouse_id>;AuthMech=3;ConnCatalog=main;ConnSchema=liquibase_harness_test_ds; | ||
| username: token | ||
| password: <dbx_token> |
Oops, something went wrong.