Merge pull request #103 from fivetran/release/v0.14.0

Release/v0.14.0

.buildkite/scripts/run_models.sh

@@ -21,4 +21,5 @@ dbt run --target "$db" --full-refresh
 dbt run --target "$db"
 dbt test --target "$db"
 dbt run --vars "{jira_using_priorities: false, jira_using_sprints: false, jira_using_components: false, jira_using_versions: false, jira_field_grain: 'field_name'}" --target "$db" --full-refresh
-dbt test --target "$db"
+dbt test --target "$db"
+dbt run-operation fivetran_utils.drop_schemas_automation --target "$db"

.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md

@@ -0,0 +1,51 @@
+## PR Overview
+**This PR will address the following Issue/Feature:**
+
+**This PR will result in the following new package version:**
+<!--- Please add details around your decision for breaking vs non-breaking version upgrade. If this is a breaking change, were backwards-compatible options explored? -->
+
+**Please detail what change(s) this PR introduces and any additional information that should be known during the review of this PR:**
+
+## PR Checklist
+### Basic Validation
+Please acknowledge that you have successfully performed the following commands locally:
+- [ ] dbt compile
+- [ ] dbt run –full-refresh
+- [ ] dbt run
+- [ ] dbt test
+- [ ] dbt run –vars (if applicable)
+
+Before marking this PR as "ready for review" the following have been applied:
+- [ ] The appropriate issue has been linked and tagged
+- [ ] You are assigned to the corresponding issue and this PR
+- [ ] BuildKite integration tests are passing
+
+### Detailed Validation
+Please acknowledge that the following validation checks have been performed prior to marking this PR as "ready for review":
+- [ ] You have validated these changes and assure this PR will address the respective Issue/Feature.
+- [ ] You are reasonably confident these changes will not impact any other components of this package or any dependent packages.
+- [ ] You have provided details below around the validation steps performed to gain confidence in these changes.
+<!--- Provide the steps you took to validate your changes below. -->
+
+### Standard Updates
+Please acknowledge that your PR contains the following standard updates:
+- Package versioning has been appropriately indexed in the following locations:
+    - [ ] indexed within dbt_project.yml
+    - [ ] indexed within integration_tests/dbt_project.yml
+- [ ] CHANGELOG has individual entries for each respective change in this PR
+    <!--- If there is a parallel upstream change, remember to reference the corresponding CHANGELOG as an individual entry.  -->
+- [ ] README updates have been applied (if applicable)
+    <!--- Remember to check the following README locations for common updates. →
+        <!--- Suggested install range (needed for breaking changes) →
+        <!--- Dependency matrix is appropriately updated (if applicable) →
+        <!--- New variable documentation (if applicable) -->
+- [ ] DECISIONLOG updates have been updated (if applicable)
+- [ ] Appropriate yml documentation has been added (if applicable)
+
+### dbt Docs
+Please acknowledge that after the above were all completed the below were applied to your branch:
+- [ ] docs were regenerated (unless this PR does not include any code or yml updates)
+
+### If you had to summarize this PR in an emoji, which would it be?
+<!--- For a complete list of markdown compatible emojis check our this git repo (https://gist.github.com/rxaviers/7360908)  --> 
+:dancer:

.github/pull_request_template.md

@@ -1,52 +1,33 @@
-**Are you a current Fivetran customer?** 
-<!--- Please tell us your name, title and company -->
+**Please provide your name and company**
 
-**What change(s) does this PR introduce?** 
-<!--- Describe what changes your PR introduces to the package and how to leverage this new feature. -->
+**Link the issue/feature request which this PR is meant to address**
+<!--- If an issue was not created, please create one first so we may discuss the PR prior to opening one. -->
 
-**Did you update the CHANGELOG?** 
+**Detail what changes this PR introduces and how this addresses the issue/feature request linked above.**
+
+**How did you validate the changes introduced within this PR?**
+
+**Which warehouse did you use to develop these changes?**
+
+**Did you update the CHANGELOG?**
 <!--- Please update the new package version’s CHANGELOG entry detailing the changes included in this PR. -->
 <!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
 - [ ] Yes
 
-**Does this PR introduce a breaking change?**
-<!--- Does this PR introduce changes that will cause current package users' jobs to fail or require a `--full-refresh`? -->
-<!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
-- [ ] Yes (please provide breaking change details below.)
-- [ ] No  (please provide an explanation as to how the change is non-breaking below.)
-
-**Did you update the dbt_project.yml files with the version upgrade (please leverage standard semantic versioning)? (In both your main project and integration_tests)** 
+**Did you update the dbt_project.yml files with the version upgrade (please leverage standard semantic versioning)? (In both your main project and integration_tests)**
 <!--- The dbt_project.yml and the integration_tests/dbt_project.yml files contain the version number. Be sure to upgrade it accordingly -->
 <!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
 - [ ] Yes
 
-**Is this PR in response to a previously created Bug or Feature Request**
-<!--- If an Issue was created it is helpful to track the progress by linking it in the PR. -->
-<!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
-- [ ] Yes, Issue/Feature [link bug/feature number here]
-- [ ] No 
-
-**How did you test the PR changes?** 
-<!--- Proof of testing is required in order for the PR to be approved. -->
-<!--- To check a box, remove the space and insert an x in the box (eg. [x] Buildkite). --> 
-<!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
-- [ ] Buildkite <!--- Buildkite testing is only applicable to Fivetran employees. --> 
-- [ ] Local (please provide additional testing details below)
-
-**Select which warehouse(s) were used to test the PR**
-<!--- To check a warehouse remove the space and insert an x in the box (eg. [x] Bigquery). --> 
-<!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
-- [ ] BigQuery
-- [ ] Redshift
-- [ ] Snowflake
-- [ ] Postgres
-- [ ] Databricks
-- [ ] Other (provide details below)
-
 **Provide an emoji that best describes your current mood**
 <!--- For a complete list of markdown compatible emojis check our this git repo (https://gist.github.com/rxaviers/7360908)  --> 
 :dancer:
 
 **Feedback**
 
 We are so excited you decided to contribute to the Fivetran community dbt package! We continue to work to improve the packages and would greatly appreciate your [feedback](https://www.surveymonkey.com/r/DQ7K7WW) on our existing dbt packages or what you'd like to see next.
+
+**PR Template** 
+- [Community Pull Request Template](?expand=1&template=pull_request_template.md) (default)
+
+- [Maintainer Pull Request Template](?expand=1&template=maintainer_pull_request_template.md) (to be used by maintainers)

CHANGELOG.md

@@ -1,3 +1,23 @@
+# dbt_jira v0.14.0 
+## 🚨 Breaking Changes 🚨
+- Fixed the `jira__daily_issue_field_history` model to make sure `component` values are correctly joined into the downstream issue models. This applied only if `components` are leveraged within the `issue_field_history_columns` variable. ([PR #99](https://github.com/fivetran/dbt_jira/pull/99))
+>**Note**: Please be aware that a `dbt run --full-refresh` will be required after upgrading to this version in order to capture the updates.
+
+## Bug Fixes
+- Updated the `int_jira__issue_calendar_spine` logic, which now references the `int_jira__field_history_scd` model as an upstream dependency. ([PR #104](https://github.com/fivetran/dbt_jira/pull/104))
+- Modified the `open_until` field  within the `int_jira__issue_calendar_spine` model to be dependent on the `int_jira__field_history_scd` model's `valid_starting_on` column as opposed to the `issue` table's `updated_at` field. ([PR #104](https://github.com/fivetran/dbt_jira/pull/104))
+  - This is required as some resolved issues (outside of the 30 day or `jira_issue_history_buffer` variable window) were having faulty incremental loads due to untracked fields (fields not tracked via the `issue_field_history_columns` variable or other fields not identified in the history tables such as Links, Comments, etc.). This caused the `updated_at` column to update, but there were no tracked fields that were updated, thus causing a faulty incremental load.
+
+
+## Under the Hood
+- Added additional seed rows to ensure the new configuration for components properly runs for all edge cases and compare against normal issue field history fields like `summary`. ([PR #104](https://github.com/fivetran/dbt_jira/pull/104))
+- Incorporated the new `fivetran_utils.drop_schemas_automation` macro into the end of each Buildkite integration test job. ([PR #98](https://github.com/fivetran/dbt_jira/pull/98))
+- Updated the pull request templates. ([PR #98](https://github.com/fivetran/dbt_jira/pull/98))
+
+## Contributors
+- [@kenzie-marsh](https://github.com/kenzie-marsh) ([Issue #100](https://github.com/fivetran/dbt_jira/issues/100))
+
+
 # dbt_jira v0.13.0
 ## 🚨 Breaking Changes 🚨:
 [PR #95](https://github.com/fivetran/dbt_jira/pull/95) applies the following changes:

README.md

@@ -56,7 +56,7 @@ Include the following jira package version in your `packages.yml` file:
 ```yaml
 packages:
   - package: fivetran/jira
-    version: [">=0.13.0", "<0.14.0"]
+    version: [">=0.14.0", "<0.15.0"]
 
 ```
 ## Step 3: Define database and schema variables

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'jira'
-version: '0.13.0'
+version: '0.14.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 vars:

integration_tests/.gitignore

@@ -2,3 +2,5 @@
 target/
 dbt_modules/
 logs/
+.DS_Store
+**/env/

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'jira_integration_tests'
-version: '0.13.0'
+version: '0.14.0'
 config-version: 2
 profile: 'integration_tests'
 
@@ -29,7 +29,7 @@ vars:
     jira_user_identifier: "user"
     jira_version_identifier: "version"
 
-  issue_field_history_columns: ['summary', 'components', 'story points']
+  issue_field_history_columns: ['summary', 'story points', 'components']
 
 seeds:
   jira_integration_tests:

integration_tests/seeds/component.csv

@@ -2,4 +2,4 @@ id,_fivetran_synced,description,name,project_id
 10004,2020-11-23 12:21:02.159,,Component 5,10001
 10003,2020-11-23 12:21:02.159,,Component 4,10001
 10002,2020-11-23 12:21:02.158,,Component 3,10001
-10019,2020-11-23 12:21:02.157,,PI Portal (B2B),10001
+10019,2020-11-23 12:21:02.157,,Component 2,10001

integration_tests/seeds/field.csv

@@ -3,3 +3,4 @@ timeoriginalestimate,2020-11-23 22:20:39.685,false,false,Original Estimate
 creator,2020-11-23 22:20:39.711,false,false,Creator
 issuerestriction,2020-11-23 22:20:39.643,false,false,Restrict to
 components,2020-11-23 22:20:40.643,false,false,Components
+summary,2020-11-23 22:20:41.553,false,false,Summary

integration_tests/seeds/field_option.csv

@@ -2,4 +2,5 @@ id,_fivetran_synced,name
 10104,2020-11-23 12:21:00.979,opt 2
 10016,2020-11-17 12:20:52.552,To Do
 10106,2020-11-17 12:20:52.227,opt 4
-10019,2020-11-19 12:20:53.110,Impediment
+10019,2020-11-19 12:20:53.110,Impediment
+10026,2020-11-20 12:20:54.221,This is the summary of the issue.

integration_tests/seeds/issue.csv

@@ -1,4 +1,4 @@
 id,_fivetran_deleted,_fivetran_synced,_original_estimate,_remaining_estimate,_time_spent,assignee,created,creator,description,due_date,environment,issue_type,key,last_viewed,original_estimate,parent_id,priority,project,remaining_estimate,reporter,resolution,resolved,status,status_category_changed,summary,time_spent,updated,work_ratio
 10011,FALSE,2020-11-06 22:23:38,,,,,2020-04-01 08:33:58,5c3326c24b248c315badf3ee,"Yes, this is an issue to test permissions",,,10103,TP-8,,,,3,10001,,5c3326c24b248c315badf3ee,,2020-05-29 11:33:53,3,2020-04-01 08:34:02,We want to test permissions,,2020-05-29 11:33:53,-1
 10015,FALSE,2020-11-06 22:23:41,,,,,2020-06-11 07:38:03,5ed0edbe2999b60c23255b4f,,,,10103,TP-12,,,,5,10001,,5ed0edbe2999b60c23255b4f,,2020-06-11 07:38:03,10003,2020-06-11 07:38:03,Part of test epic,,2020-06-11 07:38:03,-1
-10018,FALSE,2020-11-12 12:20:50,,,,,2020-11-10 19:11:45,557058:a9572f6a-3041-435a-96ed-5286e3811b33,,,,10103,TCP-1,2020-11-10 19:11:46,,,3,10005,,557058:a9572f6a-3041-435a-96ed-5286e3811b33,,2020-11-10 19:11:45,10003,2020-11-10 19:11:46,this is a story,,2020-11-10 19:11:45,-1
+10018,FALSE,2020-11-12 12:20:50,,,,,2020-11-10 19:11:45,557058:a9572f6a-3041-435a-96ed-5286e3811b33,,,,10103,TCP-1,2020-11-10 19:11:46,,,3,10005,,557058:a9572f6a-3041-435a-96ed-5286e3811b33,,2020-11-28 19:11:45,10003,2020-11-28 19:11:46,this is a story,,2020-11-28 19:11:45,-1

integration_tests/seeds/issue_field_history.csv

@@ -2,4 +2,5 @@ field_id,issue_id,time,_fivetran_synced,value
 created,10027,2020-11-10 19:19:41.472,2020-11-12 12:20:53.478,2020-11-10T19:19:41.472Z
 customfield_10104,10027,2020-11-10 19:19:41.472,2020-11-12 12:20:53.472,3.0
 summary,10027,2020-11-10 19:19:41.472,2020-11-12 12:20:53.500,"As a developer, I'd like to update story status during the sprint >> Click the Active sprints link at the top right of the screen to go to the Active sprints where the current Sprint's items can be updated"
-components,10018,2020-11-10 19:19:41.472,2020-11-12 12:20:53.500,10019
+components,10018,2020-11-10 19:19:41.472,2020-11-19 12:20:53.500,10019
+summary,10018,2020-11-10 19:19:41.472,2020-11-19 12:20:53.500,10026

models/intermediate/field_history/int_jira__issue_calendar_spine.sql

@@ -40,18 +40,23 @@ with spine as (
     {% endif %}
 ),
 
+issue_history_scd as (
+
+    select *
+    from {{ ref('int_jira__field_history_scd') }}
+),
+
 issue_dates as (
 
     select
-        issue_id,
-        cast( {{ dbt.date_trunc('day', 'created_at') }} as date) as created_on,
-
+        issue_history_scd.issue_id,
+        cast( {{ dbt.date_trunc('day', 'issue.created_at') }} as date) as created_on,
         -- resolved_at will become null if an issue is marked as un-resolved. if this sorta thing happens often, you may want to run full-refreshes of the field_history models often
         -- if it's not resolved include everything up to today. if it is, look at the last time it was updated 
-        cast({{ dbt.date_trunc('day', 'case when resolved_at is null then ' ~ dbt.current_timestamp_in_utc_backcompat() ~ ' else updated_at end') }} as date) as open_until
-
-    from {{ var('issue') }}
-
+        cast({{ dbt.date_trunc('day', 'case when issue.resolved_at is null then ' ~ dbt.current_timestamp_in_utc_backcompat() ~ ' else cast(issue_history_scd.valid_starting_on as ' ~ dbt.type_timestamp() ~ ') end') }} as date) as open_until
+    from issue_history_scd
+    left join {{ var('issue') }} as issue
+        on issue_history_scd.issue_id = issue.issue_id
 ),
 
 issue_spine as (