Skip to content

Latest commit

 

History

History
574 lines (329 loc) · 26.7 KB

CHANGELOG.md

File metadata and controls

574 lines (329 loc) · 26.7 KB
Raw

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

v0.8.0 (2022-08-05)

🐛 Fixes

  • #784 Update return type for backoff_max_tries to reflect it accepts a callable that returns an integer
  • #874 Singer metrics are now properly emitted in JSON format --Thanks, @Jack-Burnett!

📚 Documentation Improvements

  • #869 Cleanup whitespace in backoff code samples

v0.7.0 (2022-07-21)

✨ New

  • #785 Output full URL path in error messages

🐛 Fixes

  • #815 Generate correct SQL target project from cookiecutter
  • #782 Allow lists and dictionaries as types for default JSON values

📚 Documentation Improvements

  • #823 Add link to the sdk for README generation regarding Stream Maps
  • #813 Fix PyPI trove classifiers
  • #783 Document using pipx inject for nox-poetry

v0.6.1 (2022-07-01)

Fix

  • Fix missing typing-extensions for Python<3.10 (#776)

0.6.0 - (2022-06-30)

New

  • Tap and Target SDK: Adds the ability to override the logging level via LOGLEVEL environment variables. (!300) - Thanks, Eric Boucher!
  • Tap SDK: Allow sort checking to be disabled for taps with un-sortable replication keys. (#729, #730) - Thanks, Ilkka!
  • Tap SDK: Add Stream.calculate_sync_cost hook to gather "cost" (in terms of number of requests, etc.) for each stream. (#348, #704) - Thanks, Laurent Savaëte!
  • Target SDK: Add end-of-pipe clean up hooks for Sink implementations (#750) - Thanks, Alexander Butler!

Changes

  • Tap SDK: Bump minimum version of PyJWT to 2.4 (#396, !302).
  • Tap SDK: Compare replication key values when incrementing state, only if the check_sorted flag is set to True (#736, #737).

Fixes

  • Tap and Target SDK: Resolves an issue where SQLAlchemy logs would be printed to STDOUT. (!303) - Thanks, Andrey Tatarinov!
  • Tap SDK: Reduces number of log lines printed when unmapped properties are ignored from the source API (!300) - Thanks, Eric Boucher!
  • Tap and Target SDK: Remove trailing parenthesis from logged version (#766, #767).

0.5.0 - (2022-05-19)

New

  • Tap and Target SDK: The --config=ENV option now also considers environment variables from a dotenv (.env) file if present (#384, !293).

Changes

  • Target SDK: Use generic SQLALchemy markers for parameterized queries in SQL targets (#376, !287) - Thanks, Thomas Briggs!

  • Target SDK: Explicitly specify column names when inserting rows in SQL targets (#385, !294) - Thanks, Thomas Briggs!

Fixes

  • Target SDK: Fixes bug where an activate_version message caused an error when destination table does not exist (#372, !285) - Thanks, Thomas Briggs!

  • Tap and Target SDK: Do not fail --about option if tap or target is not configured (#379, !291).

0.4.9 - (2022-05-12)

New

Changes

  • Remove "errors" from non-error log line to improve log searchability (!279) - Thanks, Laurent Savaëte!

Fixes

  • Tap and Target SDK: Fixed a bug where setting stream map property to null did not remove it from SCHEMA message (#370, !286) - Thanks, Ryan Whitten!
  • Tap and Target SDK: Fixed a bug where flattening resulted in an invalid SCHEMA message (!286) - Thanks, Ryan Whitten!

0.4.8 - (2022-05-05)

Fixes

  • Target SDK: Use maxLength in SQL targets for string fields if the schema provides it (#371, !284) - Thanks, Thomas Briggs!

0.4.7 - (2022-04-28)

Fixes

  • Tap Cookiecutter: Fixed a bug where the wrong key was used to select the appropriate Tap class for SQL taps (#353, !275) - Thanks, Dan Norman!
  • Tap Cookiecutter: Fixed a bug where client.py for SQL taps was overwritten during initialization (#354, !276) - Thanks, Dan Norman!
  • Tap SDK: Fixed a bug where a parent stream emitted schema messages when it's not selected, but at least one of its child streams is (#366, !280)
  • Tap SDK: Bump pyjwt dependency to ~=2.3 (!281) - Thanks, Eric Boucher!

0.4.6 - (2022-04-21)

Fixes

  • Raise more descriptive exceptions when wrapped JSON typing classes needs to be instantiated (#55, #360, !270).
  • Support JSONPath extensions in records_jsonpath and next_page_token_jsonpath (#361, !271).

0.4.5 - (2022-04-08)

Fixes

  • Fixed a bug where setting a stream map property to null did not remove the property (#352, !263).
  • Avoid SQLAlchemy deprecation warning caused by using Engine.has_table() (#341, !264)
  • Resolve issue where "falsey" defaults like '0', '', and 'False' would not be properly applied to tap settings config. (#357, !265)
  • Return to stable poetry-core version in cookiecutter templates (#338, !260)

0.4.4 - (2022-03-03)

New

0.4.3 - (2022-02-18)

New

Changes

0.4.2 - (2022-02-04)

New

  • Add record and schema flattening in Stream Maps (!236),

Fixes

  • Resolve issues when aliasing stream maps using the keywords __alias__, __source__, or __else__ (#301, #302, !243)

0.4.1 - (2022-01-27)

Changes

  • Always sync one record per stream when invoking with --test or --test=all (#311, !241)
  • Add --test=schema option to emit tap SCHEMA messages only (!218) - Thanks, Laurent Savaëte!

0.4.0 - (2022-01-21)

New

  • Add support for SQL Taps (#74, !44)
  • Add support for SQL Targets (#263, !44, !200, !239)
  • Added Licence tracking to SDK GitLab Project (#166, !237)

0.3.18 - (2022-01-13)

New

  • Inline Mapper SDK: Support for creation of inline mapper plugins (#257, !234)
  • Tap and Target Cookiecutter: Support editable pip installs (pip install -e .) with PEP 660 (#238, !231)

Fixes

0.3.17 - (2021-12-16)

New

  • Tap SDK: Add configurable timeout for HTTP requests (#287, !217, !225) -- Thanks, Josh Lloyd!
  • Tap and Target SDK: Adds support for Python 3.10 (#293, !224)

Fixes

  • Resolve lint errors when ArrayType is used to wrap other types (!223) -- Thanks, David Wallace!

0.3.16 - (2021-12-09)

Fixes

  • Tap SDK: Fix datelike type parsing bug with nested schemas (#283, !219)
  • Tap SDK: Resolved bug in --test which caused child streams to not use record limiting (#268, !204, !220) -- Thanks, Derek Visch!

0.3.15 - (2021-12-03)

Fixes

  • Tap SDK: Fixed mapped key_properties not being passed to the emitted schema message (#281, !209)
  • Tap SDK: Fixed missing schema during development causing sync to fail #284, !212 -- Thanks, Fred Reimer!

0.3.14 - (2021-11-18)

New

  • Tap SDK: New method RESTStream.validate_response for custom validation of HTTP responses (#207, !195)
  • Tap SDK: New method RESTStream.request_decorator for custom back-off and retry parameters (#137, !195)

Changes

  • Target SDK: Split sink classes into separate modules (#264, !201)

Fixes

  • Target SDK: Document options for the target CLI and accept multiple config files as input (!183)

0.3.13 - (2021-10-28)

New

  • Target SDK: CLI flag for targets to read messages from a file instead of stdin (#249, !190) -- Thanks, Charles Julian Knight!
  • Target SDK: Add target mock classes and tap-to-target scenario tests (#198, !138)
  • Tap and Target SDK: Create expanded list of capabilities (#186, !141)

0.3.12 - (2021-10-21)

Fixes

  • Tap and Target SDK: Fix markdown table formatting in --about for multi-line settings descriptions (#240, !185)
  • Tap SDK: Clarify undocumented feature of filtering None from post_process() (#233, !187)
  • Tap and Target SDK: Add dataclasses as an explicit third-party dependency for Python 3.6 (#245, !189)
  • Tap and Target SDK: Allows --discover and --about execution without requiring settings validation (#235, !188)

0.3.11 - (2021-10-07)

New

  • Tap and Target SDK: Adds capability to print markdown docs with --about --format=markdown (!172, !180) -- Thanks, Nick Müller!

Changes

  • Tap and Target SDK: Autogenerated docstrings for arguments, return types, and exceptions raised (!166).
  • Tap and Target SDK: Support Black by default by bumping min Python version to 3.6.2. (#224, !169)

Fixes

  • Fixes a bug where tox invocations after initial setup failed (!179) -- Thanks, Jon Watson!.
  • Tap SDK: Fixes a bug in Stream.get_starting_timestamp() and Stream.get_starting_replication_key_value() calls where results where not cached breaking stream sorting (!157)

0.3.10 - (2021-09-30)

Changes

  • Tap and Target SDK: Prevents the leaking of sensitive configuration values when JSON schema validation fails (!173) -- Thanks, Kevin Mullins!.

0.3.9 - (2021-09-23)

New

Changes

  • Tap SDK: Set key_properties = [] instead of null per the Singer spec (!160) -- Thanks, Niall Woodward!

Fixes

  • Tap SDK: Fixes issue where stream map schema generation fails when overriding the value of an existing property. (#196, !165)

0.3.8 - (2021-09-16)

Fixes

  • Tap and Target SDK: Resolves 2to3 compatibility issues when installed with setuptools>=58.0.
  • Resolve issue preventing repo from being cloned on Windows.

0.3.7 - (2021-09-09)

New

Changes

  • Tap SDK: Improved record parsing and validation performance, especially with large record objects (#161, !146)
  • Tap SDK: Changed the signature of Stream.apply_catalog to reflect new catalog parsing flow (#161, !146)

0.3.6 - (2021-08-26)

New

  • Tap and Target SDK: Adds support for Python 3.9 (#66, !38)
  • Tap SDK: Added support for new authenticator classes: BasicAuthenticator, BearerTokenAuthenticator, and APIKeyAuthenticator (#185, !128) -- Thanks, Stephen Bailey!

Changes

  • Tap and Target SDK: Bumps click library version to 8.0 (#178, !140).
  • Target SDK: Improves BatchSink performance by reducing the frequency by which batches are processed. (#172, !137)

Fixes

  • Tap SDK: Improves CLI --help output (#177, !140).
  • Tap SDK: Fixes a bug in state tracking where timezone-aware timestamps are appended again with +0:00 (#176, !142) -- Thanks, Joshua Adeyemi!
  • Tap SDK: Improve performance by reusing a single authenticator instance (#168, #173, !136)

0.3.5 - (2021-08-17)

Fixes

  • Tap SDK: Fixed a bug where not using a catalog file resulted in all streams being selected but all properties being removed from the schema and records (#190, !132)

v0.3.4

New

  • Tap SDK: Added full support for selection metadata as specified by the Singer Spec, including metadata for selected, selected-by-default, and inclusion (!121)

Changes

  • Target SDK: Improved performance for Batch Sinks by skipping extra drain operations when newly recieved STATE messages are unchanged from the prior received STATE (#172, !125) -- Thanks, Pat Nadolny!

Fixes

  • Target SDK: Fixed a bug where would emit an invalid STATE payload (#188, !130) -- Thanks, Pat Nadolny!
  • Tap SDK: Fixed a bug where replication key signposts were not correctly applied for streams which defined them (#180, !129)

v0.3.3

New

  • Added JSONPath for taps to handle record extraction from API responses (!77)

Fixes

  • Resolve batch context not being reset between batches (#164, !117)

Removes

  • Removed unused DEFAULT_BATCH_SIZE_ROWS in favor of max_size for BatchSink implementations (#163, !118)

v0.3.2

Fixes

  • Resolve stream map duplicates not aliased correctly (#154, !114)

v0.3.1

New

  • Added target support for add_record_metadata config (#157, !111)

Fixes

  • Resolve target failures when dates are parsed prior to JSON Schema validation (#156, !110)
  • Resolve target failures when default_sink_class is not used (#153, !109)
  • Improved tap log messages when child property's selection metadata defaults to the parent's (#131, !91)

v0.3.0

New

  • Added Stream Map feature for inline transformation and filtering capabilities (#63, !4, !92, !103)
  • Added Target SDK, components and templates for building Singer targets (#96, !4)

Breaks

  • Removed methods deprecated in v0.2.0: get_partition_state() and get_stream_or_partition_state(). Affected developers should replace these with references to get_context_state(). (#152, !107)

v0.2.0

New

  • Added support for parent-child streams (#97, !79)
  • Added support for configurable metrics logging (#91, !79)
  • Added ability to use fewer state bookmarks by setting Stream.state_partitioning_keys to a subset of available context keys (!79)

Changes

  • Renamed the optional partition dictionary arg in method signatures to the more generic context (!79)

Deprecates

  • The methods Stream.get_partition_state() and Stream.get_stream_or_partition_state() have been deprecated in favor of the new and simpler get_context_state() (!79)

Improves

  • Code coverage is now tracked and available as a tool for SDK contributors to further improve overall stability and help prioritize unit test development. (#39, !89)

v0.1.6

Stability and bugfix release. No breaking changes.

Fixes

  • Resolved excessive logging during selection filtering. (#125, !83)
  • Resolved issue where deselected sub-fields were incorrectly included in stream records. (#126, !85) -- Thanks, Alex Levene!

Improves

  • Added improved type hints for developers, including mypy code compliance for improved stability. (#127, !86)

v0.1.5

Bugfix release. No breaking changes.

Fixes

  • Resolved tap failure when a sorted stream has non-unique replication keys. (#120, !82)

v0.1.4

Significant release with newly added features. No breaking changes.

New

  • Added support for GraphQL query variables (#115, !78)
  • Added selection rules support for record and schema messages (#7, !26)

Changes

  • Improved cookiecutter template coverage, resolved readability issues. (#116, #119, !75)

Fixes

  • Resolved tap failure when a stream is missing from the input catalog. (#105, !80)
  • Resolved bug where unsorted streams did not properly advance state bookmarks for incremental streams. (#118, !74)

v0.1.3

Significant release with newly added features. No breaking changes.

New

  • Added is_sorted stream property, which enables long-running incremental streams to be resumed if interrupted. (!61)
  • Added signpost feature to prevent bookmarks from advancing beyond the point where all records have been streamed. (!61)
  • Added get_replication_key_signpost() stream method which defaults to the current time for timestamp-based replication keys. (!61)

Fixes

  • Fixed a scenario where unsorted incremental streams would generate incorrect STATE bookmarks. (!61) -- Thanks, Egi Gjevori!
  • Fixed a problem where CI pipelines would fail when run from a fork. (!71) -- Thanks, Derek Visch!
  • Fixed fatal error when running from the cookiecutter shell script (#102, !64)

v0.1.2

Fixes bug in state handling, adds improvements to documentation.

Documentation

  • Streamlined Dev Guide (!56)
  • Added Code Samples page, including dynamic schema discovery examples (#33, !56)
  • Added links to external sdk-based taps (#32, !56)
  • Added static/dynamic property documentation (#86, !56)
  • Added "implementation" docs for debugging and troubleshooting (#71, !41)

Fixes

  • Fixes bug in Stream.get_starting_timestamp() using incorrect state key (#94, !58)

v0.1.1

Documentation and cookiecutter template improvements.

New

  • Added 'admin_name' field in cookiecutter, streamline poetry setup (!25)
  • Added meltano integration and testing options (#47, !52)
  • Added new cookiecutter .sh script to ease testing during development (!52)

Changes

  • Improved cookiecutter readme template with examples (#76, !53)

v0.1.0

First official SDK release. Numerous changes and improvements implemented, with the goal of stabilizing the SDK and making it broadly available to the community.

New

  • Added this CHANGELOG.md file (#68, !43)
  • Added standardized tap tests (!36, #78, !46)
  • Added SDK testing matrix for python versions 3.6, 3.7, 3.8 (#61, !33)
  • Added support for multiple --config= inputs, combining one or more config.json files (#53, !27)
  • Added new CLI --test option to perform connection test on all defined streams (#14, !28)
  • Added default value support for plugin configs (!12) -- Contributed by: Ken Payne

Changes

  • Promote singer_sdk.helpers.typing to singer_sdk.typing (#84)
  • Modified environment variable parsing logic for arrays (#82)
  • Renamed http_headers in Authenticator class to auth_headers (#75, !47)
  • Expect environment variables in all caps (<PLUGIN>_<SETTING>) (#59, !34)
  • Parse environment variables only if --config=ENV is passed (#53, !27)

Fixes

  • OAuth no longer applies client_email automatically if client_id is missing (#83)
  • Resolved issue on Python 3.6: cannot import 'metadata' from 'importlib' (#58)
  • Fixed issue reading from JSON file (!11) -- Contributed by: Edgar R. Mondragón
  • Look only for valid plugin settings in environment variables (!21) -- Contributed by: Edgar R. Mondragón
  • Fixed bug in STATE handling (!13) -- Contributed by: Ken Payne

Removes

  • Remove parquet sample (#81,!48)

v0.0.1-devx

Initial prerelease version for review and prototyping.