[CT-3160] minimal snowplow tracker and snowplow python tracker are incompatible for the latest versions of the snowplow python tracker

[graciegoheen]

Originally posted by @akurdyukov in #8409 (comment)

I bumped into a issue with minimal showplow after installing meltano using this guide: https://dagster.io/blog/dagster-meltano-integration-tutorial

The guide is a little bit inaccurate because it does not say that you need to install meltano package into venv the dagster is running in. And after installing meltano (and doing nothing with the code) I ran into

Stack Trace:
  File "/Users/alik/Development/dagster-env-10/lib/python3.10/site-packages/dagster/_grpc/server.py", line 295, in __init__
    self._loaded_repositories: Optional[LoadedRepositories] = LoadedRepositories(
  File "/Users/alik/Development/dagster-env-10/lib/python3.10/site-packages/dagster/_grpc/server.py", line 139, in __init__
    loadable_targets = get_loadable_targets(
  File "/Users/alik/Development/dagster-env-10/lib/python3.10/site-packages/dagster/_grpc/utils.py", line 47, in get_loadable_targets
    else loadable_targets_from_python_module(module_name, working_directory)
  File "/Users/alik/Development/dagster-env-10/lib/python3.10/site-packages/dagster/_core/workspace/autodiscovery.py", line 35, in loadable_targets_from_python_module
    module = load_python_module(
  File "/Users/alik/Development/dagster-env-10/lib/python3.10/site-packages/dagster/_core/code_pointer.py", line 135, in load_python_module
    return importlib.import_module(module_name)
  File "/opt/homebrew/Cellar/[email protected]/3.10.13/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/alik/Development/dagster_analytics/dagster_analytics/definitions.py", line 4, in <module>
    from dagster_dbt import DbtCliResource
  File "/Users/alik/Development/dagster-env-10/lib/python3.10/site-packages/dagster_dbt/__init__.py", line 2, in <module>
    from .asset_defs import (
  File "/Users/alik/Development/dagster-env-10/lib/python3.10/site-packages/dagster_dbt/asset_defs.py", line 62, in <module>
    from dagster_dbt.core.resources import DbtCliClient
  File "/Users/alik/Development/dagster-env-10/lib/python3.10/site-packages/dagster_dbt/core/__init__.py", line 5, in <module>
    from .resources_v2 import (
  File "/Users/alik/Development/dagster-env-10/lib/python3.10/site-packages/dagster_dbt/core/resources_v2.py", line 35, in <module>
    from dbt.contracts.results import NodeStatus, TestStatus
  File "/Users/alik/Development/dagster-env-10/lib/python3.10/site-packages/dbt/contracts/results.py", line 1, in <module>
    from dbt.contracts.graph.unparsed import FreshnessThreshold
  File "/Users/alik/Development/dagster-env-10/lib/python3.10/site-packages/dbt/contracts/graph/unparsed.py", line 3, in <module>
    from dbt import deprecations
  File "/Users/alik/Development/dagster-env-10/lib/python3.10/site-packages/dbt/deprecations.py", line 6, in <module>
    import dbt.tracking
  File "/Users/alik/Development/dagster-env-10/lib/python3.10/site-packages/dbt/tracking.py", line 104, in <module>
    emitter = TimeoutEmitter()
  File "/Users/alik/Development/dagster-env-10/lib/python3.10/site-packages/dbt/tracking.py", line 54, in __init__
    super().__init__(

This happens because meltano depends on snowplow-tracker [required: >=1.0.1,<2.0.0, installed: 1.0.1]. And the contract for snowplow_tracker.Emitter does not match between snowplow-tracker and minimal-snowplow-tracker.

[jtcohen6]

We opened a new issue to distinguish from #8409 (spike), and track the fact that

• minimal-snowplow-tracker and snowplow-python-tracker are currently incompatible
• this is most relevant for users of dbt-core + meltano

I see at least three potential ways of resolving this narrower issue:

1. After investigating the spike in [CT-2991] [SPIKE] Investigate minimal-snowplow-tracker #8409, decide that we actually don't want to keep maintaining our own fork, and switch to using the "real" snowplow-python-tracker within dbt-core
2. After investigating the spike in [CT-2991] [SPIKE] Investigate minimal-snowplow-tracker #8409, decide that we want to keep maintaining our own fork (for the performance reasons originally identified), but update the relevant method signatures to match the ones in newer versions of snowplow-python-tracker (>=1.0.1,<2.0.0)
3. What's proposed in Fix conflict with newer versions of Snowplow tracker #9680: Add defensive code within dbt-core that allows it to dynamically use either minimal-snowplow-tracker or snowplow-python-tracker, whichever is installed

[Daniel-SST]

@jtcohen6 is there an update on this ticket? I'm running in to this issue and this is preventing me from adopting dbt as part of our ELT pipelines.

[s-jorgenfroland]

Im running into what I think is a related issue. I started getting this error when running dbt commands (either just dbt --version or trying to run/build a model):

from dbt.cli import requires, params as p
File "/usr/local/lib/python3.10/site-packages/dbt/cli/requires.py", line 1, in
import dbt.tracking
File "/usr/local/lib/python3.10/site-packages/dbt/tracking.py", line 104, in
emitter = TimeoutEmitter()
File "/usr/local/lib/python3.10/site-packages/dbt/tracking.py", line 54, in init
super().init(
TypeError: Emitter.init() got an unexpected keyword argument 'buffer_size'

We use dbt-core v1.6.6, snowplow-plow-tracker v1.0.1 and minimal-snowplow-tracker v0.0.2.

In line 54 here https://github.com/dbt-labs/dbt-core/blob/v1.6.6/core/dbt/tracking.py#L54, buffer_size is sent as input, but is not an input parameter in the init-function of class Emitter here: https://github.com/snowplow/snowplow-python-tracker/blob/1.0.1/snowplow_tracker/emitters.py#L57. The Emitter-class init-function has batch_size and buffer_capacity.

EDIT: We have recently started using Meltano (v3.1.0). What I find a bit strange is that in our poetry.lock dbt-core depends on

minimal-snowplow-tracker = ">=0.0.2,<0.1.0"

But Meltano (v3.1.0) depends on

snowplow-tracker = ">=1.0.1,<2.0.0"

Can it create conflict, and hence result in the error we get, having dependency to both these packages. As minimal-snowplow-tracker is a fork of the other?

[dbeatty10]

We merged #9680 (option 3 in #8719 (comment)), and this will be included with the release of v1.8.

The long-term fix is still to do the spike in #8409 so we can choose between options 1 and 2.

[edgarrmondragon]

An additional option to those in #8719 (comment) is to vendor minimal-snowplow-tracker so dbt can import from dbt._vendor.snowplow_tracker, for example.