Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Port config value is being passed as an empty string on connection test #13

Open
kning opened this issue Mar 15, 2024 · 1 comment
Open
Assignees

Comments

@kning
Copy link

kning commented Mar 15, 2024

I have the following in my meltano.yml config

  - name: tap-clickhouse
    variant: akurdyukov
    pip_url: git+https://github.com/akurdyukov/tap-clickhouse.git
    config:
      host: $CLICKHOUSE_URL
      password: $CLICKHOUSE_PASSWORD
      port: 8443

And when I run meltano config tap-clickhouse test, I get the following:

Traceback (most recent call last):
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/bin/tap-clickhouse", line 8, in <module>
    sys.exit(TapClickHouse.cli())
             ^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/click/core.py", line 1077, in main
    with self.make_context(prog_name, args, **extra) as ctx:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/click/core.py", line 943, in make_context
    self.parse_args(ctx, args)
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/click/core.py", line 1408, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/click/core.py", line 2400, in handle_parse_result
    value = self.process_value(ctx, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/click/core.py", line 2362, in process_value
    value = self.callback(ctx, self, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 528, in cb_discover
    tap.run_discovery()
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 288, in run_discovery
    catalog_text = self.catalog_json_text
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 308, in catalog_json_text
    return json.dumps(self.catalog_dict, indent=2)
                      ^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 659, in catalog_dict
    result["streams"].extend(connector.discover_catalog_entries())
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/singer_sdk/connectors/sql.py", line 510, in discover_catalog_entries
    engine = self._engine
             ^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/singer_sdk/connectors/sql.py", line 309, in _engine
    self._cached_engine = self.create_engine()
                          ^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/tap_clickhouse/client.py", line 46, in create_engine
    return sqlalchemy.create_engine(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 2, in create_engine
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
    return fn(*args, **kwargs)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/sqlalchemy/engine/create.py", line 546, in create_engine
    u = _url.make_url(url)
        ^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/sqlalchemy/engine/url.py", line 842, in make_url
    return _parse_url(name_or_url)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/sqlalchemy/engine/url.py", line 905, in _parse_url
    return URL.create(name, **components)  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/sqlalchemy/engine/url.py", line 211, in create
    cls._assert_port(port),
    ^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/sqlalchemy/engine/url.py", line 221, in _assert_port
    return int(port)
           ^^^^^^^^^
ValueError: invalid literal for int() with base 10: ''
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

Plugin configuration is invalid
Catalog discovery failed: command ['/meltano_project/.meltano/extractors/tap-clickhouse/venv/bin/tap-clickhouse', '--config', '/meltano_project/.meltano/run/tap-clickhouse/tap.f748e8c9-b5ff-4f33-9268-9e43eba4fc3b.config.json', '--discover'] returned 1 with stderr:
 Traceback (most recent call last):
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/bin/tap-clickhouse", line 8, in <module>
    sys.exit(TapClickHouse.cli())
             ^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/click/core.py", line 1077, in main
    with self.make_context(prog_name, args, **extra) as ctx:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/click/core.py", line 943, in make_context
    self.parse_args(ctx, args)
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/click/core.py", line 1408, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/click/core.py", line 2400, in handle_parse_result
    value = self.process_value(ctx, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/click/core.py", line 2362, in process_value
    value = self.callback(ctx, self, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 528, in cb_discover
    tap.run_discovery()
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 288, in run_discovery
    catalog_text = self.catalog_json_text
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 308, in catalog_json_text
    return json.dumps(self.catalog_dict, indent=2)
                      ^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 659, in catalog_dict
    result["streams"].extend(connector.discover_catalog_entries())
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/singer_sdk/connectors/sql.py", line 510, in discover_catalog_entries
    engine = self._engine
             ^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/singer_sdk/connectors/sql.py", line 309, in _engine
    self._cached_engine = self.create_engine()
                          ^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/tap_clickhouse/client.py", line 46, in create_engine
    return sqlalchemy.create_engine(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 2, in create_engine
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
    return fn(*args, **kwargs)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/sqlalchemy/engine/create.py", line 546, in create_engine
    u = _url.make_url(url)
        ^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/sqlalchemy/engine/url.py", line 842, in make_url
    return _parse_url(name_or_url)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/sqlalchemy/engine/url.py", line 905, in _parse_url
    return URL.create(name, **components)  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/sqlalchemy/engine/url.py", line 211, in create
    cls._assert_port(port),
    ^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/sqlalchemy/engine/url.py", line 221, in _assert_port
    return int(port)
           ^^^^^^^^^
ValueError: invalid literal for int() with base 10: ''

The stack trace seems to suggest that something is happening in client.py when i builds the sqlalchemy URL. Would you be able to take a look?

@akurdyukov
Copy link
Owner

I tried the same with my config and it seems to work. And it seems to work in the tests. Can you please give fill tap-clickhouse config?

@akurdyukov akurdyukov self-assigned this Mar 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants