Failed import from tap-shopify. Need suggestions

[sphinks]

I try to import data (entities 'Product' and 'Order') from shopify to snowflake and use tap-shopify. However, I managed only to create tables and get issue with data import on shopify entity 'Order' ('Product' syncing smooth). What I get in logs:

INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.7647573947906494, "tags": {"endpoint": "products", "status": "succeeded"}}
INFO Skipping stream: collects
INFO Skipping stream: transactions
INFO Syncing stream: orders
INFO GET https://shopname.myshopify.com/admin/api/2020-07/orders.json?since_id=1&updated_at_min=2020-09-01+00%3A00%3A00%2B00%3A00&updated_at_max=2020-09-02+00%3A00%3A00%2B00%3A00&limit=175&status=any
INFO --> 200 OK 59714b
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.4373745918273926, "tags": {"endpoint": "orders", "status": "succeeded"}}
Traceback (most recent call last):
  File "/home/ivan/Projects/pipelinewise/.virtualenvs/target-snowflake/bin/target-snowflake", line 8, in <module>
    sys.exit(main())
  File "/home/ivan/Projects/pipelinewise/.virtualenvs/target-snowflake/lib64/python3.7/site-packages/target_snowflake/__init__.py", line 452, in main
    persist_lines(config, singer_messages, table_cache)
  File "/home/ivan/Projects/pipelinewise/.virtualenvs/target-snowflake/lib64/python3.7/site-packages/target_snowflake/__init__.py", line 189, in persist_lines
    adjust_timestamps_in_record(o['record'], schemas[stream])
  File "/home/ivan/Projects/pipelinewise/.virtualenvs/target-snowflake/lib64/python3.7/site-packages/target_snowflake/__init__.py", line 148, in adjust_timestamps_in_record
    if 'string' in schema['properties'][key]['type'] and \
KeyError: 'type'

According to source code it is function that put max date into empty date fields. Any clue, what going wrong?

[Samira-El]

It seems like the json schema of stream orders is invalid.
Can you perhaps share it here? you should be able to find it in properties.json file

[sphinks]

@Samira-El sure. Take a look, please!

[Samira-El]

It looks like many properties in the orders schema are empty, thus lacking the type property, e.g:

...
"total_price_set": {},
"total_price_usd": {
    "type": [
        "null",
        "number"
    ]
},
"total_shipping_price_set": {},
"total_tax": {
    "type": [
        "null",
        "number"
    ]
},
"total_tax_set": {},
"total_tip_received": {
    "type": [
        "null",
        "string"
    ]
}
...

The type property is important to have so that the target knows which column type to use when creating the table.

We don't maintain tap-shopify and it seems like this issue has already been raised: singer-io/tap-shopify#78

[Samira-El]

Closing as this is an issue in the tap.

[sphinks]

@Samira-El Thanks for your time and clarification. I will follow issue in tap-shopify.