Feature to add a filtering options on the API call #145

Nevsksar · 2023-09-27T16:41:30Z

Description

When dealing with enterprise workspaces in ClickUp, as ClickUp admins, we often find ourselves having access to multiple workspaces, lists, tasks, and other elements that are not needed in our pipeline. To save resources, processing time, and improve efficiency, it is beneficial for us to have the ability to filter specific elements that we want to fetch.

Noteworthy ClickUp API Bugs

During the development of this project, I encountered some inherent bugs in ClickUp's API. Primarily, these bugs prevented the passing of a single space or list as a parameter. However, a workaround was discovered: by passing the same ID twice, the API would function correctly, and there are not any duplicate values in the API response.

Issues

The following changes have been implemented and work, but they are currently coded to only fetch data from the general settings in the meltano.yml file. Consequently, it is not possible to configure different pipelines for various environments because they are not being correctly read.

For example, when using the command meltano --environment=dev run tap-clickup ..., the configurations are not being applied as expected from the dev envv because it reads only from the common configurations.

Taking this meltano.yml sample, the code always reads "spaces_id" as '90100424497' even when we run it in prod (which in this case should be '90070255202'

version: 1
default_environment: prod
project_id: 8f483ca8-e13c-4abb-8d24-666666666
environments:
- name: prod
  config:
    plugins:
      extractors:
      - name: tap-clickup
        variant: autoidm
        pip_url: tap-clickup
        config:
          workspace_id: 30979640
          spaces_id: '90070255202'
          list_ids: ''
        select:
        - space.name
        - space.id
        - task.*
      loaders:...
plugins:
  extractors:
  - name: tap-clickup
    variant: autoidm
    pip_url: tap-clickup
    config:
      workspace_id: 30979640
      spaces_id: '90100424497'
      list_ids: ''
    select:
    - space.name
    - space.id
    - task.*
  loaders:...

Could not figure out how to correctly add the new setting in the meltano.yml file. At first I was just hardcoding it to the file, then I found out about tap-clickup--autoidm.lock file and in this file I added the new configuration settings to be added when using meltano config tap-clickup -set. But I cannot figure out how to edit this into the projects code, i basically added these configs to the og file:

. . .
    {
      "name": "stream_map_config",
      "kind": "object",
      "label": "Stream Map Config",
      "description": "User-defined config values to be used within map expressions."
    },
    {
      "name": "stream_maps",
      "kind": "object",
      "label": "Stream Maps",
      "description": "Config object for stream maps capability. For more information check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html)."
    },
	    {
      "name": "workspace_id",
      "kind": "integer",
      "label": "Workspace Id",
      "description": "To limit from which workspace to fetch data pass it's id(https://dev-doc.clickup.com/333/d/h/ad-455845/c464293970a6906)."
    },
	    {
      "name": "spaces_id",
      "kind": "string",
      "label": "Spaces Ids",
      "description": "To limit from which Spaces to fetch data from, pass their ids as a list string Example: spaceid1,spaceid2,ispaced3."
    },
	    {
      "name": "list_ids",
      "kind": "string",
      "label": "Lists Ids",
      "description": "To limit from which Lists to fetch data from, pass their ids as a list string Example: listid1,listid2,listid3."
    }
  ]
}

visch · 2023-09-27T21:09:17Z

tap_clickup/tap.py

+            "workspace_id", th.IntegerType, required=False, description="Example: '20214542" # fetches the data for workspace_id
+        ),
+        th.Property(
+            "spaces_id", th.StringType, required=False, description="Example: '[45215477,4547547]" # fetches the data for workspace_id


This should be an ArrayType see https://sdk.meltano.com/en/latest/classes/typing/singer_sdk.typing.ArrayType.html

visch · 2023-09-27T21:12:53Z

tap_clickup/streams.py

    # Need this stub as a hack on _sync to force it to use Partitions
    # Since this is a child stream we want each team_id to create a request for
    # archived:true and archived:false. And we want state to track properly
    partitions = []
-
+    basepath = "/team/{team_id}/task"
+    space_ids_param = "&space_ids=" + "&space_ids=".join(map(str, space_ids)) if space_ids else "" #Dinamically create the spaces path to be passed on API call


Params should be handled via https://sdk.meltano.com/en/latest/classes/singer_sdk.RESTStream.html#singer_sdk.RESTStream.get_url_params

visch · 2023-09-27T21:14:22Z

tap_clickup/streams.py

@@ -16,7 +60,9 @@ class TeamsStream(ClickUpStream):
    primary_keys = ["id"]
    replication_key = None
    schema_filepath = SCHEMAS_DIR / "team.json"
-    records_jsonpath = "$.teams[*]"
+    # Necessary because if you have access to multiple workteams, the responses are replicated N times where N = # of worspaces you have access to
+    records_jsonpath = f"$.teams[?(@.id == {workteamid})]" 


Did they add a new concept of workteamid? Or workspace_id, could you link to that in clickups docs?

So the structure is Workspace>Space>Folder>List>Task>Subtask

In the app/platform they are called workspaces, but for the API they are called Teams, also there is a feature called "teams" (that group users together) but in the API they call it "group"

https://clickup.com/api/clickupreference/operation/CreateTeam/

visch · 2023-09-27T21:16:39Z

tap_clickup/streams.py

@@ -4,6 +4,50 @@
 import requests
 from singer_sdk.helpers.jsonpath import extract_jsonpath
 from tap_clickup.client import ClickUpStream
+import yaml #added to read information specifying spaces and workspaces


All of this needs to go away. Please go through the getting started for Meltano, and read through how singer works.

Adding extra fields to a config block added here https://github.com/AutoIDM/tap-clickup/blob/main/meltano.yml#L6 will pass all of the configs via the config CLI argument (via a json file) to the tap. This is all handled by the config options you altered in tap.py below

visch · 2023-09-27T21:16:56Z

tap_clickup/streams.py

    # Need this stub as a hack on _sync to force it to use Partitions
    # Since this is a child stream we want each team_id to create a request for
    # archived:true and archived:false. And we want state to track properly
    partitions = []
-
+    basepath = "/team/{team_id}/task"


Revert this please

visch · 2023-09-27T21:17:03Z

tap_clickup/tap.py

+            "spaces_id", th.StringType, required=False, description="Example: '[45215477,4547547]" # fetches the data for workspace_id
+        ),
+        th.Property(
+            "list_ids", th.StringType, required=False, description="Example: '[454455478,784552187]" # fetches the data for workspace_id


visch · 2023-09-27T21:17:06Z

tap_clickup/tap.py

@@ -47,6 +47,15 @@ class TapClickUp(Tap):
        th.Property(
            "api_token", th.StringType, required=True, description="Example: 'pk_12345"
        ),
+        th.Property(
+            "workspace_id", th.IntegerType, required=False, description="Example: '20214542" # fetches the data for workspace_id


visch · 2023-09-27T21:17:17Z

tap_clickup/streams.py

@@ -220,20 +267,26 @@ class TasksStream(ClickUpStream):

    name = "task"
    # Date_updated_gt is greater than or equal to not just greater than
-    path = "/team/{team_id}/task"


visch · 2023-09-27T21:17:40Z

tap_clickup/streams.py

All changes from line 0 to line 50 need to go away

…vious commit.

Addresses the requests to merge the PR AutoIDM#145

Feature to add a filtering options on the API call

130fae8

visch requested changes Sep 27, 2023

View reviewed changes

HiaggoMachado and others added 3 commits October 13, 2023 19:22

Addresses the requests to merge the PR AutoIDM#145

7caa0c0

Fixes discrepancies between branches that were not present in the pre…

8af093c

…vious commit.

Merge pull request #1 from Hiaggo/fix/pr-143-correction

1c6688a

Addresses the requests to merge the PR AutoIDM#145

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature to add a filtering options on the API call #145

Feature to add a filtering options on the API call #145

Nevsksar commented Sep 27, 2023

visch Sep 27, 2023

visch Sep 27, 2023

visch Sep 27, 2023

Nevsksar Sep 28, 2023

visch Sep 27, 2023

visch Sep 27, 2023

visch Sep 27, 2023

visch Sep 27, 2023

visch Sep 27, 2023

visch Sep 27, 2023 •

edited

Loading

Feature to add a filtering options on the API call #145

Are you sure you want to change the base?

Feature to add a filtering options on the API call #145

Conversation

Nevsksar commented Sep 27, 2023

Description

Noteworthy ClickUp API Bugs

Issues

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

visch Sep 27, 2023 • edited Loading

Choose a reason for hiding this comment

visch Sep 27, 2023 •

edited

Loading