Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide user ids in team + time entries #146

Open
wants to merge 21 commits into
base: main
Choose a base branch
from

Conversation

mlahp7
Copy link

@mlahp7 mlahp7 commented Mar 22, 2024

  • Added two new settings for clickup time entries
    • time_entry_start_date: The start date that determines how far back in time the extractor gets time entries.
    • time_entry_assignees: used to fetch time entries assigned to specific users
  • Updated the schema for time entries. It seems with the release of clickup mobile, there are sometimes null entries for certain properties. I've reached out to clickup support to validate whether this is intentional
  • Made the 'at' field of the time entry stream the replication key, since for this particular endpoint, it serves as the update_at field.

Notes:
The time_entry_start_date might be overkill but currently there isn't a query param we can use for this endpoint that allows us to query for entries that have changed. Something like date_updated_gt from the get tasks endpoint would be great here

The new default behavior for time entries will be to get the entries of ALL users in your team. If you want only a subset of assignees you can use the time_entry_assignees conf.

Copy link
Contributor

@visch visch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR looks great, does this solve for #134 ?

@mlahp7
Copy link
Author

mlahp7 commented Mar 25, 2024

PR looks great, does this solve for #134 ?

Thanks, yes it does for the most part, minus the location names that was mentioned in their issue. I don't mind tacking them on this PR.

I'm a bit new to singer and taps in general, is the best course of action to add a new setting for this as I've done with the other two configs or is there another preferred way to add these types of configurations?

@visch
Copy link
Contributor

visch commented Mar 25, 2024

PR looks great, does this solve for #134 ?

Thanks, yes it does for the most part, minus the location names that was mentioned in their issue. I don't mind tacking them on this PR.

I'm a bit new to singer and taps in general, is the best course of action to add a new setting for this as I've done with the other two configs or is there another preferred way to add these types of configurations?

If you could update the README portion of Time Entries to include where this PR would bring us to I'd be good with that. For the config's yes you're doing it right, the only thing you could do on top of that is update the configuration readme portion of the docs (if you run poetry run tap-clickup --about --format=md it'll output markdown for you to copy paste!

Copy link
Contributor

@visch visch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should utilize replication keys here so we don't pull old time entries every run.

# Formatted in ISO 8601, it must now be converted to milliseconds
start_date = datetime.strptime(self.config["time_entry_start_date"], "%Y-%m-%dT%H:%M:%SZ")
# Convert the datetime object to milliseconds
params["start_date"] = int(start_date.timestamp() * 1000)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just noticed this, we should really be using the at replication key here not just the start_date.

See https://sdk.meltano.com/en/latest/incremental_replication.html#incremental-replication , specefically self.get_starting_timestamp(context) , and instead of having a seperate start date for time_entry we should probably just drop the time_entry_start_date configuration and just document that start_date does what time_entry_start_date is currently documented to do

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added an if statement to fallback to the time_entry_start_date in the case that the state is empty on a user's first execution. Otherwise it would use the "at" field as the replication key as suggested.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not use self.get_starting_timestamp(context) here? the start_date config variable is already use in this function so we don't need time_entry_start_date either, unless you think we need two seperate start date windows?

tap_clickup/tap.py Show resolved Hide resolved
mcarriere and others added 3 commits May 17, 2024 13:48
* Per reviewer suggestion, use replication key as query for start date

* Bring back time_entry_start_date, in the case someone is just starting fresh, they would need to enter a date for the very first sync. The at field is set in milliseconds as the replication_key, we will plug that in when it is provided in the state

* Bring back if statement for initial execution falling back to config date

* Formatting
@mlahp7
Copy link
Author

mlahp7 commented May 17, 2024

PR looks great, does this solve for #134 ?

Thanks, yes it does for the most part, minus the location names that was mentioned in their issue. I don't mind tacking them on this PR.
I'm a bit new to singer and taps in general, is the best course of action to add a new setting for this as I've done with the other two configs or is there another preferred way to add these types of configurations?

If you could update the README portion of Time Entries to include where this PR would bring us to I'd be good with that. For the config's yes you're doing it right, the only thing you could do on top of that is update the configuration readme portion of the docs (if you run poetry run tap-clickup --about --format=md it'll output markdown for you to copy paste!

Done

@mlahp7 mlahp7 requested a review from visch May 17, 2024 20:51
@mcarriere
Copy link
Contributor

Is there something else we need to improve to get this merged?

# Formatted in ISO 8601, it must now be converted to milliseconds
start_date = datetime.strptime(self.config["time_entry_start_date"], "%Y-%m-%dT%H:%M:%SZ")
# Convert the datetime object to milliseconds
params["start_date"] = int(start_date.timestamp() * 1000)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not use self.get_starting_timestamp(context) here? the start_date config variable is already use in this function so we don't need time_entry_start_date either, unless you think we need two seperate start date windows?

@visch
Copy link
Contributor

visch commented Jul 1, 2024

Is there something else we need to improve to get this merged?

Took another peek and left a review!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants