Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: deprecate mixpanel /track endpoint #2833

Merged
merged 6 commits into from
Apr 15, 2024

Conversation

Gauravudia
Copy link
Contributor

@Gauravudia Gauravudia commented Nov 16, 2023

Description of the change

Resolves INT-652

  • deprecated the /track endpoint and implemented track event calls through the existing /import endpoint. This change offers a higher batch limit, supports gzip compression, enables the ingestion of historical data, and provides several other enhancements. For more details, refer to mixpanel /track vs /import comparison doc.
  • After the deprecation of the /track endpoint, all calls will be sent via the /import endpoint. As a result, in the identify call, we used to perform the merging of known (userId) and anonymous (anonymousId) users through an additional call using the /import endpoint. The earlier decision to use the /track versus /import endpoint was based on configuration. However, now that all calls will be sent using the /import endpoint, Customers should expect additional merge calls in their profile feed after this change.
  • Removing authentication with project secret for /import API as it is in the deprecating stage at Mixpanel, Instead we can use project token (which is a required field in webapp)

Type of change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

Related issues

Fix #1

Checklists

Development

  • Lint rules pass locally
  • The code changed/added as part of this pull request has been covered with tests
  • All tests related to the changed code pass in development

Code review

  • This pull request has a descriptive title and information useful to a reviewer. There may be a screenshot or screencast attached
  • "Ready for review" label attached to the PR and reviewers mentioned in a comment
  • Changes have been reviewed by at least one other engineer
  • Issue from task tracker has a link to this pull request

Summary by CodeRabbit

  • Refactor
    • Updated event tracking logic to enhance performance and accuracy.
    • Improved handling of credentials for increased security.
    • Refined event batching process for more efficient data processing.
  • Tests
    • Adjusted tests to align with the new event tracking and batching logic.
  • Chores
    • Updated internal tokens in tests for consistency and reliability.

@Gauravudia Gauravudia self-assigned this Nov 16, 2023
@Gauravudia Gauravudia requested review from a team and sivashanmukh as code owners November 16, 2023 16:30
Copy link

codecov bot commented Nov 16, 2023

Codecov Report

Attention: Patch coverage is 93.33333% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 87.82%. Comparing base (4ea55d1) to head (e921122).
Report is 59 commits behind head on develop.

Files Patch % Lines
src/v0/destinations/mp/transform.js 92.85% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #2833      +/-   ##
===========================================
+ Coverage    87.71%   87.82%   +0.11%     
===========================================
  Files          550      556       +6     
  Lines        29616    29926     +310     
  Branches      7060     7146      +86     
===========================================
+ Hits         25977    26283     +306     
- Misses        3303     3338      +35     
+ Partials       336      305      -31     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

100.0% 100.0% Coverage
0.0% 0.0% Duplication

@ItsSudip
Copy link
Member

ItsSudip commented Nov 17, 2023

Replaced the authentication method from project secret to project token. **(Not a breaking change since project token is a required field in webapp).

Why you have changed to project token? For /import endpoint we need project secret.

@Gauravudia
Copy link
Contributor Author

Gauravudia commented Nov 21, 2023

Replaced the authentication method from project secret to project token. **(Not a breaking change since project token is a required field in webapp).

Why you have changed to project token? For /import endpoint we need project secret.

Now /import supports project token, Mixpanel doc is not updated yet but i confirmed with mixpanel team and tested through postman also.
This is the reply from mixpanel team
Screenshot 2023-11-21 at 5 19 51 PM

@devops-github-rudderstack
Copy link
Contributor

This PR is considered to be stale. It has been open for 20 days with no further activity thus it is going to be closed in 7 days. To avoid such a case please consider removing the stale label manually or add a comment to the PR.

Copy link
Contributor

coderabbitai bot commented Jan 27, 2024

Important

Auto Review Skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository.

To trigger a single review, invoke the @coderabbitai review command.

Walkthrough

The changes involve significant adjustments to the Mixpanel integration within a codebase, specifically focusing on event tracking and batching logic. The removal of the TRACK_MAX_BATCH_SIZE constant indicates a shift in how batch sizes are managed, possibly moving towards a more dynamic or configuration-driven approach. Adjustments in credential handling and event processing suggest an effort to streamline operations and perhaps enhance security or efficiency. The transition from /track to /import endpoints for event tracking aligns with changes in Mixpanel's API or a strategic shift in data handling practices.

Changes

File Path Change Summary
src/v0/.../config.js Removed TRACK_MAX_BATCH_SIZE constant.
src/v0/.../transform.js Removed TRACK_MAX_BATCH_SIZE, updated setImportCredentials, modified event processing logic.
src/v0/.../util.js Updated event grouping logic, removed track property.
.../destinations/mp/util.test.js Updated tests from trackEvents to importEvents and endpoint URLs.
test/.../common.ts Updated token value within sampleDestination object.
test/.../router/data.ts Standardized export syntax for data.

Assessment against linked issues

Objective Addressed Explanation
Checking in latest GATransform artifacts under dipanjan branch of rudder-transformer repo (#1)

The primary objective from the linked issue is the integration of the latest GATransform artifacts into the master branch of the rudder-transformer repository. While the summary of changes does not explicitly mention "GATransform" artifacts, the detailed adjustments to event tracking and batching, as well as the shift in handling credentials and endpoints, suggest a significant update that could be part of integrating such artifacts. The changes align with an effort to enhance or update the data processing capabilities, which would be in line with the objective of incorporating new or updated artifacts into the repository.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit-tests for this file.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit tests for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository from git and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit tests.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • The JSON schema for the configuration file is available here.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.v2.json

CodeRabbit Discord Community

Join our Discord Community to get help, request features, and share feedback.

@devops-github-rudderstack
Copy link
Contributor

1 similar comment
@devops-github-rudderstack
Copy link
Contributor

@Gauravudia
Copy link
Contributor Author

@coderabbitai review

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between db2c170 and 5421ec0.
Files selected for processing (7)
  • src/v0/destinations/mp/config.js (2 hunks)
  • src/v0/destinations/mp/transform.js (4 hunks)
  • src/v0/destinations/mp/util.js (2 hunks)
  • src/v0/destinations/mp/util.test.js (10 hunks)
  • test/integrations/destinations/mp/common.ts (1 hunks)
  • test/integrations/destinations/mp/processor/data.ts (78 hunks)
  • test/integrations/destinations/mp/router/data.ts (6 hunks)
Files not summarized due to errors (1)
  • test/integrations/destinations/mp/processor/data.ts: Error: Message exceeds token limit
Files skipped from review due to trivial changes (1)
  • src/v0/destinations/mp/util.js
Additional comments: 10
test/integrations/destinations/mp/common.ts (1)
  • 8-8: The token value in the sampleDestination object has been updated to 'test_api_token'. Ensure this new token value aligns with the expected test configurations and authentication requirements for the /import endpoint.
src/v0/destinations/mp/transform.js (3)
  • 49-60: The setImportCredentials function has been updated to support the new authentication method using a project token. Ensure that the credentials are correctly encoded in Base64 and that the Authorization header is properly formatted.
  • 69-87: The responseBuilderSimple function has been modified to support the new /import endpoint. Verify that the event timestamp validation (within the last 5 years) and the setting of request parameters are correctly implemented for all event types.
  • 482-488: The batching logic in processRouterDest has been updated to accommodate the new /import endpoint. Ensure that the batching of events is correctly handled for engageEvents, groupsEvents, and importEvents, and that the batch size limits are respected.
src/v0/destinations/mp/util.test.js (2)
  • 139-148: > 📝 NOTE

This review was outside the diff hunks and was mapped to the diff hunk with the greatest overlap. Original lines [81-145]

The occurrences of trackEvents have been replaced with importEvents, and the endpoint URLs have been updated from /track to /import. Confirm that these changes accurately reflect the new event tracking methodology and that the tests are correctly updated to target the /import endpoint.

  • 292-297: > 📝 NOTE

This review was outside the diff hunks and was mapped to the diff hunk with the greatest overlap. Original lines [268-330]

The combineBatchRequestsWithSameJobIds function test cases have been updated to use the /import endpoint. Ensure that the logic for combining batch requests with the same job IDs is correctly implemented and that the tests accurately reflect the expected behavior.

test/integrations/destinations/mp/router/data.ts (4)
  • 442-448: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [1-1]

The import statement and the usage of overrideDestination and sampleDestination remain unchanged, ensuring that the test setup utilizes consistent destination configuration across tests.

  • 445-445: The Authorization header in the expected output for Mixpanel API calls correctly uses a base64-encoded test_api_token, aligning with the PR's objective to shift from using a project secret to a project token for authentication. This change is consistent across all test cases, ensuring that the new authentication method is thoroughly tested.

Also applies to: 511-511, 578-578, 1164-1164, 1229-1229, 1295-1295

  • 442-448: > 📝 NOTE

This review was outside the diff hunks and was mapped to the diff hunk with the greatest overlap. Original lines [2-1369]

The test cases provided in the file cover a comprehensive range of scenarios, including different types of calls and settings, ensuring thorough testing of the integration with the Mixpanel destination. The detailed configurations, including the shift to the /import endpoint and the use of project tokens for authentication, align with the PR's objectives and demonstrate a thoughtful approach to testing the changes.

  • 442-448: > 📝 NOTE

This review was outside the diff hunks, and no overlapping diff hunk was found. Original lines [1360-1369]

The test case validating the error handling for missing group keys, resulting in a 400 status code, effectively ensures that the integration correctly identifies and reports configuration errors. This test case is crucial for verifying the robustness of the error handling mechanisms in the integration.

@Gauravudia Gauravudia requested a review from ItsSudip January 29, 2024 05:16
ItsSudip
ItsSudip previously approved these changes Jan 30, 2024
@devops-github-rudderstack
Copy link
Contributor

Copy link

@devops-github-rudderstack
Copy link
Contributor

This PR is considered to be stale. It has been open for 20 days with no further activity thus it is going to be closed in 7 days. To avoid such a case please consider removing the stale label manually or add a comment to the PR.

@Gauravudia Gauravudia removed the Stale label Mar 27, 2024
@Gauravudia Gauravudia requested a review from ItsSudip March 27, 2024 11:42
ItsSudip
ItsSudip previously approved these changes Apr 3, 2024
shrouti1507
shrouti1507 previously approved these changes Apr 4, 2024
@Gauravudia Gauravudia requested a review from ItsSudip April 5, 2024 05:06
ItsSudip
ItsSudip previously approved these changes Apr 5, 2024
@Gauravudia Gauravudia requested a review from krishna2020 April 9, 2024 04:42
@Gauravudia Gauravudia requested a review from krishna2020 April 15, 2024 04:45
@Gauravudia Gauravudia dismissed stale reviews from ItsSudip and shrouti1507 via e921122 April 15, 2024 05:36
Copy link

@Gauravudia Gauravudia merged commit 1bfcd27 into develop Apr 15, 2024
16 checks passed
@Gauravudia Gauravudia deleted the refactor/mixpanel-deprecate-track-endpoint branch April 15, 2024 06:32
Gauravudia added a commit that referenced this pull request Apr 18, 2024
@Gauravudia Gauravudia restored the refactor/mixpanel-deprecate-track-endpoint branch May 21, 2024 13:34
@Gauravudia Gauravudia deleted the refactor/mixpanel-deprecate-track-endpoint branch May 21, 2024 13:48
Gauravudia added a commit that referenced this pull request May 21, 2024
* refactor: deprecate mixpanel /track endpoint

* fix: used token in place of apiSecret for merge call

* fix: no-case-declarations
Gauravudia added a commit that referenced this pull request May 22, 2024
* refactor: deprecate mixpanel /track endpoint (#2833)

* refactor: deprecate mixpanel /track endpoint

* fix: used token in place of apiSecret for merge call

* fix: no-case-declarations

* refactor: responseBuilderSimple
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants