Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: multithreaded bulk import (#1077) #1079

Merged
merged 2 commits into from
Dec 19, 2024
Merged

feat: multithreaded bulk import (#1077) #1079

merged 2 commits into from
Dec 19, 2024

Conversation

sattvikc
Copy link
Collaborator

  • feat: Add BulkImport APIs and cron

  • chore: update pull request template

  • fix: Use the correct tenant config to create the proxy storage

  • fix: PR changes

  • fix: PR changes

  • fix: PR changes

  • fix: PR changes

  • fix: PR changes

  • fix: PR changes

  • fix: Update version

  • fix: PR changes

  • fix: PR changes

  • fix: Rename DeleteBulkImportUser API path

  • fix: disable bulk import for in-memory db

  • fix: a bug with createTotpDevices

  • fix: PR changes

  • feat: Add an api to import user in sync

  • feat: Add an api to get count of bulk import users

  • fix: PR changes

  • fix: Add error codes and plainTextPassword import

  • fix: PR changes

  • feat: multithreaded bulk import

  • fix: changelog update

  • fix: add new test

  • fix: fixing unreliable mutithreaded bulk import with mysql

  • fix: review fixes

  • fix: fixing failing tests

  • feat: bulkimport flow tests

  • feat: bulk import cron starter api

  • fix: tweaking params for faster import

  • fix: tests

  • checkpoint

  • fix: remove vacuuming

  • fix: minor tweaks

  • feat: bulk inserting the bulk migration data

  • fix: fast as a lightning

  • fix: restoring lost method

  • fix: reworked error handling to comform previous approach with messages

  • fix: fixing tests

  • fix: fixing failing tests, changing version

  • chore: update changelog

  • fix: fixing issues and failing tests

  • fix: review changes

  • fix: review fixes, reworking cron start/stop


Summary of change

(A few sentences about this PR)

Related issues

  • Link to issue1 here
  • Link to issue1 here

Test Plan

(Write your test plan here. If you changed any code, please provide us with clear instructions on how you verified your
changes work. Bonus points for screenshots and videos!)

Documentation changes

(If relevant, please create a PR in our docs repo, or create a checklist here
highlighting the necessary changes)

Checklist for important updates

  • Changelog has been updated
    • If there are any db schema changes, mention those changes clearly
  • coreDriverInterfaceSupported.json file has been updated (if needed)
  • pluginInterfaceSupported.json file has been updated (if needed)
  • Changes to the version if needed
    • In build.gradle
  • If added a new paid feature, edit the getPaidFeatureStats function in FeatureFlag.java file
  • Had installed and ran the pre-commit hook
  • If there are new dependencies that have been added in build.gradle, please make sure to add them
    in implementationDependencies.json.
  • Update function getValidFields in io/supertokens/config/CoreConfig.java if new aliases were added for any core
    config (similar to the access_token_signing_key_update_interval config alias).
  • Issue this PR against the latest non released version branch.
    • To know which one it is, run find the latest released tag (git tag) in the format vX.Y.Z, and then find the
      latest branch (git branch --all) whose X.Y is greater than the latest released tag.
    • If no such branch exists, then create one from the latest released branch.
  • If added a foreign key constraint on app_id_to_user_id table, make sure to delete from this table when deleting
    the user as well if deleteUserIdMappingToo is false.

Remaining TODOs for this PR

  • Item1
  • Item2

* feat: Add BulkImport APIs and cron

* chore: update pull request template

* fix: Use the correct tenant config to create the proxy storage

* fix: PR changes

* fix: PR changes

* fix: PR changes

* fix: PR changes

* fix: PR changes

* fix: PR changes

* fix: Update version

* fix: PR changes

* fix: PR changes

* fix: Rename DeleteBulkImportUser API path

* fix: disable bulk import for in-memory db

* fix: a bug with createTotpDevices

* fix: PR changes

* feat: Add an api to import user in sync

* feat: Add an api to get count of bulk import users

* fix: PR changes

* fix: Add error codes and plainTextPassword import

* fix: PR changes

* feat: multithreaded bulk import

* fix: changelog update

* fix: add new test

* fix: fixing unreliable mutithreaded bulk import with mysql

* fix: review fixes

* fix: fixing failing tests

* feat: bulkimport flow tests

* feat: bulk import cron starter api

* fix: tweaking params for faster import

* fix: tests

* checkpoint

* fix: remove vacuuming

* fix: minor tweaks

* feat: bulk inserting the bulk migration data

* fix: fast as a lightning

* fix: restoring lost method

* fix: reworked error handling to comform previous approach with messages

* fix: fixing tests

* fix: fixing failing tests, changing version

* chore: update changelog

* fix: fixing issues and failing tests

* fix: review changes

* fix: review fixes, reworking cron start/stop

---------

Co-authored-by: Ankit Tiwari <[email protected]>
@sattvikc sattvikc changed the base branch from master to 9.4 December 19, 2024 09:17
@sattvikc sattvikc merged commit b6ee5c5 into 9.4 Dec 19, 2024
2 of 3 checks passed
@sattvikc sattvikc deleted the feat/bulk-import-base branch December 19, 2024 09:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants