DB Migrations and Backward Compatibility Issues

[lyuyangh]

This thread keeps track of everything related to DB migrations and backward compatibility issues.

[lyuyangh]

PR #335

• System outputs are now stored in Cloud Storage instead of MongoDB
• The PR is backward compatible so no DB migration is required.
• (removed in remove outdated code #422 ) code & code were added for backward compatibility reasons. They can be removed when we regenerate all system analyses next time.

[lyuyangh]

PR #365

• DB migration was performed because I couldn't make the code backward compatible when I upgraded the SDK to v0.11.2.

[lyuyangh]

PR #434 #435 #436

• Migration was performed for all three PRs in one go. Please refer to the PRs to see what was changed.

[OscarWang114]

PR #366

• DB migration was performed to create a user metadata collection and to replace all system's creator fields from emails to user ids.

[lyuyangh]

PR #439 #459

• Migration was performed for dev.
• Prod DB migration was performed on Oct 31, 2022.

[lyuyangh]

PR #490 #492

• Migration for both dev and prod performed on Nov 4, 2022.

[lyuyangh]

PR #493 (#494 also related)

• Migration for both dev and prod performed on Nov 8, 2022.
• sdk_version_used is set to "old" for all systems so all previous systems will get re-analyzed when a user clicks "Analysis".
• This migration also partially addresses an issue with custom analyses. custom_features and custom_analyses come from two sources: the dataset (if it is a datalab dataset) or the output file (provided by the user). Because of how the SDK is implemented, the latter always overrides the former if it is present. This migration brings back all custom_analyses of datasets even if a user-defined custom_analyses is present. The code still doesn't work though so new systems still suffer from this issue. When the code is fixed, I can run the migration again to fix all the exiting systems.