Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create mini datalake database #110

Merged
merged 13 commits into from
Mar 21, 2024
Merged

Create mini datalake database #110

merged 13 commits into from
Mar 21, 2024

Conversation

DJensen94
Copy link
Contributor

@DJensen94 DJensen94 commented Mar 19, 2024

Create models and handler to create mini datalake to store org and scan data

🗣 Description

This PR creates the models and code to generate the Mini Data Lake database, that will store data from the various teams and will eventually serve the Crossfeed application database data.

💭 Motivation and context

In order to expand Crossfeed's capabilities the database needs to be expanded to hold different vulnerability and scan types and hold data specific to each source, this new schema will allow us to house data from any source as it is expanded upon.

The schema will likely have some adjustments as time goes on and new sources are added.

✅ Pre-approval checklist

  • This PR has an informative and human-readable title.
  • Changes are limited to a single goal - eschew scope creep!
  • All future TODOs are captured in issues, which are referenced
    in code comments.
  • All relevant type-of-change labels have been added.
  • I have read the CONTRIBUTING document.
  • These code changes follow cisagov code standards.
  • All relevant repo and/or project documentation has been updated
    to reflect the changes in this PR.
  • Tests have been added and/or modified to cover the changes in this PR.
  • All new and existing tests pass.

✅ Pre-merge checklist

  • Revert dependencies to default branches.
  • Finalize version.

✅ Post-merge checklist

  • Create a release.

Create models and handler to create mini datalake to store org and scan data
backend/src/tools/run-syncmdl.ts Dismissed Show dismissed Hide dismissed
update some of the default parameters of the columns in the new database
rerun the backend linter
run lint on package-lock.json
run pretty-format-json pre-commit
update the cidr columns primary key to be a UUID
Connect new variables to AWS
Copy link
Collaborator

@cduhn17 cduhn17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comments

Alphabetize imports in connections and remoe blank space
Copy link
Collaborator

@cduhn17 cduhn17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking a look at those few comments.

Copy link
Contributor

@aloftus23 aloftus23 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, just make sure you check everything in the pre-approval checklist

Remove TODO comments
@schmelz21 schmelz21 added the improvement This issue or pull request will add or improve functionality, maintainability, or ease of use label Mar 20, 2024
change serial PKs to UUIDs
@schmelz21
Copy link
Collaborator

@dav3r or @jsf9k - Did either of you want/can take a look at this one? We triggered a PR review with the .yml change.

@schmelz21 schmelz21 merged commit ddcf403 into develop Mar 21, 2024
15 of 19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement This issue or pull request will add or improve functionality, maintainability, or ease of use
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants