MDSA service templates

[ian-r-rose]

In order to have a repeatable process for MDSA DIF projects, we should generate some useful templates for projects

Project management templates

There are several points during a project lifecycle that could benefit from having some templates for helping manage the project. These might be coda, google docs, or airtable:

• @jasonlally @britt-allen brainstorm jam
• Britt to fill out the discovery page
• Britt to fill out the project charter page
• Britt to create a table to capture data and systems information
• Britt to create tools checklist
• Hand-off plans - steps for making sure that code, cloud infrastructure, and processes are transitioned to the partner entity.
• Project assessment template - identify whether a partner is a good fit for our stack and maturity.
• Create MDSA notes template with prompt questions for office hours #24

Code templates

Depending upon the project, it will often be useful to have some code template projects to quickly stand-up MDSA projects in GitHub. I've used cookiecutter for this in the past, though other approaches could work as well. These projects would allow us to quickly get

• Basic directory structure for a project
• testing framework boilerplate
• pre-commit setups for linting and code-quality
• Basic configuration files
• Documentation skeletons.

There are a number of ways we could structure MDSA project code strucutre, either as separate repos or as a monorepo that addresses different components of a project. In a lot of ways, I'm thinking that a monorepo might be easiest from an organizational perspective (CalData currently has separate repos, but I might want to merge them). Whether we organize as a monorepo or not, the templates might include:

• A dbt project for data modeling
• An orchestration project (Airflow or equivalent, cf Investigate orchestration options #4) for data loading and scheduling of jobs (Not planned right now)
• A terraform project for standing up cloud resources in a replicable and auditable way. (Not planned right now)

[ian-r-rose]

Notes from DSE sync

Code templates

• We looked at cookiecutter briefly. We may want to do a light research project / due-diligence into what other projects use. e.g., what do people use with terraform?
• Do we actually want to throw terraform at people? It may be not something we want to ask clients to maintain.
• One possiblility
  • Start with a more manual process, e.g., here is a list of cloud resources to create
  • Once we start feeling the pain of that, start creating something like terraform project
• We should do some assessment of where the organization is when deciding which parts of the stack to deploy
• Research question: can cookiecutter auto-delete parts of a repo that aren't needed (e.g., delete terraform if not required for a project)
• As part of onboarding to AWS, can we dog-food some of these ideas?
• Start with a project template repo that is mostly documentation, e.g. a collection of markdown documents describing the components of the modern data stack. Really nice example from the GitLab data team: https://about.gitlab.com/handbook/business-technology/data-team/platform/ . From there we can identify which things we want to automate.

Project management templates

• We should have some sort of capabilities assessment template to help identify whether a project meets that minimum needs or is a good match.
• This can be informed by reading through actual proposals
• For the first go-round, let's try to find people who are comfortable trying the stack we have.
• @jasonlally will take a first pass at creating an onboarding template in Coda.

[ian-r-rose]

cc @jasonlally Let's spin the checklist items into specific issues (by Wednesday?)

[ian-r-rose]

Closing as complete!