Code template research

[ian-r-rose]

Code templates tools

Motivation

We are going to be doing a number of client projects with a similar stack, and plan to hand off these projects to our clients at the end of the project. The intention is to keep code for these projects in GitHub, and the code will likely involve:

1. A dbt project
2. CI/CD
3. documentation
4. (sometimes) An orchestration project
5. (sometimes) infrastructure as code

We would like to have a project template that integrates with a generation tool to rapidly make a project skeleton with the above features. Ideally, bootstrapping a new project will take under an hour.

Requirements

Some tooling requirements:

• An easy-to-use CLI for answering things like project name, repo location, etc.
• Jinja-like templating language
• Language-agnostic

Some nice-to-haves:

• Template directly from GitHub repo
• Migrations

[ian-r-rose]

Project comparison

	Cookiecutter	Cruft	Copier	Yeoman
Language	Python	Python	Python	Javascript
Configuration	JSON	JSON	Yaml	Javscript
Migrations	No	Yes	Yes	No
Templating	Jinja	Jinja	Jinja	EJS

Cookiecutter

Cookiecutter is the most popular templating tool for projects in the Python ecosystem, though it's pretty good at being a cross-language tool. It's got a no-frills CLI which asks you questions (like project name, email addresses, etc) and fills them into the appropriate places.

The main downside is that it's a one-time-use tool for a project. If the template evolves (as they tend to do), projects that were previously created using it do not get the updates (i.e., migrations). This is often okay, as projects don't necessarily need those updates, but having an option for them is not a terrible thing.

Cruft

Cruft is an interesting project in that it is a backwards-compatible extension to cookiecutter. Any cruft template is a cookiecutter template, and any cookiecutter template can be converted to a cruft template. The main goal is to bolt on migrations to a cookiecutter template so that you can update projects after the initial templating is done. Since cookiecutter is so prevalent, adopting cruft seems fairly low-risk.

Copier

Copier is a newer templating project that seems to have similar goals to cruft: allowing you to make updates to a project after the initial templating. There are some other minor differences like YAML instead of JSON for configuration, but they otherwise seem fairly similar based on the docs.

Yeoman

Yeoman is a templating tool for web applications. It seems to be a perfectly nice tool, but I don't think it's a great fit for this project. It is more tied to the npm ecosystem than the python one, and more of our tools are likely to be closer to python (dbt is a python project, as are most orchestration tools). It also requires writing javascript for configuration, which is likely a bit less accessible to our target audience.

[ian-r-rose]

Evaluation

I don't have enough information to make a call as to which tool suits our needs right now, so plan to do a bit of testing. I like that cruft is backwards-compatible with cookiecutter, but it really comes down to how easy the resulting projects are to create and maintain. Some things I'd like to look out for:

• Our template may have some jinja that we don't want to template (e.g., dbt models have jinja). How easy is it to mark some things as templated and others as not?
• Is it possible to mark some directories or files as optional depending on user inputs? e.g., if a project doesn't need an orchestration project, can we skip it?
• How easy is it to apply a migration?
• How obnoxious are whatever sidecar files we have tracking migrations?
• How pleasant is the CLI?

[ian-r-rose]

A couple of notes base on more reading cookiecutter and copier docs:

Our template may have some jinja that we don't want to template (e.g., dbt models have jinja). How easy is it to mark some things as templated and others as not?

Copier actually has a bit better of a story here. Just don't suffix the files with .jinja and we're good to go. Only .jinja files are rendered. Maybe looks a bit weird, but more explicit than having an excluded files configuration option like cookiecutter.

Is it possible to mark some directories or files as optional depending on user inputs? e.g., if a project doesn't need an orchestration project, can we skip it?

Again, I think copier has a better story here. Just put a literal jinja if statement in a file/directory name to determine whether it is included. Cookiecutter has a somewhat ugly hook apparatus for this.

How obnoxious are whatever sidecar files we have tracking migrations?

Both cruft and copier have similar files tracking history, and cookiecutter doesn't do migrations.

The above make me want to try copier for my initial pass at this. If it winds up being painful, falling back to old and reliable cookiecutter shouldn't be too hard (jinja is jinja, after all).

[ian-r-rose]

Closing as complete!