Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Skeleton Instructions for Development Guide Project Setup #97

Open
wants to merge 16 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 39 additions & 51 deletions project_setup/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,9 @@
We recommend you follow the directions below and use a skeleton for
all new repositories.

The [`skeleton`](scripts/skeleton)
helper tool included in the [`scripts`](scripts) directory can quickly setup
a new local repository. Once you've cloned and configured the repository
to your local machine, it can be published to a repository created on GitHub.
The [`gh-skeleton`][gh-skeleton] GitHub CLI
(`gh`) extension provides the ability to easily start new projects
from the existing library of skeleton repositories.

For repositories created from skeletons, run `setup-env` and
`pre-commit install` to [set up your environment](#set-up-your-environment-and-pre-commit),
Expand All @@ -19,7 +18,7 @@ branch protection - [see our branch protection guide for details](branch-protect
## Contents ##

- [Adding a skeleton configuration to an existing repository](#adding-a-skeleton-configuration-to-an-existing-repository)
- [Using the skeleton tool to start a new repository πŸ’€πŸ› ](#using-the-skeleton-tool-to-start-a-new-repository-)
- [Using the gh-skeleton `gh` extension to start a new repository](#using-the-gh-skeleton-gh-extension-to-start-a-new-repository)
- [Selecting a skeleton](#selecting-a-skeleton)
- [Cloning a selected skeleton](#cloning-a-selected-skeleton)
- [Create and publish the GitHub repository](#create-and-publish-the-github-repository)
Expand All @@ -29,77 +28,64 @@ branch protection - [see our branch protection guide for details](branch-protect
- [Setting up branch protection](#setting-up-branch-protection)
- [Setting up type-specific configuration settings](#setting-up-type-specific-configuration-settings)
- [Setting up Coveralls for Python projects](#setting-up-coveralls-for-python-projects)
- [Ansible requirement file generation tool πŸ§»πŸ› ](#ansible-requirement-file-generation-tool-)
- [Terraform IAM credentials to GitHub secrets πŸ”‘β€πŸ‘‰πŸ€«](#terraform-iam-credentials-to-github-secrets-)
- [Managing SSM parameters from files πŸ—‚πŸ‘‰β˜οΈ](#managing-ssm-parameters-from-files-%EF%B8%8F)
- [Ansible requirement file generation tool](#ansible-requirement-file-generation-tool)
- [Terraform IAM credentials to GitHub secrets](#terraform-iam-credentials-to-github-secrets)
- [Managing SSM parameters from files](#managing-ssm-parameters-from-files)

## Adding a skeleton configuration to an existing repository ##

To skeletonize an existing repository, please see the guide to
[skeletonize an existing repository](skeletonize-existing-repository.md).

## Using the skeleton tool to start a new repository πŸ’€πŸ›  ##
## Using the gh-skeleton `gh` extension to start a new repository ##

We recommend that you start your project from one of the
[skeleton projects](https://github.com/search?q=org%3Acisagov+topic%3Askeleton)
in this organization. The [`skeleton`](scripts/skeleton)
helper tool included in the [`scripts`](scripts) directory can quickly setup
a new local repository. Once you've cloned and configured the repository
to your local machine, you then publish it to a repository created on GitHub.
in this organization. The [`gh-skeleton`][gh-skeleton]
`gh` extension provides the ability to easily start new projects from the existing
library of skeleton repositories.

### Selecting a skeleton ###

First, identify a suitable skeleton project to use as the starting point
for your new repository. For a list of available skeletons, see the
[Skeleton List](skeleton-list.md) or use the following command:

```bash
./skeleton list
```

```console
Available skeletons in cisagov:

skeleton-python-library
A skeleton project for quickly getting a new cisagov Python library started.

skeleton-docker
A skeleton project for quickly getting a new cisagov Docker container started.

skeleton-generic
A generic skeleton project for quickly getting a new cisagov project started.
$ gh skeleton list

skeleton-tf-module
A skeleton project for quickly getting a new cisagov Terraform module started.

skeleton-ansible-role
A skeleton project for quickly getting a new cisagov Ansible role started.

skeleton-ansible-role-with-test-user
A skeleton project for quickly getting a new cisagov Ansible role started when that role requires an AWS test user.

skeleton-packer
A skeleton project for quickly getting a new cisagov packer project started.
Available skeletons in cisagov:

skeleton-aws-lambda
A skeleton project for quickly getting a new cisagov Python AWS Lambda started.
skeleton-ansible-role A skeleton project for quickly getting a new cisagov Ansible role started.
skeleton-ansible-role-with-test-user A skeleton project for quickly getting a new cisagov Ansible role started when that role requires an AWS test user.
skeleton-aws-lambda-python A skeleton project for quickly getting a new Python-based AWS Lambda project started.
skeleton-docker A skeleton project for quickly getting a new cisagov Docker container started.
skeleton-generic A generic skeleton project for quickly getting a new cisagov project started.
skeleton-packer A skeleton project for quickly getting a new cisagov packer project started.
skeleton-python-library A skeleton project for quickly getting a new cisagov Python library started.
skeleton-tf-module A skeleton project for quickly getting a new cisagov Terraform module started.
```

### Cloning a selected skeleton ###

Next, use the `skeleton` tool to clone, rename, and prepare the contents of
your new repository for publication. The tool will print out each command it
is issuing and its result.
Next, use the [`gh-skeleton`][gh-skeleton]
`gh` extension to clone, rename, and prepare the contents of your new repository
for publication. The tool will print out each command it is issuing
and its result.

```console
./skeleton clone [options] <parent-repo-name> <new-repo-name>
$ gh skeleton clone [options] <parent-repo-name> <new-repo-name>
# Expected output:
# Cloning into 'new-repo-name'
```

For example, to create a project based on `skeleton-ansible-role` named
`ansible-role-quantum-rng` in your local `~/projects` directory:

```bash
./skeleton clone --change-dir ~/projects skeleton-ansible-role ansible-role-quantum-rng
```console
$ gh skeleton clone --change-dir ~/projects skeleton-ansible-role ansible-role-quantum-rng
# Expected output:
# Cloning into 'repo-name'
```

This command results in:
Expand Down Expand Up @@ -145,7 +131,7 @@ Use the following commands to push the new repository to github:

## Create and publish the GitHub repository ##

Once the `skeleton` tool has run, the next step is to publish to a GitHub
Once the `gh-skeleton` `gh` extension has run, the next step is to publish to a GitHub
remote repository.

To publish your new repository on GitHub, the remote must already exist.
Expand All @@ -165,7 +151,7 @@ git push --set-upstream origin develop
## Disabling squash merging ##

Click on the settings tab for your new repository and, in the
"Options" section, make sure that "Allow squash merging" is
"Pull Request" section, make sure that "Allow squash merging" is
*unchecked*.

## Set up your environment and pre-commit ##
Expand Down Expand Up @@ -228,7 +214,7 @@ page with `Python set up for Coveralls`. The code block will have an entry for
1. Add a `New repository secret` and name it `COVERALLS_REPO_TOKEN` with the
value from Coveralls.

### Ansible requirement file generation tool πŸ§»πŸ›  ###
### Ansible requirement file generation tool ###

We have a [plethora](https://www.youtube.com/watch?v=zWld721Wk-Q) of
[ansible-roles in our organization](https://github.com/search?q=org%3Acisagov+topic%3Aansible-role+NOT+skeleton+archived%3Afalse).
Expand All @@ -245,7 +231,7 @@ usage of the tool is:
This file will now contain definitions for all the Ansible roles. Edit
the file, and remove any role that will not be required for your project.

### Terraform IAM credentials to GitHub secrets πŸ”‘β€πŸ‘‰πŸ€« ###
### Terraform IAM credentials to GitHub secrets ###

When GitHub Actions workflows require credentials to run we provide them via
[secrets](https://help.github.com/en/actions/configuring-and-managing-workflows/creating-and-storing-encrypted-secrets).
Expand Down Expand Up @@ -277,7 +263,7 @@ terraform-to-secrets 9f4ae878de917c7cf191b9861d3c1cf9224939f7
2020-02-22 15:50:41,036 INFO Success!
```

### Managing SSM parameters from files πŸ—‚πŸ‘‰β˜οΈ ###
### Managing SSM parameters from files ###

Use the [`ssm-param`](scripts/ssm-param) tool to copy files into
[SSM parameters](https://docs.aws.amazon.com/systems-manager/latest/userguide/parameter-store-about-examples.html)
Expand Down Expand Up @@ -324,3 +310,5 @@ ssm-param rm /demo/dh4096.pem /demo/private.pem /demo/public.pem
2019-08-27 16:48:02,417 INFO Deleting parameter '/demo/public.pem' in region 'us-west-1'
2019-08-27 16:48:02,795 INFO Deleting parameter '/demo/public.pem' in region 'us-west-2'
```

[gh-skeleton]: https://github.com/cisagov/gh-skeleton
26 changes: 16 additions & 10 deletions project_setup/skeleton-list.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,23 +4,26 @@ We maintain a number of [`cisagov` skeleton projects](https://github.com/search?
to make starting new repositories quicker and to simplify maintaining
configurations and dependencies across the organization.

The [`skeleton`](scripts/skeleton) helper tool is included in the [`scripts`](scripts)
directory of the [development-guide](https://github.com/cisagov/development-guide)
repository.
The [`gh-skeleton`][gh-skeleton] project is
a GitHub CLI (`gh`) extension that provides the ability to easily start new projects
from an existing library of skeleton repositories.

- To skeletonize an existing repository, please see the guide to
[skeletonize an existing repository](skeletonize-existing-repository.md).
- To skeletonize a new repository, please see the guide to
[start a new repository from a skeleton](new-repository-from-skeleton.md).
[start a new repository from a skeleton](README.md#using-the-gh-skeleton-gh-extension-to-start-a-new-repository).

## List available skeletons with the command-line tool ##

After you've run `setup-env` in this repository, the `skeleton` tool will be
available to you via the command line. To see a list of available skeletons,
use the following command:
After you've installed the [`gh` CLI][gh CLI] followed
by the [`gh-skeleton`][gh-skeleton] extension,
the tool will be available to you via the command line. To see a list of
available skeletons, use the following command:

```bash
skeleton list
```console
$ gh skeleton list
# Expected output:
# Listing of available skeleton repositories
```

## Available skeletons in cisagov ##
Expand All @@ -47,9 +50,12 @@ that role requires an AWS test user.
[`skeleton-packer`](https://github.com/cisagov/skeleton-packer):
A skeleton project for quickly getting a new cisagov packer project started.

[`skeleton-aws-lambda`](https://github.com/cisagov/skeleton-aws-lambda):
[`skeleton-aws-lambda-python`](https://github.com/cisagov/skeleton-aws-lambda-python):
A skeleton project for quickly getting a new cisagov Python AWS Lambda started.

Note: If you discover a new flavor of skeleton that doesn't yet exist, use
`skeleton-generic` to create it, or add an issue to this repository to get
assistance.

[gh-skeleton]: https://github.com/cisagov/gh-skeleton
[gh CLI]: https://github.com/cli/cli
45 changes: 25 additions & 20 deletions project_setup/skeletonize-existing-repository.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,14 @@
# Skeletonize an Existing Repository #

Skeletonizing a repository standardizes our development setup and environment,
and it enables our [Lineage GitHub Action](https://github.com/cisagov/action-lineage/)
to keep the repository updated and standardized.
Skeletonizing a repository standardizes our development setup and environment
and enables our [Lineage GitHub Action](https://github.com/cisagov/action-lineage/)
to keep the repository updated and standardized. The purpose of the
instructions here are to describe how to skeletonize an existing non-skeletonized
repository.

## About ##

Skeleton projects contain [licensing information](LICENSE), as
Skeleton projects contain [licensing information](../LICENSE), as
well as [pre-commit hooks](https://pre-commit.com) and
[GitHub Actions](https://github.com/features/actions) configurations
appropriate for the major languages that we use. This lets us standardize
Expand All @@ -15,24 +17,25 @@ appropriate for the major languages that we use. This lets us standardize

## General overview ##

The general outline of how to add a skeleton to a repository is:
The general outline of how to add a skeleton to an existing repository is:

1. [Add the skeleton as a remote](#add-a-skeleton-as-remote) to the
non-skeletonized repository
1. Pull with `--allow-unrelated-histories`
1. [Fix all the inevitable conflicts](#fix-merge-conflicts)
1. [Add the skeleton as a remote](#add-the-skeleton-as-a-remote) to the
existing non-skeletonized repository
1. Pull with `git pull--allow-unrelated-histories skeleton parent`
1. [Fix merge conflicts](#fix-merge-conflicts)
1. [Review non-conflicting changes](#review-non-conflicting-changes) to
prevent merging destructive upstream changes
1. [Update skeleton's `example` references](#update-skeletons-example-references)
1. [Set up pre-commit](#set-up-your-environment-and-pre-commit-hooks)
1. Fix additional problems that may arise
1. [Make a pull request](#make-a-pull-request)

## Add a skeleton as remote ##
## Add the skeleton as a remote ##

First, decide which of the available skeletons fits your existing repository.
To see a list of available skeletons, use the `skeleton list` command or see
the [list of skeletons](skeleton-list.md).
First, decide which of the available skeletons best fits your
existing non-skeletonized repository. To see a list of available
skeletons, use the `gh skeleton list` command or see
the [list of skeletons][skeleton-list.md].

As an example, we'll be using [`skeleton-python-library`](https://github.com/cisagov/skeleton-python-library)
in this document.
Expand Down Expand Up @@ -65,8 +68,8 @@ pre-commit install

## Fix merge conflicts ##

This merge process will almost certainly fail, resulting in merge conflicts.
The next step is to fix those conflicts and add the files once the fixes are
The merge process will almost certainly fail, resulting in merge conflicts.
The next step is to fix the conflicts and add the files once the fixes are
in place.

```sh
Expand Down Expand Up @@ -98,17 +101,17 @@ from merging in destructive upstream changes.

## Update skeleton's `example` references ##

Some skeletons need additional configuration. For example, with
`skeleton-python-library` and its module structure inside `src/example`.

This step includes such activities as:

- Update `setup.py` with non-example information
- Arrange into appropriate folders, such as `src` and `test`
- Update the `codeowners` to reflect subject matter expertise and
- Update the `.github/CODEOWNERS` file to reflect subject matter expertise and
codebase familiarity
- Aim to have at least two codeowners for every repository

Some skeletons need additional configuration, such as with
`skeleton-python-library` and its module structure inside `src/example`.

## Run pre-commit against existing files ##

The skeleton will bring along with it our standard pre-commit hook
Expand Down Expand Up @@ -150,8 +153,10 @@ repository still passes its test suite.

## Make a pull request ##

Once you've run through the configuration and testing stages, you've probably
Once you've completed the configuration and testing stages, you've probably
accumulated a number of commits on your `skeletonize` branch.

The next step is to [make a pull request](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request)
and have the team perform code reviews.

[skeleton-list.md]: https://github.com/cisagov/development-guide/blob/develop/project_setup/skeleton-list.md#available-skeletons-in-cisagov
Loading