Skip to content

Commit

Permalink
Merge pull request #14 from ukgovdatascience/make_secrets_mandatory
Browse files Browse the repository at this point in the history
Make `.secrets` creation mandatory
  • Loading branch information
ESKYoung authored Jan 15, 2021
2 parents f89fe5b + d0ec7c6 commit a497242
Show file tree
Hide file tree
Showing 23 changed files with 143 additions and 111 deletions.
6 changes: 3 additions & 3 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@
# See https://pre-commit.com/hooks.html for more hooks
repos:
- repo: https://gitlab.com/pycqa/flake8
rev: master
rev: 3.8.4
hooks:
- id: flake8
args: ["hooks", "tests", "{{ cookiecutter.repo_name }}/src"]
- repo: https://github.com/Yelp/detect-secrets
rev: v0.13.1
rev: v0.14.3
hooks:
- id: detect-secrets
args: ["--baseline", ".secrets.baseline"]
Expand All @@ -21,7 +21,7 @@ repos:
- --pin-patterns
- "[keep_output]"
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.4.0
rev: v3.4.0
hooks:
- id: check-added-large-files
args: ["--maxkb=5120"]
Expand Down
5 changes: 3 additions & 2 deletions .secrets.baseline
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
{
"custom_plugin_paths": [],
"exclude": {
"files": null,
"lines": null
},
"generated_at": "2020-10-28T08:52:26Z",
"generated_at": "2021-01-11T09:41:31Z",
"plugins_used": [
{
"name": "AWSKeyDetector"
Expand Down Expand Up @@ -58,7 +59,7 @@
}
],
"results": {},
"version": "0.13.1",
"version": "0.14.3",
"word_list": {
"file": null,
"hash": null
Expand Down
7 changes: 4 additions & 3 deletions CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,14 @@ Where this Code of Conduct says:

- "Project", we mean this `govcookiecutter` GitHub repository;
- "Maintainer", we mean the `ukgovdatascience` organisation owners; and
- "Leadership", we mean both `ukgovdatascience` organisation owners, line managers, and other leadership within GDS.
- "Leadership", we mean both `ukgovdatascience` organisation owners, line managers, and other leadership within the
Government Digital Service.

### Our Pledge

In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to make
participation in our project and our community a harassment-free experience for everyone, regardless of age, body size,
disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education,
participation in our project, and our community a harassment-free experience for everyone, regardless of age, body
size, disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education,
socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.

### Our Standards
Expand Down
16 changes: 11 additions & 5 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,9 @@ make requirements
It is better to use the above make command, rather than `pip install -r requirements.txt` and `pre-commit install`, as
the command will ensure your pre-commit hooks are up-to-date with any changes made.

The pre-commit hooks are a security feature to ensure no secrets, large data files, and output cells of Jupyter
notebooks are accidentally committed into the repository. For more information about the pre-commit hooks used in this
repository, see the [documentation][docs-pre-commit-hooks].
The pre-commit hooks are a security feature to ensure no secrets<sup>[1](#footnote-1)</sup>, large data files, and
Jupyter notebook outputs are accidentally committed into the repository. For more information about the pre-commit hooks used
in this repository, see the [documentation][docs-pre-commit-hooks].

## Code conventions

Expand Down Expand Up @@ -64,7 +64,7 @@ Use a local link to reference the [`README.md`](./README.md) file, but an extern
[gov-uk]: https://www.gov.uk/
```

We also try and wrap Markdown to a line length of 120 characters, but this is not strictly enforced in all cases, for
We also try to wrap Markdown to a line length of 120 characters, but this is not strictly enforced in all cases, for
example with long hyperlinks.

## Testing
Expand All @@ -83,7 +83,7 @@ Code coverage of Python scripts is measured using the [`coverage`][coverage] Pyt
found in `.coveragerc`. Note coverage only extends to Python scripts in the `hooks`, and
`{{ cookiecutter.repo_name }}/src` folders.

To run code coverage, and view it as a HTML report, execute the following commands in your terminal:
To run code coverage, and view it as an HTML report, execute the following commands in your terminal:

```shell
coverage run -m pytest
Expand All @@ -100,9 +100,15 @@ unless it's more appropriate to store it elsewhere, like this file.
Further information on how to write Sphinx documentation, and how to build it into a searchable website can be found
[here][docs-write-sphinx-documentation].

---

<a name="footnote-1">[1]</a>: Only secrets of specific patterns are detected by the pre-commit hooks. See
[here][docs-pre-commit-hooks-secrets-definition] for further details.

[code-of-conduct]: ./CODE_OF_CONDUCT.md
[coverage]: https://coverage.readthedocs.io/
[docs-pre-commit-hooks]: ./%7B%7B%20cookiecutter.repo_name%20%7D%7D/docs/contributor_guide/pre_commit_hooks.md
[docs-pre-commit-hooks-secrets-definition]: ./%7B%7B%20cookiecutter.repo_name%20%7D%7D/docs/contributor_guide/pre_commit_hooks.md#definition-of-a-secret-according-to-detect-secrets
[docs-updating-gitignore]: ./%7B%7B%20cookiecutter.repo_name%20%7D%7D/docs/contributor_guide/updating_gitignore.md
[docs-write-sphinx-documentation]: ./%7B%7B%20cookiecutter.repo_name%20%7D%7D/docs/contributor_guide/writing_sphinx_documentation.md
[email]: mailto:[email protected]
Expand Down
10 changes: 9 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,12 @@
.PHONY: docs docs_check_external_links example example_with_options help prepare_docs_folder prepare_example_folder requirements
.PHONY:
docs
docs_check_external_links
example
example_with_options
help
prepare_docs_folder
prepare_example_folder
requirements

.DEFAULT_GOAL := help

Expand Down
13 changes: 10 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,15 +27,16 @@ We use nudges, such as checklists in pull/merge requests, to minimise the burden
complete AQA checks. This results in faster iterative development and deployment, whilst ensuring HM Government-wide
standards on assurance are met.

We have also included [pre-commit hooks][pre-commit] to prevent accidental committing of secrets, large data files, and
Jupyter notebook outputs for security purposes.
We have also included [pre-commit hooks][pre-commit] to prevent accidental committing of
secrets<sup>[1](#footnote-1)</sup>, large data files, and Jupyter notebook outputs for security purposes.

## Getting started with `govcookiecutter` for your projects

> ⚠️ Only Unix-based systems (macOS, Linux, ...), and Python projects for GitHub or GitLab are supported — feel free to
> [contribute](#contributing) to support other operating systems/programming languages!
To use this template to start your next coding project, make sure your system meets the [requirements](#requirements-to-create-a-cookiecutter-template).
To use this template to start your next coding project, make sure your system meets the
[requirements](#requirements-to-create-a-cookiecutter-template).

Once you're all set up, open your terminal, navigate to the directory where you want your new repository to exist, and
run the following commands:
Expand Down Expand Up @@ -117,8 +118,14 @@ If you want to help us build, and improve `govcookiecutter`, view our [contribut
This template is based off the [DrivenData Cookiecutter Data Science][drivendata] project, especially around the
template data and src folder structures, and the `make help` commands in the Makefiles.

---

<a name="footnote-1">[1]</a>: Only secrets of specific patterns are detected by the pre-commit hooks. See
[here][docs-pre-commit-hooks-secrets-definition] for further details.

[aqua-book]: https://www.gov.uk/government/publications/the-aqua-book-guidance-on-producing-quality-analysis-for-government
[aqua-book-resources]: https://www.gov.uk/government/collections/aqua-book-resources
[docs-pre-commit-hooks-secrets-definition]: ./%7B%7B%20cookiecutter.repo_name%20%7D%7D/docs/contributor_guide/pre_commit_hooks.md#definition-of-a-secret-according-to-detect-secrets
[contributing]: ./CONTRIBUTING.md
[cookiecutter]: https://github.com/cookiecutter/cookiecutter
[drivendata]: http://drivendata.github.io/cookiecutter-data-science/
Expand Down
3 changes: 1 addition & 2 deletions cookiecutter.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,5 @@
"project_name": "Your new project name",
"repo_name": "{{ cookiecutter.project_name.lower().replace(' ', '-') }}",
"overview": "Brief overview of your project.",
"project_version": "0.0.1",
"create_secrets_file": ["No", "Yes"]
"project_version": "0.0.1"
}
39 changes: 0 additions & 39 deletions hooks/post_gen_project.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
from shutil import rmtree
from textwrap import dedent
import os

# Define the folder path to the 'docs/aqa_frameworks' folder, and the future `docs/aqa` folder
Expand All @@ -17,41 +16,6 @@
}


def create_secrets_file(user_option: str) -> None:
"""Create a .secrets file depending on a user option.
Args:
user_option (str): User option.
Returns:
A .secrets file created in the top-level if user_option is 'Yes', otherwise nothing.
"""

# Check if `user_option` is 'Yes'
if user_option == "Yes":

# Define a header for the `.secrets file`
secrets_file = """
# Secrets go here, and can be read in by Python using `os.getenv`:
#
# --------------------------------------------------------
# import os
#
# EXAMPLE_SECRET = os.getenv("EXAMPLE_SECRET")
# --------------------------------------------------------
#
# This file is NOT version-controlled!
# Google Cloud authentication credentials
export GOOGLE_APPLICATION_CREDENTIALS=""
"""

# Write the `.secrets` file out
with open(".secrets", "w") as f:
f.write(dedent(secrets_file).lstrip())


def select_dept_aqa_framework(user_option: str, default_option: str = "GDS") -> None:
"""Create analytical quality assurance (AQA) documents for a specific HM Government department.
Expand Down Expand Up @@ -111,9 +75,6 @@ def select_pull_merge_request_template(user_option: str, repo_host: str, default

if __name__ == "__main__":

# Create a .secrets file, if requested
create_secrets_file("{{ cookiecutter.create_secrets_file }}")

# Select the appropriate AQA framework
select_dept_aqa_framework("{{ cookiecutter.departmental_aqa_framework }}")

Expand Down
2 changes: 1 addition & 1 deletion tests/test_envrc.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# Define the cookiecutter directory
DIR_COOKIECUTTER = "{{ cookiecutter.repo_name }}"

# Initialise an empty list to store the expected filepaths
# Initialise an empty list to store the expected file paths
DIRS_EXPECTED = []

# Get all the directories within `DIR_COOKIECUTTER`, including all sub-folders (unless they are `docs`, or
Expand Down
12 changes: 6 additions & 6 deletions {{ cookiecutter.repo_name }}/.envrc
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
# --------------------------------------------------------
# import os
#
# # Example variable
# EXAMPLE_VARIABLE = os.getenv("EXAMPLE_VARIABLE")
# --------------------------------------------------------
#
Expand All @@ -12,16 +13,15 @@
# DO NOT STORE SECRETS HERE - this file is version-controlled! You should store secrets in a `.secrets` file, which is
# not version-controlled - this can then be sourced here, using `source_env ".secrets"`.

# Extract the variables to `.env` if required. Note `.env` is NOT version-controlled{% if cookiecutter.create_secrets_file == "Yes" %}, so `.secrets` will not be
# committed{% endif %}
sed -n 's/^export \(.*\)$/\1/p' .envrc {% if cookiecutter.create_secrets_file == "Yes" %}.secrets {% endif %}| sed -e 's?$(pwd)?'"$(pwd)"'?g' > .env
{% if cookiecutter.create_secrets_file == "No" %}# sed -n 's/^export \(.*\)$/\1/p' .envrc .secrets | sed -e 's?$(pwd)?'"$(pwd)"'?g' > .env # use this line instead if you have a `.secrets` file
{% endif %}
# Extract the variables to `.env` if required. Note `.env` is NOT version-controlled, so `.secrets` will not be
# committed
sed -n 's/^export \(.*\)$/\1/p' .envrc .secrets | sed -e 's?$(pwd)?'"$(pwd)"'?g' > .env

# Add the working directory to PYTHONPATH; allows Jupyter notebooks in the `notebooks` folder to import `src`
export PYTHONPATH="$PYTHONPATH:$(pwd)"

# Import secrets from an untracked file `.secrets`
{% if cookiecutter.create_secrets_file == "No" %}# {% endif %}source_env ".secrets"
source_env ".secrets"

# Add environment variables for the `data` directories
export DIR_DATA=$(pwd)/data
Expand Down
6 changes: 3 additions & 3 deletions {{ cookiecutter.repo_name }}/.pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@
# See https://pre-commit.com/hooks.html for more hooks
repos:
- repo: https://gitlab.com/pycqa/flake8
rev: master
rev: 3.8.4
hooks:
- id: flake8
args: ["src"]
- repo: https://github.com/Yelp/detect-secrets
rev: v0.13.1
rev: v0.14.3
hooks:
- id: detect-secrets
args: ["--baseline", ".secrets.baseline"]
Expand All @@ -21,7 +21,7 @@ repos:
- --pin-patterns
- "[keep_output]"
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.4.0
rev: v3.4.0
hooks:
- id: check-added-large-files
args: ["--maxkb=5120"]
Expand Down
15 changes: 15 additions & 0 deletions {{ cookiecutter.repo_name }}/.secrets
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Secrets and credentials should be stored here as environmental variables. For example:
#
# # Google Cloud authentication credentials
# export GOOGLE_APPLICATION_CREDENTIALS="path/to/credentials.json"
#
# These environment variables can then be read in by Python using `os.getenv`:
#
# --------------------------------------------------------
# import os
#
# # Google Cloud authentication credentials
# GOOGLE_APPLICATION_CREDENTIALS = os.getenv("GOOGLE_APPLICATION_CREDENTIALS")
# --------------------------------------------------------
#
# This file is NOT version-controlled!
2 changes: 1 addition & 1 deletion {{ cookiecutter.repo_name }}/.secrets.baseline
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"files": null,
"lines": null
},
"generated_at": "2020-10-28T08:52:26Z",
"generated_at": "2021-01-11T09:41:31Z",
"plugins_used": [
{
"name": "AWSKeyDetector"
Expand Down
4 changes: 2 additions & 2 deletions {{ cookiecutter.repo_name }}/CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@ Where this Code of Conduct says:
### Our Pledge

In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to make
participation in our project and our community a harassment-free experience for everyone, regardless of age, body size,
disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education,
participation in our project, and our community a harassment-free experience for everyone, regardless of age,
body size, disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education,
socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.

### Our Standards
Expand Down
16 changes: 11 additions & 5 deletions {{ cookiecutter.repo_name }}/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,9 @@ make requirements
It is better to use the above make command, rather than `pip install -r requirements.txt` and `pre-commit install`, as
the command will ensure your pre-commit hooks are up-to-date with any changes made.

The pre-commit hooks are a security feature to ensure no secrets, large data files, and Jupyter notebooks are
accidentally committed into the repository. For more information about the pre-commit hooks used in this repository,
see the [documentation][docs-pre-commit-hooks].
The pre-commit hooks are a security feature to ensure no secrets<sup>[1](#footnote-1)</sup>, large data files, and
Jupyter notebook outputs are accidentally committed into the repository. For more information about the pre-commit hooks used
in this repository, see the [documentation][docs-pre-commit-hooks].

## Code conventions

Expand Down Expand Up @@ -64,7 +64,7 @@ Use a local link to reference the [`README.md`](./README.md) file, but an extern
[gov-uk]: https://www.gov.uk/
```

We also try and wrap Markdown to a line length of 120 characters, but this is not strictly enforced in all cases, for
We also try to wrap Markdown to a line length of 120 characters, but this is not strictly enforced in all cases, for
example with long hyperlinks.

## Testing
Expand All @@ -83,7 +83,7 @@ Code coverage of Python scripts is measured using the [`coverage`][coverage] Pyt
found in `.coveragerc`. Note coverage only extends to Python scripts in the `hooks`, and
`{{ cookiecutter.repo_name }}/src` folders.

To run code coverage, and view it as a HTML report, execute the following commands in your terminal:
To run code coverage, and view it as an HTML report, execute the following commands in your terminal:

```shell
coverage run -m pytest
Expand All @@ -100,9 +100,15 @@ unless it's more appropriate to store it elsewhere, like this file.
Further information on how to write Sphinx documentation, and how to build it into a searchable website can be found
[here][docs-write-sphinx-documentation].

---

<a name="footnote-1">[1]</a>: Only secrets of specific patterns are detected by the pre-commit hooks. See
[here][docs-pre-commit-hooks-secrets-definition] for further details.

[code-of-conduct]: ./CODE_OF_CONDUCT.md
[coverage]: https://coverage.readthedocs.io/
[docs-pre-commit-hooks]: ./docs/contributor_guide/pre_commit_hooks.md
[docs-pre-commit-hooks-secrets-definition]: ./docs/contributor_guide/pre_commit_hooks.md#definition-of-a-secret-according-to-detect-secrets
[docs-updating-gitignore]: ./docs/contributor_guide/updating_gitignore.md
[docs-write-sphinx-documentation]: ./docs/contributor_guide/writing_sphinx_documentation.md
[email]: mailto:{{ cookiecutter.contact_email }}
Expand Down
7 changes: 6 additions & 1 deletion {{ cookiecutter.repo_name }}/Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
.PHONY: docs docs_check_external_links help prepare_docs_folder requirements
.PHONY:
docs
docs_check_external_links
help
prepare_docs_folder
requirements

.DEFAULT_GOAL := help

Expand Down
Loading

0 comments on commit a497242

Please sign in to comment.