Merge pull request #14 from PolicyEngine/docs-book

Add Jupyter Book documentation

.github/CONTRIBUTING.md

@@ -0,0 +1,7 @@
+## Updating data
+
+If your changes present a non-bugfix change to one or more datasets which are cloud-hosted (FRS and EFRS), then please change both the filename and URL (in both the class definition file and in `storage/upload_completed_datasets.py`). This enables us to store historical versions of datasets separately and reproducibly.
+
+## Updating the versioning
+
+Please add to `changelog.yaml` and then run `make changelog` before committing the results ONCE in this PR.

.github/changelog_template.md

@@ -0,0 +1,8 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), 
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+{{changelog}}

.github/fetch_version.py

@@ -0,0 +1,13 @@
+from policyengine_us_data.__version__ import __version__
+
+
+def fetch_version():
+    try:
+        return __version__
+    except Exception as e:
+        print(f"Error fetching version: {e}")
+        return None
+
+
+if __name__ == "__main__":
+    print(fetch_version())

.github/get-changelog-diff.sh

@@ -0,0 +1,2 @@
+last_tagged_commit=`git describe --tags --abbrev=0 --first-parent`
+git --no-pager diff $last_tagged_commit -- CHANGELOG.md

.github/has-functional-changes.sh

@@ -0,0 +1,12 @@
+#! /usr/bin/env bash
+
+IGNORE_DIFF_ON="README.md CONTRIBUTING.md Makefile docs/* .gitignore LICENSE* .github/* data/*"
+
+last_tagged_commit=`git describe --tags --abbrev=0 --first-parent`  # --first-parent ensures we don't follow tags not published in master through an unlikely intermediary merge commit
+
+if git diff-index --name-only --exit-code $last_tagged_commit -- . `echo " $IGNORE_DIFF_ON" | sed 's/ / :(exclude)/g'`  # Check if any file that has not be listed in IGNORE_DIFF_ON has changed since the last tag was published.
+then
+  echo "No functional changes detected."
+  exit 1
+else echo "The functional files above were changed."
+fi

.github/is-version-number-acceptable.sh

@@ -0,0 +1,33 @@
+#! /usr/bin/env bash
+
+if [[ ${GITHUB_REF#refs/heads/} == master ]]
+then
+    echo "No need for a version check on master."
+    exit 0
+fi
+
+if ! $(dirname "$BASH_SOURCE")/has-functional-changes.sh
+then
+    echo "No need for a version update."
+    exit 0
+fi
+
+current_version=`python .github/fetch_version.py`
+
+if git rev-parse --verify --quiet $current_version
+then
+    echo "Version $current_version already exists in commit:"
+    git --no-pager log -1 $current_version
+    echo
+    echo "Update the version number in setup.py before merging this branch into master."
+    echo "Look at the CONTRIBUTING.md file to learn how the version number should be updated."
+    exit 1
+fi
+
+if ! $(dirname "$BASH_SOURCE")/has-functional-changes.sh | grep --quiet CHANGELOG.md
+then
+    echo "CHANGELOG.md has not been modified, while functional changes were made."
+    echo "Explain what you changed before merging this branch into master."
+    echo "Look at the CONTRIBUTING.md file to learn how to write the changelog."
+    exit 2
+fi

.github/publish-git-tag.sh

@@ -0,0 +1,4 @@
+#! /usr/bin/env bash
+
+git tag `python .github/fetch_version.py`  # create a new tag
+git push --tags || true  # update the repository version

.github/workflows/ci_cd.yaml

@@ -30,6 +30,29 @@ jobs:
           user: __token__
           password: ${{ secrets.PYPI }}
           skip-existing: true
+  publish-docs:
+    name: Publish documentation
+    runs-on: ubuntu-latest
+    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0 # Fetch all history for all tags and branches
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: 3.12
+      - name: Install package
+        run: pip install -e ".[dev]"
+      - name: Build Jupyter Book
+        run: make documentation
+      - name: Deploy documentation
+        uses: JamesIves/github-pages-deploy-action@releases/v4
+        with:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          BRANCH: gh-pages
+          FOLDER: docs/_build/html
   lint:
     runs-on: ubuntu-latest
     name: Lint
@@ -47,7 +70,7 @@ jobs:
         run: black . -l 79 --check
 
   test:
-    name: Build and Test
+    name: Build and test
     runs-on: ubuntu-latest
     steps:
       - name: Checkout code
@@ -68,6 +91,8 @@ jobs:
         run: make data
       - name: Run tests
         run: pytest
+      - name: Test documentation builds
+        run: make documentation
 
   docker:
     name: Docker

.gitignore

@@ -6,8 +6,10 @@
 **/*.csv
 **/*.zip
 **/*.pkl
+**/*.tab
 !uprating_factors.csv
 !uprating_growth_factors.csv
 !incomes.csv
 !tax_benefit.csv
 !demographics.csv
+**/_build

CHANGELOG.md

@@ -0,0 +1,22 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), 
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.1.0] - 2024-09-17 18:05:27
+
+### Changed
+
+- Lightened dependency list.
+
+## [1.0.0] - 2024-09-09 17:29:10
+
+### Added
+
+- Initialized changelogging
+
+
+
+[1.1.0]: https://github.com/PolicyEngine/policyengine-us-data/compare/1.0.0...1.1.0

Makefile

@@ -19,14 +19,23 @@ docker:
 	docker buildx build --platform linux/amd64 . -t policyengine-uk-data:latest
 
 documentation:
-	streamlit run docs/Home.py
+	jb clean docs && jb build docs
 
 data:
 	python policyengine_uk_data/datasets/frs/dwp_frs.py
 	python policyengine_uk_data/datasets/frs/frs.py
+	python policyengine_uk_data/datasets/frs/extended_frs.py
+	python policyengine_uk_data/datasets/frs/enhanced_frs.py
 
 build:
 	python -m build
 
 publish:
 	twine upload dist/*
+
+changelog:
+	build-changelog changelog.yaml --output changelog.yaml --update-last-date --start-from 1.0.0 --append-file changelog_entry.yaml
+	build-changelog changelog.yaml --org PolicyEngine --repo policyengine-us-data --output CHANGELOG.md --template .github/changelog_template.md
+	bump-version changelog.yaml pyproject.toml
+	rm changelog_entry.yaml || true
+	touch changelog_entry.yaml

changelog.yaml

@@ -0,0 +1,10 @@
+- changes:
+    added:
+    - Initialized changelogging
+  date: 2024-09-09 17:29:10
+  version: 1.0.0
+- bump: minor
+  changes:
+    changed:
+    - Lightened dependency list.
+  date: 2024-09-17 18:05:27

docs/_config.yml

@@ -0,0 +1,22 @@
+title: PolicyEngine UK data
+author: PolicyEngine
+copyright: "2024"
+logo: logo.png
+
+execute:
+  execute_notebooks: off
+
+repository:
+  url: https://github.com/policyengine/policyengine-uk-data
+  branch: master
+  path_to_book: docs
+
+sphinx:
+  config:
+    html_js_files:
+    - https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.7/require.min.js
+    html_theme: furo
+    pygments_style: default
+    html_css_files:
+      - style.css
+    nb_remove_code_source: true

docs/_static/style.css

@@ -0,0 +1,2 @@
+@import url('https://fonts.googleapis.com/css2?family=Roboto+Serif:[email protected]&family=Roboto:wght@300&display=swap');
+

docs/_toc.yml

@@ -0,0 +1,5 @@
+format: jb-book
+root: intro
+chapters:
+- file: methodology.ipynb
+- file: validation.ipynb

docs/Home.py → docs/intro.md

@@ -1,16 +1,5 @@
-import streamlit as st
-from policyengine_uk_data.utils.download_docs_prerequisites import (
-    download_data,
-)
+# Introduction
 
-download_data()
-
-st.set_page_config(layout="wide")
-
-st.title("PolicyEngine-UK-Data")
-
-st.write(
-    """
 PolicyEngine-UK-Data is a package that creates representative microdata for the UK, 
 designed for input in the PolicyEngine tax-benefit microsimulation model. This tool 
 allows users to explore the data sources, validation processes, and enhancements 
@@ -22,5 +11,4 @@
 * An accurate representation of the current UK household sector *now*.
 
 This repository is dedicated to the second of those. In this documentation, we'll explain how we do that, but we'll also use our model (the first bullet) to see what we end up with when we combine the two, and measure up against other organisations doing the same thing.
-"""
-)
+