Skip to content

Commit

Permalink
Post1.4.1 (#84)
Browse files Browse the repository at this point in the history
* doc updates
* add python 3.12 and fix warnings
* add RunState enum
* update CI, tidy macros
* vuln+fixes
  • Loading branch information
virgesmith authored May 27, 2024
1 parent db69d81 commit 954ac8e
Show file tree
Hide file tree
Showing 81 changed files with 4,580 additions and 3,628 deletions.
6 changes: 3 additions & 3 deletions .github/workflows/build-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,12 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.10", "3.11"]
python-version: ["3.10", "3.11", "3.12"]
os: [ubuntu-latest, windows-latest, macos-latest]
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: "pip: Python ${{ matrix.python-version }} / ${{ matrix.os }}"
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
Expand Down
8 changes: 5 additions & 3 deletions .github/workflows/coverage.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,11 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: [ "3.10" ]
python-version: [ "3.11" ]
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: "pip: Python ${{ matrix.python-version }} coverage"
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
Expand All @@ -32,5 +32,7 @@ jobs:
run: |
python -m pytest
- name: Upload
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
run: |
bash <(curl -s https://codecov.io/bash) -Z
4 changes: 2 additions & 2 deletions .github/workflows/draft-pdf.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,14 @@ jobs:
name: Paper Draft
steps:
- name: Checkout
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Build draft PDF
uses: openjournals/openjournals-draft-action@master
with:
journal: joss
paper-path: paper/paper.md
- name: Upload
uses: actions/upload-artifact@v1
uses: actions/upload-artifact@v4
with:
name: paper
path: paper/paper.pdf
6 changes: 3 additions & 3 deletions .github/workflows/mpi-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,11 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.10"]
python-version: ["3.11"]
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: "pip: Python ${{ matrix.python-version }}"
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pypi-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ jobs:
if: "!contains(github.event.head_commit.message, 'Bump version')"
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
persist-credentials: false
- name: current_version
Expand Down
2 changes: 1 addition & 1 deletion LICENCE.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# MIT License

Copyright &copy; 2017-2023 Andrew P Smith
Copyright &copy; 2017-2024 Andrew P Smith

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
Expand Down
2 changes: 1 addition & 1 deletion docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,4 @@

## Type stubs

Type stubs were generated for the core C++ library using the `pybind11-stubgen` package, although significant editing of the output was required. See `__init__.pyi` for details.
Type stubs were generated for the core C++ library using the `pybind11-stubgen` package, with minor manual corrections. See `__init__.pyi` for details.
2 changes: 1 addition & 1 deletion docs/developer.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ The script from [codecov.io](https://codecov.io/gh/virgesmith/neworder/) uses `g

## Generating type stubs

Type stubs can be generated for the C++ module using `pybind11-stubgen`, although manual modifications are needed for the output (e.g. docstrings for overloaded functions are misplaced, numpy types need to be fixed).
Type stubs can be generated for the C++ module using `pybind11-stubgen`, although manual modifications may be needed, plus numpy types need to be fixed globally.

```sh
pybind11-stubgen _neworder_core --ignore-invalid all
Expand Down
17 changes: 3 additions & 14 deletions docs/examples/boids.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,33 +22,22 @@ Each entity travels at a fixed speed in a 2- or 3-dimensional constrained univer

(if a separation is required, the boid will not attempt to align or cohere)

The entities are stored in a pandas `DataFrame` and use `neworder.Space` to update positions. There are no explicit `for` loops in the model - all position and velocity calculations are "vectorised"<sup>&ast;</sup> for efficiency.
The entities are stored in a pandas `DataFrame` and use `neworder.Space` to update positions. There are no explicit `for` loops in the model - all position and velocity calculations are "vectorised"[^1] for efficiency.

&ast; in this context "vectorisation" merely means the avoidance of explicit loops in an interpreted language. The actual implementation may be compiled to assembly language, vectorised in the true ([SIMD](https://en.wikipedia.org/wiki/SIMD)) sense, parallelised, optimised in other ways, or any combination thereof.
[^1]: in this context "vectorisation" merely means the avoidance of explicit loops in an interpreted language. The actual implementation may be compiled to assembly language, vectorised in the true ([SIMD](https://en.wikipedia.org/wiki/SIMD)) sense, parallelised, optimised in other ways, or any combination thereof.

Run like so

```sh
python examples/boids/run.py 2d
```

The 2d version utilises a wrap-around domain and does not implement the reversion step.

or

```sh
python examples/boids/run.py 3d
```

which runs

{{ include_snippet("examples/boids/run.py") }}

and this is the 3-d implementation:

{{ include_snippet("examples/boids/boids3d.py") }}

A 2-d implementation is also provided in `examples/boids/boids2d.py`.
The 2d version utilises a wrap-around domain and so does not require the reversion step.

## Outputs

Expand Down
2 changes: 1 addition & 1 deletion docs/examples/mortality.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Mortality

We implement the example *The Life Table* from the second chapter of the book *Microsimulation and Population Dynamics* [[3]](#references). It models mortality in a homogeneous population with an age-specific mortality rate.
We implement the example *The Life Table* from the second chapter of the book *Microsimulation and Population Dynamics* [[3]](../references.md). It models mortality in a homogeneous population with an age-specific mortality rate.

This example implements the model in two different ways: firstly a discrete case-based microsimulation, and again using a continuous sampling methodology, showcasing how the latter can be much more efficient. Rather than having a class to represent an individual, as would be standard in a MODGEN implementation, individuals are stored in a *pandas* `Dataframe` which provides fast iteration over the population.

Expand Down
2 changes: 1 addition & 1 deletion docs/examples/people.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

In this example, the input data is a csv file containing a microsynthesised 2011 population of Newcastle generated from UK census data, by area (MSOA), age, gender and ethnicity. The transitions modelled are: ageing, births, deaths and migrations, over a period of 40 years to 2051.

Births, deaths and migrations (applied in that order) are modelled using Monte-Carlo simulation (sampling Poisson processes in various ways) using distributions parameterised by age, sex and ethnicity-specific fertility, mortality and migration rates respectively, which are largely fictitious (but inspired by data from the NewETHPOP[[1]](#references.md) project).
Births, deaths and migrations (applied in that order) are modelled using Monte-Carlo simulation (sampling Poisson processes in various ways) using distributions parameterised by age, sex and ethnicity-specific fertility, mortality and migration rates respectively, which are largely fictitious (but inspired by data from the NewETHPOP[[1]](../references.md) project).

For the fertility model newborns simply inherit their mother's location and ethnicity, are born aged zero, and have a randomly selected gender (with even probability). The migration model is an 'in-out' model, i.e. it is not a full origin-destination model. Flows are either inward from 'elsewhere' or outward to 'elsewhere'.

Expand Down
2 changes: 1 addition & 1 deletion docs/examples/riskpaths.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# RiskPaths

RiskPaths is a well-known MODGEN model that is primarily used for teaching purposes and described here[[5]](#references) in terms of the model itself and here in terms of implementation[[6]](#references). It models fertility in soviet-era eastern Europe, examining fertility as a function of time and union state. In the model, a woman can enter a maximum of two unions in her lifetime. The first union is divided into two sections: a (deterministic) 3 year period during which fertility is at a maximum, followed by a (stochastic) period with lower fertility.
RiskPaths is a well-known MODGEN model that is primarily used for teaching purposes and described here[[5]](../references.md) in terms of the model itself and here in terms of implementation[[6]](../references.md). It models fertility in soviet-era eastern Europe, examining fertility as a function of time and union state. In the model, a woman can enter a maximum of two unions in her lifetime. The first union is divided into two sections: a (deterministic) 3 year period during which fertility is at a maximum, followed by a (stochastic) period with lower fertility.

![riskpaths](./img/riskpaths.png)

Expand Down
163 changes: 91 additions & 72 deletions docs/macros.py
Original file line number Diff line number Diff line change
@@ -1,90 +1,109 @@
# macros for mkdocs-macros-plugin
import os
import requests
import importlib
import os
from datetime import datetime
from functools import cache
from typing import Any

import requests

_inline_code_styles = {
".py": "python",
".sh": "bash",
".h": "cpp",
".cpp": "cpp",
".c": "c",
".rs": "rs",
".js": "js",
".md": None
".py": "python",
".sh": "bash",
".h": "cpp",
".cpp": "cpp",
".c": "c",
".rs": "rs",
".js": "js",
".md": None,
}

# this is the overall record id, not a specific version
_NEWORDER_ZENODO_ID = 4031821
_NEWORDER_ZENODO_ID = 4031821 # search using this (or DOI 10.5281/zenodo.4031821) just doesnt work


@cache
def get_zenodo_record() -> dict[str, Any]:
try:
response = requests.get(
"https://zenodo.org/api/records",
params={
"q": "(virgesmith) AND (neworder)", # this is the only query that seems to work
"access_token": os.getenv("ZENODO_PAT"),
},
)
response.raise_for_status()
# with open("zenodo-result.json", "w") as fd:
# fd.write(response.text)
return response.json()["hits"]["hits"][0]
except Exception as e:
return {f"{e.__class__.__name__}": f"{e} while retrieving zenodo record"}


def write_requirements() -> None:
try:
with open("docs/requirements.txt", "w") as fd:
fd.write(f"""# DO NOT EDIT
try:
with open("docs/requirements.txt", "w") as fd:
fd.write(
f"""\
# DO NOT EDIT
# auto-generated @ {datetime.now()} by docs/macros.py::write_requirements()
# required by readthedocs.io
""")
fd.writelines(f"{dep}=={importlib.metadata.version(dep)}\n" for dep in [
"mkdocs",
"mkdocs-macros-plugin",
"mkdocs-material",
"mkdocs-material-extensions",
"mkdocs-video",
"requests"
])
# ignore any error, this should only run in a dev env anyway
except:
pass
"""
)
fd.writelines(
f"{dep}=={importlib.metadata.version(dep)}\n"
for dep in [
"mkdocs",
"mkdocs-macros-plugin",
"mkdocs-material",
"mkdocs-material-extensions",
"mkdocs-video",
"requests",
]
)
# ignore any error, this should only run in a dev env anyway
except: # noqa: E722
pass


def define_env(env):
@env.macro
def insert_zenodo_field(*keys: str) -> Any:
result = get_zenodo_record()
for key in keys:
result = result[key]
return result

@env.macro
def insert_zenodo_field(*keys: str):
""" This is the *released* version not the dev one """
try:
response = requests.get('https://zenodo.org/api/records', params={'q': _NEWORDER_ZENODO_ID, 'access_token': os.getenv("ZENODO_PAT")})
response.raise_for_status()
result = response.json()["hits"]["hits"][0]
for k in keys:
result = result[k]
return result
@env.macro
def include_snippet(filename, tag=None, show_filename=True):
"""looks for code in <filename> between lines containing "!<tag>!" """
full_filename = os.path.join(env.project_dir, filename)

except Exception as e:
return f"{e.__class__.__name__}:{e} while retrieving {keys}"


@env.macro
def include_snippet(filename, tag=None, show_filename=True):
""" looks for code in <filename> between lines containing "!<tag>!" """
full_filename = os.path.join(env.project_dir, filename)

_, file_type = os.path.splitext(filename)
# default to literal "text" for inline code style
code_style = _inline_code_styles.get(file_type, "text")

with open(full_filename, 'r') as f:
lines = f.readlines()

if tag:
tag = f"!{tag}!"
span = []
for i, l in enumerate(lines):
if tag in l:
span.append(i)
if len(span) != 2:
return f"```ERROR {filename} ({code_style}) too few/many tags ({len(span)}) for '{tag}'```"
lines = lines[span[0] + 1: span[1]]

if show_filename:
footer = f"\n[file: **{filename}**]\n"
else:
footer = ""
if code_style is not None:
return f"```{code_style}\n{''.join(lines)}```{footer}"
else:
return "".join(lines) + footer

write_requirements()
_, file_type = os.path.splitext(filename)
# default to literal "text" for inline code style
code_style = _inline_code_styles.get(file_type, "text")

with open(full_filename, "r") as f:
lines = f.readlines()

if tag:
tag = f"!{tag}!"
span = []
for i, line in enumerate(lines):
if tag in line:
span.append(i)
if len(span) != 2:
return f"```ERROR {filename} ({code_style}) too few/many tags ({len(span)}) for '{tag}'```"
lines = lines[span[0] + 1 : span[1]]

if show_filename:
title = f'title="{filename}"'
else:
title = ""
if code_style is not None:
return f"```{code_style} {title}\n{''.join(lines)}```"
else:
return "".join(lines)


# write_requirements()
Loading

0 comments on commit 954ac8e

Please sign in to comment.