rsd-summerschool

Course materials for Research Software Engineering Summer School.

Content

In this summer school, you will move beyond programming, to learn how to construct reliable, readable, efficient research software in a collaborative environment. The emphasis is on practical techniques, tips, and technologies to effectively build and maintain complex code. This is an intensive and a practical summer school. The content of each of the 10 half-day units is as follows:

Day 1: Version Control Branching Rebasing and Merging Debugging with GitBisect Forks, Pull Requests and the GitHub Flow Virtual Environments	Day 2: Research Data and Design Patterns in Python Working with files on the disk Interacting with the internet JSON and YAML Other Scientific Data Formats Refactoring Static Typing Data Classes and Validation Object Orientation Design Patterns
Day 3: Testing and Code Smell Why test? Unit testing and regression testing Negative testing Advanced Testing Techniques Debugging Continuous Integration Coding conventions	Day 4: Packaging and Documenting Software Projects Turning your code into a package Releasing code Documentation
Day 5: Programming for Speed Optimisation Profiling Scaling laws NumPy Miscellaneous libraries Cython

Prerequisites

• You need to be competent in at least one programming language, including concepts like variables, control flow, and functions. This could be through a formal course, a shorter workshop like Software Carpentry, or previous experiences.

• You need to be familiar with the basics of Git and GitHub. A good resource for attendees who are not familiar with Git and GitHub -

  • First 8 chapters of Version Control with Git by Software Carpentry

• You need to be familiar with the foundational libraries of PyData and Scientific Python ecosystem. A good resource for attendees who are not familiar with the Scientific Python ecosystem -

  • Scientific Python Lectures
    • 1.1. Python scientific computing ecosystem
    • 1.2. The Python language
    • 1.3.1. The NumPy array object
    • 1.3.2. Numerical operations on arrays
    • 1.4. Matplotlib: plotting
    • 1.6. Getting help and finding documentation

  If curious, one can also go through the remaining of the 1.3. NumPy: creating and manipulating numerical data section and 1.5. SciPy: high-level scientific computing.

• You are required to bring your own laptop. We have also provided setup instructions for you to install the software needed for the course on your computer.

Contributing to this repository

This repository contains the course notes as Jupyter notebooks converted into py:percent format. This allows to edit the files as plain text as well as jupyter notebooks. To edit them as jupyter notebooks you'll need to have installed jupytext and open the ipynb.py files as notebooks via right-click and select "open with" and "notebook" on the Jupyter file browser.

Caution

Do not run make locally on your computer! It will produce side effects on your global git configuration (because it will run the scripts from the git chapter)! Instead, follow the instructions below.

Testing it locally

The site is built using gh-actions. If you'd like to test the actions locally, you can run the actions using act command tool. By default this will run the action in a copy of the repository and you won't be able to inspect the steps that happened. If you'd like to keep the output in the current directory, use the -b (bind) flag.

$ act -b
[Build website/Build-website] 🚀  Start image=catthehacker/ubuntu:act-latest
[Build website/Build-website]   🐳  docker run image=catthehacker/ubuntu:act-latest platform= entrypoint=["/usr/bin/tail" "-f" "/dev/null"] cmd=[]
[Build website/Build-website] ⭐  Run actions/checkout@v2
[Build website/Build-website]   ✅  Success - actions/checkout@v2
[Build website/Build-website] ⭐  Run actions/cache@v2
INFO[0000]   ☁  git clone 'https://github.com/actions/cache' # ref=v2 
[Build website/Build-website]   ✅  Success - actions/cache@v2
[Build website/Build-website] ⭐  Run Install TeXLive
INFO[0000]   ☁  git clone 'https://github.com/DanySK/setup-texlive-action' # ref=0.1.1 
[Build website/Build-website]   ✅  Success - Install TeXLive
[Build website/Build-website] ⭐  Run Setup Python
INFO[0001]   ☁  git clone 'https://github.com/actions/setup-python' # ref=v2 
[Build website/Build-website]   ✅  Success - Setup Python
[Build website/Build-website] ⭐  Run Install dependencies
INFO[0001]   ☁  git clone 'https://github.com/py-actions/py-dependency-install' # ref=v2 
[Build website/Build-website]   ✅  Success - Install dependencies
[Build website/Build-website] ⭐  Run Building notes
[Build website/Build-website]   ✅  Success - Building notes
[Build website/Build-website] ⭐  Run Builds website
INFO[0001]   ☁  git clone 'https://github.com/helaili/jekyll-action' # ref=v2 
[Build website/Build-website]   🐳  docker run image=act-helaili-jekyll-action-v2:latest platform= entrypoint=[] cmd=[]
[Build website/Build-website]   ✅  Success - Builds website

Alternatively, if you want to only run the jekyll build step once you've run the whole action, you can use the official jekyll containers with:

$ docker run --rm --volume="$PWD:/srv/jekyll" --volume="$PWD/vendor/bundle:/usr/local/bundle" -p 4000:4000 -it jekyll/jekyll:latest jekyll serve

and open http://localhost:4000/rsd-engineeringcourse (or the link provided). Note that this is mounting the bundle directory where act will create them.

Migration from jupyter notebooks to py:percent

Using jupytext we've converted all the jupyter notebooks into plain text python files (py:percent) with:

# First cleaned all outputs and commited it
nbstripout --extra-keys metadata.kernelspec ch*/*ipynb
# convert them
find ./ -iname '*ipynb' -exec jupytext --opt notebook_metadata_filter="kernelspec,jupytext,jekyll" --to py:percent {} -o {}.py \;
# then deleted the ipynb
find ./ -iname '*ipynb' -delete