Modern Python3 Boilerplate with tooling

This is a template to start a python project using a Test Driven Development (TDD) approach

How to use this repo?

a) FastTrack: Just clone and use the master branch to get started right away.

Pro tip: Use the Green Use this template button to fork this repo to your github account and build your awesome Python3 application based of that. Note this will only fork the master branch.

b) LearningTrack: Clone this repo then progressively do a git checkout on all the branhes in their numerical sort order

i.e.

# clone this repo
git clone https://github.com/Grant-Steinfeld/python-starter.git

#fetch all remote branches
git fetch --all

# start at 001 - setup directory structures and essential scaffolding
git checkout 001-setup-tooling

# learn about python virtual environments and in particular the awesome alternative to venv, enter Pipenv!
git checkout 002-pipenv

# setup Pytest testing framework, really the best option when testing Python, much better than the built in unittest library
git checkout 003-pytest

# setup linters and formatters - introducing Flake8, Flake8-Bugbear and Prettier
git checkout 004-style-tools

# setup python logging with standard python logging library
git checkout 005b-logger

# setup git pre commit hooks to prevent commiting and hence pushing incorrect syntax and poor formatting to git
git checkout 005-git-pre-commit-hook

# start to write tests first and get on your way to been an awesome TDD developer!
git checkout 006-start-tdd

# deep dive into mocks in pytest by learning howto use the Monkey Patch technique!
git checkout 007-monkeypatch

# after all said and done, all these branches rollup to the master branch
git checkout master

Install the pre-requisites

1. Python version 3
2. Pipenv - Python virtual environment

Installation steps

1. Python3

Make sure you have Python installed and it's availible from your command line. You can check if it's installed and determine it's version by running:

python --version

You shoud get some output like 3.6.2 If you don't have this version of Python, please install the latest 3.x version.

To install python 3 on a Mac

brew install python3

Installation of Python3 on other platforms

To [install Python3 on RHEL](https://developers.redhat.com/blog/2018/08/13/install-python3-rhel/)

To install Python3 on Ubuntu

To install Python3 on Windows

To install Python on any other platform take a look at the Installing Python section of The Hitchhikers Guide to Python or refer to python.org

2. Pipenv - Python virtual environment

To check you have pipenv installed run the following:

pipenv --version

You should see something like version 2018.11.26 if not please setup the latest version of pipenv as follows.

To install pipenv on a Mac using brew

brew install pipenv

Installation of Pipenv on other platforms

If you have a working installation of pip, and maintain certain “toolchain” type Python modules as global utilities in your user environment, pip user installs allow for installation into your home directory. Note that due to interaction between dependencies, you should limit tools installed in this way to basic building blocks for a Python workflow like virtualenv, pipenv, tox, and similar software.

To install pipenv on anyplatform with pip

pip install --user pipenv

#or
# todo: validate this
python3 -m pip install pipenv

For more detailed instruction see here

It is a best practice to use use Python virtual environments to isolate project-specific dependencies and create reproducible environments.

Read more about Pipenv and Virtual Environments

Pipenv Features

• Pipenv is a production-ready tool that aims to bring the best of all packaging worlds to the Python world. It harnesses Pipfile, pip, and virtualenv into one single command.

• Enables truly deterministic builds, while specifying only what is needed.

• With pipenv you no longer need to use pip and venv separately.

• Setting a virtual environment to separate each project from affecting other projects and the rest of your operating system's a good idea. You may be making changes in your virtual environment that could have unintended consequences.

Learn more about Pipenv here

Intializing a pipenv Python Virtual Environment

How does one setup a Python Virtual Environment using pipenv?

What is a virutal env

It's a copy of a physically installed version of python already have, so say you have python3 install via brew or some other method, you can find this by typing

which python3
echo '/usr/local/bin/python3'

You may be asking yourself where your new virtual environment is stored?

Ordinarilly, by default, the pipenv virutal enviroments is written to a global (your user's home ) dirctory. The issue here is if you move your project directory this will corrupt the virutal environment.

So never fear!

unix/mac

export PIPENV_VENV_IN_PROJECT=1
# save this line to your ~/.bashrc or ~/.zshrc or equivalent

PowerShell on Windows 10

$env:PIPENV_VENV_IN_PROJECT=1

Creating a new Pipenv Python3 Virtual Environment

At your command line cd to the root directory of your application

#install 
pipenv install --three

You should now confirm the new local to your project, Pipenv Python Virtual Environment by output similar to this:

So great! Now pipenv created a virtual environment and created a Pipfile and a Pipfile.lock

Check!

pipenv check

Output should confirm all is good!

You can also confirm the virtual environment is setup by confirming a new file called Pipfile exists at the root directory.

Even though the pipenv virtual environment is setup, you still need to activate it. This is simply done by running:

pipenv shell

To exit the Pipenv Python Virtual environment simply type exit

Setting up tooling for Testing

Setting up the pytest unit-test framework

pytest is a no-boilerplate alternative to Python’s standard unittest module

pipenv install --dev pytest

pytest is used to write tests first and begin our journey towards Test Driven Development, been a fully-featured and extensible test tool, it boasts a simple syntax. Creating a test suite is as easy as writing a module with a couple of functions:

#contents of tests/unit/test_sample.py
def plusOne(x):
    return x + 1

def test_simple():
    assert plusOne(7) == 8

the test is run by running the pytest command.

pytest tests/unit/test_sample.py

Code stylers and formatters

Flake8 is a command-line utility for enforcing style consistency across Python projects.

learn more about flake8

Flake8, by default it includes lint checks provided by the PyFlakes project, PEP-0008 inspired style checks provided by the PyCodeStyle project, and McCabe complexity checking provided by the McCabe project. It will also run third-party extensions if they are found and installed.

To install these:

pipenv install --dev flake8 black==19.10b0

Version Control Integration - flake8 with git pre-commit hooks

Git hook scripts are useful for identifying simple issues before submission to code review.

Learn more about pre-commit

"... We run our hooks on every commit to automatically point out issues in code such as missing semicolons, trailing whitespace, and debug statements. By pointing these issues out before code review, this allows a code reviewer to focus on the architecture of a change while not wasting time with trivial style nitpicks."

quote from

Add new python packages:

pipenv install --dev pre-commit
pipenv install --dev flake8-bugbear

add [scripts] block to pipfile for pre-commit hook

[scripts]
# Install the pre-commit hook
setup_dev = "sh -c \"pre-commit install\""

Add a .pre-commit-config.yaml file. Here is the contents

Setup the pre-commit hooks

pipenv run setup_dev

Ready? Time to check git pre-commit hook works as expected!

run:

git commit README.md -m "test commit"

TIP! If this is the first time you run this, it will take 5-9 minutes depending on your local laptop or workstation's cpu/RAM horsepower.

You should see something like this output in your terminal window.

Yay!!!

After you've run pre-commit the first time, subsequent commit's will be fast ( seconds ).

The output is less verbose and like I said, it will be much much faster!

Logging

Logs provide visibility into the behavior of a running app. Logs are the stream of aggregated, time-ordered events collected from the output streams of all running processes and backing services.

Learn more about logging

A [twelve-factor app[(https://12factor.net/logs)] never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout. During local development, the developer will view this stream in the foreground of their terminal to observe the app’s behavior.

In staging or production deploys, each process’ stream will be captured by the execution environment, collated together with all other streams from the app, and routed to one or more final destinations for viewing and long-term archival. These archival destinations are not visible to or configurable by the app, and instead are completely managed by the execution environment.

Motivation to instrument logging in your code

Diagnostic logging records events related to the application’s operation. If a user calls in to report an error, for example, the logs can be searched for context.

Audit logging records events for business analysis. A user’s transactions can be extracted and combined with other user details for reports or to optimize a business goal.

The Hitchhiker's Guide to Python: Logging

Start Test Driven Development

Red-Green-Refactoring.

There are 5 basic steps as illustrated in Figure 1. below.

Figure 1. The 5 stages in the Red-Green-Refactor software development cycle

Requests originate from the BDD / Agile Design phase/thinking sessions.

Common requests are new feature stories or issue/bug fixes.

These are the 5 steps:

1. Pick a request from your project management system [5]

   1. Action it! by Read, understand the request

2. Write a test to reflect the requirement

   1. run test it must fail!! (Red)

3. Write the code

   1. run test - code until test passes (Green)

4. refine, cleanup code (Refactor)

   1. run test - if fails continue to refactor till it passes

5. rinse, lather, repeat.

Sometimes tests expect exctpions so how to make them not fail.

How to properly assert that an exception gets raised in pytest?

According to this Stackoverflow post Pytest has 2 ways to accomadate this:

1. Using pytest.raises is likely to be better for cases where you are testing exceptions your own code is deliberately raising

2. using @pytest.mark.xfail with a check function is probably better for something like documenting unfixed bugs (where the test describes what "should" happen) or bugs in dependencies.

Foot notes

[5] Project management tool include:

• Jira
• Pivotal Tracker
• ZenHug
• GitHub issues
• other

Resources

Python Testing with pytest: Simple, Rapid, Effective, and Scalable. Okken, Brian. Pragmatic Bookshelf.

License

This code is licensed under the Apache License, Version 2. Separate third-party code objects invoked within this code pattern are licensed by their respective providers pursuant to their own separate licenses. Contributions are subject to the Developer Certificate of Origin, Version 1.1 and the Apache License, Version 2.

Apache License FAQ