Basics of Virtual Environments

Virtual Environments are fun! They are great! They are my best friends.

Why? They ...

• ... prevent dependencies of one project from impacting another project.
• ... let people know what code bases and versions they need to run your stuff without hassle (they conveniently package up all of your dependencies for when other people need to run your code. :)

You’ll want at least one distinct virtual environment for each project. You might have one for each major result milestone.

Poetry

Our preferred method!

Quick How-To

poetry shell from within the package name, starts the shell (like source venv/bin/activate if you used a venv)

poetry add packagename i.e. poetry add pandas

poetry add git+https://github.com/whatever.git To add something via the git repo.

poetry config http-basic.DeepDance askdjf [your un] jksajdf [your private key] If the project is private

Toml file - you can define where it pulls from to build dependencies. It can also become a requirements.txt, conda yml.

poetry publish -0.1.0 → publishes to pypi, if your publishing is set up.

References

• Official Docs with install instructions: https://python-poetry.org/
• Which Python Package Manager Should You Use? ←a good quick review, also talks a bit about pyenv
• Poetry: "dependency management and packaging made easy" ← this guy gives a fun intro talk to poetry
• "The Packaging Lifecycle with Poetry" - Clinton Roy (Kiwi Pycon X) ← more in depth, this guy is also pretty funny

How does poetry work?

• Libraries are packaged as wheels, installed via pip
• Applications are packaged via containers
• Has a bunch of different ways to specify packages: In a path, public pypi, private, pypi, github repo, url, name + constraint (pip)

Potential Downsides

• No environment logging - You need to remember what repos are set up with poetry and where they are.
• A little too easy to communicate with private repos, doesn’t scrub log in. Sessions end really fast, so it’s not an issue in most cases, but it is something to be aware of.

Conda

• conda is a package management system developed by a company called Anaconda, which is often in use on clusters but which can also be used on a personal machine
• you can install either the whole enchilada, called anaconda, or the, uh, miniature taco, called miniconda (which includes pip for sure, plus also I think numpy, scipy, and a few other essentials); they are compared here, though at a debatably useful level of detail

Upsides

• easy package management – start with conda install <package> and it takes care of dependencies for you; keep updated with conda update <package> or, even more usefully once you have a robust environment in place, conda update --all
• virtual environments are very easy – conda create -n <name of new environment> {list of any packages you want to include from the get go}

Downsides

• it’s managed by a corporation (albeit one that operates on a freemium model, so you won’t have to pay to use it on your laptop)
• this is partially mitigated by using the conda-forge channel for installation like so conda install -c conda-forge
• conda-forge is open source (not administered by Anaconda)
• even miniconda is fairly “heavy” and the dependency solver takes O(10s) to run, so it’s not lightning speed

Mamba

• Have you ever thought about installing conda and felt “wow, this just seems too heavy for me, the dependency solver sounds like it takes foreeeeever to run, and I really value open-source projects!”? Well mamba might be the solution for you!
• mamba is a drop-in replacement for conda: you can start with mamba install <package>, for example, then create a venv with mamba create -n <new name> jax numpy matplotlib, and stay up-to-date with mamba update --all
• there is an even lighter version called micromamba. It’s self-contained. The dependency solver is lightning-fast. It’s, uh, “currently experimental”, so maybe don’t rely on it? https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html