Python Virtual Environments

Python sports a concept called the virtual environment. It is sometimes called a "virtualenv" or a "venv". You can think of a virtual environment as a kind of lightweight container for a Python application, where the application can see its own particular set of Python libraries, at specific versions. This provides significant benefits for deployment:

  • Dependencies can be precisely tracked and specified, and even kept in version control.
  • Two different Python applications with conflicting dependencies can peacefully coexist on the same machine.
  • Python packages can be installed without requiring elevated system privileges.

Let’s dig in.

Creating New VEnvs

You create a virtual environment by invoking python -m venv, adding one more argument - the name of a folder to create:[36]

$ python -m venv webappenv

What this does is create a folder named "webappenv" in the current directory. You end up with a new folder named webappenv, containing all sorts of goodies. To access them, you must run a script called "activate". In macOS and Linux, type the following in your shell:

$ source webappenv/bin/activate
(webappenv)$

For Windows, run the script webappenv\Scripts\activate.bat instead:

C:\labs> webappenv\Scripts\activate.bat
(webappenv) C:\labs>

Notice your prompt has changed: now it contains (webappenv). The script called activate did not start a new shell or session; all it did was alter your PATH and PYTHONPATH environment variables, and a few others (like the PS1 variable, which specifies how your shell renders the prompt.) You just activated your virtual environment, as we say. When it’s active for that particular shell session, we say you are working in the virtual environment.

Suppose your system Python executable is at /usr/bin/python. And suppose you’ve created the webappenv folder in /Users/sam/mywebapp/webappenv. With the virtual environment activated, you have your own local copy of the Python interpreter. You can check this with the which command on macOS and Linux:

(webappenv)$ which python
/Users/sam/mywebapp/webappenv/bin/python
(webappenv)$ python -V
Python 3.10.0b3

Or the where command on Windows:

(webappenv) C:\labs> where python
C:\labs\webappenv\Scripts\python.exe
C:\Python310\python.exe

Now, what if you want to restore your old PATH (and PYTHONPATH, etc.)? Within the virtualenv, you now have a function defined called deactivate:

(webappenv)$ deactivate
$ which python
/usr/bin/python

Now imagine you are writing a Python application: let’s call it mediatag, one designed for tagging files in a media collection. And you have not had time to test it on the latest Python version, so for now you want to just run it in Python 3.9 - installed on the system under /usr/bin/python3.9. Set up the virtual environment like so:

$ cd /Users/sam/mediatag
$ python3.9 -m venv mediatagenv
$ source mediatagenv/bin/activate
(mediatagenv)$ which python
/Users/sam/mywebapp/mediatagenv/bin/python
(mediatagenv)$ python -V
Python 3.9.6

This shows one minor benefit of virtual environments: it provides a new way to control the precise version of Python used by an application. But the main benefit for using virtual environments has to do with resolving requirements, upgrades, and dependencies with packages and libraries.

Python Package Management

The sordid history of Python library and package management is full of hidden twists, perilous turns, and dark corners hiding sinister beasts. Good news: you don’t have to worry about any of that.

Modern Python provides an application called pip, which allows you to easily install third-party Python libraries and applications. It incorporates many of the lessons learned from its predecessors, sidestepping problems that previously had to be dealt with manually. And it works very well with Python virtual environments.

The first step is to install pip. With Python 3, this is included for you automatically, and is installed in your virtual environment:

$ source venv/bin/activate
(venv)$ which pip
/Users/sam/myapp/venv/bin/pip

You can use pip install to install libraries just within this specific virtual environment. For example, requests is a high-quality HTTP library. Install it like so:

pip install requests

This is the install command. You will see some output, narrating the process of installing requests at a specific version. Once complete, you will be able to open a Python prompt and import requests.

The pip install command is also used to upgrade packages. For example, sometimes a fresh virtual environment may install a slightly stale version of pip. pip is just another package, so you can upgrade it with the -U or --upgrade option:

pip install --upgrade pip

Installed packages are, by default, fetched from Pypi - the official online Python package repository. Any package or library listed at https://pypi.python.org/pypi can be installed with pip. You can uninstall them with pip uninstall.

Now, some of these packages' files are substantial, or even compiled into object files. You definitely don’t want to put them in version control. How do you register the exact version dependencies your app has for third-party libraries? And how do you manage upgrades (and even downgrades) over time?

pip provides a good solution for this. The first part of it relies on the pip freeze command:

(venv)$ pip freeze
certifi==2021.5.30
chardet==4.0.0
idna==2.10
requests==2.25.1
urllib3==1.26.6

This prints the packages installed from Pypi, one per line, with the exact version. Notice it installs requests, and also automatically installs the other libraries that requests depends on. What you can do is place this in a file named requirements.txt:

(venv)$ pip freeze > requirements.txt
(venv)$ cat requirements.txt
certifi==2021.5.30
chardet==4.0.0
idna==2.10
requests==2.25.1
urllib3==1.26.6

This file is what you will check into version control. You can recreate the application environment, right down to the minor versions of each dependent library, simply by passing requirements.txt to pip. Whether your coworker is fetching the raw source to start development, or if the devops team sets up a CI environment that runs the automated tests, the environment is consistent and well-defined, from development to staging to production.

You can pass any file path to python -m venv. For organizational convenience, many choose to put this virtual-environment directory in the top-level folder of the repository holding the Python application. There are two schools of thought on what to name it.

One school picks a consistent name, which is used for every project. "venv" is very popular:

python -m venv venv

The idea is that every Python project will have a folder in its top level called venv to contain the virtual environment. This has several advantages. For one, you can easily activate the virtual environment for any application, just by typing source venv/bin/activate. No wasted mental cycles.

You can also configure your version control system to ignore any folder named "venv", and thereby avoid ever accidentally committing your virtual environment. (You don’t want to do that. It’s a lot of files, and the files will probably not work at all on a different machine.)

The other naming scheme is to give it a name that has something to do with the application. For example, for an application called "mywebapp", you might create it like this:

python -m venv mywebappenv

The advantage of this is that, when activated, the prompt is modified to tell you which particular virtual environment your shell prompt is using. This can be helpful if you work with many different Python applications, as it’s much more informative than a prompt that just says "(venv)".

The downside comes from the inconsistency of the folder name: keeping the folder out of version control is more error-prone, and activating the virtual environment requires the distraction of conscious thought each time. Both approaches are valid; it really comes down to which you and your teammates like better.



[36] Modern Python may be installed as python3 instead of python on your machine. If so, use python3 -m venv venv instead.


Next Chapter: What’s Next?

Previous Chapter: Logging in Python