Python Virtual Environments
Python sports a concept called the virtual environment. It is sometimes called a "virtualenv" or a "venv". You can think of a virtual environment as a kind of lightweight container for a Python application, where the application can see its own particular set of Python libraries, at specific versions. This provides significant benefits for deployment:
- Dependencies can be precisely tracked and specified, and even kept in version control.
- Two different Python applications with conflicting dependencies can peacefully coexist on the same machine.
- Python packages can be installed without requiring elevated system privileges.
Let’s dig in.
Creating New VEnvs
You create a virtual environment by invoking python -m venv, adding
one more argument - the name of a folder to create:[36]
$ python -m venv webappenv
What this does is create a folder named "webappenv" in the current
directory. You end up with a new folder named webappenv, containing
all sorts of goodies. To access them, you must run a script called
"activate". In macOS and Linux, type the following in your shell:
$ source webappenv/bin/activate (webappenv)$
For Windows, run the script webappenv\Scripts\activate.bat instead:
C:\labs> webappenv\Scripts\activate.bat (webappenv) C:\labs>
Notice your prompt has changed: now it contains (webappenv). The
script called activate did not start a new shell or session; all it
did was alter your PATH and PYTHONPATH environment variables, and a
few others (like the PS1 variable, which specifies how your shell
renders the prompt.) You just activated your virtual environment, as
we say. When it’s active for that particular shell session, we say you
are working in the virtual environment.
Suppose your system Python executable is at
/usr/bin/python. And suppose
you’ve created the webappenv folder in /Users/sam/mywebapp/webappenv. With the
virtual environment activated, you have your own local copy of the
Python interpreter. You can check this with the which command on macOS
and Linux:
(webappenv)$ which python /Users/sam/mywebapp/webappenv/bin/python (webappenv)$ python -V Python 3.11.4
Or the where command on Windows:
(webappenv) C:\labs> where python C:\labs\webappenv\Scripts\python.exe C:\Python311\python.exe
Now, what if you want to restore your old PATH
(and PYTHONPATH, etc.)? Within the virtualenv, you now have a function
defined called deactivate:
(webappenv)$ deactivate $ which python /usr/bin/python
Now imagine you are writing a Python application: let’s call it
mediatag, one designed for tagging files in a media
collection. And you have not had time to test it on the latest Python
version, so for now you want to just run it in Python 3.10 - installed
on the system under /usr/bin/python3.10.
Set up the virtual environment like so:
$ cd /Users/sam/mediatag $ python3.10 -m venv mediatagenv $ source mediatagenv/bin/activate (mediatagenv)$ which python /Users/sam/mediatag/mediatagenv/bin/python (mediatagenv)$ python -V Python 3.10.12
This shows one benefit of virtual environments: controlling the precise version of Python used by an application. But the main benefit for using virtual environments has to do with resolving requirements, upgrades, and dependencies with packages and libraries.
Python Package Management
The sordid history of Python library and package management is full of hidden twists, perilous turns, and dark corners hiding sinister beasts. Good news: you don’t have to worry about any of that.
Modern Python provides an application called pip,
which allows you to easily install third-party Python libraries and
applications. It incorporates many of the lessons learned from its
predecessors, sidestepping problems that previously had to be dealt
with manually. And it works very well with Python virtual
environments.
The first step is to install pip. With Python 3, this is included
for you automatically, and is installed in your virtual environment:
$ source venv/bin/activate (venv)$ which pip /Users/sam/myapp/venv/bin/pip
You can use pip install to install libraries just within this
specific virtual environment. For example, requests is a
high-quality HTTP library. Install it like so:
pip install requests
This is the install command. You will see some output, narrating the
process of installing requests at a specific version. Once complete,
you will be able to open a Python prompt and import requests.
The pip install command is also used to upgrade packages. For
example, sometimes a fresh virtual environment may install a slightly
stale version of pip. pip is just another package, so you can
upgrade it with the -U or --upgrade option:
pip install --upgrade pip
Installed packages are, by default, fetched from Pypi - the official online
Python package repository. Any package or library listed at
https://pypi.python.org/pypi can be installed with pip. You can
uninstall them with pip uninstall.
Now, some of these packages' files are substantial, or even compiled into object files. You definitely don’t want to put them in version control. How do you register the exact version dependencies your app has for third-party libraries? And how do you manage upgrades (and even downgrades) over time?
pip provides a good solution for this. The first part of it relies on
the pip freeze command:
(venv)$ pip freeze certifi==2023.11.17 charset-normalizer==3.3.2 idna==3.6 requests==2.31.0 urllib3==2.1.0
This prints the packages installed from Pypi, one per line, with the
exact version. Notice it installs requests, and also automatically
installs the other libraries that requests depends on. What you can do
is place this in a file named requirements.txt:
(venv)$ pip freeze > requirements.txt (venv)$ cat requirements.txt certifi==2023.11.17 charset-normalizer==3.3.2 idna==3.6 requests==2.31.0 urllib3==2.1.0
This file is what you will check into version control. You can
recreate the application environment, right down to the minor versions
of each dependent library, simply by passing requirements.txt to
pip. Whether your coworker is fetching the raw source to start
development, or if the devops team sets up a CI environment that
runs the automated tests, the environment is consistent and
well-defined, from development to staging to production.
You can pass any file path to python -m venv. For organizational
convenience, many choose to put this virtual-environment directory in
the top-level folder of the repository holding the Python
application. There are two schools of thought on what to name it.
One school picks a consistent name, which is used for every project. "venv" is very popular:
python -m venv venv
The idea is that every Python project will have a folder in its top
level called venv to contain the virtual environment. This has
several advantages. For one, you can easily activate the virtual
environment for any application, just by typing source
venv/bin/activate. No wasted mental cycles.
You can also configure your version control system to ignore any folder named "venv", and thereby avoid ever accidentally committing your virtual environment. (You don’t want to do that. It’s a lot of files, and the files will probably not work at all on a different machine.)
The other naming scheme is to give it a name that has something to do with the application. For example, for an application called "mywebapp", you might create it like this:
python -m venv mywebappenv
The advantage of this is that, when activated, the prompt is modified to tell you which particular virtual environment your shell prompt is using. This can be helpful if you work with many different Python applications, as it’s much more informative than a prompt that just says "(venv)".
The downside comes from the inconsistency of the folder name: keeping the folder out of version control is more error-prone, and activating the virtual environment requires the distraction of conscious thought each time. Both approaches are valid; it really comes down to which you and your teammates like better.
[36] Modern Python may be installed as python3 instead of python on your machine. If so, use python3 -m venv venv instead.
Next Chapter: What’s Next?
Previous Chapter: Logging in Python