Required cleanup

My man Malcolm asks:

"How do we cleanup virtual env's and requirements.txt files, after you've stopped using specific modules?"

Great question, and practical too.

I'll tell you the best strategy I've found:

(And I learned this from Kenneth Reitz, a talented Python dev who's written lots of great code you're probably using.)

The best approach I've found is to have TWO requirements files.

You have "requirements.txt", which has all your required modules, their dependencies, at their fixed versions. Generated and updated by "pip freeze".

And you have another file, called "requirements-to-freeze.txt". This is NOT written to by pip.

Why you have two:

You want to separately track the modules your app depends on...

AND their dependent modules, along with all the precise versions numbers of everything. So you can repeatably rebuild the virtual environment, on different development machines and deployment environments.

So in requirements-to-freeze.txt, you write the app's module dependencies - just the module names - one per line. For example

pyyaml
requests
django
pandas

You then run "pip install -r requirements-to-freeze.txt" to create the virtual environment. Then do "pip freeze > requirements.txt", to store the exact module numbers - it'll look something like this:

certifi==2019.3.9
chardet==3.0.4
Django==2.1.7
idna==2.8
numpy==1.16.2
pandas==0.24.2
python-dateutil==2.8.0
pytz==2018.9
PyYAML==5.1
requests==2.21.0
six==1.12.0
urllib3==1.24.1

You check BOTH these files into version control...

And imagine that later, you decide to split off the data-analysis code into a separate microservice. So you no longer need Pandas in this project.

So you remove "pandas" from requirements-to-freeze.txt...

Then you DESTROY the virtual environment...

(Necessary because "pip uninstall" doesn't remove dependencies)

And recreate it with, once again, with "pip install -r requirements-to-freeze.txt" followed by "pip freeze > requirements.txt". Now requirements.txt looks like:

certifi==2019.3.9
chardet==3.0.4
Django==2.1.7
idna==2.8
pytz==2018.9
PyYAML==5.1
requests==2.21.0
urllib3==1.24.1

See how numpy, pandas, python-dateutil, and six are all removed? So you check both of these updated requirements files into version control.

(Pip's native caching will make it run pretty fast the second time; you probably won't have to re-download anything.)