Agentically fixing 159 bugs

This past week, I have been working on a codebase with almost 200 different Python files - a pretty hefty project, as part of some consulting work.

For the most part, it's a well-written, good-quality codebase. But reading through, I kept seeing this dreaded phrase:

"except Exception:"

If you don't know, this is just about the most diabolical antipattern you can possibly use.

The better practice is to catch the most specific exception type you can, like "except ValueError" or "except duckdb.Error" or whatever is pinpointed to the actual exception you want to handle.

"except Exception" catches every unrelated exception, too, and is a great way to hide bugs from yourself until they're wreaking havoc in production.

Like I said: diabolical. And when I cracked open this codebase, I found "except Exception" a whopping 159 times.

Yikes.

My immediate priority: Systematically go through the codebase, and tighten up every exception block.

Meaning, replace that "except Exception" with the most specific exception type possible, like "except ValueError" or "except FileNotFoundError".

This is not the most straightforward process, because how do you know what exception type to replace it with? You have to study the lines of code in the "try:" block, figure out what exceptions they might throw, then look at what the "except" block is doing (and why) - all before you can finally decide the best exception to actually catch.

And since you typically have to rummage through several other source files to figure all this out... Just for one "except" block...

It can be tedious.

So while I can do it manually, instead I fired up an agentic AI coding tool, and gave it this prompt:

"This codebase repeatedly catches the Exception class. A generally better practice is to catch a more specific exception type instead. Find every place in the codebase where Exception is caught. Attempt to identify the most specific and narrow exception type which can be used, modifying the "except" clause and any related code as needed. Only allow Exception to remain if it is truly necessary for what the program and that part of the code needs."

And you know what?

It did both good, and not so good.

Some notes:

• It updated 118 out of 159 of those "except Except" blocks. About 74%. Impressive, but also surprised me that it was unable to convert so many.
• There was huge variation on repeated runs. 74% was the best run; the worst run converted about 50%.
• Many of the changes were... I want to say bureaucratic. They over-cautiously caught not just one exception, but sometimes many, without insight into what the "except" block was meant to actually accomplish.

As an example of the last one, it would sometimes come up with lines like:

except (OSError, pickle.UnpicklingError, EOFError, ValueError):

When it really just needed to catch ValueError. Just because a file handle somewhere could theoretically reach end-of-file, does not mean this except block was intended to handle that. But the agent could not tell.

More notes:

• This codebase had a fantastic and thorough test suite, which thank God, because it helped a lot. The agent was able to make changes; run pytest; identify what broke; and then fix it.
• No unit test could really tell the agent when it added too many exceptions to the catch-list. Several times, it inserted "RuntimeError" or "NotImplementedError", for example. And if you don't know, there is almost never a good reason to catch either of those; doing so will create the most baffling bugs you have ever seen, at least in single-threaded programs.
• In the first pass, it only used built-in Python exceptions, like ValueError and OSError; it mostly could not use exceptions in third-party libraries, like DockerException (from the docker library) or PyMongoError (from the MongoDB library). Though it was able to use AMQPError, from Pika (the RabbitMQ client library), for some reason.
• The only way to be confident in a given change was to look carefully at it. If the agent changed "except Exception" to "except OSError", is that the best exception to catch at that spot? Why or why not? It was almost never immediately obvious, just from looking at the try/except block itself.

So was it better to use the coding agent for this task?

Or would it be better to use grep (see P.S. below) to find each spot, and then change it myself?

Having finished now, I am glad I had the agentic tool's help. I would rather have had it, than not.

But it is also clear that without good software-engineering skills, it would have been worse than useless.

I mean that literally. It made many changes that seem fine, even made the unit tests pass, but introduced grievous bugs that would rip you to shreds later. Without the ability to recognize that, you'd be much better off not using the agent at all.

All this with a very state-of-the-art model, at least as advanced as what the general public has access to right now.

My verdict is that software development skills are more important than ever before.

Agentic coding tools, which I am now using almost daily, are only amplifying my abilities, far beyond the "vibe coders" who can barely program their phone alarm.

P.S. Here's the grep command I used to find all the "except Except" blocks, before I even started up the LLM - actually a combo of 'find' and 'grep':

find . -name "*.py" -print0 | xargs -0 grep -nrH 'except Exception'

Try it on your own favorite codebase, if you dare.