My Favorite Underused Python Idiom

While I can program solo just fine, I always prefer to develop as part of a team. Among other reasons, it teaches me a lot very fast. Some of my most significant growth as an engineer has come from having my code reviewed (or reviewing someone else's), or talking through a design on a whiteboard, or having simple conversations with my teammates about how we might implement something.

An exciting point is when I learn some new idiom or pattern that has been right under my nose the whole time... and now that I'm aware of it, will improve my Python code literally for the rest of my coding life. The best deal with mundane, everyday tasks.

A great example: opening and reading data from a file. Here's how a lot of people learn to do it: n

f = open(path, "rb")
result = do_something_with(f)
f.close()
print(f"Got result: {result}")

This works just fine. However, it has a few potential issues. First, when writing this code, it's possible I might forget to close the file object. At the very least, remembering to close it is a distraction from your focus.

The second issue is more subtle. While altering this code in the future, either myself or my teammate may add code after the second line, pushing the f.close() call down. This leads to the file being open longer than it needs to be. That may be harmless; but certainly no good can come from keeping it open longer than necessary.

And finally the worst problem: if an exception is raised before that close() method is called. That can lead to the file being open far longer than you want it to.

(This is ignoring other poor practices, like deliberately closing the file many lines of code later, long after any reading or writing is finished; or even not calling the close method at all - letting Python implicitly close it when the file object goes out of scope. Again, there are situations where doing these things is harmless, especially in smaller scripts. But in general, no good can come doing these things.)

Fortunately, Python provides an idiom that solves all these problems, using the with keyword.

with open(path, "rb") as f:
    result = do_something_with(f)
print(f"Got result: {result}")

Some of you have seen this before, but you may not yet be aware of all the nuances making this even more awesome than it appears at first glance. Here, the open function is acting as a context manager. The short version: the object returned by open() defines two methods, called __enter__ and __exit__. The former returns the file object - i.e., self - which is assigned to f. The __exit__ method is called once the with block exits; in this case, it closes the file. ¹

The punchline is that when you use with to open your files in Python, that file is automatically closed, as soon as it's no longer needed. There is no way you can forget to close it! It's the best way to open files in Python today, in most situations.

Notice result is defined inside the with block, but remains accessible once that block is exited (and the file object is auto-closed). This is the normal Python scoping rules at work. If you create (initially define) a variable in a block, it's also defined in the parent blocks - up until the function boundary. This is why the print() line is dedented. You don't want to do something like this:

# *** BAD CODE! ***
with open(path, "rb") as f:
    result = do_something_with(f)
    # Unnecessarily extending the with block keeps
    # the file open longer than it has to be.
    print(f"Got result: {result}")

Keep the with block open only until you've got what you need from the file, then break out of it and move on.

P.S. You can create your own version of open() like this:

class myopen:
    def __init__(self, path, *args, **kwargs):
        self.path = path
        self.args = args
        self.kwargs = kwargs
    def __enter__(self):
        self.file_obj = open(self.path, *self.args, **self.kwargs)
        return self.file_obj
    def __exit__(self, *optional_exception_arguments):
        self.file_obj.close()

with myopen(path, "rb") as f:
    result = do_something_with(f)
print(f"Got result: {result}")

If an exception is raised inside the with block, __exit__ will be passed some related arguments. Some context managers do something with that, but in this case we ignore them.