Building Powerful Frameworks in Python

As you read this now, I have a feeling you don't just write software to pay the bills. I know you're the kind of developer who wants your work to have meaning - to see the software you write used widely, whether that's internally in your company, or in an open-source project you put out.

So how do we acccomplish this?

One way is to create useful frameworks - libraries providing scaffolding for building specialized kinds of software, which developers will want to use when creating applications. Properly designed, a good framework will sell itself, by making it easy to solve otherwise tricky software problems. Examples include:

Web frameworks like Flask and Django
Game development engines like Pygame or pyglet
ORMs such as SQLAlchemy and SQLObject
GUI toolkits like wxPython
Networking engines like Twisted and gevent
Testing frameworks like pytest, nose, and unittest/mock

Some of these (the ORMs and testing frameworks) pertain to a certain sub-domain of a full application, while others form a mold into which the entire software system must fit (the web frameworks and game dev engines). Others are in between. Regardless, it's useful to distinguish that most libraries are not frameworks, by this definition. Requests, matplotlib, Jinja2, Pillow/PIL, and NumPy will generally count as non-framework libraries. While both categories have their own pluses and minuses, we'll focus on frameworks in what follows.

Making Hard Things Easy

Since our goal is to increase the positive impact of the code that you, personally, will write, what makes a framework a good framework... one that other developers will want to use? It matters a lot that your framework allows someone to quickly solve important, hard problems, and do so through relatively simple code. This applies whether you are building an open-source framework you want to see widely used; or (and this is probably more important in your day-to-day work) a framework you will see adopted internally, respected and used by engineering teams in your own organization.

If the framework doesn't make it much easier for people to solve problems they actually care about, you'll have a hard time recruiting people to use it at all. There's nothing more frustrating for us than to pour our heart and soul building a software system, only to find that nobody cares.

Accessible

In designing a framework for programming in a language, chances are you are an expert in that language, its features, and its ecosystem. When people who learned Python by working through some online tutorials last week can do something useful with your framework, and more intermediate users can quickly and intuitively avail themselves of everything your framework has to offer, you have leveraged your own expertise in a powerful way - effectively lending it to everyone who uses your framework.

So how do you accomplish this? After all, if your framework is only usable by the kind of people who give talks at Pycon, you've missed a massive opportunity.

Simply put: code using your framework must be able to rely on the simple syntax features of the language. That means things like creating classes and objects; defining simple functions; calling methods; and applying decorators. It does not mean doing anything with metaclasses, implementing a custom decorator, defining a function with *args or **kwargs, or using the "yield" keyword. YOU will probably do any or all of these things, as the framework author. But the user of that framework shouldn't have to. If they do, adoption will be a fraction of what it could have been.

Python's Framework Building-Blocks

Achieving this elegance requires some work from you, the framework creator. And yet, it helps to have a roadmap of what Python language features you can learn, making it easier for you to write such exceptionally high-impact code. Dig into the source of the open-source frameworks mentioned earlier, and you'll find they rely on things like:

Writing decorators
Creating context managers
Higher-order functions
Introspection
Advanced Python object concepts, like magic methods and descriptors
Generators, and using the iterator protocol

This is evident when looking at the code created using frameworks - for example, how the app.route decorator is used in this Flask endpoint handler:

@app.route("/tasks/", methods=["GET"])
def get_all_tasks():
    tasks = app.store.get_all_tasks()
    return make_response(json.dumps(tasks), 200)

Syntactically, this is simple to write - you just type "@", and call app.route with certain arguments. But as you think about what it would take to implement this decorator, you can really see how a great deal of complexity is so nicely encapsulated here.

Or how simply declaring certain member variables lets you define a database schema in this SQLAlchemy code:

class Task(Base):
    __tablename__ = 'tasks'
    id = Column(Integer, primary_key=True)
    summary = Column(String)
    description = Column(String)

# Then later, elsewhere in the code base...
all_tasks = [{"id": task.id, "summary": task.summary}
              for task in session.query(Task).all()]

Defining the Task class is simple and easy; anyone can do so, after learning the basics of Python's object model. The complexity that makes the all_tasks query work is blissfully hidden from that user.

Opinionated Guiding

Both examples show how easy it is for someone to use the framework, getting the benefit of a staggering degree of complexity and thought which they may never realize is beneath the surface. And that's a good thing. In fact, these examples illustrate another important property of a good framework: it's opinionated, expressing an opinion that's useful over a wide range of engineering domains. General-purpose programming languages let you write the same program in infinitely varied ways; a good framework will implicitly provide guidance about how to tackle the problem, letting someone cut to the chase much faster than they could with a blank slate.

White Paper Bootcamp