Unraveling dependency management in Python

This article is about two important concepts in Python: install_requires in your setup.py file, and the requirements.txt or Pipfile. It’s based on a short Twitter thread (embedded) that I posted in November. I finally got around to converting it into a blog post.

Every time I write a Python application or library, I get confused by the differences and similarities between the setup file and the requirements file: where am I defining my dependencies? What should someone need to install my library? How do I make sure production uses the same code as my local environment? Why is this so unclear?

By now, I think I figured out the difference, in the end… Let’s talk about the end goals: defining dependencies, locking the entire tree, and integration between those; and the usecases: libraries and applications.

Locking your dependency tree

Locking a dependency tree has a simple use: making sure that all environments installed from that locked tree run the same code. This is extremely important for applications. You don’t want to risk your production environment being different from your local machine, where you ran all your tests. For libraries, wedon’t care so much about this: we defer the responsibility to the application builder. That also gives them the opportunity to update child dependencies independently.

Traditionally in Python we lock dependencies with the requirements.txt file. It allows us the possibility to keep a flat list of your entire dependency tree. Nowadays we have a neat new tool called Pipfile. It exists to make sure you only need to manage your direct dependencies, where in a requirements file, you had to manage the whole tree yourself. You can even separate your development-only dependencies from the production ones, to minimalize the size of your build artifacts. The Pipfile.lock takes care of locking the tree.

If you want to lock your dependency tree to protect your production environment: keep your dependencies in the Pipfile or requirements.txt.

Defining your direct dependencies

To define our direct dependency, we have two options. This is where the two problems overlap. You can define these in the install_requires option in your setup.py file or alternatively you can track it in the Pipfile. Thesse have different uses.

The setup.py file is used to install your package, as a library. Neither the Pipfile nor the requirements.txt file are used during that process. In fact, they shouldn’t be used! These files define a dependency tree, which is way too specific for library installation. You want to defer the management of those dependencies to your library’s user, as much as possible.

When you’re defining the direct dependencies of your package, use the install_requires parameter in setup.py.

Integrating between the two

So now you get to the point where you have a setup.py for installing your package and a requirements.txt or Pipfile for local development. You’re probably mirroring a bunch of libraries between the two, so you might consider integrating them: maybe the setup.py can read from the requirements.txt?

What you’re actually trying to do here is probably wrong: if you’re building an application, you probably don’t actually need setup.py and you can stop here. If you’re building a library, you don’t have to mirror your dependencies.

Dependency management in libraries

I love keeping a Pipfile in libraries still, because especially with pipenv it’s super easy to manage your environments and dependencies. With a requirements.txt, the gains are a bit less, but they definitely still exist! Here’s how to integrate with each:

Pipfile

If you make sure your Pipfile refers to your own package as an editable dependency, it’ll automatically use the setup.py to install the dependencies. This makes sure you’re using the same way to install your module as your users will! No need to lock any versions at all.

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
sample = {editable = true, path = "."}

[dev-packages]
"flake8" = "*"

[requires]
python_version = "3.7"

Note the sample package in there. This should be your only line under [packages] when building a library. Since the package will be installed through the normal setup.py install method, you’ll still get the requirements listed in install_requires. You also get the benefits of the Pipfile, plus any development packages in dev-packages.

If you prefer (I do), you can let Pipenv add this line for you with pipenv install -e ..

Since the lockfile is separate for Pipfiles, you don’t have to worry about the lockfile for libraries. You can .gitignore it or not, it doesn’t make much of a difference.

requirements.txt

We’ll do the same here as we did with the Pipfile, just with a little bit different syntax:

-e .
flake8

The -e . here is the same as the sample package before: it installs the package in the current directory through setup.py install.

When building a library though, be careful to never run the commonly touted pip freeze > requirements.txt, as it means you’ll have to manage all dependencies yourself, or rebuild your requirements.txt. Just add any development dependencies under the -e ., and install them with pip install -r requirements.txt. You can lock them to specific versions still, if you like.

Dependency management in applications

Pipfile

This is a lot simpler: install dependencies with pipenv install and development dependencies with pipenv install --dev. Be sure to commit your lockfile into version control, and install on production with pipenv install --deploy.

requirements.txt

This one is a bit harder, since you’ll have to manage the locking yourself. You can of course use a single file for your direct dependencies (requirements.txt) and a second file for the “lock” (requirements-locked.txt). In this case, you’d generate the second file with pip freeze > requirements-locked.txt. Keep in mind though, that this freezes all installed packages, not just the ones in requirements.txt. If you drop a dependency, you’ll have to make sure to uninstall it and its dependencies from your environment manually.

setup.py

Whoa, hold it! Didn’t we just say we’re not using a setup.py for applications? Well, yes, but of course if you want to, you could. Maybe you have an application that can also function as a library, or… well, I’m sure there’s other reasons that I haven’t found yet.

In this case, you should simply use the library approach and combine it with the locking mechanisms from the other application approaches. Install on production with pipenv install --deploy, or through fully locked pip install -r requirements-locked.txt.

Closing remarks

I’ve long thought the Python packaging setup to be a complete mess. Over the course of this research I’ll admit that it’s actually quite nicely designed, but simply not very intuitively. It’s flexible to a point of confusion. Using these methods of dependency management work nicely for me, but I’m sure they don’t for everyone.

Either way, I hope this helps some people to not have to go through the mess of dependency management that I’ve gone through in earlier days.

A brief summary of the OAuth 2.0 RFC

This is the briefest useful summary I could come up with for OAuth 2.0, after reading the original RFC.

Read more…

I'm bragging about giving to charity, and you should be too!

Almost a year ago, I started working at 3D Hubs. Since I like domino-effects in life-changing events I decided that besides finally entering start up life, I was going to pursue two other goals: getting fit, as one should in the startup world; and studying computer science more.

Full of confidence, I signed up for a year long gym membership and an at-home bachelor course for computer science. I got to work and continued studying and working out for… two weeks, maybe? Then, of course, I quit. The contracts though, were quite strict: I was stuck to paying for a full year.

A few weeks ago I realised that I will soon be done paying off my self-imposed fine, and I’ve been considering what I can do with this money. I could of course buy more things with it, but that doesn’t seem quite like the wise, adulty thing to do. And then the Mozilla rebranding happened, and suddenly everyone was talking about Mozilla (and of course Firefox).

However you may feel about Firefox, it should be impossible to deny that Mozilla is doing good work. As a (somewhat) large player in the browser market they have a very effective vote in web standards, and they’re always pushing for an open web. And now they’re finally putting more effort into improving Firefox as well! I’m betting that Mozilla has a very good influence on the internet.

Setting up monthly donations to Mozilla was a two minute task, and it doesn’t take away from my monthly purchasing power. Very quickly after followed Wikimedia—I was browsing Wikipedia for information on Mozilla. And now I think I might be getting addicted to charity.

After Mozilla and Wikimedia followed Vluchtelingenwerk, a Dutch charity supporting refugees. With conservative voices seemingly getting louder and louder—especially now with Trump in office—helping organisations that support refugees, minorities, and freedom (as in speech), seems increasingy important.

Giving small amounts of money each month to charitable organisations is a really easy and incredibly effective way to do some good. So far, I’m donating about €20 each month. Don’t be fooled by small amounts: every little bit always helps, and you can always raise your gifts later.

To reallocate the money I was throwing away before, I want to continue setting up periodical donations over the next few weeks. You can keep up with my list of charities on my website.

If that list looks like bragging, it’s because I am. People need to be reminded that charitable giving is normal. I am going to continue bragging, very publicly, and I encourage you to do the same: spend your money on things you believe in, and tell the world.

Commenting: documentation and justification

Commenting is a somewhat controversial topic, with opinions spread out over mainly a single axis from those against and those in favour of commenting. The one extreme says one shouldn’t have to comment at all since your code should be readable enough to explain itself (I think of this as the arrogant approach), while the other extreme claims that reading code is always bothersome and some things will never be easily understandable so there should be plenty of comments to explain it (in my mind, the hand-holding approach). I’d like to propose a somewhat formal definition of the already existing middle ground: the Documentation-Justification Method of Commenting Code.

The way I see it, there are two distinct commenting styles: describing the usage of your code, which I’ll refer to as documentation, and elaborating on your implementation, which I call justification. I’ll use these terms in my explanation and reasoning below, but they can also be directly linked to more practical conventions, which I’ll also expand on.

Documentation describes the usage of your code and should be very obvious to any developer. Documentation tells the reader what they can expect from a class, function or otherwise. It can define what something should be used as or used for. It can also (and this may seem like a bit of a gray area here) expand on the structure of your code: why did you create x as a class, why is this a separate function. This is because it’s very important to understand the structure of the project to fully understand the usage.

Justification tells the reader why you chose to implement something the way you did. While your code should be readable enough to describe exactly what it does step-by-step with a single read-through, it will never be able to explain your reasons for choosing a certain implementation. There are many obvious things choices that don’t need justification, and you shouldn’t justify those. The usual example is x++; // increment x by one, but there’s a lot more complex code that doesn’t need commenting. It is of course up to the writer to decide what code does and doesn’t need justification.

Justification is actually a little broader than that though: it may also be used to explain a certain way to do something that is specific to your framework that may not be obvious to the reader at first (but don’t repeat the justification every time you use it). This one is a little further away from the exact meaning of “justification”, but it should still be considered the same commenting style since they are very closely related: they both comment on a certain bit of code, instead of the structure. The term justification also seems more appropriate and immediately clear about its primary purpose.

Most programming languages will offer you (at least) two methods to comment your code: block comments, often /* like so */, and line comments // like this or # this. Now that we have defined two separate commenting styles, we can directly link them to comment methods. Block comments should be used for documentation, whereas justification should use line comments. This also more clearly links them to their meaning as described here: documentation describe “building blocks” of your code and justification is more closely related to the actual lines of code.

The DJ method and more importantly this write-down comes from my own struggle with commenting methods, and my inability to pick what I should do. With a somewhat formal definition, I can enforce (or at least encourage) a certain commenting method in my own projects and those I set up in the workplace. Separating commenting into two hopefully clear styles should make the method simple enough to understand, and maybe simple enough for others to consider using.

Thanks for reading. If after reading it turns out you hate me and you want to yell at me, I recommend sending me a tweet.

Oddities of animating auto values in CSS

CSS is broken. It’s not possible to animate, in any way, an auto value. Attempting to animate to or from height: auto or width: auto will yield naught but disappointment, distress and questions: how do I fix it? Why doesn’t this work? What’s going wrong here?

Read more…