How to Write Deployment-friendly Applications

The DevOps movement gave us many ways to put Python applications into production. But how can you practically structure and configure your applications to make them indifferent to the environment they run in? How do secrets fit into the picture? And where do you put that log file?

So far, I’ve held it at PyCon US 2018 in Cleveland, USA, EuroPython 2018 in Edinburgh, UK, PiterPy #5 in Saint Petersburg, Russia, and PyCon BY 2019 in Minsk, Belarus.

Slides on Speaker Deck

General

• A lot of the talk overlaps with the 12 Factor App Manifesto.
  • Practical addendum by Kelsey Hightower: 12 Fractured Apps
• On Designing and Deploying Internet-Scale Services by James Hamilton of the Windows Live Services Platform team. Long PDF, lots of wisdom.
• Example of a modern Python application that serves 6 billion requests per month: PyPI/Warehouse.
• You probably don’t have to expose so many settings as this paper shows: Hey, You Have Given Me Too Many Knobs!

Configuration

Environment variables are the most portable and universally available way to pass key-value pairs between processes and environments.

• Setting:
  • direnv is one of many tools that will set env variables as soon as you enter a directory. If you rely on env variables for configuration, this is very handy.
  • envconsul HashiCorp’s official Consul-to-env and Vault-to-env (don’t!).
• Consuming:
  • environ-config allows you to got from a lowest possible denominator (environment variables) to well-known structured data (nested Python classes).
• Templating:
  • envsubst – part of gettext – allows for env variable → file templating. Once the barrier to the runtime environment is passed, you can have files if you really need to.
  • confd supports templating from many backends including Consul, etcd, ZooKeeper, Redis, and – of course – environment variables.
  • consul-template is HashiCorp’s official Consul templating tool.

Secrets

• Why you shouldn’t use ENV variables for secret data by Docker’s security lead Diogo Mónica.
• Behind Closed Doors: Managing Passwords in a Dangerous World by Noah Kantrowitz is a general introduction into secrets management.
• HashiCorp’s Vault is probably the currently most advanced secrets manager that is not bound to a platform or runtime.
• Encoding secrets as URLs is handy because you can atomically change multiple parts at once.
  • I like yarl for parsing and manipulating them.
  • If you need Python 2 support, hyperlink is a good option too.

Load Balancing

Load balancing is the best way to treat your applications as simple blocks and makes zero-downtime deployments, scalability, and fault tolerance Someone Else’s Problem™.

• I insist on using TLS in backend communications.
  • Defense in depth
• Excellent and exhaustive Introduction to modern network load balancing and proxying by Matt Klein, the author of Envoy.
• For client information like the protocol or the client’s IP address, your load balancer has to set certain headers and your you – or your web framework – has to interpret them.
  • Currently mostly X-Forwarded-For, X-Forwarded-Host, and X-Forwarded-Proto.
  • Since 2014 there is an RFC for a more versatile Forwarded header. Unfortunately it’s not as widely supported yet.
  • Load balancing and client information relaying is not just for HTTP: HAProxy’s PROXY protocol for TCP- and UDP-based services.
  • Clean shutdown is an important part of an application’s lifecycle. Make sure your application reacts to the signals your process- or container manager is sending to it.
    • Why Your Dockerized Application Isn’t Receiving Signals

Introspection

Being able to look directly into applications instead of relying on side-effects is a very powerful concept. And it’s quite easy to achieve even with just the Python standard library by exposing an HTTP endpoint.

• healthz: Stop reverse engineering applications and start monitoring from the inside by Kelsey Hightower (yes, yes I’m a fan) is a great primer into readiness and why it matters.
• Kubernetes documentation on Liveness and Readiness offers a decent explanation between the two.
  • This Google Cloud blog post is even more approachable: Kubernetes best practices: Setting up health checks with readiness and liveness probes.
• Mozilla’s approach with their Dockerflow.

Just Another Boundary

Language/runtime boundaries are just another boundaries. It’s worthwhile to study and apply general software engineering principles.

• Hexagonal Architecture by Alistair Cockburn.
• Boundaries by Gary Bernhardt.
• The Clean Architecture in Python by Brandon Rhodes.

Credits

• Relaxing in paradise by Simon Migaj.
• Red and gray big brick wall by Mike Setchell.
• Whispers in the street by London Scout.
• The doctors room … by Hush Naidoo.
• Lava fountaining within the Nyiragongo crater, DRC by Pierre-Yves Burgi.
• All logos and trademarks belong to their respective owners.