Scripting good practices in Python

You can make your scripts just a little nicer with some tweaks:

• Use inline dependencies if you need any.

• Acquire and store secrets in a way that won’t leak them.

• Make a clean split between print and log.

• Document and prefix your env vars.

• Support piping.

• Exit cleanly.

• Load configuration in the proper order.

It’s been easier than ever to create a Python script. We have uv, inline deps, tons of fantastique 3rd party libs for easy argument parsing/env var reading/configuration loading, and of course we can ask an AI to generate it.

When something becomes easy to do, a lot of non-experts start to do it. It’s good, but it also means they don’t know the industry standard. The AI can do a lot, but you need to ask it to do so.

So today we are going to go through a list of things you can do to make your scripts nicer.

Claude and ChatGPT are fully aware this exists, but almost never do this.

We now have a specification to list your script dependencies directly in a comment at the top of the file. If you script requires requests and keyring, you can do:

pip added support for this recently, and you can therefore install all dependencies of this script with --requirements-from-script. I would still advise doing that in a venv.

Of course, it’s even better if you use uv, since you can then just run the script transparently with:

And uv will automatically create a temporary venv, install all the packages, and run the script in one operation, very quickly.

Sometimes your scripts will require you to get some token or password that should stay safe. In that case, it is better if you don’t hardcode it or even pass it as a parameter (so it doesn’t show in the shell history).

What can you do then?

Well, you should read it from an environment variable, and if this is not provided, prompt for it with getpass.

Now, since it’s annoying to make sure the token is available every time, you can store it in the user's OS keyring, where it will be safely encrypted and stored. This is usually automatically unlocked at login, and you can use a 3rd party lib from PyPI to read from it. The example becomes:

If the secret is very big, like an entire file, you can even encrypt it with cryptography.fernet, and just save the encryption key in the keyring.

This is, of course, not bulletproof, but it’s better than 99% of the scripts you will see out there.

At first, you should just print. A simple script should stay simple, no need to make it super complicated for no reason.

But if your script becomes a bit more advanced, then you should split it into two types of feedback:

• Feedback for the user of the script, the people running it (it can be you, later). Use print().

• Feedback for the dev of the script, the people writing and debugging it (it could be you or your user trying to figure out what the hell is going on). For that, use logging.

You’ll note that by default we store a log level of 9999. That’s because we disable the logs, and just let the user messages. The logs are activated ONLY if someone requests them, otherwise, they are noise:

For a simple script, we don’t store the logs in a file. Indeed, a user who can set an env var has the knowledge to store all logs in a file by doing a shell redirection, and this saves you from the complexity of deciding WHERE to put said file yourself.

Env vars cannot be magically discovered; you need to advertise this somehow. I recommend documenting all env vars as parameters on --help. You can actually add free text to any argparse parser, even if you don’t have any parameters!

This is now documented:

Because stderr stands for “standard error”, we tend to think that it’s made for outputting errors. But it’s a bit more complicated than this. E.G: all logs go to stderr by default, even INFO.

This is because a script's output is often redirected, and the most common redirection is to split the data stream of stdout from the one on stderr, to send them in two different places (E.G: a pipe and a file).

A good rule of thumb is that operations results go to stdout, errors and logs to stderr.

How do you write to stderr?

Since this is a tad annoying to type every time, you can make a partial for it:

Since it’s nice to be able to see errors for the user from miles away, you can color them in red. That can be done manually:

But better use a lib like rich that will handle edge case like old windows setup, no tty, pipes, etc.

Whether or not to use a library depends on how feature-rich you want that to be for the time you spend, and how annoying it is to install dependencies for the target users.

It’s convenient when you use a shell to be able to pipe data to a script, so if your script is supposed to accept a big input (like a file, for example), give the option.

This must be done carefully, though, as you don’t want to read stdin if it’s coming manually from the user, so we must guard it:

This will let the user do:

All programs end their execution by returning a code, which is 0 when it went well, and between 1 and 255 if it ended with an error.

This is useful for automation since other shell commands use those codes to make decisions, and a well-behaved script should therefore choose to return codes on exit.

So you should at least use sys.exit() when you stop the script because something is going wrong:

You may call sys.exit() with a number instead to choose precisely the code, but then you should document said code in your --help to state what they are for. This is probably more trouble than most people want.

I also advocate for shipping programs that don’t print a stack trace on crash. It’s much better to hide it unless the information is requested (E.G: for debug), because they often don’t know what to do with the info, at best.

Instead, set up logging, swallow the exception, and then, if logging is activated, you will see the stack trace:

The result:

Configuration should almost always be defined by order of priority from (highest to lowest):

• Manual user input when prompted

• User passed arguments from the command line

• Env variables

• Local configuration file

• User configuration file

• System configuration file

• Default value

This means that if you have a variable at the system level, a user configuration file can override it, which can be overrided by a configuration file in the local directory, which should be also easy to override with a any env var that exist in the current session, which will be overrided by any argument passed at this run, which, finally, will be overriden by user choices right now. And if nothing is provided, you use the default value.

You don’t NEED to have that many configuration layers. You can have 1, 2, or none. It’s fine.

But if you have several of them, they will conflict, and you MUST have a clearly defined order for resolving that, and this is standard.

E.G, if I have a retry variable with a default value, that can be set by env var and CLI params, I should do:

Which effectively gives you:

❯ python script.py
RETRY = 3
❯ YOUR_SCRIPT_RETRY=5 python script.py
RETRY = 5
❯ YOUR_SCRIPT_RETRY=5 python script.py --retry 10
RETRY = 10

Yes, you should probably not. Use each improvement only if you need it. Again, a simple script should stay simple.

Plus if you feel really lazy like me, you can let pydantic do a lot of it.

And lo and behold, we have a great article to explain how to do that:

Pydantic can do what?!