Pydantic can do what?!

Pydantic has grown so Fast, and it now contains a full-featured settings loader system with support for getting values from env vars, multiple dotenv files, config files, and cloud vaults.

In fact, it can even parse CLI arguments now.

Today, you can effectively replace calls to environs/dotenv, click/typer/argparse, tomlib/yaml/json, and most parsing+validation code with a Pydantic class.

And it will cascade all those, deal with missing sources, handle priorities, understand deeply nested values, and censor secrets.

It’s quite good, I would say.

Pydantic started as a simple validation library, like marshmallow, and is mostly used as such today. You first define a schema:

Then you validate data with that schema. If it works, you get a regular object you can manipulate that you know contains exactly what you want:

And if the data is invalid, it will tell you so, and what the problem is with it:

It can also serialize/deserialize data to JSON and is compatible with type hints, for this reason, it was eventually adopted by FastAPI to define endpoints in a succinct and elegant way, and became very popular.

So they added more stuff.

Over time, Pydantic has accumulated more and more features, and one day made an innocent addition: a small class dedicated to loading values from environment variables and validating them. It has since grown, and grown, and it’s now an unexpectedly excellent settings loader system.

It has become so much more than it was at the beginning, in fact, that in 2023, the code was extracted into its own separate package: pydantic-settings.

Here is a simple example:

In a few lines, you already get so many things:

• This will load all values from a /etc/.env then a .env file if they exist.

• This will then load all values from environment variables. They override the previous values.

• This will then override those with any parameters manually passed to Settings.

• Then it will deserialize all those values into proper Python types. upload_dir will be a Path, for example.

• And finally, it will perform validation. Check if the port matches the given range. That the log level is one of the allowed values, etc.

That packs a punch.

It’s not even limited in format; you can load from any source. The lib supports JSON, TOML, and YAML out of the box, but you can inherit from PydanticBaseSettingsSource and create your own loader. You can even change the order in which sources are loaded by overloading the settings_customise_sources method on your settings class.

But the cool thing about it is that despite the fact that I have been using this for quite some time, every time I come back to the doc, there is something new.

Recently, I learned that, lo and behold, Pydantic settings can now handle command-line arguments. You just need to add one parameter to DictConfig:

And boom:

So now you load, convert, and validate cascading settings from 2 dotenv files, env vars, and the CLI. If that’s not beautiful, I don’t know what is:

And the CLI feature comes with many tricks in itself: sub-commands, kebab-case, passing lists/json to set complex values, aliases, --no-flag, positional args, mutually exclusive groups, argparse integration...

Have you noticed the token is marked as a secret? That’s why if you print it, it will show ***. You need to actively call get_secret_value() to read it, so it doesn’t end up in some log by mistake.

There is a whole part of the lib now that is dedicated to the retrieval of secrets. I can read Unix secret files, docker secrets, AWS secret manager, Google Cloud Secret Manager, and Azure Key Vault, but as you can imagine, you can create your own.

Of course, the community did, so you can also load your token and passwords from a Hashicorp vault

Ironically enough, I was a critic of Pydantic when it came out and preferred to use alternatives. I’m happy to report I’ve done a full 180 on this and use it everywhere.