CLI tools hidden in the Python standard library

Seth Michael Larson pointed out that the Python gzip module can be used as a CLI tool like this:

python -m gzip --decompress pypi.db.gz

This is a neat Python feature: modules with a if __name__ == "__main__": block that are available on Python's standard import path can be executed from the terminal using python -m name_of_module.

Seth pointed out this is useful if you are on Windows and don't have the gzip utility installed.

This made me wonder: what other little tools are lurking in the Python standard library, available on any computer with a working Python installation?

Finding them with ripgrep

I decided to take a sniff around the standard library and see what I can find.

Jim Crist-Harif pointed me to python -m site, which outputs useful information about your installation:

python3.11 -m site

sys.path = [
    '/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11',
    '/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python311.zip',
    '/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/lib-dynload',
    '/opt/homebrew/lib/python3.11/site-packages',
    '/opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages',
]
USER_BASE: '/Users/simon/Library/Python/3.11' (doesn't exist)
USER_SITE: '/Users/simon/Library/Python/3.11/lib/python/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

This showed me that the standard library itself for my Homebrew installation of Python 3.11 is in /opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11.

So I switched there and used ripgrep to find likely packages:

cd /opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11

Then ran rg:

rg 'if __name__ =' -l | grep -v 'test/' \
  | grep -v 'tests/' | grep -v idlelib | grep -v turtledemo

The -l option causes ripgrep to list matching files without showing the context of the match.

I built up those grep -v exclusions over a few iterations - idlelib/ and turtledemo/ have a bunch of matches that I wasn't interested in.

Update: Here's an alternative way of expressing that search:
rg 'if __name__ =' -l | grep -v -e 'test/' -e 'tests/' -e idlelib -e turtledemo
grep -v still means "everything that doesn't match this" - but then the multiple -e '...' patterns are used to construct a "this pattern or this pattern or this pattern" filter. This saves on having to pipe through grep -v multiple times. Thanks for the tip, dfc.

Here's the result:

tabnanny.py
pyclbr.py
netrc.py
heapq.py
fileinput.py
site.py
telnetlib.py
smtplib.py
timeit.py
__hello__.py
aifc.py
json/tool.py
asyncio/__main__.py
runpy.py
mailcap.py
tokenize.py
smtpd.py
sysconfig.py
tarfile.py
lib2to3/pgen2/tokenize.py
lib2to3/pgen2/driver.py
lib2to3/pgen2/literals.py
xmlrpc/server.py
xmlrpc/client.py
getopt.py
dbm/__init__.py
doctest.py
pickle.py
imaplib.py
compileall.py
shlex.py
ast.py
venv/__init__.py
py_compile.py
ensurepip/__main__.py
ensurepip/_uninstall.py
http/server.py
pickletools.py
poplib.py
quopri.py
calendar.py
pprint.py
symtable.py
pstats.py
inspect.py
pdb.py
platform.py
wsgiref/simple_server.py
random.py
ftplib.py
mimetypes.py
turtle.py
tkinter/dialog.py
xml/sax/expatreader.py
tkinter/colorchooser.py
tkinter/dnd.py
tkinter/filedialog.py
tkinter/messagebox.py
tkinter/simpledialog.py
tkinter/font.py
tkinter/scrolledtext.py
xml/sax/xmlreader.py
tkinter/__init__.py
code.py
difflib.py
pydoc.py
uu.py
imghdr.py
filecmp.py
profile.py
cgi.py
codecs.py
modulefinder.py
__phello__/__init__.py
__phello__/spam.py
multiprocessing/spawn.py
textwrap.py
base64.py
curses/textpad.py
curses/has_key.py
zipapp.py
cProfile.py
dis.py
webbrowser.py
nntplib.py
sndhdr.py
gzip.py
ctypes/util.py
zipfile.py
encodings/rot_13.py
distutils/fancy_getopt.py

That's a lot of neat little tools!

I haven't explored my way through all of them yet, but running python -m module_name usually outputs something useful, and adding -h frequently provides help.

A few highlights

Here are a few of the commands I've figured out so far.

http.server

To run a localhost webserver on port 8000, serving the content of the current directory:

python -m http.server

This takes an optional port. To change port, do this:

python -m http.server 8001

Pass -h for more options.

base64

python3.11 -m base64 -h

usage: .../base64.py [-h|-d|-e|-u|-t] [file|-]
        -h: print this help message and exit
        -d, -u: decode
        -e: encode (default)
        -t: encode and decode string 'Aladdin:open sesame'

asyncio

This provides a Python console with top-level await:

python -m asyncio

"asyncio REPL 3.11.4 (main, Jun 20 2023, 17:23:00) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin Use "await" directly instead of "asyncio.run()". Type "help", "copyright", "credits" or "license" for more information. >>> import asyncio >>> import httpx >>> async with httpx.AsyncClient() as client: ... r = await client.get('https://www.example.com/') ... >>> r.text[:50] '\n \n