Shell Archive From First Principles

shar utils

2026-02-14

So I’m slowly writing a little game, hacking on it now and then as time and interest both allow. (I learned how to play cribbage this winter and have been really obsessed with it ever since. Its quirky scoring rules and phased play make it interesting and challenging to reason about and write down in code.)

Initially I was sharing my progress periodically with some friends who share some of my code interests, and who indulge my cribbage fixation.

Now that the code base is +30 files I’m not sharing it in the same way because you can no longer glance at it and understand the entire thing. But initially I would just cat my files to shar(1) and upload them to a pastebin (https://termbin.com/) where my friends could simply skim the code in their browser, or easily extract the shar locally:

curl -s https://tb.wtf/blabla | sh

A shar file is a “shell archive,” a plain-text self-extracting archive that is also a valid shell script. So you just sh file.shar and it generates all the directories and files. It has extremely minimal syntax/markup so one benefit is that you can also just read the file, and see and understand its contents.

# This is a shell archive.  Save it in a file, remove anything before
# this line, and then unpack it by entering "sh file".  Note, it may
# create directories; files and directories will be owned by you and
# have default permissions.
#
# This archive contains:
#
#  Makefile
#  README.md
#  bootstrap.tcl
#  config.tcl
#  core/
[...content omitted for brevity...]
#
echo x - Makefile
sed 's/^X//' >Makefile << '60f8520832b6f80996c175fbf41f4e55'
X# target: default -- list recipes
X.PHONY: default
Xdefault:
X  @sed -n 's/^# target: //p' Makefile
[...content omitted for brevity...]
60f8520832b6f80996c175fbf41f4e55
echo c - core/
mkdir -p core/ > /dev/null 2>&1
echo c - core/game/
mkdir -p core/game/ > /dev/null 2>&1
echo x - core/game/game_actions.tcl
sed 's/^X//' >core/game/game_actions.tcl << 'c4c5e27f155b325eb56fef6526dde2df'
X# action schemas to be consumed by engine::apply
X# or as constructors for creating actions
X# and most importantly: to be created by the ui and sent to the game engine
X# mostly here for documentation and consistency
Xnamespace eval game_actions {
X  # ----- symbolic names
X  set cut_cards "cut_cards"
[...content omitted for brevity...]
c4c5e27f155b325eb56fef6526dde2df
exit

You can see from the snippet above what it does: the script removes the leading X (with sed) from each line and uses a HEREDOC style redirect to extract the file, creating directories as it goes along the way. It uses an md5sum of the filename to prevent name collisions.

Jimmy Goosebaby allegedly wrote the first version of this in the 70s and 80s, using a quirkly random string instead of the md5sum:

AR=$1
shift
for i do
    echo a - $i
    echo "echo x - $i" >>$AR
    echo "cat >$i <<'!Funky!Stuff!'" >>$AR
    cat $i >>$AR
    echo "!Funky!Stuff!" >>$AR
done

Source: https://en.wikipedia.org/wiki/Shar_(file_format)#History_and_variants

shars are useful in the niche situation where you want to share some files but are for whatever reason not able to upload or attach a proper binary archive like a tarball or a zip file. Think a web forum or even an email provider that has restrictions on file uploads. You can still always just append a plain text shell archive to your message or post.

GNU has gone on to create and maintain their own sharutils package:

https://www.gnu.org/software/sharutils/manual/sharutils.html

My macbook comes with the BSD version installed.

https://man.freebsd.org/cgi/man.cgi?query=shar

One thing that I always found kinda funny about BSD shar is how it uses md5sum to make a checksum but then ever verifies anything with it.

So I edited it to make a checksum not of the filename but of the contents of the file, and then to verify it on extraction.

cat << EOF
# This is a shell archive bla bla bla
EOF

for i
do
  echo "#   $i"
done

echo "#"

for i
do
  if [ -d $i ]; then
    echo "echo c - $i"
    echo "mkdir -p $i > /dev/null 2>&1"
  else
    md5sum=`md5 -q $i`
    echo "echo x - $i"
    echo "sed 's/^X//' >$i << '$md5sum'"
    sed 's/^/X/' $i || exit
    echo "$md5sum"
    echo "# Verify checksum"
    echo "extracted_md5=\`md5 -q $i\`"
    echo "if [ \"\$extracted_md5\" != \"$md5sum\" ]; then"
    echo "  echo \"ERROR: MD5 mismatch for $i\" 1>&2"
    echo "  echo \"  Expected: $md5sum\" 1>&2"
    echo "  echo \"  Got:      \$extracted_md5\" 1>&2"
    echo "  exit 1"
    echo "fi"
    echo "echo \"  MD5 verified: $i\""
  fi
done

Now if you (or anybody else) tries to edit a value in the shar itself, it will fail to extract because the verification will fail.

This does not make shar secure. But it’s something. And for me, the use of md5sum now feels more justified and useful.