Personal Audio Feed

In my last post I wrote about using Piper to convert test to speech on the Mac and my helper utility, for Piper, Pan. In this post I am going to explain the main reason I had to want to do this, and it is all based around creating a personal audio feed.

I Love Podcasts

First of all, I come from a place of love for podcasting. I have been avidly listening to podcasts for about twenty years now, and like many in my generation, the iPod was my first toe dip into the world of Apple. Ever since I discovered podcasts have existed I have regularly consumed them on my daily travels to and from work, school, events, and while carrying out chores around the home. It has been a constant companion for a significant part of my life, and I see no end to that as I discover more and more topics I want to enjoy and keep up to date on.

But, for me, what has always been lacking is a way to create and produce my own content from other non-podcast content I come across. When I come across a long-form piece of content or news article of interest, I do my best to read it there and then, but I don’t always have time. I used to add these to my read it later service du jour, and, when I was consulting and travelling regularly, I had plenty of opportunity to catch up on my reading over dinner for one at whatever establishment I could find to take breakfast or an evening meal in. With a reduction in travel, the opportunity to focus on reading has reduced.

Listening to podcasts however persists, and provides me with the ideal opportunity to consume such content. In recent years I was finally able to use some third party services to generate audio content from what was effectively yet another read it later service. But, the cost to value ratio was not high enough for me. Great services, but because I spend so much time making things and not just consuming things I eventually had to pass. But, the good news is that in the intervening time, the available technology and computing power was reaching a point where I could create my own text to podcast service.

The Process

The processing approach to achieve this is straight forward, but there is complexity to it and there is more than one way to accomplish this. This should be kept front of mind as you read through this post and I explain the components I have used to put this all together. Chances are, you won’t have exactly the same options available to you if you want to reproduce this, but hopefully, you’ll have enough.

Like I said, straight forward. But, oddly, to try and make this easier to follow, I’m going to tackle the process a little bit back to front and a little bit chopped around.

Personal Podcast Server

A podcast is not just an audio file. It also includes a feed definition file. Just like the RSS feed of a website, your favourite podcast has a feed of audio files. So the start for me was to identify how to run my own private podcast server. After all, this audio is just for me to consume. I’m not in a position to offer a free service to the world on this one. As those of you who follow along on my “What have I been up to this month?” posts, you may recall that earlier this year I replaced my old QNAP NAS with a Synology NAS, and one that had enough additional RAM to run some additional options such as Docker containers. One such Docker container I opted to try out to host some audio files I’d saved offline for use in my own personal podcast library was Audiobookshelf, an open source audiobook and podcast server.

In all honesty, this was not the easiest app to set up, and I would strongly advise reading the documentation. You might need to edit the ID3 metadata on your audio files and, or, the folder structure you want to store your files in, in order to get the software to recognise your files. But, once you are set up, you can have multiple libraries and multiple books or podcasts within each. It monitors the folders you set up and is configured by default to simply add files to the feed when they are added to the correct location with the appropriate data.

I had hoped to set this up to work with Overcast, my preferred podcast player on my iPhone, but unfortunately, I couldn’t find a way to work around the need for an externally recognised but home network only SSL certificate, and Overcast, like several other popular podcast players, is very particular about this. In the end, I found an alternative player called plappa, specifically designed for use with Audiobookshelf, that also allows for adjusting playback speed and does a good job of managing the pitch - an absolute must as I’ve had too many years being spoilt by the Overcast audio engine.

This gives me a way to sync up my library with my phone, and to have in place my own personal podcast server running on my home NAS.

An important point at this juncture is to note that I do not make my Synology NAS directly available on the Internet. But, I do have Tailscale running on it, and as an option on my iPhone (and Macs, etc.) This is a mesh VPN and means that I can direct access machines without punching holes in my home network. If you have not looked into Tailscale, then I would 100% recommend that you do, and you will find it will regularly help you solve access problems … and this will not be the last mention of Tailscale in this post.

Creating an MP3

We will come back to moving an MP3 file shortly simply because it is related to triggering the MP3 file creation. It does not have to be, but it is, so I will delay on that one. Meanwhile, I am going to share how I am creating the MP3 file.

If you have read my previous post, then it will have been clear from the outset that I am using Piper to take the text of the content I have identified online, and convert that into a spoken word MP3 file. I am using my Pan utility (covered in that previous post) to help with this, and I will share the code below.

If you do not already have Piper set up and configured (and Pan too), then you should work your way through that post as once you do, there will be nothing more to explain this part of the approach.

Fetching and Saving Text Content

For this step we are going to make use of a few free utilities, so we have a some preparations to carry out. I am doing this on my main Mac, which is on 24x7 and is capable of running the required tasks in the background. If you are wanting to use another operating system, then you will need to modify the approach a little.

First we are going to have a URL, and we want to capture the content found at that URL. Now there are ways to do this using a headless browser, or using cURL, but often times, the pages I want to capture the content of will have content I am not so interesting in grabbing. Advertising and superfluous navigation links spring to mind. For this reason I selected something that is quite similar to the Get Article Using Safari Reader action in Apple Shortcuts, but easier to script, and does not require you to confirm it every time you want to read a URL from a new website domain. This is a utility called Readability, and in my testing it proved reliable and blazing fast.

If you have followed the instructions in the previous post about installing Homebrew, then your package manager is all ready and waiting for you. If not, go grab it0.

With Homebrew installed, we can look at installing Readability. Because it is a utility written in a language called Go, there is a prerequisite for Go to be available on your system before installing Readability. But it is simple to install both with some simple(-ish) Homebrew commands.

Installing Go is as easy as entering the following command into the terminal.

brew install go

To install Readability, we actually get Go to do it direct from the Readability GitHub repository.

go install github.com/go-shiori/go-readability/cmd/go-readability@latest

Given a URL, Readability can be used on the command line to retrieve the ‘core’ content of a web page, and even do so as plain text. However, if you retrieve plain text, Readability does a terrible job of formatting that text. It adds in a lot of newline characters to make it easier to read, but when we run it through piper it reads them as breaks in the sentence which leads to a much more staccato playback than you might first expect. For this reason, I actually pull back the filtered HTML of the page and work with that and another utility.

What other utility, well, one of my favourite test processing utilities, Pandoc. Pandoc is a masterful tool for converting between a variety of text formats, and that includes converting from HTML to text. Pandoc does this in a way that does not introduce the additional line breaks and so provides an ideal counterpart to Readability here.

If you do not already have Pandoc installed, you can get it via Homebrew using the following command.

brew install pandoc

While these two tools are doing the heavy lifting for the text retrieval, I created the following script to process a file that contains the URL on the first line. The script grabs additional information such as the current date (in a speakable format), what website domain the content is from, the title of the page/article (using Readability to get the metadata), and the core content of the page. It then takes the resulting file and uses Pan to run it through the Piper test-to-speech utility. The script below also has plenty of comments to explain the details of what is going on.

#!/bin/zsh

# Get my path, etc.
source ~/.zshrc

# Output current date in speakable format
speakable_current_date() 
{
	local day month year suffix
	
	day=$(date +%-d)   # day of month without leading zero
	month=$(date +%B)  # full month name
	year=$(date +%Y)   # full year
	
	# Determine ordinal suffix
	case $day in
		1|21|31) suffix="st" ;;
		2|22)    suffix="nd" ;;
		3|23)    suffix="rd" ;;
		*)       suffix="th" ;;
	esac
	
	echo "${day}${suffix} ${month} ${year}"
}

# Initialise
NOW=$(date +"%Y-%m-%d-%H.%M.%S")
URL=$(head "$*")
HOST=$(printf '%s\n' "$URL" | sed -E' s~^[a-zA-Z]+://([^/]+).*~\1~; s/^www\.//')
HAZEL_PATH="/Users/stephen/hazel/inbox/my feed"
TEXT_PATH="$HAZEL_PATH/$NOW My Feed.txt"
OUTPUT_PATH="$HAZEL_PATH/$NOW My Feed.mp3"
TITLE=$($HOME/go/bin/go-readability -m "$URL" | sed -n 's/^[[:space:]]*"title":[[:space:]]*"\(.*\)"/\1/p')

# Notification of start
osascript -e "display notification \"$TITLE\" with title \"My Feed\" subtitle \"Fetching Article...\""

# Get the date
echo "$(speakable_current_date)" > "$TEXT_PATH"
# Add the source site domain
echo "$HOST" >> "$TEXT_PATH"
# Add the title of the page/article
echo "$TITLE" >> "$TEXT_PATH"
# Add the page/article body content
$HOME/go/bin/go-readability "$URL" | pandoc -f html -t plain | awk 'BEGIN{RS=""; ORS="\n\n"} {gsub(/\n/, " "); print}' >> "$TEXT_PATH"

# Convert the text file to the audio file
pan -i "$TEXT_PATH" -o "$OUTPUT_PATH"

# Clean up
rm "$TEXT_PATH"
rm "$1"

# Notification of end
osascript -e "display notification \"$TITLE\" with title \"My Feed\" subtitle \"Article added to feed\""

Triggers and File Moves

If you were looking carefully at the script in the last section, then you may have noticed a few mentions of Hazel, the automated file organiser for the Mac. This is my go to tool for setting up simple, or sophisticated file management processes. It was an ideal tool for this situation.

I created two Hazel rules that monitor a sub-folder in a folder structure i have set as an inbox area for Hazel.

The first rule watches for new .url files and then runs the script above against them to generate a new .mp3 file.

The second rule is watching for new .mp3 files and moves them into the Audiobookshelf ‘My Feed’ library folder on my NAS. I do this via SFTP so that it will always connect and transfer independently of if I have connected the server with a direct file share.

Capturing the URL

The final piece of the puzzle was capturing the URL and getting that into the .url file on my Mac. While the capture part was relatively easy, the saving to file caused me some headaches. In a way at least.

I had thought that I had a rock solid solution for taking action on the file, because in the past I had regularly used Hazel watching a Dropbox folder to trigger activities remotely. I used it to manage work on PDFs for me when using my work PC, and to trigger things like print jobs and local announcements when I was away from my home office. While I don’t sync Dropbox to my Mac any longer, I figured I could just use iCloud Drive in the same way. It turns out I was very, very, wrong.

In my testing it became readily apparent, after I traversed a few tangents that yielded no insight, that Hazel couldn’t monitor the iCloud folder as once the folder was established, within a short period of time, Hazel would note that the folder was no longer there. It was, and I had it set to remain downloaded, but if I reselected the folder it had obtained a new ID. This was a repeatable scenario, so monitoring an iCloud Drive folder for updates was ruled out.

I wondered if this was a Hazel specific issue, but after some testing with a Folder Action, I concluded it was very much an iCloud Drive/macOS constraint.

I do not use, or have a driving need for, a Dropbox subscription, so reverting to Dropbox was not a realistic option just for this, and I don’t like having multiple online file system syncs running in the background. I have had that slow things in the past at inopportune moments. As a result, the next logical step was to try accessing the Mac directly … and Tailscale comes to the rescue once more.

Tailscale is on pretty much every end computing device I have, so this actually provides me with the direct access path I need to totally circumvent the whole cloud storage approach.

My personal devices are all on the Apple ecosystem, so it made sense to use the share sheet for directing the capture of a URL and Apple Shortcuts to automate the steps to create the file.

The basic idea for the shortcut is it takes the URL shared from the share sheet and saves it to a file. If the shortcut is being run on the Mac that is running the processing (it is uniquely named, and I check the device name to determine this) it will save the file directly to the /hazel/inbox/my feed/ folder in my home directory with a plain text file name in the format yyyy-MM-dd-HH.mm.ss My Feed.url. If the shortcut is running on another device the shortcut ensures that the Tailscale VPN is enabled, then opens an SSH connection to my Mac and uses a simple command line instruction to write the file. At the end the shortcut displays a notification just so I get a confirmation that it has completed successfully.

Note: I am using an SSH key in the authentication to the Mac when writing the file remotely, and if you want to know how to set this up I have a post that leads you step-by-step through the process

Conclusion

This is very much a personal solution, but you can see how it brings together a range of tools (Readability, Pandoc, TailScale, Piper, Pan, Audiobookshelf, plappa, Docker), scripts (Apple Shortcuts, ZSH shell script with a dash of AppleScript), and a range of devices (mobile (iPhone/iPad/MacBook), desktop (Mac), NAS) to deliver the end result. Some of these tools and components could easily be swapped out to do something similar, perhaps using Linux or Windows. You could take inspiration from the use of Hazel, SSH, and Tailscale to create your own secure access automation triggers. You could even just take a local approach for grabbing content and turning it into audio files using a handful of these components.

I really do not expect anyone to try and reproduce this setup exactly, though obviously you could. Rather I hope that this last couple of posts shows how, with a bit of exploration and persistence, you can take a variety of parts and piece them together to solve your own puzzle. One step at a time.