As someone who has fully embraced the IndieWeb, owning my data and maintaining a canonical identity on the web is very important. Though I operate a lot of my online life directly out of my website, it doesn’t mean I don’t participate in other social networks. What I don’t want is for my ‘content’ to live solely within any external platform, silo or walled garden. Finding a way to archive or otherwise make this content available on my own site mitigates this. PESOS (i.e. “Publish Elsewhere Syndicate (to your) Own Site”) or what I like to call “reverse syndication” is the practice of archiving content that is originally posted elsewhere on the web, back to your site - the single source of origin. The problem is, if a lot of your content originates outside your site, you have to either deal with the overhead of manually archiving it, or, find a way to automatically do it. Here, I talk about how I (finally) found a way to semi-auto-PESOS select Mastodon posts to the site.
Traditionally, my syndication strategy has been very curated, and very manual. It’s time consuming to archive stuff I post elsewhere, and that overhead scales rapidly if I am generating a lot that originates not on my own site. It requires that I, A. remember to archive it at all, and B. that I make nuanced decisions about whether something is worth archiving to begin with. That threshold for whether something is worth it or not can vary depending on how lazy I am being, to be frank. If there was a way for me to automatically pull down posts, add the appropriate front-matter and use some logic to determine which posts I would actually want to bring back to my site, that would save me a lot of time! Not only that, but there would be things that could make it to my site that I traditionally wouldn’t have bothered to try and manually bring over. Awesome!
Setting PESOS and data ownership aside for a second though, a big motivator for me wanting to pull in content from around the Internet was Molly White’s Activity feed. I loved the idea of just having that composite feed of everything I was up to all in one mega timeline. Bringing in Mastodon posts as I explain in this devlog is one piece to that ultimate indieweb-puzzle.
Mastodon markdown archive
I’ve had this dream of automating the reverse-syndication of my Mastodon posts since April(ish) of 2024, and had searched for ways to pull in this content off and on since then. Recently though, I came across this post, “Archiving and syndicating Mastodon posts”, which seemed to exactly meet my needs. 1 So, I cloned down the repo and tried it out! 2
It didn’t go well.
I was getting an incomprehensible traceback and wasn’t sure how to troubleshoot. I thought, maybe it could be the version? So I tried a downgraded version and still no dice. Finally, I went back and actually re-read the original post and tried to follow slowly, step-by-step, and sure enough, it was something pretty simple that was the issue (isn’t it always?). Gabriel makes it pretty clear that the dist and user flags are required, and sure enough, they super are. I won’t dock any points from Gabriel for not handling those errors gracefully, after all, this is freely available and otherwise the script works great!
Planning and Mechanics
OK, so I’ve proved that I can get the script to work, now I needed to figure a few things out…
- What posts did I want to pull in? (e.g. boosts, replies, public-only, etc…)
- Where do I want these posts to live on my site?
- What front matter is important to populate for these posts? What data is available in the .json output for each post?
- Do I need to tweak any of my existing code to accomodate the posts pulled in via this script?
What to Archive?
The first challenge was part technical and part philosophical. What do I really want to archive? To start, I knew I didn’t care to archive my boosts/reblogs, so I can easily tune those out with the --exclude-reblogs flag. What about replies? Well I write a lot of (in my mind) great, well-thought-out replies, with info I’d love to hold on to and refer back to later. In fact, I’ve manually reverse-syndicated a lot of this kind of post in the past. So I knew I needed replies, just not all of them. The overwhelming majority of my reply posts are short little comments that when I thought about it, weren’t really anything I cared to bring back to my site. Especially in a format that is site-searchable and would certainly junk up other parts of the site (e.g. tags page, search results, RSS feeds, etc…)
I knew I would need to clean up any mass-export of posts that included replies, so to start I just created the full archive to see how many posts there were in total. The result was over 1000 and it became clear that I wasn't going to manually review each of those to see what was worth keeping. I needed a new approach. So, I decided to instead bring down just posts that I had marked public. More recently, when I post on Mastodon I’ve given more thought to whether that post should be marked as ‘public’ or ‘quiet public’ (i.e. ‘unlisted’). This foresight would pay off now as I wanted to only bring over a select few things that I felt were worthy of being public/exposed on my site. The problem is, I haven’t always been this mindful about what visibilty setting I post under. But it would have to do for now. So, how many posts did I have that were in fact public? Creating the filtered archive yielded only a few hundred posts - much more manageable to manually review. The compromise now was that maybe I’d miss a good reply that I had marked as unlisted some time ago, but I’ve done a decent enough job manually syndicating things I thought were worth keeping for a good while now so I’m OK if maybe I missed something.
Below are the flags I used to create my complete post archive of public toots (included replies).
go run main.go \
--user=https://shellsharks.social/@shellsharks \
--dist=./[OUTPUT FOLDER] \
--exclude-reblogs \
--filename='{{.Post.CreatedAt | date "2006-01-02"}}-shellsharks.social-{{.Post.Id}}.md' \
--porcelain \
--persist-last=./last \
--max-id=$(test -f ./last && cat ./last || echo "") \
--visibility=public
After clicking through each of the exported posts, I whittled the list down from 200-ish to about 50 posts. Not bad! Though it makes you wonder how important/worth it all of this effort is if I’m only interested in saving 50 things from nearly a year’s worth of tootin’. But, this last year I have also done a lot of manual reverse-syndicating which softened how many I needed to import now. 🤷♂️
Moving forward, I’ll have to be thoughtful about what I mark as unlisted vs. what I mark as public. Anything public will come to my site.
This is what my command looks like to import just the latest public posts…
go run main.go \
--user=https://shellsharks.social/@shellsharks \
--dist=/[DESTINATION DIR] \
--exclude-reblogs \
--filename='{{.Post.CreatedAt | date "2006-01-02"}}-shellsharks.social-{{.Post.Id}}.md' \
--porcelain --persist-first=./first --since-id=$(test -f ./first && cat ./first || echo "") \
--visibility=public
Notes
The second question was, where should these auto-syndicated posts live? In the original plan for my site, the Notes section was meant to house microblogs / social media posts, but I wasn’t sure I really wanted these auto-syndicated toots to junk up that timeline. In practice, the way I’ve used the notes feed has been for content I thought was “lower quality” than what was in my posts feed (e.g. shorter-form, less-researched, etc…). In hindsight, this was the wrong way to think about the notes feed. Over time, this approach resulted in me publishing a lot of things as Notes that should have just been Posts all along. This meant that my Notes feed had become HIGHER quality than I had intended it to be. So, rather than create yet another feed for these toots, I decided to revert back to the original, correct mode of thinking and house these toots as Notes on my site. Moving forward I plan to be less stingy about what I publish as a ‘Post’ and much less precious about what gets ‘Note’ status. If there was anything ‘lower quality’ than that, it probably doesn’t need to exist on my site.
It’s also worth mentioning that my notes layout was built with a social media post design in mind from the beginning. So, it definitely made sense in the end to just bring these toots in as notes and have them live directly within my notebook feed.
This said, they were’nt given full rights and visibility across the blog (at least not for now), as I’ll explain later…
Front Matter
Notes on my site have their own unique layout and some specific front matter metadata. To make this whole effort worth it, I needed the script to auto-populate as much of that metadata as possible. Thankfully, Gabriel solutioned for this…
This script supports a templating construct (located here /mastodon-markdown-archive/files/templates) which allows you to customize front matter. It will convert hashtags to jekyll tags, populate created dates, author names, etc… This resource shows you some of the variables/data types you can work with from the .json export. Notably, I used the “Account” and “Post” types to further enrich my post front matter. Below I show some of the more relevant tweaks to the default post.tmpl file…
layout: note
title: shellsharks.social {{.Post.CreatedAt | date "1/2/06 15:04 MST"}}
author: {{ .Post.Account.DisplayName }}
syndication: auto
syndicate-to:
- name: Mastodon
url: {{ .Post.URI }}
icon: ph ph-mastodon-logo
{{- if .Post.Application}}
posted-from: {{ .Post.Application.Name }}
{{- end}}
Customizing the date structure is super easy thanks to gomplate, and it’s nice that I can import the ‘Application’ used to publish the post and have that represented in my notes front matter metadata.
Site Tweaks
To make this all work correctly I had to tweak a few things in my existing layouts and consider how this type of content would be fed into the various streams and feeds across the site.
First, I excluded anything auto-syndicated from going into my RSS feeds. If you want to follow me on social media, you can do that, you can even subscribe to those RSS feeds directly. But I figured folks who subscribed to my blogs RSS didn’t need to have my exact social media posts fed into their RSS readers as well.
Second, a lot of these PESOS’ed toots do not have any hashtags, and thus do not come over with any tag front-matter. So I had to add a bit of logic on my tags page to make sure tagless posts didn’t show up in the list.
Third, I added some additional logic to my activity streams on my home page to not show auto-syndicated toots. They will show up in my “Infosec-Only” stream and in the “Latest Note” card however.
For all of this logic to work however, I added a flag in the front matter for these notes - syndication: auto. If I want to have these notes show up in my feeds/streams later, I can easily remove that flag and regenerate the site.
Overall though, my site handled the addition of these notes pretty well - with minimal re-engineering 😄.
Issues and the Future
I’m happy with the v1 of this capability, I think it will save me a lot of time and bring in content I never had the time to import/archive in the past. But, it isn’t without some issues, and there’s a lot I’d like to improve on into the future.
Notes are brought in with no title, and reply posts are brought in without the OP context. I might take time to retroactively go back and add this information in. Since I’m being pretty selective about what gets brought over, this shouldn’t be that big of a task.
This script solves for my Mastodon posts, but there’s a lot of other places I’d like to PESOS in a similar way (e.g. GoToSocial, Pixelfed, Bluesky, Reddit, Lemmy, etc…). Notably, this script does not work for GoToSocial or Pixelfed, so I’ll either need to tweak it myself, find a new solution, or whip up some script of my own to do it. Until then, I will continue to manually reverse-syndicate things I think are worth it.
For now, I am running this script manually, rather than having it auto-run as a GitHub action or cronjob on my local machine. In the future, I may want to automate it fully. If I do that, I’d like to find a way to auto update my changelog as well to reflect new imported notes.
Finally, I’ve considered making a separate feed that truly mirrors my Fediverse feed, importing all posts. This feed would be entirely isolated from my home page streams, search, tags, RSS feeds, etc… It would literally just be a mirror of my Mastodon toots. A project for another day…