I set up my own GoToSocial server when I decided to join the fediverse. I wanted to self-host my instance so that I could control all the settings and own all my content. I didn't really know what to expect when I started, and wasn't sure if I'd stick it out. But GoToSocial has been a breeze to maintain, and I've really enjoyed my fediverse participation.

When I started, I used a dormant vanity domain I'd previously purchased, mostly for entertainment purposes. I didn't then know how to fully integrate GoToSocial with my existing domain, so I took the easy route. And now three years later, integrating my fediverse identity back into my primary web site is way more work than I care to tackle. Unfortunately, this birfurcates my online presence across domains in ways that I now find sub-optimal.

I have backups of my GoToSocial database, but that's a big blob of data that's hard to parse or review. As a big fan of plaintext data files, and having an interest in both POSSE and PESOS, I decided to extract all my toots and store them in individual markdown files. Why? I'm not really sure! I may never do anything with them, but having a(nother) backup of my content seems useful.

I opted not to query the GoToSocial database. That would have given me the most control, but requires greater effort. Instead, I chose to read my site's RSS feed and parse that. This only shows the original public toots I created, and does not include replies. It might be neat, at a later date, to build threaded displays of conversations in which I've participated, but that's not really my primary interest right now.

There's an interesting dynamic with some fediverse users where they delete toots after a number of weeks. There seem to be two primary, and connected, motivators for this behavior: to keep the resource consumtion of their hosts under control and to embrace the ephemerality of conversations. The first issue is less important to me, hosting my own little instance. As of today, after three years of use, my SQLite databse file is 2.5GB. It costs little to keep my posts online and available. The emphemerality is a very different issue for me. Having owned my own website for approaching three decades, I've kept everything online. Why would I delete toots when I don't delete blog posts?

I'm considering splitting the difference in these two, by keeping my toots as plaintext archives in my website, and then removing them from the GoToSocial instance. That's a decision for a later day, though. Today's effort was solely about getting toots into files.

I am not a very good programmer. I never took any Computer Science classes in college, so I never formally studied any algorithms, data structures, CPU design, or the like. I grew up writing PHP, which is very permissive and flexible. I can fumble around in Python, and would like to increase my capacity in other languages. So I took this as an opportunity to practice!

I do not use an IDE, with a language server and auto-complete and function lookups. I use vim. I save often and run the script manually in another terminal window to see the results. I know that I should write tests and evaluate my code against my tests. I know that my manual REPL fumblings are inefficient in the extreme. But it works for me, for now.

I wrote two programs, one in Python and one in Golang. They do essentially the same thing. Python handles file I/O a little smoother for a caveman like myself, but I found enough examples of Golang code to help me copy/paste a solution.

I started with Python, and after a bunch of trial and error, ended up with this:

import time
import feedparser
import os
from pathlib import Path
last = 0
idx = Path("last.txt")
if idx.exists():
  with open(idx, 'r') as file:
    last = file.read().rstrip()
print(f'Starting from ID {last}')
feed = feedparser.parse(f'https://dungeoncrawler.world/@skippy/feed.rss?limit=40&min_id={last}')
for e in feed.entries:
  if 'content' not in e:
    continue
  id = e.link.split('/')[5]
  print("Parsed ID " + id)
  dirname = time.strftime("%Y/%m/%d", e.updated_parsed)
  #fname = time.strftime("%d", e.updated_parsed)
  os.makedirs(dirname, mode=0o777, exist_ok=True)
  toot = "---\n"
  toot += "permalink: " + e.link + "\n"
  toot += "date: " + e.updated + "\n"
  toot += "---\n"
  toot += e.content[0].value + "\n\n"
  for m in e.enclosures:
    if m.type.split("/")[0] == "image":
      toot += "![](" + m.href + ")\n"
  with open(f"{dirname}/{id}.md", "w") as f:
    f.write(toot)
with open('last.txt', "w") as f:
  f.write(id)

Working on my laptop, I used a Python virtualenv in which I installed the feedparser library. The GoToSocial RSS feed supports a maximum of 40 items. It also supports a min_id. A simple shell script loop allowed me to run this starting from ID 0 all the way up to my latest toot.

./archive.sh  2.27s user 0.70s system 6% cpu 47.380 total

With a working example in Python, I then tried to do the same thing with Golang. This took me considerably longer to reach success. There was a lot of reading of examples, a lot of testing in the Go playground, and a lot more REPL on my laptop. But I got it to work!

package main
import (
  "bufio"
  "fmt"
  "os"
  "path/filepath"
  "strings"
  "time"
  "github.com/mmcdole/gofeed"
)
func check(e error) {
    if e != nil {
        panic(e)
    }
}
func main() {
  // last used status ID, assume starting from absolute beginning
  ID := "0"
  file, err := os.Open("last.txt")
  if err == nil {
    defer file.Close()
    scanner := bufio.NewScanner(file)
    for scanner.Scan() {
      ID = scanner.Text()
    }
    if err := scanner.Err(); err != nil {
      fmt.Println(err)
    }
  }
  file.Close()
  fp := gofeed.NewParser()
  feed, _ := fp.ParseURL("https://dungeoncrawler.world/@skippy/feed.rss?limit=40&min_id=" + ID)
  for _, item := range feed.Items {
    if item.Content == "" {
      continue
    }
    parts := strings.Split(item.Link, "/")
    ID = parts[len(parts)-1]
    fmt.Println(ID)
    date := item.PublishedParsed.Format("2006/01/02")
    err = os.MkdirAll(date, 0755)
    check(err)
    tootFile := filepath.Join(date, ID + ".md")
    toot := "---\n"
    toot += "permalink: " + item.Link + "\n"
    toot += "date: " + item.PublishedParsed.Format(time.DateTime) + "\n"
    if item.UpdatedParsed != nil {
      fmt.Println("This one is updated " + item.Updated)
      toot += "updated: " + item.UpdatedParsed.Format(time.DateTime) + "\n"
    }
    toot += "---\n"
    toot += item.Content + "\n\n"
    for _, e := range item.Enclosures {
      if strings.Split(e.Type, "/")[0] == "image" {
        toot += "![](" + e.URL + ")\n"
      }
    }
    f, err := os.Create(tootFile)
    check(err)
    defer f.Close()
    _, err = f.WriteString(toot)
    check(err)
    f.Sync()
  }
  // update our index file
  f, err := os.Create("last.txt")
  check(err)
  defer f.Close()
  f.WriteString(ID)
  f.Sync()
}

I updated the shell script to call go run main.go:

golang: ./archive.sh  2.89s user 6.46s system 17% cpu 52.493 total

I really should have compiled it, as that would be a lot faster; but if the first iteration had to invoke the Python interpreter for ever iteration, then I guess this is a somewhat fair comparison. But really, the timings were just idle curiosity for me: this is not a time-sensitive operation and I'm not going to import the whole set of toots every time.

So now I have a collection of simple markdown files. I could iterate upon this to add better support for enclosures. I could figure out what to do about polls. I could improve a lot. But I don't think I will, because I don't particularly care at this time. When and if I decide to do something more, I'll go straight to the database and work from the source.