-1

flake.nix

View file Reviewed

··· 682 682 imports = [./nix/modules/knot-rs.nix]; 683 683 684 684 services.tangled.knot-rs.package = lib.mkDefault self.packages.${pkgs.stdenv.hostPlatform.system}.knot-rs; 685 685 - services.tangled.knot-rs.migratePackage = lib.mkDefault self.packages.${pkgs.stdenv.hostPlatform.system}.knot-migrate; 686 685 }; 687 686 nixosModules.spindle = { 688 687 lib,

+380

docs/DOCS.md

View file Reviewed

··· 387 387 2. A (sub)domain name. People generally use `knot.example.com`. 388 388 3. A valid SSL certificate for your domain. 389 389 390 390 + Already running a knot and want to move it to knot 2? 391 391 + See [Migrating to knot 2](#migrating-to-knot-2). 392 392 + 390 393 ## NixOS 391 394 392 395 Refer to the [knot ··· 721 724 > **Note:** the server will refuse to start in Secure Mode if any 722 725 > repositories have not yet been isolation-migrated. Re-run 723 726 > `migrate-isolation` if you see this error. 727 727 + 728 728 + ## Migrating to Knot 2 729 729 + 730 730 + Knot 2 is a second implementation of the knot concept. 731 731 + It serves the same repositories under the same hostname and the same `did:web`, 732 732 + and every clone URL and every `at://` reference will keep on working. 733 733 + You'll move to it by running one offline tool while the knot is stopped. 734 734 + Upgrading the old knot won't do a magic upgrade to knot 2 for you. 735 735 + This is intentional; we don't want to give anyone a nasty surprise 736 736 + if we'd switch out implementations under you or accidentally 737 737 + brick someone's janky-but-working setup by attempting to shift 738 738 + a jenga block underneath them. 739 739 + 740 740 + Here's what a switchover involves: 741 741 + 742 742 + - **Rehearsals:** try as many as you like, with the old knot still serving. 743 743 + - **Downtime:** from the `systemctl disable` in step 1 to `listening on` in step 5, 744 744 + mostly the copying, depending on how large your repositories are and how fast the disk is. 745 745 + - **Parity:** the same knot on the same hostname, 746 746 + with the same owners, members, & collaborators. 747 747 + - **Untouched stuff for reversion:** the old database, 748 748 + and under the default-copy-mode, the old repositories too. 749 749 + 750 750 + What do you get for upgrading? 751 751 + 752 752 + - Knot 2 doesn't have a SQLite database. Git is the only database, 753 753 + so your members, collaborators, & repository owners will be stored in git itself. 754 754 + - Knot 2 will serve SSH in-process, so you no longer need `sshd`, 755 755 + the `AuthorizedKeysCommand`, or the unix `git` account. 756 756 + - Knot 2 will read SSH keys live from each user's PDS. 757 757 + - Repositories created from now on will stay SHA-1, matching what you already have. 758 758 + Pass `--object-format sha256` to the migration if you'd rather modernize. 759 759 + 760 760 + The repo directories will come across whole, hooks and commit-graphs included. 761 761 + Knot 2 doesn't do repo hooks, 762 762 + and it will rewrite commit-graphs on its own maintenance schedule. 763 763 + 764 764 + ### Before we begin 765 765 + 766 766 + Get tools handy! Install both binaries: 767 767 + 768 768 + ``` 769 769 + git clone https://tangled.org/did:plc:j5hmlfdrwkvtxm7cjmu7j2is 770 770 + cd core 771 771 + nix build .#knot-rs && sudo install -m755 result/bin/knot-server /usr/local/bin/ 772 772 + nix build .#knot-migrate && sudo install -m755 result/bin/knot-migrate /usr/local/bin/ 773 773 + ``` 774 774 + 775 775 + On NixOS, the `knot-rs` module will write the config file, the service unit, 776 776 + & the state directory for you, 777 777 + so read [Migrating on NixOS](#migrating-on-nixos) alongside this guide. 778 778 + Run `./result/bin/knot-migrate` straight out of `nix build` instead of installing it. 779 779 + The knot never runs the migrator, so the module leaves it off the system entirely. 780 780 + 781 781 + Make the dir that knot 2 will use, owned by the same user that the old knot runs as, 782 782 + then write a master key for unsealing the per-repository signing keys: 783 783 + 784 784 + ``` 785 785 + sudo install -d -o git -g git /var/lib/knot 786 786 + sudo install -d -m 700 /etc/knot 787 787 + printf 'KNOT_MASTER_KEY=%s\n' "$(openssl rand -base64 32)" | sudo tee /etc/knot/knot.env > /dev/null 788 788 + sudo chmod 600 /etc/knot/knot.env 789 789 + ``` 790 790 + 791 791 + That writes `/etc/knot/knot.env`, readable only by root: 792 792 + 793 793 + ``` 794 794 + KNOT_MASTER_KEY=Xm9x2t2m2WNSVJ+9v0Cq0ftFhwEuLW/8x0aeCcaHiWM= 795 795 + ``` 796 796 + 797 797 + **Back it up somewhere off the server.** 798 798 + Seriously. 799 799 + Seriously seriously. 800 800 + The sealed key store is useless without it, 801 801 + and the keys that it seals will sign every record that the migration writes. 802 802 + 803 803 + ### Dress rehearsal, with old knot still serving requests 804 804 + 805 805 + `knot-migrate` will open its source database read-only, 806 806 + so you can rehearse as often as you like without stopping anything. 807 807 + Run it as root, since it'll read your `sshd` host key and the master key alongside the database: 808 808 + 809 809 + ``` 810 810 + sudo sh -c 'set -a; . /etc/knot/knot.env; set +a; exec knot-migrate \ 811 811 + --source-db /home/git/knotserver.db \ 812 812 + --env-file /home/git/.knot.env \ 813 813 + --host-key /etc/ssh/ssh_host_ed25519_key \ 814 814 + --plc-url https://plc.directory \ 815 815 + --target /var/lib/knot \ 816 816 + --dry-run' 817 817 + ``` 818 818 + 819 819 + The `--env-file` gives your hostname and repository scan path, 820 820 + so you don't have to repeat them. 821 821 + `--dry-run` won't touch the target, 822 822 + though SQLite will still add `knotserver.db-shm` and `knotserver.db-wal` 823 823 + beside a write-ahead-log database even to read it. 824 824 + Rehearse until the exit code is 0 and the last block of the report reads 825 825 + `scan path: writable`, `host key algorithm: ssh-ed25519`, 826 826 + and `master key: KNOT_MASTER_KEY decodes to a usable key`. 827 827 + Any line in the block with an error in it will stop the switchover. 828 828 + 829 829 + Mount each host directory at the same location inside the container if you'd rather run it that way, 830 830 + such that every path in the command is still a host path. 831 831 + Mount the target too, even though the rehearsal won't write there, 832 832 + since it will measure the free space and the write permission on the target. 833 833 + The `atcr.io` image runs as its own unprivileged `knot` user, 834 834 + so pass `--user 0:0` to read the host key. 835 835 + An image that you built yourself from `knot2/Containerfile` is distroless, 836 836 + with its binaries in `/usr/local/bin` instead: 837 837 + 838 838 + ``` 839 839 + sudo sh -c 'set -a; . /etc/knot/knot.env; set +a; exec docker run --rm \ 840 840 + --user 0:0 -v /home/git:/home/git -v /etc/ssh:/etc/ssh:ro \ 841 841 + -v /var/lib/knot:/var/lib/knot -e KNOT_MASTER_KEY \ 842 842 + --entrypoint /usr/bin/knot-migrate atcr.io/tangled.org/knot:2 \ 843 843 + <the same flags as above>' 844 844 + ``` 845 845 + 846 846 + The `:ro` on `/home/git` will work only while the old knot is still running. 847 847 + Mount it writably from step 1 onward, rehearsals included, 848 848 + even though the migration only ever reads your database. 849 849 + SQLite will create an index file beside a write-ahead-log database to read it at all. 850 850 + Under a `:ro` mount, with the knot stopped and its log checkpointed, it can't, 851 851 + and `knot-migrate` errors out with `unable to open database file`. 852 852 + `--consume-source` will move the repositories out of it, so mount it writably for that too. 853 853 + 854 854 + ### Reading the report 855 855 + 856 856 + Then read what the report says! 857 857 + 858 858 + ``` 859 859 + knot owner: did:plc:akshay 860 860 + members to grant: 3 861 861 + repos to adopt: 5 862 862 + collaborator grants: 4 863 863 + 864 864 + casbin cross-check drift: 865 865 + acl-only collaborator grants unioned in: 1 866 866 + did:plc:squid <- did:plc:boltless 867 867 + table-only collaborator grants missing from acl: 1 868 868 + did:plc:octopus <- did:plc:dawn 869 869 + repos with no acl owner marker where the owner regains push: 1 870 870 + did:plc:scallop 871 871 + 872 872 + skipped repos: 2 873 873 + did:plc:conch unrepresentable name "Test knot" 874 874 + drops collaborator grant for did:plc:boltless 875 875 + did:plc:tuna no git repository at the source path 876 876 + 877 877 + transfer mode: copy 878 878 + the filesystem checks used /var/lib/knot, since the scan path doesn't exist yet 879 879 + scan path: writable 880 880 + room to copy: 89.1GiB free is enough for the 580.0KiB that adoption will copy 881 881 + host key algorithm: ssh-ed25519 882 882 + master key: KNOT_MASTER_KEY decodes to a usable key 883 883 + 884 884 + we left the target alone. Re-run without --dry-run to migrate. 885 885 + ``` 886 886 + 887 887 + At the top is what's coming across, 888 888 + `repos to adopt`, `members to grant`, and `collaborator grants`, 889 889 + and they should match what you think you have. 890 890 + Under them is every repository that `knot-migrate` will skip, each with its reason. 891 891 + The last block is the preflight on the target, 892 892 + where `transfer mode: copy` and `room to copy: ... is enough` 893 893 + join the three lines you rehearsed against. 894 894 + 895 895 + **Skips.** If `repos to adopt` is lower than you expect, the skip list is where the rest went: 896 896 + a name with a slash, a backslash, whitespace, a control character, `..`, 897 897 + a bare `.`, or more than 100 bytes, and a record key with anything outside letters, 898 898 + digits and `.-_:~`. Rename those on tangled.org and rehearse again, or accept losing them. 899 899 + 900 900 + **Owner drift.** `knot-migrate` will refuse the whole switchover 901 901 + when the `acl` table and `repo_keys` are recorded with different owners for a repository. 902 902 + The drift section prints both DIDs, one repository per line, as `<repo> acl <did>, repo_keys <did>`. 903 903 + Settle each repository in the old knot's database, 904 904 + either by deleting the `acl` rows for the wrong owner or by correcting `repo_keys.owner_did`, 905 905 + then rehearse again. 906 906 + An extra `acl` owner marker beside the owner in `repo_keys` won't stop anything, 907 907 + since `knot-migrate` reports it and discards it. 908 908 + 909 909 + **Room on disk.** Every disk measurement happens at `<target>/repos`, 910 910 + where your repositories end up. 911 911 + Point `--target` at a directory whose parent already exists, 912 912 + and mount or create the disk you mean to fill before you rehearse. 913 913 + A rehearsal against your root filesystem measures your root filesystem. 914 914 + Under `copy`, the default, your existing repositories stay where they are, 915 915 + and `room to copy` counts a second copy of every repository it adopts. 916 916 + `--consume-source` will move them instead, in seconds, with no extra room on disk, 917 917 + and only where source and target are on one filesystem 918 918 + and this process can write the old scan path. 919 919 + Once they've moved, only a filesystem snapshot taken beforehand will bring them back, 920 920 + even from a switchover that stopped halfway. 921 921 + `knot-migrate` will refuse before it writes anything, 922 922 + so your repos never end up split across two trees. 923 923 + 924 924 + > 🦪 **Note:** if you ran Secure Mode, each repo tree has a per-owner uid, 925 925 + > so any one owner can read only their own. Root can read all of them. 926 926 + > `knot-migrate` will refuse outright when it can't read a source path, 927 927 + > and every repository is in the skip list with the error behind it. 928 928 + > It will rehearse everything else first either way, 929 929 + > and report that you need root alongside whatever else is outstanding. 930 930 + > It skips a path that isn't there at all, as usual. 931 931 + > `--skip-unreadable` will migrate the rest and leave those repositories on the old knot, 932 932 + > which is what you want when root can't read them either, 933 933 + > as with a dead disk or a stale network mount. 934 934 + > `--consume-source` will empty the old scan path of every repository that it could read, 935 935 + > so running it alongside `--skip-unreadable` leaves only the unreadable repositories on the old knot. 936 936 + 937 937 + ### Doing the thing 938 938 + 939 939 + **Step 1.** Stop the knot and disable it, 940 940 + so SQLite checkpoints its WAL and no restart of the machine can take port 5555 back from knot 2. 941 941 + Leave it installed though, for "Going back": 942 942 + 943 943 + ``` 944 944 + sudo systemctl disable --now knotserver 945 945 + ``` 946 946 + 947 947 + **Step 2.** Copy database through a read-only connection, so the log stays untouched, 948 948 + and keep the copy until the switchover is settled. 949 949 + `knot-migrate` will never write to the source database. 950 950 + The copy is there for everything else that can go wrong on the day, 951 951 + and "Going back" has a restore-command: 952 952 + 953 953 + ``` 954 954 + sudo sqlite3 "file:/home/git/knotserver.db?mode=ro" \ 955 955 + ".backup /var/tmp/knotserver-cutover.db" 956 956 + ``` 957 957 + 958 958 + **Step 3.** Run the same command that you rehearsed, without `--dry-run`, 959 959 + then give the tree to the user that the service runs as, since root wrote all of it: 960 960 + 961 961 + ``` 962 962 + sudo chown -R git:git /var/lib/knot 963 963 + ``` 964 964 + 965 965 + `knot-migrate` will write your repositories, a `config.toml`, the imported host key, 966 966 + the sealed key store, and `repo-signing-keys.json`. 967 967 + That last file is every repository's private key in the clear, 968 968 + the same keys that your old database stored, 969 969 + so move it somewhere safe and off the server once the migration is done. 970 970 + Towards the end of the copy, install 971 971 + [`knot.service`](https://tangled.org/did:plc:j5hmlfdrwkvtxm7cjmu7j2is/blob/master/systemd/knot.service) 972 972 + to `/etc/systemd/system/`, so step 5 is only a start. 973 973 + 974 974 + **Step 4.** Give port 22 to the knot, 975 975 + if you want SSH clone URLs to keep working without a port number in them. 976 976 + The order to do that in is under "Ports, Taylor's version" in the [knot 2 README](https://tangled.org/did:plc:j5hmlfdrwkvtxm7cjmu7j2is/blob/master/knot2/README.md), 977 977 + since getting it wrong will lock you out of your own server. 978 978 + Two moves there are migration-only: 979 979 + move `/etc/ssh/sshd_config.d/authorized_keys_command.conf` outside `sshd_config.d`, 980 980 + keeping it for going back, 981 981 + and uncomment `ssh_listen_addr` in `/var/lib/knot/config.toml` to set it to `"[::]:22"`. 982 982 + 983 983 + **Step 5.** Start it and follow the log, watching the index rebuild from the meta-repo, 984 984 + until `index ready` and then `listening on`: 985 985 + 986 986 + ``` 987 987 + sudo systemctl enable --now knot 988 988 + sudo journalctl -fu knot 989 989 + ``` 990 990 + 991 991 + The knot rebuilds the index before it takes a connection. 992 992 + The README's systemd section covers editing the unit for a different user, 993 993 + a different binary path, or an LFS store outside `/var/lib/knot`. 994 994 + 995 995 + Both knots serve HTTP on port 5555 by default, 996 996 + so you don't have to touch a reverse proxy already aimed at the old knot. 997 997 + Configure the proxy in the knot anyway. 998 998 + The knot will rate-limit by the address that a request arrives from, 999 999 + and every one of your users arrives from the proxy until you do. 1000 1000 + In `/var/lib/knot/config.toml`, 1001 1001 + set `xrpc.trusted_proxy_header` to the header that your proxy appends, 1002 1002 + and `xrpc.trusted_proxies` to the address that it connects from. 1003 1003 + The README's TLS section spells both out at length. 1004 1004 + An empty `trusted_proxies` leaves the knot honoring the header from every address, 1005 1005 + so fill it in whenever anything but your proxy can connect to port 5555, 1006 1006 + and keep 5555 off the public internet either way. 1007 1007 + 1008 1008 + ### Migrating on NixOS 1009 1009 + 1010 1010 + > 🦪 1011 1011 + > 1012 1012 + > NixOS is not my forte. Please read this and absolutely second-guess me. 1013 1013 + > I have of course tested this, but still, a boy gets paranoid. 1014 1014 + 1015 1015 + Steps 1 and 2 read the same, and the `knot-rs` module replaces the rest, 1016 1016 + so configure it as the [knot 2 README](https://tangled.org/did:plc:j5hmlfdrwkvtxm7cjmu7j2is/blob/master/knot2/README.md) describes. 1017 1017 + Set `settings.git.object_format` to whatever you passed `--object-format`, 1018 1018 + since the module defaults to `sha256` where the migration defaults to `sha1`. 1019 1019 + 1020 1020 + Run the migration before your first rebuild. 1021 1021 + A knot that starts on an empty state directory won't serve any repositories, 1022 1022 + and it will generate a host key of its own, 1023 1023 + which `knot-migrate` will then refuse to overwrite. 1024 1024 + A rebuild starts the knot the moment the module is on, 1025 1025 + so if you want the module in place before you migrate, add this line first: 1026 1026 + 1027 1027 + ``` 1028 1028 + systemd.services.knot-rs.wantedBy = lib.mkForce []; 1029 1029 + ``` 1030 1030 + 1031 1031 + The knot then stays stopped until you start it by hand. 1032 1032 + Migrate, `chown`, run `systemctl start knot-rs`, and delete the line at your next rebuild. 1033 1033 + 1034 1034 + If the knot already started once, `--force-host-key` writes the imported key over the one it generated. 1035 1035 + The fingerprint everybody cached under the old knot is the one that comes back, 1036 1036 + so it only surprises whoever connected while knot 2 was serving a key of its own. 1037 1037 + The `config.toml` that `knot-migrate` writes is only a record of what it chose, 1038 1038 + because the file the knot reads is the module's. 1039 1039 + 1040 1040 + Then run the migration and hand the tree to the module's user, 1041 1041 + `knot`, where the Go module used `git`, and rebuild. 1042 1042 + The `--source-repos` and `--hostname` flags replace the `--env-file` above, 1043 1043 + since the Go module configures the old knot through systemd and will never write an env file. 1044 1044 + Append `--dry-run` to rehearse it first, as often as you like: 1045 1045 + 1046 1046 + ``` 1047 1047 + nix build .#knot-migrate 1048 1048 + sudo sh -c 'set -a; . /etc/knot/knot.env; set +a; exec ./result/bin/knot-migrate \ 1049 1049 + --source-db /home/git/knotserver.db \ 1050 1050 + --source-repos /home/git \ 1051 1051 + --hostname knot.example.com \ 1052 1052 + --host-key /etc/ssh/ssh_host_ed25519_key \ 1053 1053 + --plc-url https://plc.directory \ 1054 1054 + --target /var/lib/knot' 1055 1055 + sudo chown -R knot:knot /var/lib/knot 1056 1056 + sudo nixos-rebuild switch 1057 1057 + ``` 1058 1058 + 1059 1059 + The source paths above are the Go module's defaults, 1060 1060 + where `stateDir` is `/home/git` and `repo.scanPath` is the state directory itself, 1061 1061 + so read your own out of `services.tangled.knot` if you moved either. 1062 1062 + "Check it worked" and everything after it apply unchanged, 1063 1063 + with `systemctl status knot-rs` in place of `knot`. 1064 1064 + 1065 1065 + ### Checking that it worked 1066 1066 + 1067 1067 + ``` 1068 1068 + curl -s https://knot.example.com/xrpc/_health 1069 1069 + curl -s https://knot.example.com/xrpc/sh.tangled.owner 1070 1070 + ssh-keyscan -t ed25519 your.knot.com 1071 1071 + ``` 1072 1072 + 1073 1073 + The health endpoint should answer `{"version":"knot 2.0.0"}`, 1074 1074 + `sh.tangled.owner` should answer with your own DID, 1075 1075 + and the keyscan should match the fingerprint that your users already have. 1076 1076 + Then clone a repository anonymously, and push to a repository that you own. 1077 1077 + 1078 1078 + > 🦪 1079 1079 + > 1080 1080 + > knot 2 will take SSH keys from each user's PDS on an hour's lease, 1081 1081 + > where the old knot kept a copy that could outlive the record. 1082 1082 + > So when someone's push stops working: is their key record still in their PDS? 1083 1083 + > Have they re-added the key in their Tangled settings, which writes it back? 1084 1084 + > Have they waited out the lease since? 1085 1085 + > Lower `keyfill.ttl_secs` if an hour is too long for you. 1086 1086 + 1087 1087 + ### Going back in case of emergency 1088 1088 + 1089 1089 + If you ran the migration in its default copy mode, 1090 1090 + your old data is exactly where it was. 1091 1091 + Put `sshd` back on 22, move `authorized_keys_command.conf` back into `sshd_config.d`, 1092 1092 + and start the old knot: 1093 1093 + 1094 1094 + ``` 1095 1095 + sudo systemctl disable --now knot 1096 1096 + sudo systemctl enable --now knotserver 1097 1097 + ``` 1098 1098 + 1099 1099 + Restore `/var/tmp/knotserver-cutover.db` over `/home/git/knotserver.db` only if something wrote to the original, 1100 1100 + which the migration itself never will. 1101 1101 + If you used `--consume-source`, your repositories moved instead of being copied, 1102 1102 + and only a filesystem snapshot taken before the migration 1103 1103 + or your own little dir-reverse script would bring repos back. 724 1104 725 1105 ## Troubleshooting 726 1106

+3 -2

knot2/Containerfile

View file Reviewed

··· 12 12 COPY knot2/lexicons ./knot2/lexicons 13 13 COPY knot2/crates ./knot2/crates 14 14 COPY knot2/third_party ./knot2/third_party 15 15 - RUN cargo build --release --package knot-server 16 16 - RUN strip target/release/knot-server 15 15 + RUN cargo build --release --package knot-server --package knot-migrate 16 16 + RUN strip target/release/knot-server target/release/knot-migrate 17 17 18 18 FROM gcr.io/distroless/cc-debian13:latest@sha256:1e3c6d9c255be500eb680cdea0ad07554f52ae92dfcbdf07043a2a435b4c1fe3 19 19 COPY --from=builder /src/target/release/knot-server /usr/local/bin/knot-server 20 20 + COPY --from=builder /src/target/release/knot-migrate /usr/local/bin/knot-migrate 20 21 EXPOSE 5555 2222 21 22 ENTRYPOINT ["/usr/local/bin/knot-server"] 22 23 CMD ["/etc/knot/config.toml"]

+120 -12

knot2/README.md

View file Reviewed

··· 20 20 21 21 Have fun! 22 22 23 23 + Already running the Go knot and want this instead? 24 24 + Stop it once and point `knot-migrate` at its database. 25 25 + Afterwards the knot will serve the same hostname with the same repos, 26 26 + owners, members and collaborators. 27 27 + Upgrading the Go knot won't do a magic upgrade for you. 28 28 + The walkthrough is in [the docs](https://tangled.org/did:plc:j5hmlfdrwkvtxm7cjmu7j2is/blob/master/docs/DOCS.md#migrating-to-knot-2). 29 29 + 23 30 # Running a knot2 24 31 25 32 The following is how I actually run `knot.oyster.cafe`. Please treat it as one possible setup. ··· 90 97 91 98 Moving sshd is something that can lock one out of one's own server, so do it in this order: 92 99 93 93 - 1. Open a second SSH session to the server and keep it open for this whole procedure. If step 4 goes wrong, having this session open might be the saving grace. 94 94 - 2. Edit `/etc/ssh/sshd_config` and set `Port 2200` or whatever free port one likes. Leave `Port 22` in place as well for now, so sshd listens on both. 95 95 - 3. Open the new port in the firewall if there is one. On ufw that's `ufw allow 2200/tcp` (I think!! Untested). On a cloud provider one will probably also have to deal with it / open it in their proprietary config. 96 96 - 4. Restart sshd, and **from the local computer, in a third terminal**, confirm `ssh -p 2200 root@the.server` works before continuing. 97 97 - 5. Once confirmed, only now remove `Port 22` from `sshd_config`, restart sshd once more, & give the port to the knot. When running the binary directly, that means `ssh_listen_addr = "[::]:22"`. Comparatively, in a container it entails publishing the container's 2222 as the host's 22, which is what the compose file example below does. 100 100 + 1. Open a second SSH session to the server and keep it open for this whole procedure. If step 5 goes wrong, having this session open might be the saving grace. 101 101 + 2. Find out which thing owns the port with `systemctl is-enabled ssh.socket`, since each answer sends you to a different file in step 3: 102 102 + - `enabled`: edit the socket unit. A `Port` line in `sshd_config` does nothing at all here, and editing it is the usual way to lose an afternoon. Ask me how I know. 103 103 + - `disabled`, or no systemd: edit `Port` in `sshd_config`; sshd holds the port itself. 104 104 + 3. Edit `/etc/ssh/sshd_config` and set `Port 2200` or whatever free port one likes. Leave `Port 22` in place as well for now, so sshd listens on both. Under the socket unit, `systemctl edit ssh.socket` takes a `[Socket]` section with `ListenStream=2200` instead, such that the new port joins whatever's already listening. 105 105 + 4. Open the new port in the firewall if there is one. On ufw that's `ufw allow 2200/tcp` (I think!! Untested). On a cloud provider one will probably also have to deal with it / open it in their proprietary config. 106 106 + 5. Restart sshd, or `systemctl restart ssh.socket` for the socket unit, and **from the local computer, in a third terminal**, confirm `ssh -p 2200 root@the.server` works before continuing. 107 107 + 6. Once confirmed, only now remove `Port 22` from `sshd_config`, restart sshd once more, & give the port to the knot. Under the socket unit that's an empty `ListenStream=` above the `ListenStream=2200`, since a drop-in only adds to the port it inherits. When running the binary directly, that means `ssh_listen_addr = "[::]:22"`. Comparatively, in a container it entails publishing the container's 2222 as the host's 22, which is what the compose file example below does. 98 108 99 109 If one would rather not move sshd at all, another cool option for having knot2 on port 22 is a second IP address on the remote computer. Bind sshd to one with `ListenAddress`, bind the knot to the other with `ssh_listen_addr = "<second-ip>:22"`, and just put the knot's DNS record on that second address. Leaving the knot on `[::]:22` would wildcard-bind every address on the box and would collide with sshd no matter which single IP that sshd listens on. 100 110 101 111 HTTP can stay on 5555 behind a reverse proxy, or move to 443 if one wants the knot to terminate TLS itself. 102 112 113 113 + ## Running under systemd 114 114 + 115 115 + `systemd/knot.service` at the repository root is the unit I'd install to `/etc/systemd/system/` for running the binary straight on the machine, then `systemctl enable --now knot`. 116 116 + 117 117 + It runs the knot as `git` from `/usr/local/bin/knot-server`, so edit `User=`, `Group=` and `ExecStart=` if one's setup differs. `ProtectSystem=strict` and `ReadWritePaths=/var/lib/knot` mean anything the knot writes outside the tree gets its own `ReadWritePaths=` entry, an LFS store or an ACME cache on another disk being the usual suspects. `ProtectHome=true` has to go if any of the paths is under `/home`. The two `CAP_NET_BIND_SERVICE` lines let it bind a port below 1024, and both can go when SSH stays on 2222 and HTTP on 5555. 118 118 + 103 119 ## Running with containers 104 120 105 105 - The `Containerfile` at project root builds a distroless image with just the `knot-server` binary in it: 121 121 + `knot2/Containerfile` will build a distroless image with `knot-server` and `knot-migrate` in `/usr/local/bin`. 122 122 + Its `COPY` lines start at the workspace root, 123 123 + so build it from the repository root and point `-f` at it: 106 124 107 125 ```sh 108 108 - podman build -t knot-oyster:latest . 126 126 + podman build -f knot2/Containerfile -t knot-oyster:latest . 109 127 ``` 110 128 111 129 I personally run it with a composefile. This is the file from `knot.oyster.cafe` with a few opsec adjustments: ··· 171 189 podman-compose up -d 172 190 ``` 173 191 174 174 - The `mkdir` is necessary, since the knot won't start unless the repo and LFS directories are present/writable. Podman would create the bind-mount sources for the operator, but then they belong to whichever unix user podman has rather than to the operator. 192 192 + The `mkdir` is necessary, since the knot won't start unless the repo and LFS directories are present/writable. Podman would create the bind-mount sources for the operator, but then they belong to whichever unix user podman has. 175 193 176 194 The knot creates the SSH host key on the first run at mode 600, aaand the sealed store on that same first run, because the knot's own signing key needs sealing before any repo exists. The knot purposefully won't load a host key that is group or other readable, so don't loosen those please. 177 195 178 178 - Note that this (my) config turns LFS on, since `lfs.store_path` is set. One can drop that whole `[lfs]` block if one doesn't want it. The floor of 30GiB is what I have judged for my disk (of 500GiB, doing other things at the same time), so pick something that suits one's own rather than copying mine. 196 196 + Note that this (my) config turns LFS on, since `lfs.store_path` is set. One can drop that whole `[lfs]` block if one doesn't want it. The floor of 30GiB is what I have judged for my disk (of 500GiB, doing other things at the same time), so pick something that suits one's own instead of copying mine. 179 197 180 198 Speaking of LFS, I made the directory different in the first place so that we could specify a whole separate storage medium if wanted. For example, let's say I want my actual git repos to be wicked fast, so everything *else* is on an SSD, and *only* LFS is on a massive-but-relatively-cheap HDD cluster. Wouldn't want terabytes and terabytes of massive files taking up precious SSD space in this economy! 199 199 + 200 200 + ## NixOS 201 201 + 202 202 + `nixosModules.knot-rs` renders the config file, defines the same hardened unit as above, and creates the state directory. A `config.toml` the operator writes goes unread under the module. Add the flake as an input, import the module, and write a `knot.nix`: 203 203 + 204 204 + ```nix 205 205 + { 206 206 + inputs.tangled.url = "git+https://tangled.org/did:plc:j5hmlfdrwkvtxm7cjmu7j2is"; 207 207 + outputs = {nixpkgs, tangled, ...}: { 208 208 + nixosConfigurations.knot = nixpkgs.lib.nixosSystem { 209 209 + system = "x86_64-linux"; 210 210 + modules = [tangled.nixosModules.knot-rs ./knot.nix]; 211 211 + }; 212 212 + }; 213 213 + } 214 214 + ``` 215 215 + 216 216 + ```nix 217 217 + { 218 218 + services.tangled.knot-rs = { 219 219 + enable = true; 220 220 + environmentFile = "/etc/knot/knot.env"; 221 221 + settings = { 222 222 + server = { 223 223 + hostname = "knot.oyster.cafe"; 224 224 + admins = ["did:plc:nel"]; 225 225 + ssh_listen_addr = "[::]:22"; 226 226 + }; 227 227 + atproto.plc_directory = "https://plc.directory"; 228 228 + xrpc = { 229 229 + trusted_proxy_header = "x-forwarded-for"; 230 230 + trusted_proxies = ["127.0.0.1" "::1"]; 231 231 + }; 232 232 + }; 233 233 + }; 234 234 + 235 235 + services.openssh.ports = [2200]; 236 236 + } 237 237 + ``` 238 238 + 239 239 + - `environmentFile` is where `KNOT_MASTER_KEY=` goes, and the module refuses to build without it. Write the path, never the value, so the key stays out of the nix store. Every `KNOT_*` variable the file sets overrides the matching key in `settings`. 240 240 + - Moving `services.openssh` to another port frees 22 for the knot, per the port dance above, and the module asserts the collision instead of starting two services on one port. It adds `CAP_NET_BIND_SERVICE` itself once a listen address is below 1024. Confirm a session on the new port before rebuilding, since a rebuild that moves sshd and takes 22 in one go will lock one out if the port is wrong. 241 241 + - `openFirewall` defaults to true and opens the port of each listen address that isn't loopback, so the knot's 22 opens while the default `listen_addr = "127.0.0.1:5555"` stays shut for a proxy on the same host. `services.openssh` opens its own port. 242 242 + - `stateDir` defaults to `/var/lib/knot`, and `systemd.tmpfiles` creates it and its `repos` at mode 0750 for the `knot` user. `settings.secrets.sealed_key_file` and `settings.server.ssh_host_key_file` default to `sealed-keys` and `ssh_host_key` inside it, and the module works out `ReadWritePaths=` from wherever the operator puts them. 243 243 + - The module installs the knot and nothing else. [Migrating from the Go knot](https://tangled.org/did:plc:j5hmlfdrwkvtxm7cjmu7j2is/blob/master/docs/DOCS.md#migrating-to-knot-2) runs `knot-migrate` out of `nix build`, before the module is on the machine at all. 181 244 182 245 ## TLS 183 246 ··· 277 340 278 341 Push works over both, of course . For HTTP pushing, it is up to the user to find a good Tangled-CLI or something that can put the right things in the git credential helper such that a service auth token is minted and used on push. 279 342 280 280 - A trailing `.git` on the repo name is optional, so `did:plc:nel/squid.git` goes to the same repo as `did:plc:nel/squid`. That only applies to the repo name variant though - `did:plc:barnacle.git` is read as a DID rather than as a repo-DID with a suffix, and it won't resolve. This would be made better from better DID parsing, since a `did:plc` can't have dots, only a `did:web` can. 343 343 + A trailing `.git` on the repo name is optional, so `did:plc:nel/squid.git` goes to the same repo as `did:plc:nel/squid`. That only applies to the repo name variant though - `did:plc:barnacle.git` is read as a DID with a `.git` on the end of it, and it won't resolve. This would be made better from better DID parsing, since a `did:plc` can't have dots, only a `did:web` can. 281 344 282 345 ## Things worth knowing before one commits (get it?) to a config 283 346 ··· 315 378 316 379 ## Updating 317 380 318 318 - // TODO: publish to ATCR, maybe nix something something. 381 381 + Pre-built images are at `atcr.io/tangled.org/knot:2`, 382 382 + with `knot-server` and `knot-migrate` in them. 383 383 + The image comes from [@tangled.org/knot-docker](https://tangled.org/did:plc:f5s5la5wlofsxidb3zemdune) instead of the `Containerfile` here. 384 384 + It's a Debian build, with its binaries in `/usr/bin`, 385 385 + and the `:latest` tag over there is still the Go knot, 386 386 + so one has to ask for `:2` on purpose. 387 387 + My composefile above points at a locally built image under `pull_policy: never`. 388 388 + Switching to the published image means editing the `image:` line, 389 389 + or tagging the pulled image with the name the compose file already has: 390 390 + 391 391 + ```sh 392 392 + podman pull atcr.io/tangled.org/knot:2 393 393 + podman tag atcr.io/tangled.org/knot:2 localhost/knot-oyster:latest 394 394 + podman-compose up -d 395 395 + ``` 396 396 + 397 397 + The published image bakes `KNOT_SCAN_PATH`, 398 398 + `KNOT_SEALED_KEY_FILE`, 399 399 + `KNOT_SSH_HOST_KEY_FILE` and `KNOT_MASTER_KEY_ENV` into itself, 400 400 + and an environment variable overrides the config file. 401 401 + The knot will then look for the sealed store at `/data/sealed-keys` and the host key at `/data/ssh_host_key`, 402 402 + whatever the mounted `config.toml` sets, 403 403 + find neither, 404 404 + and generate a new identity and a new host key on a path that isn't mounted by anything. 405 405 + Put my two paths back in the compose environment before switching to the published image: 406 406 + 407 407 + ```yaml 408 408 + environment: 409 409 + KNOT_SEALED_KEY_FILE: /data/secrets/sealed.bin 410 410 + KNOT_SSH_HOST_KEY_FILE: /data/ssh/host_key 411 411 + ``` 412 412 + 413 413 + The published image also runs as its own `knot` user at uid 1000, 414 414 + where the distroless image here runs as root, 415 415 + so chown the four bind mounts before the first start: 416 416 + 417 417 + ```sh 418 418 + sudo chown -R 1000:1000 repos ssh secrets lfs 419 419 + ``` 420 420 + 421 421 + Skipping the chown will stop the knot at startup with `repo.scan_path /data/repos isn't writable`, 422 422 + since root wrote every one of the directories. 423 423 + `user: "0:0"` in the compose file is the other way out, 424 424 + at the cost of running the knot as root. 425 425 + 426 426 + Building it oneself is the same three lines as always: 319 427 320 428 ```sh 321 429 git pull 322 322 - podman build -t knot-oyster:latest . 430 430 + podman build -f knot2/Containerfile -t knot-oyster:latest . 323 431 podman-compose up -d 324 432 ``` 325 433

+52

systemd/knot.service

View file Reviewed

··· 1 1 + [Unit] 2 2 + Description=tangled knot 2 server 3 3 + After=network-online.target 4 4 + Wants=network-online.target 5 5 + StartLimitIntervalSec=60 6 6 + StartLimitBurst=5 7 7 + 8 8 + [Service] 9 9 + ExecStart=/usr/local/bin/knot-server /var/lib/knot/config.toml 10 10 + User=git 11 11 + Group=git 12 12 + StateDirectory=knot 13 13 + WorkingDirectory=/var/lib/knot 14 14 + Environment=HOME=/var/lib/knot 15 15 + EnvironmentFile=/etc/knot/knot.env 16 16 + UMask=0077 17 17 + Restart=on-failure 18 18 + RestartSec=5 19 19 + TimeoutStopSec=120 20 20 + LimitNOFILE=65536 21 21 + StandardOutput=journal 22 22 + StandardError=journal 23 23 + 24 24 + AmbientCapabilities=CAP_NET_BIND_SERVICE 25 25 + CapabilityBoundingSet=CAP_NET_BIND_SERVICE 26 26 + NoNewPrivileges=true 27 27 + ProtectProc=invisible 28 28 + ProtectSystem=strict 29 29 + ProtectHome=true 30 30 + ReadWritePaths=/var/lib/knot 31 31 + PrivateTmp=true 32 32 + PrivateDevices=true 33 33 + ProtectHostname=true 34 34 + ProtectClock=true 35 35 + ProtectKernelTunables=true 36 36 + ProtectKernelModules=true 37 37 + ProtectKernelLogs=true 38 38 + ProtectControlGroups=true 39 39 + RestrictAddressFamilies=AF_INET AF_INET6 AF_NETLINK AF_UNIX 40 40 + RestrictNamespaces=true 41 41 + LockPersonality=true 42 42 + MemoryDenyWriteExecute=true 43 43 + RestrictRealtime=true 44 44 + RestrictSUIDSGID=true 45 45 + RemoveIPC=true 46 46 + PrivateMounts=true 47 47 + SystemCallFilter=@system-service 48 48 + SystemCallFilter=~@privileged @resources 49 49 + SystemCallArchitectures=native 50 50 + 51 51 + [Install] 52 52 + WantedBy=multi-user.target

+5 -21

nix/modules/knot-rs.nix

View file Reviewed

··· 82 82 description = "Package providing the knot-server binary"; 83 83 }; 84 84 85 85 - migratePackage = mkOption { 86 86 - type = types.package; 87 87 - description = "Package providing the knot-migrate binary"; 88 88 - }; 89 89 - 90 90 - installMigrateTool = mkOption { 91 91 - type = types.bool; 92 92 - default = false; 93 93 - description = '' 94 94 - Whether to instlal {option}`migratePackage` system-wide. 95 95 - Only needed if doing a one-time migration from the Go knot. 96 96 - ''; 97 97 - }; 98 98 - 99 85 user = mkOption { 100 86 type = types.str; 101 87 default = "knot"; ··· 186 172 187 173 ssh_host_key_file = mkOption { 188 174 type = absPathType; 189 189 - default = "${cfg.stateDir}/ssh_host_ed25519_key"; 190 190 - defaultText = literalExpression ''"''${stateDir}/ssh_host_ed25519_key"''; 175 175 + default = "${cfg.stateDir}/ssh_host_key"; 176 176 + defaultText = literalExpression ''"''${stateDir}/ssh_host_key"''; 191 177 description = '' 192 178 Private ssh host key the knot presents. 193 179 The knot creates one on first start when the file is absent, ··· 219 205 secrets = { 220 206 sealed_key_file = mkOption { 221 207 type = absPathType; 222 222 - default = "${cfg.stateDir}/knot.sealed"; 223 223 - defaultText = literalExpression ''"''${stateDir}/knot.sealed"''; 208 208 + default = "${cfg.stateDir}/sealed-keys"; 209 209 + defaultText = literalExpression ''"''${stateDir}/sealed-keys"''; 224 210 description = '' 225 211 Sealed store for the knot signing key. 226 212 The knot creates one on first start when the file is absent, ··· 397 383 lib.optional (tls.acme_enabled && listenPort != 443) 398 384 "services.tangled.knot-rs validates over TLS-ALPN-01, which a certificate authority reaches on TCP 443, and settings.server.listen_addr uses port ${toString listenPort}. Map 443 to that port."; 399 385 400 400 - environment.systemPackages = 401 401 - [cfg.package] 402 402 - ++ lib.optional cfg.installMigrateTool cfg.migratePackage; 386 386 + environment.systemPackages = [cfg.package]; 403 387 404 388 environment.etc."knot/config.toml".source = configFile; 405 389

+1 -4

knot2/crates/knot-migrate/src/emit.rs

View file Reviewed

··· 476 476 writable_target(destination, placement).map(|()| placement) 477 477 } 478 478 479 479 - fn writable_target( 480 480 - destination: &Path, 481 481 - placement: HostKeyPlacement, 482 482 - ) -> Result<(), HostKeyConflict> { 479 479 + fn writable_target(destination: &Path, placement: HostKeyPlacement) -> Result<(), HostKeyConflict> { 483 480 match placement { 484 481 HostKeyPlacement::Fresh => { 485 482 let blocked = match destination.parent() {

+25 -18

knot2/crates/knot-migrate/tests/migrate.rs

View file Reviewed

··· 1506 1506 chmod(&sealed, 0o500); 1507 1507 assert!( 1508 1508 matches!( 1509 1509 - emit::plan_host_key(&sealed.join("ssh_host_key"), &key, emit::HostKeyPolicy::Keep), 1509 1509 + emit::plan_host_key( 1510 1510 + &sealed.join("ssh_host_key"), 1511 1511 + &key, 1512 1512 + emit::HostKeyPolicy::Keep 1513 1513 + ), 1510 1514 Err(emit::HostKeyConflict::Uncreatable { .. }) 1511 1515 ), 1512 1516 "an absent key under a directory nobody can write is a write that fails after adoption" ··· 1560 1564 skipped: Vec::new(), 1561 1565 drift: mapping::Drift::default(), 1562 1566 }; 1563 1563 - let render = |host_key_target: Option<Result<emit::HostKeyPlacement, emit::HostKeyConflict>>| { 1564 1564 - let rehearsal = Rehearsal { 1565 1565 - host_key_target, 1566 1566 - ..ready_rehearsal() 1567 1567 + let render = 1568 1568 + |host_key_target: Option<Result<emit::HostKeyPlacement, emit::HostKeyConflict>>| { 1569 1569 + let rehearsal = Rehearsal { 1570 1570 + host_key_target, 1571 1571 + ..ready_rehearsal() 1572 1572 + }; 1573 1573 + report::Report { 1574 1574 + mapping: &mapping, 1575 1575 + orphan_alias_count: 0, 1576 1576 + phase: report::Phase::Rehearsed(&rehearsal), 1577 1577 + } 1578 1578 + .to_string() 1567 1579 }; 1568 1568 - report::Report { 1569 1569 - mapping: &mapping, 1570 1570 - orphan_alias_count: 0, 1571 1571 - phase: report::Phase::Rehearsed(&rehearsal), 1572 1572 - } 1573 1573 - .to_string() 1574 1574 - }; 1575 1580 1576 1581 [None, Some(Ok(emit::HostKeyPlacement::Fresh))] 1577 1582 .into_iter() ··· 1591 1596 1592 1597 let replacing = render(Some(Ok(emit::HostKeyPlacement::Replacing))); 1593 1598 assert!( 1594 1594 - replacing 1595 1595 - .contains("host key: the migration will replace the different key at the target"), 1599 1599 + replacing.contains("host key: the migration will replace the different key at the target"), 1596 1600 "{replacing}" 1597 1601 ); 1598 1602 ··· 1601 1605 source: rustix::io::Errno::ACCESS, 1602 1606 }))); 1603 1607 assert!( 1604 1604 - unwritable.contains("host key: the migration will write the host key over \ 1605 1605 - /srv/knot/ssh_host_key, which this process can't write"), 1608 1608 + unwritable.contains( 1609 1609 + "host key: the migration will write the host key over \ 1610 1610 + /srv/knot/ssh_host_key, which this process can't write" 1611 1611 + ), 1606 1612 "{unwritable}" 1607 1613 ); 1608 1614 ··· 1734 1740 assert!(!refused.status.success(), "{refusal}"); 1735 1741 assert!(refusal.contains("your users already trust"), "{refusal}"); 1736 1742 assert!( 1737 1737 - refusal.contains(&format!("whose fingerprint {found} your users already trust")) 1738 1738 - && refusal.contains(&importing.to_string()), 1743 1743 + refusal.contains(&format!( 1744 1744 + "whose fingerprint {found} your users already trust" 1745 1745 + )) && refusal.contains(&importing.to_string()), 1739 1746 "an operator deciding whether to force needs both fingerprints, not the word: {refusal}" 1740 1747 ); 1741 1748 assert!(

+13

knot2/crates/knot-server/src/main.rs

View file Reviewed

··· 264 264 .context("decode master key as base64")?, 265 265 ) 266 266 .context("master key from environment")?; 267 267 + if !config.secrets.sealed_key_file.exists() { 268 268 + tracing::info!( 269 269 + path = %config.secrets.sealed_key_file.display(), 270 270 + "the knot will seal a new signing identity, since it can't find a key store at this path" 271 271 + ); 272 272 + } 267 273 let secrets = Arc::new( 268 274 SealedStore::open( 269 275 &config.secrets.sealed_key_file, ··· 374 380 max_objects: ObjectCount::from(config.pack.selection_max_objects), 375 381 time_budget: Duration::from_secs(config.pack.selection_time_budget_secs), 376 382 }); 383 383 + if !config.server.ssh_host_key_file.exists() { 384 384 + tracing::info!( 385 385 + path = %config.server.ssh_host_key_file.display(), 386 386 + "the knot will generate an ssh host key, since it can't find a key at this path. \ 387 387 + Anyone who already has this knot's fingerprint cached will see it change" 388 388 + ); 389 389 + } 377 390 let host_key = knot_ssh::load_or_create_host_key(&config.server.ssh_host_key_file) 378 391 .context("load or create SSH host key")?; 379 392