RSS Amplifier

Blogs by otaku - Coding Otaku · Mar 15, 2025

Syncing NextCloud with Rclone and FZF

Read inOriginalenesdefrptjaittr
0
Sign in to vote or save

Coding Otaku · Coding Otaku

Introduction

I mentioned in my Uses page’s Cloud Host section that I use Nextcloud for cloud storage.

While I tend to like and use modular programs, I haven’t found a good workflow or programs that suites all my needs for maintaining my backups.

Syncthing could work, but I share my Nextcloud instance with other people, and also share directories and albums with them. Syncthing does not have good encrypted multi-user support, and other programs I found also have similar problems. So I’m stuck with Nextcloud for now.

The Problem

This may sound silly, but I do not want to use the GUI client of Nextcloud, there are multiple reasons for it:

  • It does not go well with my Gentoo system configuration.

    • Nextcloud is built on QT, while I have no problems with QT, all programs I have run either in a terminal or have GTK GUI, so I do not want to build qtbase just for this.
    • Nextcloud does not have a cli or -GUI flag, and the issue raised to build Nextcloud without GUI has not received enough traction.

  • I experienced weird connection drops when using the GUI, it could be just me, as I couldn’t reproduce this with other people.

  • I do not want auto-sync to happen while I am creating/editing files.

    • Auto-sync happens while I re-organise my directories too (which I do more often than I could admit).
    • I could just pause the sync, but it’s too much hassle to remember to do this always.

So the GUI program just gives me more problems, so I wanted a light-weight script that I can configure whenever and however I want.

The Solution

The solution was staring at my face the whole time, I could just use Rclone, a sync program that works with multiple providers.

Since Nextcloud offers WebDAV, I created an rclone.conf file to connect to it. It looks like this:

~/.config/rclone/rclone.conf

  [cloud-backup]
  type = webdav
  url = https://<server-address>/remote.php/dav/files/<username>
  vendor = nextcloud
  user = <username>
  pass = <app-password>

the <app=password> is generated from Nextcloud, my Nextcloud setup is encrypted and has a two factor authentication, so creating a password per app is necessary if they log in using basic auth.

Rclone also allows one to mount remote directories, but as I mentioned before, I do not want things to sync when I don’t want it to. Instead, I made use of the sync command provided by the Rclone.

The sync command, recursively copies files from the source and to the destination directory (it does more, I’m oversimplifying it), it also has an interactive mode that asks questions when in doubt.

To sync to Nextcloud, we need to do the following:

rclone sync --interactive -v "<source_directory>" "<destination_directory>" --progress

We specify the remote using the remote name provided in Rclone config file. In our case, it is cloud-backup.

For example, to restore the directory Pictures from the remote cloud-backup to the local directory $HOME/Images, the command will be:

rclone sync --interactive -v "cloud-backup:Pictures" "$HOME/Images" --progress

Now that we know how to sync, all we need to do is to do this for all the directory we want to backup, which can get frustrating easily.

People say that the command line is more efficient at doing tasks than the GUI, this is not true if you just type commands all the time. What we need is scripts and aliases that make things faster to do things.

To make the process less frustrating, let us break down what we need to do.

  1. We need to store the list of directories to backup in some file.

    • The remote and local directories may differ, so what we really need is a map of remote and local directory paths.
    • If the remote and local has the same name and is stored in the home directory, then local directory should be optional.
    • The sync order needs to be defined if a directory must backed up before another.

  2. We must handle multiple remotes, rclone listremotes can list all available remotes.

  3. Ability to backup/restore all mapped directories easily.

  4. All operations must be interactive to avoid breakage.

Storing a Map of Backup Directories

I decided to store the values in the TSV format. While it is possible to have a tab character in file names, it’s not really compatible with all file systems, so I am sticking with TSV on this one (may change in the future).

I am also setting a priority order for the directories, this is because I want all subdirectories to be synced before syncing the parent one in some cases.

The mapping file looks something like this:

~/.config/sync/cloud-backup-mapping.tsv

1	Pictures	$HOME/Images
2	Documents/Books
3	Music

There are three columns in the mapping file:

  1. The first column is the priority, it starts from 1, the 0 is treated as special, and is hard-coded in my script, we’ll get to this later.
  2. The second column is the directory path in the remote
  3. The third, optional column is the directory path in the local. If nothing is provided, it will be set as $HOME/

Did you notice that the file path is ~/.config/sync/cloud-backup-mapping.tsv? The cloud-backup is the remote name we set in the rclone.conf file. This way, we can create a mapping for each configured remote.

So the mapping file must be stored in XDG_CONFIG_HOME/sync/<remote-path>-mapping.tsv, the tool will default to the path ~/.config/sync/<remote-path>-mapping.tsv if the XDG_CONFIG_HOME environment variable is not found.

The Script

Deciding the mapping file format and path was the hard part. Now that we have one ready, let’s jump into writing a script.

Getting All Available Remotes and Selecting One

Rclone can list all available remotes using rclone listremotes. But I found that instead of returning an error code, it just prints a warning that the default configuration will be used when no remotes are found. So I decided to verify the existence of the Rclone config file before proceeding.

list remotes

BLUE="\e[1;34m"
RED="\e[1;31m"
GREEN="\e[1;32m"
RESET="\e[0m"
function check_file() {
    local file_path="${1}"
    local file_type="${2}"
    if [ ! -f "${1}" ]; then
        printf '%b \uea87 %b %s file not found: %s\n' "${RED}" "${RESET}" "${file_type}" "${file_path}" 1>&2
        exit 1
    fi
}
function get_file_path() {
    local relative_path="${1}"
    local absolute_path
    if [ -z "$XDG_CONFIG_HOME" ] || [ ! -d "$XDG_CONFIG_HOME" ]; then
        absolute_path=$(realpath "$HOME/.config/${relative_path}")
    else
        absolute_path=$(realpath "${XDG_CONFIG_HOME}/${relative_path}")
    fi
    printf '%s' "${absolute_path}"
}
rclone_config=$(get_file_path 'rclone/rclone.conf')
check_file "${rclone_config}" 'Rclone config'
remotes=$(rclone listremotes | cut -d: -f1)
if [ -z "${remotes}" ]; then
    printf '%b \uea87 %b %s %s\n' "${RED}" "${RESET}" 'No remote was found in the Rclone config: ' "${rclone_config}"
    exit 1
fi

This might look a bit long, but I will be reusing the get_file_path and check_file later, and I also wanted to add some user-friendly error messages.

The remotes can now be sent to fzf to interactively select it. If there is only one remote, fzf to select it automatically. And exit if there are no remotes at this point – this can happen if something/someone deletes the config.

select remote

function validate_selection() {
    local selection="${1}"
    local error_message="${2}"
    if [ -z "${selection}" ]; then
        [ -n "${error_message}" ] && printf '%b \uea87 %b %s\n' "${RED}" "${RESET}" "${error_message}"
        exit 137
    fi
}
remote=$(printf '%s' "${remotes}" | fzf -0 -1 --prompt 'Choose remote to sync: ')
validate_selection "${remote}" 'No remote selected.'

If nothing is selected, exit with error code 137, it means that the user cancelled the operation.

Get the Mapping File Path

The script will look slimmer from now own because we now have some helper functions.

get mapping file

mapping_file=$(get_file_path "sync/${remote}-mapping.tsv")
check_file "${mapping_file}" 'Mapping'

See? just two lines!

Backup or Restore?

Now that we have everything we need, the user now needs to decide whether to backup or restore directories. Since we use fzf anyway, we can make the decision using that.

set sync type

choice=$(printf 'Backup\nRestore' | fzf --prompt "What would you like to do using ${remote}? ")
choice=$(printf '%s' "${choice}" | tr '[:upper:]' '[:lower:]')
validate_selection "${choice}"

I converted the choice variable to lower case because I will write bash functions with the same name later, and I want to use the choice variable to run it.

Select What to Sync

This is another important part, I may want to sync everything, or a single directory, or sometimes select multiple directories.

I used the multiselect functionality of FZF to do this, with my own keybindings:

  • tab - Toggle current selection.
  • Ctrl+t - Toggle all selections.
  • Ctrl+a - Select all.

select directory to sync

directories="$(sort -n "${mapping_file}" | envsubst)"
selections=$(printf '%b' "$directories" | fzf --bind 'ctrl-a:select-all' --bind 'ctrl-t:toggle-all' --multi --prompt "Select directories to ${choice}: " | cut -f2)
validate_selection "${selections}"

The sort -n "${mapping_file}" can sort the files by their priority. The -n will do a numeric sort so that 10 will appear after 9 instead of 1.

Do the Sync

Now that we have everything we need, it is time to sync the directories. For this, I created two bash functions, backup and restore.

sync functions

function backup() {
    local remote_path="${1}"
    local local_path="${2}"
    [ -z "${local_path}" ] && local_path="${HOME}/${remote_path}"
    printf '%b \uea74 %b Backing up "%s" directory to "%s"\n' "${BLUE}" "${RESET}" "${local_path}" "${remote}:${remote_path}"
    rclone sync --interactive -v "${local_path}" "${remote}:${remote_path}" --progress
    printf '%b \uf05d %b "%s" directory has been backed up to "%s"!\n' "${GREEN}" "${RESET}" "${local_path}" "${remote}:${remote_path}"
}
function restore() {
    local remote_path="${1}"
    local local_path="${2}"
    [ -z "${local_path}" ] && local_path="${HOME}/${remote_path}"
    printf '%b \uea74 %b Restoring "%s" directory from "%s"\n' "${BLUE}" "${RESET}" "${local_path}" "${remote}:${remote_path}"
    rclone sync --interactive -v "${remote}:${remote_path}" "${local_path}" --progress
    printf '%b \uf05d %b "%s" directory has been restored from "%s"!\n' "${GREEN}" "${RESET}" "${local_path}" "${remote}:${remote_path}"
}

Most of it is repeated code, but I am leaving it as it is because I sometimes tweak just one of them depending on the situation.

The function names are identical to the choice variable we set before. So we can invoke the function using that variable… because bash allows it.

sync directories

printf 'Attempting to %s selected directories from the remote "%s".
' "${choice}" "${remote}"
while IFS=$'\t' read -u 3 -r -a selection; do
  "${choice}" "${selection[@]}"
done 3<<<"${selections}"

I am just iterating through all selected mappings, and call either the backup or the restore function based on the initial choice.

The Complete Script

~/sync.sh

#!/bin/env bash
BLUE="\e[1;34m"
RED="\e[1;31m"
GREEN="\e[1;32m"
RESET="\e[0m"
function check_file() {
  local file_path="${1}"
  local file_type="${2}"
  if [ ! -f "${1}" ]; then
  printf '%b \uea87 %b %s file not found: %s\n' "${RED}" "${RESET}" "${file_type}" "${file_path}" 1>&2
  exit 1
  fi
}
function get_file_path() {
  local relative_path="${1}"
  local absolute_path
  if [ -z "$XDG_CONFIG_HOME" ] || [ ! -d "$XDG_CONFIG_HOME" ]; then
    absolute_path=$(realpath "$HOME/.config/${relative_path}")
  else
    absolute_path=$(realpath "${XDG_CONFIG_HOME}/${relative_path}")
  fi
  printf '%s' "${absolute_path}"
}
function validate_selection() {
  local selection="${1}"
  local error_message="${2}"
  if [ -z "${selection}" ]; then
    [ -n "${error_message}" ] && printf '%b \uea87 %b %s\n' "${RED}" "${RESET}" "${error_message}"
    exit 137
  fi
}
function backup() {
  local remote_path="${1}"
  local local_path="${2}"
  [ -z "${local_path}" ] && local_path="${HOME}/${remote_path}"
  printf '%b \uea74 %b Backing up "%s" directory to "%s"\n' "${BLUE}" "${RESET}" "${local_path}" "${remote}:${remote_path}"
  rclone sync --interactive --progress "${local_path}" "${remote}:${remote_path}"
  printf '%b \uf05d %b "%s" directory has been backed up to "%s"!\n' "${GREEN}" "${RESET}" "${local_path}" "${remote}:${remote_path}"
}
function restore() {
  local remote_path="${1}"
  local local_path="${2}"
  [ -z "${local_path}" ] && local_path="${HOME}/${remote_path}"
  printf '%b \uea74 %b Restoring "%s" directory from "%s"\n' "${BLUE}" "${RESET}" "${local_path}" "${remote}:${remote_path}"
  rclone sync --interactive --progress "${remote}:${remote_path}" "${local_path}"
  printf '%b \uf05d %b "%s" directory has been restored from "%s"!\n' "${GREEN}" "${RESET}" "${local_path}" "${remote}:${remote_path}"
}
rclone_config=$(get_file_path 'rclone/rclone.conf')
check_file "${rclone_config}" 'Rclone config'
remotes=$(rclone listremotes | cut -d: -f1)
if [ -z "${remotes}" ]; then
  printf '%b \uea87 %b %s %s\n' "${RED}" "${RESET}" 'No remote was found in the Rclone config: ' "${rclone_config}"
  exit 1
fi
remote=$(printf '%s' "${remotes}" | fzf -0 -1 --prompt 'Choose remote to sync: ')
validate_selection "${remote}" 'No remote selected.'
mapping_file=$(get_file_path "sync/${remote}-mapping.tsv")
check_file "${mapping_file}" 'Mapping'
choice=$(printf 'Backup\nRestore' | fzf --prompt "What would you like to do using ${remote}? ")
choice=$(printf '%s' "${choice}" | tr '[:upper:]' '[:lower:]')
validate_selection "${choice}"
directories="$(sort -n "${mapping_file}" | envsubst)"
selections=$(printf '%b' "$directories" | fzf --bind 'ctrl-a:select-all' --bind 'ctrl-t:toggle-all' --multi --prompt "Select directories to ${choice}: " | cut -f2,3)
validate_selection "${selections}"
printf 'Attempting to %s selected directories from the remote "%s".
' "${choice}" "${remote}"
while IFS=$'\t' read -u 3 -r -a selection; do
  "${choice}" "${selection[@]}"
done 3<<<"${selections}"

If you think it looks a bit messy, that’s because it is a messy code I hacked together when I had some spare time. So it needs a lot of improvement, such as enable/disable interactive mode, support bisync (currently in beta), etc., but it does the job for me.

In future, I may create a tool similar to FZguard so that I can do many more things with fewer key strokes.

  • Stay tuned!

Read the original on codingotaku.com ↗

Comments

Nothing yet. Say the first thing.

    Sign in to join the conversation.