Administration

Common etcd admin SOPs:

• Create Cluster: Initialize an etcd cluster
• Destroy Cluster: Destroy an etcd cluster
• CLI Environment: Configure etcd client to access server cluster
• RBAC Authentication: Use etcd RBAC auth
• Reload Config: Update etcd server member list for clients
• Add Member: Add new member to existing etcd cluster
• Remove Member: Remove member from etcd cluster
• Utility Scripts: Simplify ops with bin/etcd-add and bin/etcd-rm

For more, refer to FAQ: ETCD.

Create Cluster

Define etcd cluster in config inventory:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars: { etcd_cluster: etcd }

Run etcd.yml playbook:

./etcd.yml  # initialize etcd cluster

For prod etcd clusters, enable safeguard etcd_safeguard to prevent accidental deletion.

Destroy Cluster

Use the dedicated etcd-rm.yml playbook to destroy an Etcd cluster. The default etcd_rm_data: true deletes local data and configuration. First confirm that no PostgreSQL cluster still uses it as DCS, verify a recent backup, and check the exact target name.

./etcd-rm.yml -l etcd                            # Destroy the entire Etcd cluster after confirmation
./etcd-rm.yml -l etcd -e etcd_safeguard=false   # Override only when the inventory enables the safeguard

Or use utility script:

bin/etcd-rm                           # remove entire etcd cluster

The removal playbook respects etcd_safeguard. If true, it aborts before leaving the cluster, deregistering, stopping the service, or deleting files. Its default is false, so the absence of an explicit override is not itself a confirmation.

CLI Environment

Uses etcd v3 API by default (v2 removed in v3.6+). Pigsty auto-configures env script /etc/profile.d/etcdctl.sh on etcd nodes, loaded on login.

Example client env config:

alias e="etcdctl"
alias em="etcdctl member"
export ETCDCTL_ENDPOINTS=https://10.10.10.10:2379
export ETCDCTL_CACERT=/etc/etcd/ca.crt
export ETCDCTL_CERT=/etc/etcd/server.crt
export ETCDCTL_KEY=/etc/etcd/server.key

Since v4.0, Pigsty has enabled RBAC auth for etcd by default, so user auth is still required:

export ETCDCTL_USER="root:$(cat /etc/etcd/etcd.pass)"

After configuring client env, run etcd CRUD ops:

e put a 10 ; e get a; e del a   # basic KV ops
e member list                    # list cluster members
e endpoint health                # check endpoint health
e endpoint status                # view endpoint status

RBAC Authentication

Since v4.0, Pigsty has enabled etcd RBAC auth by default. During cluster init, etcd_auth task auto-creates root user and enables auth.

Root user password set by etcd_root_password, default: Etcd.Root. Stored in /etc/etcd/etcd.pass with 0640 perms (root-owned, etcd-group readable).

Strongly recommended to change default password in prod:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars:
    etcd_cluster: etcd
    etcd_root_password: 'YourSecurePassword'  # change default

Client auth methods:

# Method 1: env vars (recommended, auto-configured in /etc/profile.d/etcdctl.sh)
export ETCDCTL_USER="root:$(cat /etc/etcd/etcd.pass)"

# Method 2: command line
etcdctl --user root:YourSecurePassword member list

Patroni and etcd auth:

Patroni uses pg_etcd_password to configure etcd connection password. If empty, Patroni uses cluster name as password (not recommended). Configure separate etcd password per PG cluster in prod.

Reload Config

If etcd cluster membership changes (add/remove members), refresh etcd service endpoint references. These etcd refs in Pigsty need updates:

Refresh etcd member config:

./etcd.yml -t etcd_conf                           # refresh /etc/etcd/etcd.conf
ansible etcd -f 1 -b -a 'systemctl restart etcd'  # optional: restart etcd instances

Refresh etcdctl client env:

./etcd.yml -t etcd_config                         # refresh /etc/profile.d/etcdctl.sh

Update Patroni DCS endpoint config:

./pgsql.yml -t pg_conf                            # regenerate patroni config
ansible all -f 1 -b -a 'systemctl reload patroni' # reload patroni config

Update VIP-Manager endpoint config (only for PGSQL L2 VIP):

./pgsql.yml -t pg_vip_config                           # regenerate vip-manager config
ansible all -f 1 -b -a 'systemctl restart vip-manager' # restart vip-manager

Add Member

ETCD Reference: Add a member

Recommended: Utility Script

Use bin/etcd-add script to add new members to existing etcd cluster:

# First add new member definition to config inventory, then:
bin/etcd-add <ip>              # add single new member
bin/etcd-add <ip1> <ip2> ...   # add multiple new members

The script attempts these operations in order:

• Validates IP address validity
• Executes etcd.yml playbook (auto-sets etcd_init=existing)
• Provides safety warnings and countdown
• Prompts config refresh commands after completion

Manual: Step-by-Step

Add new member to existing etcd cluster:

1. Update config inventory: Add new instance to etcd group
2. Notify cluster: Run etcdctl member add (optional, playbook auto-does this)
3. Initialize new member: Run playbook with etcd_init=existing parameter
4. Promote member: Promote learner to full member (optional, required when using etcd_learner=true)
5. Reload config: Update etcd endpoint references for all clients

# After config inventory update, initialize new member
./etcd.yml -l <new_ins_ip> -e etcd_init=existing

# If using learner mode, manually promote
etcdctl member promote <new_ins_server_id>

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 } # <--- only existing instance in cluster
    10.10.10.11: { etcd_seq: 2 } # <--- add this new member to inventory
  vars: { etcd_cluster: etcd }

$ bin/etcd-add 10.10.10.11

$ etcdctl member add etcd-2 --learner=true --peer-urls=https://10.10.10.11:2380
Member 33631ba6ced84cf8 added to cluster 6646fbcf5debc68f

ETCD_NAME="etcd-2"
ETCD_INITIAL_CLUSTER="etcd-2=https://10.10.10.11:2380,etcd-1=https://10.10.10.10:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.10.10.11:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

33631ba6ced84cf8, unstarted, , https://10.10.10.11:2380, , true       # unstarted new member here
429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false

$ ./etcd.yml -l 10.10.10.11 -e etcd_init=existing    # must add existing parameter
...
33631ba6ced84cf8, started, etcd-2, https://10.10.10.11:2380, https://10.10.10.11:2379, true
429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false

$ etcdctl member promote 33631ba6ced84cf8   # promote learner to follower
Member 33631ba6ced84cf8 promoted in cluster 6646fbcf5debc68f

$ em list                # check again, new member promoted to full member
33631ba6ced84cf8, started, etcd-2, https://10.10.10.11:2380, https://10.10.10.11:2379, false
429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false

Remove Member

Recommended: Utility Script

Use bin/etcd-rm script to remove members from etcd cluster:

bin/etcd-rm <ip>              # remove specified member
bin/etcd-rm <ip1> <ip2> ...   # remove multiple members
bin/etcd-rm                   # remove entire etcd cluster

Script auto-performs:

• Gracefully removes members from cluster
• Stops and disables etcd service
• Cleans up data and config files
• Deregisters from monitoring system

The underlying removal role tolerates some leave and cleanup errors. After the script finishes, inspect etcdctl member list, endpoint health, remaining quorum, and the actual service and data-directory state on the target.

Manual: Step-by-Step

Remove member instance from etcd cluster:

1. Keep the member in the inventory: The removal playbook needs its etcd_seq, cluster members, and connection endpoints
2. Clean up the instance: Run etcd-rm.yml against the target; it attempts member remove, stops the service, and cleans up according to the removal flags
3. Update the inventory: Comment out or delete the member only after the playbook succeeds
4. Reload references: Follow Reload Config for the remaining etcd members and the Patroni/VIP-Manager endpoints

# <ip> must still be in the etcd inventory group
./etcd-rm.yml -l <ip>                  # Leave the cluster and clean up automatically
# After success, remove the member from pigsty.yml and refresh the remaining members and clients

Do not delete the target from the inventory before running the removal playbook. The hosts: etcd scope in etcd-rm.yml would no longer select it, and the playbook could not derive the instance identity or cluster endpoints from inventory. There is also no need to repeat etcdctl member remove before or after the playbook.

$ bin/etcd-rm 10.10.10.12

$ ./etcd-rm.yml -l 10.10.10.12

$ etcdctl member list
429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false
33631ba6ced84cf8, started, etcd-2, https://10.10.10.11:2380, https://10.10.10.11:2379, false
93fcf23b220473fb, started, etcd-3, https://10.10.10.12:2380, https://10.10.10.12:2379, false  # <--- remove this

$ etcdctl member remove 93fcf23b220473fb # kick from cluster
Member 93fcf23b220473fb removed from cluster 6646fbcf5debc68f

Utility Scripts

v3.6+ provides utility scripts to simplify etcd cluster scaling:

bin/etcd-add

Add new members to existing etcd cluster:

bin/etcd-add <ip>              # add single new member
bin/etcd-add <ip1> <ip2> ...   # add multiple new members

Script features:

• Validates IP addresses in config inventory
• Auto-sets etcd_init=existing parameter
• Executes etcd.yml playbook to complete member addition
• Prompts config refresh commands after completion

bin/etcd-rm

Remove members or entire cluster from etcd:

bin/etcd-rm <ip>              # remove specified member
bin/etcd-rm <ip1> <ip2> ...   # remove multiple members
bin/etcd-rm                   # remove entire etcd cluster

Script features:

• Provides safety warnings and confirmation countdown
• Auto-executes etcd-rm.yml playbook
• Gracefully removes members from cluster
• Cleans up data and config files

Manage Etcd Password

The etcd_root_password parameter defines the password of the etcd root user.

To change it, reach an etcd endpoint and run the following as the admin user on an INFRA node or ETCD node:

e user passwd root  # change the etcd root user password

Then refresh every reference to the etcd root password, including the Patroni client configuration on INFRA nodes and the etcdctl client environment:

./infra.yml -t env_patroni    # refresh the etcd root password in /infra/conf/patronictl.yml
./etcd.yml  -t etcd_conf      # refresh /etc/etcd/etcd.pass and /etc/profile.d/etcdctl.sh