Ansible: Handlers Are Not Your Friend

In our last installment, we talked about the problem with Ansible dependency tracking. While annoying, the only side effect is longer run times. Ansible’s handlers are far more dangerous and problematic. I learned Ansible after spending 10 years working with Puppet. Ansible’s handlers seemed like a great way to emulate Puppet’s notify API. Unfortunately, Ansible’s handlers are not reliable and scoping means they may not cut back on repetitive processes.

Join me for a walk into madness as we collectively learn why you should avoid handlers and what you might try instead.

A Look at the Problem

Handlers seem like a great way to add conditionally run tasks to roles and playbooks in a way that deduplicates their execution. Unfortunately, given Ansible’s sequential and fault averse defaults, they can cause more problems than they solve.

Ansible’s Handlers

An Ansible playbook can contain more than one play, and each play runs the following sections in this order: pre_tasks, roles, tasks, and post_tasks, e.g.:

- hosts: all
  pre_tasks:
    - debug:
        msg: first
  roles:
    - second
  tasks:
    - debug:
        msg: third
  post_tasks:
    - debug:
        msg: fourth

Ansible’s handlers run at the end of each section. Any handlers notified in the pre_tasks will execute before the roles section executes, etc. Handlers can be defined in a handlers section of the play or in a role in the role_name/handlers/main.yaml file. Handlers load into the same global namespace, but they load and run in the order they appear in execution.

If you want better control, you can force all pending handlers to execute using a meta task:

- name: Flush all notified handlers
  meta: flush_handlers

Here we setup an example that reloads sshd when it’s config file changes:

playbook.yaml
roles/ssh/
  handlers/main.yaml
  tasks/main.yaml

With the contents:

# roles/ssh/handlers/main.yaml
---
- name: "reload sshd"
  service:
    name: sshd
    state: restarted


# roles/ssh/tasks/main.yaml
---
- name: "install sshd_config"
  template:
    src: "sshd_config.j2"
    dest: "/etc/ssh/sshd_config"
  notify:
    - reload sshd
- name: "Start/Enable sshd"
  service:
    name: sshd
    state: started
    enabled: true

# playbook.yaml
---
- hosts: all
  roles:
    - ssh

This example is pretty short and works well. However, if I inject failure into the Ansible playbook between the install sshd_config and the end of this section of the playbook, the service will not restart.

# Ansible
- hosts: all
  roles:
    - ssh
    - role_with_an_exception

When the role_with_an_exception runs, the playbook is terminated early, before the handler is flushed. Since there’s no state stored, if I fix the exception in the other role, when I re-run the playbook, /etc/ssh/sshd_config probably won’t change and sshd won’t be reloaded.

wat.

In order to ensure the sshd handlers are notified, we need to add another task to the end of the roles/ssh/tasks/main.yaml:

# roles/ssh/tasks/main.yaml
---
- name: "install sshd_config"
  template:
    src: "sshd_config.j2"
    dest: "/etc/ssh/sshd_config"
  notify:
    - reload sshd
- name: "Start/Enable sshd"
  service:
    name: sshd
    state: started
    enabled: true
- name: "Ensure sshd handlers are run"
  meta: flush_handlers

This won’t prevent failures in the ssh role from skipping the handler run, but it’s more reliable. You’ll need to do add that to the end of all the roles you want to sure up.

The Verdict: Do Not User Handlers

Unfortunately, there’s no native Ansible solution to the problem. Do NOT use Ansible handlers in production. Seriously, just don’t.

A Solution for Self-Contained Resources

If you have a foundational service like sshd that can be self-contained in its own role, it’s best to register variables and check states manually in the role.

Rewriting the previous example to use inline tasks:

# roles/ssh/tasks/main.yaml
---
- name: "install sshd_config"
  template:
    src: "sshd_config.j2"
    dest: "/etc/ssh/sshd_config"
  register: sshd_config_file
- name: "Start/Enable sshd"
  service:
    name: sshd
    state: started
    enabled: true
  register: sshd_service
- name: "Reload sshd when necessary"
  service:
    name: sshd
    state: reloaded
  when:
    - sshd_config_file is changed
    - sshd_service is not changed

This example isn’t perfect, as the change to the enabled state of sshd could prevent the reload of the daemon when a config file is changed. I might change this to be more clever to work around extra tasks:

# roles/ssh/tasks/main.yaml
---
- name: "install sshd_config"
  template:
    src: "sshd_config.j2"
    dest: "/etc/ssh/sshd_config"
  register: sshd_config_file
- name: "Manage sshd service"
  service:
    name: sshd
    state: "{{ 'reloaded' if sshd_config_file is changed else 'started' }}"
    enabled: true

(This assumes reload will start if stopped, which I think is true on systemd).

This isn’t as hip as using a primitive designed to deduplicate conditionally executed events, but it is much more reliable. It can still suffer from the problem of a failed task between the template and the service definition resulting in a service that’s not correctly configured.

What do we when we want to ensure Ansible will run a conditional task exactly once? We need to build our own.

A Solution for Shared Resources and Exactly Once Execution

Some services may have shared config. A firewall role may allow other roles to add rules for the services they provide. Flushing the handlers after each role is applied could cause the firewall to reload frequently. That may be OK, but imagine a service like Redis that may require a copy from memory to disk at stop, and another copy from disk to memory at start. If the handler restarts Redis multiple times, that’s going to cause serious churn and potentially result in errors or latency spikes.

I designed a solution to allow multiple role to “notify” a shared task that it should run. It assumes we’re using Ansible in local mode, which allows us to reference files installed on the local system. It does require the roles and playbooks be installed on all the systems in our infrastructure. The playbook harness now includes a pre_tasks and post_tasks section for setting up and executing these tasks.

# playbook.yaml
---
- hosts: all
  vars:
    post_tasks_dir: '/var/lib/ansible/post_tasks'

  pre_tasks:
    - name: "Setup for post tasks"
      include_tasks: "tasks/post_tasks/setup.yaml"

  roles:
    - ssh
    - role_with_an_exception

  post_tasks:
    - name: "Run post_tasks"
      include_tasks: "tasks/post_tasks/run.yaml"

The pre_tasks setup just creates the directory we’ll use to persist requested post_tasks.

# tasks/post_tasks/setup.yaml
---
- name: "Create directory for post tasks"
  file:
    path: "{{ post_tasks_dir }}"
    state: directory
    mode: "0700"

The run task loops through the existing post task files and calls run_one.yaml.

# tasks/post_tasks/run.yaml
---
- name: "Run post tasks"
  include_tasks: run_one.yaml
  with_fileglob: "{{ post_tasks_dir }}/*.yaml"
  loop_control:
    loop_var: post_task_yaml

A post_tasks section scans for un-run post tasks and runs them, cleaning them up once they complete successfully.

# tasks/post_tasks/run_one.yaml
---
- name: "Run the task"
  include_tasks: "{{ post_task_yaml }}"
- name: "Remove the file once it runs successfully"
  file:
    path: "{{ post_task_yaml }}"
    ensure: absent
  changed_when: false

The roles execute and use the global tasks/post_tasks/install.yaml to install tasks.

# tasks/post_tasks/install.yaml
---
- name: "Install post task"
  template:
    src: "{{ item }}"
    dest: "{{ post_tasks_dir }}/{{ task_name }}.yaml"
  with_first_found:
    - files:
        - "{{ role_path }}/post_tasks/{{ task_name }}.yaml"
        - post_tasks/{{ task_name }}.yaml

For our sshd example, we need to add a few pieces to the role. First, we need a roles/ssh/post_tasks/reload_sshd.yaml:

# roles/ssh/post_tasks/reload_sshd.yaml
---
- name: "reload sshd"
  service:
    name: sshd
    state: reloaded

In the tasks/main.yaml we need to call the global tasks/post_tasks/install.yaml task with the task name.

# roles/ssh/tasks/main.yaml
---
- name: "install sshd_config"
  template:
    src: "sshd_config.j2"
    dest: "/etc/ssh/sshd_config"
  register: result_sshd_config

- name: "install post task to reload sshd"
  vars:
    task_name: "reload_sshd"
  include_tasks: "tasks/post_tasks/install.yaml"

- name: "Start/Enable sshd"
  service:
    name: sshd
    state: started
    enabled: true

By applying this approach, all out-standing post_tasks run during the next execution of Ansible, so there’s no missed task executions!

What Did We Learn?

This solution is far from perfect, off-hand, these issues exist:

1. The ordering of the post_tasks depends on the task_names.
2. In the roles/ssh example we still have a situation where we can start and reload sshd on hosts where sshd is stopped.
3. There are two distinct tasks to accomplish this sharing state: the template step with the register, and the global post_tasks/install.yaml. If someone inserts a task between those two and it fails, we still lose state. (If someone suggests a block, slap them in the face with a trout, but that’s another show.)

Of these issues, #1 and #2 we can work-around with some more Ansible. Unfortunately, #3 is fundamentally impossible to solve without rewriting Ansible to record states, including requested, yet-to-be-executed handlers so subsequent runs would be able to provide guarantees for future runs. This is a lot of work, and would likely be rejected by the Ansible devs as “unnecessary” or only addressing “rare” events where the operator should just know not to run a playbook that throws exceptions in production.

Ansible does not remember state across runs. Handlers provide no guarantees and often leave systems in inconsistent states. When a downstream role fails on a subset of hosts, the operator has to go back and manually verify/validate the CM run.

Puppet takes flack for its eventual consistency model, but I see that as a strength. Real systems are messy. Unexpected things happen. Even rare events, when scattered across thousands of systems executing hundreds of times per day, become routine. Ansible fails to address this reality, forcing the user to account for all these edge cases themselves.