Ansible Reflections

I published two articles critical of Ansible dependencies and handlers. If you read those articles, you might be surprised that I really like Ansible. I spent 10 years bumping into all the sharp corners. In that time, I managed to create one of the most successful projects of my career, full lifecycle management of on-prem hardware with Ansible. It started as a playbook of re-usable tasks to perform firmware, kernel, and OS upgrades on hosts in our infrastructure. It soon grew with my help from my colleagues to provision, audit, decomission, and manage servers, switches, and firewalls.

Ansible made it possible for a small team of 8 to manage over 1,500 devices with enough capacity to support development efforts and innovate our own services and tooling. The support for remote management of blackbox devices like firewall, load balancers, routers, and switches provide capabilities to synchronize server changes with network and routing devices. The serial functionality and error sensitivity make it safe to point at a big batch of hosts and say “take these potentially destructive actions” and have it bail on the first sign of a problem.

Ansible is an amazing orchestration framework, but it is a poor choice for traditional CM systems where you want continual evaluation and correction to a determined baseline. In this article, we’ll explore the strengths and weaknesses.

Ansible Strengths

Beyond the extensive device support, Ansible has a few really great features. I often find myself reaching for, and missing them when working with other tooling.

Strength: JIT Template Expansion

Ansible supports Jinja2 templates almost anywhere. The linear, deterministic approach in Ansible has a spiffy side effect. Templates don’t expand when loading the YAML, rather they expand as needed. This feature creates an opportunity to build variables from intermediate variables to simplify logic in a task.

- name: "Build a variable from other variables"
  vars:
    vcpus: 48
    cores_per_cpu: "{{ 4 if vcpus | int > 24 else 2 }}"
    cores: "{{ (vcpus | int) / (cores_per_cpu | int) }}"
  debug:
    var: cores

This seems trivial here, but it can make complex operations, like transforming an array of dicts into a different structure to use for other tasks. I most commonly use this pattern to build up a global dict for tracking purposes:

- name: "Don't reinstall if we've already installed"
  vars:
    foo_instances: "{{ foo_instances | default({}) }}"
  include_tasks: install.yaml
  when: not foo_instance in foo_instances

- name: "Track this instance of foo"
  set_facts:
    foo_instances: "{{ foo_instances | default({}) | combine({ instance: true }) }}"

This pattern wouldn’t be possible without the JIT template expansions.

Strength: The Inventory, a.k.a Single Source of Truth

Ansible’s inventory of hosts, groups, and variables acts as a single source of truth from which I can derive configurations. Rather than creating an explicit list of Kafka servers, I can query the inventory to get a list of local Kafka servers for all the client applications.

If I have a group of Kafka servers in Production in the SFO1 datacenter, I would create an inventory with attributes including Build=kafka, Environment=production, DC=sfo1. Then I would create a group for all production Kafka servers in SFO1, maybe: production.sfo1.kafka. Then any service that needs Kafka configurations can use a template that expands the group:

# Example Jinja2 Template Expanding Groups
{% set myGroup = Environment + '.' + DC + '.' + Build %}
bootstrap_server = {% for node in groups[myGroup] | sort %}{{ ansible_inventory[node].FQDNP }}:{{ kafka_port }},{% endfor %}

If we add or remove Kafka nodes, all our clients adjust automatically! We could also filter that list based on properties of those Kafka servers in the inventory. I could check the Status property to make sure it’s online, if hostvars[node].Status == "online".

This Single Source of Truth model is incredibly powerful for removing toil and avoiding misconfigurations in large environments. If you don’t have a Single of Source of Truth somewhere in your infrastructure, my conservative estimate is creating and adopting one will likely reduce the “toil” in your systems by 30% or more.

Weaknesses Abound

Almost every weakness has a common thread, Ansible does not record it’s state between runs. The system is the state, so to say. This is fine for orchestration, but for consistency it’s not good enough. This shows up in the absurdity that is handlers, and creates other problems.

Weakness: Confusing and Discouraged Features

Ansible ships with features that are confusing. Despite constant feedback, the developers have not addressed these concerns beyond notices in the documentation. These include:

• block - The only use for the block statement I can condone is the rescue option. But even that can be done without a block using ignore_errors: true with register and checking if the result was an error
• import_playbook and import_tasks - Go search Stack Overflow and count the number of times those show up with when: attached and confused users. You never want to use import_playbook or import_tasks. If you think you do, take a deep breath, and repeat after me, “I don’t need import_tasks or import_playbook”
• Terrible flow control
  • No if/else - again, a big complaint on Stack Overflow
  • Loops are tricky as their length must be known ahead of time (you get used to this, but it’s hard to understand at first)
  • Make heavy use of include_tasks anytime you need to branch or perform the guts of a loop
• File management, but we’ll get to that…
• Role dependencies, but we covered that
• Limited dimension of parallelization. You can run the same tasks on a bunch of hosts in parallel, but you can’t run a bunch of tasks on the same host in parallel. “But, Brad, async?” No. Stop hitting yourself. The only acceptable use of async is to spawn a long running job and then immediately wait on it. “That’s not async!” Yes, I know, and that’s my point.

Weakness: Inventory Variable Merging

While I appreciate a lot about the Ansible inventory, there is one thing it could do better. Ansible’s merge policy for nested data structures uses a replacement strategy. This makes it harder to work with. You can change the policy in the ansible.cfg, but it should really be a property of the inventory itself as that would be a more declarative approach. Puppet’s Hiera does this much better than Ansible. I always wind up creating a Hiera clone to integrate with the inventory.

Weakness: PyYAML

Ansible uses the PyYAML implementation. There are a lot of open bugs. The developers don’t seem interested in patching some glaring issues failing to correctly implement the YAML Specification. It’s not uncommon for PyYAML to change the value and type of your data when it decodes and then encodes from a YAML file you hand managed. I found it necessary to use the Perl YAML::XS library anywhere I wanted to manipulate hand-crafted YAML files for use as Ansible variable files. Perl is lazy and has very few data types, so there’s no risk of tripping over weird PyYAML casting errors.

Weakness: File Management

I have a background in Security. I manage File Integrity Monitoring (FIM) systems for compliance iniatitives requiring timely audits of file change events. With Ansible, there is no catalog of the expected file states. Worse, there are separate action modules for managing files different ways:

• file - Manages file attributes, presence, and type
• copy - Copies a file to a place, also manages attributes
• template - Same as copy, except with template evaluation
• synchronize - Recursive directory synchronization, somehow has more features and less function than you need

In order to capture the desired checksums, you need to write a callback plugin to figure out what happened. Does the module report the checksums of all the files? If not, you enter a race in the time since the change and when you run the checksum yourself. An excellent Python programmer and Ansible afficianado at my last job was able to put together just such a callback plugin, but it was incredibly limited to the things we used and still had gaps in some places.

Contrast this with Puppet’s file resource which can do all the things the previous Ansible action modules do. Puppet indexes every file it touches in its catalog, recording the previous and current checksums. This makes me, and arguably, the entire company more efficient in areas where we need to adhere to regulatory compliance. Instead of investigating every intended and un-intended file change on every system, I can filter the file change events at the end points against the Puppet catalog. If the change event’s resulting checksum matches the checksum the FIM system detected, we can close the event immediately as an intended change!

In Which He Quotes Pirates of The Carribean

Where does this leave us? Use Ansible for orchestration. Use Ansible to build containers or packages. Use Ansible to spin up disposable infrastructure. Use Ansible to manage your blackbox appliances. Use Ansible to munge the grotesque and gargantuant hoardes of YAML Kubernetes forces down your throat. But, for the love of all that is righteous, do not use it instead of Puppet or Chef. The niche for those managing traditional server-based infrastructure maybe shrinking as everyone falls all over themselves to hand piles and piles of cash to AWS to run unnecessary YAML processing pipelines filled with repetitive and redundant YAML due to missing template primitives in Kubernetes. If you’re reading this and wondering “Should I use Ansible or Puppet?” I am here to tell you, the answer is probably both. Use Ansible to orchrestrate the lifecycle and configure Puppet on the end points. Please don’t try to turn Ansible into a traditional CM, it’s not. It can’t be. It’s not designed for it.

In other words:

“It’s terrible luck to bring Ansible on board.” “Aye, but it’s much worse not to…”