Jinja2 for better Ansible playbooks and templates

There have been posts about Ansible on this blog before, so this one will not go into Ansible basics again, but focus on ways to improve your use of variables, often, but not only used together with the template module, showing some of the more involved features its Jinja 2 -based implementation offers.

The examples are mostly taken right out of our Ansible provisioning for CenterDevice, with only slight adaptations for conciseness.

Basic Variable Access

The bare metal machines we use as the basis for our OpenStack infrastructure have different capabilities. We use this information to set up host aggregates.
The Ansible inventory sets a VENDOR_MODEL host variable for each machine:

1[nodes]  
2node01.baremetal VENDOR_MODEL="Dell R510"  
3node02.baremetal VENDOR_MODEL="Dell R510"  
4node03.baremetal VENDOR_MODEL="Dell R510"  
5node04.baremetal VENDOR_MODEL="Dell R510"  
6…
7node0x.baremetal VENDOR_MODEL="Dell R420"

For use in playbooks (and in templates) Ansible automatically puts it into the hostvars dictionary. ansible_hostname is just another regular variable expansion.

1shell: nova aggregate-add-host "{{ VENDOR_MODEL }}" "{{ ansible_hostname }}"

Sometimes, though, just expanding pre-defined variables is not good enough. So let’s move on.

Registering Variables for shell & command output

Running external programs via the shell and command modules often produces output and exit codes you may want to use in the subsequent flow of your playbook.

1- name: Create temp file for some later task  
2  command: mktemp /tmp/ansible.XXXXXXXXX  
3  register: tmp_file

Using register like this captures the result of the command execution in a new dictionary variable called tmp_file. This contains, among other things, mktemp’s exit code and its standard out and standard err output. Knowing that mktemp prints the name of the created temp file to standard out lets us use it like so:

1- name: Copy some file to the temp location  
2  sudo: True  
3  copy: src=sourcefile dest={{ tmp_file.stdout }}

Often you are interested in the exit code of a command to base decisions on. If, for example, you grep for some search term, grep informs you via its exit code if it found the term or not:

1- name: check for gpg public key  
2  sudo: true  
3  shell: gpg --list-keys | grep {{ BACKUP_GPG_PUBLIC_KEY }}  
4  register: find_gpg_public_key  
5  always_run: true  
6  failed_when: find_gpg_public_key.rc > 1

This snipped uses gpg --list-keys and grep to check if the key is already known in the gpg keychain. grep exits with exit code 0 when it found the search term and 1 when it did not. To let Ansible know that, we tell it to only treat the command as failed_when the registered output dictionary’s rc member (which stores the exit code) is greater than 1, as per grep’s man page.

Combined with the temporary file created shown earlier, the following snippet gracefully handles importing of the gpg key into the keychain on the target system if not present yet and to continue without interruption if it already is:

1- name: check for gpg public key  
2  sudo: true  
3  shell: gpg --list-keys | grep {{ BACKUP_GPG_PUBLIC_KEY }}  
4  register: find_gpg_public_key  
5  always_run: true  
6  failed_when: find_gpg_public_key.rc > 1
7 
8- name: Create temp file for gpg public key  
9  command: mktemp /tmp/ansible.XXXXXXXXX  
10  register: gpg_public_key_tmp_file  
11  always_run: true  
12  when: find_gpg_public_key.rc == 1
13 
14- name: Create gpg public key  
15  sudo: true  
16  copy: src=root/gnupg/backup@centerdevice.de.pub dest={{ gpg_public_key_tmp_file.stdout }} owner=root group=root mode=0600  
17  when: find_gpg_public_key.rc == 1
18 
19- name: Import gpg public key  
20  sudo: true  
21  command: /usr/bin/gpg --import {{ gpg_public_key_tmp_file.stdout }}  
22  when: find_gpg_public_key.rc == 1
23 
24- name: Delete temp file for gpg public key  
25  sudo: true  
26  file: path={{ gpg_public_key_tmp_file.stdout }} state=absent  
27  when: find_gpg_public_key.rc == 1  
28  always_run: true

Capturing other tasks’ output

Tasks other than command or shell also provide result output that can be registered into variables. See this example, where we set up several MySQL servers for replication automatically (the roles come from host variables, set up in the inventory):

1- name: Create replication slave user on master  
2  sudo: true  
3  mysql_user: name=repl host='%' password={{ MYSQL_REPL_PASS }} priv=*.*:"REPLICATION SLAVE" state=present login_user=root login_password={{ MYSQL_ROOT_PASS }}  
4  when: mysql_repl_role == 'master'
5 
6- name: Check if slave is already configured for replication  
7  mysql_replication: mode=getslave login_user=root login_password={{ MYSQL_ROOT_PASS }}  
8  ignore_errors: true  
9  register: slave  
10  when: mysql_repl_role == 'slave'
11 
12- name: Get the master replication status  
13  mysql_replication: mode=getmaster login_user=root login_password={{ MYSQL_ROOT_PASS }}  
14  delegate_to: mysql01.local  
15  register: repl_stat  
16  when: slave|failed and mysql_repl_role == 'slave'
17 
18- name: Change the master in slave to start the replication  
19  mysql_replication: mode=changemaster master_host=mysql01.local master_log_file={{ repl_stat.File }} master_log_pos={{ repl_stat.Position }} master_user=repl master_password={{ MYSQL_REPL_PASS }} login_user=root login_password={{ MYSQL_ROOT_PASS }}  
20  when: slave|failed and mysql_repl_role == 'slave'
21 
22- name: Activate slave to start the replication  
23  mysql_replication: mode=startslave login_user=root login_password={{ MYSQL_ROOT_PASS }}  
24  when: mysql_repl_role == 'slave'

The mysql_replication module with its mode parameter set to getmaster sets values for the File and Position keys in its output, which gets registered as a variable named repl_stat. The values are then fed into the next task which configures the slave accordingly.

Moreover, the slave variable registers the outcome of the mysql_replication call with mode=getslave, because subsequent slave setup is only needed when the machine has not been set up as a slave yet. Asking for the slave status in that case fails with an error. To prevent Ansible from aborting right then and there, ignore_errors it set to True.

Functions, Filters and Control Structures in Templates

The examples so far have used variables in the context of playbooks to control the flow of execution and capture the result of command executions. The Jinja templating engine that Ansible utilized under the hood can do much more though. It offers a wide range of control structures, functions and filters.

range() and format() in a for-loop

Consider this example, where we set up (part of) a hosts file for name lookups for OpenVPN clients. We have reserved a range of 20 IP addresses for VPN client endpoints. Instead of adding them individually, this excerpt from the hosts.j2 template file does it for us:

1{% for id in range(201,221) %}  
2192.168.0.{{ id }} client{{ "%02d"|format(id-200) }}.vpn  
3{% endfor %}

First of all, you see {% and %} as delimiters for Jinja statement execution, in contrast to the already known {{ and }} for expression evaluation.

The first line counts the numbers from 201 to 220, storing the value in the id variable for each loop iteration.

The second line first simply evaluates id as the last byte of the IP address.
Following that you see a more complicated expression: It filters the Python format string %02d into the format filter, which applies it to the value of id minus 200, leading to this nicely aligned output:

1192.168.0.201 client01.vpn  
2192.168.0.202 client02.vpn  
3…
4192.168.0.220 client20.vpn

Combining multiple values

Combining several items into some kind of list is a common task when generating configuration files. Say you have a list host names in your inventory that an application requires as a comma separated list in a configuration file. The Jinja join filter allows you to quickly provide just that:

1servers={{ groups["nodes"] | join(",") }}

This produces a list of node names, joined by a comma. The default separator is a space, so you can just use the parameterless version join() if that suits your needs. join also supports complex objects via an optional second parameter that names the attribute whose values should be joined. The official Jinja documentation has this example:

1{{ users|join(', ', attribute='username') }}

Based on a list of user-objects, this joins the values of each object’s username attribute with a comma and a space, creating nicely readable string, suitable e. g. for a display on screen.

In some cases, though, join() is not powerful enough. In one of our configuration files we need a list of host name and port number combinations. For this, a slightly more advanced for loop construct can be used:

1mqa.host.list={% for node in groups['nodes'] %}{{ node }}:5672{% if not loop.last %},{% endif %}{% endfor %}

We simply iterate over the nodes group and output each node name, followed by the string :5672 (the port number). Naïvely adding the separator right after the port number would leave us with a trailing comma at the end of the line. To prevent that, the comma is conditionally added for all but the last element by checking for the special loop.last variable that Jinja automatically injects into the. There are a few more of these, useful in different scenarios. For your convenience, here is the list of special variables as of Jinja 2.8:

• loop.index: The current iteration of the loop (1 indexed).
• loop.index0: As before, but 0 indexed.
• loop.revindex: The number of iterations from the end of the loop (1 indexed).
• loop.revindex0: As above, but (0 indexed).
• loop.first: True if first iteration.
• loop.last: True if last iteration.
• loop.length: The number of items in the sequence.
• loop.depth: Indicates how deep in a recursive loop the rendering is. Starts at level 1.
• loop.depth0: As before, but starting with level 0.
• loop.cycle: A helper function to cycle between a list of sequences. See the Jinja documentation for more details on how to use this.

default() values

This example shows the use of the default filter which can provide a default value should there be no value to expand for a variable. It is part of a template for a shell script which gets generated as a tool for initializing new OpenStack virtual machine instances. Some of the values might just be undefined for a particular instance in which case default() can provide sane defaults:

1…  
2VOL_DATA="{{ item.data_volume | default('False') }}";  
3VOL_DATA_SIZE="{{ item.data_volume_size | default(0) }}";  
4…

The resulting shell script can rely on non-empty, sensible values for all variables instead of having to check for potentially empty shell variables all over the place.

Checking for a variable’s existence

Where default() can provide default values when there are no specific ones, sometimes you may want to decide based on a variable being defined at all or not. We use this, for example, to generate the /etc/network/interfaces file for our machines. Doing this manually with a somewhat more involved network configuration is error prone and brittle. Instead, we can now simply define the network setup declaratively and let Ansible handle the rest. In the following excerpt notice how certain sections are added to the resulting file or skipped altogether, depending on the presence of the item.gateway and item.default_gateway variables:

1{% for item in NETWORK_INTERFACES %}  
2auto {{ item.vlan_name }}  
3iface {{ item.vlan_name }} inet static  
4    address {{ item.ip_net }}.{{ HOST_IP_OCTET }}  
5    netmask 255.255.255.0  
6    network {{ item.ip_net }}.0  
7    broadcast {{ item.ip_net }}.255  
8    pre-up ip link add link {{ item.dev }} name {{ item.vlan_name }} type vlan id {{ item.vlan_tag }}  
9    pre-up ip link set dev {{ item.vlan_name }} mtu 9000  
10    pre-up ethtool -K {{ item.vlan_name }} gro off  
11    post-down ip link delete {{ item.vlan_name }}  
12{% if item.gateway is defined %}  
13    post-up ip route add {{ item.ip_net }}.0/24 dev {{ item.vlan_name }} table {{ item.rt_table }}  
14    post-up ip route add default via {{ item.gateway }} table {{ item.rt_table }}  
15    post-up ip rule add from {{ item.ip_net }}.{{ HOST_IP_OCTET }} table {{ item.rt_table }}  
16    pre-down ip rule delete from {{ item.ip_net }}.{{ HOST_IP_OCTET }} table {{ item.rt_table }}  
17    pre-down ip route delete default via {{ item.gateway }} table {{ item.rt_table }}  
18    pre-down ip route delete {{ item.ip_net }}.0/24 dev {{ item.vlan_name }} table {{ item.rt_table }}  
19{% endif %}  
20{% if item.default_gateway is defined %}  
21    post-up ip route add default via {{ item.default_gateway }}  
22    pre-down ip route delete default via {{ item.default_gateway }}  
23    dns-search baremetal.{{ DOMAIN }} {{ DOMAIN }}  
24    dns-nameservers 10.0.0.{{ HOST_IP_OCTET }} 8.8.8.8  
25{% endif %}
26{% endfor %}

Conclusion

The use of a well supported and powerful template library to supplement Ansible’s automation and remote control features provides a wide range of opportunities to output pretty complex files with a reasonable amount of effort. The examples above are by no means comprehensive and represent a snapshot of some of the techniques we are using so far. Just by looking at the full list of functions, filters, control structures etc. in the Jinja documentation, it is obvious that there is much more power there, waiting to be used.

Let me wrap this up, with a word of caution: As is true with any tool, sometimes getting too clever with it can make it difficult to understand what is going on when looking at the code later on. So instead of trying to come up with the shortest way to write something in a template, you might want to consider a more verbose solution to a problem – like a simple for loop instead of a clever combination of filters and functions – if that turns out to be the more pragmatic, more readable approach. Co-workers (and your future self) might be grateful for it 6 months down the road 🙂

A side note for all German speaking readers: In the free eBook “Cloud Fibel” we get even more into Ansible and Jinja2 as well as related topics like provisioning, monitoring and virtualization. The eBook is free for registered readers .