//

Highly available VIPs on OpenStack VMs with VRRP

11.11.2016 | 6 minutes of reading time

Introduction

Virtual IPs (VIPs) are widely used to enable high availabilty for IT infrastructure services. Although the use case is very common, it is surprisingly tricky to set up in an OpenStack environment without a deeper knowledge about neutron and its port security mechanisms. In this blog post I will walk you through the creation of a pair of VMs that share a VIP via the Virtual Router Redundancy Protocol (VRRP) on an OpenStack cloud.

Neutron setup

My OpenStack project has an internal network (192.168.0.0/24) and a router connecting it to an external network (10.98.208.0/24). To setup a HA VIP I need 3 ports that I named vip-port, vm1-port and vm2-port. The VM ports will connect my virtual machines to the internal network, the VIP port actually is just a dummy port that allocates an internal IP address. Neutron will not instantiate this port, it s just a database entry that “blocks” an IP address that will later be used on the VM ports. I want both VMs to be able to communicate via their own IP and the VIP, but the neutron port security mechanisms will drop all traffic not coming from the IP and MAC address pair owned by the VM. Therefore I need to explicitly tell neutron to also allow the IP address of the VIP on the VM ports. This can be done with the --allowed-address-pair parameter on the neutron port-create command:

1$ neutron port-create --name vip-port demo-net
2Created a new port:
3+-----------------------+-------------------------------------------------------------------------------------+
4| Field                 | Value                                                                               |
5+-----------------------+-------------------------------------------------------------------------------------+
6| admin_state_up        | True                                                                                |
7| allowed_address_pairs |                                                                                     |
8| binding:host_id       |                                                                                     |
9| binding:profile       | {}                                                                                  |
10| binding:vif_details   | {}                                                                                  |
11| binding:vif_type      | unbound                                                                             |
12| binding:vnic_type     | normal                                                                              |
13| device_id             |                                                                                     |
14| device_owner          |                                                                                     |
15| fixed_ips             | {"subnet_id": "9bc8ad9b-fa50-41c3-8381-7b86e8fd554b", "ip_address": "192.168.0.14"} |
16| id                    | ae020587-e870-4e38-b72a-6c8980bb92f6                                                |
17| mac_address           | fa:16:3e:03:84:81                                                                   |
18| name                  | vip-port                                                                            |
19| network_id            | f2592a01-e31d-4bdc-8aa7-ca66f938eb83                                                |
20| security_groups       | da933f24-4b8f-4478-909a-ca19aece379d                                                |
21| status                | DOWN                                                                                |
22| tenant_id             | 619a60d46b1349f7998116abec50b088                                                    |
23+-----------------------+-------------------------------------------------------------------------------------+
24$ neutron port-create --name vm1-port --allowed-address-pair ip_address=192.168.0.14 demo-net
25Created a new port:
26+-----------------------+-------------------------------------------------------------------------------------+
27| Field                 | Value                                                                               |
28+-----------------------+-------------------------------------------------------------------------------------+
29| admin_state_up        | True                                                                                |
30| allowed_address_pairs | {"ip_address": "192.168.0.14", "mac_address": "fa:16:3e:69:b2:d2"}                  |
31| binding:host_id       |                                                                                     |
32| binding:profile       | {}                                                                                  |
33| binding:vif_details   | {}                                                                                  |
34| binding:vif_type      | unbound                                                                             |
35| binding:vnic_type     | normal                                                                              |
36| device_id             |                                                                                     |
37| device_owner          |                                                                                     |
38| fixed_ips             | {"subnet_id": "9bc8ad9b-fa50-41c3-8381-7b86e8fd554b", "ip_address": "192.168.0.15"} |
39| id                    | 9ef8a695-0409-43db-9878-bf6b555dcfee                                                |
40| mac_address           | fa:16:3e:69:b2:d2                                                                   |
41| name                  | vm1-port                                                                            |
42| network_id            | f2592a01-e31d-4bdc-8aa7-ca66f938eb83                                                |
43| security_groups       | da933f24-4b8f-4478-909a-ca19aece379d                                                |
44| status                | DOWN                                                                                |
45| tenant_id             | 619a60d46b1349f7998116abec50b088                                                    |
46+-----------------------+-------------------------------------------------------------------------------------+
47$ neutron port-create --name vm2-port --allowed-address-pair ip_address=192.168.0.14 demo-net
48Created a new port:
49+-----------------------+-------------------------------------------------------------------------------------+
50| Field                 | Value                                                                               |
51+-----------------------+-------------------------------------------------------------------------------------+
52| admin_state_up        | True                                                                                |
53| allowed_address_pairs | {"ip_address": "192.168.0.14", "mac_address": "fa:16:3e:9e:8c:66"}                  |
54| binding:host_id       |                                                                                     |
55| binding:profile       | {}                                                                                  |
56| binding:vif_details   | {}                                                                                  |
57| binding:vif_type      | unbound                                                                             |
58| binding:vnic_type     | normal                                                                              |
59| device_id             |                                                                                     |
60| device_owner          |                                                                                     |
61| fixed_ips             | {"subnet_id": "9bc8ad9b-fa50-41c3-8381-7b86e8fd554b", "ip_address": "192.168.0.16"} |
62| id                    | 928c8761-b98b-4c2f-be41-e4ab5ee82eab                                                |
63| mac_address           | fa:16:3e:9e:8c:66                                                                   |
64| name                  | vm2-port                                                                            |
65| network_id            | f2592a01-e31d-4bdc-8aa7-ca66f938eb83                                                |
66| security_groups       | da933f24-4b8f-4478-909a-ca19aece379d                                                |
67| status                | DOWN                                                                                |
68| tenant_id             | 619a60d46b1349f7998116abec50b088                                                    |
69+-----------------------+-------------------------------------------------------------------------------------+
70

To allow VRRP traffic between the VMs I will now create a security group and assign it to the VM ports we created. Just for convenience reasons I will also add rules for allowing incoming SSH (for login) and ICMP (for ping) traffic.

1$ neutron security-group-rule-create --protocol 112 vrrp
2Created a new security_group_rule:
3+-------------------+--------------------------------------+
4| Field             | Value                                |
5+-------------------+--------------------------------------+
6| direction         | ingress                              |
7| ethertype         | IPv4                                 |
8| id                | a6602f3b-9c5b-427d-a2b1-fa589e4cabd5 |
9| port_range_max    |                                      |
10| port_range_min    |                                      |
11| protocol          | 112                                  |
12| remote_group_id   |                                      |
13| remote_ip_prefix  |                                      |
14| security_group_id | 171563a6-25ce-43dd-8b4f-4fbc5109330e |
15| tenant_id         | 619a60d46b1349f7998116abec50b088     |
16+-------------------+--------------------------------------+
17$ neutron security-group-rule-create --protocol tcp --port-range-min 22 --port-range-max 22 vrrp
18Created a new security_group_rule:
19+-------------------+--------------------------------------+
20| Field             | Value                                |
21+-------------------+--------------------------------------+
22| direction         | ingress                              |
23| ethertype         | IPv4                                 |
24| id                | 717af275-c4bf-41de-9494-2942fa0eae2a |
25| port_range_max    | 22                                   |
26| port_range_min    | 22                                   |
27| protocol          | tcp                                  |
28| remote_group_id   |                                      |
29| remote_ip_prefix  |                                      |
30| security_group_id | 171563a6-25ce-43dd-8b4f-4fbc5109330e |
31| tenant_id         | 619a60d46b1349f7998116abec50b088     |
32+-------------------+--------------------------------------+
33$ neutron security-group-rule-create --protocol icmp vrrp
34Created a new security_group_rule:
35+-------------------+--------------------------------------+
36| Field             | Value                                |
37+-------------------+--------------------------------------+
38| direction         | ingress                              |
39| ethertype         | IPv4                                 |
40| id                | 1bd25b24-33a1-4f13-be8a-361a9c3bff21 |
41| port_range_max    |                                      |
42| port_range_min    |                                      |
43| protocol          | icmp                                 |
44| remote_group_id   |                                      |
45| remote_ip_prefix  |                                      |
46| security_group_id | 171563a6-25ce-43dd-8b4f-4fbc5109330e |
47| tenant_id         | 619a60d46b1349f7998116abec50b088     |
48+-------------------+--------------------------------------+
49$ neutron port-update --security-group vrrp vm1-port
50Updated port: vm1-port
51$ neutron port-update --security-group vrrp vm2-port
52Updated port: vm2-port
53

As I want to access the VMs and the VIP from outside the cloud I create three floating IPs from my external network called floating and assign them to the three ports.

1$ neutron floatingip-create floating
2Created a new floatingip:
3+---------------------+--------------------------------------+
4| Field               | Value                                |
5+---------------------+--------------------------------------+
6| fixed_ip_address    |                                      |
7| floating_ip_address | 10.98.208.217                        |
8| floating_network_id | 74621f62-b07b-4543-83e5-f6d8504b1aae |
9| id                  | 55009db5-3720-4880-943e-266690356748 |
10| port_id             |                                      |
11| router_id           |                                      |
12| status              | DOWN                                 |
13| tenant_id           | 619a60d46b1349f7998116abec50b088     |
14+---------------------+--------------------------------------+
15$ neutron floatingip-create floating
16Created a new floatingip:
17+---------------------+--------------------------------------+
18| Field               | Value                                |
19+---------------------+--------------------------------------+
20| fixed_ip_address    |                                      |
21| floating_ip_address | 10.98.208.218                        |
22| floating_network_id | 74621f62-b07b-4543-83e5-f6d8504b1aae |
23| id                  | 4377aa79-d2f4-4977-ad63-4d9f8a7f2a42 |
24| port_id             |                                      |
25| router_id           |                                      |
26| status              | DOWN                                 |
27| tenant_id           | 619a60d46b1349f7998116abec50b088     |
28+---------------------+--------------------------------------+
29$ neutron floatingip-create floating
30Created a new floatingip:
31+---------------------+--------------------------------------+
32| Field               | Value                                |
33+---------------------+--------------------------------------+
34| fixed_ip_address    |                                      |
35| floating_ip_address | 10.98.208.219                        |
36| floating_network_id | 74621f62-b07b-4543-83e5-f6d8504b1aae |
37| id                  | 73dea280-4edf-49cc-8432-c3f56d87d531 |
38| port_id             |                                      |
39| router_id           |                                      |
40| status              | DOWN                                 |
41| tenant_id           | 619a60d46b1349f7998116abec50b088     |
42+---------------------+--------------------------------------+
43$ neutron floatingip-associate 55009db5-3720-4880-943e-266690356748 ae020587-e870-4e38-b72a-6c8980bb92f6
44Associated floating IP 55009db5-3720-4880-943e-266690356748
45$ neutron floatingip-associate 4377aa79-d2f4-4977-ad63-4d9f8a7f2a42 9ef8a695-0409-43db-9878-bf6b555dcfee
46Associated floating IP 4377aa79-d2f4-4977-ad63-4d9f8a7f2a42
47$ neutron floatingip-associate 73dea280-4edf-49cc-8432-c3f56d87d531 928c8761-b98b-4c2f-be41-e4ab5ee82eab
48Associated floating IP 73dea280-4edf-49cc-8432-c3f56d87d531
49

That is all that is needed within neutron.

Create the VMs

Now I am able to create two Ubuntu VMs and conenct them to the VM ports:

1$ nova boot --flavor m1.small --image ec83428f-39e3-4675-a3c6-eff37238dbbe --key-name my_keypair --nic port-id=9ef8a695-0409-43db-9878-bf6b555dcfee vm1
2$ nova boot --flavor m1.small --image ec83428f-39e3-4675-a3c6-eff37238dbbe --key-name my_keypair --nic port-id=928c8761-b98b-4c2f-be41-e4ab5ee82eab vm2
3

I wait until the VMs are running and log in to install keepalived:

1$ ssh ubuntu@10.98.208.218
2ubuntu@vm1:~$ sudo apt install -y keepalived
3

To configure keepalived I lookup the name of the network interface on the VM:

1ubuntu@vm1:~$ ip a
21: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
3    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
4    inet 127.0.0.1/8 scope host lo
5       valid_lft forever preferred_lft forever
6    inet6 ::1/128 scope host
7       valid_lft forever preferred_lft forever
82: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
9    link/ether fa:16:3e:69:b2:d2 brd ff:ff:ff:ff:ff:ff
10    inet 192.168.0.15/24 brd 192.168.0.255 scope global ens3
11       valid_lft forever preferred_lft forever
12    inet6 fe80::f816:3eff:fe69:b2d2/64 scope link
13       valid_lft forever preferred_lft forever
14

With the interface name and the IP of the vip-port I can now continue to configure keepalived by editing /etc/keepalived/keepalived.conf:

1vrrp_instance VIP_1 {
2    state MASTER
3    interface ens3
4    virtual_router_id 51
5    priority 150
6    advert_int 1
7    authentication {
8        auth_type PASS
9        auth_pass supersecretpassword
10    }
11    virtual_ipaddress {
12        192.168.0.14
13    }
14}
15

Now I restart the keepalived service:

1ubuntu@vm1:~$ sudo systemctl restart keepalived
2

And repeat those steps on the second VM.

Checking the output of ip a on the VMs I see that one of my VMs assigned the VIP IP address 192.168.0.14 to its ens3 interface.

Testing failover

I open a terminal and ping the VIP:

1$ ping 10.98.208.217
2PING 10.98.208.217 (10.98.208.217): 56 data bytes
364 bytes from 10.98.208.217: icmp_seq=0 ttl=59 time=23.294 ms
464 bytes from 10.98.208.217: icmp_seq=1 ttl=59 time=20.879 ms
564 bytes from 10.98.208.217: icmp_seq=2 ttl=59 time=20.152 ms
6...
7

In a second terminal I log in to the VM that currently has the VIP listed under the ens3 and stop the keepalived:

1ubuntu@vm1:~$ sudo service keepalived stop
2

If everything worked I should now see that the ping in my first terminal is still working and checking ip a on both VMs shows that the VIP failed over to the other VM.

share post

Likes

0

//

More articles in this subject area\n

Discover exciting further topics and let the codecentric world inspire you.

//

Gemeinsam bessere Projekte umsetzen

Wir helfen Deinem Unternehmen

Du stehst vor einer großen IT-Herausforderung? Wir sorgen für eine maßgeschneiderte Unterstützung. Informiere dich jetzt.

Hilf uns, noch besser zu werden.

Wir sind immer auf der Suche nach neuen Talenten. Auch für dich ist die passende Stelle dabei.