Skip to content

High Availability

You may want to deploy Drovio Server for High Availability to prevent any downtime that would lower the user experience. Drovio Server handles that using Hazelcast clustering.

Deploy multiple instances

Deploy Drovio Server to at least two different machines. Make sure each instance is configured the same way.

You will have to host the database on a separate machine, so it is reachable by both instances. Drovio Server exclusively supports PostgreSQL 14+.

Pool sizes

Carefully set the two pool size settings (pool_options.main.max_size and pool_options.licensing.max_size) on each instance: the overall sum should be lower than the maximum number of connections your database can handle. For example, if max_connections = 60 and you have 2 instances, set both max_size values to 15 on each instance.

You can retrieve the max connections value using the PostgreSQL tool psql, with show max_connections;.

Enable clustering

Open the cluster properties file:

/etc/drovio-server/cluster.properties

C:\ProgramData\Drovio\Drovio Server\cluster.properties

Then:

  • Set enabled to true.
  • Make sure config_path points to the cluster.xml file (it should be in the same folder as cluster.properties).
  • Set host to the machine's private IP (the address used to communicate with the other cluster members), or 0.0.0.0 to listen on all available interfaces.
  • You can leave port at 0: Vert.x picks an available port automatically and Hazelcast advertises it to the other members during discovery. Set a fixed port only if your firewall rules require it.

Configure clustering

Open the cluster XML file:

/etc/drovio-server/cluster.xml

C:\ProgramData\Drovio\Drovio Server\cluster.xml

You need to change what is inside the <network></network> tag (the other tags should not be modified). You then have multiple options to make your instances discoverable by each other.

You must have multicast enabled on your network for it to work.

<network>
  <join>
    <multicast enabled="true">
      <multicast-group>224.2.2.3</multicast-group>
      <multicast-port>54327</multicast-port>
    </multicast>
  </join>
</network>

Set the static IPs of the other members (make sure multicast is disabled).

<network>
  <join>
    <multicast enabled="false"/>
    <tcp-ip enabled="true">
      <member>10.12.38.2</member>
      <member>10.12.38.3</member>
    </tcp-ip>
  </join>
</network>

<network>
  <join>
    <multicast enabled="false"/>
    <aws enabled="true">
      <!-- optional, if access-key and secret-key are not set, IAM role is used (recommended) -->
      <access-key>my-access-key</access-key>
      <secret-key>my-secret-key</secret-key>
      <!-- optional, if not set, default IAM role is used (recommended) -->
      <iam-role>role</iam-role>
      <!-- optional, default is us-east-1 -->
      <region>us-west-1</region>
      <!-- optional, default is ec2.amazonaws.com. If set, region shouldn't be set -->
      <host-header>ec2.amazonaws.com</host-header>
      <!-- optional, only instances in this group will be discovered -->
      <security-group-name>hazelcast-sg</security-group-name>
      <!-- optional, only instances with this tag and value will be discovered -->
      <tag-key>type</tag-key>
      <tag-value>hz-nodes</tag-value>
    </aws>
  </join>
</network>

The simplest way to get started with AWS EC2 discovery is to:

  • Create an IAM Role with the ec2:DescribeInstances permission.
  • Assign it to each instance.
  • Just use region, tag-key and tag-value.

More configuration options: hazelcast-aws README.

If the machines have multiple network interfaces (e.g. when a VPN is used), you can tell Hazelcast which interface to use with the <interface></interface> tag:

<network>
  <join>
    […]
  </join>
  <interfaces enabled="true">
    <!-- the IP address associated to your network interface -->
    <interface>192.168.1.20</interface>
  </interfaces>
</network>

More information: Hazelcast: discovery mechanisms.

Start the cluster

Start each instance, one by one. Watch the logs and look for an entry like the one below before starting another instance (generally ~10 seconds between each start):

Members [2] {
  Member [192.168.1.105]:5701 - 899898be-b8aa-49aa-8d28-40917ccba56c
  Member [192.168.1.105]:5702 - d6b81800-2c78-4055-8a5f-7f5b65d49f30 this
}

Update the cluster

When you need to update Drovio Server, proceed one by one to prevent downtime and ensure shared data between the instances is not lost. Stop the instance manually, look at the other instance's logs and wait until you see an entry like this before proceeding with the update:

Members [1] {
  Member [192.168.1.105]:5702 - d6b81800-2c78-4055-8a5f-7f5b65d49f30 this
}

Failover (Linux only)

Your cluster is now up and running, but the instances have different IPs. You will want to configure a basic IP failover so one IP can point to either instance. On Linux, keepalived comes in handy for that.

The principle is rather simple:

  • For 2 instances, you have 3 IP addresses, one of them shared by both instances.
  • You define which instance is the master and which one is the backup.
  • The shared IP (e.g. 10.0.12.16) points to the master instance (e.g. 10.0.12.17).
  • When the master instance is down, the shared IP now points to the backup instance (e.g. 10.0.12.18).

You need to deploy keepalived on all your Drovio instances for that to work.

Failover is the quickest solution when you want to make sure the service is always available. When clustering is enabled, Drovio Server internally load-balances a lot of operations. However, HTTP queries always land on the same instance, and the WebSocket connections Drovio Server maintains with active clients also operate on the same instance. You may thus want to load-balance the whole installation as the load and users stack up.

Load balancing (Linux only)

Load balancing can be achieved with HAProxy or Nginx (Plus). Drovio Server is an HTTP + WebSocket application. You need at least one additional machine on which HAProxy or Nginx is deployed.

Warning

In this scenario, when the load-balancing machine is down, the whole service is down. You might want to deploy at least 2 additional machines instead and combine them with a failover technique such as the one described above (e.g. HAProxy + keepalived).

AWS Elastic Load Balancing

If the Drovio Server instances are deployed to Amazon AWS, you can set up a quick load balancer using AWS ELB. Failover, load balancing, WebSockets… everything is handled and just requires a few clicks. Refer to the AWS documentation, Application Load Balancer.

Draining connections

When you need to stop an instance (for maintenance), temporarily remove it from the target group and wait until the connection is drained before actually stopping the instance. If you don't, ELB will send HTTP 502 replies for about 1 minute because the keep-alive connection with the stopped instance is broken.

AWS Route 53 multivalue answer

If your domain name is managed by AWS Route 53, another solution is the multivalue answer routing policy. Associated with health checks, the DNS answers queries with either instance IP that isn't down (up to 8) in a round-robin fashion. See multivalue vs simple policies.

Note

The connection between a Drovio client and the cluster takes longer to recover than with the ELB method (ELB is ~5 seconds, while Route 53 multivalue requires 30 seconds to 1 minute). However, during the instance switch, current screen sharing sessions won't be interrupted, thanks to Drovio's P2P architecture.