How to Cluster and Load Balance Geocortex Essentials

Overview:
This article will explain how to cluster two or more Geocortex Essentials servers together, in order to provide high availability Sites.

Solution:

Before you begin

Before you begin, you should review our knowledge base article for Essentials environment best practices, to ensure that you are properly planning your deployment.  

If you run into any issues with these instructions, or if you would like some assistance getting this set up, please contact your account manager to request assistance.  Configuration of clustering and load balancing is not covered by Support and may require an engagement with our Professional Services team.

Some prerequisites that you should have ready to go include:

1. A network file share for your Sites and temporary data folders
2. If necessary, a user account that has permission to access the file share (that you can use for your App Pools.  See below for details)
3. A load balancer to distribute connections between your servers

This article assumes that you are setting up three new Essentials servers. If you are migrating from a single server to a new multi node cluster, you will want to set up the first two nodes, and then add your third node when the other two are clustered. We recommend only installing the REST Manager component on one of the nodes, since REST Manager does not support load balancing.
 

Part 1 - Configure the Sites folder and the temp folder

1. Create two folders on the file share, one for the Sites, and one for the Essentials temp directory. The identity you run the EssentialsAppPool4 application pool under must be able to write to the temp folder and read from the Sites folder. The identity you run the EssentialsAdministrationAppPool4 application pool under must be able to read and write to the temp folder and the Sites folder.
2. On the first server, edit the web.config file for the REST application (located here by default: C:\Program Files (x86)\Latitude Geographics\Geocortex Essentials\Default\REST Elements\REST\Web.config)
3. Look for the key called "gcx.io.fileSystem.path" and edit the value to point to the UNC path for your file share.  For example, the default will look like this:

<add key="gcx.io.fileSystem.path" value="C:\Program Files (x86)\Latitude Geographics\Geocortex Essentials\Default\REST Elements\Sites" />

Edit it to look like this, but use your UNC path instead of the generic path shown:

<add key="gcx.io.fileSystem.path" value="\\myfileshare\Essentials\Sites" />

4. Directly under that line, you will need to add a new key for the path to your temp folder. The temp folder, like the Sites folder, needs to be available to all nodes in the cluster. The new key should look like this, but use your UNC path instead of the generic path shown:

<add key="gcx.io.tempFileManager.commonTempFolder" value="\\myfileshare\Essentials\Temp" />

5. Save the file.
6. Repeat steps 2-5 for all other servers.
7. On the node with Essentials Manager installed, you will need to repeat steps 3-5 in the REST Manager web.config file (located here by default: C:\Program Files (x86)\Latitude Geographics\Geocortex Essentials\Default\REST Elements\Manager\Web.config).
8. Save the REST Manager web.config file.
9. Copy the contents from your local Sites folder to the network share.
10. At this point, if you log into Essentials Manager, you should be able to create a new Site (If not, see the troubleshooting section at the end of this document). Try creating a Site, then check to make sure the Site appears in the file share.  
11. Check the REST endpoint on the other nodes in the cluster to make sure that you can see the Site from all the nodes.  If you can, proceed on to the next section.

Part 2 - Set the URL for the load balancer in the Post Installation Configuration

1. Run the Post Installation Configuration tool on the node where you've installed REST Manager.
2. Navigate to Configure Geocortex Essentials REST API.
3. Click the Hosts button.
4. Add the URL that points to your load balancer.
5. Click OK, then click Finish.

Part 3 - Configure the load balancer

Configuration of your load balancer is largely up to you, however we have had success in the past configuring the ARR (Application Request Routing) module for IIS. This can be installed on one of your Essentials servers, or on a separate server. You can find our guide for configuring ARR here.

Ideally, all of the servers hosting Essentials would always return exactly the same response. However, in the event of a failure, the load balancer would not necessarily know that the response it’s about to return is actually an error message. We recommend that either the load balancer be configured to detect invalid content, or that it periodically tests the participating Essentials servers to determine whether or not they are available.

A reliable test that the load balancer can perform is to request the Essentials Site with the “f=json” parameter appended. This is the same as what is requested by a Viewer.

When the load balancer detects an issue with a node, it should stop routing request to that node. Ideally, it should also alert an administrator of the failure.

Part 4 - Configuring Geocortex Identity Server (Optional)

If you are planning on securing your Sites with Geocortex Identity Server, you will need to load balance the Identity Servers on all your nodes so that changes you make to your user accounts are synchronized. If you are not planning on using Identity Server, you can skip this step. Our guide on load balancing Identity Server can be found here.

Part 5 - Upgrading

When it comes time to upgrade, use the following procedure to avoid downtime on your Sites:

1. Set the load balancer to point to a single node (Not the one with Essentials Manager on it).
2. Upgrade Essentials on the other two nodes.
3. Point the load balancer to the 2 upgraded nodes, excluding the non-upgraded node.
4. Load Essentials Manager to update the Sites and Viewers to the new version (if applicable).
5. Upgrade the remaining node to the new version of Essentials.

Part 6 - Troubleshooting

You may run into issues with one or more of these steps. If you do, here are some troubleshooting steps to help:

I get an error in Essentials Manager after setting my Sites folder in the web.config files

If you are seeing an error accessing the share, it may be a permissions issue. By default, the application pools will attempt to access the shares as the identity they've been assigned. You can change the identity to that of a domain user, or a user that has permission to the folder and the share:

1. Open IIS Manager.
2. Expand your server, then select Application Pools.
3. Right click on the EssentialsAppPool4 application pool, and select Advanced Settings.
4. Find Identity in the list of settings and click it (By default, it will be set to ApplicationPoolIdentity).
5. Click on the ellipsis ("...").
6. Click Custom account:, click the Set button.
7. Enter credentials of an existing, valid user with permissions to the folder and the share, click OK.

8. Repeat steps 3-7 for all nodes in the cluster.
9. Repeat steps 3-7 for the EssentialsAdministrationAppPool4 on the node where Essentials Manager is installed.
10. Load Essentials Manager, the error should be gone.

I occasionally get a serialization error when running reports or workflows

This can happen in cases where the workflow gets executed on one node, but in the process of passing the serialized workflow from the client to the server, the load balancer passes it to one of the other nodes in the cluster. To work around this, you should enable sticky sessions on the load balancer. This setting causes the load balancer to bind the session on the client side to a specific node in the cluster, ensuring that when you run a workflow, the interaction between the client and server only hits a single node in the cluster.

The exact method for enabling this varies depending on the load balancer you are using.

My cluster nodes are available, but my Sites are not

This can happen if the file share or load balancer are unavailable. You can test the share by opening a file explorer window on one of the Essentials servers and navigating to the share UNC path. If it fails to load, that indicates an issue with the share, or with the network connection between the Essentials server and the file share. Try loading the share from different points and see if you get a consistent failure from all points.

If you are having issues loading the Sites through the load balancer, try loading them directly from the servers. If the Sites load from the servers, but not through the load balancer, this indicates an issue with the load balancer.

I'm not sure if my nodes are all in the correct cluster

In some cases (particularly if you are upgrading from a standalone server to a cluster of servers), you may not be sure if your nodes are all participating in the same cluster. Additionally, after an outage or maintenance you may want to verify your cluster health. The simplest way to do this is to navigate to http://localhost:19200/_plugin/head/index.html on each of the nodes, and make sure that all three nodes are showing up, and that the cluster health is green (per the screenshot below).  
 

If a node or nodes are missing, check the following file to see which cluster they belong to: C:\Program Files\Latitude Geographics\Geocortex Core\Bin\Geocortex.Platform.Host.exe.config

Find the following elements, make sure that the cluster name and shared secret match the other nodes in the cluster:

<add key="SharedSecret" value="MH+XMvNFyrh3aRuARv/zYg=="/>
<add key="ClusterName" value="MyCluster"/>

If they don't, and this is a new node, you can copy and paste the elements from the other cluster node into this file and overwrite the incorrect values. If it's been running this way for a while, you should contact support so we can help you investigate the issue.