Recently, while I was working on one of our new Kentico MVC sites that is hosted on a Windows Server with IIS, I ran into an issue with the Kentico Web Farm. The automatically updated search index files were not matching between the Kentico CMS app, MVC app or the API app we had set up. The Kentico Web Farm needs to synchronize changes to the search index files by copying them to each server in the Web Farm when changes are made. As it turns out, there is a good explanation for this issue. In this blog post I’ll walk you through the Kentico settings and web.config settings you can tweak to fix the problem.
On this project we had set up the MVC app to load a search page when an initial request is made. This initial request will show the first page of results on page load. Once a user is on the search page, all other requests for data, except the initial page load request, uses the API app. On initial request, by searching through our master template search bar, we were seeing a certain number of results returned, let’s say 45 for the search query “test”. Once on the search page, there was another search bar that allows users to search for something else. If we used the API search bar to search for “test” again we were seeing 46 results.
Troubleshooting Kentico Web Farms
We use the MVC app to load pages upon request and the API app for requesting new data on those pages. It was a lengthy process of troubleshooting and testing to determine what was causing the issue and what an appropriate/maintainable/long-lasting solution would be.
My first step was to perform a smart search index rebuild and verify that the index files were updated. I looked in this location for the CMS, MVC and API apps:
The date modified column within file explorer will show you what you need to know. If there aren’t any index files that were just modified for any of the 3 apps then that app’s index files are not being updated. This wasn’t an issue for me though as the index files for all three apps were updated as expected. There were 45 search results returned for “test” by the MVC and API project. Problem solved! I just needed to rebuild the search index.
However, I knew this wasn’t the solution. How did the indexes get out of sync in the first place? When content on the site is updated the search index automatically updates the index files for all 3 apps. As I suspected, the indexes were out of sync again a week later.
At this point, I started to suspect that the web-farm setup was causing the issue. The Kentico web-farm keeps the CMS, MVC, and API apps in sync. These were the Kentico Web Farm settings we were using (Settings -> Versioning & Synchronization -> Web farm):
You will need a Web Farm license that allows for 3 Web Farms, otherwise you’ll run into all sorts of problems that are hard to diagnose if you aren’t familiar with how they work. One weird issue that is caused by this scenario is clearing Kentico’s system cache. If you don’t have enough licenses you will only be clearing the cache for the Web Farm servers that are healthy. This gets very confusing when you are testing. You may see issues with any of the other “Synchronized” items in the above image as well. We had 3 Web Farm licenses, so this wasn’t the issue, or was it? Would there ever be a time when we needed more than 3 licenses?
Well, as it turns out, when developers run the site locally, via Visual Studio and IIS Express, they are connecting to the same database. Kentico adds the computer name to the list of web-farm servers if you are running automatic web-farms. When this happens Kentico shuts down the Web Farm because there are too many Web Farm servers running as the site is only licensed for 3 servers. Now there are 4 servers. Kentico will no longer synchronize changes between the servers and you will be left confused.
If anyone changes content throughout the CMS site none of the changes will be synchronized over to the API or MVC servers. You might think this is a non-issue because, after all, how often do we spin-up our localhost environments for extended periods of time? As it turns out, we do this quite often during long debug sessions. When in this phase, our QA team sometimes performs heavy testing, which requires them to change a lot of content throughout the site. Some changes, like page content changes, will update the search index but the updates will not show up on the front-end of the site because the Web Farm isn’t synchronizing the changes across the servers. Start writing those bug reports!
There are a few ways to handle this problem. First, you can delete the localhost app. Once you delete the extra servers the web farm will synchronize changes again, including the changes that were made while the Web Farm was out of commission. The tasks will wait in a queue until the Web Farm starts again if the servers still exist in the Web Farm app listing. This seems tedious and hard to remember though.
Second, you can increase the number of Web Farms your Kentico license can handle. Hmm, is this necessary though? Do our localhost environments need to be in sync like the CMS, MVC and API apps?
A better solution would be to change the Web Farm mode from Automatic to Manual. When in manual mode you must specify the servers you want the web farm to synchronize. By default, in automatic mode, Kentico will add servers to the Web Farm even if your site isn’t licensed for them. According to the Kentico documentation: “If a server doesn't have a server name defined in the web.config, it uses its machine name.” That’s fine but it may be hard to tell which server is the MVC app, the API app, and the CMS app when there are issues.
An even better solution is to set the Web Farm server names in each project’s web.config file using the following app setting key:
<add key="CMSWebFarmServerName" value="localhostmvcsite" />
After I defined the server names, switched to manual Web Farm mode, and added each server to the servers list in Kentico I saw this:
This solved our localhost/development server issues on our end because no new localhost servers were being added to the Web Farm in manual mode. However, there is another issue that I haven’t fully discussed yet.
We weren’t the only people complaining about mismatched search results. Our client was experiencing the same issues, only they weren’t doing any localhost development against their copy of the same site. Their site was also being hosted on a Windows Server with IIS. I rebuilt the search indexes and the search results were matching again but of course, they didn’t match a week later. What the heck is going on now?
This is what I expected to see after a long weekend:
This is what I actually saw:
What happened to the apisite and mvcsite servers? From the Kentico 10 troubleshooting Web Farms guide:
Details for automatic
- After 24 hours in the Not responding state, the system removes the server from the system along with its web farm tasks.
This is a problem because a server will only be re-added to the Web Farm once a request is made. In the image above, I made a request to the cmssite server because I logged in to check on the Web Farm. Therefore, the cmssite server was added back into the Web Farm and is displayed in the healthy state.
Content administrators may also be logging in after a long weekend to create and update content on the cmssite server while the other two servers don’t exist in the Web Farm. When content editors make changes to page content and the search index is updated, the servers that were removed from the Web Farm don’t get the same Web Farm tasks created for them. As a result, the search index files are not updated for the removed servers. This results in some obvious confusion because the search indexes may never match again even if the servers are added back into the Web Farm. Therefore, users may see different results for different requests to different Web Farm servers. This issue is different, but it has the same solution. Switch your web farm to manual mode because the server stays in the system until you manually remove it.
One more thing to note when troubleshooting a Web Farm. If a Web Farm server is not responding you will need to make a web request to that server to transition to healthy. If it isn’t healthy, and you make updates to the content of a page, synchronization tasks will get created for the “Not Responding” servers but those updates will not be synced until the servers are in the “Transitioning” or “Healthy” state.
I hope this helps some developers out there that are having issues with Web Farms. If you have any questions or additional advice about web farms please let me know in the comments.