Maximize IIS application pool availability

Your website is screaming fast, with lots of effort devoted to optimizations and caching.

EXCEPT when it first comes into service cold … or when it recycles or restarts! Then, you are in for a world of hurt as your site struggles to cope with incoming traffic without the benefit of full caches. Or perhaps, your application preloads the caches on start but takes minutes to initialize as it loads all your data, resulting in an outage-like production delay.

Either way, unplanned restarts and recycles become a serious problem.

This challenge is well-known.

But, solving this problem is an entirely different kind of flying, altogether … and few know how to fully implement the solution in their IIS website.

In this guide, I am going to show you exactly how you can configure your IIS application pools to achieve maximum availability, with:

• 100% warm application performance from the first request … even after a recycle or cold start.
• Zero restart/recycling downtime in production.

At this point, you are probably saying: surely you can’t be serious?

Yes, I am serious. And don’t call me Shirly.

The goal

First, let’s define exactly what we want to accomplish:

1. Make sure our application is 100% warm for best performance before it receives traffic.
2. Be able to recycle our IIS application pool with zero downtime/startup delay..

These goals initially appear at odds with each other. Application warmup increases cold start performance, but causes a startup delay … which causes downtime when recycling in production. Recycling faster in production means no time for warmup. No warmup means fast recycle, but slow performance due to empty caches.

You can read more about the costs of recycling, and how to minimize it, in our comprehensive How to restart, reset, and recycle IIS guide.

Suffice it to say, we’ll need to break this cycle to achieve both goals.

Secret sauce: overlapped recycle + application initialization

To understand what we are after, let’s dig into the overlapped recycle. The overlapped recycle is the process that IIS uses to gracefully swap a new IIS worker process with the old worker process. Without losing any requests in the process.

During the overlapped recycle, a new worker process is started and new requests are sent to it. The old process then finishes out existing requests and shuts down.

This is great for zero-downtime recycling!

(Check out the IIS recycle guide for the advantages of recycling vs. IISRESET and other methods of restarting IIS).

That said, IIS considers the new worker process ready BEFORE the application has started. This means either:

1. Your application starts cold without any of its data built up (and has poor performance).
   
   OR

1. New requests experience a long delay (while your new application starts up).

If your application startup takes 30 seconds, that could effectively spell downtime during peak traffic.

If your application starts cold, without warming caches, it could get easily overwhelmed. Or worse, overwhelm the backend database or downstream services as hundreds or thousands of requests request backend data that is not in the cache.

So, ideally what we want is:

1. Overlapped recycle.
2. Old worker process continues to serve new traffic until the new process is FULLY ready.
3. New process fully initializes the app/warms caches BEFORE it’s ready.
4. BONUS: We want the app to always remain WARM when it’s in service.

The great news is, we can have ALL that! Thanks to a number of new features in IIS 8.0+.

To do this properly though, we’ll need to consider a number of aspects of your app pool recycling and application restarts, and get the details right.

Here is how:

1. Install the Application Initialization module.
2. Auto-start your application pool to have it start without waiting for a request.
3. Set up application initialization to warm-up your application when the application pool starts.
4. Prevent your application pool from shutting down unexpectedly.
5. Set up an optimal recycling strategy for your application pool.

We’ll also include a handy tool at the end that automates this entire process for a specific application.

Let’s go!

Install the Application initialization module

Application Initialization, you ask, what is it? It’s not a big building with patients (that’s a hospital).

It’s the module that helps IIS do application warmup when the worker process starts. Application Initialization is a built-in feature (IIS 8.0+) that we’ll need to get this set up.

You can do this from cmdline:

dism /online /enable-feature /featurename:IIS-ApplicationInit

To check that you have this installed, do:

appcmd list config /section:globalModules | findstr warmup.dll

You should see the warmup module registered:

Alternatively you can check/install the Application Initialization role from the Server Manager or ”Turn Windows Features On or Off” in Windows 10:

NOTE: We've seen cases where installing the feature still leaves the module missing. To fix: If the module is missing, uninstall and reinstall the AppInit feature.

Now that you’ve got this out of the way, let’s get into the good stuff.

Enable “AlwaysRunning” mode for your application pool

This makes sure that IIS always maintains a ready worker process for your application pool (even if there is no traffic yet).

To enable this, we’ll need to set the new startMode attribute to “AlwaysRunning”:

appcmd set apppool TestApp /startMode:AlwaysRunning

Which sets the applicationPool configuration:

<add name="TestApp" startMode="AlwaysRunning" autoStart="true" ... >

Details on https://docs.microsoft.com/en-us/iis/configuration/system.applicationhost/applicationpools/add/.

In the next section, we’ll make sure this process is actually warmed up and READY to receive traffic.

Enable application initialization to warm-up our application

This will make sure that when the IIS worker process starts for our application pool, it will also initialize our application and trigger our warmup code.

To do this, we’ll need to use the “preloadEnabled” attribute in the application definition.

appcmd set app "TestApp/" /preloadEnabled:true

Here it is in our application definition:

<site name="TestApp" id="4" serverAutoStart="true">
   <application path="/" applicationPool="TestApp" serviceAutoStartEnabled="false" serviceAutoStartProvider="" preloadEnabled="true" ...>

Now that we’ve got this, IIS will always start a worker process for this application pool. The application initialization module will then start each application marked with “preloadEnabled” to initialize them BEFORE the worker process is considered ready for service.

Now, we just need to make sure to have the application initialization and warmup code in your Application_Start handler in global.asax:

When you do this, and the worker process starts or recycles, it will immediately preload the application. Here is what this looks like if you were to break into your startup code in the debugger:

By default, there will also be a special request to the “/” root url, so you can put first-request warmup code in Application_BeginRequest instead. If you do this, you’ll likely want to do it the first time only, something like this:

private static bool s_inited = false;
private static object s_initLock = new object();

public void Application_BeginRequest(Object source, EventArgs e)
{
  // low-lock "is initialization done?" check
  if (!s_inited)
  {
    lock(s_initLock)
    {
      if (!s_inited)
      {
        // Do the first request initialization
        InitializeApplication();
        s_inited = true;
      }
    }
  }
}

We’ll look at how to configure the warmup for other parts of your application next.

Configure application initialization for optimal warm-up

By default, application initialization will make a fake “/” request to your application. This triggers your Application_Start event, and allows you to run your application initialization and warmup code.

However, by default, it will NOT:

1. Make any other requests to your application.
2. Run any ASP.NET modules as part of the fake “/” request.
3. Trigger compilation of your aspx pages or MVC views.
4. Annoyingly, it will also NOT re-run your application warmup after an application restart (e.g. if web.config changes). This is a big bummer because your app could restart during peak traffic and not receive the benefit of warmup.

If all your warmup is inside Application_Start, this is OK. However, you may also want to pre-compile parts of your application, and warm up individual pages or controllers/actions. To do this, you can also have application initialization make some requests to parts of your application that need additional warmup.

Let’s fix that:

appcmd set config "TestApp/" /section:applicationInitialization /doAppInitAfterRestart:true /skipManagedModules:false 

And then add any pages or urls that you want to be part of your warmup:

appcmd set config "TestApp/" /section:applicationInitialization /+[initializationPage='/Login',hostname='www.testapp.com'] 

The hostname is whatever you want to show up in the warmup request.

Here is our resulting configuration:

<applicationInitialization skipManagedModules="false" doAppInitAfterRestart="true">
    <add initializationPage="/Login" hostName="www.testapp.com" />
</applicationInitialization>

NOTE: The initializationPage url must be APP-relative, not website relative. So, if your app is /products, the url you would set for /products/login would be /login.

You can read more about the options here. The docs say to commit the changes to the applicationHost.config level, which is tied to the server, not to the application’s web.config. However, we think these settings belong with the application because they specify application-specific warmup urls.

With this configured, Application Initialization will now make all the requests you have configured, and run the full managed module stack for each request as expected.

In each of those requests, you’ll have a full (although fake) set of request information:

So, what kinds of stuff should you preload?

We recommend to configure preload requests to the following:

1. High trafficked actions in your app. Especially if these actions are output cached (you can prime your cache BEFORE the traffic hits for super-high performance)
2. Any page or action that has its own data/caches that it initializes once.
3. Any pages/actions that use compiled views. This is a big one actually, because compilation can be a significant overhead. Compilation can also cause CPU overload if multiple requests cause multiple folders with aspx pages or razor cshtml views to be batch-compiled at once. With warm-up, you can pre-compile all of them in sequence without overloading your server, and before your app goes into service.

Don’t go crazy of course, thinking you need to hit every page. If in doubt, just hit your most high traffic/known-heavy pages.

If you have some pages that experience long first-time compile or init overhead, add them too.

Something is probably better than nothing here.

Prevent your application pool from shutting down

You’ve gone through the trouble of making sure IIS initializes and warms up our application, for peak warm performance.

Great work, time to celebrate?

Not so fast. It turns out that even with the startMode of “AlwaysRunning”, IIS will still honor the idle timeout setting. This causes your worker process to shut down after a period of inactivity.

That means, for lower traffic or intermittent traffic applications, your entire warmup advantage could be lost. Your application could be doing expensive initialization every 20 minutes or so … As the worker process shuts down due to intermittent traffic and *immediately* starts back up.

Note that this action is NOT overlapped! Meaning that the old process is gone and the new one is doing potentially long application initialization, during which your application is effectively blocked or down.

Here it is in action:

That’s the exact thing we DIDN’T want!

To prevent this, one more application pool configuration change:

appcmd set apppool "TestApp" /processModel.idleTimeout:00:00:00

This results in this application pool configuration:

<add name="TestApp" autoStart="true" startMode="AlwaysRunning" ...>
  <processModel idleTimeout="00:00:00" idleTimeoutAction="Terminate" startupTimeLimit="00:01:30" logEventOnProcessModel="IdleTimeout" />

This keeps the worker process running, and warm, even through periods of inactivity.

Use idleTimeoutAction = Suspend to save memory if you have a lot of websites or application pools

There is another option available in IIS 8.5, which lets you keep the idle timeout. Instead of shutting down the worker process, you can have it get suspended (paged out from memory to disk).

The advantage of this option is that we can keep the worker process and the applications in a warm state, but without actively taking up memory. This can be helpful if you have a lot of websites/application pools, that are active intermittently, and you are trying to save memory. Then, you can do this instead:

appcmd set apppool "TestApp" /processModel.idleTimeout:00:10:00 /processModel.idleTimeoutAction:Suspend 

This results in this application pool configuration:

<add name="TestApp" autoStart="true" startMode="AlwaysRunning" ...>
  <processModel idleTimeout="00:10:00" idleTimeoutAction="Suspend" startupTimeLimit="00:01:30" logEventOnProcessModel="IdleTimeout" />

You can see this in action in the Application EventLog whenever your application goes idle. IIS logs Event ID 2310 indicating “A worker process with process id of '%1’ serving application pool 'TestApp' paged out due to inactivity. Application Pool timeout-action configuration was set to %2 minutes. A worker process will resume when needed.”

Check out Idle Worker Process Page-Out for a bit more detail.

Optimize application pool recycling

We are in the home stretch. I mean it.

We know that overlapped application pool recycling + AlwaysRunning + application warmup is a great combo strategy to keep your application always hot in production.

This final section deals with the recycling timing itself. By default, IIS configures your application pool to recycle after 29 hours of uptime.

But, that’s not important right now.

The point is that YOUR website MAY have a good time to recycle it’s application pool, because you know your own traffic pattern. Sure, some large international websites truly have peak traffic 24 hours/day. Still, most sites do have natural peaks and valleys corresponding to the usage patterns of their more regional audiences.

For these sites, the default time period based recycle can strike right in the middle of peak traffic.

So, we generally advise our LeanSentry customers to move away from a time period-based recycle to a scheduled recycle at off times. For example, to recycle at 2am every day:

appcmd set apppool TestApp /recycling.periodicRestart.time:00:00:00 /+recycling.periodicRestart.schedule.[value='02:00:00'] 

Finally, if your application suffers from memory leaks, you can also consider adding a private memory threshold to recycle if it approaches your desired memory limit. It’s not ideal - you do want to analyze your memory usage and optimize it if at all possible - but it works in a pinch.

ADVANCED: Apply application changes without a cold restart

Ok, you may be getting impatient here. I mean, how many hoops do you need to jump through to get a 100% warm, always-available application here, amirite?

Bear with me. After all, if you did all this work, just to have your application re-do it’s startup during peak traffic in production ... that would be no bueno.

The problem here is that despite keeping our application pool warm … there are still some things that can cause the ASP.NET application to restart.

These things include:

1. Recompilations (e.g. due to changes in cshtml or aspx pages)
2. Changes to application web.config
3. Changes to /BIN, App_Code, and other key files

Let’s assume that you are not changing the web.config and application files in production normally, unless you are either hot-patching it or deploying. And you are not going to be doing that unless you need to. Ideally the server is going to be out of service … or it’s an emergency. Regardless, you are probably going to want the app to restart then.

Unfortunately, the ASP.NET application does not support an overlapped restart, like IIS does for the application pool recycle. This means, while old requests get finished by the old application, the new requests get immediately sent to the new application which is still initializing. So, you don’t get the benefit of having a pre-warmed up come online only when ready.

So, what can we do?

The options here are simple.

If you are not changing the web app, it won’t restart, so you are good.

If you are changing the web application, and want to minimize restarts, you have some limited options:

appcmd set config "TestApp" /section:httpRuntime /fcnMode:Disabled 
appcmd set config "TestApp" /section:compilation /numRecompilesBeforeAppRestart:1000 

This prevents the application from restarting on any changes, other than root web.config.

It is believed that there is no way to turn off IIS restarting the ASP.NET application if the root web.config changes. This is actually not true!

There is a way, by changing the server-wide IIS registry key (requires you to recycle any relevant application pool to take effect).

reg add HKLM\System\CurrentControlSet\Services\W3SVC\Parameters /v ConfigPollMilliSeconds /t REG_DWORD /d 2147483647 

What this does is basically tell IIS to not monitor changes to web.config files or any other files, AT ALL. It also won’t see any changes to any files the web server caches, e.g. static files or files compressed by IIS static compression.

With that in mind, it is possible to have a crazy but maximally defensive strategy to handle changes to the web application without restarting.

And that is, to not detect ANY changes to configuration and content, and then to apply those changes manually by recycling the application pool instead!

While this gives up the very useful ability to make auto-detected changes to the application and files, it provides one crucial benefit:

The ability to trigger an overlapped application restart to apply any changes!

Of course, if you are using this strategy, you have to seriously consider the implications of IIS not detecting any changes to your files.

To those good with that, I just want to tell you both good luck. We are all counting on you.

Conclusion

This was probably harder than you expected, but look at what we’ve been able to accomplish:

1. Your application is always on, and always warmed up for maximum performance.
2. Your application stays warm despite restarts and recycles.
3. You have no “experienced” startup delay when recycling takes place.
4. You have minimum restart overhead by reducing unnecessary recycling.

Not to mention, I was able to include at least 5 jokes from my favorite movie “Airplane”. Or was it six? I can’t tell. Maybe you can take a guess? Not for another two hours?

ConfigureWarmup "MyWebSite/app" [/warmupUrls:url1,url2,url3…] [/testWarmup]

Final thought: if you are having to recycle your application pool due to hangs, high CPU usage, or memory leaks, use LeanSentry to diagnose and fix those issues. This will be the biggest thing you can do to improve your availability in the long run.