Expert guides

Maximum IIS application pool availability

The complete guide to converting your IIS site to 100% warm, always available application that can recycle with zero downtime.

Category Restarting and recycling IIS
Tags IIS, Recycle, Warmup
Improve IIS application pool availability

Your website is screaming fast, with lots of effort devoted to optimizations and caching.

EXCEPT when it first comes into service cold … or when it recycles or restarts! Then, you are in for a world of hurt as your site struggles to cope with incoming traffic without the benefit of full caches. Or perhaps, your application preloads the caches on start but takes minutes to initialize as it loads all your data, resulting in an outage-like production delay.

Either way, unplanned restarts and recycles become a serious problem.

Reset, restart and recycle IIS the right way

Of course, you are probably doing your best to minimize production recycling, but you know how that goes. Your application pool still occasionally recycles in production, and sometimes you have to recycle it when your website is experiencing hangs, memory leaks, or other issues.

For the comprehensive how-to on recycling IIS, be sure to check out our Reset, restart and recycle IIS guide.

This challenge is well-known.

But, solving this problem is an entirely different kind of flying, altogether … and few know how to fully implement the solution in their IIS website.

In this guide, I am going to show you exactly how you can configure your IIS application pools to achieve maximum availability, with:

  • 100% warm application performance from the first request … even after a recycle or cold start.
  • Zero restart/recycling downtime in production.

At this point, you are probably saying: surely you can’t be serious?

Yes, I am serious. And don’t call me Shirly.

The goal

First, let’s define exactly what we want to accomplish:

  1. Make sure our application is 100% warm for best performance before it receives traffic.
  2. Be able to recycle our IIS application pool with zero downtime/startup delay..

These goals initially appear at odds with each other. Application warmup increases cold start performance, but causes a startup delay … which causes downtime when recycling in production. Recycling faster in production means no time for warmup. No warmup means fast recycle, but slow performance due to empty caches.

Vicious cycle of cold start performance, and startup delay

You can read more about the costs of recycling, and how to minimize it, in our comprehensive How to restart, reset, and recycle IIS guide.

Suffice it to say, we’ll need to break this cycle to achieve both goals.

Secret sauce: overlapped recycle + application initialization

To understand what we are after, let’s dig into the overlapped recycle. The overlapped recycle is the process that IIS uses to gracefully swap a new IIS worker process with the old worker process. Without losing any requests in the process.

During the overlapped recycle, a new worker process is started and new requests are sent to it. The old process then finishes out existing requests and shuts down.

This is great for zero-downtime recycling!

(Check out the IIS recycle guide for the advantages of recycling vs. IISRESET and other methods of restarting IIS).

IIS application pool recycle has lowest downtime

That said, IIS considers the new worker process ready BEFORE the application has started. This means either:

  1. Your application starts cold without any of its data built up (and has poor performance).

    OR
  1. New requests experience a long delay (while your new application starts up).

If your application startup takes 30 seconds, that could effectively spell downtime during peak traffic.

If your application starts cold, without warming caches, it could get easily overwhelmed. Or worse, overwhelm the backend database or downstream services as hundreds or thousands of requests request backend data that is not in the cache.

So, ideally what we want is:

  1. Overlapped recycle.
  2. Old worker process continues to serve new traffic until the new process is FULLY ready.
  3. New process fully initializes the app/warms caches BEFORE it’s ready.
  4. BONUS: We want the app to always remain WARM when it’s in service.

The great news is, we can have ALL that! Thanks to a number of new features in IIS 8.0+.

To do this properly though, we’ll need to consider a number of aspects of your app pool recycling and application restarts, and get the details right.

Here is how:

  1. Install the Application Initialization module.
  2. Auto-start your application pool to have it start without waiting for a request.
  3. Set up application initialization to warm-up your application when the application pool starts.
  4. Prevent your application pool from shutting down unexpectedly.
  5. Set up an optimal recycling strategy for your application pool.

We’ll also include a handy tool at the end that automates this entire process for a specific application.

Let’s go!

But I saw somewhere that application initialization doesn’t work?

The internet is full of posts by people trying to set up application initialization. These posts usually end with “I followed all the instructions, but it still does not work”. This is sad, because it probably scared a lot of other people from succeeding with application warmup for their sites.

I think there are several reasons why this happened.

First, there was the confusion over which features to use. The IIS team shipped a module called Application Initialization in IIS 7.5, just to pull it back and include a different feature in IIS 8.0. The ASP.NET team also provided an app warmup feature called serviceAutoStart in ASP.NET 4.0. This feature sort of became obsolete when IIS 8.0 built in the app warmup support.

You get the idea.

Second, to get the intended benefits, you really need to get all the details right. And this is why I wrote this guide.

Install the Application initialization module

Application Initialization, you ask, what is it? It’s not a big building with patients (that’s a hospital).

It’s the module that helps IIS do application warmup when the worker process starts. Application Initialization is a built-in feature (IIS 8.0+) that we’ll need to get this set up.

You can do this from cmdline:

dism /online /enable-feature /featurename:IIS-ApplicationInit

To check that you have this installed, do:

appcmd list config /section:globalModules | findstr warmup.dll

You should see the warmup module registered:

Appcmd command to confirm warmup.dll module is installed

Alternatively you can check/install the Application Initialization role from the Server Manager or ”Turn Windows Features On or Off” in Windows 10:

Install Application Initialization role service

NOTE: We've seen cases where installing the feature still leaves the module missing. To fix: If the module is missing, uninstall and reinstall the AppInit feature.

Now that you’ve got this out of the way, let’s get into the good stuff.

Enable “AlwaysRunning” mode for your application pool

This makes sure that IIS always maintains a ready worker process for your application pool (even if there is no traffic yet).


To enable this, we’ll need to set the new startMode attribute to “AlwaysRunning”:

appcmd set apppool TestApp /startMode:AlwaysRunning

Which sets the applicationPool configuration:

<add name="TestApp" startMode="AlwaysRunning" autoStart="true" ... >

Details on https://docs.microsoft.com/en-us/iis/configuration/system.applicationhost/applicationpools/add/.

In the next section, we’ll make sure this process is actually warmed up and READY to receive traffic.

Enable application initialization to warm-up our application

This will make sure that when the IIS worker process starts for our application pool, it will also initialize our application and trigger our warmup code.

To do this, we’ll need to use the “preloadEnabled” attribute in the application definition.

appcmd set app "TestApp/" /preloadEnabled:true

Here it is in our application definition:

<site name="TestApp" id="4" serverAutoStart="true">
   <application path="/" applicationPool="TestApp" serviceAutoStartEnabled="false" serviceAutoStartProvider="" preloadEnabled="true" ...>

What about serviceAutoStartEnabled?

Note the serviceAutoStartEnabled attribute in the application configuration. I mention this because there is some confusion on how serviceAutoStartEnabled and serviceAutoStartProvider settings relate to the preloadEnabled pathway that we are using. The bottom line is, the preloadEnabled attribute is the new way to do application warmup for regular ASP.NET applications in IIS 8.0+.

We’ll be using that INSTEAD of the older serviceAutoStartEnabled feature. That feature may still make sense for non-http applications, and you can read about in Scott Gu’s really old post.

Now that we’ve got this, IIS will always start a worker process for this application pool. The application initialization module will then start each application marked with “preloadEnabled” to initialize them BEFORE the worker process is considered ready for service.

Now, we just need to make sure to have the application initialization and warmup code in your Application_Start handler in global.asax:

Application warmup code in Application_Start inside global.asax

When you do this, and the worker process starts or recycles, it will immediately preload the application. Here is what this looks like if you were to break into your startup code in the debugger:

IIS calls into Application_Start via a special InitializeApplication pathway

Why can’t I access the HttpContext inside Application_Start?

Note that IIS (really webengine4.dll, the ASP.NET Integrated Pipeline engine), is not coming into this code via a request code path. Instead, it’s using the special “InitializeApplication” path. This also means that your Application_Start code will not be able to access any HttpRequest or HttpResponse in the app init code. This is because there is really no “request” associated with this stage.

For reasons why, you can read my old blog post on this on mvolo.com. Incidentally, in it I presciently suggest application warmup as a potential future feature.

By default, there will also be a special request to the “/” root url, so you can put first-request warmup code in Application_BeginRequest instead. If you do this, you’ll likely want to do it the first time only, something like this:

private static bool s_inited = false;
private static object s_initLock = new object();

public void Application_BeginRequest(Object source, EventArgs e)
{
  // low-lock "is initialization done?" check
  if (!s_inited)
  {
    lock(s_initLock)
    {
      if (!s_inited)
      {
        // Do the first request initialization
        InitializeApplication();
        s_inited = true;
      }
    }
  }
}

We’ll look at how to configure the warmup for other parts of your application next.

Configure application initialization for optimal warm-up

By default, application initialization will make a fake “/” request to your application. This triggers your Application_Start event, and allows you to run your application initialization and warmup code.

However, by default, it will NOT:

  1. Make any other requests to your application.
  2. Run any ASP.NET modules as part of the fake “/” request.
  3. Trigger compilation of your aspx pages or MVC views.
  4. Annoyingly, it will also NOT re-run your application warmup after an application restart (e.g. if web.config changes). This is a big bummer because your app could restart during peak traffic and not receive the benefit of warmup.

If all your warmup is inside Application_Start, this is OK. However, you may also want to pre-compile parts of your application, and warm up individual pages or controllers/actions. To do this, you can also have application initialization make some requests to parts of your application that need additional warmup.

Let’s fix that:

appcmd set config "TestApp/" /section:applicationInitialization /doAppInitAfterRestart:true /skipManagedModules:false 

And then add any pages or urls that you want to be part of your warmup:

appcmd set config "TestApp/" /section:applicationInitialization /+[initializationPage='/Login',hostname='www.testapp.com'] 

The hostname is whatever you want to show up in the warmup request.

Here is our resulting configuration:

<applicationInitialization skipManagedModules="false" doAppInitAfterRestart="true">
    <add initializationPage="/Login" hostName="www.testapp.com" />
</applicationInitialization>

NOTE: The initializationPage url must be APP-relative, not website relative. So, if your app is /products, the url you would set for /products/login would be /login.

You can read more about the options here. The docs say to commit the changes to the applicationHost.config level, which is tied to the server, not to the application’s web.config. However, we think these settings belong with the application because they specify application-specific warmup urls.

With this configured, Application Initialization will now make all the requests you have configured, and run the full managed module stack for each request as expected.

Application Initialization triggers BeginRequest in your global.asax as it makes warmup requests

In each of those requests, you’ll have a full (although fake) set of request information:

Warmup requests provide full request context.

So, what kinds of stuff should you preload?

We recommend to configure preload requests to the following:

  1. High trafficked actions in your app. Especially if these actions are output cached (you can prime your cache BEFORE the traffic hits for super-high performance)
  2. Any page or action that has its own data/caches that it initializes once.
  3. Any pages/actions that use compiled views. This is a big one actually, because compilation can be a significant overhead. Compilation can also cause CPU overload if multiple requests cause multiple folders with aspx pages or razor cshtml views to be batch-compiled at once. With warm-up, you can pre-compile all of them in sequence without overloading your server, and before your app goes into service.

Don’t go crazy of course, thinking you need to hit every page. If in doubt, just hit your most high traffic/known-heavy pages.

If you have some pages that experience long first-time compile or init overhead, add them too.

Something is probably better than nothing here.

Prevent your application pool from shutting down

You’ve gone through the trouble of making sure IIS initializes and warms up our application, for peak warm performance.

Great work, time to celebrate?

Not so fast. It turns out that even with the startMode of “AlwaysRunning”, IIS will still honor the idle timeout setting. This causes your worker process to shut down after a period of inactivity.

That means, for lower traffic or intermittent traffic applications, your entire warmup advantage could be lost. Your application could be doing expensive initialization every 20 minutes or so … As the worker process shuts down due to intermittent traffic and *immediately* starts back up.

Note that this action is NOT overlapped! Meaning that the old process is gone and the new one is doing potentially long application initialization, during which your application is effectively blocked or down.

Here it is in action:

IIS shuts down application pool due to idleTimeout

That’s the exact thing we DIDN’T want!

To prevent this, one more application pool configuration change:

appcmd set apppool "TestApp" /processModel.idleTimeout:00:00:00

This results in this application pool configuration:

<add name="TestApp" autoStart="true" startMode="AlwaysRunning" ...>
  <processModel idleTimeout="00:00:00" idleTimeoutAction="Terminate" startupTimeLimit="00:01:30" logEventOnProcessModel="IdleTimeout" />

This keeps the worker process running, and warm, even through periods of inactivity.

Use idleTimeoutAction = Suspend to save memory if you have a lot of websites or application pools

There is another option available in IIS 8.5, which lets you keep the idle timeout. Instead of shutting down the worker process, you can have it get suspended (paged out from memory to disk).

The advantage of this option is that we can keep the worker process and the applications in a warm state, but without actively taking up memory. This can be helpful if you have a lot of websites/application pools, that are active intermittently, and you are trying to save memory. Then, you can do this instead:

appcmd set apppool "TestApp" /processModel.idleTimeout:00:10:00 /processModel.idleTimeoutAction:Suspend 

This results in this application pool configuration:

<add name="TestApp" autoStart="true" startMode="AlwaysRunning" ...>
  <processModel idleTimeout="00:10:00" idleTimeoutAction="Suspend" startupTimeLimit="00:01:30" logEventOnProcessModel="IdleTimeout" />

You can see this in action in the Application EventLog whenever your application goes idle. IIS logs Event ID 2310 indicating “A worker process with process id of '%1’ serving application pool 'TestApp' paged out due to inactivity. Application Pool timeout-action configuration was set to %2 minutes. A worker process will resume when needed.”

IIS Event 2310: worker process paged out

Check out Idle Worker Process Page-Out for a bit more detail.

Optimize application pool recycling

We are in the home stretch. I mean it.

We know that overlapped application pool recycling + AlwaysRunning + application warmup is a great combo strategy to keep your application always hot in production.

This final section deals with the recycling timing itself. By default, IIS configures your application pool to recycle after 29 hours of uptime.

But, that’s not important right now.

Why do IIS application pools recycle after 29 hours?

This was before my time on the IIS team. But, I overheard Wade Hilmo - a lead developer on IIS7 - tell the story at TechEd 2007. They came up with that number during IIS 6.0 development, as the first prime after 24. The idea was to help the application pool recycle on a kind-of daily schedule but at different times of the day.

Wade, if you are reading this, I still think that this makes total sense. Many Classic ASP (and let’s face it, modern ASP.NET) applications needed to periodically purge due to memory leaks and corrupted states. We could not reasonably anticipate the ideal schedule for each customer, so we just made sure it happens at regular intervals but different times of day.

The point is that YOUR website MAY have a good time to recycle it’s application pool, because you know your own traffic pattern. Sure, some large international websites truly have peak traffic 24 hours/day. Still, most sites do have natural peaks and valleys corresponding to the usage patterns of their more regional audiences.

For these sites, the default time period based recycle can strike right in the middle of peak traffic.

So, we generally advise our LeanSentry customers to move away from a time period-based recycle to a scheduled recycle at off times. For example, to recycle at 2am every day:

appcmd set apppool TestApp /recycling.periodicRestart.time:00:00:00 /+recycling.periodicRestart.schedule.[value='02:00:00'] 

Finally, if your application suffers from memory leaks, you can also consider adding a private memory threshold to recycle if it approaches your desired memory limit. It’s not ideal - you do want to analyze your memory usage and optimize it if at all possible - but it works in a pinch.

ADVANCED: Apply application changes without a cold restart

Ok, you may be getting impatient here. I mean, how many hoops do you need to jump through to get a 100% warm, always-available application here, amirite?

Bear with me. After all, if you did all this work, just to have your application re-do it’s startup during peak traffic in production ... that would be no bueno.

WARNING

This section shows you how to configure your webserver for zero cold application restarts in production, at the cost of having to manually apply configuration changes. You instead do that with an overlapped application pool recycle. The web server will not detect any of the config or file changes it normally does.

The problem here is that despite keeping our application pool warm … there are still some things that can cause the ASP.NET application to restart.

These things include:

  1. Recompilations (e.g. due to changes in cshtml or aspx pages)
  2. Changes to application web.config
  3. Changes to /BIN, App_Code, and other key files

Let’s assume that you are not changing the web.config and application files in production normally, unless you are either hot-patching it or deploying. And you are not going to be doing that unless you need to. Ideally the server is going to be out of service … or it’s an emergency. Regardless, you are probably going to want the app to restart then.

Unfortunately, the ASP.NET application does not support an overlapped restart, like IIS does for the application pool recycle. This means, while old requests get finished by the old application, the new requests get immediately sent to the new application which is still initializing. So, you don’t get the benefit of having a pre-warmed up come online only when ready.

So, what can we do?

The options here are simple.

If you are not changing the web app, it won’t restart, so you are good.

If you are changing the web application, and want to minimize restarts, you have some limited options:

appcmd set config "TestApp" /section:httpRuntime /fcnMode:Disabled 
appcmd set config "TestApp" /section:compilation /numRecompilesBeforeAppRestart:1000 

This prevents the application from restarting on any changes, other than root web.config.

It is believed that there is no way to turn off IIS restarting the ASP.NET application if the root web.config changes. This is actually not true!

There is a way, by changing the server-wide IIS registry key (requires you to recycle any relevant application pool to take effect).

reg add HKLM\System\CurrentControlSet\Services\W3SVC\Parameters /v ConfigPollMilliSeconds /t REG_DWORD /d 2147483647 

What this does is basically tell IIS to not monitor changes to web.config files or any other files, AT ALL. It also won’t see any changes to any files the web server caches, e.g. static files or files compressed by IIS static compression.

With that in mind, it is possible to have a crazy but maximally defensive strategy to handle changes to the web application without restarting.

And that is, to not detect ANY changes to configuration and content, and then to apply those changes manually by recycling the application pool instead!

While this gives up the very useful ability to make auto-detected changes to the application and files, it provides one crucial benefit:

The ability to trigger an overlapped application restart to apply any changes!

Of course, if you are using this strategy, you have to seriously consider the implications of IIS not detecting any changes to your files.

To those good with that, I just want to tell you both good luck. We are all counting on you.

Conclusion

This was probably harder than you expected, but look at what we’ve been able to accomplish:

  1. Your application is always on, and always warmed up for maximum performance.
  2. Your application stays warm despite restarts and recycles.
  3. You have no “experienced” startup delay when recycling takes place.
  4. You have minimum restart overhead by reducing unnecessary recycling.

Not to mention, I was able to include at least 5 jokes from my favorite movie “Airplane”. Or was it six? I can’t tell. Maybe you can take a guess? Not for another two hours?

Congratulations!

ConfigureWarmup tool

To help you configure this, we also created a tool that will make all these changes for you for any application (and application pool) on your server.

Read more and download this tool here.

Here is how to use:

ConfigureWarmup "MyWebSite/app" [/warmupUrls:url1,url2,url3…] [/testWarmup]

/warmupUrls: optional, list of urls you want to warm up each time. If missing, will just do the default “/” request.

/testWarmup: if present, runs a test of your warmup strategy by making requests to your application across a recycle and measuring any downtime, errors, or request delays. Then displays your grade!

If you implement this strategy, please comment below and let us know how it went! We’d love to hear your success story, or hear any additional data or concerns you may have with this approach.





More resources


Cannot use SAAS monitoring / need an on-premise solution?

Talk to us about LeanSentry On-Premise.

Want to automate LeanSentry deployment in a cloud environment?

Read this.

Need expert assistance with an urgent performance issue?

Get an quick consultation from one of our performance engineers.