Hyper-V Dynamic Memory: An Illustrated Guide

Even though Hyper-V Dynamic Memory technology has been with us since Service Pack 1 for 2008 R2, it remains one of the less discussed, and therefore more poorly understood components of Hyper-V. Despite many claims and common practices to the contrary, Dynamic Memory is perfectly suitable for most production virtualization workloads. That suitability is a discussion for other articles. This post is intended to explain what Hyper-V Dynamic Memory is, how it functions, and how it compares to memory management techniques in other hypervisors.

As we progress through this post, I’ll use the following screen shot as a reference point:

 

What is Hyper-V Dynamic Memory?

I don’t know of any other virtualization technology that manages memory in a comparable fashion, so if you’re coming to Hyper-V from another hypervisor, you’ll want to pay attention to this part. Most other hypervisors use a different approach to memory management that is more universal, but has problems of its own.

Comparison: Oversubscription Model in Other Hypervisors

This is the more common hypervisor memory-management scheme that Hyper-V does not use:

 

In this mode, the virtual machine always believes that it has 8 GB of memory to use as it sees fit. In reality, the hypervisor may or may not be silently paging any amount of that memory to disk. Because of the isolation between the guest operating system and the hypervisor, there’s no way for the host to be certain that it is making the correct choices for what memory to page. Of course, the hypervisor can follow some semantic patterns, such as assuming the guest’s kernel will be in the first few megabytes of memory and skipping that, and it can perform some heuristic analysis and not page memory that it knows the guest is frequently accessing. No matter what, this approach is not without guesswork. Also, hypervisors that handle memory in this fashion are not bound by the needs of the guests at all. It always has the option to ignore semantics and heuristics, sending active memory to disk in order to make room for the memory needs of other guests. This can potentially lead to some very heavy disk thrashing and severe performance problems.

The primary takeaway from the oversubscription model is that the guest does not know that anything is out of the ordinary and will access its memory as though it were normal physical memory. The secondary takeaway is that performance problems may be caused by memory oversubscription.

Hyper-V Dynamic Memory

In contrast to the coercive nature of the memory oversubscription model, Hyper-V Dynamic Memory is cooperative. The hypervisor and its guests work together in memory management. This requires the usage of what is called a balloon driver. If you’re used to other hypervisors that employ the oversubscription model, you might have heard of this before. Some of them also employ a balloon driver to reduce the need for blind paging. One of the differences is that Hyper-V uses the industry-standard procedure for hot-add memory to grow the virtual machine’s allocation as necessary, then it employs the balloon driver to reclaim memory that the guest is no longer actively using.

Using the guest from the screen shot at the start of this post, we’ll work through a normal process.

This is what the guest sees when it first boots:

 

The guest’s memory manager believes that it has a maximum of 512 MB of memory to work with because that is its Startup value, so it works to keep its demand below that level. As you can see in the screen shot, this guest does know about the ceiling specified by Dynamic Memory. However, it will behave as though 512 is all that it has. That doesn’t stop applications from requesting more memory, of course. As they ask for more, the Demand on the memory manager increases — this is called pressure. This is noticed by the Dynamic Memory driver running in the guest, which tells Hyper-V about it. If Hyper-V has more memory to give, then the Hot Add mechanism kicks in and more is added. If possible, it is assigned with at least enough to cover demand.

Our guest now looks like this:

 

The maximum memory in this guest has changed to 865 MB. If applications place greater demand on memory and there is more to give, the total can increase all the way up to the maximum, which is set at 2.0 GB.

Sometimes, memory needs to be taken away from a guest. That process is quite a bit different. The following illustrates a situation in which that has happened:

 

The above screenshot pair was taken from inside a guest and from its view in Hyper-V Manager, respectively, for a different virtual machine on a host that doesn’t have enough memory to satisfy all of its guests needs simultaneously. As you can see, the maximum that this guest sees is the same as the maximum memory that has been reserved for it in Dynamic Memory. But, that’s not what’s assigned. In case the text in the images is too small to read, these are the numbers:

• The guest operating system’s memory manager reports that it has 1.5 GB maximum memory
• The guest operating system’s memory manager reports that 1.4 GB of the 1.5 GB maximum is in use
• Hyper-V Manager reports that the guest is currently demanding 975 MB of memory
• Hyper-V Manager reports that it has assigned 602 MB of memory to the guest

The most important thing to remember when you encounter situations like this is that when the memory information displayed in the guest operating system contradicts what is displayed in Hyper-V Manager, Hyper-V Manager is always right. Ultimately, Hyper-V owns all of the physical memory and is only loaning it out to the guests. What they do with it is their business, but they can’t get any more than Hyper-V is willing to provide.

So, what’s really going on in this guest? Why does it appear to have 800 MB more memory than Hyper-V says that it does? The root cause of this behavior lies in the fact that the Hot Add function for memory has no Hot Remove opposite. Once memory has been added to a guest (or a physical machine using Hot Add memory), that memory must remain assigned until that system is powered down. No matter what happens, the maximum memory as seen by the guest cannot be reduced until it is turned off. That doesn’t mean that Hyper-V can’t take memory from the guest. It means that Hyper-V must use its own techniques. Its implementation is the aforementioned balloon driver.

There isn’t a built-in tool in 2012 R2 that I know of that can display this information. What you see below is a screen shot taken with RAMMap. Look it over, then check the following paragraph for an explanation of what you see.

 

“Driver locked” memory is where the memory claimed by the balloon driver lives. Other drivers might also be using some of the memory that you see here, but even RAMMap can’t go any deeper than this. I’m not sure if that’s because Windows prevents it from going further or if it’s just due to the unique behavior of drivers. Whereas the operating system will reclaim all memory allocated by a user-mode application when it exits even if it was leaking, a driver can completely stop running without releasing its memory. So, there may not be any process tied to this memory that RAMMap or Task Manager can query, but the memory manager knows that it isn’t available.

So, where did that 900+ MB of memory go? In truth, Hyper-V has whatever isn’t taken by the other drivers in that guest — which, in this case, is most of it. For this system, Hyper-V has given most of its memory away to other virtual machines. It might also be keeping some for the management operating system.

After taking the above screenshot, I turned off another virtual machine on the same host that was consuming 2 GB of RAM. I then refreshed RAMMap, and this is what I saw:

 

Task Manager’s graphs didn’t look much different than they did before, but RAMMap shows over a 300MB difference. When the other guest shut down, Hyper-V was free to redistribute the memory that it was using. This guest got 300MB of it. To facilitate that, the balloon driver deflated — it claimed less memory from the guest’s memory manager. The guest still sees a 1.5 GB maximum, but it also saw that 300 MB had become free to use elsewhere (and quickly distributed it).

Hyper-V Dynamic Memory Features Quick Summary

From the above, several important points can be highlighted:

• Hyper-V never oversubscribes memory. There is no point in which the guest’s memory management is not involved with the mechanics of Dynamic Memory. The balloon driver notifies the guest of exactly how much memory it is taking away, but it can take no more than what the memory manager grants. Just as memory consumed by the balloon driver is unavailable to other processes, memory in other processes is unavailable to the balloon driver.
• Hyper-V Dynamic Memory does not page guest memory to disk. I should probably put a * by that statement (and the previous one as well), but it’s generally true. Smart Paging is a very rare exception to the rule. That aside, the guest operating system’s memory manager is free to page all of its memory that it wants to, but Hyper-V will not.
• The balloon driver is required. Because of the cooperative nature of Dynamic Memory, the balloon driver must exist in the guest and be functional. This is a bit of a non-issue for all currently-supported Windows and Windows Server operating systems, because they all come with some version of Integration Services and this particular bit can’t be disabled. The necessary Integration Components for Linux have been included in the core packages for some time now, so many Linux distributions will instantly support Dynamic Memory. Many have conditional requirements, such as the memory increments that can be used. Check for your distribution(s) on the TechNet site. I am personally using Dynamic Memory with the 15.10 version of Ubuntu Server with great success.
• The maximum amount of memory that the guest memory manager sees may increase, but it cannot decrease until the virtual machine is turned off. This fact is a persistent tripping point for those not accustomed to Dynamic Memory. Remember that Hyper-V Manager’s memory display is always right (unless you’re behind on patches and have that bug which shows a lot more memory than is even possible). You’ll need a tool such as RAMMap to get a better idea of what’s going on from inside the guest.
• Hyper-V only adds memory when pressure is applied to the guest’s memory manager. This is what makes Dynamic Memory inappropriate for some applications. Most modern programs simply ask for more memory when they need it and release memory when they’re done with it. These will be fine in a Dynamic Memory environment. A few query the memory manager for available quantities, then claim a portion. The memory manager in a guest with Dynamic Memory will not report memory that has not yet been assigned to the guest.
• Hyper-V only reclaims memory from guests when it is necessary to do so. Sometimes, you’ll see that a guest’s Memory Demand is lower, perhaps even much lower, than its Assigned Memory. That’s because Hyper-V is perfectly content to allow virtual machines to keep what they have until another guest needs it.

 

Hyper-V Dynamic Memory Pros and Cons

Hyper-V Dynamic Memory is certainly different than the methodologies used in other hypervisors, and people will argue until the end of time over what method is “best”. I find such arguments tiring and devoid of merit. What I expect you to understand is that every approach has strengths and weaknesses. What’s best for you may not be best for whomever wants to argue with you about it. This is a neutral assessment to help you decide what is most applicable to your situation.

Dynamic Memory Pros Against Other Methods

We’ll start with the benefits of Dynamic Memory.

• When host memory is low, Hyper-V signals the balloon drivers to start inflating. This causes the memory managers in the guests to start diverting memory into the balloon driver, taking it from other processes where possible. This ensures that the guests control their own memory assignment, which means that they can selectively page unused and rarely used memory to disk. Other hypervisors often have their own balloon drivers and will utilize them where possible, but they can also continue taking memory from the guests even when they don’t really have any more to give. The upshot is that the maximum negative performance impact that Hyper-V Dynamic Memory can have on guests is lower than that of oversubscribing hypervisors.
• Except for the rare instances in which Smart Paging is in effect, Dynamic Memory will never page the content’s of a guest’s memory to disk. This is important because, if the guest is accessing memory that it doesn’t realize is on disk, it expects it to respond within nanoseconds, not the milliseconds necessary to retrieve data from magnetic media. The upshot on this is the same as the previous bullet.
• Dynamic Memory is automatically available in most operating systems.

Dynamic Memory Cons Against Other Methods

Next, we’ll look at the weaknesses of Dynamic Memory when compared to the oversubscription model.

• Because Dynamic Memory does not page, it cannot achieve the densities of other methods. Obviously, having guests so densely packed that their memory is always being paged by the hypervisor is a fairly horrid situation, and I doubt there are any administrators that allow such a condition to persist for any longer than is absolutely necessary. With that as a given, it can be nice to have the option sometimes. Hyper-V does not give that to you.
• If Hyper-V doesn’t have enough memory to start a virtual machine, it stays off. We are again excluding situations in which Smart Paging would take over. The non-paging approach of Dynamic Memory could cause Live Migrations to fail in overcommitted clusters and it could stop a host from restoring all of its guests after an external condition such as a power outage. Other hypervisors will not face these limitations, which makes them more resilient under some adverse conditions.

 

Recommendations

There are a number of recommendations that can be extracted from this knowledge:

• Build with available density in mind. This is really a virtualization truism; Microsoft loves to remind everyone that even their competitors don’t recommend putting their hypervisors in a memory-overcommitted situation. If you’re using Hyper-V, you’ll really need to take this to heart.
• Be prepared to tinker. I’ve seen too many people try Dynamic Memory once, find they don’t like something about it, and then never use it again. That is a good way to waste memory.
• Follow manufacturer recommendations. Some vendors will tell you not to use Dynamic Memory at all (ask them why — Exchange makes sense; web servers do not). A lot of software vendors will tell you to use Reservations. Since Dynamic Memory can’t take away any more than the guest’s memory manager is willing to part with, it doesn’t have a true Reservation setting. If the software vendor coded their application to not release memory that it needs, then Dynamic Memory doesn’t need a Reservation setting. Unfortunately, most software vendors do not code their applications in accordance with their own requirements (which is a little mean on my part; it’s often easier said than done). Set the Minimum to match the vendor’s recommendation for Reservation.
• Use caution when using Dynamic Memory with databases and read the documentation. Microsoft’s SQL Server can support Dynamic Memory in some situations, but that doesn’t necessarily mean that you’ll want to use it. I disagree with any notion that SQL Servers should never use Dynamic Memory. Most SQL deployments have periods of heavy usage and periods of light usage just like everything else in the world, so locking their resources away even when they’re not using them defeats one of the main purposes of virtualizing.

These are some generic recommendations that should work well for common server applications. Check out our 10 Best Practices and Strategies for Hyper-V Dynamic Memory, and we’ve also got a guide on Hyper-V Dynamic Memory configuration. If you want to learn more about a Hyper-V Memory you can read this post which I had done a while back.