Advanced Troubleshooting of Hyper-V Replica – Part 2

In Part 1 of this series, we learned about the different possibilities which may cause communication issues between Primary and Replica Server. In this part, we continue to focus on other possibilities and also explain a mechanism which can be used to rule out network port or firewall issues.

• Ports not opened between Primary and Replica Server

As you know by reading the Hyper-V Replica articles I wrote recently that Primary and Replica Servers communicate using specific network ports. These ports are predefined in the Windows Firewall Rules which are created on Primary and Replica Servers when you enable the Hyper-V Role for the first time. By default, these two firewall rules are disabled. These two firewall rules must be enabled when you configure the Replica Server as shown in the below screenshot:

If you are using Active Directory as the authentication mechanism, you must enable “Hyper-V Replica HTTP Listener (TCP-IN)” rule and “Hyper-V Replica HTTPS Listener (TCP-IN)” if you are using the certificate-based authentication as shown in the above screenshot.

Windows Firewall Rule “Hyper-V Replica HTTP Listener (TCP-IN)” is configured to use network port 80 whereas “Hyper-V Replica HTTPS Listener (TCP-IN)” is configured to use network port 443. These ports must be opened in the Windows Firewall for Primary Server to communicate with the Replica Server.

Note: These are the default network ports used by the Hyper-V Replica Server but you can configure network ports based on your requirement.

Hyper-V replication is one-way (e.g. Primary Server will replicate virtual machine contents to the Replica Server but not vice-versa unless you use reverse replication). So replication notifications including the replica copies will be sent from Primary Server to Replica Server using the network ports defined in the Windows Firewall. Hence, these ports must be opened at the “Replica Server” to allow communication from Primary Servers.

In a large network environment where there a number of network devices between Primary and Replica Server, it is difficult to check the port status on each network device. Hence, to rule out any port or firewall issues between the Primary and Replica Servers, use the mechanism explained below which shows a replication connection being established from a Primary Server to a Replica Server successfully. You can actually use the Netstat command on both Primary and Replica Servers to ensure communication is taking place.

There are two screenshots shown below; one is from Hyper-V Replica Server and other one is from Hyper-V Primary Server. Screenshot “LABEL1” shows the NetStat command output from a Replica Server and “LABEL2” screenshot, which is taken from a Primary Server, shows the NetStat command output and processes with PID in the Task Manager.

LABEL1 – NetStat Output from Replica Server Accepting Connection from a Primary Server

LABEL2 – Netstat Output from Primary Server Connecting to Replica Server 

As you can see in the above screenshot “LABEL1” (which is from a Replica Server) which confirms the following points:

• Replica Server is listening on Port 80 (first and last lines shown in the “Local Address” column in the Netstat ouput).
• Replica Server IP Address is 192.168.1.11 which is shown in the “Local Address” column and a connection has been established with Primary Server (192.168.1.2) over multiple dynamic ports (“Remote Computer” column).
• “Connection Status” column shows “ESTABLISHED” (2^nd, 3^rd, 4^th and 5^th lines in the LABEL1 screenshot).

Basically, “LABEL1” screenshot says that the Primary Server has a connection established with Replica Server. Replica Server acts as a Server-Side component which listens for network traffic over network port 80 and Primary Server acts as a Client-Side component which uses dynamic ports to establish a connection with the Replica Server.

Now, look at the “LABEL2” screenshot which displays the NetStat output and Task Manager from Primary Server. As you can see, Primary Server (192.168.1.2) using dynamic port (62715, 62716, 62717 and 62718) which appears in the “Local Address” column in the NetStat ouput has established a connection with the Replica Server (192.168.1.11) over network port 80 (2^nd, 3^rd, 4^th and 5^th lines in the NetStat output).

The “1744” indicates the process ID which is responsible for establishing the connection with the Replica Server. If you look at the Task Manager, you can see PID “1744” is used by VMMS.exe process as indicated in the red circle of the screenshot “LABEL2” above.

This is a successful connection between Primary Server and Replica Server. This indicates that there are no issues with the network port and firewalls configured on both Primary and Replica Servers. If there are any issues with the Hyper-V replication, this could be related to other components. Since NetStat output on both the servers shows connection established, it confirms that the network devices are not blocking the required network ports for Hyper-V replication!

• Replication Disabled on Replica Server or Primary Server is not authorized to replicate to Replica Server

Remember that replication must be enabled on the Replica Server before it can accept the replication packets. To make sure the replication is enabled on the Replica Server, you can execute “Get-VMReplicationServer” PowerShell cmdlet on the Replica Server as shown in the below screenshot:

As you can see in the above screenshot, “RepEnabled” column in the table shows “False” and “AllowAnyServer” column shows “True”. This indicates that the Replica Server can accept connections from any Primary Server but the replication is actually “disabled” on the Replica Server. In other words, this Replica Server is not accepting replication.

The Issue could also be related to the Trust Group membership if you have configured one at Replica Server. If the Trust Group membership has been modified by an administrator then Hyper-V Primary Server might not be allowed to replicate to the Replica Server. You can check the Trust Group membership on the Replica Server to make sure Replica Server allows a connection from a specific Primary Server.

Conclusion

In this article we learned about a mechanism which we can use to check the connection status on both Primary and Replica Servers. Using this mechanism, we can rule out any issues related to the network ports or firewalls. Mechanism explained above makes it easy to avoid checking port status at every network device sitting between Primary and Replica Servers.

Finally, we learned about running Get-VMReplicationServer PowerShell cmdlet to make sure replication is enabled on the Replica Server and Primary Server is allowed a replication connection.

In the third and final part of this article series, we are going to touch upon a few more possibilities causing Hyper-V Replication to fail. Stay tuned!