How to Use Azure ExpressRoute Global Reach to Interconnect Datacenters

Wide Area Market is in a change right now. When talking to my customers, I see more and more of those customers moving away from classic MPLS and Dark Fibre networks, and with that, many of them are thinking about the value of services like Microsoft Azure ExpressRoute.

With this blog post, I would like to give you some insides into how ExpressRoute is evolving and becoming more valuable in addition to a simple MPLS or Datacenter Interconnect to Azure.

First, let’s get some frequently asked questions away.

What is Azure ExpressRoute?

ExpressRoute is an Azure Service that lets you connect your on-premises networks and network colocations to the Microsoft Edge Network via private connection between you, your connectivity provider, and Microsoft. These connection locations are called Microsoft Enterprise Edge or MSEE and they are distributed around the globe in around 65+ network colocations in different datacenters.

With ExpressRoute you can either access Microsoft Azure Services or even Microsoft 365 Services. Microsoft still does not recommend accessing Microsoft 365 trough ExpressRoute but there is another interesting option on how to use ExpressRoute in extending to regular Azure Service access.

Later in the article when we dig deeper into our main topic, you will learn how to leverage ExpressRoute to connect your datacenters and colocation independently from provider availability in that location. There is a SKU for ExpressRoute called Global Reach, which enables customers to build and extend their backbone through the Microsoft Global Network.

How Does Azure ExpressRoute Work?

As already explained, ExpressRoute is a private connection between the customer and the Microsoft Edge Network. ExpressRoute comes in different flavors.

ExpressRoute or ExpressRoute Direct?

There are two options on how to do the physical interconnect. The first and regular option is to use ExpressRoute through a Network Provider as shown in the example below.

The other option is to use ExpressRoute direct. Here you eliminate the need of a Network Provider and establish a connection with Microsoft yourself.

With a regular ExpressRoute you are limited to what your provider can offer you in regards to locations and you are limited in a maximum of 10GBE bandwidth. Also, you have additional costs on the provider interconnect between you and Microsoft but it is much easier to implement and you do not need high-end routing equipment, colocation space in the MSEE Edge location, or dramedies knowledge on provider peering.

With ExpressRoute Direct you can achieve much higher bandwidth, currently up to 100 GBE. For that, you need to follow some peering policies and technical requirements shown below.

• Microsoft Enterprise Edge Router (MSEE) Interfaces:
  • Dual 10 or 100 Gigabit Ethernet ports only across router pair
  • Single Mode LR Fiber connectivity
  • IPv4 and IPv6
  • IP MTU 1500 bytes
• Switch/Router Layer 2/Layer 3 Connectivity:
  • Must support 1 802.1Q (Dot1Q) tag or two Tag 802.1Q (QinQ) tag encapsulation
  • Ethertype = 0x8100
  • Must add the outer VLAN tag (STAG) based on the VLAN ID specified by Microsoft – applicable only on QinQ
  • Must support multiple BGP sessions (VLANs) per port and device
  • IPv4 and IPv6 connectivity. For IPv6 no additional sub-interface will be created. IPv6 address will be added to existing sub-interface.
  • Optional: Bidirectional Forwarding Detection (BFD) support, which is configured by default on all Private Peerings on ExpressRoute circuits

The table below shows the key difference between ExpressRoute with a Service / Network Provider and ExpressRoute direct.

ExpressRoute using a service provider	ExpressRoute Direct
Utilizes service providers for fast onboarding and connectivity into existing provider infrastructure	Requires 100 Gbps/10 Gbps infrastructure and full management of all layers
Integrates with hundreds of providers including Ethernet and MPLS	Direct/Dedicated capacity for regulated industries and massive data ingestion
Circuit SKUs: 50 Mbps 100 Mbps 200 Mbps 500 Mbps 1 Gbps 2 Gbps 5 Gbps 10 Gbps	Customer may select a combination of the following circuit SKUs on 100 Gbps ExpressRoute Direct: 5 Gbps 10 Gbps 40 Gbps 100 Gbps Customer may select a combination of the following circuit SKUs on 10 Gbps ExpressRoute Direct: 1 Gbps 2 Gbps 5 Gbps 10 Gbps
Optimized for single tenant and single customer	Optimized for single tenant with multiple business units and multiple work environments Network Providers are not allowed to leverage ER direct for their customers. They are supposed to become regular ExpressRoute providers.

Additional insides about ExpressRoute Direct can be found in Microsoft’s official article.

ExpressRoute SKUs

ExpressRoute comes in three different SKUs for available services and limitations. 

• ExpressRoute Local / Kommune
• ExpressRoute Standard
• ExpressRoute Premium

A detailed pricing guide is also available.

ExpressRoute Global Reach is an Add On to ExpressRoute Premium SKU and needs to be ordered in addition. 

I will explain to you the technical concept later in the blog. For those curious in pricing, a detailed guide is available

What is an ExpressRoute Circuit?

To explain an ExpressRoute Circuit you need to know about the other components of an ExpressRoute in Azure first.

• ExpressRoute Gateway: when you want to connect an ExpressRoute Circuit to a virtual network, you need some kind of Gateway. That gateway is called an ExpressRoute Gateway and another “mode” of the Azure Virtual Network Gateway.
• Peerings: To establish the routing through an ExpressRoute Circuit you need to configure peerings on the circuit to establish the BGP connection. There are two peering types.
  • Private Peering: Azure virtual machines and IaaS resources, and Azure PaaS resources that can leverage private Endpoints or VNet integration, that are deployed within a virtual network can be connected and leverage the ExpressRoute Private Peering. The Azure Private Peering can be considered as a trusted extension of a customer core network into Microsoft Azure Datacenters. With private peering, you set up a bidirectional interconnect between your network and your virtual network in Azure. The peering IPs used on the private peering are private IPv4 and later this year IPv6 addresses.
  • Microsoft Peering: Connectivity to Microsoft online services such as Office 365 and Azure PaaS services are made available via the Microsoft Peering. Microsoft enables bi-directional connectivity between a customer WAN and Microsoft cloud services through the Microsoft global backbone and Microsoft Routing Domain named with AS# 12076. Microsoft Peering can only use public IP addresses owned by the customer or customer connectivity provider. To enable Microsoft Peering, you need to agree to all with that peering connected rules.

Now let’s talk about the circuit itself. The circuit is a so-called NNI, a Network to Network Interconnect.

A network-to-network interface (NNI) is a physical interface that connects two or more networks and defines inter signaling and management processes. It enables the linking of networks using signaling, Internet Protocol (IP) or Asynchronous Transfer Mode (ATM) networks.

How do I set up an Express Route in Azure?

There are three different parts to set up an ExpressRoute.

The first one is the Setup within the Azure Portal. Microsoft published a pretty good guide on how to do it.

The next part would be to setup the provider part of your ExpressRoute, which is highly dependent on your provider. I linked you examples from two providers.

• Microsoft Azure ExpressRoute with Megaport
• Microsoft Azure ExpressRoute with Equinix

Afterward, you need to configure the peering in ExpressRoute and the routing in your Router or Network.

Using ExpressRoute with Global Reach to Interconnect Datacenters

Now, let’s move to the main topic. How can I use ExpressRoute to interconnect my colocation and datacenters.

Why Would you Use ExpressRoute Instead of a Global Network Provider?

There are three main reasons why you maybe want to decide on a combination of ExpressRoute and a local interconnect provider.

• Provider availability: When you look into provider availability, you will sooner or later notice that not every provider is available in every region or in every datacenter. When you are in a local datacenter or a region with a limited amount of network providers, you normally have high costs to make your network provider available in that datacenter or on-prem location. Let me show you an example from the peering Database. With ExpressRoute you can select any provider who can make ExpressRoute connections available in that region or datacenter. You do not need the same provider in every location.
  • Networks in Equinix Dusseldorf

• • Networks in ITENOS Berlin

• Longterm contracts: When you want to interconnect datacenters you mostly need to agree to some kind of long term contract starting with 12 months or more contract time. With providers like Megaport, Equinix, Interxion, and others, you mostly have a pay as you go agreement which can be canceled every month. It is the same with Microsoft ExpressRoute. You can use that interconnect as pay as you go.
• Provider Lock In: When working with Network Providers you normally commit to a provider and to change afterward is a huge migration with high time and financial investment. Many customers don’t have that flexibility and overpay on networking.

 

What do I Need to Enable ExpressRoute for Global Interconnect?

ExpressRoute by default is already a global service but with the main SKUs it can only connect you to Azure Production Regions / Datacenter and not interconnect networks from different providers. To interconnect network providers, you need the ExpressRoute Global Reach Add On.

Looking on an ExpressRoute without Global Reach, the interconnect would look like the following.

When enabling Global Reach, the routing behavior changes as shown below.

While an Azure Production Region is normally three milliseconds from an ExpressRoute Edge, the traffic with Global Reach stays within the Microsoft Edge Network.

Now, let’s think about how that could work with an interconnect strategy. In our scenario, we have the following locations and interconnects.

• Europe provided by British Telecom
• Hong Kong provided by Equinix
• Japan provided by NTT

Normally you would need to ask all of those providers to build an interconnect,  or get another colocation or office location to interconnect all these networks yourself. As you can see in the picture, with ExpressRoute Global Reach you can do that trough the Microsoft Backbone and use it as a service from Microsoft Azure. The additional fact is, you do not leverage additional cloud services from Microsoft. You can just use Microsoft as a backbone provider. The figure below shows our scenario simplified.

After you configured the ExpressRoutes with the local providers of your choice you need to setup ExpressRoute Global Reach. There is one downside. ExpressRoute Global Reach is not available in every country where you have Azure Regions. Mostly because there are law or tax regulations which make Microsoft with Global Reach a network service and last-mile network provider. In those cases, Microsoft is mostly solving that over time with special government agreements.

There is a workaround for those countries where Global Reach is not available, like in South Africa, India or Brazil. I will explain the workaround later in the blog.

How to Set Up Global Reach

ExpressRoute Global Reach can only be set up through PowerShell or Azure CLI. There are two options when you set up Global Reach. I will link you both setup guides below.

• ExpressRoutes are in the same Subscription 
• ExpressRoutes are in different Subscriptions or Tenants 

Afterward, you need to verify the configuration. That must be done via PowerShell or Azure CLI too.

$ckt1 = Get-AzExpressRouteCircuit -Name "Your_circuit_1_name" -ResourceGroupName "Your_resource_group"

If you simply run $ckt1 in PowerShell, you see CircuitConnectionStatus in the output. It tells you whether the connectivity is established, “Connected”, or “Disconnected”. For more information, you can consult this detailed guide.

To disconnect you also run a command which looks like following.

$ckt1 = Get-AzExpressRouteCircuit -Name "Your_circuit_1_name" -ResourceGroupName "Your_resource_group"
Remove-AzExpressRouteCircuitConnectionConfig -Name "Your_connection_name" -ExpressRouteCircuit $ckt_1
Set-AzExpressRouteCircuit -ExpressRouteCircuit $ckt_1

For more information, you can consult this detailed guide.

What is an Alternative when Global Reach is not available?

To use private peering with ExpressRoute Global Reach, it needs to be enabled in the country. As already explained, that’s not the case everywhere.

You can use the global transit architecture with Azure Virtual WAN or an overlay network using a Network Virtual Appliance. In that case, you create an ExpressRoute and do not use the ExpressRoute private Peering. You configure Microsoft Peering with that ExpressRoute. What you have then is the public IP addresses from Azure virtual WAN and the Network Virtual Appliance in that peering. That would enable you to build an IPSec Tunnel through the ExpressRoute to the VPN Gateway. With virtual WAN you would then be able to route through the Microsoft Backbone to the other ExpressRoute Gateways. With an NVA you can leverage User Defined Routes to establish the same transit architecture.

The schematic setup for a solution with Azure virtual WAN could look like following.

With virtual WAN such a solution comes out of the box as soon as ExpressRoute Global Reach becomes available. You only need to enable it and a few minutes later Virtual WAN will switch from IPSec VPN to the connected ExpressRoute. Afterward, you can just decommission the VPN Tunnel.

Closing Thoughts

Hopefully, my post gives you a brief introduction to the possibilities you have with ExpressRoute besides a simple connection to Azure. If you would like to read more about Wide Area Networks with Microsoft Azure, please leave a comment and describe the scenarios you are looking for.