AWS Direct Connect Technical Deep Dive (IX)
In the previous posts, we looked at how to use a site‑to‑site VPN to connect your on‑premises network to AWS, and as we saw, it is very easy to set up. So what’s the fuss about Direct Connect (DX), and why would we need one?
To give you a one‑word answer, a VPN connects through the Internet. As you would expect, that comes with some limitations. Latency can be high, and the throughput is capped at around 1.25 Gb/s (per tunnel). So what if we need something more resilient and with much higher throughput?
That is where AWS Direct Connect comes in. As the name suggests, it is a Dedicated Direct Connection (DX Connection) to AWS, giving you a dedicated network link with better performance and reliability compared to a traditional VPN over the Internet.
As always, if you find this post helpful, press the ‘clap’ button. It means a lot to me and helps me know you enjoy this type of content. If I get enough claps for this series, I’ll make sure to write more on this specific topic.
Avoiding Some Confusion
Before we proceed, I want to point out that Direct Connect can be confusing because there are several ways to get it. The easiest way is by directly cross‑connecting to AWS’s network equipment, but for that, you need to have your own network kit in the same co‑location facility as an AWS Direct Connect location.
The confusion usually starts when you don’t have a presence in one of those facilities, and you work through a provider/partner instead. Depending on the provider/partner, they might extend a Layer 2 circuit all the way to your site, or they might offer a Layer 3 service like MPLS. To make things even more confusing, you can also get what’s called a hosted connection through a partner, which allows you to start with sub‑1 Gb/s speeds.
We will structure this post in a way that starts with a dedicated connection first, and then we will look at the other options. Most of the underlying components and the way we establish the BGP sessions remain the same (mostly 🙂)
AWS Direct Connect Overview
Before we go deeper into the topic, let’s start at a very basic level. AWS has what are called Direct Connect locations. These are simply co‑location facilities where AWS has installed its own equipment that is connected directly to the AWS global backbone network. You can find the list of locations here.
For example, if you have workloads in a data centre like Equinix LD5 in London, and you want those workloads to connect to your resources in AWS, you can take advantage of the Direct Connect service available in that same facility. Equinix is a co‑location provider, and LD5 is just the name of one of their data centres. It’s important to understand that you’re not directly connecting to your AWS cloud resources. You’re connecting to AWS’s own equipment inside that facility, and that equipment is what connects you to the AWS backbone. By ordering a physical link (a cross‑connect) from your kit to theirs, you gain direct access to AWS's backbone network. This is called a 'Dedicated Connection'.
If you don’t have a presence in one of the co‑location facilities where AWS provides Direct Connect, don’t worry. You can still get a Dedicated Connection through a service provider. These providers already have infrastructure in those facilities and can provide you with a Direct Connect service through them.
Alternatively, you can use what AWS calls a Hosted Connection. A Direct Connect Partner provides a hosted connection. One of the reasons you might choose a hosted connection is if you don’t want to commit to the full 1 Gb/s of a dedicated port. Hosted connections let you start with smaller speeds such as 50 Mbps, 100 Mbps, 200 Mbps, 300 Mbps, 400 Mbps, or 500 Mbps. This is often useful when you are just getting started or don’t need high throughput. With dedicated connections, on the other hand, the smallest option you can order starts at 1 Gb/s.
Once you have the cross‑connect in place between your equipment and AWS’s (directly or via a partner), the next step is to establish BGP sessions over that link. This is how routes are exchanged between your network and AWS. We will go through this in detail in the upcoming sections.
Direct Connect Pricing
At a high level, Direct Connect pricing is based on port hours and data transfer. As long as you have a dedicated connection or hosted connection, you pay for the port by the hour. The pricing is mostly consistent across regions, except for Japan.
As of writing this post, a 1 Gbps dedicated port costs $0.30 per hour, 10 Gbps is $2.25 per hour, 100 Gbps is $22.50 per hour, and 400 Gbps is $85 per hour.
For hosted connections, pricing starts as low as $0.03 per hour for 50 Mbps, $0.06 for 100 Mbps, and goes up to around $0.20 per hour for 500 Mbps. Hosted options at 1 Gbps or higher are also available, depending on the partner.
In addition to port charges, there’s a data egress charge for any traffic leaving AWS through Direct Connect. Data transfer into AWS is free. For detailed and up-to-date pricing, it's best to refer to the official pricing page here.
Dedicated - Colocation Cross-Connect
As mentioned at the beginning, we’ll start with the dedicated Direct Connect option, where you already have your network equipment in the same facility as an AWS Direct Connect location. Here is a list of all AWS Direct Connect locations and campus data centres where AWS Direct Connect is accessible using a standard cross-connect.
A dedicated connection provides you with your own physical port on AWS’s equipment within a Direct Connect location. This port is not shared with anyone else and connects directly to the AWS global backbone. Dedicated connections are available in fixed port speeds of 1 Gbps, 10 Gbps, 100 Gbps, and 400 Gbps, giving you options based on how much bandwidth you need. You can order and manage Direct Connect via the dedicated 'Direct Connect' service in the AWS console.
Assuming you already have a presence in a data centre where AWS offers Direct Connect, all you need to do is order a Direct Connect (or multiple) and download an LoA from AWS and give it to the data centre provider. LoA stands for Letter of Authorization, and AWS provides this document so the data centre knows that AWS allows you to connect to their equipment.
Using that LoA, you order what’s called a cross‑connect. A cross‑connect is just a physical cable inside the data centre that connects your equipment to AWS’s equipment. If you are getting a Direct Connect connection directly from AWS, you need to meet certain networking requirements.
- Direct Connect speed would be 1 Gbps, 10 Gbps, 100 Gbps or 400 Gbps
- Your network must use single‑mode fibre with the correct transceiver type for the speed you choose.
- 1000BASE‑LX (1310 nm) for 1 Gbps
- 10GBASE‑LR (1310 nm) for 10 Gbps
- 100GBASE‑LR4 for 100 Gbps
- 400GBASE‑LR4 for 400 Gbps Ethernet.
- Depending on the AWS Direct Connect endpoint serving your connection, auto‑negotiation on your on‑premises device might need to be enabled or disabled for a dedicated connection.
- You also need to support 802.1Q VLAN encapsulation across the entire path, including any intermediate devices.
- Finally, your device must support BGP and BGP MD5 authentication.
- Link Aggregation Group (LAG) support (optional)
Layer 2 and LAG
The dedicated links that you receive terminate on a network port on either side, and these are layer 2 ports. The link you receive is just a single link connected to two switch ports on either side. To add some redundancy or to increase throughput, you can use a Link Aggregation Group (LAG). All connections in the LAG must terminate at the same AWS Direct Connect device. A LAG can handle individual link failures, but it does not protect you from device failures.
Layer 3 Sub-Interfaces
The cross‑connect we order to AWS is a Layer 2 link. To carry multiple logical connections over that single link, we use 802.1Q VLAN tagging. On our router or switch, we create Layer 3 subinterfaces, each with its own VLAN tag.
These subinterfaces are then used to build BGP sessions for the different virtual interfaces we configure, such as private, public, or transit. This is a nice segue to our next important topic - Virtual Interfaces or VIFs.
Direct Connect Virtual Interfaces (VIF)
Let's assume we have the direct connect in place. What do we do next? Let’s say we have a single VPC with a VGW attached to it. How do we access the resources inside that VPC from our on‑prem network? Having the physical connection is just the start. On top of that link, we need to establish a BGP session to advertise and receive routes. The way we do this is by creating virtual interfaces. AWS Direct Connect supports three types of VIFs.
- Private VIF - used to access private resources in your VPC through a Virtual Private Gateway (VGW)
- Public VIF - used to access AWS public services such as S3 over the Direct Connect.
- Transit VIF - used when connecting to a Transit Gateway (via Direct Connect Gateway) to reach multiple VPCs or networks through one virtual interface.
Assuming you’re using a Cisco router and the interface that connects to Direct Connect is something like Gi0/1, then the virtual interfaces are just subinterfaces you would normally create for VLAN tagging.
interface Gi0/1.10
description PUBLIC-VIF
encapsulation dot1q 10
ip address 125.25.25.1 255.255.255.252
interface Gi0/1.15
description PRIVATE-VIF
encapsulation dot1q 15
ip address 169.254.15.10 255.255.255.252
interface Gi0/1.20
description TRANSIT-VIF
encapsulation dot1q 20
ip address 169.254.20.10 255.255.255.252Here, for example, .10 is the VLAN ID assigned to the subinterface, and we use this for establishing a Layer 3 connection with AWS over the Direct Connect link.
Private VIF
A Private VIF is used to connect to private resources inside a VPC. If you remember from our earlier posts, we initially connected to our VPC using a site‑to‑site VPN. In that setup, we terminated the VPN on a Virtual Private Gateway (VGW). The Private VIF works in a similar way. It connects to the VGW, allowing your on‑prem network to access resources in the VPC over the Direct Connect link instead of the Internet.
Please note that each Private VIF can be associated with only one VGW, which means you can't use a single Private VIF to connect to multiple VPCs (unless you use a Direct Connect Gateway or DXGW). There are better ways to manage multi‑VPC connectivity, which we’ll cover in the next section. Another important thing to keep in mind is that the VPC you’re connecting to must be in the same region as the Direct Connect location. You can't use a Private VIF to reach VPCs in other regions (again, unless you use a Direct Connect Gateway or DXGW)
So to recap, if you have a single VPC in eu-west-2 and want to connect to it reliably from your on-prem workloads, you’ll get a Direct Connect between AWS and your device by means of a cross-connect. Once the link is in place, you create what’s called a private VIF on the AWS side and a sub-interface on your router. On the sub-interface/private VIF, you assign point-to-point IPs and establish a BGP session. You can advertise up to 100 prefixes to the VGW, and in turn, the VGW advertises the VPC CIDR back to you. If you have another VPC in the same region, you repeat the same steps, create a new private VIF, a new sub-interface, and a new BGP session.
Please be mindful when setting up a setup like this. In this setup, your on‑prem device will advertise the CIDR of each VPC to the other, which means your on‑prem router could unintentionally become a transit path for VPC‑to‑VPC communication. This not only adds unnecessary complexity but can also lead to significant data transfer charges, especially for traffic that never needed to leave AWS in the first place.
In cases like this, VPC peering would be a better fit, as it allows VPCs to talk directly without routing through your on‑prem network. Keep this in mind when designing your architecture.
Public VIF
As the name suggests, a Public VIF is used to connect to AWS services that are available over the internet. Things like S3 buckets, DynamoDB, or any other service that uses public IP addresses. Instead of sending your traffic over the Internet, you can use Direct Connect to reach these services through a dedicated, more reliable path.
To make this work, first create a sub-interface on the router, create a public VIF on AWS, and finally, announce your own public IP prefix to AWS, and in return, AWS announces their public IP ranges to you over the BGP session. This way, your traffic to AWS public services stays off the Internet and flows over your Direct Connect.
For the BGP peer IPs, you must use public IP addresses, and you also need to announce at least one public prefix. If you own a public BGP ASN, you can use it. If not, you can use a private ASN in the range 64512–65534. On the AWS side, BGP is established using ASN 7224.
Once the BGP session comes up, AWS will send you all available local and remote region public prefixes. Direct Connect also does inbound packet filtering to ensure the source IP of your traffic matches your advertised prefixes. In other words, you can only send traffic from prefixes that you’ve advertised over the public VIF.
Since AWS announces public prefixes from all regions over the Public VIF, you can access AWS services in any region through your Direct Connect link. So even if your Direct Connect location is in London, you can still reach public AWS services hosted in regions like Frankfurt or Tokyo.
Transit VIF
The next type is called a Transit VIF, which you can use to connect your Direct Connect to a Transit Gateway using a Direct Connect Gateway (DX Gateway) in between. Hopefully, by now, you’re familiar with TGW, which allows you to attach multiple VPCs and VPNs to centralize your network architecture.
With a Transit VIF, you connect to the DX Gateway, which then connects to the TGW. This gives your on‑prem network access to any VPC attached to the Transit Gateway. You can probably already see the advantage over a Private VIF, which only connects to a single VPC. Let’s move straight into the Direct Connect Gateway next, and we’ll cover all of this in more detail.
Direct Connect Gateway (DXGW)
Next, let’s take a closer look at Direct Connect Gateway. You can use a DXGW with either a Private VIF or a Transit VIF, but not both at the same time. It’s important to choose the right type of VIF based on your use case. Also, keep in mind that DXGW does not support Public VIFs.
You can have 30 Virtual interfaces (private or transit) per AWS Direct Connect gateway. You can also have 200 Direct Connect gateways per account.
DXGW and Private VIF
Earlier, we saw that with a Private VIF, you can only connect to a single VPC, and you're limited to 50 VIFs (private) per Direct Connect connection. That means even with a dedicated Direct Connect, you can only connect to a maximum of 50 VPCs. Another key limitation is that Private VIFs only support VPCs in the same region; you can’t connect to VPCs in other regions.
Direct Connect Gateway helps solve these limitations. Instead of one VPC per Private VIF, you can now connect up to 20 VPCs to a single Private VIF through a DXGW. And since you can have up to 50 Private VIFs per Direct Connect, that gives you the ability to connect to up to 1000 VPCs using a single Direct Connect connection. Even better, DXGW supports cross-region VPC attachments, so you can connect to VPCs in other regions without needing multiple Direct Connect locations. (Please note if you have 1000 VPCs, this may not be the right way to do this, but we will look at using TGW in the next section)
You associate a Direct Connect Gateway with the Virtual Private Gateway (VGW) of the VPC. After that, you create a Private Virtual Interface (VIF) on your Direct Connect connection and attach it to the DX Gateway.
To associate a VGW with a DXGW, go to the AWS Direct Connect console, select your DX Gateway, and choose 'View details'. Under the 'Gateway associations' tab, click 'Associate gateway', select the VGW you want to link, and confirm the association.
You can also associate a Direct Connect Gateway with a Virtual Private Gateway from another AWS account. The VGW owner sends an association proposal, and the DX Gateway owner must accept it for the connection to be established.
Another very important thing to remember is that DXGW is not transitive. It’s not a router in the data plane; it's a globally available control plane construct. So even though we’ve associated multiple VPCs with the DXGW, as shown in the diagram, these VPCs cannot talk to each other through the DXGW. The DXGW only facilitates connectivity between your on‑prem network and the remote VPCs. It does not enable VPC‑to‑VPC communication. Please keep this in mind when designing your architecture.
DXGW and Transit VIF
Similarly, you can use a Transit VIF with a Direct Connect Gateway to connect to Transit Gateways across regions. Just remember, a single DXGW can only support either Private VIFs or Transit VIFs, not both at the same time.
Using this setup, you can connect multiple Transit Gateways to a single DXGW and extend your on‑prem connectivity to multiple VPCs across different regions. You can associate up to six Transit Gateways per Direct Connect Gateway, and each Transit VIF can receive up to 200 prefixes from AWS to your on‑prem network. Additionally, a single Transit Gateway can be associated with up to 20 Direct Connect Gateways.
A Quick Recap
We’ve gone through quite a bit so far, so let’s quickly recap. We looked at the three types of virtual interfaces, Private, Public, and Transit, and how each one connects to different parts of the AWS network. Private VIFs connect to individual VPCs via VGWs, while Transit VIFs connect to Transit Gateways, which in turn can connect to multiple VPCs. Public VIFs are used to access AWS public services like S3 or DynamoDB across all regions.
We also covered how Direct Connect Gateway helps scale connectivity by allowing you to attach multiple VPCs or Transit Gateways, even across different regions or accounts. Just keep in mind that Direct Connect Gateway is not transitive, so traffic won’t automatically flow between VPCs connected to the same DXGW; you’d need something like VPC peering or TGW peering for that.
Also, remember, so far we only looked at getting a dedicated cross‑connect directly to AWS with no third party involved. The requirement for that is having your network gear in the same co‑location facility as AWS’s Direct Connect presence. In the next sections, we’ll look at how to get a dedicated Direct Connect through a network service provider instead.
Dedicated -Network Service Provider Layer 2 Extension
If you don’t have any network infrastructure within a Direct Connect location but still want the benefits of a dedicated connection, you can work with a network service provider to extend Layer 2 connectivity from your location to the AWS Direct Connect facility. This setup gives you the same capabilities as a local cross‑connect without needing to be physically present at the Direct Connect site.
With a Layer 2 extension, you get the same capabilities as if you had a direct cross-connect in a co-location. This includes support for features like Link Aggregation Groups (LAG), MACsec encryption, VLAN tagging, BFD, and full BGP configuration.
You still request the connection through the AWS Direct Connect console, just like before. Once the request is approved, you forward the Letter of Authorization (LOA) to your provider so they can complete the cross-connect setup on their end.
Dedicated - Network Service Provider Layer 3 Managed
If you're using an existing Layer 3 service like MPLS to connect remote sites and data centres, this managed option allows you to reach AWS through your current network. You don’t need to have physical presence at a Direct Connect location or perform any additional network configuration, your provider handles the routing to AWS.
The provider establishes BGP sessions, and the customer configures BGP with the provider edge.
Hosted Connection
A hosted connection is a Direct Connect service that a Direct Connect partner can provision for you using their pre‑provisioned network circuits. A Hosted Connection can support one private, public, or transit VIF. Each Hosted Connection supports only a single VIF, and you can obtain multiple VIFs by configuring multiple Hosted Connections.
With hosted connections, you can get direct connect speeds of 50 Mbps, 100 Mbps, 200 Mbps, 300 Mbps, 400 Mbps, 500 Mbps, 1 Gbps, 2 Gbps, 5 Gbps, 10 Gbps, and 25 Gbps. Note that only those AWS Direct Connect partners who have met specific requirements may create a 1 Gbps, 2 Gbps, 5 Gbps, 10 Gbps, or 25 Gbps hosted connection. Hosted Connections are ideal for smaller environments or for getting started with Direct Connect without committing to higher capacities.
Layer 2 Hosted Connection (Virtual Cross-Connect)
As shown in the diagram, the connection between the AWS Direct Connect equipment and the partner device is referred to as an Interconnect. An Interconnect is a 1G or 10G Ethernet fibre-optic port that’s made available only to approved Direct Connect Partners. 802.1Q VLANs provide separation between different Hosted connections on an Interconnect.
You establish a BGP session directly to the direct connect endpoint. It is the responsibility of the Direct Connect Partner to assign VLANs when allocating Hosted connections to customers, and to deliver each VLAN to the appropriate customer.
With hosted connections, once you request and accept a connection through a Direct Connect partner, it appears under your Direct Connect connections in the AWS console. You’ll need to select the VLAN ID through the partner’s portal as part of the setup. After that, the connection becomes visible in your AWS account, and you can create a virtual interface (only one VIF is allowed per hosted connection). Once the VIF is created, you can configure BGP peering to start exchanging routes.
Connection port speeds can only be changed by your AWS Direct Connect Partner. If the Partner supports upgrade/downgrade of the hosted connection, you are no longer required to delete and then recreate a connection in order to upgrade or downgrade an existing hosted connection's bandwidth. AWS uses traffic policing on hosted connections, which means that when the traffic rate reaches the configured maximum rate, excess traffic is dropped.
A good example of this setup is Megaport. You can get a ‘port’ from Megaport and connect it to your own network kit. This port is just a cross‑connect between you and Megaport, but once it’s in place, you can access any service connected to Megaport’s network. One of those services is an AWS Hosted Connection.
You can log in to their portal and order a hosted connection directly. Megaport assigns a VLAN for the connection, and you’ll typically pay a monthly fee to Megaport for the port itself, plus hourly port charges to AWS for the hosted connection.
Layer 3 Hosted Connection (MPLS)
You can also get a Hosted Connection through a network service provider using their existing Layer 3 service, like MPLS. So, if you already have MPLS across your network and the provider supports Direct Connect, they can provision the connection for you. In this setup, you establish BGP with the provider.
AWS Direct Connect SiteLink
SiteLink, a new feature of AWS Direct Connect, makes it easy to send data from one Direct Connect location to another, bypassing AWS Regions. Before SiteLink, you won't be able to route traffic directly between Direct Connect locations. Now, you can create global, reliable, and pay-as-you-go connections between the offices and data centres in your global network by sending data over the fastest path between AWS Direct Connect locations.
To use SiteLink, you first connect your on‑prem networks to any Direct Connect location. Then, you create Virtual Interfaces (VIFs) on those connections and enable SiteLink. Once the VIFs are all attached to the same Direct Connect Gateway (DXGW), you can begin sending data between those sites using SiteLink.
Closing Thoughts & References
That wraps up our deep dive into AWS Direct Connect. We’ve covered everything from dedicated and hosted connections to topics like Direct Connect Gateway, VIFs, and SiteLink. Direct Connect can feel overwhelming at first, but once you understand the core building blocks and how they fit together, the picture becomes much clearer. I hope this post helped break it down in a simple and practical way. As always, if you have questions or spot anything that needs correction, feel free to reach out.