ACI: Configuring a shared external Layer-3 connection for all Tenants

As with most things with ACI, we have a tremendous amount of flexibility in the configuration options to meet different requirements. In this post, we’ll explore options that allow multiple Tenants to use a common, shared L3Out (routing table) for the entire fabric (as opposed to using a L3OUT per VRF).

Assumptions:

• Only non-overlapping IP addresses can be leaked between tenants.
• Subnets leaked from multiple consumer networks, which are leaked into a shared VRF, must be unique and cannot overlap.

Prerequisites for this design:

To configure shared L3Outs, you must meet the following prerequisites:

• Configure ACI Fabric BGP route reflectors
• Configure Fabric Access Policies (ACI Leaf Interfaces that connect your External Routers will need to be configured)
• Tenant Configuration is complete (Application Profile, EPG, BD, Contracts / Filters, and VRF already exist)

Caveats for this design:

• Contracts for shared service must have the scope set to Global. The default scope is VRF and will not work for shared services.
• Transit routing between shared L3Outs in different tenants is not supported

 

HW/SW requirements:

• APIC release 1.2 or later is required for Option 3 (Inter-VRF Shared L3out)

• APIC release 1.1.3 or later is required for Option 2 if dynamic routing is required. Prior to 1.1.3, you could only use static routing with option 2.

• As of APIC release 1.2, shared services can be performed with a shared subnet defined under the bridge domain (BD). Prior to this, shared services required that the provider subnet be defined under the EPG that was to be shared.

• No hardware requirements

 

For Shared external Layer-3 connectivity to networks outside of the ACI fabric, customers have three validated options:

1. BD in Common tenant: Configure a shared L3 out for the fabric with static/dynamic routing (iBGP/OSPF/EIGRP) in Tenant Common. This option implies that all Endpoint groups (EPGs) are configured in respective user Tenant(s), while the Bridge Domains (BDs), subnets, and VRFs are all configured in the Tenant common where the L3out is configured.
2. BD in User tenant: Configure a shared L3 out for the fabric with static/dynamic routing in Tenant Common. This option implies that all Endpoint groups (EPGs),  Bridge Domains (BDs), and subnets are configured within the customer’s respective user Tenant(s), while the VRF is configured in the Tenant common where the L3out is configured.
3. Inter-VRF Leaking with Shared L3out: Configure a shared L3out for the fabric with static/dynamic routing in Tenant Common. This option implies that all Endpoint groups (EPGs),  Bridge Domains (BDs), subnets and VRFs are configured within the customer’s respective user Tenant(s), while only L3out is configured in the common tenant.

 

Option 1 –BD in Common tenant

Create a VRF, bridge domains, subnets and L3Out in the common tenant.
Create endpoint groups (EPGs) in individual tenant spaces.

Static and/or Dynamic routing can be used on the L3out.

In this configuration, tenants share the same VRF and cannot have overlapping IP addresses.  Since BDs and subnets are part of common tenant, the all subnets are visible to all tenants. In this case, all subnets are centrally located, and do not need to be configured or allocated everytime an EPG is created or deleted.

For Option 1, the tasks needed to implement are:

Configure in Tenant Common

1. Configure VRF under Tenant common
2. Configure L3 outside connection under Tenant common and associate it with the VRF configured in step 1
3. Configure Bridge Domains (BDs) and subnets under Tenant common. Associate Bridge Domains (BDs) with the VRF and L3 outside connection

Configure in User Tenant

1. Configure the Application Profile(s) under each user Tenant.
2. Under each user Tenant, configure Endpoint groups (EPGs) and associate each Endpoint group (EPG) with the relevant Bridge Domain (BD) which was configured (above) in step 3 (under Configure in Tenant Common)

 

Option 2 –BD in User tenant:

Create a VRF and L3Out in the common tenant.
Create bridge domains and endpoint groups in individual tenant spaces.

Static and/or Dynamic routing can be used on the L3out from release 1.1.3 and later. Only static peering was allowed before 1.1.3.

In this configuration, tenants share the same VRF and cannot have overlapping IP addresses. The bridge domain with the subnets are configured under the individual tenant spaces and are not visible to other tenants.

For Option 2, the tasks needed to implement are:

Configure in Tenant Common

1. Configure VRF  under Tenant common
2. Configure L3 outside connection under Tenant common and associate it with the VRF configured in step 1

Configure in User Tenant

1. Configure your Bridge Domains (BDs) and subnets under each user Tenant. Associate Bridge Domains (BDs) with the VRF in Tenant common and the L3 outside connection that you configured in (above) step 2
2. Configure the Application Profile(s) under each user Tenant.
3. Under each user Tenant, configure EPG and associate EPG with the BD configured in step 1

 

Option 3 – Inter-VRF Leaking with Shared L3out:

Create separate tenants with separate VRFs, bridge domains, and endpoint group. Each tenant has its own VRF and can use overlapping IP addresses. A contract is exported from the tenant that is providing the shared service (tenant common in our case). Route leaking between VRFs is performed to provide connectivity between the consumer and provider.

This option is supported from 1.2 onwards.

For Option 3, the tasks needed to implement are:

Configure in User Tenant

1. Configure your VRF, Bridge Domains (BDs) and subnets under each user Tenant. Associate Bridge Domains (BDs) with the VRF in User tenants.
   1. Ensure the BDs are marked as both “advertised externally and shared”
2. Configure the Application Profile(s) under each user Tenant.
3. Under each user Tenant, configure EPG and associate EPG with the BD configured in step 1
4. Consume the contract from Tenant Common (this can be done via a VzAny/vrf-level consume, via consuming the contract on a per-EPG basis, or by consuming a contract interface).

Configure in Tenant Common

1. Create a contract (in the common Tenant) for each Tenant. This contract should have a global scope.
2. Configure L3 outside connection under Tenant common and associate it with the VRF in tenant common.
3. On your L3EPG, provide the appropriate contracts

 

Below we will now configure L3Outs in common tenant for different options.

Option 1 –BD in Common tenant.
Using iBGP routing for a shared L3_Out

As seen below, this design will allow us to have multiple user Tenants share a common L3 Out, which is located in Tenant common. The main benefit for this design is that all Tenants on the fabric do not need a separate L3 out.

The Endpoint groups (EPGs) are configured inside of each of the respective Tenants, as usual. However, the Bridge Domains (BDs), the subnets, and the shared VRF are defined in Tenant common.

Potential drawbacks of this design include:

1. Shared VRF instance (Shared Routing table) – All Tenants not using their own, individual L3 Out would need to use a common routing table. This implies that all IP addresses for all Tenants would need to be unique and cannot overlap.

From a shared services perspective, the ACI Fabric will only advertise routes via BGP to external neighbors if the following conditions are met:

• The Bridge Domain (BD) and subnet must be configured in Tenant common and attached to a VRF in Tenant common
• Subnet must be marked as public (this makes it eligible to be advertised outside of the fabric)
• At least one interface on the fabric must be up and operational for the Endpoint group (EPG) which is associated with the Bridge Domain (BD).

Show ip bgp neighbor advertised command on Leaf-1 (Border Leaf connected to ASR)

Show ip bgp neighbor routes command on ASR (Connected to ACI Border Leaf)

Option 2 –BD in User tenant.
Using static routing for a shared L3_Out

As seen below, this design will allow us to have multiple tenants share a common L3 Out, which is located in Tenant common. The main benefit for this design is that all Tenants on the fabric do not need a separate L3 out.

The Endpoint groups (EPGs), Bridge domains (BDs), and subnets are configured inside of the respective user Tenant, as usual. However, the shared VRF is defined in Tenant common.

Some potential drawbacks of this design include:

1. Shared VRF instance (Shared Routing table) – All Tenants not using their own, individual L3 Out would need to use a common routing table. This implies that all IP addresses for all Tenants would need to be unique and cannot overlap.

Option 3 – Inter-VRF Leaking with Shared L3out.
(Using EIGRP routing for a shared L3Out)

For this option, all of our VRFs, BDs and EPGs will be placed in respective Tenants. In the below diagram, we can see two user tenants (Tenant1 and Tenant2) utilizing a shared L3Out from the Common tenant. The L3Out in Tenant Common is using the default VRF. While the VRFs in our user tenants (Tenant1 and Tenant2) do not have L3outs attached to them, they could (we just didn’t configure these to keep the post shorter 😉

The use-case for this type of design is multi-tenant fabric, with Tenants that are fully self-contained, but have the ability to share an external connection, such as access to the DC network, or internet routes. With this design, you could have X number of Tenants, in theory, and you would not need to configure an L3out for all of them; you could just share the L3out from Tenant Common (as long as IP addresses do not overlap).

In the diagram below, you will see that the VMs in each respective Tenants have the ability to communicate with hosts outside of the ACI fabric via the shared L3out. The 9.9.9.1 and 9.9.9.2 addresses represent networks learned from our shared, L3out.

 

In order to configure this, you will need to do the following:

Tenant1 has Bridge Domain ‘bd1’ configured with subnet gateway 101.1.1.1/24. This subnet needs to be enabled with the below options:

• Advertised Externally – to advertise these gateway subnets out to Shared L3Out to the internet
• Shared between VRFs – To leak the subnets to common tenant.
• NOTE – There is no associated L3out listed on the BD; when we use an Inter-vrf Shared L3out, we do not need to associate the user Tenant BDs with the L3out in Tenant Common. If you had a Tenant-specific L3out, it would still be associated to your BDs in your respective Tenants.

Tenant > Networking > Bridge Domains > YourBD > L3 Configurations

 

Tenant2 has Bridge Domain ‘bd1’ configured with subnet gateway 102.1.1.1/24. This subnet needs to be enabled with the below options:

• Advertised Externally – to advertise these gateway subnets out to Shared L3Out to the internet
• Shared between VRFs – To leak the subnets to common tenant.
• NOTE – There is no associated L3out listed on the BD; when we use an Inter-vrf Shared L3out, we do not need to associate the user Tenant BDs with the L3out in Tenant Common. If you had a Tenant-specific L3out, it would still be associated to your BDs in your respective Tenants.

Tenant > Networking > Bridge Domains > YourBD > L3 Configurations

 

Moving into our Common Tenant, we will need to define a Shared contract for our L3 services. This contract will need to have a scope of Global. Our contract, ‘SharedL3’ is providing the contract to allow any-to-any communication.

Tenant > Security Policies > Contracts > YourContract

 

Tenant > Networking > External Routed Networks > L3out > Networks > L3EPG

From our L3out, we will need to provide the SharedL3 contract that we previously created. This contract will be consumed in our user Tenants and is required to allow the routes to be leaked from our Common Tenant to the user Tenants.

 

Next, we will create an external EPG for your L3out. This EPG (or L3EPG) refers to external IP prefixes coming in on the L3out. In this example, we are allowing all subnets into the external EPG with 0.0.0.0/0 route as below.

The options used below are:

• External Subnets for the External EPG – this option tells the L3out to allow this subnet in the external EPG
• Shared Route Control Subnet – This indicates that this network, if learned from the outside through this VRF, can be leaked to the other VRFs (if they have a contract with this external EPG). It is used for route leaking to other tenants (Tenant1 and Tenant2 in this case). This subnet type is similar to ‘export route control’ with one exception: the Aggregate Shared Routes option applies to any subnet, not just the 0.0.0.0/0 subnet. For example, if you configure subnet 192.168.0.0/16 with the aggregate shared routes option, this matches the 192.168.0.0/16 subnet and all 192.168.0.0 subnets with longer prefix lengths. This is equivalent to configuring an IP prefix-list with the le 32 keyword (less than or equal to).
• Shared Security Import Subnet – sets the classifier for the subnets in the VRF where the routes are advertised. Shared security-import subnets are used with shared L3Out configuration, not used for routing control. This setting configures an ACL in the VRF that is consuming the shared L3Out.

Tenant > Networking > External Routed Networks > L3out > Networks > L3EPG

 

Next, from Tenant1, we will configure a VzAny contract which will consume the ‘SharedL3’ contract at the VRF level. In our case, for simplicity, we have configured vzany contract, but the same could be established by consuming regular contracts for EPG (since the provider is in common) or we could use ‘Consumed Contract Interface’ (if the provider is not in common tenant but rather is in another user created shared tenant).

 

 

Tenant2 will also be configured with VzAny consuming the ‘SharedL3’ contract from Tenant Common at the VRF level. In our case, for simplicity, we have configured vzany but the same could be established by consuming regular contracts for EPG (since the provider is in common) or we could use ‘Consumed Contract Interface’ (if the provider is not in common tenant but rather is in another user created shared tenant).

 

Verification:

Now that we have configured our Inter-VRF Shared L3out, we will perform a few pings from VMs inside of each respective Tenant to validate that the configuration is working.

A ping test from a VM in Tenant2 shows that it can reach networks behind the Shared L3out. The 9.9.9.0/24 network is only reachable via the Shared L3out from Tenant Common.

 

Below are the highlighted (RED) routes from the ACI Border leaves (the common:default VRF, the Tenant1 VRF, and the Tenant2 VRF. In addition, we will show the VRF specific routing tables on our external Nexus 7K.

• 0/0 (default route) is originating from Nexus 7K towards the ACI border leafs.
• 9.9.9.1/32 and 9.9.9.2/32 are loopbacks on the Nexus 7K that represent the external networks learned from the Shared L3out in Tenant Common
• 101.1.1.0/24 is subnet configured in BD Tenant1
• 102.1.1.0/24 is subnet configured in BD Tenant2

Border Leaf:

Leaf202# show ip route vrf common:default
IP Route Table for VRF “common:default”
‘*’ denotes best ucast next-hop
‘**’ denotes best mcast next-hop
‘[x/y]’ denotes [preference/metric]
‘%<string>’ in via output denotes VRF <string>

0.0.0.0/0, ubest/mbest: 2/0
*via 99.1.1.4, vlan7, [170/51456], 02:27:58, eigrp-default, external
*via 99.1.1.3, vlan7, [170/51456], 02:27:58, eigrp-default, external
9.9.9.1/32, ubest/mbest: 1/0
*via 99.1.1.3, vlan7, [90/128576], 02:27:58, eigrp-default, internal << External Routes
9.9.9.2/32, ubest/mbest: 1/0
*via 99.1.1.4, vlan7, [90/128576], 02:27:58, eigrp-default, internal << External Routes
99.1.1.0/24, ubest/mbest: 1/0, attached, direct
*via 99.1.1.2, vlan7, [1/0], 02:28:00, direct
99.1.1.2/32, ubest/mbest: 1/0, attached
*via 99.1.1.2, vlan7, [1/0], 02:28:00, local, local
101.1.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive << Tenant1 BD1 Subnet
*via 10.0.88.66%overlay-1, [1/0], 00:05:12, static
102.1.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive << Tenant2 BD1 Subnet
*via 10.0.88.66%overlay-1, [1/0], 00:24:03, static
109.109.109.0/24, ubest/mbest: 1/0
*via 99.1.1.4, vlan7, [90/3072], 02:27:58, eigrp-default, internal
201.1.1.1/32, ubest/mbest: 1/0
*via 10.0.8.93%overlay-1, [1/0], 02:27:45, bgp-65001, internal, tag 65001
202.1.1.1/32, ubest/mbest: 2/0, attached, direct
*via 202.1.1.1, lo3, [1/0], 02:28:00, local, local
*via 202.1.1.1, lo3, [1/0], 02:28:01, direct
Leaf202#

Tenant1:

Leaf202# show ip route vrf Tenant1:mainvrf
IP Route Table for VRF “Tenant1:mainvrf”
‘*’ denotes best ucast next-hop
‘**’ denotes best mcast next-hop
‘[x/y]’ denotes [preference/metric]
‘%<string>’ in via output denotes VRF <string>

0.0.0.0/0, ubest/mbest: 2/0
*via 99.1.1.3%common:default, vlan7, [20/51456], 00:05:20, bgp-65001, external, tag 65001
*via 99.1.1.4%common:default, vlan7, [20/51456], 00:05:20, bgp-65001, external, tag 65001
9.9.9.1/32, ubest/mbest: 1/0 << External Routes
*via 99.1.1.3%common:default, vlan7, [20/128576], 00:05:20, bgp-65001, external, tag 65001
9.9.9.2/32, ubest/mbest: 1/0 << External Routes
*via 99.1.1.4%common:default, vlan7, [20/128576], 00:05:20, bgp-65001, external, tag 65001
51.1.1.0/24, ubest/mbest: 1/0, attached, direct
*via 51.1.1.2, vlan8, [1/0], 04:26:07, direct
51.1.1.2/32, ubest/mbest: 1/0, attached
*via 51.1.1.2, vlan8, [1/0], 04:26:07, local, local
51.51.51.1/32, ubest/mbest: 1/0
*via 51.1.1.3, vlan8, [90/128576], 04:22:29, eigrp-default, internal
51.51.51.2/32, ubest/mbest: 1/0
*via 51.1.1.4, vlan8, [90/128576], 04:22:47, eigrp-default, internal
99.1.1.0/24, ubest/mbest: 1/0, attached, direct
*via 99.1.1.2%common:default, vlan7, [20/0], 00:05:20, bgp-65001, external, tag 65001
101.1.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.0.88.66%overlay-1, [1/0], 03:47:58, static
109.109.109.0/24, ubest/mbest: 1/0
*via 99.1.1.4%common:default, vlan7, [20/3072], 00:05:20, bgp-65001, external, tag 65001
201.1.1.1/32, ubest/mbest: 1/0
*via 10.0.8.93%overlay-1, [1/0], 04:26:06, bgp-65001, internal, tag 65001
202.1.1.1/32, ubest/mbest: 2/0, attached, direct
*via 202.1.1.1, lo4, [1/0], 04:26:07, local, local
*via 202.1.1.1, lo4, [1/0], 04:26:07, direct
Leaf202#

Tenant2:

Leaf202# show ip route vrf Tenant2:mainvrf
IP Route Table for VRF “Tenant2:mainvrf”
‘*’ denotes best ucast next-hop
‘**’ denotes best mcast next-hop
‘[x/y]’ denotes [preference/metric]
‘%<string>’ in via output denotes VRF <string>

0.0.0.0/0, ubest/mbest: 2/0
*via 99.1.1.3%common:default, vlan7, [20/51456], 00:24:17, bgp-65001, external, tag 65001
*via 99.1.1.4%common:default, vlan7, [20/51456], 00:24:17, bgp-65001, external, tag 65001
9.9.9.1/32, ubest/mbest: 1/0 << External Routes
*via 99.1.1.3%common:default, vlan7, [20/128576], 00:24:17, bgp-65001, external, tag 65001
9.9.9.2/32, ubest/mbest: 1/0 << External Routes
*via 99.1.1.4%common:default, vlan7, [20/128576], 00:24:17, bgp-65001, external, tag 65001
52.1.1.0/24, ubest/mbest: 1/0, attached, direct
*via 52.1.1.2, vlan9, [1/0], 04:16:07, direct
52.1.1.2/32, ubest/mbest: 1/0, attached
*via 52.1.1.2, vlan9, [1/0], 04:16:07, local, local
52.52.52.1/32, ubest/mbest: 1/0
*via 52.1.1.3, vlan9, [90/128576], 03:07:12, eigrp-default, internal
52.52.52.2/32, ubest/mbest: 1/0
*via 52.1.1.4, vlan9, [90/128576], 03:07:14, eigrp-default, internal
99.1.1.0/24, ubest/mbest: 1/0, attached, direct
*via 99.1.1.2%common:default, vlan7, [20/0], 00:24:17, bgp-65001, external, tag 65001
102.1.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.0.88.66%overlay-1, [1/0], 03:46:00, static
109.109.109.0/24, ubest/mbest: 1/0
*via 99.1.1.4%common:default, vlan7, [20/3072], 00:24:17, bgp-65001, external, tag 65001
201.1.1.1/32, ubest/mbest: 1/0
*via 10.0.8.93%overlay-1, [1/0], 04:16:06, bgp-65001, internal, tag 65001
202.1.1.1/32, ubest/mbest: 2/0, attached, direct
*via 202.1.1.1, lo5, [1/0], 04:16:07, local, local
*via 202.1.1.1, lo5, [1/0], 04:16:07, direct
Leaf202#

N7K:

LabCore02# show ip route vrf l3shared
IP Route Table for VRF “l3shared”
‘*’ denotes best ucast next-hop
‘**’ denotes best mcast next-hop
‘[x/y]’ denotes [preference/metric]
‘%<string>’ in via output denotes VRF <string>

9.9.9.1/32, ubest/mbest: 1/0
*via 99.1.1.3, Vlan99, [90/130816], 02:29:32, eigrp-99, internal
9.9.9.2/32, ubest/mbest: 2/0, attached
*via 9.9.9.2, Lo99, [0/0], 21:59:13, local
*via 9.9.9.2, Lo99, [0/0], 21:59:13, direct
99.1.1.0/24, ubest/mbest: 1/0, attached
*via 99.1.1.4, Vlan99, [0/0], 22:01:26, direct
99.1.1.4/32, ubest/mbest: 1/0, attached
*via 99.1.1.4, Vlan99, [0/0], 22:01:26, local
101.1.1.0/24, ubest/mbest: 2/0 << Tenant1 BD1
*via 99.1.1.1, Vlan99, [170/51456], 00:06:10, eigrp-99, external
*via 99.1.1.2, Vlan99, [170/51456], 00:06:10, eigrp-99, external
102.1.1.0/24, ubest/mbest: 2/0 << Tenant2 BD1
*via 99.1.1.1, Vlan99, [170/51456], 00:06:20, eigrp-99, external
*via 99.1.1.2, Vlan99, [170/51456], 00:06:18, eigrp-99, external
109.109.109.0/24, ubest/mbest: 1/0, attached
*via 109.109.109.1, Vlan999, [0/0], 02:42:46, direct
109.109.109.1/32, ubest/mbest: 1/0, attached
*via 109.109.109.1, Vlan999, [0/0], 02:42:46, local
201.1.1.1/32, ubest/mbest: 1/0
*via 99.1.1.1, Vlan99, [90/130816], 02:28:41, eigrp-99, internal
202.1.1.1/32, ubest/mbest: 1/0
*via 99.1.1.2, Vlan99, [90/130816], 02:28:55, eigrp-99, internal
LabCore02#