Do you have a need to migrate a Policy Based Routing policy to ACI? Have a need to redirect a subset of traffic to a specific route? The ACI way of ‘Policy Based Routing’ is to use a service graph. Service graphs have had a varied history with regard to ACI, and some have written them off as being too complicated. Introduced in 2.2, we are now able to utilize ‘unmanaged’ service graphing.

Unmanaged service graphs have fewer dependencies than their original managed mode counter parts. They also ease the creation of ACI policy to accommodate L4-7 service devices such as Firewalls and Load-balancers. Rather than manually stitching ACI objects (EPGs and BDs) together, a service graph will be inserted at the contract level.

Service Graphing is also the only way in to configure Policy Based Redirection and effectively Policy Based Routing. 

Note: This article will be updated to include different topologies and redundancy considerations. In this state, it is meant as a jumping off point to get started. We would recommend to test this configuration in the lab before implementing in production.

Scenario

We have a requirement to route traffic from a specific EPG(s) to a specific L3 External destination while keeping the rest the traffic from other EPGs using the VRF’s default (or less specific) route. We want the rest of the specific EPG’s traffic to take the default GW as well.

Adding Static routes (or any routes), in this case, would introduce more preferred routes than the default and thus point all traffic toward the specific path for this destination (and likely blocked by policy), which we don’t want to do. This type of scenario is useful for bypassing firewalls for things like storage or replication traffic.

Prerequisites

We’ll assume there is already at least (1) L3 External, and at least (2) Endpoints in different EPGs to test traffic with. These objects are shown in the diagram below.

We’ll create a second L3 External, an unmanaged L4-7 Device, a Service Graph template, and then apply the SG template which will create a few more objects. We’ll end up with something like the next diagram.

Now, before we really dive deep, the diagram above is just a logical, policy-centric view of the end state.

Other points to keep in mind: this L3 External can have some regular, more specific routes in addition to our special destination that we don’t want in the routing table. This second L3 External is relatively flexible in terms of what you can configure on it. Our specific destination, that we don’t want in the routing table, will be 222.222.222.222, a nice, easy to notice destination. I have configured a /32, but this can be any size network.

Getting Started

The first step we will take is to create the second L3 External. I won’t go into creating an L3 External from scratch as that is covered in depth in many other places. Instead, we will focus on the specific L3 External EPGs that are needed.

1. Create the New L3 External and L3 EPGs

For our purposes we will need (2) L3-EPGs: One for the route control for our connected router/device along with any normal/general EPGs reachable via this L3 External, and a second for the Specific destination we want to forward traffic to without appearing in the routing table. Static routing will be used here, but there is no reason a dynamic protocol couldn’t be used instead.

The Regular-Destination L3-EPG (below) will be used for route control for connecting to our external device.

The Special-Destination L3-EPG (below) will be used to policy-based route the traffic without appearing in the routing table.

The Special-Destination subnet is only configured as ‘External Subnet for the External EPG’. This means that it will be used only for policy matching for traffic instead of route control.

2. EPGs and Contracts

For the EPGs and Contracts, we have (2) EPGs, each with it’s own BD and subnet. Each EPG has contracts to the default (0.0.0.0/0) L3-EPG of the COAST L3 External. ‘epg1’ and ‘epg2’ will also have a contract to the Regular-Destination L3 EPGs in our Specific L3 External while ‘epg1’ will have a contract to our Special-Destination L3 EPG. This is reflected in the final state diagram from above.

The Service Graph

With the basics out of the way, we can dig into the polices and objects of the service graph.

There are many parts of the service graph: the device, the graph template, the device selection policy, the PBR redirect policy, the contract, and finally the connectivity in the form of a BD and/or EPG. We’ll go through each of these as we build out the service graph.

3. The L4-7 device

The device defines a few things about the actual device we will be using as a ‘service’ device. Typically this would be a Firewall or Load-balancer or something similar, but both of those will act similar to a Router, which we are going to be using in this example. The device specifies where our external device connects to the fabric (which leafs and interfaces), how these interfaces should be labeled or abstracted for use in the service graph (inside and outside or other such similar concepts), and how the fabric should expect traffic to flow through the device (routed or transparent).

3.a To start creating a new L4-7 device, follow the screenshot above then right click the Devices folder and click ‘Create L4-7 Devices’.

The first thing we do when creating a new device, in this instance, is uncheck the ‘Managed’ check box. This removes a bunch of options that are required when using device packages and managers and makes configuring the L4-7 device more straight forward. Another thing to note here is that in the context of ACI a ‘Device’ is a logical device or cluster, so when we reference the L4-7 Device, this is the logical pair or group of devices that are acting to provide the service.

3b. The next step is to fill in the Name. The name should be something a little more descriptive of what the device is. Since my lab doesn’t have many devices I can be generic with ‘PBR_DEV’, but the hostname of the actual device or if using the same device for an L3 Ext, the name of the L3 Ext might be better.

We’ll cover the next few options and what we’re doing and what’s possible with them (more can be found here.)

• Service Type: in our case the service type does not matter, but this selection will merely limit what and where the device is shown in later Service Graph template options.

• Device Type: we are choosing physical because our device is attached through a physical domain and not a VMM domain. If the device was on a virtual domain, we would select the VMM domain and then the VM of the Service device on the domain. This helps validate the interface selections and then determines where VLANs will get programmed once the graph gets applied.

• Promiscuous Mode: Promiscuous mode is only used for Virtual Devices in specific situations. We’ll leave it unchecked.

• Context Aware: We choose single context, but in unmanaged mode, this option does not matter. In managed mode, multiple contexts can be used when sharing a single concrete device across tenants and in other ways depending on the device package and device.

• Function Type: we select GoTo because we are sending traffic to a next hop that isn’t the fabric, this can also be described as ‘Routed Mode’. The fabric will not expect the traffic to come directly back to it’s own interfaces and may need some sort of forwarding information.

3c. Under Devices, we’ll add (1) device with (2) interfaces. In this case, the interfaces actually go to (2) separate physical devices. This configuration allows us to Load-balance and achieve Redundancy between the (2) devices/interfaces and which are configured similarly to an Active/Active service because, in this case, both devices will have the route(s) to the Special-Destination. We can configure (2) separate logical devices, but this results only in more configuration rather than reaping any particular benefit.

3d. In this section, we also configure where the (2) interfaces connect into the fabric and give them labels or names that are referenced in the last part of the modal menu.

These are the interfaces we’ll use later on when applying the service graph template. This abstracts any of the interface we configured above to a single configuration item which I have labeled ‘1arm’.

The interfaces are grouped per function. Because this is a 1-armed approach and both interfaces lead to the same function on our external device, they are put under the same cluster interface. We then configure the encapsulation that will be used on those interfaces. This VLAN should be present in the physical domain chosen.

3e. Once everything is filled out, we can hit the finish button and go to the next step. Your Navigation menu should look similar to the screenshot below, indicating all the objects were created properly.

Typically, at this stage, there should not be any faults.

4. Service BD and PBR Policy

Before we continue with the L4-7 polices we will need to create a Bridge Domain for the L4-7 devices to connect into and the Service Graph to use as well as give the fabric information on how to forward traffic to the devices.

4a. Creating the BD is simple, it simply requires one setting changed to accommodate redirection. Create a BD like normal, I’ve named mine ‘pbr’;

• Uncheck Endpoint Dataplane Learning
• Add a Subnet for communication with the external device

That’s it for the BD. *Note: DP learning is automatically disabled for BDs when the Service Graph gets initiated since version 3.1.

4b. For the PBR policy, we need the MAC or Virtual MAC and the IP Address(es) that we will be forwarding traffic to. We configure that under Policies > Protocol > L4-7 Policy-Based Redirect.

4c. Create a new Redirect Policy, add a name, and leave the rest of the settings at default for now, add the MAC and IPs for the devices that ACI will forward traffic to. We’ll cover what the other options are in the redirect policy in a future update, but they are not needed for our deployment in a single Pod at this time.

This will be used a bit later.

5. Create Service Graph Template and PBR

After the L4-7 device, BD, and PBR policies are created, we can go ahead with creating the Service Graph Template. This step is defining a template. In some ways, it’s similar to defining an Access Policy Group: it defines a set of properties that are applied together, but they are not applied to a physical device/node interface until the policy is specified by an Interface Selector and then applied to a Switch. The Service Graph template defines which traffic goes to which interface on the selected devices and in which direction (provider to consumer or vice versa) policies and actions will be applied.

5a. Navigate to Services > L4-7 > Service Graph Templates and Right click to create a new template.

Give the template a name, then drag the device/function that we created in the first step to the right. Select ‘Routed’ and the click the ‘Route Redirect’ checkbox.

5b. After creating the Service Graph Template, it can be applied. The easiest way to create all the associated policies is to use the ‘Apply Service Graph Template’ wizard by right clicking the Service Graph Template we just finished creating.

In the Wizard, we select the consumer side and the provider side EPGs. It does not matter much whether the EPG or L3 Ext EPG is consumer or provider, but it will have an implication on which side of the graph we apply the PBR policy.

5c. We’re going to just create a new contract, but you can also choose an existing contract if you have one configured. Specific ports can also be chosen, but I don’t want to filter anything at this time. Clicking next will bring you to the Graph configuration screen.

This part of the configuration deals with specific policies and actions for each side of the graph. Since in the last screen we used the EPG as the Consumer, we want to redirect traffic going from EPG-1 to the Special Destination, we will place the redirect policy on that connector.

For the Provider connector, once we redirect the traffic to the device, we will use a different path for the return traffic back into the fabric, we do not need to configure anything here and select some place holder values (in this case the same BD as the consumer side).

5d. For both Consumer and Provider connectors, select the PBR bd we created earlier. For the Cluster Interfaces, select 1arm.

5e. For only the direction where our internal traffic is coming from, select the Redirect Policy that we created earlier. In our case the Redirect Policy will be on the Consumer side.

5f. After ensuring all of the options are selected, go ahead and hit finish. This wizard will create and apply the contract as well as a device selection policy. At this point, if all the configuration items were correct, traffic should start flowing from your EPG to the Special destination without any routes for the Special Destination in the routing table.