ACI with IPv6

This is a running list of ACI/IPV6 support notes that I will add to as I learn more items.  Most of the items here have been taken from Cisco ACI Infrastructure Fundamentals Release 5.1(x), Networking and Managemcnt Connectivity.  In addition I’ve also added items here that I’ve obtained by querying the field.

Supported:

• IPv4 only, IPv6 only, or dual stack configuration of in-band and out-of-band interfaces
• Neighbor Discovery using ICMPv6 messages known as router advertisements (RA) and router solicitations (RS), and Duplicate Address Detection (DAD)
• Stateless Address Auto configuration (SLAAC) and DHCPv6
• Bridge domain forwarding
• IPv4 only, IPv6 only, or dual stack configuration of in-band and out-of-band interfaces

Limitations:

• Multicast Listener Discovery (MLD) snooping is not supported
• For IPv6 management, only static addresses are permitted; dynamic IPv6 pools are not supported for IPv6 management
• IPv6 tunnel interfaces (Intra-Site Automatic Tunnel Addressing Protocol, 6to4 and so forth) are not supported within the fabric; IPv6 tunnel traffic run over the fabric is transparent to the fabric

Other Items to know:

General:

• An administrator can manually specify one or more complete 128-bit IPv6 global unicast addresses on an interface in compressed or uncompressed format. For example, the administration can specify the address in one of the following formats: ‘2001:0000:0000:0001:0000:0000:0000:0003’, ‘2001:0:0:1:0:0:0:3’, ‘2001:0:0:1::3’. In the ACI fabric naming property, an IPv6 address is always represented in the compressed format. In the above example, the Relative Name is: 2001:0:0:1::3. The administrator can choose any mask length as appropriate for the address
• An administrator can also specify an ACI fabric IPv6 global unicast address in EUI-64 format. As specified in RFC2373, Extended Unique Identifier (EUI) enables a host to assign itself a unique 64-bit IPv6 interface identifier (EUI-64). The IPv6 EUI-64 format address is obtained by incorporating the switch MAC address within the 128-bit IPv6 global unicast address. This feature of IPv6 eliminates the need for manual configuration or DHCP.
• An IPv6 address for a bridge domain or Layer 3 interface specified in the EUI-64 format is formed this way: <IPv6 prefix>::/<mask>/eui64 where the mask is <=64. For example, 2002::/64/eui64 is what the administrator specifies, and the switch assigns the address as 2002::222:bdff:fef8:19ff/64. The switch uses the switch MAC address to create the EUI-64 address. The formed IPv6 address is contained in the operAddr field of the ipv6If object.
• The EUI-64 format can only be used for pervasive bridge domain and Layer 3 interface addresses. It cannot be used for other IP fields in the fabric such as an external server address or for DHCP relay
• Bridge domain subnets and Layer 3 external interface IP addresses can be IPv6 global addresses with a mask ranging from /1 to /127. A bridge domain can contain multiple IPv4 and IPv6 subnets. To support IPv4 and IPv6 address on the same L3 external interface, the administrator creates multiple interface profiles. When an EPG or external EpP gets deployed on the switch, the presence of a manually configured link-local address for the equivalent bridge domain/L3 Interface or an IPv6 address for the subnet/address field results in the creation of ipv6If interface in the switch.

Link Local Address:

• One Link-Local Address (LLA) can be assigned to an interface. The LLA can be autogenerated or configured by an administrator. By default, an ACI LLA is autogenerated by the switch in EUI-64 format. An administrator must configure at least one global address on the interface for an autogenerated LLA to be generated on the switch. The autogenerated address is saved in the operllAddr field of the ipv6If MO. For pervasive SVIs the MAC address used is the same as the configured interface MAC address. For other kinds of interfaces the switch MAC address is used. An administrator has the option to manually specify a complete 128-bit IPv6 link-local address on an interface in compressed or uncompressed format
• The switch hardware tables are limited to one LLA per Virtual Routing and Forwarding (VRF) instance.
• Each pervasive bridge domain can have a single IPv6 LLA. This LLA can be set by an administrator, or can be automatically configured by the switch when one isn’t provided. When automatically configured, the switch forms the LLA in the modified EUI-64 format where the MAC address is encoded in the IPv6 address to form a unique address. A pervasive bridge domain uses one LLA on all the leaf nodes
• For external SVI and VPC members, the LLA is unique for every leaf node
• LLAs can be changed to manual (non-zero manually specified link-local addresses) or auto (by manually setting the specified link-local address to zero) anytime in the lifecycle of the interface
• LLAs specified by an administrator must conform to the IPv6 link-local format (FE80:/10)
• The IPv6 interface MO (ipv6If) is created on the switch upon the creation of the first global address on the interface, or when an administrator manually configures an LLA, whichever happens first
• An administrator-specified LLA is represented in the llAddr property in the bridge domain and Layer 3 interface objects in the logical model.
• The LLA used by the switch (either from llAddr or autogenerated when llAddr is zero is represented in the operLlAddr property in the corresponding ipv6If object
• Operational LLA-related errors like duplicate LLAs are detected by the switch during Duplicate Address Detection process and recorded in operStQual field in the ipv6If object or raise faults as appropriate
• Apart from the llAddr fields, an LLA (FE80:/10) cannot be a valid address in any other IP address field in the APIC (such as external server addresses or bridge domain subnets) as these addresses cannot be routed

Static Routes:

• Local Routes: Any /128 address configured on an interface leads to a local route that points to the CPU
• Direct routes: For any configured address on a pervasive BD, the policy element pushes a subnet route pointing to an IPv4 proxy tunnel destination on the spine. For any configured address on a non-pervasive Layer 3 external interface, the IPv6 manager module automatically pushes a subnet route pointing to the CPU
• Static routes pushed from PE: Used for external connectivity. The next hop IPv6 address for such routes can be on a directly connected subnet on the external router or a recursive next hop that can be resolved to a real next hop on a directly connected subnet. Note that the interface model does not allow an interface as a next hop (though it is supported in the switch). Used to enable shared services across tenants, the next hop for shared-services static routes is located in the shared services Virtual Routing and Forwarding (VRF) instance, which is different from the tenant VRF, where the route is installed on the ingress leaf switches.

•  

Neighbor Discovery:

• ND-specific Neighbor Solicitation or Neighbor Advertisement (NS or NA) and Router Solicitation or Router Advertisement (RS or RA) packet types are supported on all ACI fabric Layer 3 interfaces, including physical, Layer 3 sub interface, and SVI (external and pervasive). Up to APIC release 3.1(1x), RS/RA packets are used for auto configuration for all Layer 3 interfaces but are only configurable for pervasive SVIs.
• Starting with APIC release 3.1(2x), RS/RA packets are used for auto configuration and are configurable on Layer 3 interfaces including routed interface, Layer 3 sub interface, and SVI (external and pervasive).
• ACI bridge domain ND always operates in flood mode; unicast mode is not supported.

The ACI fabric ND support includes the following:

• Interface policies (nd:IfPol) control ND timers and behavior for NS/NA messages.
• ND prefix policies (nd:PfxPol) control RA messages.
• Configuration of IPv6 subnets for ND (fv:Subnet).
• ND interface policies for external networks.
• Configurable ND subnets for external networks, and arbitrary subnet configurations for pervasive bridge domains are not supported.

Configuration options include the following:

• Adjacencies
  • Configurable Static Adjacencies: (<vrf, L3Iface, ipv6 address> –> mac address)
  • Dynamic Adjacencies: Learned via exchange of NS/NA packets
• Per Interface
  • Control of ND packets (NS/NA)
  • Neighbor Solicitation Interval
  • Neighbor Solicitation Retry count
• Control of RA packets
  • Suppress RA
  • Suppress RA MTU
  • RA Interval, RA Interval minimum, Retransmit time
• Per Prefix (advertised in RAs) control Lifetime, preferred lifetime
  • Prefix Control (auto configuration, on link)
• DAD (Duplicate Address Detection) States:
  • NONE—This is the state when the address is initially created before attempting the DAD.
  • VALID—This is the state that represents the address has successfully passed the DAD process without detecting the address as a duplicate address.
  • DUP—This is the state that represents the address is found as duplicate on the link.
• Support for SLAAC (Stateless Address Autoconfiguration ) and DHCPv6
  • SLAAC only
  • DHCPv6 only
    • SLAAC and DHCPv6 stateless used together use SLAAC for address configuration only, but uses DHCPv6 for DNS resolution and other functions.

Other Items Learnt from the field:

• Flooding is required in a MSO stretched BD for IPv6 (BD stretch but no flooding did not work for IPv6, even if IPv4 was fine)
• DHCP Relay – for the DHCPv6 Relay to work we had to create an ND Interface Policy with enabled “managed config” in Controller State and bind it to the BDs (Tenants -> <tenant>  -> Networking -> Bridge Domains -> <bd> -> Policy -> L3 Configurations -> ND Policy)
• COPP:  the ipv6 support is similar to what ipv4 is supported, ie if it supports ipv4 it support ipv6. Nothing special or unusual about ACI. All the caveats and requirements for ipv6 stay true when enabling it in the fabric:
  • Ex: use a 32 but router-id for BGPv6, disable strict adjacency check for isis
• 0.0.0.0/0 and ::/0 have the same PCTag values.  If customer needs to have different behavior for IPv4 deault route (EXT EPG) and IPv6 Default (EXT EPG), then they need to use EtherType in filters to drive different behavior
• ERSPAN over IPv6 is not supported
• Floating L3Out with VMM domain does not support dual stack
• BGP IPv6 address-family is not supported over v4 sssions and vice versa.  Customer needs to configure two separate BGP sessions.
• L3 Outs:  Static, BGP, OSPFv3. (SVI’s and Sub-Interfaces)
• IPv6 Prefixes are learnt from Border Leaves to User Leaves by bgp address-family VPNv6.   IPv6 on underlay is not supported. 
• Depending on your needs, you may want to adjust the Forwarding Scale Profiles.  Please also see the article I had written previously on aci tcam utilization.  

References:

Cisco ACI Infrastructure Fundamentals Release 5.1(x), Networking and Managemcnt Connectivity