Updates:
01/21/2022: Added Notes on Multisite and Multipod parameters. Also, how to look at APIC initial setup parameters from ssh to APIC.
This is a rather old topic, relevant only for day 0 setup of ACI. However, now that a lot of new customers are implementing ACI, this topic is again coming up and I’ve had a number of folks reach out to me to discuss this. For that reason, I decided to do a short write-up on this.
When you initially setup your fabric, one of the parameters you need to put in for the initial setup is the TEP Pool subnet. Below is a screenshot of the initial APIC setup screen. (this screenshot is a bit old but the concept is still the same. I’m just copying this screenshot from a much older presentation that I used to use during teaching ACI classes internally)
IPs from TEP Pools are used internally in ACI for many purposes. As an example, if you configure a VPC domain between 2 leaves you will notice that a virtual IP is assigned to that VPC leaf pair. The virtual IP will be assigned a /32 host IP from that TEP Pool.
When ACI needs an internal IP for any purpose, it takes a /27 range from the TEP Pool and assigns an IP from there. A /27 block gives you 32 host IPs. If ACI needs another IP for some other purpose, ACI will not assign an IP from the previous /27 pool (even though it might have been hardly utilized), but create a new /27 range and take an IP from there. In other words the assigned /27 ranges are only used for similar functions. This is all done internally by ACI and you don’t have to think about it or do anything special to make it work.
So, the 2 questions that often comes up are:
a) How big should my TEP Pool be ?
b) Does the TEP Pool that I assign have to be unique ?
Let’s discuss each of those questions and answer them here.
a) How big should my TEP Pool be ?
The default TEP value during APIC setup is 10.0.0.0/16. Since it’s broken up internally in /27 pools, we should get 227-16 /27 subnets, i.e. 2,048 /27 pools. That seems like a lot and it is, but then you need to make sure you position this properly depending on how big you want your fabric to grow in the future. Remember that the TEP Pool is a day 0 setup parameter and cannot be changed later in a non disruptive way. You could always backup your fabric, blow it off, change it and then restore the fabric back, but in a production network that is not practical or possible. So, the question becomes does my TEP Pool need to be unique, if not, I don’t care, it’s only used internally in ACI, why should I bother. We’ll tackle that question next, but first let me get you a ACI TEP Pool calculator that you can use to size your fabric TEP Pool usage. It’s basically just a excel sheet and you fill in the values in the RED row. This gives you an approximate value on how big your TEP Pool should be. Looking at the bottom example, you will see that If the fabric had 200 leaves, 4 spines and 5 APICs with VPC ports on the 48 port leaf front panel ports, you would need to have a /17 TEP Pool.
b) Does the TEP Pool that I assign have to be unique ?
If you had multiple Pods in a ACI Fabric, (multipod), each pod will require a unique TEP Pool. If you had different Fabrics even if they were tied together with MultiSite Orchestrator as Multisite ACI Fabrics, then you could get away using the same TEP pool on the different ACI Fabrics. It used to be that before ACI version 4.1.2, if you were also implementing Remote Leaf, the TEP Pool would have to be advertised out for that Remote Leaf connectivity, If 2 Fabrics had the same TEP pool, then that would obviously cause a problem. After 4.1.2, the Routable TEP pool feature came out which makes this a non issue.
The other item to think about is “what about my TEP Pool overlapping with IPs that are already used in the enterprise ?”
Generally the best approach is to have a unique TEP Pool that is not overlapping (being used in the enterprise). Ofcourse, in practicality this is often not possible. Enterprises often have RFC 1918 space heavily utilized. One way to get around this is to use RFC 6598 space (100.64/10) that Service Providers use internally for Carrier Grade NAT. Ofcourse if you are a Service Provider, you probably won’t do that either. So, then the question comes up, what if I overlap my internal TEP Pool in ACI with IP Pool I’m already using in my network. For the most part this should be o.k. with one caveat. You will not be able to reach the APIC OOB from a host that belongs to the same subnet as the subnet that is defined for the ACI TEP pool. That is because the packet will reach the APIC but then get routed internally to the Fabric instead of going back to the host where the packet initiated from. So, if you had MSO or a NMC where you do Network Management/Monitoring from then the IP subnet of that NMC cannot overlap with the IP subnet assigned to the Internal ACI TEP Pool. Below diagram will make it clear why this is the case.
The key point to look in the above diagram is that:
- NMC is at 10.0.1.0/24 which overlaps with ACI TEP Pool of 10.0.0.0/16
- Packet from NMC host reaches APIC OOB of 10.29.198.168
- On the return trip, APIC has a route for 10.0.0.0 pointing to 10.0.0.30 on bond0.3967 which is internal leaf Anycast, so packet goes there instead of going back to originating host
📗Note: Added 01/21/2022:
Parameters for MultiSite and Multipod:
Multisite:
Multipod:
🦠You can quickly look at the parameters of the APIC you configured by ssh’ing to your APIC and then doing:
cat /data/data_admin/sam_exported.config as shown in the screenshot below:
