NDO/CNC Fabric Integration with AWS GWLB for protected apps -

Introduction
High Level Architecture for GWLB
2.a.Routing Table Discussion
2.b. Packet Flow from Internet to Protected APP
2.c. Packet Flow: East/West to Protected APP
Integration of GWLB topology with NDO/CNC Fabric
3.a. Discussion on Importing App VPC into CNC
3.b. Packet Flow from outside VPC to Protected APP
3.c. Routing across TGW for Protected Apps Example
Tips for Configuration Section
4.a. Edge Routing for IGW
4.aa. GWLB Cross Zone Load Balancing and MTU
4.b.Target Group for GWLB
4.c. Firewall Related Tips
4.d. Creating GWLBe
4.e. Make sure to import into CNC with read only
4.f. Sticky configuration in GWLB and Appliance Mode in TGW
References

Introduction

Referencing AWS post, GWLB (Gateway Load Balancer) was introuduced in AWS towards the later part of 2020. AWS Gateway Load Balancer (GWLB),is a service that makes it easy and cost-effective to deploy, scale and manage the availability of third-party virtual appliances such as firewalls, intrusion detection and prevention systems and deep packet inspection systems in the cloud.

In essence, GWLB makes it very easy to deploy protected apps that require both North/South and East/West traffic going back and forth through a firewall.

In this article I will discuss the architecture for GWLB followed by how to integrate GWLB with NDO/CNC Fabric.

High Level Architecture for GWLB

A basic high level architeture that can be deployed for GWLB is depicted below.
📙Keep in mind that GWLB architecture can be integrated with AWS Transit Gateways for added resiliency. Please see reference section for links that describe these architectures.

Figure 1: Basic High Level Architecture for GWLB

An explanation of the basic architecture:

Firewalls are deployed in a subnet (or multiple subnets) in a NetworkAppliance VPC. The Firewalls don’t necessarily have to be clustered. The Firewalls could also have added functionality like IPS/IDS (item 1 in figure)
The intent is to redirect all flows, North/South and East West for protected apps through the firewalls. (item 2 in figure)
A target group is created pointing to the firewalls, a GWLB is spun up in a GWLB subnet and associated with that target group. (item 3 in figure).
Once the firewall configuration is completed, a geneve tunnel will be established between the GWLB and the Firewalls. All Data traffic for protected app will traverse the geneve tunnel. There are plenty of 3rd party firewall vendors that support firewalls with geneve tunnel feature. (item 4 in figure)
A GWLB Service is created in the GWLB subnet. The service is associated with the GWLB (item 5 in figure)
A GWLB Interface VPC endpoint is created in the app VPC in a GWLBe subnet. This endpoint is associated with the GWLB Service in the NetworkAppliance VPC. This action creates a AWS Private Link from the GWLBe to the GWLB Service which in turn has already been associated with the GWLB (item 6 in figure)

⚠️ Keep in mind that the NetworkAppliance VPC and the APP-VPC needs to be in the same AWS Region (us-east-1 in the figure). Also the GWLBe and the GWLB Service needs to be in the same AWS Region/Zone (us-east-1a in figure).

Routing Table Discussion

Each subnet in the NetworkAppliance VPC and APP VPC should be configured with custom routing table for maximum control. Do not use the same default VPC routing table for the subnets.
Please reference the figure below for the discussion of routing tables configured.

Figure 2: Discussion of Routing Tables.

In the Network Appliance VPC:

The Firewall Jumphost needs Internet Access (based on the design shown in the figure).
Firewalls and GWLB don’t need public access. For that reason the Firewall Jumphost subnet has a default route pointig to the IGW.

In the App VPC:

App1 Jump Host has a route pointing to the IGW.
App1 Jump Host does not have any route pointing to GWLBe (Global Load Balancer Endpoint) as we don’t want this traffic to go through the firewall.
The Protected APP App1 is in a Public Subnet, since it needs to reach the internet for patching. Also, it needs to be accessed from the internet (maybe https). However notice that the default route is pointed to the GWLBe (Gateway Load Balancer Endpoint denoted by vpce-xxxxx).
The Protected APP App1 also has a route for APP2-subnet pointing to the GWLBe
Also, notice that APP2-subnet is a private subnet (which resides in the APP VPC which can be in any zone) has a route for APP1 subnet pointed to the Next Hop of GWLBe.
The IGW is configured with ingress routing (also known as edge routing). The edge routing rule says that for destination 10.0.20.0/24 (Protected APP Subnet), the Next Hop is the GWLBe. ( In the Tips for Configuration Section, I will show you how to do this)

Packet Flow from Internet to Protected App

Discussion of Packet Flow From Internet to Protected APP
Please reference the figure below for a discussion of packet flow from Internet to Protected App.

Figure 3: Packet Flow from Internet to Protected APP

Packet comes from Internet destined to the App. The IGW route Table has a more specific route for the APP Subnet and directs it to the GWLBe
Once Packet reaches the GWLBe, it is forwarded to the GWLB Service on the NetworkAppliace VPC through the AWS Private Link.
The GWLB Service is associated with the GWLB and the packet goes to the GWLB and then gets forwarded to the Firewall through the geneve tunnel
The Firewall applies it’s rules and if allowed, the packet is sent back to the GWLB
Once packet reaches GWLB, it goes through the GWLB Service to ghe GWLBe through the AWS Private Link.
The GWLBe subnet has a local route for the APP VPC (in this case 10.0.0.0/16), so packet gets forwarded to the App.

Discussion of Packet Flow from Protected APP to Internet
Please reference the figure below for a discussion of packet flow from Protected APP to Internet.

Figure 4: Return Packet to Internet

App1 Route Table has a default route pointing to GWLBe, so the packet goes to GWLBe
The packet is shunted to GWLB Service through the Private Link. The GWLB service is tied to the GWLB so, the packet reaches the GWLB.
GWLB forwards the packet to the firewall through the geneve tunnel
The Firewall applies it’s rules and forwards the packet back to GWLB
The packet then goes through the private link back to GWLBe
GWLBe has a default route to the internet, so packet goes through IGW back to source in Internet

Below figure shows the results of the configuration.

I am sitting on a Ubuntu Jump Server which is behind my home network. I curl ip.me to find my public IP assigned at this time.
I then curl to the Public IP of APP1 and it is successful.
I ssh to the Public IP (ssh is allowed by the firewall rules in my example).
I check on the Firewall for connections and observe that the flow did go through the firewall.

Figure 5: Example of sucessful N/S traffic Flow

Packet Flow: East/West to Protected APP

Using the same logic as above, you can reason out how East/West Redirect works. Please reference the figures below for this.

The Figure below shows Packet Flow from App2 to App1 (Protected App)

Figure 6: Packet Flow from App2 to App1 (Protected App)

The figure below shows Return Packet from App1 (Protected App) to App2

Figure 7: Packet Flow from App1 (Protected App) to App2

Below figure shows the results of the configuration.

Figure 8: Example of sucessful E/W traffic Flow

Integration of GWLB topology with NDO/CNC Fabric

Once the GWLB topology is setup in AWS, integrating it with NDO/CNC fabric can be done in 2 easy steps.

Import Brownfield APP VPC into CNC. Please use Read Only Mode for this import.
On NDO Import VRF from CNC
📙 Please reference the previous writeups on how to do these imports.

Discussion on Importing App VPC into CNC

When Importing APP VPC into CNC, it is important to use a separate TGW Gateway Subnet for the TGW Attachment. This is best practice and gives you granular control for routing configuration which is needed in this solution.
Make sure to import the VPC into CNC with Read Only. CAPIC fully managed topology creates a universal route per VPC and you don’t want to do this. You also don’t want 2 points of control for Routing and Security Policies. Access Control and routing control for the Protected APP will need to be done from AWS Console.
There is no need to import the NetworkAppliance VPC (though you can). Treat this as a blackbox which is in control of the security group.
Once integrated you should be able to reach your Protected App from anywhere (Internet, CNC Fabric VPC/Vnet in same/otgher clouds, onPrem)

Packet Flow from outside VPC to Protected APP

There are 3 things that need to be done from a routing prospective for this to work.

Create a subnet in APP VPC in same zone as APP. Use this Subnet for the TGW attachment Subnet for that zone. In this subnet create a route which says that Next Hop to reach the APP is the GWLBe. Also, put a route for the greenfield subnet pointing to the infra tgw
In the GWLBe subnet put a route for the Source (greenfield subnet) pointing to Infra TGW
In the APP Subnet put a route for the Source (greenfield subnet) pointing to the GWLBe (we just put a 0/0 pointing to GWLBe instead)

The figure below shows the Packet Flow from Greenfield VPC to Protected APP in Brownfield VPC.

Figure 9: Packet Flow from Greenfield VPC to Protected APP in Brownfield VPC

Packet from Greenfield (destined to Protected APP1) comes from Infra TGW through attachment into the Gateway Subnet
Since Gateway subnet has a more specific route point to GWLBe the packet goes to GWLBe
Packet gets shunted to GWLB Service/GWLB through the Private Link
Packet from GWLB goes through geneve tunnel to Firewall.
Firewall inspects the rules and if allowed, sends the packet back to GWLB
Packet reaches the GWlBe through the Private Link
Since GWLBe subnet has a local route for VPC, it knows how to get to APP1 (Protected APP)

The figure below shows the Return Packet Flow from Protected APP in Brownfield VPC to Greenfield VPC.

Figure 10: Return Packet Flow from Protected APP in Brownfield VPC to Greenfield VPC.

Return packet from Protected App1 goes to the GWLBe, because of the static route in the App1 subnet
GWLBe forwards the packet to GWLB Service/GWLB through the AWS Private Link
GWLB forwards the packet to firewall through the geneve tunnel
Firewasll inspects the traffic and if allowed, sends it back to GWLB
GWLB Service sends the packet back to GWLBe in APP VPC
Once packet reaches the GWLBe, the packet is forwarded to the source through the Infra TGW (because of the static route in GWLB subnet for source pointing to Infra Transit Gateway)

Below figure shows that Packet from greenfield VPC reaches the Protected APP1 through the firewall.

Figure 11: greenfield VPC reaches the Protected APP1 through the firewall.

📙 Adding this Routing across TGW for Protected Apps example section on 2/13/2023.**

Routing across TGW for Protected Apps Example

As you might have gathered by now, the important thing to keep in mind for packet flow to work correctly in this scenario, you have to configure the routing tables correctly in the AWS Subnets. It’s not really material that you are using CNC based AWS Fabric. Just make sure that you have double-checked your routing tables correctly and you should be good to go.

Please look at an example below, where there is no CNC Fabric. We will just use AWS configurations to show how the routing tables should be configured.

Figure 11a: An example of how to configure routing tables

In the figure above, note the following:

There are 2 VPCs, Consumer VPC and App-VPC
The ec2 with ip of 10.12.12.12/28 wants to communicate with the Protected APP of 10.0.20.239/24
The requirement is that any communication to the protected app needs to be redirected through the firewalls
Consumer APP Subnet is in us-east-1b zone
Protected APP Subnet is also in us-east-1b zone
GWLBe-subnet (gwlb endpoint) is in us-east-1a zone
Traffic from Consumer APP to Protected APP needs to go through a TGW.
TGW has attachments to every zone in every VPC in /28 subnets

Please study the routing tables for each subnet as shown in the figure.

Verifying the flow.
To Verify that traffic is going through the firewall, we can ping from Consumer APP to Protected APP and capture packets on the ASA firewall as shown below.

Figure 11b: Pinging from Consumer EC2 to Protected APP

Figure 11c: Setting up access-list to capture packets

Figure 11d: Capturing packets on ASA proves that the packets are getting redirected through it

Tips for Configuration Section

Edge Routing for IGW

To configure Edge Routing (ingress routing) for IGW:

Create a New Route Table.
Associate it with the VPC
Click on Edit Edge Association
Associate IGW
Add Route for Protected App, pointing to GWLBe

Figure 12: Configuring Edge Routing for IGW

GWLB Cross Zone Load Balancing and MTU

In the examples shown here, we show that the Firewalls reside in one Availibilty Zone. Obviously, in a production Network, you will want the Firewalls to be in Multiple Availability Zone for HA purposes.

For this you need to do the following:

Create Multiple Firewall subnets in different Availaibilty Zones, for instance FW-us-east-1a, FW-us-east-1b. Then install Firewalls in each AZ you selected.
When creating the target group, make sure to include the IP of all the FW data interfaces
When creating GWLB, choose all the FW subnets in the different Availaiblity Zones. After creating the GWLB, edit the GWLB Attributes and enable Cross-zone load balancing
⚠️ You cannot add on a zone later

Figure 12a: Enabling Cross Zone Load Balancing

Create your GWLB Endpoint Service and tie your GWLB to it.
Create your GWLB Endpoint in the app VPC and it will install 1 GWLB Endpoint with ENIs in the corresponding zones.

GWLB & MTU:
The maximum transmission unit (MTU) is the size of the largest data packet that can be transmitted through the network. The Gateway Load Balancer interface MTU supports packets up to 8,500 bytes. Packets with a size larger than 8500 bytes that arrive at the Gateway Load Balancer interface are dropped.

Please see: https://docs.aws.amazon.com/elasticloadbalancing/latest/gateway/gateway-load-balancers.html

Target Group for GWLB

GWLB has to be associated withg a target group. Make sure that you use UDP 6081 for target group.

Figure 13: Use UDP 6081 (geeve) for target group

For the Health Checks in target group, use appropriate protocol. Below figure shows usage of port 443. Also make sure that health checks pass.

Figure 14: Health Checks for Target Group

Firewall Related Tips

When deploying firewall from Marketpalce, choose a firewall that supports Geneve. There are plenty of Firewalls that support Geneve, including Cisco, Checkpoint, Palo Alto.

Figure 15: Many Firewall vendors support geneve

For Firewall Security Group, you have to allow Geneve and the health check protocols. In addition all user traffic is tunnled through the Geneve Tunnel, so, the main ingress interface will need to allow all traffic coming from the GWLB. Controls on what host/protocol access is allowed can be configured for the vni interface (where the user traffic is de-encapsulated)

Figure 15a: Security Group for Firewall

📙
When deploying the ASA firewall make sure to create 2 ENIs for it. The first ENI should be the mgmt-interface, 2nd ENI should be FW-Data. You might want to hardcode the FW-Data interface IP.
After bringing up ASA FW, you can access it from AWS EC2 Serial Console connect and set the enable password (type in enable and it will prompt you for setting a password). After that you can ssh in from jumpbox

Sample Configuration for ASA Firewall:

Username admin2 password SuperSekret!!! Priv 15
http server enable
http 0 0 data

!
interface Management0/0
 no management-only
 nameif management
 security-level 100
 ip address dhcp setroute 
!
interface TenGigabitEthernet0/0
 nameif data
 security-level 50
 ip address 192.168.20.191 255.255.255.0 
!
interface vni1
 proxy single-arm
 nameif vni
 security-level 50
 vtep-nve 1
!
nve 1
 encapsulation geneve
 source-interface data
!
access-group inbound in interface data
access-list inbound remark For Health Checks
access-list inbound permit tcp any any eq http

access-group vni-inbound in interface vni
access-list vni-inbound remark For user data that you want to permit - add as needed
access-list vni-inbound extended permit tcp any any eq ssh
access-list vni-inbound extended permit icmp  any any

same-security-traffic permit intra-interface
route data 192.168.10.0 255.255.255.0 192.168.20.1 1

Debugging Tips for ASA Firewall

#Looking at logs:

term mon
config t
logging on
logging monitor debug
logging off
terminal no mon

# Looking at Connections
show conn

#Capture on Interface
capture test interface data real-time
no capture test
no capture /all

#Capture with filters:
access-list cap1 extended permit ip host 10.0.20.239 any4 
access-list cap1 extended permit ip host 10.0.30.53 any4 

capture test1 access-list cap1 interface data real-time

capture test1 access-list cap1 interface vni real-time

In case you have issues with ssh to firewall from jumphost, you can use:

ssh -oHostKeyAlgorithms=+ssh-rsa -i private_key admin2@fw_mgmt_IP

Creating GWLBe

For GWLBe, make sure to associate the GWLB Service Name. Also, make sure to put in correct subnet

Figure 16: Associating GWLBe with GWLB Service Name and putting in correct subnet

Make sure to import APP VPC into CNC with read only

Figure 17: Importing into CNC with read-only

Sticky configuration for GWLB Target Group and Appliance Mode in TGW

Since the Firewalls are not in Cluster Mode, but rather standalones, Target Group has Sticky Configuration Options as shown below.

Figure 18: Sticky Configuration for Target Group

In certain topology be aware that you may need to turn on TGW Appliance Mode. TGW flows have affinity to the same AZs and this can cause broken flows when using stateful appliances, depending on the topology. In the topology shown above, it is not necessarty to configure this. However be aware of this. The configuration cannot be changed via the AWS Console. It can only be changed from AWS CLI or SDKs.
The AWS CLI commands for configuring Appliance Mode for TGW Attachments are shown below

# to view
aws ec2 describe-transit-gateway-vpc-attachments --transit-gateway-attachment-ids  <tgw-attach-xyx>
# to change
aws ec2 modify-transit-gateway-vpc-attachment --transit-gateway-attachment-id <tgw-attach-xyx> --options ApplianceModeSupport=enable

Example of configuring/viewing TGW Attachment Appliance Mode is shown below:

Figure 19: TGW Appliance Mode AWS CLI commands

📙 Update as of 2/13/2023. Transit Gateway Appliance Mode can now be modified directly from AWS Console as shown below.

Figure 22: Modifying Transit Gateway Attachment for Appliance Mode from AWS Console

References

ASA configs:
https://www.cisco.com/c/en/us/td/docs/security/asa/asa918/configuration/general/asa-918-general-config/interface-vxlan.html?bookSearch=true#task_kff_ctk_5rb
https://www.cisco.com/c/en/us/td/docs/security/asa/asa918/configuration/general/asa-918-general-config/interface-vxlan.html?bookSearch=true#ID-2126-000000a0
https://www.cisco.com/c/en/us/td/docs/security/asa/asa94/asav/quick-start/asav-quick/asav-aws.html
https://blogs.cisco.com/security/building-a-scalable-security-architecture-on-aws-with-cisco-secure-firewall-and-aws-transit-gateway
http://www.cisco.com/c/en/us/td/docs/security/asa/asa94/asav/quick-start/asav-quick/asav-aws.html

Palo Alto Example:
https://github.com/Pal oAltoNetworks/AWS-GWLB-VMSeries/tree/main/terraform
https://github.com/PaloAltoNetworks/AWS-GWLB-VMSeries

Checkpoint Relalted:
https://blog.checkpoint.com/2020/11/10/check-point-cloudguard-integrates-with-aws-gateway-load-balancer-at-launch/
https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk174447
https://downloads.checkpoint.com/fileserver/SOURCE/direct/ID/112148/FILE/CP_CloudGuard_Network_for_AWS_Gateway_Load_Balancer_Security_VPC_for_Transit_Gateway_DeploymentGuide.pdf

F5:
https://community.f5.com/t5/technical-articles/big-ip-integration-with-aws-gateway-load-balancer-overview/ta-p/290821