NDO/CNC Fabric Integration with AWS GWLB for protected apps

Table of Contents:

  1. Introduction
  2. High Level Architecture for GWLB
    2.a.Routing Table Discussion
    2.b. Packet Flow from Internet to Protected APP
    2.c. Packet Flow: East/West to Protected APP
  3. Integration of GWLB topology with NDO/CNC Fabric
    3.a. Discussion on Importing App VPC into CNC
    3.b. Packet Flow from outside VPC to Protected APP
  4. Tips for Configuration Section
    4.a, Edge Routing for IGW
    4.b.Target Group for GWLB
    4.c. Firewall Related Tips
    4.d. Creating GWLBe
    4.e. Make sure to import into CNC with read only
    4.f. Sticky configuration in GWLB and Appliance Mode in TGW
  5. References

Introduction

Referencing AWS post, GWLB (Gateway Load Balancer) was introuduced in AWS towards the later part of 2020. AWS Gateway Load Balancer (GWLB),is a service that makes it easy and cost-effective to deploy, scale and manage the availability of third-party virtual appliances such as firewalls, intrusion detection and prevention systems and deep packet inspection systems in the cloud.

In essence, GWLB makes it very easy to deploy protected apps that require both North/South and East/West traffic going back and forth through a firewall.

In this article I will discuss the architecture for GWLB followed by how to integrate GWLB with NDO/CNC Fabric.

High Level Architecture for GWLB

A basic high level architeture that can be deployed for GWLB is depicted below.
📙Keep in mind that GWLB architecture can be integrated with AWS Transit Gateways for added resiliency. Please see reference section for links that describe these architectures.
file
Figure 1: Basic High Level Architecture for GWLB

An explanation of the basic architecture:

  • Firewalls are deployed in a subnet (or multiple subnets) in a NetworkAppliance VPC. The Firewalls don’t necessarily have to be clustered. The Firewalls could also have added functionality like IPS/IDS (item 1 in figure)
  • The intent is to redirect all flows, North/South and East West for protected apps through the firewalls. (item 2 in figure)
  • A target group is created pointing to the firewalls, a GWLB is spun up in a GWLB subnet and associated with that target group. (item 3 in figure).
  • Once the firewall configuration is completed, a geneve tunnel will be established between the GWLB and the Firewalls. All Data traffic for protected app will traverse the geneve tunnel. There are plenty of 3rd party firewall vendors that support firewalls with geneve tunnel feature. (item 4 in figure)
  • A GWLB Service is created in the GWLB subnet. The service is associated with the GWLB (item 5 in figure)
  • A GWLB Interface VPC endpoint is created in the app VPC in a GWLBe subnet. This endpoint is associated with the GWLB Service in the NetworkAppliance VPC. This action creates a AWS Private Link from the GWLBe to the GWLB Service which in turn has already been associated with the GWLB (item 6 in figure)

⚠️ Keep in mind that the NetworkAppliance VPC and the APP-VPC needs to be in the same AWS Region (us-east-1 in the figure). Also the GWLBe and the GWLB Service needs to be in the same AWS Region/Zone (us-east-1a in figure).

Routing Table Discussion

Each subnet in the NetworkAppliance VPC and APP VPC should be configured with custom routing table for maximum control. Do not use the same default VPC routing table for the subnets.
Please reference the figure below for the discussion of routing tables configured.
file
Figure 2: Discussion of Routing Tables.

In the Network Appliance VPC:

  • The Firewall Jumphost needs Internet Access (based on the design shown in the figure).
  • Firewalls and GWLB don’t need public access. For that reason the Firewall Jumphost subnet has a default route pointig to the IGW.

In the App VPC:

  • App1 Jump Host has a route pointing to the IGW.
  • App1 Jump Host does not have any route pointing to GWLBe (Global Load Balancer Endpoint) as we don’t want this traffic to go through the firewall.
  • The Protected APP App1 is in a Public Subnet, since it needs to reach the internet for patching. Also, it needs to be accessed from the internet (maybe https). However notice that the default route is pointed to the GWLBe (Gateway Load Balancer Endpoint denoted by vpce-xxxxx).
  • The Protected APP App1 also has a route for APP2-subnet pointing to the GWLBe
  • Also, notice that APP2-subnet is a private subnet (which resides in the APP VPC which can be in any zone) has a route for APP1 subnet pointed to the Next Hop of GWLBe.
  • The IGW is configured with ingress routing (also known as edge routing). The edge routing rule says that for destination 10.0.20.0/24 (Protected APP Subnet), the Next Hop is the GWLBe. ( In the Tips for Configuration Section, I will show you how to do this)
Packet Flow from Internet to Protected App

Discussion of Packet Flow From Internet to Protected APP
Please reference the figure below for a discussion of packet flow from Internet to Protected App.
file
Figure 3: Packet Flow from Internet to Protected APP

  1. Packet comes from Internet destined to the App. The IGW route Table has a more specific route for the APP Subnet and directs it to the GWLBe
  2. Once Packet reaches the GWLBe, it is forwarded to the GWLB Service on the NetworkAppliace VPC through the AWS Private Link.
  3. The GWLB Service is associated with the GWLB and the packet goes to the GWLB and then gets forwarded to the Firewall through the geneve tunnel
  4. The Firewall applies it’s rules and if allowed, the packet is sent back to the GWLB
  5. Once packet reaches GWLB, it goes through the GWLB Service to ghe GWLBe through the AWS Private Link.
  6. The GWLBe subnet has a local route for the APP VPC (in this case 10.0.0.0/16), so packet gets forwarded to the App.

Discussion of Packet Flow from Protected APP to Internet
Please reference the figure below for a discussion of packet flow from Protected APP to Internet.
file
Figure 4: Return Packet to Internet

  1. App1 Route Table has a default route pointing to GWLBe, so the packet goes to GWLBe
  2. The packet is shunted to GWLB Service through the Private Link. The GWLB service is tied to the GWLB so, the packet reaches the GWLB.
  3. GWLB forwards the packet to the firewall through the geneve tunnel
  4. The Firewall applies it’s rules and forwards the packet back to GWLB
  5. The packet then goes through the private link back to GWLBe
  6. GWLBe has a default route to the internet, so packet goes through IGW back to source in Internet

Below figure shows the results of the configuration.

  • I am sitting on a Ubuntu Jump Server which is behind my home network. I curl ip.me to find my public IP assigned at this time.
  • I then curl to the Public IP of APP1 and it is successful.
  • I ssh to the Public IP (ssh is allowed by the firewall rules in my example).
  • I check on the Firewall for connections and observe that the flow did go through the firewall.

file
Figure 5: Example of sucessful N/S traffic Flow

Packet Flow: East/West to Protected APP

Using the same logic as above, you can reason out how East/West Redirect works. Please reference the figures below for this.

The Figure below shows Packet Flow from App2 to App1 (Protected App)
file
Figure 6: Packet Flow from App2 to App1 (Protected App)

The figure below shows Return Packet from App1 (Protected App) to App2
file
Figure 7: Packet Flow from App1 (Protected App) to App2

Below figure shows the results of the configuration.
file
Figure 8: Example of sucessful E/W traffic Flow

Integration of GWLB topology with NDO/CNC Fabric

Once the GWLB topology is setup in AWS, integrating it with NDO/CNC fabric can be done in 2 easy steps.

  1. Import Brownfield APP VPC into CNC. Please use Read Only Mode for this import.
  2. On NDO Import VRF from CNC
    📙 Please reference the previous writeups on how to do these imports.

Discussion on Importing App VPC into CNC
  • When Importing APP VPC into CNC, it is important to use a separate TGW Gateway Subnet for the TGW Attachment. This is best practice and gives you granular control for routing configuration which is needed in this solution.
  • Make sure to import the VPC into CNC with Read Only. CAPIC fully managed topology creates a universal route per VPC and you don’t want to do this. You also don’t want 2 points of control for Routing and Security Policies. Access Control and routing control for the Protected APP will need to be done from AWS Console.
  • There is no need to import the NetworkAppliance VPC (though you can). Treat this as a blackbox which is in control of the security group.
  • Once integrated you should be able to reach your Protected App from anywhere (Internet, CNC Fabric VPC/Vnet in same/otgher clouds, onPrem)
Packet Flow from outside VPC to Protected APP

There are 3 things that need to be done from a routing prospective for this to work.

  • Create a subnet in APP VPC in same zone as APP. Use this Subnet for attaching the AZ in the VPC This subnet is called the TGW Gateway Subnet. In this subnet create a route which says that Next Hop to reach the APP is the GWLBe.
  • In the GWLBe subnet put a route for the Source (greenfield subnet) pointing to Infra TGW
  • In the APP Subnet put a route for the Source (greenfield subnet) pointing to the GWLBe

The figure below shows the Packet Flow from Greenfield VPC to Protected APP in Brownfield VPC.
file
Figure 9: Packet Flow from Greenfield VPC to Protected APP in Brownfield VPC

Packet from Greenfield (destined to Protected APP1) comes from Infra TGW through attachment into the Gateway Subnet
Since Gateway subnet has a more specific route point to GWLBe the packet goes to GWLBe
Packet gets shunted to GWLB Service/GWLB through the Private Link
Packet from GWLB goes through geneve tunnel to Firewall.
Firewall inspects the rules and if allowed, sends the packet back to GWLB
Packet reaches the GWlBe through the Private Link
Since GWLBe subnet has a local route for VPC, it knows how to get to APP1 (Protected APP)

The figure below shows the Return Packet Flow from Protected APP in Brownfield VPC to Greenfield VPC.
file
Figure 10: Return Packet Flow from Protected APP in Brownfield VPC to Greenfield VPC.

Return packet from Protected App1 goes to the GWLBe, because of the static route in the App1 subnet
GWLBe forwards the packet to GWLB Service/GWLB through the AWS Private Link
GWLB forwards the packet to firewall through the geneve tunnel
Firewasll inspects the traffic and if allowed, sends it back to GWLB
GWLB Service sends the packet back to GWLBe in APP VPC
Once packet reaches the GWLBe, the packet is forwarded to the source through the Infra TGW (because of the static route in GWLB subnet for source pointing to Infra Transit Gateway)

Below figure shows that Packet from greenfield VPC reaches the Protected APP1 through the firewall.
file
Figure 11: greenfield VPC reaches the Protected APP1 through the firewall.

Tips for Configuration Section
Edge Routing for IGW

To configure Edge Routing (ingress routing) for IGW:

  • Create a New Route Table.
  • Associate it with the VPC
  • Click on Edit Edge Association
  • Associate IGW
  • Add Route for Protected App, pointing to GWLBe

file
file
file
Figure 12: Configuring Edge Routing for IGW

Target Group for GWLB

GWLB has to be associated withg a target group. Make sure that you use UDP 6081 for target group.

file
Figure 13: Use UDP 6081 (geeve) for target group

For the Health Checks in target group, use appropriate protocol. Below figure shows usage of port 443. Also make sure that health checks pass.
file
file
Figure 14: Health Checks for Target Group

Firewall Related Tips

When deploying firewall from Marketpalce, choose a firewall that supports Geneve. There are plenty of Firewalls that support Geneve, including Cisco, Checkpoint, Palo Alto.

file
Figure 15: Many Firewall vendors support geneve

For Firewall Security Group, you have to allow Geneve and the health check protocols.
file
Figure 16: Security Group for Firewall

Sample Configuration for ASA Firewall:

Username admin2 password SuperSekret!!! Priv 15
http server enable
http 0 0 data

!
interface Management0/0
 no management-only
 nameif management
 security-level 100
 ip address dhcp setroute 
!
interface TenGigabitEthernet0/0
 nameif data
 security-level 50
 ip address 192.168.20.191 255.255.255.0 
!
interface vni1
 proxy single-arm
 nameif vni
 security-level 50
 vtep-nve 1
!
nve 1
 encapsulation geneve
 source-interface data
!
access-group inbound in interface data
access-list inbound permit tcp any any eq http 
access-group inbound in interface data
same-security-traffic permit intra-interface
route data 192.168.10.0 255.255.255.0 192.168.20.1 1

Debugging Tips for ASA Firewall

#Looking at logs:

term mon
config t
logging on
logging monitor debug
logging off
terminal no mon

# Looking at Connections
show conn

#Capture on Interface
capture test interface data real-time
no capture test
no capture /all

#Capture with filters:
access-list cap1 line 1 extended permit ip host 172.16.0.40 any4
access-list cap1 line 1 extended permit ip host 172.16.0.40 any4

capture test1 access-list cap1 interface data real-time

In case you have issues with ssh to firewall from jumphost, you can use:

ssh -oHostKeyAlgorithms=+ssh-rsa -i pr admin2@fw_mgmt_IP
Creating GWLBe

For GWLBe, make sure to associate the GWLB Service Name. Also, make sure to put in correct subnet

file
file
Figure 16: Associating GWLBe with GWLB Service Name and putting in correct subnet

Make sure to import APP VPC into CNC with read only

file
Figure 17: Importing into CNC with read-only

Sticky configuration for GWLB Target Group and Appliance Mode in TGW

Since the Firewalls are not in Cluster Mode, but rather standalones, Target Group has Sticky Configuration Options as shown below.

file
Figure 18: Sticky Configuration for Target Group

In certain topology be aware that you may need to turn on TGW Appliance Mode. TGW flows have affinity to the same AZs and this can cause broken flows when using stateful appliances, depending on the topology. In the topology shown above, it is not necessarty to configure this. However be aware of this. The configuration cannot be changed via the AWS Console. It can only be changed from AWS CLI or SDKs.
The AWS CLI commands for configuring Appliance Mode for TGW Attachments are shown below

# to view
aws ec2 describe-transit-gateway-vpc-attachments --transit-gateway-attachment-ids  <tgw-attach-xyx>
# to change
aws ec2 modify-transit-gateway-vpc-attachment --transit-gateway-attachment-id <tgw-attach-xyx> --options ApplianceModeSupport=enable

Example of configuring/viewing TGW Attachment Appliance Mode is shown below:

file
file
file
Figure 19: TGW Appliance Mode AWS CLI commands

References

ASA configs:
https://www.cisco.com/c/en/us/td/docs/security/asa/asa918/configuration/general/asa-918-general-config/interface-vxlan.html?bookSearch=true#task_kff_ctk_5rb
https://www.cisco.com/c/en/us/td/docs/security/asa/asa918/configuration/general/asa-918-general-config/interface-vxlan.html?bookSearch=true#ID-2126-000000a0
https://www.cisco.com/c/en/us/td/docs/security/asa/asa94/asav/quick-start/asav-quick/asav-aws.html
https://blogs.cisco.com/security/building-a-scalable-security-architecture-on-aws-with-cisco-secure-firewall-and-aws-transit-gateway
http://www.cisco.com/c/en/us/td/docs/security/asa/asa94/asav/quick-start/asav-quick/asav-aws.html

Palo Alto Example:
https://github.com/Pal oAltoNetworks/AWS-GWLB-VMSeries/tree/main/terraform
https://github.com/PaloAltoNetworks/AWS-GWLB-VMSeries

Checkpoint Relalted:
https://blog.checkpoint.com/2020/11/10/check-point-cloudguard-integrates-with-aws-gateway-load-balancer-at-launch/
https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk174447
https://downloads.checkpoint.com/fileserver/SOURCE/direct/ID/112148/FILE/CP_CloudGuard_Network_for_AWS_Gateway_Load_Balancer_Security_VPC_for_Transit_Gateway_DeploymentGuide.pdf

F5:
https://community.f5.com/t5/technical-articles/big-ip-integration-with-aws-gateway-load-balancer-overview/ta-p/290821

GWLB Related:
https://aws.amazon.com/blogs/aws/introducing-aws-gateway-load-balancer-easy-deployment-scalability-and-high-availability-for-partner-appliances/
https://docs.aws.amazon.com/elasticloadbalancing/latest/gateway/elb-gateway.pdf
https://aws.amazon.com/blogs/networking-and-content-delivery/best-practices-for-deploying-gateway-load-balancer/
https://aws.amazon.com/blogs/networking-and-content-delivery/centralized-inspection-architecture-with-aws-gateway-load-balancer-and-aws-transit-gateway/
https://d1.awsstatic.com/architecture-diagrams/ArchitectureDiagrams/gateway-load-balancer-inspection-east-west-ra.pdf

Go To TOP


Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.