Site icon

Terraform with Cisco Nexus Dashboard Orchestrator for building Hybrid Cloud and end to end services

Table of contents

  1. Introduction
  2. What We Will Demo in this Article
  3. A Brief Introduction to Terraform
    1. Characteristics of Terraform
    2. Important Pointers of Terraform
    3. Terraform Providers
    4. Terraform HCL code structure
    5. What Order are Resources built
    6. Terraform Backends
    7. Provisioners
    8. Executing The HCL Code
    9. Types of IAC Tools
    10. Terraform Commands
    11. Items to Remember for Terraform
    12. Security Considerations when uploading Terraform HCL code to git
  4. Let’s begin with our deployment
    1. Pre-Requisites
    2. Download & Install Terraform
    3. git clone the sample code
    4. modify override.tf files
    5. create aci tenant in aws tenant account
      1. Important Note on parallelism env for Terraform with NDO
      2. Checking from NDO
      3. Checking the terraform Module graph with terraform graph
    6. create ec2 instances in the ACI/AWS tenant infrastructure
    7. spinning up Elastic Kubernetes Cluster on ACI/AWS tenant infrastructure
  5. References

Introduction

In the past, setting up data center infrastructure has been a very manual process. Humans have to physically rack and stack the Data Center Infrastructure. Then the infrastructure has to be configured so that connectivity to Servers/Services/Security can be properly implemented. Finally the Servers/Services has to be manually provisioned and segregated based on the requirements.

If more servers and services are needed on a temporary or pemnanent basis, the process has to be manually repeated. Expanding data centers to the cloud is a perfect solution to speed up the process. With Hybrid Cloud Data Centers, you can easily expand or pull back your Data Center resources in the cloud as needed.

Cloud ACI with Cisco Nexus Dashboard Orchestrator gives you the capability to create hybrid cloud data centers easily. The Data Center infrastructure across physical and cloud (or any combination of physical and/or multiple cloud provider clouds) can be configured through the Nexus Dashboard Orchestrator. The end result is that you can have your services securely anywhere (physical/cloud or any combinaton) and it is oblivious to the end user.

Once the infrastructre is ready, you can use the Cloud providier’s console or API based provisioning methods for spining up compute and services. Provisioning using the cloud console is generally done for quick one time configurations. However in the cloud, services are often spun up and destroyed depending on the needs. Automation for this is essential for any devops based operations so that the required services can be spun up quickly and destroyed consistently and without errors.

Every cloud provider has their own automation method. Some examples are shown below:


Though these cloud automation/deployment tools are excellent for the particular cloud provider, Terraform works across all cloud providers. Terraform also works with NDO. The implication of this is that you can build your entire Data Ceter Infrastructre across physical and different cloud Providers targetting Terraform for NDO. You can then use Terraform to spin up the required services in the cloud and you have an end to end Hybrid Data Center in minutes. This also gives you the capabiity to break the infrastructure and services down as needed (to save on costs). Further, the infrastructure will be always consistent and error free since you don’t have to do manual provisioning.

What We Will Demo in this Article

In this article, we will do the following:

  1. Spin up a full ACI Tenant in AWS using Terraform with NDO
  2. Spin up Web Servers on the AWS Tenant / ACI Infrastructure using Terraform
  3. Spin up a full Kubernetes Cluster (Elastic Kubernetes Services) on the ACI Infrastructure on AWS using eksctl utility

I have purposely only used AWS in this demo to keep it simple. You can easily modify the provided scripts to extend the tenant to physical ACI Fabric as needed. You can follow though this writeup to get very familiar with the process and modify the scripts as you wish.

For this demo, we will run the free (non cloud) version of Terraform on our local desktop.

Go To TOP

A Brief Introduction to Terraform

Terraform is a multi-cloud automation software product created by Hashicorp®. Recently Hashicorp announced it’s initial Public Offering (Nov 29th, 2021)

Some of the highlights of Terraform are listed below:

You can see a full list of Terraform Feature Matrix at: https://www.datocms-assets.com/2885/1602500234-terraform-full-feature-pricing-tablev2-1.pdf


Figure 1: Terraform Feature Matrix

Characteristics of Terraform

*Note: SaltStack can run both with agent and agentless

Important Pointers of Terraform


Figure 2: Terraform is a Declarative Model, Ansible is an Imperative Model.

Terraform Providers

Terraform List Of Providers can be found here: https://registry.terraform.io/browse/providers



Figure 3: Terraform Providers

Terraform HCL code structure
What Order are Resources built

Terraform builds a dependency graph from the Terraform configurations, and walks this graph to generate plans, refresh state, and more.

Terraform Backends
Provisioners
Executing The HCL Code

Once you write the initial Terraform Code, you can deploy and modify the code as shown in the flow diagram below.

Figure 4: Executing the Terraform Code

Types of IAC Tools

It’s important to realize that all IAC tools have their place and each is suitable for it’s use case. Terraform in no way obsoletes any of the existing tools. Depending on the use case, the proper IAC tool needs to be selected.


Figure 5: Different IAC Tools

IAC tools can also be used in combination. The illustration below shows Terraform being used to spin up AWS EC2 and utilizing Terraform Local Provisioner to make ansible calls to configure the EC2 tools (perhaps install some application on the EC2). As mentioned before Terraform Remote Provisioner can be also used to configure the EC2, but as a practise, it’s not recommended.


Figure 6. Making Ansible calls from Terraform

Terraform Commands

A list of Terraform Commands available can be seen below:

aciadmin@ubuntu-jump:~/Terraform/AWS-EKS-Terraform$ terraform -help
Usage: terraform [global options] <subcommand> [args]

The available commands for execution are listed below.
The primary workflow commands are given first, followed by
less common or more advanced commands.

Main commands:
  init          Prepare your working directory for other commands
  validate      Check whether the configuration is valid
  plan          Show changes required by the current configuration
  apply         Create or update infrastructure
  destroy       Destroy previously-created infrastructure

All other commands:
  console       Try Terraform expressions at an interactive command prompt
  fmt           Reformat your configuration in the standard style
  force-unlock  Release a stuck lock on the current workspace
  get           Install or upgrade remote Terraform modules
  graph         Generate a Graphviz graph of the steps in an operation
  import        Associate existing infrastructure with a Terraform resource
  login         Obtain and save credentials for a remote host
  logout        Remove locally-stored credentials for a remote host
  output        Show output values from your root module
  providers     Show the providers required for this configuration
  refresh       Update the state to match remote systems
  show          Show the current state or a saved plan
  state         Advanced state management
  taint         Mark a resource instance as not fully functional
  test          Experimental support for module integration testing
  untaint       Remove the 'tainted' state from a resource instance
  version       Show the current Terraform version
  workspace     Workspace management

Global options (use these before the subcommand, if any):
  -chdir=DIR    Switch to a different working directory before executing the
                given subcommand.
  -help         Show this help output, or the help for a specified subcommand.
  -version      An alias for the "version" subcommand.

Go To TOP

Items to Remember for Terraform

Items to Remember for Terraform

As mentioned previously, Terraform is stateful and sensitive information like passwords and keys are stored in the state files. For this reason, if you want to upload your code to git, you need to ensure that you don’t upload the sensitive information.

I would recommend the following:

aciadmin@ubuntu-jump:~/gitstuff/terraform-NDO-AWS_with_ec2_eks$cat .gitignore
#  Local .terraform directories
**/.terraform/*

# .tfstate files (including backup state file)
*.tfstate
*.tfstate.*

# .tfvars files
*.tfvars

# no creds
*.pem
*.key

# terraformrc file
terraform.rc

# override.tf files with sensitive information
*override.tf

Go To TOP

Let’s begin with our deployment

Pre-Requisites
  curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
  unzip awscliv2.zip
   sudo ./aws/install
  aws configure --profile tfuser     ( # tfuser is the username you created in AWS for this)
export AWS_DEFAULT_PROFILE=tfuser
Download & Install Terraform

Terraform comes in a single binary. To download it,


Figure 7: Installing Terraform

Check with terraform version

Figure 8: terraform version

git clone the sample code

Clone the sample git code to your ubuntu VM

git clone https://github.com/soumukhe/terraform-NDO-AWS_with_ec2_eks.git


Figure 9: Clone the sample code from git

Change Directory to "terraform-NDO-AWS_with_ec2_eks/"
a tree command in that directory will show you the folowing file structures

Figure 9: Structure of cloned git repo

modify override.tf files

Go to each of the main directories and modify the "override.tf" files.

Below you can see the contents of "override.tf" in thge aciAWS_infra directory.

#  use this override.tf to put in confidential data

#  Populate values based on your AWS values
variable "awsstuff" {
  type = object({
    aws_account_id    = string
    aws_access_key_id = string
    aws_secret_key    = string
  })
  default = {
    aws_account_id    = "populate_me"
    aws_access_key_id = "populate_me"
    aws_secret_key    = "populate_me"
  }
}

#  Populate values based on your ND cofigiration
variable "creds" {
  type = map(any)
  default = {
    username = "populate_me"
    password = "populate_me"
    url      = "https://ip_of_nd/"
    domain   = "put_in_auth_domain_defined_in_ND" # if you are using local user, comment this out.  
                                                  # Make sure to also comment out in variables.tf file.
  }
}
create aci tenant in aws tenant account
/*
Values of variables to override default values defined in variables.tf
These are my sample variable values.  Please change based on your requirements.
*/
aws_site_name = "AWS10" # the site name for the AWS site as seen on ND

schema_name   = "SM-Terraform-Tenant-Schema" # give it a name for the schema as you wish
template_name = "shared-template"            # use a template name as you wish
vrf_name      = "vrf1"                       # use a vrf name as you wish
bd_name       = "bd1"                        # use a bd name as you wish
anp_name      = "anp1"                       # use a ANP name as you wish
epg_name      = "epg1"                       # use a EPg name as you wish
region_name   = "us-east-1"                  # Make sure that you choose a region that was enabled in cAPIC initial setup

cidr_ip = "10.140.0.0/16" # CIDR IP as you wish for the VPC to be created in AWS tenant account

subnet1 = "10.140.1.0/24" # subnet should belong to CIDR
zone1   = "us-east-1a"    # az should be the 1st az in the chosen region.

subnet2 = "10.140.2.0/24" # subnet should belong to CIDR
zone2   = "us-east-1b"    # az should be the 2nd az in the chosen region.

subnet3 = "10.140.3.0/24" # subnet should belong to CIDR
zone3   = "us-east-1b"    # az should be the 2nd az in the chosen region.  Only 2 zones are allowed per region currently in cAPIC

epg_selector_value = "10.140.0.0/16" # EPG Selector to ensure proper Security Rules as defined by ACI Contracts

user_association = "soumukhe" #  the user to get associated with the tenant

📗

Important Note on parallelism env for Terraform with NDO

Make sure to source the file parallelism.env
⚠️ if you don’t do this, NDO based "terraform apply" will show many errors and will not work

Check with "terraform version"

Figure 11: terraform version

Now do "terraform apply"
You will see the terraform plan output and it will propmt you to accept with a yes
The bottom part of "terraform apply" output is shown below:

Figure 12: terraform apply (bottom part)

your ACI Infrastructure on AWS Tenant Account will now be setup
Bottom part of output is shown below:

Figure 13: output of "terraform apply"

📗 Note: you can destroy the Teanant by terraform destroy
However for my script it seems you have to run "terraform destroy" 3 times. This is probably due to some interpolation error in my script, that I will have to investigate

Figure 14: terraform destroy (unfortunately in my script I have to run it 3 times)


In case you tested with "terraform destroy" please make sure to do "terraform apply" again to deploy the ACI/AWS tenant

Checking from NDO

You can now check from NDO to see what the tenant looks like.


Figure 15: Clicking on deployed view

The deployed view will look like below

Figure 16: Deployed View from NDO

Checking the terraform Module graph with terraform graph

To accomplish this do the following:

terraform graph > graph.dot
sudo apt install graphviz
cat graph.dot | dot -Tsvg > graph.svg
scp the file to your local workstation
Then open graph.svg in browser

The output should look like below:


Figure 16: viewing output of "terraform graph"

Go To TOP

create ec2 instances in the ACI/AWS tenant infrastructure]


Figure 18: spinning up 2 ec2 instances in ACI/AWS tenant infrastructure


Figure 19: output of run (bottom part)



Figure 20: curling to the public IPs

For ssh, user name is ec2-user

Figure 21: ssh to one of the ec2 instances


You can use "terraform destroy" to destroy the ec2 instances

Go To TOP

spinning up Elastic Kubernetes Cluster on ACI/AWS tenant infrastructure]
1. Download and extract the latest release of eksctl with the following command.
curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp

2. Move the extracted binary to /usr/local/bin.
sudo mv /tmp/eksctl /usr/local/bin

3. Test that your installation was successful with the following command.
eksctl version
sudo apt install kubectl

This will spit out the subnet-ids for your ACI/AWS tenant environment as shown below:

Figure 22: using Terraform to get subnet ID of subnets

Using the output from "terraform plan":

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: acme-test-cluster
  region: us-east-1
  version: "1.21"

vpc:
  id: vpc-XXX                           # put in vpcID from output of terraform output
  subnets:
    public:
      us-east-1a: { id: subnet-YYY }    # put in subnetID from output of terraform output
      us-east-1b: { id: subnet-ZZZ }    # put in subnetID from output of terraform output
    private:   
      us-east-1a: { id: subnet-YYY }    # put in subnetID from output of terraform output
      us-east-1b: { id: subnet-ZZZ }    # put in subnetID from output of terraform output

managedNodeGroups:
#nodeGroups:
  - name: acme-test-cluster-workers
    minSize: 3
    maxSize: 6
    desiredCapacity: 3
    instanceType: m5.2xlarge
    labels: {role: worker}
    ssh:
      publicKeyPath: ~/.ssh/id_rsa.pub
    tags:
      nodegroup-role: worker
    iam:
      withAddonPolicies:
        externalDNS: true
        #certManager: true
        #albIngress: true

# usage:  eksctl create cluster --config-file ./cluster_config.yaml
#         eksctl delete cluster --config-file ./cluster_config.yaml

My modification of cluster-config.yaml file looks like below:

Figure 23: My modified cluster-config.yaml

export AWS_DEFAULT_PROFILE=tfuser
aws sts get-caller-identity

The script will take about 20 minutes to create the cluster

snip


Figure 24: Creating EKS Cluster

Verifiation:

eksctl get clusters
kubectl get nodes
kubectl config view   (and other kubectl commands)


Figure 25: eksctl get clusters


Figure 26: kubectl get nodes

To delete the cluster use the following command:

eksctl delete cluster --config-file ./cluster_config.yaml


Figure 28: Deleting EKS cluster

References

terraform related:
https://www.terraform.io/
https://registry.terraform.io/browse/providers
https://www.datocms-assets.com/2885/1602500234-terraform-full-feature-pricing-tablev2-1.pdf
https://registry.terraform.io/providers/CiscoDevNet/mso/latest/docs
https://registry.terraform.io/providers/CiscoDevNet/aci/latest/docs
https://github.com/lhercot/terraform-mso-demo_aws/blob/master/main.tf
https://github.com/lhercot/terraform-mso-demo_azure/blob/master/main.tf

eksctl related:
https://eksctl.io/introduction/
https://docs.aws.amazon.com/eks/latest/userguide/network_reqs.html
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_cluster
https://learn.hashicorp.com/tutorials/terraform/eks?_ga=2.50616617.1278539655.1636448383-980409436.1623122455
https://www.fairwinds.com/blog/terraform-and-eks-a-step-by-step-guide-to-deploying-your-first-cluster
https://joachim8675309.medium.com/create-eks-with-an-existing-vpc-8e31d95ccc5b
https://joachim8675309.medium.com/create-an-amazon-vpc-for-eks-597481514bcc
https://aws.amazon.com/premiumsupport/knowledge-center/eks-kubernetes-object-access-error/
https://stackoverflow.com/questions/53266960/how-do-you-get-kubectl-to-log-in-to-an-aws-eks-cluster
https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html
https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html
https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
https://aws.plainenglish.io/making-sense-of-amazon-eks-vpc-networking-fa50ddfeda3d
https://eksctl.io/usage/eks-private-cluster/

From other blogs:
https://blogs.cisco.com/datacenter/flexible-hybrid-cloud-networking-with-infrastructure-as-code-and-cisco-nexus-dashboard?oid=pstcsm027527
https://blogs.cisco.com/developer/terraform-with-aci-1

Something to investigate:
https://dcappcenter.cisco.com/consul.html

Go To TOP

Exit mobile version