Routable Pod Networks using vSphere with Tanzu- Another Reason to Choose it as Your Cloud-Native Platform

While Enterprises are moving more workloads as Containers into the Kubernetes Platform, they need additional features in the Platform that are typically not part of the default Kubernetes Native configuration Patterns.

One such common feature requirement is the capability to expose Pod IPs to External Services.

In simple, don’t SNAT the Pod IP to the Node IP while egressing the traffic from the Pods to the External Services.

Even though a few Container Network Interface (CNI) providers have the feature, they need additional routing configurations to enforce the behavior.

Here comes the seamless Operation Experience of vSphere with Tanzu. The recent release of vSphere with Tanzu version 7.0.3 offers the Routable Pod Network (Non-SNATed egress traffic) option for the Network traffic initiated from the Pods deployed into the Platform. Cloud-Native Platform Operators can turn on the feature while provisioning the Supervisor Clusters.

The USP of the vSphere with Tanzu Platform is its flexibility to provision Non-SNAT and SNAT capable Supervisor Namespaces in the same Supervisor Cluster a.k.a Workload Control Plane (WCP).

Once SNAT Disabled Supervisor Namespace is available, you can provision Tanzu Kubernetes Grid Service Clusters (Confirment Kubernetes Cluster Service of vSphere with Tanzu Platform). To Enable Routable Pod IPs, TKGs Cluster should use the newly introduced antrea-nsx-routed CNI plugin. TKGs Instances deploy in a Supervisor Cluster Namespace using antrea-nsx-routed CNI plugin, enable Non-SNAT egress Network traffic between the TKGs Cluster Pods and the External Services.

antrea-nsx-routed

The optional CNI plugin antrea-nsx-routedavailable for TKGs Clusters helps to deploy Anrea CNI with the following configurations. (The config. parameters are for ref. only and no need to manually change anything at the antrea-nsx-routedCNI specs).

• Inter-node Pod traffic is not encapsulated

trafficEncapMode: noEncap

• In the noEncap mode, the Pod traffic to the external network does not SNAT.

trafficEncapMode: noEncap

Also, you need to take the following Network requirements into account while choosing antrea-nsx-routed as the CNI plugin

• Pod’s IP addresses are routable and accessible externally from the cluster

In the current version, antrea-nsx-routedwill allocate a /24 pod CIDR to each TKGs Nodes from the pod CIDR defined for the Workload Control Plane or the pod CIDR configured at the Namespacelevel. You may allocate large enough pod CIDR at the Cluster level or Namespace level to fulfill the TKGs Node requirement.

When the CNI is antrea-nsx-routed, the pods.cidrBlock field must be empty in the TKGs Cluster manifest.

Note: vSphere version 7.0.3 added a new capability to define the Network Parameters like Pod CIDR, Ingress CIDR, Egress CIDR, Loadbalancer Flavor at the Supervisor Namespace level to override the cluster-wide Network Configurations.

In this blog, I will show the configuration option to disable SNAT at the Supervisor Cluster level , Custom Network Configuration Options at the Supervisor Namespace Level, and the CNI specifications of the TKGs (Confirment Kubernetes Cluster Service of WCP) manifest to Provision a Kubernetes Cluster with Routable Pod IPs

The intended readers are those familiar with the vSphere with Tanzu and the Workload Control Plane enablement workflow.

Lets jump into the config steps. The following screenshots explain the configuration steps needed to Enable Supervisor Clusterwith Routable Pod Network as its default egress Network traffic pattern. The configuration currently works with the vSphere version 7.0.3 deployment with NSX-T as its network backbone.

Configuration Steps

1. Start the Workload Management Workflow

Start the Workload Management Workflow

2. Select NSX as the Networking Stack

Select the Networking Stack

3. Select the Compatible vSphere Cluster to Setup as Supervisor Cluster

Select the Compatible vSphere Cluster

4. Select the rightStorage for the Supervisor Cluster based on the Pre-defined Storage Policy

Select the Storage Policy for the Supervisor Cluster

5. Provide the Management Network Parameters for the Supervisor Cluster. In vSphere 7.0.3 , you can optionally select DHCP as Network Mode for the Management Network based on the environment requiremnt.

Provide the Management Network

6. Configure the Workload Network. This configuration step defines the Default Egress Traffic Pattern of the vSphere with the Tanzu Platform.

Apply the following configuration requirements

• UnCheck the NAT Mode
• Unchecking the NAT mode disables the Egress CIDR Parameter
• Provide the remaining Workload Network Configuration parametrs as per the environment

Configure the Workload Network

7. Choose the Content Library hosting the TKGs Node OVAs

Ref: https://docs.vmware.com/en/VMware-vSphere/7.0/vmware-vsphere-with-tanzu/GUID-6519328C-E4B7-46DE-BE2D-FC9CA0994C39.html

Choose the Content Library-Step1

Choose the Content Library-Step2

8. Select the Supervisor Controlplane Flavor based on the Cluster Resource requirement

Select the Supervisor Controlplane Flavor

Finish the Workflow to initiate the Workload Control Plane deployment.

9. Once the Supervisor Cluster Config Status turns running, you may continue with the remaining Cluster Configuration

Supervisor Cluster Config Status

10. Login to the Supervisor Cluster using kubectl after downloading the kubectl-vsphere plugin from the URL https://<Control Plane Node Address> and verify the Node Status

Sample Command:

kubectl vsphere login --server 192.168.29.2  --insecure-skip-tls-verify
kubectl get nodes

Now its the time to create a Supervisor Namespaceand Deploy a TKGs Cluster with antrea-nsx-routed CNI to verify the Egress Network Traffic pattern.

11. Create a Supervisor Namespace. As mentioned earlier, with vSphere with Tanzu version 7.0.3, you could apply custom Network Configuration at Namespace level to override the Cluster-wide Workload Management Network Configurations. In this demonstration, I opted to override the cluster network settingsto show this excellent feature.

Create a Supervisor Namespace

11.1 Provide the Custom Network Parameters for the Namespace

Please note,as in the case of WCP enablemengt workflow;

• UnCheck the NAT Mode
• Unchecking the NAT mode disables the Egress CIDR Parameter

11.2 Add Permission to the Namespace

Add Permission

11.3 Add Storage Policy for the Namespace

Add Storage Policy

11.4 Add VM Classes from the VM Service Menu

Add VM Classes

11.5 Add the Content Library hosting the TKGs Node OVAs from VM Service Menu

Add the Content Library

12. Log into the Supervisor Cluster and Verify the Newly Created Namespace

Sample Command:

kubectl vsphere login --server 192.168.29.2  --insecure-skip-tls-verify

13. Describe the Namespace to Veify the SNAT configuration

kubectl describe ns routed-pod-ns

Sample Output:

You can see the NSX-T t1Router Id in which the Static Route for the Pod CIDR IPs of the Namespace will be configured. Also note that, there is no SNAT reference in the Namespace description.

Namespace description

14.Provision a TKGs Cluser Specs.

Note the following Mandatory Specs to provision a TKGs Cluster to configure routed egress IPs for the Pods

cni.name : antrea-nsx-routed

pods.cidrBlocks: <Empty>

15. Apply the TKGs Cluster Manifest to the Namespace

kubectl apply -f tkgs-routed-pod.yaml -n routed-pod-ns

15.1 After a few minutes the TKGs Cluster Ready Status turns True

Sample Output:

valex@tkgs-wrkstn:~$ kubectl get tkc  -n routed-pod-ns
NAME          CONTROL PLANE   WORKER   TKR NAME                           AGE   READY   TKR COMPATIBLE   UPDATES AVAILABLE
tkgs-routed   3               2        v1.21.2---vmware.1-tkg.1.ee25d55   10m   True    True

16.Login to the newly Created TKGs Cluster

Sample Command:

kubectl vsphere login --server 192.168.29.2  --tanzu-kubernetes-cluster-name tkgs-routed  --tanzu-kubernetes-cluster-namespace routed-pod-ns  --insecure-skip-tls-verify

17. Create a Namespace in the TKGs Cluster

kubectl create ns test-app

18. You may need to Create a Cluster Role Bindingto use the Pod Security Policy vmware-system-privilegedto deploy Privileged Containers like the one used here

Sample Command

kubectl create clusterrolebinding default-tkg-admin-privileged-binding  --clusterrole=psp:vmware-system-privileged --group=system:authenticated

18. Deploy a Test Pod into the TKGs Cluster Namespace to validate the egress traffic behavior.

In the demonstration, the image for the test App contains tools like ping for illustration purposes.

Sample Command:

kubectl create deploy wpcustom --image=harbor.cnrlab.tanzu/tkgs-test/wpcustom:v1 -n test-app

19. List the newly created Pod and note down its IP Address

Sample Command:

You may see the IP address of the test Pod 192.168.147.3

20.In the demo environment, to illustrate the Egress Network Flow of the Traffic initiating from the Pod, Opened a remote shell to the Container and executed the Ping Command to an External Linux Workstation having the IP address 10.105.18.176

20.1 Opened Remote Shell to the Pod:

Sample Command:

valex@tkgs-wrkstn:~$ kubectl exec -it wpcustom-7ffb58bd4c-qkpm6 -n test-app -- /bin/bash
root@wpcustom-7ffb58bd4c-qkpm6:/var/www/html#

20.2 executed the Ping Command to an External Linux Workstation having the IP address 10.105.18.176

Executed the Ping Command

21. I executed tcpdump at the Destination Workstation to verify the Source IP Address of the ICMP Packets

The tcpdumpoutput at the Destination shows IP Address of the Test Pod and the Non-SNATed traffic egressing from the vSphere with the Tanzu Platform.

The Supervisor Cluster configuration to disable SNATing of egress Network traffic works equally well for vSphere Podstoo, if you find a use case to deploy your Applications as vSphere Native Pods.

Awesome…you got a glance at another Cool feature of the vSphere with the Tanzu Platform. Well… still curious to know the egress traffic flow of another scenario?
Suppose you define antreaas the CNI in the TKGs Cluster instead of antrea-nsx-routedand deployed into a Namespace configured for Non-SNATtraffic. What will be the Source IP of the egress traffic initiating from the Pods deployed into the TKGs Cluster, showing at the external Destination?
You probably guess it correctly. The Source IP will be the TKGs Cluster Node IP of the Pod.
Enjoy Tanzu…See you with another exciting topic soon…