Routable Pod Networks using vSphere with Tanzu- Another Reason to Choose it as Your Cloud-Native Platform
While Enterprises are moving more workloads as Containers into the Kubernetes Platform, they need additional features in the Platform that are typically not part of the default Kubernetes Native configuration Patterns.
One such common feature requirement is the capability to expose Pod IPs to External Services.
In simple, don’t SNAT the Pod IP to the Node IP while egressing the traffic from the Pods to the External Services.
Even though a few Container Network Interface (CNI) providers have the feature, they need additional routing configurations to enforce the behavior.
Here comes the seamless Operation Experience of vSphere with Tanzu. The recent release of vSphere with Tanzu version 7.0.3 offers the Routable Pod Network (Non-SNATed egress traffic) option for the Network traffic initiated from the Pods deployed into the Platform. Cloud-Native Platform Operators can turn on the feature while provisioning the Supervisor Clusters.
The USP of the vSphere with Tanzu Platform is its flexibility to provision Non-SNAT and SNAT capable Supervisor Namespaces in the same Supervisor Cluster a.k.a Workload Control Plane (WCP).
Once SNAT Disabled Supervisor Namespace is available, you can provision Tanzu Kubernetes Grid Service Clusters (Confirment Kubernetes Cluster Service of vSphere with Tanzu Platform). To Enable Routable Pod IPs, TKGs Cluster should use the newly introduced antrea-nsx-routed CNI plugin. TKGs Instances deploy in a Supervisor Cluster Namespace using antrea-nsx-routed CNI plugin, enable Non-SNAT egress Network traffic between the TKGs Cluster Pods and the External Services.
The optional CNI plugin
antrea-nsx-routedavailable for TKGs Clusters helps to deploy Anrea CNI with the following configurations. (The config. parameters are for ref. only and no need to manually change anything at the
- Inter-node Pod traffic is not encapsulated
- In the noEncap mode, the Pod traffic to the external network does not SNAT.
Also, you need to take the following Network requirements into account while choosing
antrea-nsx-routed as the CNI plugin
- Pod’s IP addresses are routable and accessible externally from the cluster
In the current version,
antrea-nsx-routedwill allocate a
/24 pod CIDR to each TKGs Nodes from the pod CIDR defined for the Workload Control Plane or the pod CIDR configured at the
Namespacelevel. You may allocate large enough pod CIDR at the Cluster level or Namespace level to fulfill the TKGs Node requirement.
When the CNI is
pods.cidrBlock field must be empty in the TKGs Cluster manifest.
Note: vSphere version 7.0.3 added a new capability to define the Network Parameters like Pod CIDR, Ingress CIDR, Egress CIDR, Loadbalancer Flavor at the Supervisor Namespace level to override the cluster-wide Network Configurations.
In this blog, I will show the configuration option to disable SNAT at the Supervisor Cluster level , Custom Network Configuration Options at the Supervisor Namespace Level, and the CNI specifications of the TKGs (Confirment Kubernetes Cluster Service of WCP) manifest to Provision a Kubernetes Cluster with Routable Pod IPs
The intended readers are those familiar with the vSphere with Tanzu and the Workload Control Plane enablement workflow.
Lets jump into the config steps. The following screenshots explain the configuration steps needed to Enable
Supervisor Clusterwith Routable Pod Network as its default egress Network traffic pattern. The configuration currently works with the vSphere version 7.0.3 deployment with NSX-T as its network backbone.
- Start the Workload Management Workflow
NSX as the Networking Stack
3. Select the Compatible vSphere Cluster to Setup as Supervisor Cluster
4. Select the
rightStorage for the Supervisor Cluster based on the Pre-defined Storage Policy
5. Provide the Management Network Parameters for the Supervisor Cluster. In vSphere 7.0.3 , you can optionally select
Network Mode for the Management Network based on the environment requiremnt.
6. Configure the Workload Network. This configuration step defines the Default Egress Traffic Pattern of the vSphere with the Tanzu Platform.
Apply the following configuration requirements
- UnCheck the NAT Mode
- Unchecking the NAT mode disables the Egress CIDR Parameter
- Provide the remaining Workload Network Configuration parametrs as per the environment
7. Choose the Content Library hosting the
TKGs Node OVAs
8. Select the Supervisor Controlplane Flavor based on the Cluster Resource requirement
Finish the Workflow to initiate the Workload Control Plane deployment.
9. Once the Supervisor Cluster Config Status turns
running, you may continue with the remaining Cluster Configuration
10. Login to the Supervisor Cluster using
kubectl after downloading the
kubectl-vsphere plugin from the URL https://<Control Plane Node Address> and verify the Node Status
kubectl vsphere login --server 192.168.29.2 --insecure-skip-tls-verifykubectl get nodes
Now its the time to create a
Supervisor Namespaceand Deploy a TKGs Cluster with
antrea-nsx-routed CNI to verify the Egress Network Traffic pattern.
11. Create a Supervisor Namespace. As mentioned earlier, with vSphere with Tanzu version 7.0.3, you could apply custom Network Configuration at Namespace level to override the Cluster-wide Workload Management Network Configurations. In this demonstration, I opted to
override the cluster network settingsto show this excellent feature.
11.1 Provide the Custom Network Parameters for the Namespace
Please note,as in the case of WCP enablemengt workflow;
- UnCheck the NAT Mode
- Unchecking the NAT mode disables the Egress CIDR Parameter
11.2 Add Permission to the Namespace
11.3 Add Storage Policy for the Namespace
11.4 Add VM Classes from the VM Service Menu
11.5 Add the Content Library hosting the TKGs Node OVAs from VM Service Menu
12. Log into the Supervisor Cluster and Verify the Newly Created Namespace
kubectl vsphere login --server 192.168.29.2 --insecure-skip-tls-verify
13. Describe the Namespace to Veify the
kubectl describe ns routed-pod-ns
You can see the NSX-T
t1Router Id in which the
Static Route for the Pod CIDR IPs of the Namespace will be configured. Also note that, there is no SNAT reference in the Namespace description.
14.Provision a TKGs Cluser Specs.
Note the following Mandatory Specs to provision a TKGs Cluster to configure
routed egress IPs for the Pods
cni.name : antrea-nsx-routed
15. Apply the TKGs Cluster Manifest to the Namespace
kubectl apply -f tkgs-routed-pod.yaml -n routed-pod-ns
15.1 After a few minutes the TKGs Cluster Ready Status turns
valex@tkgs-wrkstn:~$ kubectl get tkc -n routed-pod-nsNAME CONTROL PLANE WORKER TKR NAME AGE READY TKR COMPATIBLE UPDATES AVAILABLEtkgs-routed 3 2 v1.21.2---vmware.1-tkg.1.ee25d55 10m True True
16.Login to the newly Created TKGs Cluster
kubectl vsphere login --server 192.168.29.2 --tanzu-kubernetes-cluster-name tkgs-routed --tanzu-kubernetes-cluster-namespace routed-pod-ns --insecure-skip-tls-verify
17. Create a Namespace in the TKGs Cluster
kubectl create ns test-app
18. You may need to
Create a Cluster Role Bindingto use the Pod Security Policy
vmware-system-privilegedto deploy Privileged Containers like the one used here
kubectl create clusterrolebinding default-tkg-admin-privileged-binding --clusterrole=psp:vmware-system-privileged --group=system:authenticated
18. Deploy a
Test Pod into the TKGs Cluster Namespace to validate the egress traffic behavior.
In the demonstration, the image for the test App contains tools like
ping for illustration purposes.
kubectl create deploy wpcustom --image=harbor.cnrlab.tanzu/tkgs-test/wpcustom:v1 -n test-app
19. List the newly created Pod and note down its IP Address
You may see the IP address of the
test Pod 192.168.147.3
20.In the demo environment, to illustrate the Egress Network Flow of the Traffic initiating from the Pod, Opened a remote shell to the Container and executed the Ping Command to an External Linux Workstation having the IP address 10.105.18.176
20.1 Opened Remote Shell to the Pod:
valex@tkgs-wrkstn:~$ kubectl exec -it wpcustom-7ffb58bd4c-qkpm6 -n test-app -- /bin/bashroot@wpcustom-7ffb58bd4c-qkpm6:/var/www/html#
20.2 executed the Ping Command to an External Linux Workstation having the IP address 10.105.18.176
21. I executed
tcpdump at the Destination Workstation to verify the Source IP Address of the ICMP Packets
tcpdumpoutput at the Destination shows IP Address of the Test Pod and the Non-SNATed traffic egressing from the vSphere with the Tanzu Platform.
The Supervisor Cluster configuration to disable SNATing of egress Network traffic works equally well for
vSphere Podstoo, if you find a use case to deploy your Applications as vSphere Native Pods.
Awesome…you got a glance at another Cool feature of the vSphere with the Tanzu Platform. Well… still curious to know the egress traffic flow of another scenario?
Suppose you define
antreaas the CNI in the TKGs Cluster instead of
antrea-nsx-routedand deployed into a Namespace configured for
Non-SNATtraffic. What will be the
Source IP of the egress traffic initiating from the Pods deployed into the TKGs Cluster, showing at the external Destination?
You probably guess it correctly. The Source IP will be the TKGs Cluster Node IP of the Pod.
Enjoy Tanzu…See you with another exciting topic soon…