NodePortLocal: What Magic ‘Local’ adds to the Default NodePort Configuration?
A while ago, One of the well-known Kubernetes CNI Provider Antrea, Introduced a feature called ‘NodePortLocal’.
As per the product document, NodePortLocal (NPL) is a feature that runs as part of the Antrea Agent, through which each port of a Service's backend Pod can be reached from the external network using a port of the Node on which the Pod is running. NPL enables better integration with external Load Balancers, which can take advantage of the feature instead of relying on NodePort Services implemented by Kube-Proxy.
Here comes the AVI Kubernetes Operator, a.k.a AKO handy. AKO helps integrate NSX-T Advanced Load Balancer (Previously known as AVI Vantage) as the External LoadBalancer Provider for Kubernetes Clusters and utilize the Antrea CNI NodePortLocalfeature.AKO started supporting the feature NodePortLocal from Version 1.5.2 onwards with Antrea CNI Provider.
AKO is a Kubernetes operator to automate the Corresponding Configurations at the AVI platform while creating the relevant Kubernetes Network Objects.
AKO supports creating Ingress / Routes (L7 Functions) and Load Balancer (L4 Functions)Services.
The blog's objective is to provide a quick reference to understand the difference between the NodePort and NodePortLocal Modes to Integrate External Load Balancer Providers with the K8S Clusters. Also, the article shows the Advantage of Enabling the NodePortLocal Mode while using AVI with Antrea CNI.
Considering the availability of detailed How-To documentsabout deploying the AVI Platform and AKO, I opted to point to the relevant references instead of duplicating the instructions.
AKO and AVI Configuration
You may refer to the following URLs for the steps to deploy AKO and NSX-T Advanced Load Balancer Platform.
You Can find the HELM Value file used to deploy the reference Environment for the article in the below Repository.
https://github.com/emailtovinod/ako-values-file.git[Irrespective of the version reference in the URL, the process is similar for the latest releases of AKO too. You may use the relevant version numbers while executing the Commands]
There is an alternate procedure for VMware TANZU Kubernetes Grid (TKG) clusters to Deploy AKO using kapp-controller during the Cluster Boot Strapping. Refer to the following URL to know the process.
You can find the guidelines to modify the AKO Deployment Configuration to enable NodePortLocal in the below URL.
Antrea Configuration to Enable NodePortLocal
As of TKG Version 1.5.1, NodePort is the default mode. You need to add the following Configuration Parameter to the TKG Cluster Configuration Manifest to change it to NodePortLocal.
For those who deploy Antrea into the Non-TKG Kubernetes Platform, you need to apply the following configuration in the Antrea ConfigMap to enable the NodePort Local Feature.
kind: ConfigMap
apiVersion: v1
metadata:
name: antrea-config-dcfb6k2hkm
namespace: kube-system
data:
antrea-agent.conf: |
featureGates:
# True by default starting with Antrea v1.4
# NodePortLocal: true
nodePortLocal:
enable: true
# Uncomment if you need to change the port range.
# portRange: 61000-62000Now let us see how the Integration of External LoadBalancer Providers works with Kubernetes Clusters in NodePort and NodePortLocal Modes.
1.1 NodePort Configuration Using Kube-Proxy
1.2 External Load Balancer Integration with NodePort Configuration
Pros:
- Default Supported Configuration of most of the CNI Providers
- Works for a majority of Use Cases.
Cons:
- NodePort Configuration Exposes the port on all the nodes of the Kubernetes Cluster, even if the pods are not on it. [ Though the behavior can be controlled via Node Labels in AKO at additional operational overhead]. It means Kube-Proxy does secondary hop of load balancing to re-distribute the traffic amongst the Pod and increases the east-west traffic in the Cluster.
- Session Persistence does not work. For load balancers wanting to do consistent hash or session persistence directly to the Application pod from outside the Kubernetes clusters, NodePort is not a feasible solution.
- NodePort Configuration exposes a range of ports on all Kubernetes nodes irrespective of the Pod scheduling. It may hit the port range limitations as the number of services (of type nodePort) increases.
2.0 NodePortLocal Configuration Using Antrea
2.1 Antrea Adds' nodeportlocal.antrea.io' annotation to the Pods of the Service
2.2 External Load Balancer Integration with NodePortLocal Configuration
3.0 How NodePortLocal Mode Works for the Pods Schedule to the Same Node?
3.1 Pod ‘webapp-cd787c9bc-ntfr8’ exposes Port 61000
3.2 Pod 'web app-cd787c9bc-brch2' exposes to Port 61001 on the same Node
3.3 Avi Virtual Service Pool Member Configuration for the Pods Scheduled into the Same Node
It is one of the key advantages of the NodePortLocal. NodePortLocal Mode integration also helps External Load Balancers effectively distribute the Network Traffic amongst the Pods scheduled to the Same Kubernetes Nodes. Kubernetes Native parameters like externalTrafficPolicy=Local do not help to achieve the function.
4. Conclusion
Pros:
- NPL removes the
Hop— Kube-Proxy configured Network Rule- at the cluster node and will direct traffic to the Pod's node port instead of the service's node port. - Pod's node port will only exist on nodes where the Pod is running, and it helps to reduce the east-west traffic and encapsulation Overhead.
- Even in Scenarios like multiple pods of the same deployment running on the same Node, the NodePortLocal Enabled External LoadBalancers (e.g.AVI) can effectively distribute the traffic to the Pods.
- Better Session Persistence.
Cons:
- Currently, Limited CNIs Supporting the Feature
To meet the Critical Cloud-Native Workload requirements, Antrea and Avi add theNodePortLocalfeature. It can configure without any additional,NSX-T Advanced LoadBalancer License overhead compared to the default NodePort Mode to achieve consistent Ingress Traffic distribution amongst the Pods.
