Yet Another Harbor Registry Architecture for vSphere with Tanzu -YAHA!!!

While vSphere with Kubernetes gaining adoption across all sort of business domains, the product team released the latest update to add a couple of features to the Tanzu Kubernetes Grid Services (TKGs).

The latest update provides the following key features to the Tanzu Kubernetes Grid services.

• Native Registry Service trust
• HTTP/HTTPS proxy support
• Custom disk creation on TKGs nodes

Native Trust to the Integrated Registry Service

In this blog, I explore a design approach that focuses on the Native Registry Service Trust feature.

Container Image Registry plays an essential role in the Cloud Native Development environment. One of the Kubernetes deployment’s mandatory configuration requirements is establishing the trust between the Container Runtime in the Nodes and the Container Image Registry. It allows the Container Runtime of the Nodes to pull the Images once the Kubernetes scheduler schedule the Pods on the respective Nodes.

The problem with the environments where registries use self-signed certificates is that the CA certs need to be distributed across the nodes manually or during the cluster bootstrap phase.

vSphere with Tanzu provides an Integrated secure Registry based on Harbor. In vSphere with Kubernetes and NSX-T, with a Mouse Click, you could enable the Integrated registry. The registry provides native authentication as well as automatically create Projects corresponding to the Supervisor Cluster Namespaces.

Even though the Supervisor Cluster Nodes ( ESXi hosts) trust the Integrated Registry CA Certificates, the Tanzu Kubernetes Service (TKGs) Nodes do not automatically trust the Integrated Registry’s CA Certificate. To establish trust between TKGs nodes and Integrated Harbor Registry, the platform admin needs to copy the CA Certificate manually across the TKGs nodes. Since the TKGs nodes can be scaled-out and scaled-in based on the compute requirement, the consistent distribution of the CA Certificates is a painful operational task.

The latest patch/update for the vSphere with Tanzu helps distribute the Integrated Registry Certificate during the TKGs bootstrap(Kubeadm Config) and avoid the manual process to populate the Integrated Registry CA Certificate across the TKGs nodes.

Though the native Harbor Registry of the vSphere/VCF with Tanzu provides a Secure and Integrated experience, it has only a limited set of features. The Integrated registry has not yet ported all upstream Harbor Registry features.

If those users are looking for a feature-rich OCI registry — for example OCI Artifact Repository, Image Synchronization, Image Scanning, Flexible Authentication Model, etc. — still need to deploy either Harbor or another Non-Harbor OCI registry. But if the user chooses to configure the external registry with a self-signed certificate, they end up with the requirement to port the CA certificate to all TKGs cluster nodes.

To save the situation, let us discuss an alternative Image Registry Infrastructure design.

Harbor is a Complete OCI registry, and it comes with a whole bunch of features to provide OCI registry requirements of a CI/CD environment. One of the cool features of the Harbor is its Replication Feature. The Replication Feature allows configuring Images or OCI Artifacts replication from one Harbor to another Harbor or another non-harbor OCI registry. The Replication Function can invoke using the following methods.

• Manual Replication
• Cron based Scheduler
• Event-based (for example: An Image Push is an Event) Replication

Let us see how the event-based replication feature of Harbor could help to provide a solution to the problem.

Fig:1 External Harbor Registry as Part of the CI/CD Infrastructure

As illustrated in the Diagram, Consider deploying an External Instance of the Harbor Registry, with all its cool features. This Instance of the harbor registry could act as a Point of Truthfor the Images and OCI artifacts in the CI/CD infrastructure. The External Harbor can run as a Kubernetes Service on the TKG clusters (preferred way) or on a Virtual Machine.

There are multiple types of distributions — TKG Extensions, Bitnami Helm Charts, Harbor Compose, etc. — that are available to deploy Harbor Registry. You may refer to some of the URLs provided at the end of the article, for the installation steps.

Here I brief a design where External Harbor functions as the Central OCI Registry with the following Configurations.

• Create the Projects corresponding to the Projects in the Integrated Harbor registry at the External Harbor Registry too.
• Images and OCI artifacts push to the External Harbor Registry’s respective Projects as part of the CI/CD workflow.
• Configure an endpointto define the Integrated Harbor Registry as the Destination.
• Event-based Replication Pushes the Image changes at the External Registry to the corresponding Projects in the Integrated Harbor Registry.

Configuration Steps

To configure the Replication from the external to the Integrated registry, you need to get the Administrator username and password of the integrated Harbor Registry.

Due to the RBAC model of the vSphere with Tanzu platform, the SSO users — even the administrator user- cannot access the objects like secrets, configmaps etc…

To get the Secret containing the Integrated Harbor Administrator Credentials, you need to execute the `kubectl` commands from the Supervisor Cluster Control plane Nodes. It allows you to perform tasks with the Supervisor Cluster-Admin privileges.

Step -1

1.1 Find the Namespace of the Integrated Harbor Registry from the VCenter Workload Management View.

Fig:2 Integrated Harbor Registry Namespace

Step-2

2.1 Login to the VCenter using your Credentials:

ssh root@<VCenter IP>

2.2 Execute the following Command after switching to shellto fetch the root User Passwordof the Supervisor Control Plane VMs.

/usr/lib/vmware-wcp/decryptK8Pwd.py

You get a really long Password (keep it safe!!! that is your super-secret password of the Supervisor Control Plane root user.)

2.3 Login to one of the Supervisor Control Plane VMs Management Interface (Preferably the etcd Master — The VM with two Management CIDR IPs)

ssh root@<Supervisior Control Plane VM Management IP>

Step-3

3.1 List the Secrets in the Integrated Registry Namespace

kubectl get secrets -n vmware-system-registry-1675912230

The Harbor Administrator Credentials are in the secretwith the name harbor-<id>-controller-registry

Please note, you see a different id in your installation. Replace the idwith the one, you find in your environment.

3.2 Execute the following Command to Extract the Harbor Administrator User Name.

kubectl get secret harbor-1675912230-controller-registry  -n vmware-system-registry-1675912230 -o jsonpath='{.data.harborAdminUsername}' | base64 -d | base64 -d

3.3 Execute the following Command to Extract the Harbor Administrator Password

kubectl get secret harbor-1675912230-controller-registry  -n vmware-system-registry-1675912230 -o jsonpath='{.data.harborAdminPassword}' | base64 -d | base64 -d

Step-4

Now it is time to configure the Replication in the External Harbor Registry.

As a reference configuration for this article, I created a Supervisor Cluster Name Space,tkgs-demo(Using the Workload Management Option of the vCenter).

A project with the same name automatically created in the Integrated Harbor Registry.

4.1 Create a Project with the same name in the External Harbor Registry

Fig:3 Create a Project in the External Harbor

4.2 Define the Integrated Harbor as an EndPointin the External Harbor Registry.

Provide the Integrated Harbor URL and Credentials while configuring the End Points. Test and Verify the Successful Communication between the registries. Please note, uncheck the box Verify Remote Certto skip the Certificate verification.

Fig:4 End Point Configuration

1. Select Harbor as the Provider
2. Give a Name for the endpoint
3. Provide the URLof the Integrated Harbor Registry
4. Enter the Integrated Harbor Administrator User Name (Ref 3.2)
5. Enter the Integrated Harbor Administrator Password (Ref 3.3)
6. Uncheck Verify Remote Certto skip the Integrated Harbor Certificate Verification

After Configure the parameters Test Connection and press OK

4:3 Configure the Replication from the External Harbor Registry to the Integrated Harbor Registry.

Fig:5 Replication Rule Configuration

1. Give a Name for the Replication Rule
2. Choose Push based replication
3. Provide the Name of the Project to Create a Source Filter to omit the other Projects from the Scope of the Replication. Optionally you can create additional filer rules based on Tags, Label, etc.
4. Select the EndPoint created in the previous step as the Destination
5. Enter the Destination Namespace
6. Select Event-Based as the Trigger Mode.

7 &8. Optionally Select the remaining two Tick boxes to define this (External)Instance of the Harbor as the Source of Truth for the images.

Step:5

Now is the time to test the Event-Based Replication from the External Registry to the Integrated Registry.

5:1 Login to the External Harbor Registry Using the Docker Client

docker login <External Harbor URL/IP>

5:2 Tag an Image with the External Harbor Repository Path

docker tag <ImageID> <External Harbor URL/IP>/tkgs-demo/<Image Name>:<Tag>

5:3 Push the Image to the External Registry

docker push <External Harbor URL/IP>/tkgs-demo/<Image Name>:<Tag>

5:4 Verify the Image in the External Registry

5:5 Login to the Integrated Harbor Registry and Verify the availability of the Image in the Project.

Step:6

In this step, deploy a Pod using the replicated Image. ( A basic smoke test!!!)

• *It is assuming that there is a TKGs cluster running in the namespace tkgs-demo and you currently logged in and switched to the TKGs context.

6.1 Creare a Namespace in TKGs (for example: tkgs-demo)

kubectl create ns tkgs-demo

6.2 Create an Image Pull Secret to authenticate with the Integrated harbor Registry

kubectl create secret docker-registry regcred --docker-server=<Integrated Harbor IP> --docker-username=<TKGs User Name> --docker-password=<TKGs Password> --docker-email=<*Optional Email> -n tkgs-demo

• *Please note, TKGs is using Pod Security Policy to control the security-sensitive aspects of the pod specification. You may create appropriate Role Bindings to allow the Successful creation of the Pods in TKGs.

Ref: https://docs.vmware.com/en/VMware-vSphere/7.0/vmware-vsphere-with-tanzu/GUID-4CCDBB85-2770-4FB8-BF0E-5146B45C9543.html

6.3 Create a Pod manifest with Image Pull Secret

6.4 Check the status of the Pod

**Sample Output
[root@centos-wkstn tkgs]# kubectl  get po -n tkgs-demo
NAME            READY   STATUS    RESTARTS   AGE
tkgs-demo-pod   1/1     Running   0          91s

Conclusion

In the article, I explored a possible Registry Infrastructure design, which works in tandem with the new features available in Tanzu Kubernetes Grid Services part of the vSphere/VCF with Tanzu. Even though the integrated Harbor Registry provides the essential functions for a CI/CD environment, many users prefer to deploy an External OCI registry to utilize the additional features. The design’s objective is to avoid the error-prone process of distributing the External Registries’ CA Certificateacross the TKGs cluster.

In the design, I considered an External Harbor Registry as the Source of Truth for the CI/CD infrastructure. Suppose you wish to go with a non-harbor registry in its place, you may configure an additional, Scheduler based Pull Replicationbetween the Non-Harbor and External Harbor Registry. In that case, the External Harbor can act as an Intermediate Registry, where it pulls the Images from the Non-Harbor registry in a scheduled mode and Push to the TKGs Integrated Harbor Registry using Event-based replication.

Referral URLs:

vSphere with Tanzu Configuration and Management

New Proxy Support, Registry Service Trust, and Separate Disks on vSphere with Tanzu

Harbor - Harbor Installation and Configuration

https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.2/vmware-tanzu-kubernetes-grid-12/GUID-extensions-harbor-registry.html

https://github.com/bitnami/charts/tree/master/bitnami/harbor