Building Red Hat UBI Images for vSphere Kubernetes Service

Introduction

Broadcom announced support for Red Hat Enterprise Linux 9 as a supported operating system for vSphere Kubernetes Service (VKS) Cluster nodes in VKS 3.6 release. VKS 3.6 also introduces Image Baker Tool for creation of custom operating images for consumption in VKS Environments.

Image Baker provides a build tool that lets you create custom, reproducible Kubernetes node images with full control over the OS ( sysctl & kernel configuration) along with the packages.

Image Baker introduces

Part of the VCF CLI toolchain – Delivered as a plugin for the VMware Cloud Foundation CLI, making it easy to integrate into existing automation flows.
Declarative image specification – You define the image’s configuration (OS, Kubernetes version, registry sources) in a structured spec file.
OCI-based build workflow – The tool uses a BuildKit-style approach, pulling a base image (like a universal base image) and layering in all required components systematically.
Clean, compliant images – The images are lean, predictable, and easier to secure and audit.

In this blog we’ll walk through creation of Red Hat UBI Image in an internet restricted environment for consumption in VKS Environments.

Prerequisites

Ubuntu 24.04 VM with 1 vCPU , 4 GB RAM and 100 GB disk
Build-essential and zlib1g-dev package installed
Docker Engine
Harbor Registry
open-vmdk
Buildkit

Firewall rules

Source	Destination	Protocol	Port
Image Baker VM	DNS Server	UDP	53
Image Baker VM	NTP Server	UDP	123
Image Baker VM	Harbor Registry	TCP	80
Image Baker VM	Harbor Registry	TCP	443
Image Baker VM	Red Hat CDN(s)	TCP	443

Refer to Public CIDR Lists for Red Hat for FQDN/IP Address for Red Hat CDN(s).

Download Packages on an Internet Connected Machine

Download vcf cli v9.0.2 from Broadcom Repository
Install vcf cli on the machine to download the required plugins

sudo install vcf-cli.tar.gz /usr/local/bin/vcf

Download vcf cli plugins to a tar bundle

			
vcf plugin download-bundle --plugin kubernetes-release --to-tar /home/pj/vcf-cli-plugins.tar

Download buildkit binaries from GitHub

			
wget https://github.com/moby/buildkit/releases/download/v0.27.1/buildkit-v0.27.1.linux-amd64.tar.gz

Download open-vmdk git repository fom GitHub

git clone https://github.com/vmware/open-vmdk

Pull Red Hat UBI Image and VMware Kubernetes Spec Image

			
docker pull registry.access.redhat.com/ubi9/ubi
docker pull projects.packages.broadcom.com/vsphere/iaas/kubernetes-release/1.35.0/kubernetes-distribution-image:v1.35.0_vmware.2-vkr.4

View pulled images

			
docker images
                                                                                                                                                                                                                                                                                                                                                                                        
IMAGE                                                                                                                        ID             DISK USAGE   CONTENT SIZE   EXTRA
projects.packages.broadcom.com/vsphere/iaas/kubernetes-release/1.35.0/kubernetes-distribution-image:v1.35.0_vmware.2-vkr.4   08999113df77       5.33GB         2.38GB
registry.access.redhat.com/ubi9/ubi:latest                                                                                   b8923f58ef6a        312MB           80MB

Save the Red Hat UBI Image and VMware Kubernetes Spec Image as a tar archive

			
docker save 08999113df77 | gzip > v1.35.0_vmware.2-vkr.4.tar.gz      
docker save b8923f58ef6a | gzip > ubi9.tar.gz

Transfer all the packages to the internet restricted environment . The packages to be transferred are

VCF cli tar archive
VCF cli plugins tar archive
Red Hat UBI Image tar archive
VMware Kubernetes Spec Image tar archive
open-vmdk git repo folder
buildkit tar archive

Prepare Image Baker in Internet Restricted Setup

As all download packages are transferred to a restricted internet environnment lets setup vcf cli to create the Red Hat UBI Image

Retrive the certificate for the Harbor Registry

			
openssl s_client -showcerts -connect bitnami.workernode.lab:443  < /dev/null | awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/{print}' > ca.crt

Copy the certificate to the local CA certificates directory and add the certificate to system trust store

			
cp ca.crt  /usr/local/share/ca-certificates/ca.crt
update-ca-certificates

Create a dedicated directory for the Harbor certificate in the Docker configuration location and copy the Harbor Registry file into it

			
mkdir -p /etc/docker/certs.d/bitnami.workernode.lab/
cp ca.crt /etc/docker/certs.d/bitnami.workernode.lab/ca.crt

Restart the Docker service to apply the changes

systemctl restart docker

Load the UBI Image and Kubernetes spec image into docker

			
docker load -i v1.35.0_vmware.2-vkr.4.tar.gz
Loaded image ID: sha256:08999113df775e0c0c105360e66ccb03ffc0a4a7d239941a46c8d10dc3a2cf5a
docker load -i ubi9.tar.gz
Loaded image ID: sha256:b8923f58ef6aebe2b8f543f8f6c5af15c6f9aeeef34ba332f33bf7610012de0c

Tag the Images

			
docker tag 08999113df77 kubernetes-distribution-image:v1.35.0_vmware.2-vkr.4
docker tag b8923f58ef6a ubi:latest

List the Images

			
docker images
                                                                                                                                            
IMAGE                                                  ID             DISK USAGE   CONTENT SIZE   EXTRA
kubernetes-distribution-image:v1.35.0_vmware.2-vkr.4   08999113df77       5.33GB         2.38GB
ubi:latest                                             b8923f58ef6a        312MB           80M

docker login bitnami.workernode.lab

Create project in Harbor for vcf cli plugins and push the vcf plugins bundle

			
vcf plugin upload-bundle --tar /home/pj/vcf-cli-plugins.tar --to-repo bitnami.workernode.lab/vcf_cli/plugins

Create project in Harbor for Red Hat UBI Image . Tag and push the image to Harbor Registry

			
docker tag b8923f58ef6a bitnami.workernode.lab/ubi9/ubi
docker push bitnami.workernode.lab/ubi9/ubi

Create a project in Harbor for Kubernetes Spec Image . Tag and push the image to Harbor Registry

			
docker tag 08999113df77 bitnami.workernode.lab/vkd-image-builder/kubernetes-distribution-image:v1.35.0_vmware.2-vkr.4
docker push bitnami.workernode.lab/vkd-image-builder/kubernetes-distribution-image:v1.35.0_vmware.2-vkr.4

Extract buildkit and configure buildkit service

			
##Extract buildkit binaries 
tar -C /usr/local -xzf buildkit-v0.27.1.linux-amd64.tar.gz
##Configure buildkit service
sh -c 'cat > /etc/systemd/system/buildkit.service' << 'EOF'
[Unit]
Description=BuildKit
Requires=buildkit.socket
After=buildkit.socket
Documentation=https://github.com/moby/buildkit
[Service]
Type=notify
ExecStart=/usr/local/bin/buildkitd --addr fd://
[Install]
WantedBy=multi-user.target
EOF
sh -c 'cat > /etc/systemd/system/buildkit.socket' << 'EOF'
[Unit]
Description=BuildKit
Documentation=https://github.com/moby/buildkit
[Socket]
ListenStream=%t/buildkit/buildkitd.sock
SocketMode=0660
[Install]
WantedBy=sockets.target
EOF
##Enable buildkit service 
systemctl enable --now buildkit.socket

		

Browse to open-vmdk folder and run below command to install open-vmdk .

			
#cd open-vmdk/
make 
make install

Install vcf cli on the machine

sudo install vcf-cli.tar.gz /usr/local/bin/vcf

Update the vcf cli plugin repository to Harbor Registry plugin inventory arfifact and install the kubernetes-release plugin

			
vcf plugin source update default --uri bitnami.workernode.lab/vcf_cli/plugins/plugin-inventory:latest
vcf plugin source list
vcf plugin list
vcf plugin install kubernetes-release

Provide Red Hat Subscription username and password

			
export RH_USERNAME=testing
export RH_PASSWORD=testing@123

Run Image Baker in to build Red Hat UBI Image

Create a configuration file for the image creation

			
apiVersion: imageconfiguration.vmware.com/v1alpha1
kind: Image
metadata:
    name: rhel-9-amd64-v1.35.0---vmware.2-vkr.4
spec:
    osSpec:
        name: rhel
        version: "9"
        image: bitnami.workernode.lab/ubi9/ubi:latest
    kubernetesSpec:
      image: bitnami.workernode.lab/vkd-image-builder/kubernetes-distribution-image:v1.35.0_vmware.2-vkr.4
    diskSize: 30Gi
    sysctls:
        - key: "net.ipv4.tcp_tw_reuse"
          value: "1"
        - key: "net.ipv4.ip_forward"
          value: "1"
        - key: "net.core.somaxconn="
          value: "1024"
        - key: "net.core.rmem_max"
          value: "16777216 "
        - key: "net.core.wmem_max"
          value: "16777216 "
        - key: "net.ipv4.tcp_max_syn_backlog"
          value: "8192"
        - key: "vm.max_map_count"
          value: "1048575"
        - key: "net.ipv4.ip_forward"
          value: "1"
    kernelParameters:
        - name: "verbose"
          value: "1"
        - name: "maxcpus"
          value: "4"
		- name: "rhgb"
          value: "splash"
	packages:
    present:
      - name: nfs-utils
      - name: curl
      - name: jq

		

Run image baker for creation of image

vcf kr bake -f config-packages.yaml --log-level DEBUG --verbose 9

Image Baker process would pull the OCI Image from Harbor Registry and installs the kubernetes packages along with the packages defined in the configuration yaml . It would also set the sysctl & kernel configuration to the image.

Image Baker takes about 20 minutes to generate the image and saves the image in artficats/ directory on the system.

A successful creation prints the below message

			
{"time":"2026-02-17T12:53:10.973516988+05:30","level":"DEBUG","msg":"Custom OVF properties generated successfully","file":"cache/custom_ovf_properties.json"}
{"time":"2026-02-17T12:53:33.857200117+05:30","level":"INFO","msg":"generated OVA file in the artifacts directory."}
{"time":"2026-02-17T12:53:33.857585129+05:30","level":"INFO","msg":"OVA package generated successfully"}
{"time":"2026-02-17T12:53:33.857593285+05:30","level":"INFO","msg":"Image baking completed successfully","image":"rhel-9-amd64-v1.35.0---vmware.2-vkr.4","output":"artifacts/rhel-9-amd64-v1.35.0---vmware.2-vkr.4"}
{"time":"2026-02-17T12:53:33.85760159+05:30","level":"DEBUG","msg":"Clearing cache"}
{"time":"2026-02-17T12:53:34.164060236+05:30","level":"DEBUG","msg":"Cache cleared successfully"}

		

View created image in the artficats/ directory on the system

			
ls -altrh artifacts/
total 12K
drwxr-xr-x 3 root root 4.0K Feb 17 12:46 .
drwxr-x--- 8 pj   pj   4.0K Feb 17 12:50 ..
drwxr-xr-x 2 root root 4.0K Feb 17 12:53 rhel-9-amd64-v1.35.0---vmware.2-vkr.4
ls -altrh artifacts/rhel-9-amd64-v1.35.0---vmware.2-vkr.4/
total 6.9G
drwxr-xr-x 3 root root 4.0K Feb 17 12:46 ..
-rw-r--r-- 1 root root  35K Feb 17 12:50 package_list.json
-rw-r--r-- 1 root root 231K Feb 17 12:50 kernel_config
-rw-r--r-- 1 root root  407 Feb 17 12:50 system_info.json
-rw-r--r-- 1 root root  341 Feb 17 12:50 repo_sources.tgz
-rw-r--r-- 1 root root  757 Feb 17 12:50 kernel_tunables.tgz
-rwxr-xr-x 1 root root  581 Feb 17 12:50 manifest.json
-rw-r--r-- 1 root root 3.5G Feb 17 12:53 rhel-9-amd64-v1.35.0---vmware.2-vkr.4-disk1.vmdk
-rw-r--r-- 1 root root 179K Feb 17 12:53 rhel-9-amd64-v1.35.0---vmware.2-vkr.4.ovf
-rw-r--r-- 1 root root  239 Feb 17 12:53 rhel-9-amd64-v1.35.0---vmware.2-vkr.4.mf
-rw-r--r-- 1 root root   64 Feb 17 12:53 rhel-9-amd64-v1.35.0---vmware.2-vkr.4.ova.sha256
-rw-r--r-- 1 root root 3.5G Feb 17 12:53 rhel-9-amd64-v1.35.0---vmware.2-vkr.4.ova
drwxr-xr-x 2 root root 4.0K Feb 17 12:53 .

		

The introduction of Image Baker in VKS 3.6 marks a significant milestone for vSphere Kubernetes Service. For too long, creating node images has been the wild west of platform engineering—full of manual steps, fragile scripts, and inconsistent results.

Image Baker changes all that by giving teams:

Full control over node composition
Repeatable builds via declarative specs
Support for enterprise OS standards like RHEL

In the next blog I would showcase creation of VKS Clusters using the Red Hat image created with the image baker workflow and deploy a sample application on the VKS Cluster.

Disclaimer: All posts, contents and examples are for educational purposes in lab environments only and does not constitute professional advice. No warranty is implied or given. The user accepts that all information, contents, and opinions are my own. They do not reflect the opinions of my employer.

Building Red Hat UBI Images for vSphere Kubernetes Service

Comments

Leave a comment Cancel reply

Building Red Hat UBI Images for vSphere Kubernetes Service

Share this:

Comments

Leave a comment Cancel reply