Old Kubernetes upgrade

This case focuses on the upgrade of an ageing Kubernetes cluster running on AWS EKS. The cluster had been originally deployed by application developers, and over time had fallen several versions behind the supported range. With the Kubernetes version no longer maintained by AWS, the client needed to upgrade - but lacked a clear plan to do so safely, without risking downtime or service disruption.

As is often the case with developer-provisioned infrastructure, the environment lacked formal maintenance procedures, runbooks, or structured versioning policies. This increased the risk of incompatibility during the upgrade process, especially given the presence of breaking changes between the installed version and the desired target version.

In Kubernetes, breaking changes typically involve API deprecations or removals. If a resource manifest uses a removed API version, workloads will fail to deploy on the upgraded cluster. Even deprecated APIs that still function today are effectively a ticking time bomb - expected to fail in future releases.

Upgrade Planning and Compatibility Checks

The first step was to assess the current cluster version and consult AWS’s official EKS upgrade guidance for that release. This was critical, as EKS clusters involve more than just Kubernetes versioning; core system components like CoreDNS, kube-proxy, and the AWS VPC CNI plugin also require validation and upgrade.

To identify workload compatibility risks, we used the open-source tool kube-no-trouble. This scans existing cluster resources to detect deprecated or removed Kubernetes APIs. A full inventory of affected workloads was produced, and the associated manifest files were updated to use supported API versions. All changes were committed to source control for ongoing traceability and safe redeployment.

Handling Helm Compatibility

One additional challenge came from Helm releases. Helm stores release metadata that can reference outdated APIs. If this metadata is not updated, future Helm upgrades can fail - even if the underlying manifests are corrected.

To address this, we used mapkubeapis, a Helm plugin designed to update stored release state by remapping deprecated APIs to their replacements. This ensured Helm upgrades would remain compatible with the new Kubernetes version and preserved release integrity.

Summary of Steps Taken

The upgrade strategy was executed through three structured phases:

1. Following AWS EKS’s legacy version upgrade path, starting from the cluster’s current version.

2. Scanning for and replacing deprecated Kubernetes APIs using kube-no-trouble.

3. Migrating Helm release metadata using mapkubeapis to ensure compatibility.

   
   

Once all adjustments were made, the cluster was upgraded without incident. In parallel, a full cluster review was performed to bring the platform in line with operational best practices, correcting several architectural oversights that had accumulated over time.