The first post on this blog, and a reasonable summary of how we work.
Most of our engagements start the same way. A team has a Kubernetes platform that works — mostly — and nobody left on staff who built it. Deploys take longer than they used to. The AWS bill has drifted upward without an obvious cause. There’s an upgrade everyone knows is overdue and nobody wants to be the one to run.
We do fixed-scope work on exactly that problem. So rather than open this blog with an introduction nobody would read, here’s the actual checklist we work through in the first few days of a cluster handover. It’s the same list whether the platform is EKS, OpenShift, or something assembled from kubeadm and optimism.
1. How far behind is the control plane, really?
Version skew is the single most common reason an “upgrade project” turns into a quarter of work. Start with the honest numbers:
kubectl version -o json | jq '.serverVersion.gitVersion'kubectl get nodes -o custom-columns=\NAME:.metadata.name,\VERSION:.status.nodeInfo.kubeletVersion,\OS:.status.nodeInfo.osImage
What we’re looking for isn’t just the version — it’s the spread. A control plane at 1.29 with nodes at 1.26 means the supported skew window has already been breached, and the upgrade path is now serial rather than parallel. Each minor version has to be walked through in order, and on EKS that’s a distinct maintenance window per hop.
The second question is what breaks on the way. Deprecated API usage is the usual landmine:
kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis
That metric tells you which deprecated APIs are still being called and by what. It’s considerably more reliable than grepping manifests, because it catches the controller you forgot was installed three years ago.
2. Who can actually reach AWS, and how?
On EKS, the failure mode we see most often is credentials that work but shouldn’t. Node instance profiles carrying permissions that individual workloads inherit by default — meaning any pod on that node can assume them.
kubectl get sa -A -o json | jq -r ' .items[] | select(.metadata.annotations."eks.amazonaws.com/role-arn") | "\(.metadata.namespace)/\(.metadata.name)\t\(.metadata.annotations."eks.amazonaws.com/role-arn")"'
If that returns very little on a busy cluster, workloads are almost certainly falling back to the node role. Moving to IRSA — or EKS Pod Identity on newer clusters — is usually a week of careful work and one of the highest-value security changes available. It also makes the blast radius of a compromised pod something you can reason about, which matters when a customer security questionnaire lands.
While we’re here, we check whether the Terraform that provisioned the cluster still matches reality:
terraform plan -detailed-exitcode
An exit code of 2 on a cluster nobody has “changed” is a useful conversation starter. Drift accumulates through console clicks made during incidents, and every one of them is a change that will silently disappear the next time someone runs an apply.
3. On OpenShift: what’s pinned, and what’s drifting?
OpenShift clusters have a different failure profile. The platform itself is well-managed; the trouble is usually in what’s been installed on top of it.
oc get clusterversionoc get subscription -A -o custom-columns=\NS:.metadata.namespace,\NAME:.metadata.name,\CHANNEL:.spec.channel,\APPROVAL:.spec.installPlanApproval
That last column is the one that matters. A Subscription set to Automatic approval will upgrade its operator whenever the channel publishes a new version — which means an operator can move underneath you during a cluster upgrade, at the worst possible moment. We generally move production Subscriptions to Manual and pin the channel, so operator upgrades become a decision rather than an event.
We install through OLM wherever an operator exists, rather than Helm or raw manifests. It’s not dogma — it’s that OLM gives you a dependency graph, a defined upgrade path, and a cluster-wide view of what’s installed and where it came from. Helm gives you a tarball and good intentions. On day one they look equivalent. On day four hundred, when someone needs to know why a CRD version changed, they’re not remotely equivalent.
4. What does the deploy pipeline actually do?
Most CI/CD archaeology comes down to one question: can you tell, from a running pod, exactly which commit produced it?
kubectl get deploy -A -o json | jq -r ' .items[] | "\(.metadata.namespace)/\(.metadata.name)\t\(.spec.template.spec.containers[0].image)"' \ | grep -E ':(latest|master|main)$'
Anything that returns from that grep is a deployment you cannot roll back with confidence. Mutable tags mean the image running in production is whatever was last pushed under that name, and the relationship to source control is a matter of trust rather than record.
The fix is unglamorous — immutable tags, digest pinning, and a deployment annotation carrying the commit SHA — and it takes a couple of days. It also converts “roll back the bad release” from an investigation into a command.
5. Where is the money going?
Cost work usually gets framed as a finance request, but the useful version of it is an engineering question: what is running that nobody asked for?
The three findings that come up almost every time are requests set far above actual usage (so the cluster scales out to satisfy reservations nobody needs), storage volumes left behind by deleted workloads, and non-production clusters running around the clock for a team that works one timezone’s business hours.
None of that is exotic. It’s just that nobody owns it, so it compounds quietly.
What this looks like as a piece of work
Everything above is a few days of investigation followed by a scoped set of changes. That’s deliberately how we sell it: a defined piece of work with a stated outcome and an end date, not an open-ended retainer that becomes part of your run rate.
Typical first engagements look like a cluster upgrade path with the deprecated-API work done up front, an IRSA migration, an OpenShift day-2 review covering operator pinning and upgrade readiness, or a CI/CD rebuild that makes deployments traceable and reversible.
If any of the commands above returned something you’d rather they hadn’t, that’s usually a reasonable place to start a conversation.
Theta Cloud Consulting is a trading name of Theta Consulting Services Ltd. We are not affiliated with Red Hat, and OpenShift is a trademark of Red Hat, Inc.