Tag: non-human identities

Kubernetes clusters are expensive to run and expensive to understand. Most engineering teams know their monthly bill; almost none know which workload, team, or feature is responsible for which portion of it. That information gap is where cloud waste lives.

The FinOps Foundation’s State of FinOps 2026 report documents the gap precisely: 98% of FinOps practitioners are now managing AI and cloud spend together, and pre-deployment cost visibility is the top desired capability across organizations of all sizes. Teams that have built this visibility are cutting their Kubernetes bills by 20 to 40 percent without removing features or downgrading performance.

This guide covers the specific practices, tools, and architecture decisions that make that possible.

Why Kubernetes Costs Are Hard to Manage

Traditional cloud cost allocation works at the service or resource level. Kubernetes adds two layers of abstraction: pods share nodes, and nodes are grouped into clusters. A single node bill might represent traffic from a dozen different applications owned by three different teams.

Without active cost attribution, the bill is opaque. You know you spent $40,000 on compute in March. You do not know that $18,000 of that came from a batch job that runs once a day and could run overnight on Spot instances at one-fifth the cost.

The three root causes of Kubernetes waste:

1. Overprovisioning: Teams request more CPU and memory than workloads use, because the cost of over-requesting is invisible and the cost of under-requesting is an outage.
2. Idle capacity: Nodes that stay running overnight and on weekends for workloads that only run during business hours.
3. Unattributed spend: No namespace-level or label-level cost breakdown means no team feels accountable for their portion of the bill.

Step 1: Get Cost Visibility Before You Optimize:

You cannot optimize what you cannot see. The first step is establishing namespace-level and workload-level cost attribution.

GKE Cost Allocation (Now Generally Available) : Google Kubernetes Engine’s cost allocation feature, which became generally available in 2025, breaks down billing by cluster, namespace, and label, and exports that data to BigQuery. If you are on GKE, this is your starting point. Enable it today.

In your GKE cluster settings, enable the Cost Allocation feature under Networking. Configure a BigQuery export in your billing settings. Within 24 to 48 hours you will have namespace-level cost data you can query directly.

A basic BigQuery query to see cost by namespace:

SELECT namespace, SUM(cost) as total_cost FROM `billing_export.gke_cost_allocation` 
WHERE DATE(usage_start_time) >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY) GROUP BY 
namespace ORDER BY total_cost DESC;

For Multi-Cloud or Self-Managed Clusters : Tools like Kubecost, OpenCost (CNCF open-source), and Finout provide namespace and label-level cost attribution across AWS EKS, Azure AKS, and self-managed clusters. Kubecost’s free tier covers a single cluster; the paid tier adds multi-cluster rollup and anomaly detection.

The minimum label taxonomy to enforce across all workloads:

1. team: the owning engineering team
2. service: the product or service name
3. environment: production, staging, development
4. cost-center: the budget code for chargeback

Step 2: Rightsize Before You Buy More

Most Kubernetes performance problems are attributed to insufficient resources, so teams over-provision. The data consistently shows the opposite: the average Kubernetes cluster runs at 20 to 30 percent CPU utilization and 40 to 60 percent memory utilization under normal load.

Vertical Pod Autoscaler (VPA) for Rightsizing Recommendations : VPA in recommendation mode (not enforcement mode) analyzes actual pod resource usage and recommends right-sized requests and limits without changing anything automatically. Run it for two weeks, review the recommendations, and apply changes manually to critical workloads.

To deploy VPA in recommendation mode for a deployment:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
spec.
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Off"  # Recommendation only, no automatic changes

Check recommendations after 14 days:

kubectl describe vpa my-app-vpa

Teams that right-size based on VPA recommendations typically reduce their compute requests by 30 to 40 percent while maintaining the same performance profile.

Horizontal Pod Autoscaler (HPA) for Bursty Workloads: If your workloads have predictable traffic patterns (higher during business hours, lower at night), HPA with custom metrics can scale down to minimum replicas during off-peak hours automatically. Combined with cluster autoscaler removing idle nodes, this is the single highest-ROI optimization for most teams.

Step 3: Shift Non-Critical Workloads to Spot or Preemptible Instances

Spot instances (AWS) and Preemptible VMs (GCP) cost 60 to 90 percent less than on-demand instances. They can be terminated with 2 minutes of notice. That constraint rules them out for stateful or latency-critical workloads, but opens significant savings for everything else.

Workloads that are suitable for Spot:

1. Batch processing jobs
2. CI/CD pipeline workers
3. Data transformation and ETL
4. Non-critical background workers
5. Development and staging environments

The Kubernetes node pool configuration for Spot on GKE:

gcloud container node-pools create spot-pool \  --cluster=my-cluster \  --spot \  --machine-type=n2-standard-4 \  --num-nodes=0 \  --enable-autoscaling \  --min-nodes=0 \  --max-nodes=20

Use node selectors or tolerations to schedule appropriate workloads onto the spot pool while keeping production workloads on on-demand nodes.

Step 4: Add AI Spend to Your FinOps Scope

The FinOps Foundation’s 2026 survey found that 98% of FinOps teams are now managing AI spend, making it the fastest-growing cost category under FinOps oversight. If your Kubernetes clusters are running ML inference workloads or AI-adjacent services, those costs need the same attribution and optimization treatment as your application workloads.

Specific controls for AI workloads on Kubernetes:

1. GPU cost allocation: Tag GPU node pools separately and require workloads to justify GPU requests. GPU nodes cost 3 to 8 times more than equivalent CPU nodes.
2. Inference scheduling: Batch inference workloads to run during off-peak hours when Spot availability is higher and cost is lower.
3. Model caching: Cache loaded models in memory rather than loading them on each request. Model load time is pure GPU cost with no output.
4. Cost per inference: Track cost per model query, not just per pod. This connects infrastructure cost to product usage in a way engineers and product managers can both act on.

Step 5: Implement Chargeback to Create Accountability

The most durable cost control is not a technical optimization. It is making teams financially aware of what they consume.

Chargeback allocates actual cloud costs to the teams or cost centers responsible for them. Showback is the lighter version: teams see their costs but are not charged internally. Both work; chargeback creates stronger behavioral change.

A minimal chargeback implementation:

1. Export namespace-level cost data weekly to a shared dashboard (BigQuery + Looker Studio, or Kubecost’s cost center report)
2. Send each team lead a weekly cost summary email for their namespaces
3. Set budget alerts at 80% and 100% of monthly targets per namespace
4. Review cost anomalies in your weekly engineering sync, not in a separate FinOps meeting

Teams that see their costs consistently make different infrastructure decisions than teams that do not. The change is not dramatic; it is cumulative. Over six months, awareness alone reduces waste by 10 to 15 percent.

What 30% Cost Reduction Actually Looks Like

Based on implementations across multiple clients, the savings stack roughly as follows:

1. Rightsizing via VPA recommendations: 15 to 25% reduction in compute spend
2. Spot/Preemptible for non-critical workloads: 10 to 20% of total cluster cost
3. HPA + cluster autoscaler for off-peak scaling: 5 to 10% reduction
4. Chargeback-driven behavioral change: 5 to 15% over six months

The exact number depends on your current state. Teams with no optimization in place and no cost attribution tend to see the largest gains quickly. Teams that are already using autoscaling and have some attribution in place see smaller but still meaningful reductions.

The work is not technically complex. It is operationally consistent. The teams that achieve 30% reductions are the ones that treat infrastructure cost as an engineering metric, not an accounting problem.

Need help building a FinOps practice for your Kubernetes environment? Talk to our engineering team at Codelynks.www.codelynks.com/contact

Related Blogs: RAG vs Fine-Tuning in 2026: The Best Strategy for Your Enterprise AI