👋 Everything about EKS & AI Infrastructure Newsletter "#56" ☁️❤👨‍💻

Dear EKS & AI Infrastructure enthusiasts,

Welcome to Everything about EKS & AI Infrastructure #56.

The infrastructure we’re building right now has a strange quality: it’s both harder and easier than it’s ever been. Harder because the problems are genuinely difficult—disaggregating LLM inference phases across GPU types, sharing KV cache peer-to-peer without external dependencies, coordinating resilience systems with autoscalers in ways Kubernetes was never designed for. Easier because the tools to solve these problems are emerging faster than we can evaluate them, and half the time we’re not sure if we’re engineering solutions or just wiring together what the ecosystem handed us this week.

This edition focuses on performance engineering in modern AI systems, but it’s also about something less tangible: the growing gap between provisioning infrastructure and using it intelligently. We can spin up GPU clusters in minutes now. The hard part is making sure 40-60% of that capacity doesn’t sit idle in silos while other teams wait in queue. The hard part is knowing when to adopt the new thing and when to let it bake. The hard part is building systems we’ll still understand six months from now when the hype cycle has moved on and we’re the ones on-call.

If you’re running EKS, managing GPU fleets, or just trying to keep up with how fast the AI infrastructure layer is evolving, this edition is for you—not because it has all the answers, but because it names the questions we’re all sitting with right now.

1. Performance Engineering in Modern AI Systems 🌩️

🌩️Chamber (YC W26): Why 40-60% of your GPUs sit idle (by Andreas Blomquist)

Chamber just emerged from Y Combinator’s Winter 2026 batch with a sharp thesis: despite massive AI infrastructure investments, 40-60% of enterprise GPU capacity sits idle in siloed allocations. Teams reserve GPUs they don’t fully use, other teams wait in queue, and nobody has centralized visibility across clusters to fix it. The annual waste? Over $240 billion industry-wide.

Chamber’s approach is straightforward: intelligent scheduling with preemptive queuing (high-priority jobs pause lower ones, resuming automatically), continuous hardware health monitoring to catch silent GPU failures before they kill training runs, and team-based fair-share policies where unused allocation automatically lends to others. The promise: run ~50% more workloads on the same GPUs without manual intervention. If you’re managing GPU fleets on EKS with multiple teams and projects competing for capacity, this is the kind of tooling that addresses the allocation problem, not just the visibility problem. Worth watching.

1. Starred Content ⭐

⭐ Accelerate Generative AI Inference with NVIDIA Dynamo and Amazon EKS (by Michael Silva and Fernando Schroder Rodrigues)

Scaling LLM inference in production hits the same walls everywhere: high latency, poor GPU utilization, and costs that spiral when you move beyond experiments. This AWS Machine Learning blog walks through deploying NVIDIA Dynamo on EKS to address all three. Dynamo disaggregates prefill (compute-bound, short-lived) and decode (memory-bound, long-running) phases across different GPU nodes, schedules resources dynamically based on real-time demand, and routes requests to minimize KV cache recomputation. The result: higher GPU throughput, lower latency, and better resource efficiency for multi-node inference workloads.

The deployment uses an AI on EKS blueprint from AWS Labs that provisions everything—VPC, EKS cluster with GPU nodes (g6, p5, p6), Karpenter for just-in-time scaling, EFA for low-latency inter-node transfers, and storage integrations (EFS, FSx for Lustre, S3 Mountpoint CSI). The post includes architecture diagrams, benchmarks, and a complete walkthrough from infrastructure setup to model serving. If you’re running vLLM or Triton on EKS and fighting latency or utilization issues at scale, Dynamo’s disaggregated architecture is a sharp answer—especially for workloads with varying sequence lengths or bursty traffic patterns

⭐ Deploying Qwen Image Edit on Amazon EKS with GPU Acceleration (by Gary Stafford)

Gary Stafford walks through deploying Qwen Image Edit 2511—an open-weight image editing model—on EKS with GPU acceleration for production-grade image processing at scale. The architecture uses a two-container pattern: FastAPI service on GPU nodes for inference, React UI on CPU for user-facing workflows, both behind an ALB. A DaemonSet preloads the 4-bit quantized model from S3 to node-local EBS volumes, so pod restarts don’t trigger multi-GB downloads and cold starts stay low.

The 2511 release improved character consistency, multi-person editing, integrated LoRA capabilities, and geometric reasoning—useful for batch industrial design, material replacement, and multi-reference object placement. The post covers Kubernetes patterns (health checks, resource requests, GPU scheduling), storage trade-offs (S3 for artifacts, EBS for fast access), and how to wire it all up with Terraform and Helm. If you’re running GPU workloads on EKS and need to serve generative models with low-latency startup and predictable scaling, this is a solid reference implementation with open-source code on GitHub.

⭐LMCache P2P: Sharing KV cache across instances without external services (by LMCache Lab and Tencent)

Most production LLM deployments on EKS waste compute because each inference pod builds its own KV cache from scratch, even when serving identical prefixes. LMCache’s new P2P CPU memory sharing, built with Tencent, lets instances share KV cache directly using RDMA—no Redis, no S3, just peer-to-peer transfers coordinated by a lightweight controller. The results: 4× faster time-to-first-token, 5× faster query completion, and massive reduction in redundant prefill work for agentic workflows and multi-turn conversations.

If you’re running vLLM or SGLang on EKS with GPU node groups and fighting inference costs, this changes the economics. Instead of scaling horizontally and duplicating cache, you pool memory across the fleet and let pods pull cached prefixes from peers. The system assumes cache storage scales with active nodes (limited by cluster RAM, not external store size), which maps cleanly to EKS autoscaling patterns. LMCache is open source and integrates with existing inference engines, so you can deploy it as a DaemonSet or sidecar and start seeing gains without rearchitecting your serving stack.

⭐ Enhance Kubernetes High Availability with ARC and Karpenter Integration (by Ajay Desai and Jeremy Cowan)

When an AWS Availability Zone degrades, Amazon Application Recovery Controller (ARC) can shift traffic away from the impacted zone to keep apps running. EKS now supports ARC zonal shift and zonal autoshift, but Karpenter doesn’t natively integrate with it—so when ARC shifts traffic, Karpenter can still provision nodes in the bad zone, wasting capacity and breaking resilience. This AWS Containers blog introduces an open-source Kubernetes controller that bridges the gap.

The controller listens for ARC zonal shift events (manual or autoshift), extracts the impaired zone ID, and dynamically reconfigures Karpenter NodePools to avoid it. When the shift ends, it restores normal behavior. This ensures capacity provisioning aligns with traffic management, so you don’t launch GPU or compute nodes in zones that ARC just declared unhealthy. The controller is a reference implementation (not officially supported by AWS) meant as a stop-gap until Karpenter natively integrates with ARC. If you’re running Karpenter on EKS with multi-AZ workloads and care about resilience, this is worth testing—especially for GPU clusters where zone-level failures can waste expensive capacity. Full code on GitHub.

1. Announcements 📢

📢 Something Big Is Happening (by Matt Shumer)

Matt Shumer’s essay hit 30 million views in a day. Fortune and Business Insider covered it. It’s not predictions—it’s what already happened to him. He describes software in plain English, leaves for four hours, comes back to work that’s better than what he would’ve built. Not a demo. The actual thing. He says AI writes much of the code at AI labs now, models help build the next models, and the loop researchers warned about is already running.

I’m including this because 50 million people read it, teams are forwarding it around, and it’s the conversation right now. Whether it’s hype or real, it names something a lot of us managing EKS clusters and GPU fleets have been feeling but not saying: the gap between “AI helps me code” and “AI does the work while I review” closed fast. If you’re running infrastructure for these workloads or watching agents handle deploys you used to own, read it—not as a prediction, but as a snapshot of where things are.

Community & Career 🤝

🤝 Headlamp Karpenter plugin v0.2.0: EKS Auto Mode support (by Arshad Zackeriya)

Arshad Zackeriya just contributed EKS Auto Mode support to the Headlamp Karpenter plugin, so you can visualize and manage Karpenter resources directly in the Headlamp UI whether you're running self-managed Karpenter or letting AWS handle it. If you're using Headlamp as your Kubernetes dashboard and running EKS Auto Mode, this closes the gap—no more switching between tools to see node provisioning state, NodePool configs, or EC2NodeClass details. Clean open-source contribution from someone actually using the stack.

🤝 eksguide.com: A Free, Open-Source Amazon EKS Learning Resource (by Matt Lewis)

Matt Lewis just published eksguide.com—a free, open-source guide to Amazon EKS with curated blog posts, videos, workshops, and hands-on walkthroughs. The content covers EKS Auto Mode, storage classes, sample deployments, and Argo CD Capability, all based on building actual prototypes instead of rehashing docs. Matt’s observation tracks with what a lot of us have seen: despite simpler container options on AWS, EKS adoption keeps growing because teams want upstream-compatible, CNCF-conformant Kubernetes—and AWS has steadily reduced operational burden with managed node groups, add-ons, Auto Mode, and now Capabilities.

If you’re learning EKS or onboarding teammates, this is a solid starting point that filters out noise and focuses on practical patterns. It’s open on GitHub, so contributions are welcome. Bookmark material for anyone running or evaluating EKS.

🤝 CNCF Kubernetes Book Club: AI Agents in Kubernetes - Live Session (hosted by Carlos Santana & Sevi Karakulak)

The CNCF Kubernetes Book Club ran a live hands-on session on AI Agents in Kubernetes this past Thursday—real-time coding, debugging on the fly, and figuring things out together. The session continued the journey inspired by Lin Sun and Christian Posta’s book on practical AI agent patterns. If you missed it, catch up via the recording or join the #kubernetes-book-club channel on CNCF Slack to stay in the loop for future sessions. If you’re exploring how agents interact with Kubernetes APIs or trying to wire up agentic workflows on EKS, this series is worth following.

🤝 From MCP to Multi-Agents: The Evolution of Agentic AI (AWS Developer Podcast with Mike Chambers and Romain Jourdan)

Mike Chambers sat down with Romain Jourdan on the AWS Developer Podcast to unpack how last year didn’t go the way anyone expected—and what that means for agentic AI. The conversation covers the shift from Model Context Protocol (MCP) to multi-agent systems, what’s actually working in production versus what’s still hype, and where the ecosystem is heading next. If you’ve been following the agent explosion (or just trying to figure out what parts of it matter for real infrastructure work), this is a good grounded conversation from people actually shipping on AWS. Worth a listen if you’re thinking about how agents fit into your platform stack.

🤝 Setu: Bridging Kueue and Karpenter for Gang Scheduling (by Sanjeev Ganjihal)

Kueue and Karpenter don’t talk to each other. Kueue admits workloads, then Karpenter reactively spins up nodes—which means distributed training jobs can start before all GPU nodes are ready, wasting expensive compute time. Setu fixes this by sitting between them: it provisions the exact nodes needed (GPU type, instance, AZ) before Kueue admits the workload. If any node fails, everything rolls back. All-or-nothing gang provisioning.

This matters for distributed training (PyTorch, JAX, MPI) where partial starts break jobs, and for batch workloads where 60-90 second cold-start kills SLAs. If you’re running Kueue + Karpenter on EKS and fighting scheduler-autoscaler race conditions, Sanjeev Ganjihal just open-sourced the fix. Code on GitHub.

1. Highlights ✨

✨ Full-Stack Starter Template for Amazon Bedrock AgentCore

AWS just released a ready-to-deploy starter template for building AI agents on Bedrock AgentCore. It wires together AgentCore Runtime, Gateway, Memory, and Code Interpreter with a React frontend and Cognito auth—all defined with CDK. Instead of cobbling together AgentCore services from scratch or reverse-engineering examples, you get a working chat application that shows how the pieces integrate, with infrastructure-as-code you can deploy, modify, and learn from. If you’re exploring agentic workflows on AWS or trying to stand up a proof-of-concept quickly without manual API stitching, this is a sharp starting point that handles the plumbing so you can focus on agent logic and tooling.

✨Building a Multi-Agent SRE Assistant for Amazon EKS with Bedrock AgentCore (by Tolgahan Demirbaş)

When an EKS cluster goes sideways, you’re juggling CloudWatch logs, Prometheus metrics, and Kubernetes events across multiple tools trying to figure out what broke. This AWS blog shows how to build an SRE agent on Bedrock AgentCore that does the correlation for you. The architecture uses AgentCore Gateway to expose K8s, Prometheus, and CloudWatch APIs as MCP tools, with specialized agents for logs, metrics, and events working under a supervisor to investigate incidents end-to-end.

Instead of manually connecting a pod restart to a memory spike and the right runbook, the agent queries APIs, spots patterns, and surfaces recommendations. Tolgahan includes full source code on GitHub with Terraform to deploy the whole stack. If you’re running EKS and spending too much time hunting through scattered signals during incidents, this is a practical reference for how agents can actually help with reliability work—not just chat about it.

🎉 Sponsor Section

At the moment, we don’t have a sponsor for this edition, but we look forward to working with companies and organizations that support the EKS & AI Infrastructure community in future editions. If you or your company is interested in sponsoring, please contact us at 📧 thecloudtechforall@gmail.com

📝 Words from the Author

I’ve been thinking about something uncomfortable this week: we’re optimizing for speed when the real problem might be that we’ve forgotten how to stop.

Look at what we’re celebrating in this edition—emulation that makes LLM tuning 17× faster, agents that deploy infrastructure in 14 days instead of months, tools that turn natural language into finished code while you’re asleep. All genuinely impressive. All genuinely useful.

But here’s what keeps nagging at me: when did “faster” become the only axis that matters?

I don’t mean this as nostalgia. I’m not romanticizing the days of manually patching AMIs or SSH-ing into nodes at 2 a.m. Those things sucked, and I’m glad we’ve moved past them.

What I’m questioning is different.

We’ve built systems so fast that we’re deploying infrastructure we don’t fully understand, shipping features we didn’t stop to validate, and scaling complexity faster than our ability to reason about it. The bottleneck isn’t typing anymore—it’s judgment. And judgment doesn’t compress well.

I see this in my own work. I can scaffold an entire platform in an afternoon now, complete with observability, autoscaling, and GitOps wired in. It feels productive. But if I’m honest, half the time I’m not building something the team actually needs—I’m building what the tools make easy to generate. The path of least resistance used to be “do nothing.” Now it’s “generate something and see if it sticks.”

That’s not inherently bad. But it changes the failure mode.

We used to fail by moving too slowly. Now we’re starting to fail by moving too fast—shipping before we’ve thought, deploying before we’ve questioned, scaling before we’ve understood.

And the scary part? The tools don’t care. They’ll happily generate the next thing, and the next, as long as you keep prompting. There’s no built-in moment where the system says, “Hey, are you sure this is worth building?”

That’s supposed to be us.

So here’s the uncomfortable question I’m sitting with:

If we can build almost anything now, what does it mean to build the right thing?

Not “right” as in technically correct. Right as in: does this actually help? Does it simplify or just shift complexity? Will we be able to run this six months from now, or are we creating a maintenance nightmare disguised as velocity?

I don’t have clean answers. But I know this:

The skill that matters most right now isn’t learning the next tool or framework. It’s learning when to say no. When to delete instead of add. When to slow down and ask “why are we building this?” before the agent finishes generating it.

Speed is a superpower. But only if we remember that the most important decision isn’t how fast we can ship—it’s whether we should ship at all.

If you’re feeling the same tension between “we can do this so much faster now” and “I’m not sure we’re building the right things anymore,” you’re not alone. Maybe that discomfort is the signal we should be listening to.

Thank you for reading Everything about EKS & AI Infrastructure Newsletter #56

Happy Building! 😎