👋 Everything about EKS & AI Infrastructure Newsletter "#92" ☁️❤👨‍💻

Dear EKS & AI Infrastructure enthusiasts,
Welcome to Everything about EKS & AI Infrastructure #92.

If one theme ran through this week, it was giving AI agents a safe place to run code. Google donated Agent Substrate, gVisor and OpenRL to the CNCF. Baseten launched Sandboxes with microVM isolation and instant pause and resume. Agent Sandbox v1.0.10 shipped with new file operations and a network policy change you should check before you upgrade. Different teams are building the same thing: an isolated, stateful home for every agent.

On the EKS side, we look at keeping things running when they fail. A two-part guide walks through EKS Auto Mode high availability, from a single pod to a whole Availability Zone. An AWS sample shows Auto Mode replacing a broken GPU node mid-training and resuming from a checkpoint in about 12 minutes. A community experiment shows how one CPU request can make Auto Mode add a node.

For inference, RunAnywhere built its own engine to serve a 320B model on a single Trainium2 machine. A community builder compared six serving engines on one NVIDIA L4. Kinference adds a traffic controller in front of vLLM. We also cover AI agents that help run Kubernetes, from incident response designs to Rescale’s test triage agent on EKS.

And if you’re celebrating, Happy Diwali from all of us. 🪔

Let’s dive in.

Performance Engineering in Modern AI Systems 🌩️

🌩️Serving GLM-5.3-Flash on One Trainium2 Box with NKI Layer Megakernels
Aman Swar and Sanchit Monga, RunAnywhere

Coding agents send very long prompts, and most of each prompt is text the model has already seen. Serving many of them at once on a big model is hard. GLM-5.3-Flash is a very large model: 320B parameters, of which only about 18B are used for each token, and it can read a million tokens of context. The authors say that when they started, no AWS-supported or major open-source engine could run it on AWS Trainium, Amazon’s own AI chip. So they built one, called trnRT, and ran it on a single trn2.48xlarge machine with 16 Trainium2 chips.

Their main idea is to stop the chip from pausing between small jobs. Normally one layer of the model runs as a chain of tiny programs, called kernels. Between each pair, the chip waits and hands over results. They merged each half-layer into one kernel. At batch size 8, the time to produce each step dropped from 32.2 ms to 24.2 ms, about 25% faster. They also keep the model’s earlier context cached in memory, so a returning agent doesn’t start from zero, and a router sends each agent back to the same instance.

Starred Content ⭐

⭐ Building highly available and resilient applications on Amazon EKS Auto Mode
Camilo Cortes, AWS Senior Solutions Architect

EKS Auto Mode looks after your nodes, but keeping your app up when something fails is still your job. This two-part guide goes from small failures to big ones. Part 1 covers a pod or a node. The foundation is to spread pods across zones and keep spare room in each zone, so the other zones can take the load without waiting one to two minutes for a new node. Probes handle a pod that stops responding. A pod that is up but returning errors is harder, and the Application Load Balancer’s Automatic Target Weights send it less traffic. When a node fails, Auto Mode repairs it on its own. That repair ignores Pod Disruption Budgets and pauses if more than 20 percent of the nodes in a NodePool are unhealthy. So spread replicas across nodes too. Auto Mode also replaces healthy nodes on purpose, for example when a new node image arrives about once a week. Pod Disruption Budgets and NodePool disruption budgets set the pace, and a clean shutdown stops requests from being dropped.

Part 2 covers a whole zone going bad. Set alarms per zone with CloudWatch anomaly detection. Then take the zone out of traffic with ARC zonal shift, which works at the cluster level in EKS Auto Mode. You turn it on once per cluster. During a shift, the zone’s nodes get a taint, its pods leave the EndpointSlices, and load balancer targets there are deregistered. No pods are evicted or nodes terminated. The guide also covers scaling and rollouts while a zone is out, and how to rebalance pods afterward.

It ends with three choices that each cost something: who decides when traffic moves, how strictly pods stay spread, and how much spare capacity you pay for. The catch: spare capacity is billed even when idle, and the descheduler and one spread setting (nodeTaintsPolicy: Honor) are not documented by AWS for Auto Mode. Test both in a non-production cluster first.

⭐ sample-eks-gpu-node-auto-repair
aws-samples (AWS)

Training an AI model on GPUs can take hours, and a GPU can break halfway through. This AWS sample shows what EKS Auto Mode does when that happens. You don’t install anything extra. The authors tested it on EKS Auto Mode with Kubernetes 1.36. They break a GPU on purpose with a fault called XID 79, which means the GPU has stopped responding. A monitoring agent on the node notices it and marks the node as unhealthy. Auto Mode waits 10 minutes, and you can’t change that wait. Then it removes the bad node and starts a new one. The training job restarts and carries on from its last saved checkpoint, so it doesn’t begin again from zero. Afterward, a command called kubectl ekslogs collects the node’s logs without SSH access.

The timings are the useful part. The fault is noticed in under a second. The new node is ready in about 90 seconds. Everything together, from the fault to training again, takes about 12 minutes. There are two versions to try. The small one fine-tunes a small model on cheap single-GPU g6e nodes. It takes about 35 minutes to set up and costs roughly US$6 to 8 an hour. The large one fine-tunes a bigger model on two p5en.48xlarge machines.

The catch: your training must run longer than those 12 minutes, or the node may be removed as idle before the repair starts. Don’t request hugepages-2Mi in these pods, because it blocks new nodes from starting, including the replacement. Only the XID 79 fault is tested, and p5en machines are costly and hard to get. Run the clean command when you finish so you stop paying.

⭐ Managing a hybrid fleet with Nutanix Kubernetes Platform and Amazon EKS (in Spanish)
Guillermo Ruiz Esteban (AWS) and Paul Zerdilas Herrera (Nutanix)

Many companies run some Kubernetes clusters in their own data center and new ones on EKS. As the number of clusters grows, upgrades, certificates and security rules get harder to keep the same everywhere. This two-part guide uses Nutanix Kubernetes Platform (NKP) as one central “hub” that manages both. Part 1 connects an existing EKS cluster to the hub, which the authors call “attached.” Your own tools, such as Terraform or eksctl, keep owning the cluster, and the hub sends shared add-ons to it using Git (GitOps with Flux CD). As an example, they push a Gatekeeper policy that only allows container images from approved registries, and test it first in dry-run mode before blocking anything. App traffic does not go through the hub, so if the hub goes down, your apps keep serving users.

Part 2 covers three more EKS jobs. It gives one app read access to only one S3 folder using EKS Pod Identity. It creates a brand-new EKS cluster from the hub using Cluster API, with an IAM role that can only create clusters in one workspace. It also gives pods their own IP range, separate from the nodes, using VPC CNI custom networking. Each step ends with a check that what should fail actually fails.

The catch: you need NKP, and its license tier depends on what the hub manages. Neither part covers upgrades or monitoring in depth. Changing an EKS cluster’s access mode to API_AND_CONFIG_MAP can’t be undone. Both parts are in Spanish.

1. Announcements 📢

📢Baseten launch week: Sandboxes, Grid and Loops
Amir Haghighat, Paul Sinai, Nicolas Lecomte, Sai Maddali, Rachel Rapp, Kenzie Amack, Kimbrian Canavan, Raymond Cano, Ardin Lo, Baseten

Baseten, an AI inference company, launched three products in one week to cover more than serving models. Sandboxes is the one most relevant to platform teams. An agent that runs code needs a safe place to do it. Each sandbox gets its own microVM, a small virtual machine with its own kernel, which Baseten says isolates better than containers. When the agent goes idle, the sandbox saves its files, memory and running processes and drops compute to zero. It resumes in about 25 ms, and Baseten says it can scale to thousands of sandboxes within seconds. Two more pieces are still to come: Agent Drive, a shared filesystem, and Agent Proxy, which controls what agents can reach from outside the sandbox and injects credentials so the agent never sees them.
Here is why it matters on EKS. Normal pods on a node share that node’s kernel, so a microVM per agent is a stronger wall between untrusted code and everything else. The same three questions apply to any agent platform you build: where agent state lives, how fast a new sandbox starts, and who controls the agent’s network and credentials. Baseten’s answers are one reference point

Grid sits in front of coding agents such as Claude Code and Codex. It is one gateway for model access, budgets and safety checks. A router picks the model for each task, and for models Baseten serves, it sends long agent sessions back to a warm KV cache (the model’s saved memory of the prompt). Loops is now generally available. It lets you fine-tune and run reinforcement learning on open models through a Tinker-compatible API at 256k context, billed per token. A trained adapter can go to production with one click, or you can download the weights as Safetensors.

📢 Open infrastructure for autonomous agents comes to CNCF

Google is donating three projects it started to the Cloud Native Computing Foundation (CNCF): Agent Substrate, gVisor and OpenRL. All three were accepted into the CNCF Sandbox last month and are now onboarding with community maintainers. The reason is a new kind of workload. Agents run code for a few seconds, then wait on a model, an API or a person, and they run commands nobody reviewed. In standard Kubernetes, one Pod per agent holds memory and CPU while it waits. Containers also share the node’s kernel with everything else on the node.

Each project covers one part of that problem. Agent Substrate keeps each agent’s memory, files and identity apart from the machine that runs it. When an agent pauses, its state is saved and the machine goes back to a warm pool, and the agent resumes on any free one. You choose Cloud Hypervisor microVMs or gVisor for isolation. gVisor is an application kernel written in Go that handles system calls in userspace, so untrusted code never talks to the host kernel. OpenRL is a self-hosted reinforcement learning loop on Kubernetes that uses vLLM for sampling and llm-d for routing and GPU time-slicing. Weights and data stay on your cluster.

The open source part is the real news. Agent Substrate is open source for any Kubernetes cluster, which includes EKS, though the page never mentions or tests EKS. gVisor needs no nested virtualization and runs on any Linux node in any cloud. Moving to the CNCF also matters for gVisor itself. Ant Group, Modal and Tines have committed to join its maintainers. The page says neutral governance opens the way to upstream I/O improvements that Linux maintainers declined while Google was the sole owner. The catch: the performance numbers are Google’s own claims, with a restore time under 500 ms and 10× density but no published method. Sandbox is the earliest CNCF stage. OpenRL has validated only Gemma and Qwen, and TPU support is on the roadmap.

Community & Career 🤝

🤝Six Serving Engines on One EC2 G6: Gemma 4 E2B on an NVIDIA L4
xbill9, AWS Community Builder

Say you have one small AI model and one GPU. Which tool should you use to serve it? The author tested six on one AWS g6.4xlarge, which has a single NVIDIA L4 GPU: llama.cpp, vLLM, SGLang, mistral.rs, PyTorch and LiteRT-LM. Every tool got the same model, the same questions and the same load test, from 1 to 64 requests at once. The quality of the answers was almost the same on all of them, so the real difference is speed.

With one request at a time, five of the tools are about equally fast, at roughly 135 to 145 tokens per second (a token is a small piece of a word). The difference shows up when many people use it at once. At 64 requests, vLLM produces 3,716 tokens per second, SGLang 2,967, mistral.rs 1,825 and llama.cpp 1,056. Adding Google’s small helper model, which guesses ahead so the main model can check faster, pushes vLLM to 4,516. That works out to about $0.08 per million output tokens at full load, based on the on-demand price. The helper model doesn’t always win. It made mistral.rs a little slower, and it made SGLang slower at 64 requests.

The catch: this is one small model, one GPU and one set of test prompts. The cost is a calculation from the price list, not a real bill. The tools ran directly on EC2, not on EKS, so treat the numbers as a starting point before you plan your pods.

🤝 AI-Powered Kubernetes Incident Response on AWS: A Production-Ready System Design
Muhammad Umair Ashraf, Cloud Data Engineer

When something breaks on EKS, one failure can fire hundreds of alerts. Then an engineer has to check events, logs, metrics and runbooks by hand to find the cause. This article designs a system where an AI agent does that checking first. Prometheus and Alertmanager send alerts to SNS and then SQS, with a dead-letter queue for failures. A DynamoDB table gives each incident a fingerprint, so ten pods failing from one cause start one investigation, not ten. Step Functions runs the workflow. A Strands agent on Amazon Bedrock then looks at pod status, logs, events and metrics through read-only MCP tools for EKS, Grafana and CloudWatch. It also reads your runbooks from a Bedrock Knowledge Base. It returns a report that separates what it observed from what it only suspects.

The safety design is the strongest part. The agent only proposes an action. A separate policy engine checks it before anything runs: is the environment approved, are enough healthy replicas left, and has the restart limit been hit. Risky actions go to a human for approval. The agent gets no write access, and logs and alert text are treated as data, never as instructions. In the author’s worked example, the agent suggests restarting a crash-looping pod, but the policy engine rejects it because too few replicas are healthy. The on-call engineer gets the evidence and a likely cause, which is a database connection failure. The author advises starting with read-only investigations in non-production, then adding a few low-risk fixes. Past incidents can be replayed to check whether the diagnosis was right.

🤝 Kubernetes CPU Requests in Amazon EKS Auto Mode
Jesus Aguirre, AWS Community Builder

A Pod can sit in Pending even when your nodes are healthy, because Kubernetes places Pods by what they ask for, not by how much CPU they actually use. Jesus Aguirre tested this on EKS Auto Mode. A Pod requesting 500m CPU (half a CPU) ran on an existing c6a.large node. He recreated it asking for 4 CPUs. No existing node could fit that, so the Pod went Pending with a FailedScheduling event saying Insufficient cpu. Auto Mode then created a NodeClaim, which is Kubernetes’ record of a request for a node. It launched a c6a.2xlarge On-Demand instance, the node became Ready, and the Pod was scheduled. The container only ran sleep 3600, so it never used four CPUs. The request alone caused the new node.

The useful part is the trace you can repeat in your own cluster. Start at the Pod’s events, then the NodePool that limits what can be launched, then the NodeClaim. Follow its provider ID to the Node, and use that to find the EC2 instance. The article also warns that an instance’s advertised vCPUs are not what Pods can use, because system reservations and other Pods’ requests take a share. It shows a custom NodePool that limits nodes to chosen instance types, and a Pod nodeSelector that targets it.

1. Highlights ✨

✨ Kinference: a traffic controller for LLM inference on Kubernetes
Buoyant, Inc.

When several teams share GPU pools running vLLM, someone has to decide where each request goes, stop one team from taking all the capacity, and track who used how much. Kinference is the layer that does this on top of a vLLM setup you already have. vLLM still runs the models, and Kinference decides where the traffic goes.

Apps talk to a gateway that speaks the OpenAI and Anthropic API formats, so they only change their endpoint. Behind it, a controller sends each request to a vLLM pod, using live queue length and KV cache state (the KV cache is the model’s short-term memory of a prompt). It also limits tokens per team, tracks cost per team, and checks latency against the targets you set. The page’s policy examples include sending simple prompts to cheaper models, bursting to the cloud, keeping prompts with personal data local, and running batch work only on idle GPUs. You write these rules as Kubernetes custom resources (InferencePolicy, InferencePool and InferenceWorkerSet). It installs with Helm and comes with Prometheus and Grafana dashboards. It works with any model vLLM can run.

✨ How Rescale built an AI triage agent with Strands Agents and AI-DLC
Isha Doshi, Ayush Ahuja, Evan Richardson, Gustavo Frizzo, Jimin Kim, Shree Krishnamurthi, Mark Whitney, Michael Dillon and Matt Martin

Every night Rescale runs hundreds of end-to-end tests, and these tests decide whether the weekly release can go out. When tests failed, engineers had to sort each failure by hand, collect logs from several places, and then start debugging. Until that was done, the release could be stuck for several teams. So they built an AI agent with the Strands SDK that runs as a container on Amazon EKS, as one more step in their Argo Workflows pipeline. For each failed test, the agent uses Claude on Amazon Bedrock to decide whether the cause is infrastructure, a code bug, a flaky test or the environment. It looks up similar past failures, reads pod logs through MCP, and writes a report with the files, functions and a suggested fix. It does not change anything on its own. An engineer reviews every report.

The team built it in a five-day workshop with AWS using AI-DLC, a step-by-step method for building software with AI. Two plans changed along the way. The agent was meant to run on Amazon Bedrock AgentCore but moved to EKS, and log lookup moved from a third-party service to Amazon CloudWatch. By day four the whole pipeline ran in staging.

The results are honest. At first the agent agreed with human triage only 44% of the time, and today it agrees 68% of the time. The time a release is blocked while failures are analysed dropped by 20%. The catch: about one in three failures is still sorted differently from a human, so people still check the reports.

✨ Agent Sandbox v1.0.10
kubernetes-sigs/agent-sandbox maintainers

AI agents write and run their own code, so each one needs a safe, separate place to run it. Agent Sandbox is an open-source Kubernetes project that gives every agent its own sandbox Pod. It has two parts. Inside each sandbox runs a small helper called sandboxd, which runs commands and handles files for the agent. In front of all the sandboxes sits the sandbox-router, which takes each incoming request and sends it to the right sandbox.

This release adds three main things. First, the router now accepts gRPC-Web, the version of gRPC that browsers can use, so a web app can talk to a sandbox directly. Second, sandboxd can now create folders, list a folder and everything inside it, move and copy files, write a file only if it doesn’t already exist, and unpack a tar archive. Third, sandboxes that use Kata or gVisor for stronger isolation run inside a small VM, so a normal kubectl port-forward can’t reach them. The Go SDK can now reach them through the router instead, from a laptop or a CI runner. There are new examples too, including Hyperlight micro-VMs and reinforcement learning shared by several users.

The catch is a change to the router’s default network rules, and you need to act on it. Before, the router could send traffic to sandboxes only on ports 8888 and 9090. Now it can use any TCP port, but only to Pods that carry the label agents.x-k8s.io/sandbox-name-hash. So if your router sends traffic to any Pod without that label, those requests will start to fail, and you need to add your own NetworkPolicy to allow them. The rules also now let traffic reach ports 443 and 6443 (the Kubernetes API server) at any address. The project made this change because the old rules broke on some clusters, for example with Cilium. To lock this back down, set networkPolicy.apiServerCIDRs in the Helm chart, or add a rule that allows only your API server’s addresses. Also note that the new file actions need sandboxd v1.0.10. Older versions quietly do the wrong thing, for example writing a plain file when you asked for a folder, so upgrade sandboxd first.

✨ New digital courses on how to optimize AI compute and containers on AWS
Sophy Chen and Art Baudo, AWS Training and Certification

AWS Skill Builder added two free, advanced-level courses for platform and ML teams. The first, Boost Amazon EKS Efficiency, runs 1.5 hours. It covers three ways to cut EKS cost. EKS Auto Mode automates cluster infrastructure. AWS Graviton chips run containers for less, and the course covers migrating workloads to them. EC2 Spot Instances use spare capacity, and the course covers how to handle Spot interruptions. The course also covers Kubernetes scaling and ends with a demonstration of all three together.

The second, Assembling the Complete AI Stack, runs 2 hours. It helps you choose AWS hardware for each stage of an AI pipeline, from data processing to training and inference. That includes Graviton, Trainium, Inferentia, GPU instances, and Intel and AMD instances. It teaches a way to pick instances by model size, plus cost and capacity planning for EC2. The hands-on parts are building an internal AI assistant, a startup inference deployment, and model customization.

The catch: both are advanced, so expect to need Kubernetes and EC2 basics. The AI hardware course is about EC2 choices in general and is not EKS-specific. The post doesn’t mention prerequisites or certificates.

🎉 Sponsor Section

At the moment, we don’t have a sponsor for this edition, but we look forward to working with companies and organizations that support the EKS & AI Infrastructure community in future editions. If you or your company is interested in sponsoring, please contact us at 📧 thecloudtechforall@gmail.com

📝 **Words from the Author
**Our code talk on agent sandboxing is finally coming together. For the past few weeks, most of my evenings and weekends have gone into it: building the demos, breaking them, fixing them, and then breaking them again in new ways. Seeing the whole flow run end to end for the first time felt like a small win, and I can’t wait to share it with you.

A code talk is different from a normal session. There are very few slides to hide behind. You write and run real code in front of the room, so every demo has to work live, on stage, on conference Wi-Fi, with the clock running. That means you don’t just prepare the content. You also prepare for everything that could go wrong, like a cluster that takes longer to come up, an image that won’t pull, or a network rule that changed in the latest release. Agent sandboxing moves fast too, so part of the work has been keeping up with new releases and making sure what we show is still current on the day.

Working on this reminded me of something we don’t talk about enough. When you watch a 30 or 60 minute session, you only see the finished version. You don’t see the weeks of research behind it, the outline rewritten again and again, the demo that failed the night before, the dry runs where the timing was way off, or the hours spent trimming a story until it makes sense to someone seeing it for the first time. One clean hour on stage often takes many, many hours off stage.

I have been doing this for a while now, as a speaker and as an organizer of the AWS User Group Vadodara for more than six years. So I can say from experience that the effort is much more than most people realise. It is the same for every speaker you see, whether at a big conference or at a small local meetup on a Saturday morning. Most of them do it alongside a full-time job, unpaid, simply because they want to share what they learned.

So here is a small request. The next time you attend a talk, a workshop or a meetup, take a moment to thank the speaker. Send them a message, leave feedback, ask a good question, or share something you tried after their session. It costs you a minute, and it means a great deal to the person who spent weeks getting it ready. If you have been thinking about giving a talk yourself, start small, start local, and know that the community will be there to support you.

Finally, to everyone celebrating, Happy Diwali! 🪔 Diwali is about light winning over darkness, and new beginnings. I hope this festival brings you and your families happiness, good health and a bright year ahead. Take a real break from the laptop, enjoy the sweets, light a few diyas, and spend time with the people who matter most. The clusters will still be there when you get back.

See you in the next edition.

Adit