👋 Everything about EKS & AI Infrastructure Newsletter "#88" ☁️❤👨‍💻

Dear EKS & AI Infrastructure enthusiasts,

Welcome to Everything about EKS & AI Infrastructure #88.

A node group update finishes at 2 a.m. Every node is Ready. Every dashboard is green. Your manifest still promises an even spread across three Availability Zones. Except the pods never went back. One zone sits empty for nearly 17 hours, and nothing tells you, because nothing broke. The scheduler only checks that rule when a pod is first placed, and the autoscaler never touches pods that are running fine. Every component did exactly what it was designed to do. The only thing between “everything looks healthy” and “half your replicas are gone when a zone fails” is a setting someone chose, or never chose, long before tonight.

That’s the thread running through this edition. Yogesh Bangar found his subnets draining IPs far faster than his pod count explained, because a default quietly reserves a block of 16 per slot on every node. Ray Train on SageMaker HyperPod will happily see a node repaired and still end your job, because max_failures defaults to 0. And the newest vLLM images in AWS’s Deep Learning Containers are the Ubuntu ones, which AWS says it can’t promise to patch. The gap between what the platform does by default and what you assumed it does is where the incidents live.

And on the other side of the same coin: the speed work this week is loud. One AWS sample takes vLLM model loading from 28 minutes to 35 seconds with FSx for Lustre and GPUDirect Storage. Strands Decider lets a small model pick from your options instead of writing a sentence you then have to parse. FluxServe is betting that diffusion language models need their own serving engine, with 1.6x higher throughput in its own tests. The HyperPod Inference Gateway stops sending requests round-robin and scores each pod on six live signals, and AWS reports up to 82% lower first-token latency. Ryan Cormack puts a number on SnapStart against Provisioned Concurrency for Lambda containers, and the cost gap is bigger than you’d guess.

Then there is the question every platform team now gets: what happens when the thing running code in your cluster is an agent? The AWS Agent Sandbox puts a gVisor wall and a domain allowlist around it, and the EKS migration assessment agent shows the same building blocks doing something useful with your code. Around it all sit the practical calls: which EKS deployment model fits each edge site, how to flip a feature flag with an AppConfig sidecar without a redeploy, and Claude now available with in-country inference in India.

Performance Engineering in Modern AI Systems 🌩️

🌩️AWS Strands Decider 2B: a model that picks from your options instead of writing an answer

Think of the difference between an essay exam and a multiple-choice test. Most agent pipelines have steps that are really multiple choice: which of five queues does this ticket go to, should this request go to the cheap model or the expensive one, did the tool call succeed or fail. Today many teams run a full LLM for those steps, let it write a sentence, then parse the sentence and hope it did not invent a sixth option.

Strands Decider (Apache 2.0, built on Qwen3.5-2B) takes the model’s text-writing head off and puts on a small head of about a million parameters that scores the options you give it. The answer can only be one of your options, so there is no parsing and no made-up label. A scoring head also does not need the token-by-token generation loop, so it should be much cheaper and faster per decision. At 2B parameters the weights are roughly 4 GB in fp16, which fits on a small GPU. It joins OpenAI’s Decisions API and Simple Jev from Featherless as a new category, but it is the one you can run on your own cluster today.

The first test to run is simple. Take a few hundred labelled examples from a routing or classification step you already run, and compare agreement and p95 latency against your current LLM call. If the agreement holds, you have just removed a large model call from your hot path.

🌩️AWS sample: 48x faster vLLM model loading on EKS with FSx for Lustre and GPUDirect Storage

When a vLLM pod starts on EKS, it has to load the whole model into GPU memory before it can answer a single request. For a big model this is the slowest part of the start. The AWS sample tests it on an EKS cluster with one p5en.48xlarge GPU node (8 H200 GPUs) and DeepSeek-V4-Pro-0813, which has 892.7 GB of weights. With the default loader, loading took about 28 minutes. With the setup in this sample, it took 35 seconds, which is 48x faster. So if traffic spikes and you add a replica, the new GPU node waits under a minute instead of half an hour.

Here is how it works. The model files sit on Amazon FSx for Lustre, and the vLLM pod mounts them through the FSx CSI driver as a normal Kubernetes volume. Normally the data goes from storage to the CPU and then to the GPU, and the CPU is the slow part. NVIDIA GPUDirect Storage (GDS) skips the CPU and sends the data straight from FSx into GPU memory. Two things have to be right for it to work. You need the InstantTensor loader in vLLM, because the other GDS loader quietly falls back to the slow CPU path when the model is split across 8 GPUs. And you need to spread the model files across all of FSx’s storage servers when you first write them, or you only read from one server.

There are some trade-offs. A bigger FSx filesystem did not help much: doubling it gave only about 10% faster loading for twice the cost, so the smaller one was the better buy. The GDS setup also needs work on the GPU node itself, including building a kernel module, and the authors list a privileged pod as a known issue. It is a proof-of-concept benchmark that measures loading time only, not inference speed, so review it for security and cost before you use it for production.

🌩️FluxServe: an open-source engine for serving diffusion language models

Most language models write one word at a time. Take the sentence “The cat sat on the mat.” A normal model writes “The”, then “cat”, then “sat”, one step each, so six steps. A diffusion language model works differently. It starts with six blanks, “_ _ _ _ _ _”, and fills in several of them in the same step, then fixes its guesses. This is a simplified example, but it shows the idea: more words come out per step.

The problem is that tools like vLLM and SGLang are built for the one-word-at-a-time kind of model. They don’t take full advantage of a model that fills in many blanks at once. FluxServe is a new, free, open-source engine made for diffusion models, led by Youpeng Zhao. In the team’s own tests against SGLang, it was on average 1.6x faster. For one model, it produced over 1,000 tokens per second on a GPU while each user still got about 100 tokens per second. That is about 60% more than SGLang, which means one GPU can serve more users at the same speed.

If you run LLMs on EKS, this does not change anything today, but it is one to watch. The tests came from the FluxServe team, used one coding dataset, and ran on NVIDIA GH200 GPUs, so check it on the instances you use. It is also early, and support for newer GPUs and smaller model formats is still on the roadmap.

1. Starred Content ⭐

⭐AI on EKS: Agent Sandbox on EKS

Many AI agents write code and then run it. If that code runs in a normal pod, two things can go wrong. The code might break out and affect the node it runs on, or it might send your data to a website you never approved. This AWS Labs solution adds a stronger wall around the code and a list of websites the code is allowed to reach.

Here is a tiny example. An agent is asked to analyze a sales CSV, and the model writes some Python. The sandbox lets it install a package from PyPI and call Amazon Bedrock, because both are on the allowed list. When the code tries to send the file to evil.example.com, the lookup fails and nothing leaves the pod. When it tries to connect to a raw IP address, that gets blocked too. The project’s own test checks exactly these five outcomes: PyPI install passes, Bedrock call passes, code runs, unlisted domain blocked, raw IP blocked.

The choice you have to make is between two EKS modes. On standard EKS you get both protections: gVisor as the stronger wall, and Cilium for the website list. You also run more yourself, including a Karpenter node pool, and the first pod on each node takes about 60 to 90 seconds to start. On EKS Auto Mode, AWS runs more for you and domain filtering is built in, but gVisor is not supported, so the wall around the code is weaker. If you only want to add isolation to something you already run, start with the basic blueprint.

⭐Fix pod distribution drift in Amazon EKS with the Kubernetes descheduler

A node group update or a Spot interruption can empty all the workers in one AZ. The pods that were there move to the other two AZs, and when capacity comes back, they stay where they landed. Every node looks healthy, but the running pods no longer match the spread constraint in your manifest. The scheduler only checks that constraint when a pod is first placed, so nothing moves them back.

Ramya D, Himanshu Bansal and Tushar Mishra tested this with about 1,000 pods across three AZs. They took one AZ out, and the split went from even to roughly 500/500/0 in 90 seconds. After the AZ returned, the skew stayed at 227 for nearly 17 hours. Pods from autoscaled workloads recovered on their own, because new pods landed in the empty AZ. Workloads running at steady size never recovered, which is where most of your pods probably sit.

The fix is the Kubernetes descheduler, set to act on soft spread constraints. Two passes brought the split back to about 333/334/333 with 183 evictions, against a minimum of about 180. Every eviction is a pod restart, so put a PodDisruptionBudget on everything you include. Nothing here is specific to EKS, because the descheduler is a Kubernetes SIG project and runs on any cluster.

⭐One Amazon EKS, many edges: how to choose your edge container strategy on AWS

A customer asked the authors for one container standard across sites that look nothing alike. Some ran on VMware, some on Nutanix, some on bare metal, and a few had no reliable connection to the cloud. Sathish Kumar Prabakaran, Karan Thanvi and Ayush Ragesh say the mistake is asking which platform to run at each site, because you get a different answer every time and the fleet fragments. Instead, separate the standard from the deployment model. The standard is Amazon EKS, meaning the Kubernetes API, the workload definition, the tooling and the pipeline. It stays the same everywhere. Only the way it runs at each site changes. Supporting pieces like load balancers, storage and networking will still vary by site.

The method is two questions, in this order. First, can the site rely on a steady connection to an AWS Region? If not, because it is air-gapped, on a ship or has unreliable links, the answer is EKS Anywhere, which is self-managed. If the site is connected, ask what hardware is already there. If it has VMs or bare metal, use EKS Hybrid Nodes, which attaches your machines to an AWS-managed control plane. If it has nothing, or you want AWS to own the hardware, use EKS on AWS Outposts. In a simple example, an air-gapped plant gets EKS Anywhere, ten connected stores with their own servers get Hybrid Nodes, and a new warehouse with no servers gets Outposts. All three run the same manifests and the same Amazon EKS Distro, so it is one fleet hosted three ways. Their rule is that any operational work a site does not need is waste, so let AWS run the control plane wherever the site allows it.

Check the limits and prices before you commit. EKS Anywhere and Hybrid Nodes are Linux only, and only Outposts extended clusters run Windows nodes. On cost, EKS Anywhere is open source and free to run, with an optional enterprise subscription at $24,000 per cluster for one year or $18,000 per cluster per year for three. Hybrid Nodes costs $0.10 per cluster per hour plus a per-vCPU-hour charge starting at $0.020. Outposts has the same cluster charge plus hardware capacity fees and a multi-year commitment. For existing fleets, the post suggests starting with a few representative sites. Hybrid Nodes lets you attach machines you already own without replacing hardware, and you shift workloads over gradually.

⭐AI-powered EKS migration assessment with Amazon Bedrock AgentCore

Before an app moves to EKS, someone has to read its code, map its dependencies, find the blockers and estimate the work. Doing that by hand is slow, and different reviewers reach different answers. Pradip Kumar Pandey and Pratap Nanda built an agent that does the first pass. You give it an app’s source code and container files from OpenShift, Azure, WebSphere, JBoss, self-managed Kubernetes or Docker Swarm. It returns a readiness score from 0 to 100, a list of blockers by severity, and a migration plan for EKS. It runs on Amazon Bedrock AgentCore with the Strands Agents SDK. You can upload files, point it at a Git URL, or push to S3 from a CI/CD pipeline, and then ask follow-up questions in a chat.

Here is the example from the post, an order-management-service on WebSphere. The agent finds 12 plaintext passwords and recommends Secrets Manager with the External Secrets Operator. It finds a 1 TB shared NFS path and recommends EFS with the CSI driver. It finds host networking and recommends an ALB Ingress Controller and a NetworkPolicy. For sizing, an app with 4 CPUs and 16 GB becomes 3 replicas at 1 CPU and 2Gi each, with autoscaling between 3 and 10. Five crontab entries become Kubernetes CronJobs. The full report has eight sections, from the executive summary to a runbook, a risk table and YAML snippets for the Deployment, Ingress, ExternalSecret, CronJob, NetworkPolicy and IRSA ServiceAccount. The post estimates each assessment at $0.18 to $0.57 plus about $96 a month for the always-on UI.

The design is the part to copy if you are building agents yourself. The agent runs a chain of separate tools: clone the repo, extract the current state, scan the code, scan dependencies, check EKS compatibility, then write the plan. Each tool is a Python function marked with @tool, so adding a new check means adding a function, and Strands registers it automatically. AgentCore Runtime gives each assessment its own isolated session and scales to many apps at once. AgentCore Memory lets the agent recall patterns from earlier assessments, and AgentCore Observability records traces, tool latency and token use in CloudWatch.

⭐AWS Deep Learning Containers: tested images for vLLM, SGLang, Ray and vLLM-Omni

Running AI on EKS usually means building your own GPU image, then keeping CUDA, NCCL, EFA and the serving framework in step with each other. The AWS Deep Learning Containers repo does that work for you. It publishes pre-built Docker images for PyTorch, TensorFlow, vLLM, vLLM-Omni, SGLang, Ray and llama.cpp, with the versions listed in each release note. For example, Ray Train v1.0 ships PyTorch 2.13.0, CUDA 13.0.2, EFA 1.47.0, the AWS NCCL OFI plugin, GDRCopy, flash-attn, Transformer Engine and DeepSpeed already installed. One image runs as either the Ray head or a worker under KubeRay on any EKS cluster, including HyperPod-EKS. The Ray LLM image does multi-node LLM serving on EKS with Ray Serve and vLLM.

The two new AWS blog posts show the same idea with vLLM-Omni, which extends vLLM to audio, image and video models. In Part 1, an SM_VLLM_MODEL environment variable tells the container which model to load. It runs Qwen3-TTS, which streams speech back while it is still generating, over one two-way connection. In Part 2, the same image runs on two endpoints, one with FLUX.2-klein for images and one with Wan2.1-VACE for video. Both posts run on SageMaker, but the repo also publishes EC2 tags, such as omni-cuda-v1.7, without the SageMaker routing layer. Part 1 also uses SageMaker instance pools, a priority list of GPU types for one endpoint, and it needs quota for every type in the list.

Before you pull a tag, check the base OS. The README says the Ubuntu-based vLLM and SGLang images cannot be guaranteed security patches, because AWS has no Ubuntu Pro licensing for them. AWS recommends the Amazon Linux 2023 images instead. The newest highlights are easy to mix up here: vLLM 0.30.0 and SGLang 0.5.20 are Ubuntu images, while the AL2023 vLLM server image server-cuda-v2.5 also carries vLLM 0.30.0. Versions also move quickly. The AL2023 vLLM server image went from vLLM 0.27.1 to 0.30.0 in about a month, so pin exact tags and plan how often you update.

1. Announcements 📢

📢Amazon SageMaker HyperPod Inference Gateway: GPU-aware routing as a single EKS add-on

Most inference setups send requests to model server pods in turn, using round-robin, without checking what each pod is doing. A pod with a full KV cache or a long queue still gets its share, and that shows up as slow time to first token. The new HyperPod Inference Gateway replaces that with routing based on live inference signals. It deploys as one EKS managed add-on on existing HyperPod infrastructure and needs no application changes. AWS reports up to 82% lower first-token latency, and 97 to 98% lower p99 time to first token in mixed-hardware and burst traffic scenarios.

The detail that matters is how it chooses. The Endpoint Picker scores every model server pod on six signals: KV cache utilization, queue depth, LoRA adapter residency, prefix cache hit rate, predicted latency and running requests. In practice, a request can go to the pod that already holds its prompt prefix in cache or already has its LoRA adapter loaded, so the work isn’t repeated. A Body-Based Router reads the model name from each request and sends it to the right GPU pool, so one private endpoint per cluster serves many models. It works with any OpenAI-compatible server, including vLLM and SGLang.

Per-cluster routing is available now in Regions that support the HyperPod inference add-on. Cross-cluster and cross-region routing with a central fleet gateway, global rate limiting and cost-tier-aware traffic shaping are listed as coming soon.

📢SageMaker HyperPod adds managed Ray on EKS, and Gyanesh Kumar’s workshop shows how it recovers from a failure

Training on many GPUs fails in boring ways. A GPU errors out in hour six, or one pod loses its storage, and the whole job sits waiting. Ray on Kubernetes already worked through KubeRay, but teams still did a lot by hand: writing YAML, rebuilding images, opening the dashboard with port-forward, and setting up Prometheus and Grafana. HyperPod now handles much of that on EKS, using open-source KubeRay and standard Ray, so existing scripts run unchanged. It replaces failed nodes, spots jobs that stall without an error, saves checkpoints to local disk and then S3, and gives you dashboards and notebooks that connect to a live cluster.

The workshop shows this with a simple test. Training runs on two GPU nodes and saves a checkpoint after every epoch. Then the author reboots one node in the middle of the run. HyperPod fixes the node, KubeRay replaces the pod, and Ray Train restarts the job from its last checkpoint. For example, if the node fails during epoch 4, the job continues from the checkpoint at the end of epoch 3, so you lose a few minutes instead of the whole run.

The catch is in the training code. Ray Train’s retry setting, max_failures, defaults to 0, so by default one failure ends the job even though HyperPod repaired the node. The workshop sets it to 10 and tells the job to load the latest checkpoint each time it starts. So HyperPod keeps healthy hardware available, but your training code decides whether the job continues. Before you rely on automatic recovery, check that your teams set max_failures and save checkpoints.

Community & Career 🤝

🤝Yogesh Bangar on why his EKS subnets were running out of IPs

Yogesh Bangar noticed his VPC subnets were using up IP addresses much faster than his pod count could explain. Counting running pods didn’t help, because pods weren’t the problem. Each node reserves IP addresses ahead of time, before any pod needs them, so the number of IPs in use can be far higher than the number of pods running.

The cause was Prefix Delegation, which Bangar says is the default on EKS Auto Mode. With it on, every ENI slot reserves a block of 16 IPs (a /28) as soon as the node starts, whether or not 16 pods ever land there. A cluster with several NodePools full of lightly loaded nodes ends up with a lot of reserved IPs that nothing is using. His fix was a custom NodeClass with ipv4PrefixSize: "32", which gives out one IP per pod only when the pod is scheduled. Reserved IPs then matched the real pod count. The trade-off is slightly slower pod startup during traffic bursts, which was fine for his workloads.

To check whether you have the same problem, compare your node count against the used IP count for the subnet in the VPC console. A large gap points to this. Also, the built-in default NodeClass and the system and general-purpose NodePools can’t be edited, so you need to create your own NodeClass and point a NodePool at it.

🤝Ryan Cormack tests Lambda SnapStart for container images against Provisioned Concurrency

Lambda cold starts hurt most on apps that take a while to boot, such as a NestJS API on Node. Provisioned Concurrency fixes that by paying to keep environments ready around the clock. SnapStart now works with container images too, as long as the runtime client supports its restore API, and the Lambda Web Adapter does. Lambda runs your init once when you publish a version, saves a snapshot of that initialized environment, and starts every new environment from the snapshot. Cormack’s setup has no Lambda-specific code in the app. It is a normal web server in a container, with the Web Adapter and two routes that run before the snapshot and after the restore.

He ran the same NestJS image three ways, with 10 forced cold starts per memory size. SnapStart cut cold starts from about 1.2 to 1.4 seconds to 0.56 to 0.76 seconds. When he added about a second of CPU work at init, the regular function rose to 2.0 to 2.3 seconds, while SnapStart stayed at 0.4 to 0.55 seconds, because that work is already in the snapshot. Provisioned Concurrency answered warm requests in 2 to 8 ms. But when 10 requests arrived at once with one provisioned environment, the other 9 each paid a 1.75 to 3.0 second cold start. On cost, for a 1 GB function, one cached SnapStart snapshot is $3.90 a month. One provisioned environment is $9.63. In his example API, the monthly cost is $8.20 regular, $14.48 with SnapStart and $53.51 with Provisioned Concurrency set to 5. Provisioned Concurrency only breaks even with on-demand at about 80% utilization.

The trade-off will sound familiar to anyone running containers on EKS: you own the image now. AWS patches the managed runtimes, but it cannot patch what is inside your container, so you rebuild and redeploy on a schedule, automate base image updates and scan in ECR. There are other limits. SnapStart needs a version or alias, each publish takes about 45 seconds, and unique IDs, credentials and connections have to be recreated after restore. It cannot be combined with Provisioned Concurrency on the same function. It is also fast, not instant, with 400 to 600 ms against single-digit milliseconds, so strict per-request latency still points to Provisioned Concurrency.

1. Highlights ✨

✨Amazon Bedrock brings Claude Opus 5, Sonnet 5 and Haiku 4.5 to in-country inference in India

Teams in India that need data processed locally have had a hard time using Claude on Bedrock. Now Claude Opus 5, Claude Sonnet 5 and Claude Haiku 4.5 are available through India geographic cross-Region inference. You call Bedrock from your source Region, and the request is routed only between ap-south-1 (Mumbai) and ap-south-2 (Hyderabad). This is on top of the global cross-Region inference that was already supported. The point is capacity as well as location: your requests draw on both Indian Regions, which helps throughput during traffic peaks.

The detail to get right is what “in-country” means. Processing stays in India, but not in one Region, because prompts and outputs may move between Mumbai and Hyderabad. Customer data is not stored in the destination Region, and billing, quota, CloudWatch logs and CloudTrail records all stay in the source Region. Bedrock keeps no model inputs or outputs by default. The post adds that certain models require human review by AWS if content is flagged by automatic safety classifiers. If your compliance team needs a single-Region guarantee, this is the point to raise with them, and Guardrails and intelligent prompt routing also work with it.

For an app running on EKS, the change is mostly one value. Call bedrock-runtime in ap-south-1 and use the India profile ID, for example in.anthropic.claude-sonnet-5 or in.anthropic.claude-opus-5. It works with the Messages, InvokeModel and Converse APIs. The role your pods use also needs permission to invoke the geographic profile, and the post links the IAM setup. In the console, pick the profile marked IN, such as “IN Anthropic Claude Opus 5”.

✨Daniel Abib on feature flags in ECS and EKS with the AWS AppConfig sidecar

Containers are meant to stay the same once built, so changing how an app behaves usually means a new image and a redeploy. Environment variables need a pod restart, and reading config from an external store means writing your own polling code. Daniel Abib’s fix is to run the AWS AppConfig Agent as a sidecar in the same pod. The agent fetches the flags, caches them, and serves them on localhost:2772. Your app makes a plain HTTP GET, with no SDK and no credentials in the code. On EKS the agent gets its AWS access through IRSA, and it polls AppConfig every 45 seconds by default. If AppConfig is unreachable, it keeps serving the cached values.

Here is the example from the post. A flag called discount_enabled is set to off, with discount_percentage at 15. The backend runs on EKS with 3 replicas. The frontend runs on ECS. When you switch the flag on in AppConfig, the backend starts applying 15% off and the frontend shows the promotion, with no new image and no restarts. The post also shows multi-variant flags, which give different values to different users, for example 15% for platinum customers, 10% for new users and 5% for everyone else.

A flag change still takes time to reach your pods. The post rolls changes out gradually, with a sample strategy of 15 minutes, and the agent polls on top of that, so a flip is not instant. The best practices are to roll out gradually and use CloudWatch alarms to roll back automatically. Add validators so a bad value, like a discount over 100, never ships. Keep “off” as the safe default, and read the flag from the agent on every request instead of caching it in your code. The post also says the sample is not production-ready, and you should add HTTPS, authentication, private networking and non-root containers.

🎉 Sponsor Section

At the moment, we don’t have a sponsor for this edition, but we look forward to working with companies and organizations that support the EKS & AI Infrastructure community in future editions. If you or your company is interested in sponsoring, please contact us at 📧 thecloudtechforall@gmail.com

📝 Words from the Author

The popular prediction is that agents will run the platform, so platform teams will shrink. I think it goes the other way. When software writes and ships code faster than people can read it, whatever stands between that code and your production cluster stops being a convenience and becomes the only thing that matters.

For ten years, platform engineering meant a paved road for developers: templates, a portal, a golden path. The developer was a person who read the docs and followed the signs. An agent does not. It tries the road, then the side door, then the window, because getting the task done is exactly what we asked of it. So the platform’s customer changes, from someone who follows the path to something that tests every edge of it.

That moves the work from the portal to the rules. A golden path becomes a boundary: what this agent can call, which domains it can reach, what credentials it gets, how long it lives, and who approves when it asks for more. Look at this edition. A domain allowlist around agent code, a retry setting that decides whether a job survives, a default that decides how many IPs a node grabs. None of these is a new product category. It is platform work, pointed at a different user.

On EKS, most of the pieces already exist: namespaces, short-lived service account roles, NetworkPolicy, quotas, admission policy and PodDisruptionBudgets. The hard part is not building something new. It is turning the defaults from allow to deny and checking that they actually hold. My bet is that the best platform teams of the next few years will look less like portal builders and more like the people who write the rules an agent has to ask permission to change.

Happy building. 😎