đ Everything about EKS & AI Infrastructure Newsletter "#85" âď¸â¤đ¨âđť
Memory, cold starts, and what infrastructure keeps forgetting â plus a big week for AWS User Group Vadodara.
Dear EKS & AI Infrastructure enthusiasts,
Welcome to Everything about EKS & AI Infrastructure #85.
A lot of what we build in AI infrastructure comes down to one simple problem: computers forget things they shouldnât, and remembering the right things at the right time is expensive. A GPU forgets a modelâs weights the moment it restarts, so it has to download them all over again. A load balancer forgets which server already âknowsâ your question, so it sends it somewhere that has to start from scratch. A crashed pod takes its own error logs down with it before anyone can even look at them.
This issue is full of people fixing exactly that. Amazon SageMakerâs new routing feature makes sure repeat requests land on the same server so it doesnât have to redo work it already did. A new caching feature on SageMaker HyperPod keeps model files sitting locally on the machine, so restarting a server takes seconds instead of 30 minutes. A new AWS tool grabs a crashed podâs evidence and saves it before it disappears, so someone (or something) can actually figure out what went wrong. We also cover a fresh GPU benchmark (the new G7 instances vs older ones), a deep look at splitting AI inference into two coordinated pieces on Kubernetes instead of one, a builder who rebuilt his entire AI-serving setup three times chasing lower cost, a tool that makes Spark run faster on GPUs without any code changes, a real security pattern for AI agents built around an actual hack that happened, and a good discussion on why specialized AI engines keep losing out to the popular ones like vLLM.
Performance Engineering in Modern AI Systems đŠď¸
đŠď¸Prefix-aware routing on Amazon SageMaker Inference
Hereâs the setup: every prompt to an LLM usually has a big fixed chunk at the start (system instructions, a document, chat history) and a small changing chunk at the end (the actual user message). Serving frameworks like vLLM can cache the computation for that fixed chunk so it doesnât get redone on every request. But this only works if the same fixed chunk keeps arriving at the same machine. If youâre running multiple instances behind a load balancer and it spreads requests randomly, that same 3,000-token chunk shows up on a different machine each time, so none of them ever gets to reuse the cache. Amazon SageMakerâs new PREFIX_AWARE routing fixes exactly this: it looks at the start of each request and consistently sends matching ones to the same instance, so that machineâs cache actually gets used.
The results show why this matters: on Llama 3.1 70B, time to first response dropped by up to 77% and the cache started getting reused in over 80% of requests, up from about 25%. The one catch: this only works if your requests are formatted identically every time (same field order, no extra spaces), because the routing looks at the raw text of the request, byte for byte, to decide what counts as a âmatch.â If your app formats the same prompt slightly differently between requests, theyâll get routed to different machines and youâll lose the benefit. Worth checking if youâre running chatbots, RAG apps, or coding assistants on SageMaker.
đŠď¸Benchmarking small LLM inference on SageMaker AI: G7 vs G5 and G6
This post benchmarks two 30B MoE models across the older G5/G6 GPU instances and the new Blackwell-based G7, and the interesting bit is that G7 wins even though itâs running on half the GPUs and less total memory than the others. The reason comes down to how MoE models actually work: when generating each token, they only wake up a small subset of âexpertâ sub-networks instead of the whole model. Because of this, the bottleneck isnât how much raw compute the GPU has, itâs how fast it can move data in and out of memory. G7 simply has more memory bandwidth, so it doesnât sit around waiting on data as much.
The second reason G7 pulls ahead is precision support. Models can be shrunk down to run in a very compact 4-bit number format (called NVFP4) to save memory and speed things up, but only Blackwell GPUs (G7) can actually run that format natively in hardware. On G5 and G6, the same 4-bit weights still work, but the GPU has to do extra work to handle them since it wasnât built for that format, so the potential speedup mostly goes unused. The practical takeaway: if youâre picking GPU instances for a mixture-of-experts model, itâs not just about âmore GPUsâ or âmore memory,â itâs about whether the hardware actually understands the number format your model is running in.
đŠď¸Deploying Disaggregated LLM Inference Workloads on Kubernetes
When an LLM answers a prompt, it actually does two very different jobs: first it reads and understands your whole prompt (this is called âprefill,â and itâs compute-heavy, like a burst of hard math), then it generates the answer one word at a time (this is âdecode,â and it depends more on fast memory access than raw compute). Normally both jobs run on the same GPU, which is wasteful because the GPU is never great at both at once. âDisaggregated servingâ means splitting these into two separate services, each running on hardware suited to its actual job, so nothing sits idle waiting on the wrong kind of work.
The tricky part is coordinating these two services so they still work as one system. The post compares two ways to do this on Kubernetes. The simple way (using something called LeaderWorkerSets) treats the âprompt-readingâ service and the âanswer-generatingâ service as two totally separate things, so Kubernetes doesnât know theyâre supposed to work together, meaning you have to manually keep them in balance yourself, or things break, like if you scale up the prompt-reading side but forget to scale up the answer-generating side, the answers have nowhere to go and everything backs up. NVIDIAâs newer tool called Grove fixes this by treating both services as parts of one system from the start, so scaling and coordination happen automatically and safely, without needing someone to manually keep the two halves in sync.
- Starred Content â
âEKS Node Diagnostics MCP with DevOps Agent Integration
AWS DevOps Agent is good at investigating things it can already see, like a crashing pod or a CloudWatch metric spike. But a lot of what actually goes wrong on an EKS node lives outside that view: kubelet logs, container runtime logs, firewall rules, network config, kernel messages. None of that is visible through the Kubernetes API or CloudWatch. This solution builds a bridge for that: a custom MCP server that tells the agent how to reach into the node itself (using SSM, AWSâs remote command tool) and pull over 20 different log sources.
The interesting part is how the pipeline is built, not just the idea. The agent asks for logs from a specific instance, an SSM script grabs everything and zips it into S3 (encrypted), and then a processing step goes through the logs, tags each error with a severity level, and gives it a stable ID before handing it back to the agent. That last step matters a lot: dumping raw logs on an AI agent doesnât work well, but handing it pre-sorted, labeled findings does. Itâs still a proof of concept, so test it somewhere safe before pointing it at anything that matters.
âMulti-Model LLM Inference on EKS, Across Three Generations of Silicon â Chris Jaimon
Chris Jaimon built a system to serve LLMs cheaply on spot instances, and rebuilt it three times as he ran into real problems. Round one used AWSâs Inferentia2 chips because theyâre cheap and easy to find on spot, but he hit two walls: each user only got about 8 tokens/sec of output (he wanted 10+), and whenever a spot instance got killed, bringing a new one back online took almost an hour, because it had to re-download the model and recompile it for the chip. His fix for the cold start was simple but effective: pre-load the already-compiled model into S3 ahead of time, so a new instance just pulls the ready-made version instead of rebuilding it from scratch. That took the recovery time from an hour down to minutes.
Since scaling up Inferentia2 further wouldâve gotten too expensive, he switched chips instead. He moved a big mixture-of-experts model (Llama 4 Maverick) to Trainium2, which has enough onboard memory to hold the whole model without needing to shrink it, and added a third setup using regular NVIDIA GPUs (L40S) for a smaller, faster model. Now traffic gets split between the two live models through a single gateway (Envoy AI Gateway): about 80% goes to the fast small model, 20% to the bigger one, and if a spot instance running either model goes down, the gateway automatically shifts traffic away without the app noticing. The thing that stands out most: every throughput number in the post came from him actually running the benchmark himself (using a tool called LLMPerf), not from trusting whatever the chip vendor claims.
- Announcements đ˘
đ˘Ray Serve Deep Learning Containers on Amazon EKS
TorchServe is officially unmaintained now, no more security patches, no more compatibility fixes as PyTorch and CUDA move forward. That leaves any team still on it fully responsible for keeping the GPU stack, framework, and serving layer aligned themselves. AWSâs answer is the new Ray Serve DLC: a prebuilt, tested container that already bundles PyTorch, the CUDA runtime, and Ray Serve together, the same âpull and runâ model DLCs already offered for training, now extended to inference.
The walkthrough deploys Qwen3-VL-2B on a single g5.xlarge (one A10G, 24GB VRAM), loading the model in float16 specifically to fit that memory ceiling, and the whole endpoint is one @serve.deployment-decorated class with ray_actor_options={"num_gpus": 1}, no archiver, no handler hierarchy. The gotcha worth flagging for anyone reproducing this: the pod reports Ready a minute or two before Ray Serve actually starts answering requests, since the model is still loading onto the GPU, so a failed first curl doesnât mean the deployment is broken. Itâs single-node only here; multi-node model parallelism or replica autoscaling means layering KubeRay on top.
đ˘Model caching for Amazon SageMaker HyperPod inference
Every time an inference pod scales out on HyperPod, it has to pull the container image from ECR and then download the model weights, both over the network, before it can serve a single request. For a big model like DeepSeek-R1 thatâs 30+ minutes, and it happens all over again for every new pod during a scale-out event, which means your autoscaler can react in seconds but your actual capacity doesnât show up for half an hour. Model caching fixes this by pre-loading both the weights and the image onto node-local NVMe ahead of time, so a pod reads from local disk at ~7 GB/s instead of pulling over the network.
The mechanism worth understanding: itâs two independent caches (weights and image), each backed by its own CRD, and both use âpreferredâ rather than ârequiredâ scheduling, so a pod that lands on a node without a warm cache just falls back to the normal download path instead of failing. Benchmarks show ~60% faster scale-out for 57â145GB models. The catch to watch for: weights are cached per-node, so NVMe usage scales with node count, not a shared cache, and your instanceâs NVMe capacity has to actually exceed your model size or caching silently wonât fit.
Community & Career đ¤
đ¤Run Apache Spark up to 10x faster with DataPelago on Amazon EKS
Spark was built for CPU-only execution, so even when infra teams provision GPU nodes alongside their EKS clusters, Spark itself has no path to actually use them without a rewrite. DataPelagoâs Nucleus engine solves this by inserting a hardware-abstraction layer right after Sparkâs Catalyst optimizer produces its physical plan: it inspects each operation (scans, joins, aggregations, sorts) and decides at runtime whether that specific operation runs faster on GPU or on vectorized CPU, with automatic fallback to standard Spark if a stage canât be accelerated.
The mechanism that actually earns the speedup numbers is kernel fusion: Nucleus fuses multiple Spark operations into single GPU kernels, which is why it beats even NVIDIAâs own cuDF library by 3.7â10.5x on projection/filtering and up to 38.6x on string-heavy hash joins, since cuDF pays the cost of separate kernel launches and memory copies that Nucleus avoids. It deploys as a JAR plugin with zero code changes and no data migration, which is the detail that matters for anyone evaluating this: production customers report 2â10x speedups and 50â80% cost cuts without touching cluster topology, security config, or data formats.
đ¤Secure Agentic AI Deployment on Kubernetes â Kashish Verma
Kashish Verma grounds the whole piece in the real May 2025 GitHub MCP attack: an attacker planted hidden instructions in a public repoâs issue, and when a user asked their agent to âaddress the issues,â the agent pulled private repo data into context and leaked it via a public PR, fully authorized, using a completely legitimate tool call. Thatâs the core problem this piece addresses: least-privilege Kubernetes RBAC (ServiceAccounts, Roles) stops an agent from touching resources it shouldnât, but it says nothing about an agent being tricked by content itâs allowed to read.
The architecture worth studying is the sidecar pattern applied to secrets: instead of the agentâs own code holding a long-lived token, a sidecar container fetches credentials from a secrets manager (e.g., Vault) and exchanges them for short-lived tokens at call time, so a compromised agent process never holds anything long-lived. A custom ToolPermission CRD extends this to tool-level granularity Kubernetes RBAC has no native concept of (e.g., read_balance allowed, make_payment not), and an LLM-as-a-judge step in the same sidecar screens tool responses for injected instructions before they reach the agentâs context, the piece Kubernetes RBAC alone canât provide since the GitHub attack succeeded through a fully permitted tool call. GitOps (ArgoCD/Flux) ties it together org-wide by making the reviewed pipeline the only path to deploy any of this, closing the gap where a manually configured agent gets one step wrong.
- Highlights â¨
â¨The oldest architecture in computing â Werner Vogels
Vogels uses his time with Kiro Crew (AWSâs open-source agent workspace) as a jumping-off point to compare agent memory design to how brains manage memory: distributed storage across specialized units, background consolidation, and graceful decay of what isnât reinforced. The one concrete architectural detail worth noting is that Kiro Crewâs memory system spans markdown files, a local vector database, and a key-value index together, with recent memories kept in full detail while older ones compress or get pruned, which is the same âactive forgettingâ principle any long-running agent system eventually needs or it spends more cycles sorting old context than doing useful work.
â¨Kaichao You on why specialized inference engines keep losing to vLLM
Alex Cheemaâs viral complaint was that at least five specialized inference engines shipped in a single month, and asked why fragment the ecosystem instead of building on vLLM or SGLang. Kaichao Youâs response reframes what vLLM actually is: not just a way to run a model on a chip, but a unified interface sitting at the intersection of models, hardware, and inference techniques, all three of which move fast independently. A specialized engine optimizes one point in that space; vLLM has to keep tracking all three simultaneously, which is a much harder, more durable problem to solve.
The concrete example he gives is TileRT, a megakernel engine built specifically for decode. Rather than competing head-on, TileRT ended up pairing with vLLM in a prefill-with-vLLM, decode-with-TileRT split, the pattern You says heâs watched repeat for three years: specialized projects either get their techniques absorbed into vLLM or they disappear, because the ecosystemâs gravity is stronger than any single optimization. Worth remembering next time a new âvLLM killerâ makes the rounds: the interesting question isnât whether itâs faster at one thing, itâs whether it plugs into the ecosystem or tries to replace it.
đ Sponsor Section
At the moment, we donât have a sponsor for this edition, but we look forward to working with companies and organizations that support the EKS & AI Infrastructure community in future editions. If you or your company is interested in sponsoring, please contact us at đ§ thecloudtechforall@gmail.com
đ Words from the AuthorWe organized our 5th AWS Community Day Vadodara this year, and I still canât quite believe weâre here.
Six years ago this started as a small monthly meetup with a handful of people in a room. Yesterday we had Darko MesaroĹĄ, an AWS Distinguished Developer Advocate, standing on our stage, talking to over 700 people who showed up on a weekend.
Darko spoke about Kiro Crew, and honestly it connected with something Iâd already been chewing on. If you read Werner Vogelsâ piece in this issue, heâs circling the exact same idea from his side of the world: what does it actually mean for an agent to remember something well, to consolidate what matters and let the rest fade instead of dragging every bit of old context forward forever. Hearing Darko talk through it live, in the same week I was putting this issue together, made that whole thread click in a way reading it alone hadnât.
Dhaval Nagarâs keynote, âThe Model Is Not the System,â was the one thatâs stuck with me the most since. His point was simple but it landed hard: AI coding agents mean engineers might not need to remember every CLI flag or Terraform syntax anymore, but that doesnât mean the job gets smaller, it means the job moves up a level. You still have to understand whatâs actually being built, where the permission boundaries are, what happens when something fails, what data is flowing where. He walked through real AWS architecture examples showing how to let an agent move fast without quietly handing it architectural ownership along with the keyboard. Itâs the kind of talk that reframes something youâve been vaguely uneasy about into something you can actually name and design against.
But I need to say this plainly: this wasnât a one-day thing. This was three months of work. Planning started back in June, and it did not let up until people were walking through the doors on Saturday. I did not do this alone, not even close. Every single organizer and volunteer on this team gave up weekends, stayed late, chased down a hundred small things that nobody in the audience will ever see, and made sure it all just worked. Three months of that kind of effort is what a single day of âwow, that was amazingâ is actually built on.
So to everyone who put in that time: thank you. This one is yours as much as itâs mine.
Happy building. đ
