š Everything about EKS & AI Infrastructure Newsletter "#86" āļøā¤šØāš»
Where agent code runs, how to stop wasting GPUs, and what a token really costs
Dear EKS & AI Infrastructure enthusiasts,
Welcome to Everything about EKS & AI Infrastructure #86.
A lot of AI infrastructure comes down to one plain problem: GPUs are the most expensive thing you run, and they spend a surprising amount of time waiting. A GPU waits while a model checkpoint loads. It waits when one long prompt blocks everyone else. When one GPU fails in a training job, the healthy ones sit idle while the pods restart. And when an AI agent writes code, that code has to run somewhere safe, which is one more thing to build and pay for.
This issue is full of people fixing exactly that. Periodic Labs shows how they keep a cluster 95% busy during reinforcement learning, including a sandbox that runs on the spare CPUs of the same GPU nodes. An AWS guide shows how to add NVIDIAās resiliency tools to EKS training, so a GPU fault costs seconds and saving a checkpoint stops blocking training. PACER, a new AWS sample, lets GPU nodes share checkpoints with each other, and a 131 GiB model loaded into 8 GPUs in about six seconds in its test. We also cover benchmarks for splitting prompt reading and answer writing onto separate GPUs, and a look at how hard it is to share one GPU between several workloads on Kubernetes.
The second half is about agents. There are two guides to running agent code in Lambda MicroVMs, one from AWS and one from a builder who made a browser-based Claude Code environment for his iPad, plus a playground of 20 live demos. Edera shows how to snapshot and fork agents on your own cluster. TypeSafeās new Jev model makes small decisions, like routing a request or checking whether a command is risky, without writing any text, at a fraction of the cost of an LLM call. The SageMaker updates cover routing by KV cache, faster cold starts, and a list of backup GPU types for when your first choice has no capacity. There is also a bank that cut its cost per token by 60% on Kubernetes, and a tool that grades your EKS clusters against best practices.
Performance Engineering in Modern AI Systems š©ļø
š©ļøDavid B Chase: Automating GPU sharing with mutating admission
GPU sharing lets several workloads use one GPU, but each sharing method on Kubernetes is set up in a different place. Some use an environment variable, some use resources.limits, some use a pod annotation, and some need a separate object. A mutating admission webhook is a small service that edits pods as they are created, so it could add these settings for developers. David B Chase goes through each method and asks how easy it is to automate this way.
The easiest are KAI Scheduler, which only needs an annotation and a scheduler name, and MPS, which only needs an environment variable. HAMi and Volcano are harder because they use resources.limits, the same field Kubernetes uses for quotas, so an edit can clash with them. MIG can only pick from fixed GPU slices, because a custom split means reconfiguring the node. Time-slicing canāt be automated this way at all, since it is set in a cluster-wide ConfigMap and not on the pod. DRA is the most capable but the hardest, because the webhook has to create a separate ResourceClaim object and link the pod to it.
A webhook can add the setting, but something else has to decide how much GPU each workload gets, such as a label or a queue. If you use HAMi or Volcano, test the edits against your namespace ResourceQuota first.
š©ļøLMCache on Amazon SageMaker HyperPod: Disaggregated Prefill and Decode
Reading a prompt (prefill) and writing the answer (decode) need different things from a GPU, and on the same GPU a long prompt slows token generation for everyone else. Splitting them onto separate GPU pools fixes that, but only if the KV cache can move between pools faster than recomputing it. Xuan Lu, Yihua Cheng, Vinay Arora, and Swapnil Palod show how LMCache handles this. It runs inside vLLM and pushes the KV cache GPU to GPU over NIXL, libfabric, and EFA with GPU-Direct RDMA. A router only sends prompts above 4,096 tokens down the split path, and shorter ones run end to end on a decoder.
On Llama 3.3 70B, the split setup cut per-token latency by 22% to 66% on H100 and 28% to 48% on H200, depending on concurrency. Output throughput rose by up to 35% on H100 and up to 64% on H200, and median end-to-end latency improved by 14% to 41%. The cost is a higher time to first token, which the authors put at a few milliseconds.
The comparison isnāt equal on hardware. The split setup used 2 nodes (16 GPUs), while the colocated baseline used 1 node (8 GPUs). Throughput up 64% on double the GPUs means less output per GPU than the baseline. So compare cost per token, not only latency, before you adopt it. The test also used one workload (4,096-token prompts, 256-token outputs, 50 requests per run) and one prefiller with one decoder. The authors say colocated serving still wins for batch work that cares about first-token time, low concurrency, and short prompts.
š©ļøPACER: Peer-Accelerated Cache with EFA Replication (AWS sample)
Loading a big model checkpoint from S3 onto many GPU nodes is slow, and every node ends up fetching the same bytes. PACER is a Rust program that runs as a DaemonSet, one per node, and acts like S3 for that node. Unmodified S3 clients such as boto3, the AWS CLI, s5cmd, or a PyTorch dataloader point at it. A read is served from the nodeās own RAM or NVMe first. If a peer node has the object, it comes straight from that peerās memory over one-sided EFA RDMA, so the peerās CPU isnāt used. If nobody has it, PACER streams it from S3 and fills the cache on the way. Writes go through to S3 and are also cached, so saving a checkpoint pre-warms it for the next reader.
The sharp detail is that the layout, not the network, gives most of the speedup. The README loads a 131.4 GiB Llama 3.3 70B checkpoint into 8 GPUs in about 6.3 seconds, which is 20.8 GiB/s. The normal path takes about 33 seconds, and about 18 of those are spent in vLLMās per-parameter weight loader, at 0.9 GiB/s across 729 tensors per rank. PACER avoids that by re-laying out the checkpoint ahead of time into one contiguous slab per rank and RDMA-writing it into GPU memory that the serving process has registered. The author also says a single daemon serves about 100 GiB/s from registered memory on a p6-b200.
Two limits matter before you plan around the 6.3 seconds. It needs a re-laid-out checkpoint and a cooperating client that registers GPU memory. Plain S3 clients get the node-local and peer cache, and the postās zero-copy path is the part that needs the extra work. The number also comes from one configuration on one date (Sept 12), with a Ā±20% spread, since a load ends when its slowest of 8 ranks finishes.
- Starred Content ā
āPeriodic Labs: AI Infrastructure at Periodic
Periodic Labs trains models with reinforcement learning (RL), where the model attempts a task, uses tools, and learns from the result. One attempt can run for hours, while a training step takes only minutes, and the code the model writes runs on the same machines as the training. Their first setup had no isolation, and it crashed when the modelās code asked for 80 GB of RAM. Their current stack trains 4.1x faster than plain Megatron, generates text 2.5x faster on a 1T-parameter model, and keeps the cluster 95%+ busy, with a peak of 1,300 H200 GPUs.
The bug worth understanding is a slowdown across 32 GPUs that work on the same batch and wait for each other at every step. Whenever one GPU finishes a response, it does a heavy CPU task to send back a record of how the model routed each token, and all 32 GPUs wait for it. With responses around 200 tokens long, there is about a 15% chance that at least one of the 32 finishes on any given step, so the stall hit roughly 1.5 times every 10 steps. Their fix sends only the newly generated part each turn instead of the whole history. They also moved prefill (reading the prompt) and decode (writing the answer) onto separate GPUs, because prefill on one GPU was slowing everyone else down. That made each rollout about 2x shorter.
For code execution they built their own sandbox, pbox, that runs on the spare CPUs of the same GPU nodes the job already has. Its lifetime is tied to the job, so when the job ends the CPUs go back to the pool, and there is no separate sandbox cluster sitting idle. It moved files 4.5x faster (p50 latency) and had 3.3x the throughput of a hosted sandbox provider, and about 32 CPUs per GPU node was enough. They used Slurm for scheduling and cleanup, which EKS doesnāt give you, so on EKS you would have to build the job-lifetime and cleanup part yourself.
āFault tolerant distributed training on Amazon EKS using NVRx
On a multi-node training job, one GPU fault sets off a chain reaction: the healthy workers hit NCCL timeouts, the pods restart out of sync, and the cluster burns GPU hours without making progress. Saving checkpoints the normal way adds a second loss, because every rank waits for the write, which cost up to 40% of wall time in their runs. The post adds NVIDIAās Resiliency Extension (NVRx), a pip package for PyTorch, to an FSDP job on EKS. It covers three pieces: async checkpointing, in-process restart, and in-job restart with ft_launcher. Each piece is separate, so you can adopt only the ones you need.
The numbers show what each piece fixes. Async checkpointing kept training efficiency at 99%+ from 16 to 64 H100s, while synchronous saves stayed at 57% to 61%. The write to FSx for Lustre took about 275 seconds however many GPUs you had, so the loss is set by storage and not by cluster size. For faults, they injected 5 identical faults on 2 nodes. In-process restart recovered in about 10 seconds with no container restarts, ft_launcher took 17 seconds, and plain Kubernetes restarts took 270 seconds per fault. The plain Kubernetes case fell apart because one crashed rank left the other waiting 60 seconds for NCCL to time out, and then both pods restarted out of sync and went into CrashLoopBackOff.
Two settings need care. Async checkpointing hides the write fully only if the gap between checkpoints is longer than the write time. At every 100 steps (about 280 seconds apart, roughly the write time) the gain shrank to 29.6% efficiency for async against 14.7% for sync. And set ft_launcherās heartbeat timeout above your longest legitimate gap between steps. The post uses 900 seconds, with 1,200 seconds for the first heartbeat while the model loads.
āRunning self-hosted AI agent sandboxes with AWS Lambda MicroVMs
When an AI agent writes a shell command or queries a database, that code has to run somewhere isolated. Otherwise one userās session can leak credentials or files into anotherās. AWS shows a way to give every agent session its own Lambda MicroVM, a Firecracker virtual machine that boots from a saved snapshot, so there is no startup wait. It can run for up to 8 hours and scale its CPU and memory up to 4x without being recreated. The example uses Claude Managed Agents self-hosted sandboxes, so the database credentials and network rules stay in your own AWS account.
The design choice worth learning is how the VM gets started. The orchestration service sends a webhook when a session is ready. A small launcher Lambda checks the signature, calls RunMicrovm, and drops duplicate retries by tracking the event ID in DynamoDB. The VM boots, does the sessionās work, and terminates itself. The post says not to use a worker that keeps polling for work. A MicroVM with no incoming traffic is suspended by its idle policy, which stops the polling, and turning that policy off means paying for a VM that does nothing between sessions. Credentials are also split, so the launcher only holds the webhook signing secret and the VM only holds the key it needs to claim its session.
If you run agent sandboxes as pods on EKS today, this is the main alternative to compare against. You get a VM boundary per session and no idle cost, but you leave your clusterās network policies and tooling behind and live within the 8-hour limit. One thing to change before copying the sample: its launch code uses ALL_INGRESS and INTERNET_EGRESS network connectors, which allow all inbound traffic and outbound internet access. That is a lot of freedom for a sandbox that runs agent-written code, so restrict both.
- Announcements š¢
š¢Jev: a model that makes decisions and doesnāt write text
An AI agent makes many small judgment calls. Which model should handle this? Is this shell command risky? Is the task finished? Today each call goes to an LLM that writes its answer one token at a time, which is slow and costly when only a few answers are possible. Jev, released by TypeSafe AI on September 15, 2026, doesnāt write text at all. You send it some text plus questions with fixed answer types: pick one option, score on a scale, or answer yes/no (TypeSafe calls this last type āNoulā). It returns the answers with probabilities, and independent questions run in parallel.
The probabilities are the useful part. If engineering gets 91% and billing gets 9%, you can route automatically. If itās 52% to 48%, your code can escalate to a person or a stronger model. Your code owns the thresholds, and Jev only supplies the judgment. TypeSafe reports 70 to 500 ms latency and $0.042 per million input tokens with free output, and it claims up to 200x faster and 400x cheaper than comparable LLM workflows. Those figures come from TypeSafeās own tests and are best cases. Its ācannot hallucinateā claim only means the answer stays inside your list of options, and it can still pick the wrong option from that list.
For EKS teams, the fit is as a cheap router and gate around your LLM. It can choose between a fast and a strong model, check a tool call before an agent runs it (LangChain has middleware for this), or check whether a task is really done. It canāt write, summarize, do arithmetic, or handle decisions that need several dependent steps. It is also closed-weights, early-access, and text-only, so itās an API you call and not something you run on your own GPUs. Start with one low-risk decision. Run it beside your current logic without letting it act, plot accuracy against confidence, and set your thresholds from that data.
š¢Amazon SageMaker HyperPod Inference Gateway: smarter routing for GPU inference on EKS
The default Kubernetes load balancing (round-robin) sends requests to pods without knowing how busy each GPU is. A pod with a nearly full KV cache (the memory that holds the conversation so far) or a long queue still gets new requests, while other pods sit idle. AWSās new addon installs on a HyperPod EKS cluster and looks at live metrics from each pod before choosing where a request goes. It checks KV cache usage, queue length, whether the LoRA adapter is already loaded, whether the prompt prefix is already cached, and how many requests are running. Itās built on the open-source Gateway API Inference Extension.
How much you gain depends on how uneven your setup is. On mixed GPU types, P95 first-token latency dropped 97% to 98% for Llama-3.1-8B and Qwen3-32B, and throughput rose 50% for Qwen3-32B. With bursty traffic on Llama-3.1-70B, P99 dropped 98%, and with shared prompt prefixes the drops were smaller (26% at P95, 43% at P99). On a uniform fleet with steady traffic (Qwen3-235B), the result was the same as round-robin.
The addon is for HyperPod clusters, and the cross-cluster routing layer is still ācoming soonā. Since the routing logic is the upstream open-source project, self-managed EKS teams can build the same pattern, as long as their model servers expose KV cache and queue metrics. Check how uneven your replicas and traffic really are first, because a uniform fleet gets no benefit.
š¢Amazon SageMaker Inference: 2026 year-to-date launches in review
AWS lists 13 SageMaker inference launches this year, seven for managed endpoints and six for HyperPod Inference, which runs on your own EKS GPU clusters. The HyperPod six go after the usual pain points: slow scale-outs, repeated work on shared prompts, and long prompts blocking other users. They are a simple operator add-on, a two-level KV cache (CPU memory per node, plus Redis shared across nodes, with AWS claiming up to 40% lower latency), data capture for logging, separate prefill and decode GPUs, Hugging Face, NVMe and Route 53 features, and model caching.
Two of them have details worth knowing. Model caching pre-loads weights onto local NVMe (about 7 GB/s reads) and pre-pulls container images with a DaemonSet, which saves 5 to 7 minutes per pod start and gave about 60% faster scale-out on 57 GB to 145 GB models. Disaggregated prefill and decode runs prompt processing and token generation on different GPU pools and passes the KV cache between them over EFA with GPU-Direct RDMA. That stops one long prompt from slowing everyone elseās tokens, and you can scale each pool on its own.
Model caching has limits that the headline numbers hide. Each node keeps its own copy, NVMe space is finite, and if you change a model at the same path, the cache does not notice. Use a new path or version for every model update, and check that your GPU nodes have enough local NVMe before you turn it on. Disaggregated prefill and decode also needs EFA-capable instances, and it shipped in Inference Operator v3.2.
š¢Instance preference lists for Amazon SageMaker AI training jobs
A training job pinned to one GPU instance type has to wait, or fail, when that type has no capacity. Many teams wrote retry scripts to resubmit on other types. SageMaker now lets you give an ordered list of up to five instance types for training and processing jobs, and it launches on the first one with capacity. If none has capacity, the job waits in a queue and retries when capacity opens up.
Two details matter in practice. First, the wait limit MaxPendingTimeInSeconds covers the whole list, not each type, and it only applies if the list includes accelerated instances (ml.p, ml.g, or ml.trn). Second, you can attach a Flexible Training Plan reservation to one entry, for example a reserved p5, so it is tried first, with on-demand p4d and p4de as fallback. You can also set a different instance count for each entry, such as 2 g6.48xlarge or 4 g5.48xlarge, to get roughly equal total compute.
SageMaker does not check whether the types on your list work together (GPU architecture, EFA support, driver versions), so your training container must run on every one. You also pick the counts that make slower types comparable, and the post doesnāt say how to size them. Processing jobs canāt use training plans.
š¢SageMaker AI adds Nemotron 3.5 Lightning 30B-A3B to model customization
SageMaker AI can now fine-tune NVIDIAās Nemotron 3.5 Lightning, a 30B-parameter mixture-of-experts model that uses only 3B parameters per token. NVIDIA built it for always-on agents, and it is small enough to run on a single GPU. The recipes cover supervised fine-tuning, DPO, and reinforcement learning with either verifiable rewards or AI feedback, each with LoRA or full fine-tuning. Gemma 4 and Qwen 3.6 are already supported, and Ministral 3 and Qwen 3.8 are in the open recipes repo.
The useful part is that the recipe stays the same however you run it. In serverless mode you pick the model in SageMaker Studio and point it at your dataset in S3. SageMaker finds the accelerators and releases them when the job finishes, so you donāt size a cluster. If you have your own capacity, you run the same open-source recipe on a HyperPod cluster in your VPC, with checkpoints in your S3, and you can edit the recipe before you launch. AWSās Whatās New page lists serverless availability in US East (N. Virginia), US West (Oregon), Asia Pacific (Tokyo), and Europe (Ireland).
NVIDIAās model card says the BF16 weights are a starting point for customization and not meant for production serving. For fast inference it points to its NVFP4 release, so plan a quantization step for your fine-tuned model before you serve it.
Community & Career š¤
š¤EKS Guardian: an operational review skill for AWS DevOps Agent
EKS clusters drift away from best practices over time, and nobody notices until something breaks. EKS Guardian is a skill for AWS DevOps Agent that checks for this. You ask āRun an EKS operational review for all clustersā, and it finds your clusters across accounts and regions. It then checks each one against the AWS EKS Best Practices Guide in 12 areas, including Security, Reliability, Karpenter, Networking, Scalability, Upgrades, and Cost.
The useful part is that it looks at how the cluster behaved, not just how it is configured. It pulls 7 days of CloudWatch metrics, control plane logs, and CloudTrail events. The output is an A to F grade per cluster, an executive summary, a list of fixes, and a comparison against earlier reviews. It is read-only and never changes your cluster or AWS resources.
A DevOps Agent skill is a set of instructions the agent follows, not a fixed rules engine, so the grades depend on how the agent reads the evidence. Run it first on a cluster you know well and check whether the grade and findings match what you already know. Only then trust it on the rest of the fleet.
š¤edjgeekās Claude Code sandbox for iPad, built on Lambda MicroVMs and S3 Files
edjgeek (singledigit on GitHub) wanted a dev environment he could open from an iPad browser, with three conditions: it must be strong enough for Claude Code, it must cost nothing while idle, and his files must survive the VM being deleted. His solution is one SAM template. You sign in with Cognito, a MicroVM starts just for you, and a terminal in your browser connects to a shell inside it. Your home directory lives in S3 Files, so the VM can be thrown away without losing anything.
The key design detail is how the home directory gets attached. Every userās VM boots from the same shared image snapshot, so a home directory canāt be baked into it. Instead, the token Lambda creates an S3 Files access point for each user (scoped to /users/<their-cognito-sub>) and passes it to the VMās /run and /resume hooks, which mount it as /home/coder. The idle policy suspends the VM after 2 hours with no traffic, and a browser keepalive every 15 seconds keeps it awake while a tab is open. If you return within 30 minutes, the VM resumes from its memory snapshot, and after that you get a fresh VM with the same files. There is also an 8-hour hard cap. Terminal authentication is handled by the platformās proxy with a 55-minute token, so the server inside the VM does no auth work.
He hit a few gotchas. Mounting S3 Files needs the ALL OS capability (CAP_SYS_ADMIN), which can only be set when the image is created, so forgetting it means deleting and rebuilding the image. Enabling the /validate hook lets the platform learn which disk pages startup uses and prefetch them, and he says that cut the first mount from about 26 seconds to a few. His own cost estimate is about $37 a month for 2 hours a day at an 8 GB baseline with bursting. He later moved to a 4 GB baseline, and an August update says the sandbox now also runs Codex CLI and Kiro CLI next to Claude Code.
š¤kagent-eks-setup: ask your EKS cluster questions in plain English
KAgent is a tool that lets you type a question like āwhat pods are running?ā and get a real answer from your cluster. It runs the kubectl commands for you, so you donāt have to remember them. This repo shows how to set it up on EKS, using OpenAI as the AI brain. You install it with Helm in small steps: first the basics, then the AI agent, and you check each step before the next.
The neat part is how it keeps the OpenAI key safe. The key is stored in AWS Secrets Manager, not in your files. A small tool called External Secrets Operator copies it into the cluster when needed. To let it read the key, the setup uses IRSA, which gives a pod its AWS access through an IAM role, so you never store AWS passwords in the cluster. The UI opens with kubectl port-forward, so nothing is public.
š¤Lambda MicroVM Demo Playground: 20 live demos of stateful, isolated sandboxes
For years the advice was to keep workloads stateless. But AI agents that run code, developer sandboxes, CI/CD workers, and Monte Carlo runs all execute code you canāt fully trust, and they often need to keep state between steps. A normal VM isolates well but boots slowly and costs a lot to run at scale. Lambda MicroVMs try to fix that. Each one has its own kernel, memory, and disk, boots from a snapshot almost instantly, and suspends when idle. Antonās playground shows this with 20 demos, including agent sandboxes with tools, code REPLs, a full Linux terminal, VS Code and Jupyter workstations, a SQL database, container build-and-run, and CI/CD workers.
The most useful detail for security is egress, meaning what the sandbox can reach. The playgroundās overview page says you can run a MicroVM with public access, inside a private VPC, or with no outbound access at all. Inbound traffic can be turned off or protected by a short-lived access token. This matters because code that canāt reach the internet canāt send your data out. The state stays for as long as the MicroVM lives, up to 8 hours, and a MicroVM can use up to 4x its starting CPU and memory when it needs to. It runs any standard container image.
If you run sandboxes as pods on EKS today, this is the alternative to compare against. You get a hardware isolation boundary per session, but you leave your clusterās tooling behind and live within the 8-hour limit.
- Highlights āØ
āØKubernetes can run AI inference. But can it count the real cost?
China Merchants Bankās infrastructure team won the CNCF End User Case Study Contest for running AI training and inference on Kubernetes. They manage nearly 10,000 mixed accelerator cards using Kueue for queues and quotas, KEDA for autoscaling, Prometheus for metrics, HAMi for sharing GPU capacity, and Fluid for faster dataset access. They now manage 99% of their AI compute this way, average utilization went from 35% to over 60%, and the cost of processing 1 million tokens dropped 60% under comparable conditions.
The same column has a useful counterpoint from Val Bercovici, chief AI officer at WEKA. He argues that cost per token depends on things Kubernetes doesnāt track: the mix of requests, how full the KV cache is, the balance between prefill and decode, and how memory and bandwidth are used inside the GPU after a pod is already running. He thinks Kubernetes stays, but without a better resource model it becomes a tax on inference economics, and he expects a new scheduling and memory layer around it. Note that the bankās projects are mostly about how GPUs are allocated and shared, which is the layer Kubernetes can already see. Bercovici is pointing at the layer below it.
If you track GPU utilization as your main cost signal, add token-level numbers next to it, such as KV cache usage, queue depth, and time to first token. A GPU can look busy while each token costs too much to serve.
āØEdera Project Lunchbox: snapshot, fork, and isolate AI agents on your own Kubernetes
Running thousands of AI agent sessions at once is expensive, because each one holds memory even while it waits, and one compromised agent shouldnāt reach the others. Edera puts each agent in its own microVM with its own Linux kernel on your existing Kubernetes cluster. Outside access is controlled below the guest at the runtime and hypervisor layer, so an agent canāt give itself more network access. On top of that come three tools: snapshots, live forks, and narrow permissions, which Edera calls capabilities, such as talk-to-github.
The forking is the interesting part. You freeze an agent at a decision point and clone its live state, say 10 times, and all the clones share the same memory through copy-on-write until one of them changes a page. That means an RL job or a ātry 10 pathsā agent doesnāt pay for 10 full environments, and the clones canāt read each otherās diverging memory. Snapshots work similarly for cost. You install toolchains once, snapshot the warm state, and restore it instantly, and you can suspend an agent that is waiting for a human so it stops using compute.
Today this is a Python SDK. Edera describes Sandbox, Snapshot, Fork, and Capability as Kubernetes-native resources it wants to build later. The post itself says that spreading forks across nodes and Kubernetes memory accounting for shared pages are hard problems it hasnāt finished. Kubernetes 1.37 has pod-level checkpoint and restore in alpha, but that leaves the implementation to the container runtime and has no agent-level tools on top. If you plan large parallel agent or RL workloads on EKS, ask Edera how pod memory requests and limits work when forks share pages.
āØArchitecting Flexible Agentic AI on Amazon EKS, with NVIDIA and LangChain
AWS Loft is running an event on building agentic AI that you own and run on Amazon EKS. It includes a hands-on session with LangChain, plus talks on NVIDIA open models and sovereign AI architecture. The stated point is portability: the same stack can be deployed in-region, on-premises, or air-gapped. Teams with data residency or air-gap requirements are the ones who will care about that.
āØGyanesh Kumarās takeaways from the 2026 AI Infra Summit (Santa Clara, Sept 15 to 17)
Gyanesh Kumar, Field CTO for AI Startups at AWS, says the common thread at the summit was that gains now come from tuning every layer together (chips, networking, memory, software) and not from adding more GPUs, with Peter DeSantis of Amazon and Olivia Wu of Meta making that point. Two takeaways matter for EKS teams: memory and data movement, more than raw compute, now limit what you can run, and agents that work in long, multi-step, tool-calling loops need different infrastructure than simple request and response. That is why Vivek Gangasani of AWS argued that inference needs purpose-built systems that treat cold starts, batching, and memory management as core problems.
š Sponsor Section
At the moment, we donāt have a sponsor for this edition, but we look forward to working with companies and organizations that support the EKS & AI Infrastructure community in future editions. If you or your company is interested in sponsoring, please contact us at š§ thecloudtechforall@gmail.com
š Words from the AuthorThis is edition 86, and the pattern I see in this issue is that the interesting work in AI infrastructure has moved. It used to be about how to run the model. Now itās about everything around the model: how fast it loads, how much of the GPU sits idle, where agent code runs, and what each decision costs.
Here is my contrarian view. The next big gain may come from smaller models that do less. Jev is the example in this issue. It canāt write a sentence. It picks an answer from a list you give it and tells you how sure it is, and your code decides what to do next. Most steps in an agent are small decisions like that, and we keep sending them to a model built to write essays. I wouldnāt rip out your LLMs over this, and Jev is closed-weights and early access, with speed claims from its own maker. But the pattern will outlast Jev: let a small, fast model make the small call, and save the big model for work that needs language.
My second view is about EKS. Kubernetes is becoming the floor, not the platform. AWS now ships inference routing and caching as managed add-ons, while builders create the same pieces in the open, like PACER and Edera. What Kubernetes still canāt see is KV cache, prefill against decode, and what happens inside the GPU. The teams that win will add a layer that can.
As for this newsletter, I try to keep it practical. Each item says what it is, gives the one detail that matters, and tells you what to check before you use it. This issue leans on agents and sandboxes because thatās where I get the most questions. If you want a topic covered, reply and tell me.
Happy building. š
