đ Everything about EKS & AI Infrastructure Newsletter "#83" âď¸â¤đ¨âđť
Every fix this edition starts by admitting where the last one was quietly wrong.
Dear EKS & AI Infrastructure enthusiasts,
Welcome to Everything about EKS & AI Infrastructure #83.
What struck me putting this one together is how many of these pieces exist because something else, something that looked reasonable at the time, turned out not to be. llmfitâs whole changelog this release is a list of memory estimates that were confidently wrong, hybrid attention miscounted, pre-quantized models flagged as too big when they werenât. The Qwen3.8-Flash-Next sizing writeup exists because most peopleâs first guess about what fits on a Spark was wrong, and the reason took real digging to find. Curvineâs tiered cache exists because the assumption that each replicaâs cache only serves itself was costing everyone a redundant prefill they didnât need to pay for.
Even the two pieces that arenât about model serving follow the pattern. Break-glass access for EKS exists because the standard advice, keep a spare admin account around, quietly breaks in exactly the scenario youâd need it for, if the same event that took down your identity provider also touched that account. And GPU-accelerated Spark on EMR is really a correction too, that GPUs are a niche tool for AI workloads and donât belong in a plain old data pipeline, when it turns out a big chunk of ordinary ETL was leaving real speed on the table the whole time.
None of these are dramatic reversals, theyâre closer to someone actually measuring a thing everyone had been assuming about. Thatâs most of whatâs in here this week.
Performance Engineering in Modern AI Systems đŠď¸
đŠď¸Qwen3.8-Flash-Next fits on one DGX Spark, but only if you pick the right build
When Qwen3.8-Flash-Next dropped, the first thing that showed up on a lot of timelines was people saying it wouldnât fit on a single DGX Spark. It does fit, but the interesting part isnât that it fits, itâs why almost every build of it doesnât, and what makes the one that does fit different from the rest.
The sizing problem comes down to one specific piece of the model: a 51B-parameter N-gram lookup table that stays in higher precision even after the routed experts around it get quantized down. That table alone is enough to push most standard builds past what a Sparkâs 128GB of unified memory can hold. BF16 comes in at 335 GiB. FP8 gets it down to 172.8 GiB, still too big. Even NVFP4, usually the aggressive option, lands at 135 GiB, still just over the line. The only build that actually works is a GGUF version, specifically the UD-IQ1_S quantization, because itâs the only one that also compresses that N-gram table instead of leaving it untouched. That gets the whole thing down to 67.5 GiB, which loads in about 30 seconds and leaves real headroom, using about 72.5 of the 121 GiB actually available. Running on llama.cpp, that build does about 34.5 tokens per second single-stream.
A few things came out of testing this on an eight-card RTX PRO 6000 box that genuinely surprised whoever ran the benchmarks. First, two GPUs beat four for single-user latency, TP2 hit 81.5 tokens per second while TP4 dropped to 64.6. Thatâs counterintuitive until you think about why: without NVLink, splitting a sparse mixture-of-experts model that only has 6B active parameters across four cards costs more in communication overhead between the cards than it saves by spreading the compute out. Throughput tells a different story though, TP4 with 32 concurrent streams reaches 805 tokens per second, so the right topology genuinely depends on whether youâre optimizing for one user or for volume.
Second, keeping that N-gram table on the GPU rather than fetching it from host memory turned out to be a deliberate architectural choice, not just a workaround. It buys about 16% better performance for a single user, but it collapses how many concurrent users you can support from 74x down to 10x. Thatâs a real tradeoff a team has to make consciously depending on what theyâre actually building.
And third, probably the most broadly useful finding here: speculative decoding benchmarks lie to you if you test them with random synthetic tokens instead of real text. The MTP acceptance rate came in at 84.4% on synthetic data, which looked great, but dropped to 55.3% on real prose. If you size your capacity planning off the synthetic number, you will be wrong, and youâll find out you were wrong at the worst possible time, in production, under real load.
The problem here is one every team running multiple vLLM replicas eventually hits. Each replica keeps its own isolated KV cache, so if a request lands on a different Pod than the one that processed an identical prompt earlier, thatâs a full cold prefill regardless of the fact that another replica already did the work seconds ago. On a cost-efficient instance like a 48GB G6e GPU, the memory left for the cache after model weights is limited to begin with, and it shrinks further with bigger models or more concurrency, so this problem gets worse exactly when youâre trying to save money by using smaller instances.
The fix is a three-tier hierarchy, GPU HBM as L0, CPU memory via LMCache as L1, and a new shared layer, L2, built on Curvine, a distributed cache filesystem that pools node-local NVMe across the cluster into one namespace every Pod mounts as ReadWriteMany. Thatâs the actual innovation here, a KV block one replica writes becomes immediately readable by every other replica over the shared mount, so a cache miss on your Pod isnât necessarily a cache miss on the cluster. The router sits in front and does prefix-aware or KV-aware matching to land requests on the replica most likely to already hold the relevant blocks.
The numbers back it up. Cross-Pod reads showed a 100% hit rate on a request the receiving Pod never itself prefilled, with the TTFT win scaling from 1.7x at 1,000 tokens up to 2.7x at 2,500 tokens, a cold 774ms request finishing in 287ms. Below roughly 1,000 tokens, though, the L2 round-trip costs about as much as just recomputing, which is exactly the case for relying on L0 and L1 instead. The practical framing they land on is: writes are basically free since they hit local NVMe at near-disk speed, reads are the expensive part since they cross the Pod network, so getting the routerâs prefix matching right is what actually determines whether you see the 2.7x or eat the full network round-trip.
đŠď¸AWS Neuron 2.32 adds MXFP8 MoE training kernels and variable-size collectives for Trn2 and Trn3
The headline item here is that blockwise MoE layers can now train end to end in MXFP8, forward and backward pass both, with matched kernels rather than a mix of precisions stitched together. If youâve been following the quantization tradeoffs in the Qwen sizing piece above, this is the training-side version of the same idea, doing the math in a lower precision format without needing a separate correction pass, which is normally where the real engineering effort in a quantized training pipeline goes. Thirteen new NKI Library kernels ship alongside this for MoE training and sparse attention specifically, including context encoding for DeepSeek-V3.2âs sparse multi-head latent attention, so this isnât a generic precision bump, itâs built around the architectures teams are actually training right now.
The other piece worth knowing about is variable-size all-gather, reduce-scatter, and all-to-all collectives on Trn2 and Trn3, where each rank can contribute or receive a different number of elements instead of everyone being forced to a fixed size. That matters specifically for MoE and sparse attention workloads, where different experts or different ranks naturally end up holding different amounts of data, and previously youâd pad everything to a uniform size and waste bandwidth doing it. The vLLM Neuron plugin also moved to vLLM 0.24.0 and now ships in all Neuron Deep Learning AMIs and containers by default, so if youâre running inference on Trainium through vLLM, that upgrade path is already there waiting.**
đŠď¸**Inference Perfâs accompanying paper just got published in the Journal of Open Source Software
If youâve spent any time trying to benchmark LLM inference on Kubernetes, you already know the actual problem isnât running the benchmark, itâs that everyoneâs numbers mean something slightly different. One teamâs âthroughputâ counts prefill differently than anotherâs. Latency gets measured at different percentiles, against different traffic patterns, with different definitions of what counts as a successful request. You end up comparing apples to some other, vaguely apple-shaped fruit, and nobody can tell you why their vLLM deployment looks faster than someone elseâs on paper but slower in production.
Inference Perf, a project under the Kubernetes Serving Working Group, exists specifically to close that gap. Yuan Tang and the team, Ashok Chandrasekar, Sachin Mathew Varghese, Jason Kramberger, Brendan Slabe, and Chen Wang, just had the accompanying paper published in JOSS, which is a meaningful marker for a project like this. Itâs not a flashy release announcement, itâs the kind of peer-reviewed documentation that makes a benchmarking tool something you can actually cite and standardize on, instead of just another script someone on your team wrote and half-trusts.
The part worth sitting with is what âstandardizationâ actually buys a platform team here. Right now, if youâre running inference on EKS and comparing model servers, or comparing your own setup against someone elseâs published numbers, youâre implicitly trusting that both sides measured the same thing the same way. A shared, community-maintained tool under Kubernetes SIGs means that trust doesnât have to be implicit anymore. If your team ends up adopting it for internal benchmarking, the numbers you get become comparable to what the rest of the ecosystem is publishing, which is a much stronger position to be evaluating GPU sizing or model server choices from than a one-off internal script.
đŠď¸AWS Neuron 2.31 brings the UltraServer Operator to EKS in public beta
The operator is the headline for anyone running Trainium on EKS, not the compiler changes. It automates UltraServer discovery, workload allocation, and resource claim generation for Trainium UltraServer workloads, which is exactly the kind of glue code teams have historically had to write themselves to get Kubernetes to understand a multi-chip UltraServer topology as a single schedulable unit. Before something like this exists, mapping a workload onto the right UltraServer boundary is manual bookkeeping, and getting it wrong either wastes capacity or breaks the interconnect assumptions the workload was written for.
The rest of the release is compiler and kernel-level work that compounds with it. The Neuron Compiler ships a redesigned code generation backend on by default for Trn2 and Trn3, and NKI 0.5.0 adds MX FP8 scale dtype support along with tensor indirection for indexed access patterns, useful specifically for MoE routing where youâre gathering from a variable set of experts rather than a fixed dense layer. Fourteen new experimental NKI kernels ship alongside this too, covering MoE training collectives, DeepSeek MLA projection, and ring attention, so if your team is on Trainium and evaluating either of those architectures, the low-level kernel support is already there rather than something youâd need to write against raw NKI primitives yourself.
- Starred Content â
âHybrid ML inferencing on EKS with FSx for NetApp ONTAP cuts cold start from 228 seconds to 101
The problem this solves is one most teams running inference on Kubernetes hit eventually and then work around badly. Model weights, tokenizer files, and compiled CUDA kernels have to survive pod restarts, but without shared persistent storage, every new pod re-downloads and recompiles all of it from scratch, so scaling out looks fast on paper and takes minutes in practice. This writeup from AWS and NetApp walks through solving it with FSx for ONTAP mounted through the Trident CSI driver as a shared, read-only PersistentVolume, so every pod hits the same cached copy instead of racing to populate its own.
The numbers make the case better than the architecture diagram does. On a g6.4xlarge with an L4 GPU serving a fine-tuned Qwen2.5-1.5B model through vLLM, cold start took 228 seconds, almost all of it in a 177-second image pull plus model download and CUDA kernel compilation. Warm start, with weights and compiled kernels already sitting on the FSx volume, dropped total startup to about 101 seconds, and the remaining time is basically just Karpenter provisioning a node and PCIe pushing weights into VRAM. Thatâs not a marginal win, thatâs the download and compilation phases disappearing entirely, and it was consistent across three separate runs, not a one-off best case.
The part worth remembering if youâre running a hybrid setup is NetApp SnapMirror doing block-level replication straight from an on-premises ONTAP system into the FSx volume, no S3 hop, no manual transfer step. Train where your data has to live for compliance or hardware reasons, serve where your traffic actually is, and Trident handles snapshots and rollback the same way it handles the initial provisioning, through a plain kubectl command if a model version underperforms.
âRay on SageMaker HyperPod removes the YAML and kubectl tax for running Ray on EKS
The problem being solved here is specific and familiar if youâve stood up KubeRay yourself: running Ray on Kubernetes has always meant writing RayCluster YAML by hand, rebuilding Docker images for every dependency change, setting up kubectl port-forward just to see the Ray Dashboard, and manually wiring Prometheus and Grafana for observability. None of that is hard exactly, itâs just tax, the kind of platform work that eats a week before a data scientist gets to run their first distributed job. HyperPod now handles all of it from SageMaker Studio, cluster creation, dashboard access, and job submission without a single kubectl command, while still running standard open-source KubeRay underneath, so existing Ray scripts donât need to change.
The part that actually matters for reliability is what happens when things go wrong at scale. HyperPodâs node health monitoring now extends to Ray Train specifically, with a per-node Job Monitoring Agent that catches hung jobs, the failure mode where one pod dies from a network partition and every other pod blocks indefinitely at the next collective operation with no error message, just GPUs sitting allocated and idle for hours until someone happens to check. Thatâs a real cost problem at scale thatâs easy to miss without dedicated detection, and itâs now surfaced automatically through CloudWatch and a Ray Train Grafana dashboard, with tiered checkpointing pulling recovery from local HyperPod storage before falling back to S3.
On the serving side, Ray Serve deployments on HyperPod get the same Managed Tiered KV Cache architecture that Curvine implementation used above, L1 in CPU memory per node, L2 on shared tiered storage for cross-instance reuse, wired in with minimal code changes. Combined with a JumpStart model loader that pulls weights straight into a Ray Serve endpoint without manual container setup, this closes a meaningful gap between âI want to serve a JumpStart model with Rayâ and actually having a production endpoint with cache reuse across replicas.
**
â**Waferâs GPU performance engineering curriculum is the most complete learning path Iâve seen for this
Most âlearn GPU programmingâ resource lists are either a scattered pile of blog posts with no order to them, or a textbook that stops at fundamentals and leaves you on your own once you actually need to write something fast. This one is neither. Itâs a tiered curriculum from Wafer that runs from fundamentals through to what frontier labs are doing, starting with PMPP and the GPU Mode lecture series, moving through matrix multiplication, tensor cores, and attention kernels, and only then getting into production inference systems and multi-GPU work. That ordering matters more than it sounds, jumping straight to âwrite a fast attention kernelâ without the matmul and memory fundamentals underneath it is how you end up copying patterns you donât actually understand.
The part most useful for an infra team rather than someone writing kernels day to day is the production inference systems section, vLLM, SGLang, TensorRT-LLM, continuous batching from Orcaâs original paper through to how itâs explained today, and speculative decoding approaches like Medusa and EAGLE. Understanding what continuous batching or PagedAttention is doing underneath the model server youâre deploying changes how you reason about GPU utilization and where your real bottlenecks are, even if you never write a kernel yourself. Worth flagging when you share it: this is a multi-week curriculum, not a single sitting, frame it that way.
âA backup way to get into your Kubernetes cluster when your normal login system breaks
Hereâs the problem in plain terms. Most companies running Kubernetes on AWS (through EKS) donât log engineers in directly. Instead, someone signs into a company identity system (like Okta or a similar login provider), and that system hands them temporary AWS credentials, which then get mapped to permissions inside the cluster. Itâs convenient day to day. But if that login system ever breaks, has an outage, a certificate expires, something gets misconfigured during a migration, youâre stuck. You canât log into the cluster to fix the problem, because the thing thatâs broken is the exact thing you need working in order to log in. Itâs a locked door with the only key inside the room.
AWSâs existing advice is basically: keep one special admin account around, safely tucked away, for emergencies like this. The trouble is that account usually doesnât require multi-factor authentication (MFA, the second security check like a phone code, on top of a password), and it often lives in the same system that just broke. If your company migrated to a new AWS account structure and thatâs what caused the outage, that emergency account may have gotten deleted or renamed in the process too. So the backup plan can fail for the same reason the main plan did.
This post lays out a proper fix. You set up a completely separate emergency login path ahead of time, one that doesnât depend on your companyâs normal login system at all. It lives in its own AWS account, requires MFA every time, and only works if that MFA check happened recently (not just âI logged in with MFA three days ago and never logged outâ). Critically, it also stamps the operatorâs real name onto every action they take during the emergency, so thereâs no ambiguity later about who did what. The tradeoff is that this backup path only works from the command line, not through the AWS website, because of how that name-stamping technically works.
The part Iâd actually flag as the most important lesson here, more than the technical setup itself: you have to actually test that this emergency door works, and test that itâs actually locked to everyone else. That means two tests. One where you try the emergency login the right way, with MFA, and confirm it works. And one where you deliberately try it the wrong way, without MFA, and confirm it gets rejected. Most teams build something like this and never check the second half, so they donât find out the lock was never really engaged until the day they actually needed it. The company in this writeup ran both tests across more than 50 AWS accounts before trusting the system, and it paid off, they migrated their whole setup without a single cluster becoming unreachable.
âGPU-accelerated Spark on EMR runs 3.7x faster on G7 instances, and somehow costs less too
Quick bit of context before the numbers: Apache Spark is the tool most teams use to process huge datasets across many machines at once, things like joining tables together or aggregating billions of rows. AWS ran a standard industry benchmark for this kind of workload (called TPC-DS, basically a stress test with 103 realistic queries) on two kinds of machines: normal CPU-based servers, and a newer type with an NVIDIA GPU attached. The GPU version finished the whole benchmark in 4.7 minutes. The CPU versions took between 17 and 17.5 minutes. Same code, no rewrites, the GPU one just runs faster because a plugin automatically hands off the heavy math to the graphics card instead of the regular processor.
The part that actually surprised me is the cost. Youâd expect the GPU machine to be a splurge, since it costs about two and a half times more per hour to rent. But because it finishes the job so much faster, youâre paying that higher rate for a much shorter time, and it actually comes out up to 31% cheaper overall for the same amount of work. So this isnât really a âpay more for speedâ story, itâs closer to âpay less because you needed less time.â
Itâs not a universal win though, and thatâs the useful part. The GPU won on 102 out of 103 test queries, but the size of the win varied a lot depending on what kind of work each query did. Queries that involved heavy grouping and joining data (the kind of thing GPUs are naturally good at doing many calculations at once) sped up the most, one query went from 64 seconds down to under 6. But there was one query, a really trivial one that barely touched any data, where the CPU actually won, because thereâs a small setup cost every time you hand work off to the GPU, and for that tiny query it wasnât worth it. So the lesson isnât âswitch everything to GPU,â itâs âthe heavy stuff benefits a lot, the tiny stuff doesnât benefit at all,â and knowing which is which matters more than the flashy 3.7x headline number.
- Announcements đ˘
đ˘Kubernetes v1.37 âGarhwalâ is out, and DRA finally feels finished
Every Kubernetes release promises something big, and most of the time that something big is one flashy feature you can put in a headline. This one didnât work that way. DRA, Dynamic Resource Allocation, didnât get a single new feature this cycle. It got four smaller graduations, and itâs only once you sit with all four together that you realize they add up to something that finally works the way people have wanted it to for a couple of years now.
Start with device taints, which are now stable. If youâve ever had a GPU start throwing ECC errors in the middle of the night, you know the annoying part isnât fixing the GPU, itâs getting the scheduler to stop sending pods to a node thatâs about to become a problem. Before this, your options were rough: drain the whole node, or write something custom to keep the scheduler away from it. Now you can taint the device itself. The scheduler just stops considering it, and everything else on that node keeps running normally.
The second piece is quieter but just as useful. Devices can now report their own status back through the API. The example that makes this click is networking. Before v1.37, a pod could request a network device through DRA, but nothing else in the cluster had any way of knowing what IP address that device actually got. It existed, it worked, but it was invisible to everything downstream. Now the status is there for anything that needs to read it.
Hereâs the part I think matters most if youâre running GPU sharing through something like HAMi. What graduated in 1.37 is the request side of things, the part where Kubernetes understands what a workload is asking for and how to schedule it against available devices. Thatâs now genuinely upstream, which is a real milestone. But enforcement, the part where something actually makes sure a container canât see or touch resources it wasnât granted, still isnât in core Kubernetes. DRA can make the promise. Something else, like HAMi-core, still has to be the thing that holds the container to it at runtime.
One more thing worth calling out before anyone touches a real cluster with this: eventRecordQPS: 0 used to mean disabled, and now it means unlimited. Thatâs a meaningful behavior change hiding in a config value most people set once and forget about. And the API server can now return 429 during its own startup and recovery process, so if youâve written a controller that assumes every request either succeeds or fails outright, it needs to learn to back off and retry instead of falling over.
đ˘NVIDIA is reportedly buying Hugging Face for $12.9 billion
Take this one with a bit of caution for now, because as of writing thereâs no signed agreement, and some of the reporting is explicit that the talks could still fall apart before anything closes. But the reporting itself is solid enough, coming from multiple outlets independently, that itâs worth sitting with even before itâs confirmed one way or the other.
The logic behind why NVIDIA would want this isnât complicated once you say it out loud. Almost everyone who downloads an open-weight model from Hugging Face has to actually run it somewhere, and overwhelmingly that somewhere is NVIDIA hardware. So owning the platform where those models get distributed isnât really about the platformâs own revenue, which is reportedly only around $150 million a year, nowhere close to what would normally justify a $12.9 billion price tag. Itâs about position. Right now OpenAI, Google, Amazon, and Anthropic are all building their own inference silicon, which is the one real long-term threat to NVIDIAâs grip on the market. Owning the biggest open-weight hub is a hedge against that.
The part Iâd actually sit with, if this does close, is what happens to the word âopenâ itself. Right now, Hugging Face gets described as vendor-neutral, and that neutrality is doing a lot of quiet work in how people trust the platform. Once the biggest chip company in the world owns the shelf that everyoneâs models sit on, âvendor-neutralâ stops being an accurate description even if nothing about how the platform actually operates changes on day one. Whether that matters in practice will depend entirely on what NVIDIA does with it, and NVIDIA does actually have a reasonably good track record of leaving open source projects itâs acquired alone. But itâs worth watching, not assuming either way.
The API for this launched back in April, and it already did the hard part, benchmarking your model across instance types and configurations on real GPU infrastructure. What it didnât do was make that accessible to anyone who wasnât already comfortable setting the right parameters and reading raw benchmark output to figure out what âgoodâ looks like. The UI closes that gap with preset use-case profiles, Interact for chat-style short-input workloads, Generate for longer outputs, Summarize for high input-to-output ratios, so youâre picking a traffic pattern instead of guessing at token distributions and concurrency settings yourself.
The part worth understanding is whatâs actually happening underneath when you pick an optimization goal. Minimize latency triggers kernel-tuned deployments where the model architecture supports it. Maximize throughput kicks off a training job first to train a draft model for speculative decoding before it even deploys the benchmark endpoints. Minimize cost just finds the most cost-efficient configuration for your expected traffic. All three benchmark on real infrastructure using NVIDIA AIPerf with multi-run confidence intervals, not a simulated estimate, and the optimization job tears down the endpoints it created once itâs done, so youâre not left paying for benchmark infrastructure you forgot about.
The practical shift here is who gets to make this decision. Previously this was API-only, which meant it lived with whoever already knew the parameter space. Now a technical lead evaluating cost-performance trade-offs, or an ML engineer without deep infra background, can run the same benchmarking infrastructure through a guided workflow and get a ranked, deployable configuration out the other end, with the advanced API path still there if you want fine-grained control.
Community & Career đ¤
đ¤OpenChoreo treats agents as first-class users of the platform, not something bolted on after
âInternal developer platformâ has become one of those phrases that gets used so often itâs stopped meaning much on its own, mostly because most explanations of what one actually is stay at the level of an architecture diagram with no working system behind it. OpenChoreo is worth a closer look because itâs an actual running thing you can install and break, not a slide deck. Itâs open source, Apache 2.0, came out of WSO2, and is now in the CNCF Sandbox.
Structurally it looks like what youâd expect from a serious IDP: a Backstage-powered developer portal sitting on top of built-in CI and GitOps, with control, CI, data, and observability kept as separate planes rather than one tangled system. None of that alone would be worth a writeup. What made me stop and pay attention is that it exposes MCP servers as part of the platform itself. That means an agent interacts with OpenChoreo the same way a developer does, through the platformâs own interface, rather than through some API wrapper someone bolted on after the fact because agents became popular. As more teams start putting agents somewhere in their deploy path, âdoes the platform treat agents as first-class citizens or as an afterthoughtâ is going to become a real question people ask when evaluating an IDP, not just a nice-to-have feature.
đ¤llmfit hits v1.1.11, and the fixes tell you what kind of tool this actually is
llmfit, built by Alex Jones, answers one specific question: given hundreds of models across different providers and quantization formats, which ones will actually run on the hardware sitting in front of you, one command, no manual GGUF-size math. At 34.1k stars itâs clearly hit a nerve, the same nerve as the âwill this fit on a single DGX Sparkâ question that keeps coming up around new model releases, except llmfit tries to answer that question generically for any model and any GPU rather than requiring someone to work it out by hand each time.
The fixes in this release are the interesting part, because they show where the hard edges of that problem actually live. Hybrid attention wasnât being accounted for correctly in KV cache sizing, which means any model mixing local and global attention layers, increasingly common, was getting a wrong memory estimate. Pre-quantized models that already fit were being incorrectly flagged as insufficient, and Ollamaâs size-less family tags were causing smaller models to get misidentified as much larger ones. Each of those is a specific way that âjust estimate the memoryâ turns out to be genuinely hard once quantization formats, attention architectures, and inconsistent model naming all interact.
The community layer is worth noting too. This release folds in benchmark contributions for an NVIDIA GB10 DGX Spark class card with 122 models tested, plus results for several Apple Silicon and AMD GPUs from community contributors, which is exactly the kind of crowdsourced hardware coverage that makes a tool like this actually trustworthy across the long tail of GPUs people are running, not just the handful AWS or NVIDIA officially benchmark.
- Highlights â¨
â¨Amazon EKS now supports up to 10 external OIDC identity providers per cluster
This one solves a problem a lot of teams have quietly worked around for a while. If youâve got employees, contractors, and CI/CD systems authenticating through different identity providers, the old options were either consolidate everyone into one provider or stand up an identity broker in front of the cluster. Now you can associate each provider directly with the cluster, up to 10 of them, each configured and managed independently, and your existing IAM authentication keeps working alongside all of them. No additional cost, available everywhere EKS runs, and it goes through the same AssociateIdentityProviderConfig API you were already using for a single provider, so thereâs no new mental model to learn, just a limit that got lifted.
â¨EKS Capability for Argo CD now supports custom configuration via argocd-cm
This closes a real gap in the managed Argo CD capability. By default, Argo CD has no built-in health logic for Custom Resources, so an Application can report healthy while its resources are still provisioning, and sync waves can move forward before those resources are actually ready. Thatâs exactly the kind of thing that silently breaks a rollout and takes a while to trace back. You can now define custom health checks for your CRs, so a database resource, for example, can hold an Application at progressing until itâs genuinely ready, and itâs configured the same way youâd do it in upstream Argo CD, through the standard argocd-cm ConfigMap, so nothing new to learn if youâve run this before. Built-in health checks for ACK and kro resources also come for free, no extra config needed.
â¨HPA can scale to zero now, and itâs on by default
This one is simple enough to explain in a couple of sentences, which is rare for anything involving the autoscaler. Set minReplicas to 0 on a workload thatâs driven by object or external metrics, and Kubernetes will actually let it drop to zero running pods when thereâs nothing to do, then bring it back the moment work shows up. It doesnât work for CPU or memory-based scaling, for the obvious reason that you need a running pod to measure CPU or memory in the first place. But for anything that spends a lot of its life idle, queue consumers, batch jobs, GPU pods sitting around waiting for a request, this is about as direct a line to a lower cloud bill as a Kubernetes feature gets.
đ Sponsor Section
At the moment, we donât have a sponsor for this edition, but we look forward to working with companies and organizations that support the EKS & AI Infrastructure community in future editions. If you or your company is interested in sponsoring, please contact us at đ§ thecloudtechforall@gmail.com
đ Words from the Author
My friend Vinod Nair wrote something this week on EC2âs 20th anniversary that stuck with me, and I want to react to it rather than just point at it. His read is that the last five years of EC2âs timeline are really a story about compute economics shifting under us â silicon choice going from a footnote to a line item, Capacity Blocks changing how GPU planning actually works, Nitro Isolation Engine giving formal verification a real answer to âwhere do the model weights live.â Read his full piece, itâs sharper than anything Iâd write cold.
The line that stayed with me is the silicon point. Trainium3 and Graviton5 arenât AWS chasing a trend â they exist because general-purpose CPU and merchant GPU pricing stopped being the only option, and once that stops being true, âwhich instance typeâ stops being a procurement question and becomes an architecture question. Vinodâs other point, about Capacity Blocks mapping more cleanly to how training actually gets scheduled than on-demand or reserved instances ever did, is the same shift in a different place: the primitives are finally catching up to how the workloads actually behave, instead of forcing the workload to bend around a billing model built for something else.
But the part I keep returning to is his framing that EC2 shipped with one instance type and left room to grow. Twenty years later, the discipline hasnât changed even though everything built on top of it has â ship the primitive, let the workload tell you whatâs next. Itâs a good reminder that most of what looks like AI infrastructure innovation right now is really just that same discipline being applied one layer up.
Happy building. đ
