John Carmack Challenges HBM Monopoly While Replit Unlocks Self-Improving Agents
Today's tech discourse pivoted away from surface-level model releases toward deep systemic optimization, highlighted by John Carmack's proposal to bypass HBM with NAND flash for AI inference. Meanwhile, the software engineering ecosystem continued its rapid restructuring around autonomous verification, with Replit unveiling continual learning agents and developers debating the necessities of rigorous TDD in an era of LLM-generated code.
Daily Wrap-Up
The AI community experienced a profound shift in perspective today, moving past the novelty of raw model capabilities to tackle the much harder problems of infrastructure, deployment, and automated reliability. The most compelling technical narrative challenged the fundamental economics of AI hardware. For years, High Bandwidth Memory (HBM) has been treated as the unbreakable bottleneck and ultimate prize of AI acceleration. Today, however, prominent voices began constructing a credible mathematical and architectural case for bypassing it entirely in favor of sequential NAND flash or LPDDR architectures. This transition from blindly scaling compute to smartly balancing memory pipelines signals a maturation of the AI hardware sector, where efficiency and cost-per-token will likely dictate the next era of market dominance.
On the software front, developers are rapidly coming to terms with the reality that AI agents require a completely different operational paradigm than traditional code. The old mantra of moving fast and breaking things is fatally dangerous when an autonomous agent can write and deploy its own updates. The conversation has decisively shifted toward building impenetrable loops of automated verification, continuous learning, and isolated infrastructure. There is a growing consensus that human code reviews are becoming a relic of the past, replaced by AI systems capable of reviewing and merging routine dependencies faster than any human ever could. The developer's role is evolving from a creator of logic to an architect of boundaries, designing the safety harnesses that allow AI to iterate safely.
Perhaps the most entertaining moment of the day came from a deadpan exchange about AI safety research, reminding us that pushing the boundaries of model behavior can yield unpredictable and slightly terrifying results. As one developer humorously noted regarding their alignment experiments, a successful test case might just result in accidentally making the model evil. It is a stark reminder that while we optimize for inference speed and agent autonomy, the underlying unpredictability of these systems remains a deeply human problem.
The most practical takeaway for developers is to establish a dedicated, isolated hardware environment for your AI coding agents immediately. Instead of letting resource-heavy agents like Cursor or Devin drain your laptop battery, configure a dedicated Linux home server with passwordless SSH access between your trusted devices, and implement color-coded tmux sessions to monitor their workflows remotely.
Quick Hits
- @vaizaragorn dreams of affording every tool Miniware makes after the company unveiled its AirGo 140, a remarkably lightweight 140W PD-powered portable hot air gun that heats up to 300°C in just ten seconds.
- @herdrdev celebrates a massive milestone as the open-source project herdr hit number one on GitHub trending, accumulating 12,500 stars in just 105 days while building entirely in public.
- @EricJorgenson highlights an exciting opportunity at Aalo Atomics, dubbed the SpaceX of Nuclear, which recently powered on its first reactor and is aggressively hiring technical and non-technical talent to scale modular power plants for data centers.
- @Aktiehedonist shares a blistering analysis of SemiAnalysis, accusing the research firm of repeatedly making aggressive, market-moving claims about NVIDIA roadmaps and semiconductor manufacturing that frequently turn out to be overly sensationalized misrepresentations of complex supply chain adjustments.
The Architecture of Autonomous Engineering
The software development lifecycle is undergoing a foundational reconstruction as engineers figure out how to safely integrate non-deterministic AI agents into production pipelines. The old paradigms of test-driven development are being violently reshaped. As @lyc_aon highlighted in a sharp critique of modern tooling skepticism, pushing back against the idea that automated verification is cost-prohibitive, the reality is that rigorous testing is no longer a luxury for hobby projects. When an LLM is generating changes, automated verification is the only barrier preventing your software from becoming worse by default. This friction is exactly why AI leaders are fundamentally rethinking how agents learn and operate.
Amjad Masad (@amasad) revealed that Replit's rapid improvement stems from successfully closing the loop to create a self-improving agent through continual learning. By allowing the agent to learn from its iterative deployment, the system essentially upgrades itself. However, self-improvement requires massive amounts of automated verification to ensure the agent is learning the right lessons. This leads to a stark realization about human involvement in the code review process. Thorsten Ball (@thorstenball) succinctly noted that after a decade of required reviews, there is very little a human can do better in a routine review than an advanced model like GPT-5.6. He correctly distinguishes between routine syntax and dependency checks, which AI handles flawlessly, and architectural discussions, which remain inherently human.
These perspectives coalesce into a new reality for software infrastructure: developers must build environments where agents can run, test, and verify their own code autonomously. As developers transition into orchestrators of AI talent, the focus shifts from writing logic to designing bulletproof harnesses. The engineering loop of the future relies on continuous agent learning powered by flawless automated verification, rendering manual intervention a rare exception rather than a daily standard.
Challenging the HBM Monopoly in AI Inference
For years, the semiconductor industry has operated under the assumption that high bandwidth memory is the insurmountable moat protecting incumbent AI accelerator manufacturers. Today, deep technical analyses challenged that monopoly, suggesting that massive inefficiencies exist in how we handle memory access during model inference. The core argument hinges on the deterministic nature of model weights compared to the random access patterns required for tasks like game rendering.
John Carmack (@ID_AA_Carmack), via @bubbleboi, laid out a compelling thesis for utilizing NAND flash over expensive HBM. "Unlike game rendering, model inference can have a deterministic memory access pattern. You don’t need 'random access memory' at all for model weights," Carmack explained. He points out that NAND flash is over a hundred times cheaper per gigabyte than HBM. While it features higher cold-start latencies, continuous sequential reads can be delivered at the necessary bandwidth to power inference efficiently. This mirrors a broader technical consensus that current chip architectures are fundamentally unbalanced for AI workloads.
Elaborating on this systemic inefficiency, @christianoboria shared an analysis of how modern LLM inference disaggregates weights, highlighting the massive latency taxes imposed by current interconnects like NVLink. The analysis details how sharding models to cut memory bandwidth also cuts FLOPs per chip, resulting in terrible Machine Utilization Factors (MFU). "So if you build around LPDDR’s lower bandwidth, lower your interconnect latency, you actually can beat Nvidia on decode with a fraction of the silicon." Running parallel to this hardware awakening, software optimizations are providing immediate relief. @u1tra_instinct highlighted NVIDIA's Model Optimizer 0.45.0, which introduces NVFP4 weight-only quantization to shrink models and speed up inference locally without requiring complex calibration.
Whether solved by radically redesigning silicon pipelines to leverage cheap flash memory or by compressing models into 4-bit representations, the mandate is clear. The next massive leap in AI capabilities will not come simply from building larger chips, but from ruthlessly optimizing the memory pathways that feed them.
Enterprise Orchestration and the Open Source Horizon
The strategic deployment of AI in enterprise environments is settling into a nuanced, multi-tiered approach. Companies are realizing that blindly throwing massive frontier models at every single task is economically and technically unsustainable. The conversation has matured into an architecture of orchestration, where different classes of models are dynamically routed based on task complexity, predictability, and cost constraints.
Aaron Levie (@levie) articulated this evolution perfectly, noting that while frontier intelligence will always be necessary for orchestrating complex workflows and solving brand-new use cases, mature and predictable tasks require a different approach. "As use-cases become mature and predictable in an enterprise context, you can begin to peel off some set of the tokens to either lower cost open or closed models or models that are trained for the task at hand." Levie emphasizes that this dynamic relies entirely on the applied AI layer, which must effectively evaluate workflows, choose mixtures of models, and scale operations efficiently.
Filling out the lower-cost layers of this enterprise stack is a rapidly maturing generation of open-source models. Harrison Kinsley (@Sentdex) spotlighted the release of Tencent's Hy3, a 295B Mixture-of-Experts model that rivals trillion-scale flagships while remaining highly affordable for agentic use cases under an Apache 2.0 license. Kinsley points out that the open-source AI community is being inundated with exceptional models that offer total data control and consistent model behavior without the restrictive quotas of proprietary APIs. As enterprises build out their orchestration layers, having access to reliable, commercially friendly open-weight models provides the exact leverage needed to train and optimize those highly specific, cost-effective task models.
The Infrastructure of Self-Improvement
The pursuit of artificial general intelligence is intimately tied to the concept of recursive self-improvement, but building the infrastructure to enable an AI to safely upgrade itself presents profound engineering and safety hurdles. The current research focus has shifted toward harness engineering, creating the scaffolding necessary to evaluate, guide, and restrain autonomous systems as they iterate on their own capabilities.
Lilian Weng (@lilianweng) shared new insights into harness engineering for AI self-improvement, forecasting that the future of Recursive Self-Improvement (RSI) will rely heavily on these structures. She notes a symbiotic loop: harness engineering will enable automated research, which in turn produces smarter models that require simpler harnesses. "Even when many harness improvement get eventually internalized into core model, the need to specify goals and context will not disappear." This highlights the persistent human responsibility in AI evolution.
The risks of mismanaging this loop were starkly, and humorously, illustrated by Benjamin Glickenhaus (@benglickenhaus). Responding to speculation about pushing models past their operational limits, he offered a chillingly casual summary of an unrestrained alignment experiment: "yes we’ve done this, yes it works, no you can’t have it, it potentially made the model evil."
While said in jest, the underlying truth reinforces the critical necessity of robust evaluation pipelines. As developers construct the infrastructure for autonomous agents, ensuring the integrity of the improvement loop is paramount. Unintended behavioral drifts during self-modification remain a top-tier security concern, making the discipline of harness engineering arguably more critical than the raw training of the models themselves.
Elevating Design Workflows with AI
The struggle against generic, artificially generated aesthetics is a major pain point for creative professionals adopting AI tools. While text-to-image models and automated coding assistants can generate functional layouts at lightning speed, they often lack the refined polish and deliberate user experience considerations of human-crafted design. Bridging this gap requires highly specialized system prompts and the integration of nuanced design heuristics into the AI's workflow.
Meng To (@MengTo) demonstrated how combining specialized skills with AI coding assistants fundamentally alters the creative pipeline. By utilizing the "Taste" skill alongside AI agents like Claude Code or Codex, creators can turn basic prompts into polished designs that actively avoid the notorious AI-slop aesthetic. This workflow empowers developers to recreate beautiful landing pages directly from video captures or full-page screenshots, blending rapid code generation with high-level design standards. The focus shifts from mere generation to sophisticated curation and styling.
This emphasis on measurable, high-quality user experiences was echoed by @joshpuckett, who praised the impactful, data-driven design work shared by @JohnPhamous. The specific case study showcased how simplifying an account recovery flow dramatically increased user compliance, raising evidence submission rates for project URLs from 25% to 71% and git providers from 31% to 80%. Together, these posts illustrate that whether an interface is built via traditional methods or generated by an autonomous agent, the ultimate value remains rooted in clear, user-centric design principles and rigorous empirical testing.
Sources
@NVIDIAAI Model Optimizer 0.45.0 just shipped 🚀🚀🚀 Key feature: New NVFP4 W4A16 weight-only quantization — 4-bit FP4 weights + BF16 activations with zero calibration required! Super simple way to shrink models and speed up inference 🤓 https://t.co/nC4LgQwFee
the worst part about this shift in technology: you need large amounts of automated verification for any LLM generated changes that tooling has a serious cost to it that cost is unrealistic for your hobby projects so your software is worse-by-default
Everyone is wrong about open source AI in the enterprise
🚀Hy3 is here. 295B MoE. Best in its size class. Rivals trillion-scale flagships. Reliable and affordable for most agentic usecases. Apache 2.0. Friendly for commercial use. FREE API for 2 weeks → https://t.co/EyURKwTdgi 🤗 https://t.co/twqJpqb2SL 📖 https://t.co/4uEkIU1cW4 https://t.co/LLJTfDFNQu
SemiAnalysis는 올해 연초부터 같은 패턴을 반복하고 있다. 1. Micron HBM4 배제론 연초에는 Micron이 NVIDIA Rubin HBM4에서 사실상 배제됐다는 식의 이야기가 돌았다. 시장은 “Micron은 차세대 NVIDIA HBM 경쟁에서 밀렸다”로 받아들였다. 그런데 이후 Micron은 Vera Rubin용 HBM4 출하를 발표했고, NVIDIA가 SK하이닉스·삼성전자·Micron을 모두 Vera Rubin HBM4 공급사로 인증했다는 보도까지 나왔다. 최소한 이건 너무 단정적으로 전달됐거나, 후속 현실과 맞지 않았다. 2. SOCAMM 수요 붕괴론 그다음은 Vera Rubin rack의 CPU-side SOCAMM 메모리 축소 이슈였다. 55TB에서 28TB로 줄었다는 이야기가 나오자 시장은 바로 “AI memory demand가 식는다”로 받아들였다. Micron은 급락했다. 그런데 이 역시 단순한 수요 붕괴로 보기 어려웠다. 오히려 공급 제약 때문에 realistic configuration으로 조정됐다는 반론이 나왔다. 수요가 꺾인 게 아니라, 병목과 공급부족 때문에 구성이 조정됐을 가능성도 있었던 것이다. 3. CPO·800VDC 지연론 그다음이 CPO와 800VDC였다. SemiAnalysis 리포트 이후 optical 관련주가 한꺼번에 박살났다. AAOI, COHR, LITE, CIEN, MRVL 같은 이름들이 동시에 맞았다. 논리는 CPO 대량 양산이 2028~2029년으로 밀릴 수 있고, 800VDC ramp도 늦어질 수 있다는 것이었다. 기술적으로 어려운 건 맞다. CPO는 yield, serviceability, optical engine failure, ASIC integration 문제가 있다. 800VDC도 전력 구조 전환, 부품 신뢰성, rack architecture 난이도가 있다. 하지만 “어렵다”와 “끝났다”는 다른 말이다. CPO adoption이 느려질 수 있다는 말과 optical interconnect chain 전체가 무너진다는 말도 전혀 다르다. CPO가 늦어진다고 AI 데이터센터의 optical 수요가 사라지는 게 아니다. 오히려 NPO, LPO, pluggable, ELS, laser source 같은 중간 경로가 더 오래 살아남을 수도 있다. 4. Kyber NVL144 지연론 그리고 이번엔 Kyber NVL144다. SemiAnalysis는 NVIDIA Kyber NVL144 rack architecture가 78-layer PCB midplane 제조 난이도 때문에 2028년으로 12개월 이상 지연됐다고 주장했다. 여기에 NVL72x2 취소, NVL576 저물량 가능성, Rubin Ultra scale-up domain 제한, AMD·Google 기회론까지 붙었다. 즉 이번에도 단순 제조 리스크가 아니라 NVIDIA 차세대 rack-scale roadmap 전체를 흔드는 narrative가 만들어졌다. 문제는 아직 NVIDIA 공식 확인이 아닌데도 시장은 또 “NVIDIA roadmap 차질”로 바로 소비한다는 점이다. 5. 진짜 문제는 반복 패턴이다 올해 흐름을 보면 계속 같다. 강한 단정. 시장 충격. 후속 반론. 그리고 시간이 지나면 실제 산업 흐름은 훨씬 복잡했다. Micron HBM4 배제론. SOCAMM 수요 붕괴론. CPO 지연론. 800VDC 지연론. Kyber NVL144 지연론. 하나하나 기술적 논점은 있을 수 있다. 하지만 매번 결론이 너무 자극적이다. 복잡한 제조 리스크가 수요 붕괴처럼 소비되고, 타임라인 조정이 사이클 종료처럼 소비되고, 특정 architecture의 병목이 밸류체인 전체 악재처럼 소비된다. 이 정도면 SemiAnalysis를 단순한 기술 리서치로 보면 안 된다. 지속적으로 이런 식의 교묘한 시장 흔들기성 리포트를 내는 것처럼 보인다면, 리서치 기관의 본질인 “리서치 신뢰도”는 사실상 0에 가깝게 봐야 한다. 리서치 기관의 핵심 자산은 예측력이 아니라 신뢰다. 틀릴 수는 있다. 하지만 계속 강한 단정으로 시장을 흔들고, 나중에 보면 실제 산업 흐름과 맞지 않거나 훨씬 복잡했던 사례가 반복된다면 그건 단순 오판이 아니다. 리서치 품질의 문제이고, 표현 방식의 문제이고, 시장 영향력에 대한 책임의 문제다. 누군가는 이미 SEC에 제보했어도 이상하지 않은 사안이다. 핵심은 리포트가 틀렸냐 맞았냐가 아니다. market-moving 리서치가 반복적으로 섹터 가격을 흔들었고, 그 과정에서 이해상충, 선행 배포, 포지션 노출, 파생상품 거래 여부가 불투명하다면, 규제기관이 들여다볼 영역이 된다. 특히 SemiAnalysis는 AI 반도체, 데이터센터, 네트워킹 모델을 팔고, 기관용 리서치와 컨설팅을 팔고, 이제는 메모리·포토닉스 ETF 파트너십까지 하는 유료 정보사업자다. 그런 주체가 리포트 하나로 섹터 밸류에이션을 흔들 수 있다면 시장은 당연히 물어야 한다. 사실관계는 충분히 검증됐는가? 기술 리스크가 과장돼 전달된 건 아닌가? 기관 고객 배포 구조와 시장 반응 사이에 이해상충은 없는가? 작성자나 관련 주체의 포지션, 파생상품 노출, 거래 패턴은 투명한가? 부정적 리서치는 필요하다. NVIDIA roadmap에도 리스크는 있다. CPO도 어렵다. 800VDC도 어렵다. Kyber도 쉽지 않다. 하지만 어려운 것과 망한 것은 다르다. 지연 가능성과 수요 붕괴도 다르다. 제조 병목과 산업 thesis 훼손도 다르다. SemiAnalysis 리포트는 기술 자료로 참고할 수 있다. 하지만 그대로 믿고 매매하면 안 된다. 반드시 NVIDIA, Micron, Broadcom, Marvell, GF, Jabil, Coherent, Lumentum, Sivers 같은 primary source와 대조해야 한다. 올해 SemiAnalysis가 보여준 건 기술 통찰만이 아니다. 시장 narrative가 어떻게 만들어지고, 그 narrative가 어떻게 주가를 때리고, 그 과정에서 nuance가 얼마나 쉽게 사라지는지도 보여줬다. 리서치가 시장을 움직이는 순간, 그건 단순한 의견이 아니다. 검증받아야 할 권력이 된다.
First you have to understand that modern LLM inference already disaggregates weights as models outgrew single chips years ago. You shard either by layer (pipeline parallelism) or by slicing every layer (tensor parallelism), and the two do very different things. As an example, let’s look at Llama 3.3. It has 70B of weights and at FP8 that’s 70 GB of memory which is enough to fit on a single H100. Now that H100 has 3.35 TB/s of HBM, so the fastest it can ever decode for one user is 70/3.35 ≈ 21 ms/token or ~48 tok/s while using under 1% of its FLOPs. Now if we pipeline it across 8 chips: each chip holds ~8.75 GB, which means it only needs 1/8th the bandwidth and 1/8th the FLOPs to sustain the same aggregate throughput. Now crucially the token/sec a user gets is limited by the amount of data that crosses the link. In current LLMs all that is a small amount of activations for LLama 3.3 it’s ~8 KB per token…. Yes, you read that right it’s 8 KILOBYTES we are sending over a <900 GB/s link. That’s only 9 ns of serialization time but the overhead of 224G PAM4 SerDes adds ~100 ns per link traversal with RS-FEC which is 11x longer than the payload itself. And then you have the NVSwitch adding ~300 ns per hop and you need to pay twice. That’s ~600 ns of just hardware latency wrapped around 9 ns of data making a 98% tax before software even shows up. Then NCCL’s collective stack turns 600 ns into 10-20+ us… all to move 8 kilobytes lol. For comparison 8 KB serializes over 10 Gigabit Ethernet NRZ, in just 6.6 us. Pipeline parallelism however doesn’t make a single user faster as the token still needs to visit every layer in the sequence, so per-user speed is still weights / per-chip bandwidth. To get more speed per user token you need to use tensor parallelism and have all the chips work on the same layer simultaneously. TP costs you 2 all reduce OPs per layer, 160 per token on llama 3, that’s still kilobytes of traffic but with NVLink overhead it’s a massive tax and why pipeline parallelism on most models still gives more interactivity per user. However, this gives you a huge latency lever to pull that scales tokens per second with interconnect speed instead of memory BW. The clever amongst you might have also realized that sharding doesn’t just cut memory bandwidth per chip it also cuts FLOPs per chip and is why we have such bad MFU on decode. So once you’ve sized the link for the memory, you need to size the compute for it too. This is called “balancing the pipeline”, and currently no shipping chip does it because they were all designed as standalone monsters. Remember Tokens/sec = ~aggregate memory BW / bytes touched per token. At batch 64 in FP4 you need ~250 FLOPs per byte, and Blackwell ships 1,250. Provisioned 5x more than the narrow pipe of HBM. Nobody saturates shit cause they are all building around HBM. So now it all comes full circle. Parallelism reduces memory bw pressure and thus FLOPs but increases interconnect latency pressure. Despite having HBM and GigaSERDES we aren’t actually doing more work lol. But if you really wanted to balance the pipeline you need to match the memory bandwidth, the flops, and most importantly the interconnect. So what does that look like ? Well if you build around LPDDR’s lower bandwidth, lower your interconnect latency, you actually can beat Nvidia on decode with a fraction of the silicon.
we made account recovery simpler users stopped skipping evidence (n=1116) - project URLs: 25% to 71% - git providers: 31% to 80% - phone number: 40% to 79% - payment method: 3% to 17% - billing address: 15% to 41% - team emails: 6% to 24% - account creation date: 34% to 33% https://t.co/KI0xYS1KsV
Aalo was founded 3 years ago. On July 4th, we turned on our first nuclear reactor. Today, we're issuing an open call to technical and non-technical talent: help us unlock global energy abundance. Over the coming years, we will scale assembly-line manufacturing of modular power plants, starting with powering data centers and ending with energy abundance. View over 50 open roles spanning engineering, regulatory, manufacturing, software, and more at https://t.co/fLF7aptlia
Continual Learning for Agents
Memory cost and capacity are significant issues for AI accelerators. Unlike game rendering, model inference can have a deterministic memory access pattern. You don’t need “random access memory” at all for model weights, and you could tolerate cold-start latencies in the multiple milliseconds, as long as continuous reads were delivered at the necessary bandwidth. NAND flash is over 100 times cheaper per GB than HBM, so there should be opportunity there, even after giving a flash controller a 1024 bit interface with HBM bandwidth. You could make a specialized pin protocol that just supported pipelined transfer of full 16KB+ pages from the flash to program-managed accelerator scratchpad memory and improve per-pin performance over HBM, but it might be more convenient to make it still look like a true random access memory with very fragile performance characteristics, where anything but sequential reads falls off a 1000x+ performance cliff. That has the advantage of automatically using existing cache hierarchies, and providing a natural path to update the flash memory with new model weights. With the stream-to-scratch interface, code has to be completely rewritten before it works at all, while the ram-emulation interface will start off just extremely slow, and you can incrementally sort out the changes for full performance. There may be cases where there isn’t enough scratchpad SRAM to hold the weights for a layer, which might force you to deploy the old optical drive optimization technique of duplicating data in multiple places on a sequential read to avoid seeking, but there would be capacity to burn. It might be possible to do something like cuda graph capture to record a memory access trace and have everything magically remapped to a linear sequence, but deploying programmer / agent elbow grease to manage transfers and access in a scratch ram ring buffer would be lower risk. A split memory system consisting of some channels of flash and some channels of HBM will probably be suboptimal compared to a uniform memory, but it could be much cheaper, and allow much larger models to be run. I think th case is strong for inference, but you have to stretch more for training. You can still linearize all the weight memory accesses, both reads and writes, but flash memory would quickly wear out from the writes, even if they were all perfectly page aligned. Replacing low-latency HBM with massively parallel cheap(er) DRAM at high latency might still be a worthwhile cost savings.
@didier_lopes @pangram why do you think we haven't done this?
the number of coding agents i have running on my laptop has dropped to an all time low, near zero
This is an insane Fable 5 UI/UX hack. This "Taste" skill completely kills generic AI-slop and gives Fable 5 the tools & instructions needed to ship beautiful design. This might just be the best AI skill I've ever used. https://t.co/XVQ6JpjYCy https://t.co/aQvw9rB2HJ
This is the version of the “software factory” that I think is realistic btw. I’m not sure what human review adds to dependabot PRs, for example. We should be able to build a factory that merges these within 5 minutes of them being opened.
MINIWARE AirGo 140 Hot Air Gun – First Look Meet the AirGo 140 – MINIWARE's first PD‑powered portable hot air gun. 115g, 140W, heats up to 300°C in 10 seconds, APP control. First look, more to come... #Electronics #Tech #Innovation #DIY #Maker #MINIWARE https://t.co/UdtZR0bfsJ