Seagate and SK hynix cut AI reply times by 95% by moving chatbot memory onto hard drives

In a test run by Seagate and SK hynix, the wait for an AI model to begin its reply fell from 35 seconds to under two. The two companies stored the model’s memory of each conversation on a mix of fast flash drives and cheaper hard drives, so the chip running the model no longer had to rebuild that memory every time a user came back. The same change cut the chip’s workload by 76% and its power use by 52%.

The results come from a white paper written by Kyung Soo Lee and Jin Na Yang of SK hynix and Thomas Prohofsky of Seagate. Its subject is the KV cache, short for key-value cache. As a model reads a conversation, it works out how each word relates to the others. The KV cache stores those calculations so the model can continue without reading everything again. The longer the conversation, the larger the cache, and the paper argues it has now outgrown the memory built to hold it.

“The challenge with inference is scale,” said Jason Feist, Senior Vice President, Cloud Business at Seagate Technology, in an email interview with The Source Code. Inference is the work a model does each time someone uses it. “As AI moves into production, systems must handle more concurrent requests, longer interactions, and retain more context.”

GPU memory runs out within minutes once AI agents get busy

AI chips such as NVIDIA’s H100 keep the cache in their own onboard memory, which is extremely fast and very small. The paper estimates that a busy chip can produce about 92TB of cache a day. The H100 has 80GB of memory, enough for roughly 1.2 minutes of it. A server’s main memory, at one terabyte, holds about 16 minutes.

AI agents make the problem larger. These are systems that carry out tasks over many steps, often by talking to other agents, and NVIDIA figures cited in the paper say they generate up to 15 times more text than a conventional chatbot. The memory that holds all this is also costly. Omdia prices quoted by the authors put the fastest chip memory at about $14 per gigabyte and standard server memory at about $8.50.

When memory fills, the system deletes the oldest conversations. If one of those users returns, the chip has to work through their entire conversation again. That makes the user wait, takes capacity away from other users and burns more electricity. “When context has to be repeatedly rebuilt, valuable GPU capacity is spent recomputing information the system has already processed,” Feist said. “What starts as a memory-capacity challenge quickly becomes a compute and cost challenge.”

Anyone who uses these tools will recognise the effect. A developer steps away from a coding assistant for lunch. A customer reopens a support chat the next morning. A finance team runs an automated workflow that stretches across several weeks. Each time they pick up again, the system may have to reread everything from the start.

Hard drives keep the memory while flash drives serve it

The test setup was modest. Four SK hynix flash drives and twelve 30TB Seagate hard drives, 360TB in total, were connected to a single NVIDIA H100 over a high-speed network link that lets the chip read storage directly. NVIDIA’s Dynamo software decided where each piece of cache was kept. The model was a small, publicly available version of DeepSeek, fed long prompts of about 21,000 tokens, the units of text AI models process, and asked for short answers of 100 tokens.

The system works like a filing cabinet and a desk. A user’s cache is first stored on the hard drives. When that user returns, the cache is copied to the flash drives and then loaded onto the chip, and every later reply in that session is served from flash. “SSDs can support active, latency-sensitive retrieval, while high-capacity hard drives can retain much larger amounts of context for longer periods,” Feist said.

The first reply after a break is the slow one. Compared with a system using flash drives alone, the hybrid setup took 7.6 seconds longer to answer the first question. By the 50th question, the difference had fallen to 0.35 seconds. Data also moved slowly at the start. Less than two minutes into the test, the hybrid system read data at 3GB per second against 25GB for flash alone. After about 17 minutes, it had reached 22GB against 26GB.

What the hybrid setup offers in exchange is space and a lower price. At the paper’s estimate of 92TB of cache a day, the hybrid pool could hold almost four days of it before filling up, against 8.3 hours for a flash-only pool, about 11 times as much. Storage cost about $0.04 per gigabyte, against $0.19 for flash alone, a saving of about 75%. “This is particularly valuable for long-running, multi-turn, and agentic workloads, where interactions may pause and resume, or the same context may be used repeatedly,” Feist said.

The 95% figure mostly shows the value of keeping memory at all

The headline improvement compares the hybrid system with one that stores no cache at all. The flash-only system in the same tests answered in about 0.2 seconds, faster than the hybrid. When the number of users grew from 10 to 50, the system with no stored cache slowed from 11 seconds to 209 seconds per first reply, while the flash-only system stayed at about 0.16 seconds. The largest gain comes from saving the cache somewhere. Hard drives make it affordable to save far more of it for far longer, with a slower first reply as the price.

The test was also small. It used one chip and a small model, and the authors say the setup was built to show what is possible, with no attempt to optimise it. Results for more than 12 hard drives were estimated. The network connection between the storage servers limited how much faster things could get, and the hard drive server ran on an old processor. The paper also contradicts itself on flash capacity: the equipment list shows 15.36TB, while the charts on storage time and cost use 32TB. The 92TB-a-day estimate is itself based on Microsoft’s figures for a 72-chip rack, so the hours quoted are best treated as a rough guide.

The authors’ employers also have an interest in the answer. Seagate sells hard drives. SK hynix sells flash drives, and it is also one of the world’s largest makers of the fast chip memory the paper describes as too expensive for this job. One reading is that this memory has become more valuable for running models than for storing old conversations.

Storage now has to be designed alongside the AI chips

Feist said storage can no longer be an afterthought. “The system needs to know where KV-cache data resides, when it should move between storage tiers, and how it can be shared across the system,” he said. In practice, software has to track every piece of cache and move it between chip, flash and hard drive as users come and go.

The connection matters as much as the drives. The direct link used in the test answered in under a tenth of a millisecond. Standard cloud-style storage, which works through web requests, adds several milliseconds to each request. The paper also found that the cache is read and written in small, scattered pieces, so the storage has to handle that kind of access well.

The software is catching up. NVIDIA’s Dynamo and the open-source LMCache project both manage this kind of storage, and the authors cite Amazon Web Services as evidence that the largest cloud providers already use flash and hard drives together for AI. Their plans for further work include warning the hard drives in advance when a user is about to return, comparing the setup with cloud storage, and compressing the cache so it takes up less space. Each is aimed at the same weak point: the slow first read from a hard drive.

AI budgets will increasingly depend on how long a model can remember

Larger models make the pressure worse. The paper notes that a 72-billion-parameter Qwen model needs almost six times more cache per word than its 7-billion-parameter sibling. Storage prices widen the gap further. Forward Insights data in the paper puts flash at 16 times the cost of hard drives per terabyte in 2026, up from a ten-year average of about 10 times, and Omdia puts server memory at 44 times the cost of flash.

“Organisations should plan beyond GPU capacity and think about compute, memory and storage as parts of the same AI infrastructure,” Feist said. “Over the next three to five years, the priority should be infrastructure that can adapt as AI workloads evolve.”

For companies weighing this up, the useful questions come from their own users: how long they pause, how often they return, and how much of an earlier conversation the model actually needs.