SAN FRANCISCO - OpenAI has been looking beyond Nvidia for some of its artificial intelligence processing needs, people familiar with the company told Reuters, signaling a significant tactical shift that centers on chips optimized for inference. The change, which sources say dates back to last year, highlights growing attention on the part of AI developers to hardware that can return answers to users more quickly for particular workloads.
According to eight individuals with knowledge of the situation, OpenAI's review of alternatives is driven by mounting concern about the speed of certain Nvidia hardware when handling inference tasks - the moment a trained model generates a response to a user's request. While Nvidia maintains a dominant position in GPUs used to train large AI models, inference is emerging as a distinct and competitive segment of the market.
The shift in OpenAI's procurement strategy has occurred alongside prolonged negotiations between the two companies over a financial and strategic arrangement. In September, Nvidia announced an intention to invest as much as $100 billion in OpenAI under a structure that would give Nvidia a stake and provide OpenAI with cash to acquire advanced chips. That announcement had been followed by expectations the deal would close in a matter of weeks, yet sources said talks have continued without resolution for several months.
As those discussions have stalled, OpenAI has pursued partnerships for hardware alternatives. Sources said the company struck deals with AMD and other suppliers that produce GPUs intended to contest Nvidia's position. At the same time, changes in OpenAI's product roadmap influenced the types of compute resources it needs, which in turn complicated the investment talks with Nvidia, a person familiar with the discussions said.
Nvidia's chief executive Jensen Huang publicly dismissed reports of friction between the companies, calling the notion "nonsense," and reiterated the chipmaker's plans for a large investment in OpenAI. In a statement, Nvidia added that "customers continue to choose NVIDIA for inference because we deliver the best performance and total cost of ownership at scale." An OpenAI spokesperson likewise said the company relies on Nvidia to power the vast majority of its inference fleet and that Nvidia delivers the best performance per dollar for inference.
Despite those public stances, seven sources told Reuters that OpenAI is not satisfied with how quickly some Nvidia hardware can produce answers for certain use cases - specifically workloads such as software development and integrations where AI must interact with other pieces of software. One source said OpenAI anticipates that the new class of hardware could account for roughly 10% of its future inference computing needs.
To address those speed concerns, OpenAI explored chips from startups focused on designs that embed large amounts of static random-access memory - SRAM - directly on the chip. Embedding sizable SRAM close to compute units can reduce memory access latency and thus improve responsiveness for chatbot and coding workloads. Two companies named by sources as targets for discussion were Cerebras and Groq.
Sources described a competitive dynamic in which Nvidia, seeking to strengthen its position across these evolving requirements, entered into a $20 billion licensing deal with Groq. That arrangement, the sources said, effectively curtailed OpenAI's talks with Groq. Industry executives described Nvidia's hiring of Groq designers and licensing of Groq technology as moves to broaden the company's portfolio in an area where SRAM-heavy architectures could offer advantages for inference.
In response to Reuters' questions, Nvidia said Groq's intellectual property complemented Nvidia's product roadmap. The company also approached other firms developing SRAM-centric chips about potential commercial or strategic relationships, according to people familiar with the discussions.
Cerebras declined an acquisition approach and instead finalized a commercial agreement with OpenAI that was announced last month. Cerebras did not comment for this report. Groq, which had held discussions with OpenAI about providing computing capacity and attracted investor interest at a valuation reported by sources to be about $14 billion, also declined to comment. By December, sources said Nvidia had moved to license Groq's technology in a non-exclusive all-cash deal while hiring key chip designers away from Groq.
Technical factors underlie much of the debate over hardware choice. The sources described inference as a memory-intensive activity relative to training because inference workloads spend comparatively more time fetching data from memory than carrying out raw mathematical operations. Nvidia and AMD's GPU designs use external memory architectures; fetching data across the memory interface introduces latency that can slow how swiftly a model can answer user prompts.
On the other hand, chips that cram more SRAM onto the same silicon as the compute fabric can cut down those access delays and therefore improve responsiveness for latency-sensitive tasks. The internal conversation at OpenAI, according to sources, made this trade-off particularly evident with Codex, OpenAI's product for generating computer code. Some OpenAI staff attributed portions of Codex's performance shortcomings to constraints tied to GPU-based hardware, one of the people said.
OpenAI's chief executive, Sam Altman, told reporters on a January 30 call that customers using OpenAI's coding models will "put a big premium on speed for coding work." He added that the company's commercial agreement with Cerebras is one avenue to meet that demand, and that speed is less critical for casual ChatGPT users.
Competing AI services, including Anthropic's Claude and Google's Gemini, also highlight the range of approaches now in play. Sources said those platforms benefit from deployments that leverage Google's internal tensor processing units - TPUs - which are designed specifically to perform calculations required in inference and can present performance advantages over general-purpose GPUs in certain scenarios.
As OpenAI reassessed its reliance on Nvidia hardware, the chipmaker pursued its own strategy to secure technology and talent in the SRAM-focused segment. People familiar with the matter described Nvidia as having approached Cerebras and Groq about acquisition possibilities, moves that appear aimed at ensuring access to designs that emphasize embedded memory and low-latency inference performance.
Cerebras instead opted for a commercial tie-up with OpenAI. Groq engaged in talks with OpenAI and drew investor interest but then entered into a licensing arrangement with Nvidia by year-end, according to sources. The licensing deal would permit other firms to license Groq's technology, yet Groq has shifted emphasis toward offering cloud-based software after key design staff were hired by Nvidia.
Implications and context
The developments outlined by sources suggest a testing of market positions as AI applications multiply and diversify. Nvidia continues to claim strength in inference at scale, and OpenAI publicly affirms Nvidia's role across much of its fleet. Yet the private sourcing conversations and commercial agreements signal a push by a major AI developer to secure a mix of hardware optimized for specific low-latency workloads.
How these procurement choices and strategic moves evolve could influence competition among chip designers, the deployment options for cloud providers, and the user experience for software that relies on fast inference, such as coding assistants and applications that require rapid model-to-software interaction.
Data disclosures
The reporting in this article reflects details provided by multiple sources with knowledge of the companies' internal discussions and commercial arrangements. Companies named in this piece provided the statements attributed to them; several private firms declined to comment.