Elon Musk establishes the world's most powerful AI training cluster
Elon Musk announced that the world’s most powerful AI training cluster, named the Memphis Supercluster, has been established and is now operational for his AI company, xAI.
Elon Musk, a multi-company leader, manages several prominent companies: the automotive giant Tesla, space technology firm SpaceX, satellite internet provider Starlink, social media platform X (formerly known as Twitter), brain chip company Neuralink, and AI company xAI. Recently, the tech magnate announced that the world’s most powerful AI training cluster, named the Memphis Supercluster, has been established and is now operational for xAI.
Training the Most Powerful AI
For those unfamiliar, xAI is a very young AI company founded in 2023. The company is developing a large language model named Grok, which is offered to paid subscribers via a chatbot of the same name on X. Musk founded xAI as a direct competitor to OpenAI, aiming to create the most advanced artificial intelligence.
Nice work by @xAI team, @X team, @Nvidia & supporting companies getting Memphis Supercluster training started at ~4:20am local time.
— Elon Musk (@elonmusk) July 22, 2024
With 100k liquid-cooled H100s on a single RDMA fabric, it’s the most powerful AI training cluster in the world!
In line with this goal, the Memphis Supercluster has been brought online. This AI training cluster consists of 100,000 liquid-cooled Nvidia H100 GPUs, all interconnected via a single RDMA (remote direct memory access) structure. These GPUs are currently highly sought after, with everyone wanting to acquire thousands of them.
Nvidia plans to release its more powerful H200 chips later this year. Additionally, next-generation, more advanced Blackwell-based B100 and B200 GPUs will enter mass production next year.
Meanwhile, the Memphis Supercluster, described as the world’s most powerful AI training cluster, will be used to train the most powerful AI in the world, according to Musk. Grok 1.5 has already been released, and Grok 2 is expected to debut in the coming months. Musk previously announced that Grok 3 would be released in late 2024. In his latest update, he stated that this “most powerful AI” would complete its training by December.
The Memphis Supercluster is on an unprecedented scale. The newly established cluster easily surpasses all systems on the Top500 list. Frontier, which tops this list in terms of computational power, uses 37,888 AMD GPUs. Other supercomputers like Aurora (60,000 Intel GPUs) and Microsoft Eagle (14,400 Nvidia H100 GPUs) are also significantly outperformed.
