AI World Drama – A New Player, Although a One-Hit Wonder?
For the past year, OpenAI had been the center of attraction in the AI world with its advancements and achievements in the field. However, recently, a new contender has emerged, causing some commotion. While their claim of having the best open-source large language model (LLM) made headlines, things didn’t quite add up. An individual by the name of Matt Shumer announced their achievement on September 5, but their story ultimately unraveled.
Data-Driven Insights
Matt Shumer made headlines when he announced having the top open-source LLM in the world with a 70B model. According to benchmark scores, the model seemed more advanced than Llama 405B and on par with commercial LLMs. However, it got worse as it became obvious that something was off with the scores.
Inconsistencies started unfolding when the model uploaded to HuggingFace didn’t work as expected. Instead of a clear explanation for the malfunction, Matt apologized and stated that they had re-trained the model to resolve the issue. Confusion arose as users, including other AI experts and developers, began questioning whether the initial claim was reliable.
The Model Drama
The most surprising aspect was the similarity between the “newly uploaded” model and a previous one. Incredibly, it seemed an attempt was made to dupe users into believing their LLM had been better than previously claimed. Subsequently, a thorough study was conducted, revealing disturbing similarities between the models.
A number of curious users replicated the benchmark’s results and discovered they came up short. To counter these claims, the user who created the LLM model (Matt Shumer) retrained it – again! Still, most people have doubts regarding honesty. Furthermore, Matt declared they will re-train their LLM, thus implying a loss of effectiveness, yet again!
The Reason Behind This Mess
The AI world has turned out to be more uncertain, leaving experts in discomfort. No clear reasons will ever emerge to support honesty in such matters. Here’s an in-depth glance at Matt’s explanations while also reviewing a couple dozen Twitter tweets about his adventure, from uploading an alternate version, or possibly sharing Claude model prompts for any model API and not addressing his original issue with incorrect uploaded model.
Evidently, a full-stop hasn’t been reached while in the ongoing pursuit seeking a truthful response to its model drama. This investigation should ultimately yield some conclusion regarding whether there is much substance or whether we’d all rather stay ignorant given that such matters usually concern financial profits.
