Elon Musk’s xAI has made a bold move over the weekend, releasing its Grok large language model as an open source resource. While this may seem like a strategic move to differentiate from rival company OpenAI, it begs the question – does contributing the code for a model like Grok truly benefit the AI development community? The answer, as always, is not a simple one.
Grok is a chatbot created by xAI to fulfill a similar role as ChatGPT or Claude – you ask a question and it provides an answer. However, this large language model has been imbued with a sassy tone and additional access to Twitter data, setting it apart from other models in the market.
While it is nearly impossible to accurately evaluate these systems, experts agree that Grok is on par with previous medium-sized models such as GPT-3.5. Whether this is considered impressive given its short development time or underwhelming given the resources and hype surrounding xAI is subject to individual interpretation.
Regardless, Grok is a modern and fully functional LLM with substantial capacity. The more access the development community has to the inner workings of such models, the better. However, the issue lies in defining the term “open” – is it simply a way for companies or billionaires to claim the moral high ground?
This isn’t the first time the terms “open” and “open source” have been called into question or misused in the field of AI. And we aren’t just talking about a minor technical discrepancy, like choosing a less “open” usage license (Grok uses the Apache 2.0 license).
The truth is, AI models are significantly different from traditional software when it comes to being labeled as “open source.” It may be relatively straightforward to release the source code for a word processor, allowing the community to suggest improvements or create their own version. One of the key benefits of open source software is its transparency and proper attribution to its original creators. However, in the case of AI models, this level of transparency and credit is often unachievable.
When designing an AI model, a complex and largely unknowable process is involved, where vast amounts of training data are used to create a statistical representation. This process cannot be examined or improved in the same way that traditional code can. While it may hold immense value, it can never truly be “open” in the same sense. The standards community is actively discussing and defining what “open” will look like in this context.
This has not stopped AI developers and companies from claiming their models as “open,” a term that has lost much of its meaning in this context. Some may refer to their model as “open” simply because it has a public-facing interface or API, while others may consider it as such if they release a paper detailing the development process. However, arguably the closest an AI model can come to being truly “open source” is when its developers release the weights – a precise representation of the nodes and neural networks used for computations. Even then, essential data such as the training dataset and process are often excluded, making it impossible to recreate the model from scratch. Some projects may go further.
All of this is before considering the fact that creating or reproducing AI models requires significant funding and engineering resources, making it predominantly exclusive to companies with substantial resources.
So where does xAI’s Grok release fall on this spectrum? As an open-weights model, anyone is free to download, adjust, improve, or distill it. This is certainly a positive development. It appears to be one of the most extensive models currently available for anyone to access in this manner, with a whopping 314 billion parameters for curious engineers to work with and test through various modifications.
However, the enormous size of the model also comes with significant drawbacks – it requires hundreds of gigabytes of high-speed RAM to use in its raw form. If you don’t already have access to a highly advanced AI inference rig with a dozen Nvidia H100s, the download link is essentially useless.
While Grok may be competitive with other modern models, it is still significantly larger, meaning it requires more resources to achieve the same results. There will always be a hierarchy of size, efficiency, and other metrics, and while it is still valuable to have such a model available, it is more of a “raw material” than a finished product. It is also unclear whether this is the latest and best version of Grok, as opposed to the finely-tuned version accessible to certain individuals through X.
Overall, it is an excellent development that this data has been released, but it is not quite the game-changing move that some had hoped for.
It’s hard not to question Musk’s motives in making this move. Is his burgeoning AI company genuinely committed to open-source development, or is this merely a way to ruffle the feathers of OpenAI, with whom Musk currently has a prominent dispute?
If xAI is genuinely dedicated to open-source development, this will only be the first in a series of releases, with future iterations taking into account feedback from the community. They should release other essential information, clarify the training data process, and further explain their approach. However, if the release is merely a ploy for Musk to use in online debates, it may still be valuable, but it is unlikely to garner much attention or reliance from the AI world after the next few months of experimentation.
[…] CEO, Jensen Huang, makes a thought-provoking statement on the importance of building foundation models for humanoid robots in the field of AI. The humanoid form has become a hot topic in the world of […]