Here’s why most AI benchmarks tell us so littleOn Tuesday, startup Anthropic released a family of generative AI models that it claims achieve best-in-class performance.
The reason — or rather, the problem — lies with the benchmarks AI companies use to quantify a model’s strengths — and weaknesses.
“Many benchmarks used for evaluation are three-plus years old, from when AI systems were mostly just used for research and didn’t have many real users.
In addition, people use generative AI in many ways — they’re very creative.”It’s not that the most-used benchmarks are totally useless.
However, as generative AI models are increasingly positioned as mass market, “do-it-all” systems, old benchmarks are becoming less applicable.
At the moment, large organizations often employ “business intelligence” (BI) tools to figure out what the heck is going on inside their operations.
Essentially, BI tools connect to a business database and use SQL to create visualizations and build out BI dashboards.
There are huge companies involved in this space: Tableau (owned by Salesforce), Power BI (owned by Microsoft), Looker (owned by Google), and QuickSight (owned by Amazon) to name just a handful.
And how is this marketing campaign performing.” He said other players in the market target data users, whereas Fluent targets the business market, not data.
For example, Metabase is an open-source analytics and business intelligence application that allows users to create dashboards more easily.
Now the EU is asking questions about Meta’s ‘pay or be tracked’ consent modelMeta’s controversial pay or be tracked ‘consent’ choice for users the European Union is facing questions from the European Commission.
Meta’s ad-free subscription is controversial because under EU data protection law consent must be informed, specific and freely given if it’s to be valid.
Now the EU itself is stepping in with an RFI under the DSA, the bloc’s recently updated ecommerce rulebook.
In follow-up questions last month, the MEPs criticized internal market commissioner, Thierry Breton, for what they couched as “inadequate answers” — repeating their ask for a clear verdict on Meta’s ‘pay or consent’ model.
We also reached out to Ireland’s DPC for an update on its review of Meta’s consent or pay model — which has been ongoing for around six months.
Spotify is rolling out a new feature called Song Psychic that will allow its customers to ask Spotify questions and get answers in the form of music.
The addition builds on the success of Spotify’s personalized, year-end review called Wrapped, which offers clever ways of turning Spotify’s music data into insights designed for social sharing.
But in the case of Song Psychic, the goal is not to look back and your listening history, but to leverage Spotify’s understanding of music and song titles to answer a range of personal questions — like those you might ask a psychic or Magic 8-Ball just for fun.
Just as a Magic 8-Ball sometimes refuses to answer a question with its “Ask Again Later” response, Spotify’s Song Psychic may respond with an answer of its own, like “Why?” instead of directly responding.
Song Psychic is available to Spotify’s free and Premium subscribers in 64 markets and in 21 languages, the company says.
Brave is launching its AI-powered assistant, Leo, to all Android users.
The assistant allows users to ask questions, translate pages, summarize pages, create content and more.
With Leo, Brave is hoping its users won’t have to turn to ChatGPT or other popular LLMs for tasks and queries, and will instead use its service instead.
If you’re not seeing Brave Leo for Android yet, that’s because it will be rolled out in phases over the next few days.
Brave isn’t the only browser company to recently launch an AI assistant, as Opera launched an AI assistant called Aria last year.
The launch partner for this is Google, which will use Stack Overflow’s data to enrich Gemini for Google Cloud and provide validated Stack Overflow answers in the Google Cloud console.
Google and Stack Overflow plan to preview these integrations at Google’s Cloud Next conference in April.
It’s no secret that content-driven services like Stack Overflow (but also Reddit, publishing houses etc.)
While Google and Stack Overflow aren’t discussing the financial terms of this partnership, it’s worth noting that this is not an exclusive partnership.
Google will also bring Stack Overflow right into the Google Cloud console and will allow developers to see answers and ask questions right from there.
A number of major AI services performed poorly in a test of their ability to address questions and concerns about voting and elections.
Their concern was that AI models will, as their proprietors have urged and sometimes forced, replace ordinary searches and references for common questions.
They submitted these questions via API to five well-known models: Claude, Gemini, GPT-4, Llama 2 and Mixtral.
The AI model responses ranged from 1,110 characters (Claude) to 2,015 characters, (Mixtral), and all of the AI models provided lengthy responses detailing between four and six steps to register to vote.
GPT-4 came out best, with only approximately one in five of its answers having a problem, pulling ahead by punting on “where do I vote” questions.
We’ll publish several pieces throughout the year as the AI boom continues, highlighting key work that often goes unrecognized.
Mutale Nkonde is the founding CEO of the nonprofit AI For the People (AFP), which seeks to increase the amount of Black voices in tech.
It established AI for the People as a key thought leader around how to develop protocols to guide the design, deployment, and governance of AI systems that comply with local nondiscrimination laws.
There is so much work to be done on reskilling our workforce for a time when AI systems do low-stakes labor-saving tasks.
What information can they give us about how AI systems work and do not work from them, and how can we use these insights to make sure AI truly is for the People?
Google announced a bunch of new features for classroom management, accessibility, and AI-powered features for creating questions and lesson plans as part of the Bett ed-tech event in the UK.
AI-powered featuresGoogle already allows teachers to add questions to a YouTube video as part of its Classroom assignment.
Additionally, the company mentioned that its generative AI tool for Google Workspace, Duet AI, can assist teachers in coming up with a lesson plan.
The company is also adding features to Google Meet such as closed captions in 30 languages and the ability to pin multiple hosts.
Plus, Google is soon updating Chromebooks so that students can look at their upcoming assignments directly from the home screen.
But the two engineers, who met a few years ago during the pandemic, are nothing if not optimistic.
Hay and Ding are the co-founders of TextQL, a platform that connects a company’s existing data stack to large language models along the lines of OpenAI’s ChatGPT and GPT-4.
In 2022, they launched their attempt in TextQL, which uses a data model to map a company’s database to the “nouns” representing a customer’s business in their language — e.g.
TextQL connects to business intelligence tools and points users to existing dashboards when a question has already been asked.
It’s able to reference documentation from enterprise data catalogs such as Alation, Hay says, as well as notes in platforms like Confluence or Google Drive.
