The AI industry is obsessed with Chatbot Arena, but it might not be the best benchmark | TechCrunch

Chatbot Arena is a crowdsourced AI benchmarking tool that lets users test AI models.

It lets anyone on the web ask a question (or questions) of two randomly-selected, anonymous models.

LMSYS is a non-profit run by students and faculty at Carnegie Mellon , UC Berkeley’s SkyLab and UC San Diego .

In March , the organization released a data set containing a million conversations between users and models.

Chatbot Arena is framed as an empirical test, but it amounts to a relative rating of models.

The voting isn’t accounting for people’s ability to spot hallucinations from models, nor differences in their preferences.

LMSYS is trying to balance out these biases by using automated systems — MT-Bench and Arena-Hard-Auto — that use models themselves to rank the quality of responses from other models.

LMSYS and Chatbot Arena provide real-time insights into how different AI models perform outside the lab.

The platform is sponsored in part by organizations, one of which is a VC firm, with horses in the AI race.

Google ’s Gemini and Mistral AI models are among those being tested on the platform.