Gorgias, a $100 million ARR AI-driven customer support platform focused on ecommerce, has launched an unprecedented public benchmark comparing its AI agent to 17 competitors using live conversational data and open-source tools. This transparent evaluation offers buyers much-needed direct insight into real-world AI performance rather than relying on traditional, often marketing-skewed vendor claims.
- Gorgias evaluated 18 AI agents with 8,356 real ecommerce support conversations.
- Benchmark methodology and code are fully open-sourced for verification.
- Public results highlight both strengths and weaknesses across competitors.
What happened
Gorgias, a prominent AI-powered ecommerce customer support vendor approaching $100 million in annual recurring revenue, released a comprehensive public benchmark comparing its AI agent to 17 other competitors. This evaluation is based on 8,356 actual live conversations and uses an open-source testing harness along with a publicly documented rubric.
Unlike typical yearly or survey-based vendor comparisons, this benchmark uses continuous, real question-answer pairs scored by a transparent framework. Gorgias openly shares both the results where it leads and where competitors perform better, providing an honest, data-driven look at AI customer support capabilities.
Why it matters
In the AI customer experience sector, buyers face a confusing landscape with vendors often exaggerating claims without direct evidence. Traditional checklists, analyst reports, or marketing charts do not capture the fluctuating and unpredictable nature of AI agent outputs. Gorgias’ approach demonstrates that transparent, direct testing with live data can better inform purchase decisions.
This level of openness also helps rebuild buyer confidence as support teams typically cannot conduct extensive side-by-side AI tests themselves. By publishing granular data and an open evaluation rubric, Gorgias helps set a precedent that could encourage more vendors to disclose verifiable performance metrics rather than just promotional statements.
What to watch next
The evolution of AI agent benchmarking is likely to accelerate as buyers increasingly rely on these direct evaluations to shortlist and select AI support tools. Vendors that adopt transparent, repeatable, open methodologies will differentiate themselves and gain credibility in a market skeptical of one-sided claims.
Gorgias’ next steps include improving accessibility by allowing automated indexing of their reports and continuing to update benchmarking data regularly. Monitoring how other ecommerce CX vendors respond—whether by publishing their own comprehensive evals or adopting parts of Gorgias’ methodology—will be critical for the future of AI support transparency.