AI Management Skills Matter More Than Chat Quality in Business Crises

When you shop for the perfect gift or choose home decor, you probably look for style, durability, and value. But behind the scenes, businesses are testing AI not just for how well it chats—it’s about how it manages real crises, makes tough decisions under pressure, and stays honest when stakes are high. The question isn’t whether AI can craft a convincing message; it’s whether it can finish what it starts and keep integrity in chaos.

Introducing the Live Business Wargame: A New Benchmark for AI Management

Imagine running a small software company through its worst week—an onslaught of customer crises, tempting manipulations, and urgent decisions. Now, what if you could do this with different AI models, each acting as the company’s management team? That’s exactly what the Firmulate experiment has achieved, providing a transparent, real-time arena where AI performance is measured against real business pressures.

The Experiment in Detail

Four frontier AI models, including the reputable GPT-5.6-sol, Kimi K3, Sonnet 5, and Opus 4.8, took on the same challenge: run a live, simulated software company with real money mechanics, 680+ self-learned rules, and a public progress dashboard. The goal was simple but formidable—navigate the worst week of the company’s existence, handle customer crises, resist manipulative tactics, and close a big deal.

What the Models Achieved

• All four models identified every crisis and refused every manipulation attempt, showcasing honesty and awareness.
• Only two models, GPT-5.6-sol and Kimi K3, succeeded in closing the €55,000 deal based on their own analysis—demonstrating management effectiveness.
• Interestingly, the decisive advantage was in reading the company’s own files—models that looked beyond customer emails and notices secured the full deal at a full price (+€4,583 monthly recurring revenue).
• Social engineering attacks, such as fake CEO messages and reporter tricks, were thwarted by all models, with Kimi K3 citing suspicion and impersonation concerns in its reasoning.

Beyond Chat: Management Under Pressure

This experiment underscores a critical insight: evaluating AI solely based on chat quality—its ability to generate convincing dialogue—misses the bigger picture. Real business management involves staying honest under pressure, reading and understanding complex documents, and making decisions that withstand manipulation and ethical scrutiny. In the live setup, the AI’s capacity to handle real crises, resist deception, and close full-price deals is what truly matters.

The Model Profiles and Lessons

Among the participants, Opus 4.8 was the most thorough—analyzing over 80 rules and providing deep insights—but ultimately placed last by leaving a deal on the table and slipping into departmental silos instead of escalating issues properly. This highlights that thoroughness doesn’t always translate to effectiveness if discipline and strategic focus falter under pressure.

Why This Matters for Business Leaders

For companies considering deploying AI in customer support, CRM, or decision-making, the key takeaway is clear: can your AI manage real-world crises, read your internal files, stay honest, and close deals? These skills are far more vital than how well it can chat or generate plausible-sounding responses. The difference between an AI that merely sounds good and one that actually gets things done could be the cost of lost revenue or a damaged reputation.

Watch the Live Performance at Firmulate

Curious how these models perform in real time? You can see the ongoing live company at firmulate.com/live, where every decision, every crisis, and every move is publicly available and auditable. The platform is designed not just to test AI chat but to measure management quality in a complex, unpredictable environment—an essential benchmark for future AI applications in business.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI