"My favourite thing about UserTrace is how realistic the simulations feel. It makes the whole testing process way more efficient."

Wisnu Waskitho Putra Senior Data Analyst at RedDoorz

Company

About RedDoorz is Southeast Asia's largest and fastest-growing technology-driven hospitality brand, offering affordable, standardized hotel stays across Indonesia, Singapore, the Philippines, and Vietnam.
Industry Hospitality
Company size 501-1,000

Context and challenge

RIA is RedDoorz's hotel booking chatbot. It lets customers book hotel rooms straight from a WhatsApp chat, and it's a key channel for acquiring direct customers.

As the usage of RIA scaled up, running it became expensive for the team.

"As we scaled, we ran into another challenge. Cost," said Nittin, who works in product analytics at RedDoorz. "Our model worked well but was expensive. So we wanted to move to a cheaper model without compromising on quality."

RedDoorz team wanted to explore open-source alternative for RIA. They had tried this once before for another chatbot, however, that whole migration eventually had failed in production because of latency issues.

"We had tried switching once before, but it failed in production due to latency issues," Nittin said. "So this time we wanted to do our homework and avoid the same problem."

Every few weeks a new model claims to be faster, cheaper, or smarter. Be it open source or closed one.

Model benchmarks only tell you a part of that claim. They don't tell you how that model will perform for your product and users.

Testing had been mostly manual

RedDoorz's evaluation up to this point was manual UAT, QA, and conversion funnel data.

"We look at user funnel using regex to detect each step," said Wisnu, senior data analyst at RedDoorz. "It worked, but it was hard for us to get a clear picture of the full bot's performance."

Nittin, who works in product analytics at RedDoorz, shared the problem from a different angle: "We weren't really evaluating what matters for an AI bot. For example, tone, correctness of information, consistency, or the hallucinations."

Evaluating between models just based on sample prompts will not give the whole picture. And the team already knew what happens when you don't test it properly: it breaks under real usage.

Two things made this particular model swap hard to test:

Tool calling. RIA checks availability, applies pricing, and confirms bookings. Any replacement model has to keep every one of those calls working.

"Since our bot is WhatsApp integrated with heavy tool calling, testing new models ourselves was hard," Nittin said.

Stress Testing. Sample prompts don't reveal what breaks under real traffic, which is exactly where the previous attempt had failed.

"The challenge was stress testing," Wisnu said. "We want to know how the bot performs under a heavy load. If we had to do it manually, it will be very hard for us."

What UserTrace did

The first step of model evaluation was focused on deciding what to test, especially what is relevant for RedDoorz.

UserTrace worked with the RedDoorz team to define the scenarios and evaluation criteria that matter for a booking agent, including, latency, context awareness, booking flow correctness, and information accuracy.

"That's where UserTrace made a real difference," Nittin said. "They helped us understand exactly what to test and how to test it properly."

Post this, UserTrace ran the candidate models against the test scenarios, also on real production conversations: replaying actual customer chat logs through the platform to see how each model performed across the usecases.

Here, the tool calls were mocked to allow the complexity of the scenarios and conversations is enabled, be it user booking history, pricing or even availability flows which is how it would happen in the live bot.

The evaluation revealed that another model version delivered better latency and overall performance for RedDoorz's production use case. It wasn't just the one model RedDoorz had in mind initially. As Nittin noted, UserTrace "evaluated performance for our specific use case, not just for the model we suggested, but a couple of others as well, which was incredibly useful." And, Wisnu mentioned, "complete with the mock tool scope, so we can make a decision based on real data."

The team received detailed interactive reports to understand not only that which model did better but what aspects and which conversation what happened. what could be done better.

Outcome

The evaluation surfaced a candidate that delivered better latency and stronger overall performance for RedDoorz's production use case than the model the team had originally shortlisted, and it was the smallest model tested.
It also came out ahead of the model RIA was already running, at every percentile measured: a median of 1,256 ms against 1,600 ms, about 21% quicker.

The tail mattered more. At P99 the gap widened to roughly 35%, which is exactly the condition that had broken the team's earlier migration: not the average response, but the slow ones under load.

Two larger models were also evaluated. Both were substantially slower on RedDoorz's tool-calling workload, one of them by more than four times at the median. Parameter count didn't predict production behaviour, which is exactly why the evaluation was run on real conversations rather than benchmarks.

All of these insights came before a single engineering sprint was committed.

"The initial results look promising and we are now preparing to release it to production," Nittin said.