Runaway incident: Daniela (reservation scenario) entered a tool-calling loop, invoking faq_lookup with the same query dozens of times per second, stuck at 'Performing' in Sidekiq for minutes with 1-of-12 busy. Root cause was two interacting factors: 1. The previous commit removed scenario_agent.register_handoffs( assistant_agent) to prevent ping-pong. In practice, the scenario LLM uses handoff_to_orchestrator as a safety valve when it cannot advance. Without it, the LLM kept calling other available tools (faq_lookup) indefinitely. 2. max_turns was 100. A runaway loop could burn 100 LLM + tool cycles before Sidekiq's timeout fired, which meant real token spend in a single bad turn could blow a day's budget. Both restored/fixed: - max_turns: 100 -> 15. Plenty for normal flows; hard ceiling on any runaway. The LLM simply ran out of turns and had to emit a final response instead of looping further. - scenario -> orchestrator handoff: re-registered. Ping-pong risk is contained by max_turns AND by explicit prompt rules in the scenario instruction forbidding gratuitous handoffs (added to Daniela prompt in earlier commit). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| assistant | ||
| contact_memories | ||
| copilot | ||
| inter | ||
| lifecycle | ||
| llm | ||
| notifications | ||
| onboarding | ||
| payments | ||
| reservations | ||
| tools | ||
| open_ai_message_builder_service.rb | ||
| tool_registry_service.rb | ||