Observed incident 2026-04-19 14:34: ResponseBuilderJob sat 156s
'Performing' in Sidekiq without ever emitting [Captain V2] Agent result,
while the client waited on WhatsApp. The runner.run() call never
returned — presumably an HTTP hang on the LLM side (OpenAI slow,
network flake, or retry storm inside ruby-llm).
Post-hoc protections (tool_loop_detected, max_turns) can't fire because
they only inspect result after run() returns. Adding a 45s hard timeout
on the run() block guarantees we bail out, trigger bot_handoff, and
respond to the client instead of hanging forever.
Rescue Timeout::Error separately so the log message is specific and
the user-facing message says "demorou mais do que o esperado".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Problema observado em teste real 2026-04-19 11:24:
usuário forneceu suíte+data+hora pra Daniela. Em vez de chamar
generate_pix, Daniela chamou handoff_to_jasmine. Jasmine respondeu
"Vou te transferir pra Daniela..." — mentira, a conversa ficou
parada com a Jasmine.
Sequência dentro de UM único run:
jasmine.handoff_to_daniela_reservas_agent
-> daniela.handoff_to_jasmine (!)
-> jasmine responde "vou te transferir..."
O prompt da Daniela tem "🚨 NUNCA FAÇA HANDOFF DE VOLTA PRA JASMINE"
mas o LLM ignora a proibição quando a ferramenta está registrada.
A única solução robusta é não registrar a ferramenta.
Historicamente tivemos medo de remover a back-edge porque sem ela
a Daniela (quando confusa) ficava em loop chamando faq_lookup —
incidente que queimou créditos reais. Esse medo não vale mais:
commit f3f8a8d5c adicionou TOOL_LOOP_THRESHOLD=3 +
MAX_TURNS_PER_MESSAGE=15 que disparam bot_handoff automático em
qualquer loop de tool. A proteção contra runaway existe por
OUTRA via agora, então podemos remover a back-edge com segurança.
Efeito esperado:
- scenario termina a resposta sozinho (sem ping-pong)
- scenario confuso/em loop -> rate limit corta -> humano recebe
Memory: atualizado feedback_never_touch_captain_without_safety_caps.md
refletindo a nova invariante.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Três camadas de proteção contra runaway token burn no AgentRunnerService:
1. MAX_TURNS_PER_MESSAGE = 15
Cap dentro de uma única chamada run(). Já estava aplicado;
agora extraído como constante nomeada.
2. MAX_TURNS_PER_CONVERSATION = 30
Cap ao longo da vida da conversa. Contador em
conversation.custom_attributes['captain_turn_count']. Ao atingir,
dispara bot_handoff automático e responde com mensagem de
transferência pra humano.
3. TOOL_LOOP_THRESHOLD = 3
Detecta a mesma (tool_name, args) invocada 3+ vezes no resultado
de um único run (sintoma do loop faq_lookup que queimou tokens
em 2026-04-19). Ao detectar: dispara bot_handoff e aborta o turno.
trigger_bot_handoff! aciona conversation.bot_handoff! quando
disponível, removendo a conversa do pipeline automático.
Motivação: dois incidentes reais de queima de crédito OpenAI em
2026-04-19. Ver memory/feedback_never_touch_captain_without_safety_caps.md
pras invariantes completas.
Tests atualizados: mock_result agora stuba :messages (usado pelo
novo tool_loop_detected?) e max_turns esperado é 15.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Runaway incident: Daniela (reservation scenario) entered a tool-calling
loop, invoking faq_lookup with the same query dozens of times per
second, stuck at 'Performing' in Sidekiq for minutes with 1-of-12 busy.
Root cause was two interacting factors:
1. The previous commit removed scenario_agent.register_handoffs(
assistant_agent) to prevent ping-pong. In practice, the scenario LLM
uses handoff_to_orchestrator as a safety valve when it cannot
advance. Without it, the LLM kept calling other available tools
(faq_lookup) indefinitely.
2. max_turns was 100. A runaway loop could burn 100 LLM + tool cycles
before Sidekiq's timeout fired, which meant real token spend in a
single bad turn could blow a day's budget.
Both restored/fixed:
- max_turns: 100 -> 15. Plenty for normal flows; hard ceiling on any
runaway. The LLM simply ran out of turns and had to emit a final
response instead of looping further.
- scenario -> orchestrator handoff: re-registered. Ping-pong risk is
contained by max_turns AND by explicit prompt rules in the scenario
instruction forbidding gratuitous handoffs (added to Daniela prompt
in earlier commit).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two behavioural regressions caught in live testing with a real customer
conversation:
1. Ping-pong scenario -> orchestrator -> scenario
build_and_wire_agents was calling scenario_agents.register_handoffs(
assistant_agent), which exposed handoff_to_jasmine as a tool INSIDE
every scenario. Daniela (reservation scenario) kept calling it mid
flow, the orchestrator resumed the turn, and customers got messages
like "Vou te encaminhar para a Daniela..." after ALREADY being with
Daniela. The back-edge is removed. When a customer legitimately
changes topic mid-scenario, pick_starting_agent on the next turn
already routes back to the orchestrator based on conversation state,
so no manual handoff from the scenario side is needed.
2. FAQ_PRICE_PATTERNS was hijacking legitimate routing responses
The previous regex matched the bare words "pernoite", "sinal",
"diaria" WITHOUT requiring a numeric price nearby. A legitimate
handoff response like "Vou transferir para a Daniela para confirmar
a Stilo para pernoite" tripped the guardrail, which then substituted
the response with raw FAQ content about rates. Narrowed to: R$
values, numbers followed by "reais", and the explicit price-noun
variants (preco/preço/valor/preços/valores/custo/custa). Incidental
mentions of stay types no longer trigger.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
# Pull Request Template
## Description
## Type of change
typo fix
## How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide
instructions so we can reproduce. Please also list any relevant details
for your test configuration.
## Checklist:
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my code
- [x] I have commented on my code, particularly in hard-to-understand
areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] Any dependent changes have been merged and published in downstream
modules
# Pull Request Template
## Description
Instruments captain v2
## Type of change
- [x] New feature (non-breaking change which adds functionality)
## How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide
instructions so we can reproduce. Please also list any relevant details
for your test configuration.
Local testing:
<img width="864" height="510" alt="image"
src="https://github.com/user-attachments/assets/855ebce5-e8b8-4d22-b0bb-0d413769a6ab"
/>
## Checklist:
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my code
- [x] I have commented on my code, particularly in hard-to-understand
areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] Any dependent changes have been merged and published in downstream
modules
---------
Co-authored-by: Shivam Mishra <scm.mymail@gmail.com>
