PoC validado com conta ChatGPT Plus e client_id do Hermes. Device flow
OAuth funciona, gera access_token + refresh_token auto-refresh. Chat e
function calling funcionaram em gpt-5.4, gpt-5.4-mini, gpt-5.2 e
gpt-5.3-codex.
Descobertas pro adapter final:
- Endpoint: /responses (não /chat/completions)
- Streaming obrigatório (stream: true)
- store: false obrigatório
- Sem temperature/top_p (modelos reasoning)
- input[] no lugar de messages[]
- instructions top-level no lugar de system role
- Tools sem wrapping function: {}
- Output via events response.output_item.done (não response.completed)
Pasta scripts/captain_codex_poc/ está excluída do Rubocop (scripts
standalone, não rodam em contexto Rails).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reorganized db/seed_prompts/ into three clear bins:
_prod_snapshot/ — 16 prompts pulled from iachat_production
(4 Jasmines + 12 scenarios). Read-only baseline.
_staging_current/ — 6 prompts active in iachat-v2 right now
(Jasmine + 5 scenarios, including
outras_unidades and Reclamacoes_Ouvidoria
which were created on this branch).
target/ — empty for now. Source of truth: the seed
migration only writes from here. Files we
review and approve land here, then deploy
pushes them to prod.
Updated the seed migration to walk target/ and to support both
generic scenarios (apply to every unit) and unit-scoped scenarios
(file prefixed with assistant slug, only that unit). Empty files
are skipped — useful for staged rollouts.
This guarantees no prompt ships to prod by accident: only what
ends up in target/ is applied.
The orchestrator prompt (Jasmine) and scenario instruction (Daniela)
live in the database. When we merge this branch to main and deploy to
production, the prod DB will keep its OLD prompts — the new ones would
only exist in staging. That defeats the point of merging.
Fix: commit the current staging prompts as .md files under
db/seed_prompts/ and add a data migration that syncs them into the DB
on deploy. Idempotent (no-ops when content already matches).
From now on, prompt changes follow the same workflow as code: edit the
.md file, migration resyncs on deploy. The DB row becomes a mirror of
the file, not the source of truth.
- RetentionSummaryBadge in the "Previous conversations" sidebar:
tiered status (First contact / Active / Recurring / Sleeping /
At risk / Inactive) + counts of interactions, one-shots, Pix.
- Retention tab in Captain Reports: KpiCards, FlowCard, CohortMatrix
(12x13 heatmap with CSV export).
- Five new filters on the contacts list: recurring, last interaction,
days since, interactions count, reservations paid.
- Full pt_BR + en i18n under CAPTAIN_REPORTS.RETENTION.*
- Spec for InteractionCalculatorService covering gap behavior,
one-shot classification, internal-label exclusion, multi-conversation
grouping across the 30h window.
- Docs: docs/captain-retention-indicators.md with business rules,
column reference, endpoint shape, and backup SQL queries.
Exposes two JSON endpoints under /api/v1/accounts/:id/captain/reports:
- GET /retention — aggregate KPIs (active/recurring/sleeping/at-risk/
churned, new vs returned in period, Pix generated/paid/conversion,
retention rates at 30d and 90d)
- GET /retention/cohort — monthly cohort matrix, 12 months lookback,
12 months of offset. Each cell is % of the cohort that interacted in
month M+N. SQL-aggregated with DATE_TRUNC + DISTINCT so it is a
single query even on large histories.
Lays the data + job foundation for tracking customer interactions,
recurrence, and Pix conversion on Contact. Design decisions negotiated
with Rodrigo (see docs to come):
Rules:
- Gap of 30h from last message defines separate interactions
- Qualified interaction = >=2 customer msgs + >=2 attendant msgs,
both with textual content (>= 2 letters)
- One-shot consultation = >=1+1 but below the qualified threshold
(tracked as secondary KPI)
- Excludes contacts labeled `equipe_interna`
- is_recurring = interactions_count >= 2
- pix_generated_count counts all PixCharges; reservations_paid_count
only counts those with status = paid
Surface area:
- Migration adds denormalized stats to contacts + indexes for fast filtering
- Captain::ContactStats::InteractionCalculatorService computes the stats
for a single contact (pure, no persistence)
- Captain::Retention::RecalculateContactStatsJob persists them for one
contact (idempotent)
- Captain::Retention::RecalculateAllContactStatsJob runs daily at 3am BRT,
enqueues per-contact jobs for everyone active in the last 120 days
- Event-driven refresh: CaptainListener#conversation_resolved enqueues
recalc; Captain::PixCharge after_create/after_update enqueues recalc
on status change
No UI yet — that's the next layer.
Previous commit made the extractor reject any reservation-shaped fact
without a literal payment confirmation in the conversation. That
killed the useful middle ground: a customer who requests a Pix and
hasn't paid yet is still a concrete signal worth remembering (for
follow-up, interest mapping, CRM). We were going from "hallucinated
reservation" to "nothing remembered".
Add the intermediate pattern:
- Payment confirmed → "Reservou X para Y em DD/MM/AAAA"
- Pix generated, no payment yet → "Solicitou Pix para X em DD/MM/AAAA, aguardando pagamento"
- Just a price quote → nothing
The "aguardando pagamento" suffix is required so the downstream recall
never confuses it with a closed reservation.
Three UX bugs from staging testing:
1. Duplicate Pix link in WhatsApp — the tool's formatted_message
embedded the full link + instructions, so the LLM copied it into
its own response on top of the dedicated link message sent by
dispatch_direct_link_message. The tool now returns a short
summary with no URL; dispatch is the single source of the link.
2. "Reserva confirmada!" sent before payment — the scenario prompt
used the word "confirmação" loosely, which the LLM read as the
reservation being closed. Now the prompt forces "pré-reserva /
aguardando pagamento" until the Pix is actually paid, and the
dispatched link message explains that the reservation is only
secured after payment clears.
3. Memory extraction wrote "Reservou Hidromassagem para pernoite
em 22/04/2026" when the customer only received a Pix link and
replied "obrigado". Tightened the extraction prompt so
padrao_comportamental of a reservation requires a literal
payment confirmation — Pix generated alone no longer qualifies.
When Inter integration fails ("Unit not configured for Pix", missing
certs, etc.), the tool was returning success=true with the error
message as formatted_message. The LLM interpreted that as success and
hallucinated "Pix generated" to the customer — and never triggered the
generate_reservation_link fallback.
Switch the rescue path from tool_feedback_response (success=true) to
error_response (success=false) so the Daniela scenario correctly falls
back to the reservation-link tool as documented in her prompt.
Root cause of the staging test failure:
- Tool asked for CPF then name separately, two back-and-forth turns.
- When the user replied with just "Rodrigo Borba Machado" (no "nome:"
prefix), NAME_WITH_LABEL_REGEX didn't match, so the contact.name
stayed as the emoji "😅‼️". The tool kept returning missing_name and
the LLM eventually hallucinated success without another generate_pix
call.
Changes:
- missing_identity_response combines nome + CPF into one prompt when
both are missing.
- extract_name_from_qa_pattern finds the last outgoing message asking
for "nome completo" and takes the next incoming message as the name
candidate.
- extract_name_run_from_text pulls the leading alphabetic run from the
message so "Rodrigo Borba Machado, 00251938131" parses the name
correctly alongside the CPF.
Consolida o trabalho desta branch de abril/2026 em um bloco pronto pra
testar em staging antes do merge pra main.
## Correções de memória semântica
- ExtractionService: Princípio Zero + Regra de Ouro (ação consumada vs intenção).
- Cenário Daniela_Reservas: Passo 0 de classificação (consulta/intenção/fora).
## Roleta da Sorte (end-to-end)
- Schema Supabase + 7 RPCs atômicas (server-side, idempotentes).
- Services: Offer, Redeem, WeeklyReport.
- Jobs: OfferRouletteJob (hook em ConfirmationService após Pix pago),
NotifyRevealed + Scheduler de fallback.
- Tool manual GenerateRoletaLinkTool + endpoint público /roleta/notify.
- Dashboard /captain/roleta com Resgate + Relatório + anomaly detection.
## Cenário Reclamacoes_Ouvidoria
- Triagem P1-P4, framework LAST, Three-level listening, Self-check.
- Sem compensação material, detecção de cliente frustrado eleva prioridade.
## Analytics
- Funil de conversão /captain/funnel: 5 etapas via regex, zero LLM.
- Detector de churn via ChurnOutreach* (cron dias úteis 10h-17h BRT).
## Trabalho pré-existente incluído
- Captain Executive Reports (ceo_digest, mattermost_delivery).
- get_reserva_preco_tool, Lifecycle ajustes, Reservations UI polimentos.
## Outros
- .gitignore: patterns pra credenciais.
- Migrations de scenarios idempotentes.
- i18n completa pt_BR+en pra roleta/funnel.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Observed incident 2026-04-19 14:34: ResponseBuilderJob sat 156s
'Performing' in Sidekiq without ever emitting [Captain V2] Agent result,
while the client waited on WhatsApp. The runner.run() call never
returned — presumably an HTTP hang on the LLM side (OpenAI slow,
network flake, or retry storm inside ruby-llm).
Post-hoc protections (tool_loop_detected, max_turns) can't fire because
they only inspect result after run() returns. Adding a 45s hard timeout
on the run() block guarantees we bail out, trigger bot_handoff, and
respond to the client instead of hanging forever.
Rescue Timeout::Error separately so the log message is specific and
the user-facing message says "demorou mais do que o esperado".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Problema observado em teste real 2026-04-19 11:24:
usuário forneceu suíte+data+hora pra Daniela. Em vez de chamar
generate_pix, Daniela chamou handoff_to_jasmine. Jasmine respondeu
"Vou te transferir pra Daniela..." — mentira, a conversa ficou
parada com a Jasmine.
Sequência dentro de UM único run:
jasmine.handoff_to_daniela_reservas_agent
-> daniela.handoff_to_jasmine (!)
-> jasmine responde "vou te transferir..."
O prompt da Daniela tem "🚨 NUNCA FAÇA HANDOFF DE VOLTA PRA JASMINE"
mas o LLM ignora a proibição quando a ferramenta está registrada.
A única solução robusta é não registrar a ferramenta.
Historicamente tivemos medo de remover a back-edge porque sem ela
a Daniela (quando confusa) ficava em loop chamando faq_lookup —
incidente que queimou créditos reais. Esse medo não vale mais:
commit f3f8a8d5c adicionou TOOL_LOOP_THRESHOLD=3 +
MAX_TURNS_PER_MESSAGE=15 que disparam bot_handoff automático em
qualquer loop de tool. A proteção contra runaway existe por
OUTRA via agora, então podemos remover a back-edge com segurança.
Efeito esperado:
- scenario termina a resposta sozinho (sem ping-pong)
- scenario confuso/em loop -> rate limit corta -> humano recebe
Memory: atualizado feedback_never_touch_captain_without_safety_caps.md
refletindo a nova invariante.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Três camadas de proteção contra runaway token burn no AgentRunnerService:
1. MAX_TURNS_PER_MESSAGE = 15
Cap dentro de uma única chamada run(). Já estava aplicado;
agora extraído como constante nomeada.
2. MAX_TURNS_PER_CONVERSATION = 30
Cap ao longo da vida da conversa. Contador em
conversation.custom_attributes['captain_turn_count']. Ao atingir,
dispara bot_handoff automático e responde com mensagem de
transferência pra humano.
3. TOOL_LOOP_THRESHOLD = 3
Detecta a mesma (tool_name, args) invocada 3+ vezes no resultado
de um único run (sintoma do loop faq_lookup que queimou tokens
em 2026-04-19). Ao detectar: dispara bot_handoff e aborta o turno.
trigger_bot_handoff! aciona conversation.bot_handoff! quando
disponível, removendo a conversa do pipeline automático.
Motivação: dois incidentes reais de queima de crédito OpenAI em
2026-04-19. Ver memory/feedback_never_touch_captain_without_safety_caps.md
pras invariantes completas.
Tests atualizados: mock_result agora stuba :messages (usado pelo
novo tool_loop_detected?) e max_turns esperado é 15.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Runaway incident: Daniela (reservation scenario) entered a tool-calling
loop, invoking faq_lookup with the same query dozens of times per
second, stuck at 'Performing' in Sidekiq for minutes with 1-of-12 busy.
Root cause was two interacting factors:
1. The previous commit removed scenario_agent.register_handoffs(
assistant_agent) to prevent ping-pong. In practice, the scenario LLM
uses handoff_to_orchestrator as a safety valve when it cannot
advance. Without it, the LLM kept calling other available tools
(faq_lookup) indefinitely.
2. max_turns was 100. A runaway loop could burn 100 LLM + tool cycles
before Sidekiq's timeout fired, which meant real token spend in a
single bad turn could blow a day's budget.
Both restored/fixed:
- max_turns: 100 -> 15. Plenty for normal flows; hard ceiling on any
runaway. The LLM simply ran out of turns and had to emit a final
response instead of looping further.
- scenario -> orchestrator handoff: re-registered. Ping-pong risk is
contained by max_turns AND by explicit prompt rules in the scenario
instruction forbidding gratuitous handoffs (added to Daniela prompt
in earlier commit).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two behavioural regressions caught in live testing with a real customer
conversation:
1. Ping-pong scenario -> orchestrator -> scenario
build_and_wire_agents was calling scenario_agents.register_handoffs(
assistant_agent), which exposed handoff_to_jasmine as a tool INSIDE
every scenario. Daniela (reservation scenario) kept calling it mid
flow, the orchestrator resumed the turn, and customers got messages
like "Vou te encaminhar para a Daniela..." after ALREADY being with
Daniela. The back-edge is removed. When a customer legitimately
changes topic mid-scenario, pick_starting_agent on the next turn
already routes back to the orchestrator based on conversation state,
so no manual handoff from the scenario side is needed.
2. FAQ_PRICE_PATTERNS was hijacking legitimate routing responses
The previous regex matched the bare words "pernoite", "sinal",
"diaria" WITHOUT requiring a numeric price nearby. A legitimate
handoff response like "Vou transferir para a Daniela para confirmar
a Stilo para pernoite" tripped the guardrail, which then substituted
the response with raw FAQ content about rates. Narrowed to: R$
values, numbers followed by "reais", and the explicit price-noun
variants (preco/preço/valor/preços/valores/custo/custa). Incidental
mentions of stay types no longer trigger.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two orthogonal cost optimizations to the Captain agent pipeline:
1. Hierarchical model routing (optimization A)
Captain::Scenario now overrides agent_model to read a dedicated
InstallationConfig CAPTAIN_OPEN_AI_MODEL_SCENARIO, falling back to the
global CAPTAIN_OPEN_AI_MODEL used by the orchestrator (Assistant).
Rationale: the orchestrator (Jasmine) does cheap triage (is this a
reservation intent? a greeting? escalate to human?) — a smaller model
handles this well. Scenarios (Daniela — reserva) run complex flows with
tool calling, strict taxonomies, and JSON schema output — they benefit
from a stronger model.
Config in this install: CAPTAIN_OPEN_AI_MODEL=gpt-4o-mini (orchestrator)
and CAPTAIN_OPEN_AI_MODEL_SCENARIO=gpt-4o (scenarios). Estimated ~60%
cost reduction vs everything on gpt-4o, preserving quality where it
matters for the business flow.
2. Conversation-level memory cache (optimization B)
MemoryPromptInjector now persists the computed memory block on
conversation.custom_attributes[captain_cached_memory_block]. First turn
computes once (embedding + pgvector query + XML formatting); subsequent
turns reuse. The customer's profile does not change during an open
conversation, so re-running the pipeline on every turn was pure waste.
Graceful fallbacks:
- Cache write failure → per-service-instance in-memory fallback still
applies.
- Cache read failure → fresh recall runs (no regression).
- Contact mismatch → invalidates cache, fresh recall runs.
When a new conversation starts, custom_attributes is empty → fresh
recall populates the cache for that conversation's lifetime.
Estimated ~80% reduction in embedding + pgvector calls during
multi-turn conversations.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Real-world test triggered a Sidekiq worker hang on conv 67 after a
message was routed through Daniela: two ResponseBuilderJobs (msg 1318
and 1319) started, emitted typing_on, then never returned. Sidekiq
showed 2/12 workers stuck for 10+ minutes — indefinite.
Root cause likely: Agents::Runner evaluates the orchestrator
instructions lambda multiple times per turn, and our wrapped lambda
calls MemoryPromptInjector#append_memory_block each time. Inside,
RecallService invokes OpenAI embedding API (2s timeout) and pgvector.
Ruby's Timeout.timeout has documented holes on net/http syscalls — if
the embedding API stalls at the socket level, the worker hangs forever
even though the timeout "fired".
Two fixes:
1. Per-message cache in the injector instance: the same
message_text is embedded + queried once, not N times per turn.
Dramatic reduction in network calls + DB queries during a single
agent run. Every call after the first returns the cached block
instantly.
2. Absolute rescue at append_memory_block top level:
rescue StandardError => e; return base_prompt. Even if the whole
memory pipeline throws, the base system prompt passes through and
the agent keeps responding. Memory is NEVER allowed to block a
response — that was already the design intent but the lambda caller
path didn't honor it rigorously enough.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User feedback revealed a fundamental design issue: the memory model was
accumulating contradictory "Prefere X" facts because a single choice was
being treated as a permanent preference. Result: 3 different
"Prefere suite X" entries coexisting, all at 90% confidence, with
reservation patterns over time (2hrs, 4hrs, pernoite) all claiming to be
the customer's "preferred" duration.
Corrections:
1. ExtractionService prompt — preferencia now requires EXPLICIT
declaration words ("prefiro", "gosto mais de", "sempre escolho",
"adoro", "favorita"). A mere choice in one conversation is NO LONGER
extracted as preferencia — instead it goes to padrao_comportamental
WITH THE DATE in the content (e.g. "Reservou Alexa para pernoite em
23/05/2026"). This makes memory temporal and auditable instead of
imposing fake consistency.
2. Reference date is passed to the LLM prompt via the latest message
timestamp, used as the anchor date the LLM must embed in every
padrao_comportamental content.
3. ContradictionCheckerService — dual threshold:
- cosine < 0.15 → auto-supersede without LLM (pure duplicate)
- 0.15 to 0.6 → ask LLM if contradicts, supersede if yes
- > 0.6 → ignore, unrelated facts
Previously only the middle band existed, so near-duplicate facts like
two "aniversário 23/05" entries or three "prefere suite X" entries
were never cleaned up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Real test revealed gpt-4o-mini was still:
- Hallucinating suite names ("Aluba" doesn't exist — we only have
Alexa, Stilo, Hidromassagem)
- Extracting cadastral data as memory ("Rodrigo has a CPF", "Name is X")
despite the per-type NÃO examples
Added two sections at the top of the prompt:
1. Business canonical data — explicit whitelist of suite names (Alexa,
Stilo, Hidromassagem) and stay types. Anything else = discard, NO auto-
normalization. LLM must not guess.
2. Cadastral data absolute rule — explicit list of fields that are
profile data, not memory: name, CPF/RG/passport, email/phone/address,
birth date. Plus 5 concrete ❌ examples of what was being wrongly
extracted in the wild.
Existing 9 specs still pass (stub at call_llm; prompt change is
semantic, not structural).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Real-world test revealed the LLM extractor (gpt-4o-mini) was using type
labels too loosely: a customer's QUESTION about parking ("tem
estacionamento?") was classified as 'reclamacao'. Similarly cortesia
generica ("obrigado") was becoming 'feedback_positivo', and transactional
events (CPF informed, reservation made) were becoming memories when they
should be ignored.
Rewrote build_prompt with:
- Per-type strict definition (what it IS)
- YES/NO examples for each of the 9 types, with the most common pitfalls
explicitly shown as NO
- 7 absolute rules, including: questions are never complaints, generic
courtesy is never feedback, agent actions are never customer memory,
transactional events are not long-term facts
- Confidence threshold guidance (>=0.9 only if totally explicit, 0.7-0.89
for strong inference, <0.7 discard)
- "If in doubt, discard — quality > quantity. Most transactional
conversations should return empty facts list"
Existing 9 specs still pass (stub call_llm, so prompt changes don't
affect unit test assertions).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Real-world observation: OpenAI embedding API takes 200-400ms typical,
plus pgvector query overhead, the 500ms budget was being exceeded
frequently, silently dropping memory recall. Agent typing delay is
already 2-15s humanized, so a 2s recall budget is well within UX
tolerance and gives ~4-5x margin over typical embedding latency.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Documents the Rails console procedure to toggle
captain_contact_memory_extraction_enabled and
captain_contact_memory_recall_enabled on Account#custom_attributes,
including rollout phasing (extraction-first, then recall), rollback,
bulk enablement, and post-activation verification queries.
The UI toggles in Captain Settings are deferred: the existing
FeatureToggle component is coupled to the captain_features hash and
cannot be reused for custom_attributes-backed flags without a new
component and a new account-update store action. Scope and
implementation notes for that follow-up are included at the end of the
document.
Task 5.4 of Captain Semantic Memory epic (Phase 5).
