Some customers using WhatsApp inboxes with account-level webhooks were
reporting receiving duplicate `message_created` webhook deliveries for
every incoming message. Upon inspection, here's what we found
- Both payloads are identical.
- No errors appear in the application logs
- Webhook URL is only configured in one place.
This meant, the system was sending the webhooks twice. For some context,
there's a know related issue... Meta's WhatsApp Business API can deliver
the same webhook notification multiple times for a single message. The
codebase already acknowledges this — there's a comment in
`IncomingMessageBaseService#process_messages` noting that "multiple
webhook events can be received against the same message due to
misconfigurations in the Meta business manager account." A deduplication
guard exists, but it doesn't actually work under concurrency.
### Rationale
The existing dedup was a three-step sequence: check Redis (`GET`), check
the database, then set a Redis flag (`SETEX`). Two Sidekiq workers
processing duplicate Meta webhooks simultaneously would both complete
the `GET` before either executed the `SETEX`, so both would proceed to
create a message. The `source_id` column has a non-unique index, so the
database wouldn't catch the duplicate either. Each message then
independently fires `after_create_commit`, dispatching two
`message_created` webhook events to the customer.
```
Worker A Worker B
│ │
▼ ▼
Redis GET key ──► nil Redis GET key ──► nil
│ │
│ ◄── both pass guard ──► │
│ │
▼ ▼
Redis SETEX key Redis SETEX key
│ │
▼ ▼
BEGIN transaction BEGIN transaction
INSERT message INSERT message
DELETE Redis key ◄─┐ │
COMMIT │ DELETE Redis key
│ COMMIT
│ │
└── key gone before ───┘
B's commit lands
▼ ▼
after_create_commit after_create_commit
dispatch MESSAGE_CREATED dispatch MESSAGE_CREATED
│ │
▼ ▼
WebhookJob ──► n8n WebhookJob ──► n8n
(duplicate!)
```
There was a second, subtler problem visible in the diagram: the Redis
key was cleared *inside* the database transaction, before the
transaction committed. This opened a window where neither the Redis
check nor the database check would see the in-flight message.
The fix collapses the check-and-set into a single `SET NX EX` call,
which is atomic in Redis. The key is no longer eagerly cleared — it
expires naturally after 24 hours. The database lookup
(`find_message_by_source_id`) remains as a fallback for messages that
were created before the lock expired.
```
Worker A Worker B
│ │
▼ ▼
Redis SET NX ──► OK Redis SET NX ──► nil
│ │
▼ ▼
proceeds to create returns early
message normally (lock already held)
```
### Implementation Notes
The lock logic is extracted into `Whatsapp::MessageDedupLock`, a small
class that wraps a single `Redis SET NX EX` call. This makes the
concurrency guarantee testable in isolation — the spec uses a
`CyclicBarrier` to race two threads against the same key and asserts
exactly one wins, without needing database writes,
`use_transactional_tests = false`, or monkey-patching.
Because the Redis lock now persists (instead of being cleared
mid-transaction), existing WhatsApp specs needed an `after` hook to
clean up `MESSAGE_SOURCE_KEY::*` keys between examples. Transactional
fixtures only roll back the database, not Redis.
Currently, we do not support reactions, ephemeral messages, or the request_welcome event for the WhatsApp channel. However, if this is the first event we receive in Chatwoot (i.e., there is no previous conversation or contact in Chatwoot), it will create a contact and a conversation without any messages. This confuses our customer, as it may appear that Chatwoot has missed some messages. There are multiple cases where this might be the first event we receive in Chatwoot. One quick example is when the user has sent an outbound campaign from another tool and their customers reacted to the message.
Another event like this is request_welcome event. WhatsApp has a concept for welcome messages. You can send an outbound message even though the user has not send a message. You can receive notifications through a webhook whenever a WhatsApp user initiates a chat with you for the first time. (Read the Welcome message section: https://developers.facebook.com/docs/whatsapp/cloud-api/phone-numbers/conversational-components/ ). Although this can help the business send a pro-active message to the user, we don't have it scoped in our feature set. For now, I'm ignoring this event.
Fixes https://linear.app/chatwoot/issue/CW-3018/whatsapp-handle-request-welcome-case-properly
Fixes https://linear.app/chatwoot/issue/CW-3017/whatsapp-handle-reactions-properly
* chore: update to ruby 3.1.3
* chore: ping docker version to alpine3.16 for nodev16.x
Starting with Node 17, nodejs switched to OpenSSL3. The docker builds
are installing node18.xx with alpine-3.1.3.
From Node.js 17's announcement post:
If you hit an ERR_OSSL_EVP_UNSUPPORTED error in your application
with Node.js 17, it’s likely that your application or a module you’re
using is attempting to use an algorithm or key size which is no longer
allowed by default with OpenSSL 3.0. A new command-line option,
--openssl-legacy-provider, has been added to revert to the legacy
provider as a temporary workaround for these tightened restrictions.
Looks like a webpack issue. This is fixed in webpacl 5+ and we are on
webpack4 at the moment.
Solutions
Upgrade webpack.
Pin nodejs version to be 16.x.x
Use --openssl-legacy-provider as a workaround.
Pin docker version to alpine3.16 branch to have node16.x by default
ref:
https://github.com/chatwoot/chatwoot/pull/5555#issuecomment-1379778532
* chore: update webmock
* chore: fix ruby gem path in dockerfile
* chore: switch to node16 in circleci
* chore: update ruby version in linux installer script
* chore: update ruby version in linux installer script
* chore: fix circleci
* chore: fix circleci
* feat: upgrade node version to 16.x in linux installer
* chore: update systemd files
Co-authored-by: Sojan Jose <sojan@chatwoot.com>
Process field statuses received in webhook WhatsApp cloud API
ref: #1021
Co-authored-by: Sojan <sojan@pepalo.com>
Co-authored-by: Nithin David <1277421+nithindavid@users.noreply.github.com>
