230 lines
6.8 KiB
Markdown
230 lines
6.8 KiB
Markdown
# Meta Webhook Proxy
|
||
|
||
## Problem
|
||
|
||
Some VPS providers silently drop inbound TCP connections from Meta's webhook servers (AS32934) due to overzealous DDoS protection. This causes 15–20% WhatsApp message loss. A reverse proxy on a clean provider (e.g., DigitalOcean) eliminates the drops completely.
|
||
|
||
```
|
||
Meta (WhatsApp) → proxy.example.com (clean provider) → your Chatwoot instance
|
||
```
|
||
|
||
## Architecture
|
||
|
||
The proxy is a single nginx server that routes requests based on the first path segment:
|
||
|
||
```
|
||
https://proxy.example.com/<upstream_host>/webhooks/whatsapp/%2B<phone>
|
||
```
|
||
|
||
This is the URL you configure in Meta's App Dashboard as the webhook callback URL. The proxy extracts `<upstream_host>`, checks it against an allowlist, and forwards the request to `https://<upstream_host>/webhooks/whatsapp/%2B<phone>`.
|
||
|
||
### Multi-tenant
|
||
|
||
One proxy serves multiple Chatwoot instances. Each upstream is identified by its domain in the URL path — no separate config per tenant beyond adding the host to the allowlist.
|
||
|
||
## Setup
|
||
|
||
### Prerequisites
|
||
|
||
- A server (Ubuntu 22.04/24.04) on a provider with clean Meta connectivity (DigitalOcean, AWS, etc.)
|
||
- A DNS A record pointing your proxy domain to the server IP (e.g., `proxy.example.com → 1.2.3.4`)
|
||
- SSH root access to the server
|
||
|
||
### 1. Install nginx and certbot
|
||
|
||
```bash
|
||
ssh root@proxy.example.com
|
||
|
||
apt-get update
|
||
apt-get install -y nginx certbot python3-certbot-nginx
|
||
```
|
||
|
||
### 2. Create a temporary HTTP-only config
|
||
|
||
Certbot needs nginx running to perform the ACME challenge, but the full config references SSL certs that don't exist yet. Start with an HTTP-only config:
|
||
|
||
```bash
|
||
cat > /etc/nginx/sites-available/cw-proxy << 'EOF'
|
||
server {
|
||
listen 80;
|
||
server_name proxy.example.com;
|
||
|
||
location /.well-known/acme-challenge/ {
|
||
root /var/www/html;
|
||
}
|
||
|
||
location / {
|
||
return 404;
|
||
}
|
||
}
|
||
EOF
|
||
```
|
||
|
||
Enable the site and reload:
|
||
|
||
```bash
|
||
rm -f /etc/nginx/sites-enabled/default
|
||
ln -sf /etc/nginx/sites-available/cw-proxy /etc/nginx/sites-enabled/cw-proxy
|
||
nginx -t && systemctl reload nginx
|
||
```
|
||
|
||
### 3. Obtain the SSL certificate
|
||
|
||
```bash
|
||
certbot certonly --webroot -w /var/www/html -d proxy.example.com \
|
||
--non-interactive --agree-tos -m your-email@example.com
|
||
```
|
||
|
||
Certbot installs a systemd timer that auto-renews the certificate before it expires.
|
||
|
||
### 4. Deploy the full proxy config
|
||
|
||
Replace the temporary config with the full proxy configuration:
|
||
|
||
```bash
|
||
cat > /etc/nginx/sites-available/cw-proxy << 'EOF'
|
||
# Allowlist of upstream Chatwoot hosts
|
||
# Add new hosts here to enable proxying
|
||
map $upstream_host $upstream_allowed {
|
||
default 0;
|
||
chatwoot.example.com 1;
|
||
# chatwoot.other.com 1;
|
||
}
|
||
|
||
server {
|
||
listen 80;
|
||
server_name proxy.example.com;
|
||
|
||
location /.well-known/acme-challenge/ {
|
||
root /var/www/html;
|
||
}
|
||
|
||
location / {
|
||
return 301 https://$host$request_uri;
|
||
}
|
||
}
|
||
|
||
server {
|
||
listen 443 ssl;
|
||
server_name proxy.example.com;
|
||
|
||
resolver 1.1.1.1 8.8.8.8 valid=300s;
|
||
resolver_timeout 5s;
|
||
|
||
ssl_certificate /etc/letsencrypt/live/proxy.example.com/fullchain.pem;
|
||
ssl_certificate_key /etc/letsencrypt/live/proxy.example.com/privkey.pem;
|
||
|
||
# Extract upstream host from first path segment, proxy the rest
|
||
location ~ ^/([^/]+)(/.*)$ {
|
||
set $upstream_host $1;
|
||
set $upstream_path $2;
|
||
|
||
# Reject hosts not in the allowlist
|
||
if ($upstream_allowed = 0) {
|
||
return 403;
|
||
}
|
||
|
||
proxy_pass https://$upstream_host$upstream_path$is_args$args;
|
||
proxy_set_header Host $upstream_host;
|
||
proxy_set_header X-Forwarded-For $remote_addr;
|
||
proxy_set_header X-Real-IP $remote_addr;
|
||
proxy_ssl_server_name on;
|
||
}
|
||
|
||
location / {
|
||
return 404;
|
||
}
|
||
}
|
||
EOF
|
||
```
|
||
|
||
Test and reload:
|
||
|
||
```bash
|
||
nginx -t && systemctl reload nginx
|
||
```
|
||
|
||
### 5. Verify
|
||
|
||
```bash
|
||
# Root path → 404
|
||
curl -s -o /dev/null -w "%{http_code}" https://proxy.example.com/
|
||
# Expected: 404
|
||
|
||
# Unknown host → 403
|
||
curl -s -o /dev/null -w "%{http_code}" https://proxy.example.com/unknown.host/webhooks/whatsapp/test
|
||
# Expected: 403
|
||
|
||
# Allowed host → proxied (502 if upstream is unreachable, 200 if live)
|
||
curl -s -o /dev/null -w "%{http_code}" https://proxy.example.com/chatwoot.example.com/webhooks/whatsapp/test
|
||
# Expected: 502 or 200
|
||
```
|
||
|
||
### 6. Configure Meta webhook URL
|
||
|
||
In the Meta App Dashboard, set the webhook callback URL to:
|
||
|
||
```
|
||
https://proxy.example.com/<your-chatwoot-domain>/webhooks/whatsapp/%2B<phone>
|
||
```
|
||
|
||
For example, if your Chatwoot is at `chatwoot.example.com` and the phone number is `+5511999999999`:
|
||
|
||
```
|
||
https://proxy.example.com/chatwoot.example.com/webhooks/whatsapp/%2B5511999999999
|
||
```
|
||
|
||
Meta will send both verification (GET) and delivery (POST) requests to this URL. The proxy passes them through transparently.
|
||
|
||
## Adding a new upstream
|
||
|
||
1. SSH into the proxy server
|
||
2. Edit `/etc/nginx/sites-available/cw-proxy`
|
||
3. Add the new host to the `map` block:
|
||
```nginx
|
||
map $upstream_host $upstream_allowed {
|
||
default 0;
|
||
chatwoot.example.com 1;
|
||
chatwoot.newclient.com 1; # ← add this line
|
||
}
|
||
```
|
||
4. Test and reload:
|
||
```bash
|
||
nginx -t && systemctl reload nginx
|
||
```
|
||
5. Set the Meta webhook callback URL for the new instance to:
|
||
```
|
||
https://proxy.example.com/chatwoot.newclient.com/webhooks/whatsapp/%2B<phone>
|
||
```
|
||
|
||
## Removing an upstream
|
||
|
||
1. Remove or comment out the host from the `map` block
|
||
2. `nginx -t && systemctl reload nginx`
|
||
3. Update the Meta webhook callback URL to point directly at the Chatwoot instance (or to a different proxy)
|
||
|
||
## Key nginx directives
|
||
|
||
| Directive | Purpose |
|
||
|-----------|---------|
|
||
| `map $upstream_host $upstream_allowed` | Allowlist of permitted upstream hosts. Only hosts set to `1` are proxied; all others get 403. |
|
||
| `proxy_ssl_server_name on` | Enables SNI so the TLS handshake uses the correct hostname for the upstream's certificate. |
|
||
| `resolver 1.1.1.1 8.8.8.8 valid=300s` | Required because `proxy_pass` uses a variable (`$upstream_host`), so nginx cannot resolve DNS at config load time. Uses Cloudflare and Google DNS. |
|
||
| `proxy_set_header Host $upstream_host` | Sets the Host header to the upstream domain so reverse proxies (Traefik, etc.) route correctly. |
|
||
|
||
## Failure modes
|
||
|
||
All recoverable — Meta retries with exponential backoff for up to 36 hours:
|
||
|
||
| Failure | What happens | Recovery |
|
||
|---------|-------------|----------|
|
||
| Proxy down | Connection refused | Meta retries |
|
||
| Upstream down | 502 Bad Gateway | Meta retries |
|
||
| SSL expired | TLS handshake error | Meta retries |
|
||
|
||
## Important notes
|
||
|
||
- **Do not rate-limit.** Meta sends webhook deliveries from many IPs in AS32934. Bursts of 10+ requests per second are normal.
|
||
- **SSL auto-renewal** is handled by the certbot systemd timer. Verify with `systemctl status certbot.timer`.
|
||
- The `%2B` in the URL is the URL-encoded `+` sign for the phone number's country code.
|