diff --git a/META-WEBHOOK-PROXY.md b/META-WEBHOOK-PROXY.md deleted file mode 100644 index 9062d14d5..000000000 --- a/META-WEBHOOK-PROXY.md +++ /dev/null @@ -1,229 +0,0 @@ -# Meta Webhook Proxy - -## Problem - -Some VPS providers silently drop inbound TCP connections from Meta's webhook servers (AS32934) due to overzealous DDoS protection. This causes 15–20% WhatsApp message loss. A reverse proxy on a clean provider (e.g., DigitalOcean) eliminates the drops completely. - -``` -Meta (WhatsApp) → proxy.example.com (clean provider) → your Chatwoot instance -``` - -## Architecture - -The proxy is a single nginx server that routes requests based on the first path segment: - -``` -https://proxy.example.com//webhooks/whatsapp/%2B -``` - -This is the URL you configure in Meta's App Dashboard as the webhook callback URL. The proxy extracts ``, checks it against an allowlist, and forwards the request to `https:///webhooks/whatsapp/%2B`. - -### Multi-tenant - -One proxy serves multiple Chatwoot instances. Each upstream is identified by its domain in the URL path — no separate config per tenant beyond adding the host to the allowlist. - -## Setup - -### Prerequisites - -- A server (Ubuntu 22.04/24.04) on a provider with clean Meta connectivity (DigitalOcean, AWS, etc.) -- A DNS A record pointing your proxy domain to the server IP (e.g., `proxy.example.com → 1.2.3.4`) -- SSH root access to the server - -### 1. Install nginx and certbot - -```bash -ssh root@proxy.example.com - -apt-get update -apt-get install -y nginx certbot python3-certbot-nginx -``` - -### 2. Create a temporary HTTP-only config - -Certbot needs nginx running to perform the ACME challenge, but the full config references SSL certs that don't exist yet. Start with an HTTP-only config: - -```bash -cat > /etc/nginx/sites-available/cw-proxy << 'EOF' -server { - listen 80; - server_name proxy.example.com; - - location /.well-known/acme-challenge/ { - root /var/www/html; - } - - location / { - return 404; - } -} -EOF -``` - -Enable the site and reload: - -```bash -rm -f /etc/nginx/sites-enabled/default -ln -sf /etc/nginx/sites-available/cw-proxy /etc/nginx/sites-enabled/cw-proxy -nginx -t && systemctl reload nginx -``` - -### 3. Obtain the SSL certificate - -```bash -certbot certonly --webroot -w /var/www/html -d proxy.example.com \ - --non-interactive --agree-tos -m your-email@example.com -``` - -Certbot installs a systemd timer that auto-renews the certificate before it expires. - -### 4. Deploy the full proxy config - -Replace the temporary config with the full proxy configuration: - -```bash -cat > /etc/nginx/sites-available/cw-proxy << 'EOF' -# Allowlist of upstream Chatwoot hosts -# Add new hosts here to enable proxying -map $upstream_host $upstream_allowed { - default 0; - chatwoot.example.com 1; - # chatwoot.other.com 1; -} - -server { - listen 80; - server_name proxy.example.com; - - location /.well-known/acme-challenge/ { - root /var/www/html; - } - - location / { - return 301 https://$host$request_uri; - } -} - -server { - listen 443 ssl; - server_name proxy.example.com; - - resolver 1.1.1.1 8.8.8.8 valid=300s; - resolver_timeout 5s; - - ssl_certificate /etc/letsencrypt/live/proxy.example.com/fullchain.pem; - ssl_certificate_key /etc/letsencrypt/live/proxy.example.com/privkey.pem; - - # Extract upstream host from first path segment, proxy the rest - location ~ ^/([^/]+)(/.*)$ { - set $upstream_host $1; - set $upstream_path $2; - - # Reject hosts not in the allowlist - if ($upstream_allowed = 0) { - return 403; - } - - proxy_pass https://$upstream_host$upstream_path$is_args$args; - proxy_set_header Host $upstream_host; - proxy_set_header X-Forwarded-For $remote_addr; - proxy_set_header X-Real-IP $remote_addr; - proxy_ssl_server_name on; - } - - location / { - return 404; - } -} -EOF -``` - -Test and reload: - -```bash -nginx -t && systemctl reload nginx -``` - -### 5. Verify - -```bash -# Root path → 404 -curl -s -o /dev/null -w "%{http_code}" https://proxy.example.com/ -# Expected: 404 - -# Unknown host → 403 -curl -s -o /dev/null -w "%{http_code}" https://proxy.example.com/unknown.host/webhooks/whatsapp/test -# Expected: 403 - -# Allowed host → proxied (502 if upstream is unreachable, 200 if live) -curl -s -o /dev/null -w "%{http_code}" https://proxy.example.com/chatwoot.example.com/webhooks/whatsapp/test -# Expected: 502 or 200 -``` - -### 6. Configure Meta webhook URL - -In the Meta App Dashboard, set the webhook callback URL to: - -``` -https://proxy.example.com//webhooks/whatsapp/%2B -``` - -For example, if your Chatwoot is at `chatwoot.example.com` and the phone number is `+5511999999999`: - -``` -https://proxy.example.com/chatwoot.example.com/webhooks/whatsapp/%2B5511999999999 -``` - -Meta will send both verification (GET) and delivery (POST) requests to this URL. The proxy passes them through transparently. - -## Adding a new upstream - -1. SSH into the proxy server -2. Edit `/etc/nginx/sites-available/cw-proxy` -3. Add the new host to the `map` block: - ```nginx - map $upstream_host $upstream_allowed { - default 0; - chatwoot.example.com 1; - chatwoot.newclient.com 1; # ← add this line - } - ``` -4. Test and reload: - ```bash - nginx -t && systemctl reload nginx - ``` -5. Set the Meta webhook callback URL for the new instance to: - ``` - https://proxy.example.com/chatwoot.newclient.com/webhooks/whatsapp/%2B - ``` - -## Removing an upstream - -1. Remove or comment out the host from the `map` block -2. `nginx -t && systemctl reload nginx` -3. Update the Meta webhook callback URL to point directly at the Chatwoot instance (or to a different proxy) - -## Key nginx directives - -| Directive | Purpose | -|-----------|---------| -| `map $upstream_host $upstream_allowed` | Allowlist of permitted upstream hosts. Only hosts set to `1` are proxied; all others get 403. | -| `proxy_ssl_server_name on` | Enables SNI so the TLS handshake uses the correct hostname for the upstream's certificate. | -| `resolver 1.1.1.1 8.8.8.8 valid=300s` | Required because `proxy_pass` uses a variable (`$upstream_host`), so nginx cannot resolve DNS at config load time. Uses Cloudflare and Google DNS. | -| `proxy_set_header Host $upstream_host` | Sets the Host header to the upstream domain so reverse proxies (Traefik, etc.) route correctly. | - -## Failure modes - -All recoverable — Meta retries with exponential backoff for up to 36 hours: - -| Failure | What happens | Recovery | -|---------|-------------|----------| -| Proxy down | Connection refused | Meta retries | -| Upstream down | 502 Bad Gateway | Meta retries | -| SSL expired | TLS handshake error | Meta retries | - -## Important notes - -- **Do not rate-limit.** Meta sends webhook deliveries from many IPs in AS32934. Bursts of 10+ requests per second are normal. -- **SSL auto-renewal** is handled by the certbot systemd timer. Verify with `systemctl status certbot.timer`. -- The `%2B` in the URL is the URL-encoded `+` sign for the phone number's country code. diff --git a/docs/BAILEYS_VPN_SETUP.md b/docs/BAILEYS_VPN_SETUP.md deleted file mode 100644 index f9f160c87..000000000 --- a/docs/BAILEYS_VPN_SETUP.md +++ /dev/null @@ -1,170 +0,0 @@ -# VPN Setup for Baileys API (Gluetun + Mullvad) - -## Problem - -Hosting providers like Hostinger have their IP ranges mass-blocked by Meta. This causes WhatsApp connections through Baileys to fail. The solution is to route **only** Baileys traffic through a VPN, keeping everything else (Rails, Sidekiq, Postgres, Redis) on the regular network. - -## Architecture - -``` -┌─────────────────────────────────────────────────────┐ -│ Docker Network (coolify) │ -│ │ -│ ┌──────────┐ ┌──────────────────────────────┐ │ -│ │ Rails │ │ Gluetun (VPN tunnel) │ │ -│ │ Sidekiq │───▶│ :3025 ──▶ Baileys API │ │ -│ └──────────┘ │ (network_mode: service) │ │ -│ │ │ │ │ │ -│ │ │ │ VPN (WireGuard) │ │ -│ │ │ ▼ │ │ -│ │ │ Mullvad SP (Brazil) │ │ -│ │ └──────────────────────────────┘ │ -│ │ │ │ -│ ▼ ▼ │ -│ ┌──────────┐ ┌──────────┐ │ -│ │ Redis │◀───────│ Baileys │ │ -│ │ Postgres│ │ (via VPN)│ │ -│ └──────────┘ └──────────┘ │ -└─────────────────────────────────────────────────────┘ -``` - -**Gluetun** creates a WireGuard VPN tunnel. Baileys shares Gluetun's network via `network_mode: "service:gluetun"`, so all WhatsApp traffic exits through the VPN IP. Internal Docker traffic (Redis, etc.) is exempted via firewall subnet rules. - ---- - -## Step 1 — Get Mullvad Credentials - -1. Go to and generate an account (no email needed — save the 16-digit account number). -2. Add credit (€5/month, accepts card, PayPal, crypto). -3. Go to , log in, select **Linux**, click **Generate key**, choose **Brazil > São Paulo**, and download the `.conf` file. -4. From the downloaded file, note: - - `PrivateKey` — e.g. `KLaIt4oAaI6Iz4iQhSS9/0UBlbvfmG1LC/NWGXW/DH4=` - - `Address` — use **only the IPv4** address, e.g. `10.67.152.85/32` (discard the IPv6 address) - -> **Important:** Gluetun does not support IPv6. Only use the IPv4 address from the `.conf` file. - ---- - -## Step 2 — Docker Compose Changes - -### 2.1 Add the `gluetun` service (baileys-api compose) - -```yaml -gluetun: - image: qmcgaw/gluetun - restart: always - cap_add: - - NET_ADMIN - ports: - - '3025:3025' - environment: - - VPN_SERVICE_PROVIDER=mullvad - - VPN_TYPE=wireguard - - WIREGUARD_PRIVATE_KEY= - - WIREGUARD_ADDRESSES=/32 - - SERVER_COUNTRIES=Brazil - - SERVER_CITIES=Sao Paulo - - FIREWALL_OUTBOUND_SUBNETS=172.16.0.0/12,10.0.0.0/8,192.168.0.0/16 - - DNS_KEEP_NAMESERVER=on - healthcheck: - test: - - CMD-SHELL - - 'wget -qO- https://ipinfo.io/ip' - interval: 30s - timeout: 10s - retries: 5 - networks: - - coolify -``` - -Key environment variables: - -| Variable | Purpose | -|---|---| -| `FIREWALL_OUTBOUND_SUBNETS` | Allows internal Docker traffic (Redis, etc.) to bypass the VPN. Must cover all private subnets used by Docker. | -| `DNS_KEEP_NAMESERVER` | Keeps Docker's internal DNS so containers can resolve hostnames like `redis`. Without this, you get `getaddrinfo ENOTFOUND` errors. | - -> **Do NOT set `OWNED_ONLY=yes`** — Mullvad does not have owned servers in São Paulo, only rented ones. This filter would match zero servers. - -### 2.2 Modify the `baileys-api` service - -Apply these changes to the existing baileys-api service: - -```yaml -baileys-api: - # ... existing config ... - network_mode: 'service:gluetun' # Route all traffic through Gluetun - depends_on: - gluetun: - condition: service_healthy # Wait for VPN to be up - # REMOVE any 'ports' section — port 3025 is now exposed by gluetun - # REMOVE any 'networks' section — network_mode is incompatible with networks -``` - -> **`condition: service_healthy`** is critical. Without it, baileys-api starts before the VPN tunnel is established, causing Redis connection timeouts. - -### 2.3 Update `BAILEYS_PROVIDER_DEFAULT_URL` in Chatwoot - -In the Rails and Sidekiq services (or Coolify environment variables), change: - -``` -# Before -BAILEYS_PROVIDER_DEFAULT_URL=http://baileys-api:3025 - -# After -BAILEYS_PROVIDER_DEFAULT_URL=http://gluetun:3025 -``` - -Since baileys-api now shares Gluetun's network, external services must address it via `gluetun` hostname. - -### 2.4 Declare the shared network - -If the baileys-api compose is a separate stack, declare the shared network: - -```yaml -networks: - coolify: - external: true - name: coolify -``` - ---- - -## Step 3 — Verify - -After deploying, run from the server's SSH terminal: - -```bash -# Check container health -docker ps | grep -E "gluetun|baileys" - -# Get the VPN exit IP (replace with your actual gluetun container name) -docker exec wget -qO- https://ipinfo.io/ip - -# Compare with the server's real IP -curl -s https://ipinfo.io/ip -``` - -If the two IPs are **different**, the VPN is working correctly. - ---- - -## Troubleshooting - -| Symptom | Cause | Fix | -|---|---|---| -| `Redis client error` / connection timeout on Redis | Baileys starts before VPN is ready | Add `depends_on` with `condition: service_healthy` | -| `Redis client error` / connection refused | Internal Docker traffic blocked by VPN firewall | Add `192.168.0.0/16` to `FIREWALL_OUTBOUND_SUBNETS` | -| `getaddrinfo ENOTFOUND redis` | Docker DNS not working inside VPN | Set `DNS_KEEP_NAMESERVER=on` in gluetun | -| `no server found: ... city sao paulo; owned servers only` | Mullvad has no owned servers in São Paulo | Remove `OWNED_ONLY=yes` from gluetun env | -| `interface address is IPv6 but IPv6 is not supported` | IPv6 address in `WIREGUARD_ADDRESSES` | Use only the IPv4 address (remove the `fc00:...` part) | -| `REDIS_URL` with hardcoded IP (e.g. `172.19.0.2`) | Docker internal IPs change on restart | Always use hostnames (e.g. `redis://redis:6379`) | - ---- - -## VPN Expiration - -If the Mullvad subscription expires: -- **WhatsApp stops working** (Baileys can't connect through the VPN). -- **Everything else keeps running** normally (Rails, Sidekiq, Redis, Postgres are not affected). -- To restore: renew Mullvad, or revert the docker-compose changes to bypass the VPN entirely.