## Description
Refactored the `Message#search_data` method to prevent exceeding
OpenSearch's 1000 field limit during reindex operations.
**Problem:** The previous implementation serialized entire ActiveRecord
objects (Inbox, Sender, Conversation) with all their JSONB fields,
causing dynamic field explosion in OpenSearch. This resulted in
`Searchkick::ImportError` with "Limit of total fields [1000] has been
exceeded".
**Solution:** Whitelisted only necessary fields for search and
filtering, and flattened JSONB `custom_attributes` into key-value pair
arrays to prevent unbounded field creation.
Linked to: CW-5861
## Type of change
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality not to work as expected)
- [x] This change requires a documentation update
## How Has This Been Tested?
- Verified rubocop passes with no offenses
- Code review of search field usage from
`enterprise/app/services/enterprise/search_service.rb`
- Analyzed actual search queries to determine required indexed fields
**Still needed:**
- Full reindex test on staging/production environment
- Verify search functionality still works after reindex
- Confirm field count is under 1000 limit
## Changes Made
### Before
- Indexed 1000+ fields (entire AR objects with JSONB)
- `inbox` = full Inbox object (23+ fields + JSONB)
- `sender` = full Contact/User/AgentBot object (10+ fields + JSONB)
- `conversation` = full push_event_data
- Dynamic JSONB keys creating unlimited fields
### After
- ~35-40 controlled fields
- Whitelisted search fields: `content`, `attachment_transcribed_text`,
`email_subject`
- Filter fields: `account_id`, `inbox_id`, `conversation_id`,
`sender_id`, `sender_type`, etc.
- Flattened `custom_attributes`: `[{key, value, value_type}]` format
- Helper methods: `search_conversation_data`, `search_inbox_data`,
`search_sender_data`, `search_additional_data`
## Checklist:
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my code
- [x] I have commented on my code, particularly in hard-to-understand
areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged and published in downstream
modules
## Post-merge Steps
After merging, the following steps are required:
1. **Reindex all messages:**
```bash
bundle exec rails runner "Message.reindex"
```
2. **Verify field count:**
```bash
bundle exec rails runner "
client = Searchkick.client
index_name = Message.searchkick_index.name
mapping = client.indices.get_mapping(index: index_name)
fields = mapping.dig(index_name, 'mappings', 'properties')
puts 'Total fields: ' + fields.keys.count.to_s
"
```
3. **Test search functionality** to ensure queries still work as
expected
---------
Co-authored-by: Vishnu Narayanan <iamwishnu@gmail.com>
Co-authored-by: Pranav <pranav@chatwoot.com>
Previously, email replies were handled inside workers. There was no
execution logs. This meant if emails silently failed (as reported by a
customer), we had no way to trace where the issue happened, the only
assumption was “no error = mail sent.”
By moving email handling into jobs, we now have proper execution logs
for each attempt. This makes it easier to debug delivery issues and
would have better visibility when investigating customer reports.
Fixes
https://linear.app/chatwoot/issue/CW-5538/emails-are-not-sentdelivered-to-the-contact
---------
Co-authored-by: Sojan Jose <sojan@pepalo.com>
Co-authored-by: Shivam Mishra <scm.mymail@gmail.com>
Changes searchkick callback behavior to check `should_index?` before
creating reindex jobs, preventing unnecessary job creation for messages
that don't need indexing (activity messages, unpaid accounts, etc.).
Previously, `callbacks: :async` created reindex jobs for all messages
(~5,100/min or 7.3M/day in production), which were then filtered by
`should_index?` inside the job worker - resulting in 98% wasted jobs,
Redis memory pressure, and avoidable p0 alerts.
Now, `should_index?` is checked before job creation via `after_commit`
callback, reducing job creation to actual incoming/outgoing messages
from paid accounts.
Changes:
- Disable automatic searchkick callbacks
- Add manual `after_commit` callback with `should_index?` condition
- Add specs to verify callback behavior
Expected impact:
- 98% reduction in sidekiq job creation (~7.3M → ~150K jobs/day)
- Reduced redis memory usage
- Same async indexing behavior for eligible messages
With this change, the indexing would be separate from the search, so you
need to enable indexing on the cloud and run it. It should start
indexing the messages to ElasticSearch/OpenSearch. Once indexing is
completed, we can turn on the feature for the customer.
Make sure that the following is done when you deploy.
Set POSTGRES_STATEMENT_TIMEOUT=600s before you run the indexing.
1. Make sure that the account with advanced_search has
advanced_search_indexing enabled
```rb
Account.feature_advanced_search.each do |account|
account.enable_features(:advanced_search_indexing)
account.save!
end
```
2. Enable indexing for all accounts with paid subscription.
```rb
Account.where("custom_attributes ->> 'plan_name' IN (?)", ['Enterprise', 'Startups', 'Business']).each do |account|
account.enable_features(:advanced_search_indexing)
account.save!
end
```
3. Run indexing for all the messages.
```rb
Message.reindex
```
Co-authored-by: Vishnu Narayanan <iamwishnu@gmail.com>
Co-authored-by: Muhsin Keloth <muhsinkeramam@gmail.com>
This PR delivers the first slice of the voice channel: inbound call
handling. When a customer calls a configured voice
number, Chatwoot now creates a new conversation and shows a dedicated
call bubble in the UI. As the call progresses
(ringing, answered, completed), its status updates in real time in both
the conversation list and the call bubble, so
agents can instantly see what’s happening. This focuses on the inbound
flow and is part of breaking the larger voice
feature into smaller, functional, and testable units; further
enhancements will follow in subsequent PRs.
references: #11602 , #11481
## Testing
- Configure a Voice inbox in Chatwoot with your Twilio number.
- Place a call to that number.
- Verify a new conversation appears in the Voice inbox for the call.
- Open it and confirm a dedicated voice call message bubble is shown.
- Watch status update live (ringing/answered); hang up and see it change
to completed in both the bubble and conversation
list.
- to test missed call status, make sure to hangup the call before the
please wait while we connect you to an agent message plays
## Screens
<img width="400" alt="Screenshot 2025-09-03 at 3 11 25 PM"
src="https://github.com/user-attachments/assets/d6a1d2ff-2ded-47b7-9144-a9d898beb380"
/>
<img width="700" alt="Screenshot 2025-09-03 at 3 11 33 PM"
src="https://github.com/user-attachments/assets/c25e6a1e-a885-47f7-b3d7-c3e15eef18c7"
/>
<img width="700" alt="Screenshot 2025-09-03 at 3 11 57 PM"
src="https://github.com/user-attachments/assets/29e7366d-b1d4-4add-a062-4646d2bff435"
/>
<img width="442" height="255" alt="Screenshot 2025-09-04 at 11 55 01 PM"
src="https://github.com/user-attachments/assets/703126f6-a448-49d9-9c02-daf3092cc7f9"
/>
---------
Co-authored-by: Muhsin <muhsinkeramam@gmail.com>
We now support searching within the actual message content, email
subject lines, and audio transcriptions. This enables a faster, more
accurate search experience going forward. Unlike the standard message
search, which is limited to the last 3 months, this search has no time
restrictions.
The search engine also accounts for small variations in queries. Minor
spelling mistakes, such as searching for slck instead of Slack, will
still return the correct results. It also ignores differences in accents
and diacritics, so searching for Deja vu will match content containing
Déjà vu.
We can also refine searches in the future by criteria such as:
- Searching within a specific inbox
- Filtering by sender or recipient
- Limiting to messages sent by an agent
Fixes https://github.com/chatwoot/chatwoot/issues/11656
Fixes https://github.com/chatwoot/chatwoot/issues/10669
Fixes https://github.com/chatwoot/chatwoot/issues/5910
---
Rake tasks to reindex all the messages.
```sh
bundle exec rake search:all
```
Rake task to reindex messages from one account only
```sh
bundle exec rake search:account ACCOUNT_ID=1
```
The term "sorcerer’s apprentice mode" is defined as a bug in a protocol
where, under some circumstances, the receipt of a message causes
multiple messages to be sent, each of which, when received, triggers the
same bug. - RFC3834
Reference: https://github.com/chatwoot/chatwoot/pull/9606
This PR:
- Adds an auto_reply attribute to message.
- Adds an auto_reply attribute to conversation.
- Disable conversation_created / conversation_opened event if auto_reply
is set.
- Disable message_created event if auto_reply is set.
---------
Co-authored-by: Muhsin Keloth <muhsinkeramam@gmail.com>
This PR adds a composite index (:account_id, :content_type, :created_at)
on the table messages.
This index is added as a temporary fix for performance issues in the
CSAT responses controller where we query messages with account_id,
content_type and created_at. The current implementation
(account.message.input_csat.count) times out with millions of messages.
TODO: Create a dedicated csat_survey table and add entries when surveys
are sent, then query this table instead of the entire messages table for
better performance.
We made so many improvements for CSAT via https://github.com/chatwoot/chatwoot/pull/11485. However, we missed showing message content in the dashboard for CSAT URLs created in non-widget inboxes. This PR fixes the issue by ensuring that CSAT-configured messages are passed along with CSAT responses, otherwise defaulting to the translation.
Show captain messages under the name of the assistant which generated
the message.
- Add support for `Captain::Assistant` sender type
- Add push_event_data for captain_assistants
- Add activity message handler for captain_assistants
- Update UI to show captain messages under the name of the assistant
- Fix the issue where openAI errors when image is sent
- Add support for custom name of the assistant
---------
Co-authored-by: Muhsin Keloth <muhsinkeramam@gmail.com>
Co-authored-by: Sivin Varghese <64252451+iamsivin@users.noreply.github.com>
* fix: handle protocol messages in IncomingMessageBaileysService and ignore it
* fix: ignore protocol messages before create conversation and contact
* feat: add flag to edited messages and content history tracking
* refactor: update message model to use edit_history instead of content_history for tracking edits
* refactor: update message model to track previous content instead of edit history
* test: fix message content update test to correctly track original content
---------
Co-authored-by: gabrieljablonski <contact@gabrieljablonski.com>
* chore: lint files
* chore: suppress warning
* chore: disable suggest extensions
* chore: do not stage changes in pre-commit
* chore: remove git add from FE lint and `-a` flag from rubocop on husky
The `before_type_cast` method sometimes returns a string for
`message_type`, creating inconsistencies in different payloads. This
pull request will remove all `before_type_cast` usage and replace it
with `to_i` methods.
- Adding a new index on (account_id,created_at,message_type) based on
our performance improvement exercise. This index significantly improves
the page load speeds of messaging reports.
This PR addresses several items listed in our rubocop_todo by implementing the necessary corrections and enhancements. As a result, we are now able to remove the rubocop_todo file entirely, streamlining our codebase and ensuring adherence to our coding standards.
fixes: https://linear.app/chatwoot/issue/CW-1806/chore-rubocop-audit
We observed an issue in production where the external webhook for an API inbox was failing. This, in turn, calls message update to update message status to failed. This causes a loop because rails trigger after_update callbacks even for empty commits.
Ref: rails/rails#44500
This PR logs additional information in content_attributes of a message in case it is unsupported. This info can be used by the client to render a fresh UI
Updating the `unattended` tab to include conversations where the customer responded and is awaiting an agent's response.
Previously it showed only the conversations where the first response was pending.
Co-authored-by: Pranav Raj S <pranav@chatwoot.com>
- This renames the unread_since scope to created_since, which more accurately describes what the scope returns.
- The EXTRACT(EPOCH FROM created_at) > (?) clause was also simplified and rewritten as created_at > ?, which is equivalent:
* logging the messages id for message validation exception
* Update the processed_message_content validation over length
* codeclimate
* specs failing for contacts
