## Notion document
https://www.notion.so/chatwoot/Email-IMAP-Issue-30aa5f274c928062aa6bddc2e5877a63?showMoveTo=true&saveParent=true
## Description
PLAIN IMAP channels (non-OAuth) were silently retrying failed
authentication every minute, forever. When credentials are
wrong/expired, Net::IMAP::NoResponseError was caught and logged but
channel.authorization_error! was never called — so the Redis error
counter never incremented, reauthorization_required? was never set, and
admins were never notified. OAuth channels already had this handled
correctly via the Reauthorizable concern.
Additionally, Net::IMAP::ResponseParseError (raised by non-RFC-compliant
IMAP servers) was falling through to the StandardError catch-all,
flooding
Estimated impact before fix: ~70–75 broken IMAP inboxes generating
~700k–750k wasted Sidekiq jobs/week.
## Type of change
- [ ] Bug fix (non-breaking change which fixes an issue)
## Checklist:
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented on my code, particularly in hard-to-understand
areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged and published in downstream
modules
When an IMAP server returns malformed or partial protocol responses,
Inboxes::FetchImapEmailsJob can raise Net::IMAP::ResponseParseError,
Net::IMAP::ResponseReadError, or Net::IMAP::ResponseTooLargeError.
Previously these errors fell through to the generic StandardError
handler and were captured repeatedly in Sentry.
This PR updates app/jobs/inboxes/fetch_imap_emails_job.rb to treat those
parser/read exceptions as handled IMAP failures by adding them to the
existing IMAP/network rescue block, so they are logged and retried on
the next scheduled run without exception tracking noise.
fixes:
https://chatwoot-p3.sentry.io/issues/7132037793/events/104fb9b4d80a4fb6ba3861c44c6c9b83/
How to reproduce:
- Use an IMAP server/inbox that intermittently returns malformed or
truncated protocol responses during search/fetch.
- Run Inboxes::FetchImapEmailsJob for that channel and observe the
raised parser/read exception.
How this was tested:
- bundle exec ruby -c app/jobs/inboxes/fetch_imap_emails_job.rb
- bundle exec rubocop app/jobs/inboxes/fetch_imap_emails_job.rb
Currently, auto-assignment runs only during conversation creation or
update events. If no agents are online when new conversations arrive,
those conversations remain unassigned.
With this change, unassigned conversations will be automatically
assigned once agents become available. The job runs every 15 minutes and
uses a fair distribution threshold of 100 to prevent a large number of
conversations from being assigned to a single available agent. This will
be customizable later.
This PR adds the following changes
1. Add `Imap::GoogleFetchEmailService` and
`Google::RefreshOauthTokenService`. The
`Google::RefreshOauthTokenService` uses
`OmniAuth::Strategies::GoogleOauth2` which is already added as a packge
2. Update `Inboxes::FetchImapEmailsJob` to handle Google inboxes
3. Add SMTP settings for Google in `ConversationReplyMailerHelper` to
allow sending emails
## Preview
#### Incoming emails
#### Outgoing email
---------
Co-authored-by: Muhsin Keloth <muhsinkeramam@gmail.com>
Fixes the follow cases
- The ensure block released the lock even on LockAcquisitionError
- Custom timeout was not allowed
This also refactored the with_lock method, now the key has to be constructed in the parent function itself
Co-authored-by: Sojan Jose <sojan@pepalo.com>
This allows a user to add/update a custom regex and a cue while defining custom attributes(Only applicable for type- text).
While adding/editing custom attributes, the values are validated against the attribute definition regex, and if it is incorrect, a cue message or default error message is shown and restricts invalid values from being saved.
Fixes: #6866
The Inboxes::FetchImapEmailsJob is designed to fetch the entire email object and check if its message id is already in the database. However, this process is resource-intensive and time-consuming, as fetching the full email object takes a significant amount of time.
On average, fetching 100 emails can take approximately 3-4 minutes to complete depending on the IMAP server. Since we are not using server flags to identify which emails have already been fetched (to avoid compatibility issues with flags in different email services), we have to fetch all previously available emails. Currently we fetch all the messages that were created from yesterday. This becomes problematic with services like Gmail, which throttle requests based on bandwidth usage.
To address this issue, I have updated the logic as follows:
Fetch the sequence IDs of all the mails in the "Inbox" that were created from yesterday.
Use the FETCH command to fetch only the message-ids using BODY.PEEK[HEADER.FIELDS (MESSAGE-ID)] with the sequence IDs we got in the previous step. This is a faster operation with lower bandwidth usage, as it only returns the sequence ID and message ID.
Check if the message IDs are already present in the database for the selected inbox.
If not present, fetch the entire email object using the FETCH command and create the message.
If the message ID is already present, ignore it and move on to the next message-id.
I have also addressed the issue of duplicate emails appearing in conversations when two fetch email jobs occur simultaneously. I have added a lock for the channel to ensure that the job gracefully exits without waiting for the current job to complete.
Fixes#7247Fixes#6082Fixes#8314
Co-authored-by: Sojan <sojan@pepalo.com>
* fix: mail logger for source-id email-id and message-id
* added mail provided to the tracking
* change the logger verb from info to error
* app/jobs/inboxes/fetch_imap_emails_job.rb
* codeclimate fix
* remove extra filter
* fixes: specs
- Adds the backend APIs required for Microsoft Email Channels
Co-authored-by: Pranav Raj S <pranav@chatwoot.com>
Co-authored-by: Sojan <sojan@pepalo.com>
This change allows the user to configure both IMAP and SMTP for an email inbox. IMAP enables the user to see emails in Chatwoot. And user can use SMTP to reply to an email conversation.
Users can use the default settings to send and receive emails for email inboxes if both IMAP and SMTP are disabled.
Fixes#2520
