The Inboxes::FetchImapEmailsJob is designed to fetch the entire email object and check if its message id is already in the database. However, this process is resource-intensive and time-consuming, as fetching the full email object takes a significant amount of time.
On average, fetching 100 emails can take approximately 3-4 minutes to complete depending on the IMAP server. Since we are not using server flags to identify which emails have already been fetched (to avoid compatibility issues with flags in different email services), we have to fetch all previously available emails. Currently we fetch all the messages that were created from yesterday. This becomes problematic with services like Gmail, which throttle requests based on bandwidth usage.
To address this issue, I have updated the logic as follows:
Fetch the sequence IDs of all the mails in the "Inbox" that were created from yesterday.
Use the FETCH command to fetch only the message-ids using BODY.PEEK[HEADER.FIELDS (MESSAGE-ID)] with the sequence IDs we got in the previous step. This is a faster operation with lower bandwidth usage, as it only returns the sequence ID and message ID.
Check if the message IDs are already present in the database for the selected inbox.
If not present, fetch the entire email object using the FETCH command and create the message.
If the message ID is already present, ignore it and move on to the next message-id.
I have also addressed the issue of duplicate emails appearing in conversations when two fetch email jobs occur simultaneously. I have added a lock for the channel to ensure that the job gracefully exits without waiting for the current job to complete.
Fixes#7247Fixes#6082Fixes#8314
Co-authored-by: Sojan <sojan@pepalo.com>
fdbb3bf4b1