811eb66615
# Pull Request Template ## Linear links: - https://linear.app/chatwoot/issue/CW-4479/if-image-is-sent-by-the-customer-send-it-to-openai ## Description This pull request adds “Captain image support” to Chatwoot. It introduces multimodal message handling so that when a customer sends an image, Captain can forward the file to OpenAI’s vision endpoint, generate a caption/analysis ## Type of change Please delete options that are not relevant. - [x] New feature (non-breaking change which adds functionality) ## How Has This Been Tested? <img width="891" alt="image" src="https://github.com/user-attachments/assets/c7cc98ed-cc44-4865-a53a-83d129e2fe2c" /> ## Checklist: - [ ] My code follows the style guidelines of this project - [ ] I have performed a self-review of my code - [ ] I have commented on my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] Any dependent changes have been merged and published in downstream modules --------- Co-authored-by: Pranav <pranav@chatwoot.com>
59 lines
1.8 KiB
Ruby
59 lines
1.8 KiB
Ruby
class Captain::OpenAiMessageBuilderService
|
|
pattr_initialize [:message!]
|
|
|
|
def generate_content
|
|
parts = []
|
|
parts << text_part(@message.content) if @message.content.present?
|
|
parts.concat(attachment_parts(@message.attachments)) if @message.attachments.any?
|
|
|
|
return 'Message without content' if parts.blank?
|
|
return parts.first[:text] if parts.one? && parts.first[:type] == 'text'
|
|
|
|
parts
|
|
end
|
|
|
|
private
|
|
|
|
def text_part(text)
|
|
{ type: 'text', text: text }
|
|
end
|
|
|
|
def image_part(image_url)
|
|
{ type: 'image_url', image_url: { url: image_url } }
|
|
end
|
|
|
|
def attachment_parts(attachments)
|
|
image_attachments = attachments.where(file_type: :image)
|
|
image_content = image_parts(image_attachments)
|
|
|
|
transcription = extract_audio_transcriptions(attachments)
|
|
transcription_part = text_part(transcription) if transcription.present?
|
|
|
|
attachment_part = text_part('User has shared an attachment') if attachments.where.not(file_type: %i[image audio]).exists?
|
|
|
|
[image_content, transcription_part, attachment_part].flatten.compact
|
|
end
|
|
|
|
def image_parts(image_attachments)
|
|
image_attachments.each_with_object([]) do |attachment, parts|
|
|
url = get_attachment_url(attachment)
|
|
parts << image_part(url) if url.present?
|
|
end
|
|
end
|
|
|
|
def get_attachment_url(attachment)
|
|
return attachment.external_url if attachment.external_url.present?
|
|
|
|
attachment.file.attached? ? attachment.file_url : nil
|
|
end
|
|
|
|
def extract_audio_transcriptions(attachments)
|
|
audio_attachments = attachments.where(file_type: :audio)
|
|
return '' if audio_attachments.blank?
|
|
|
|
audio_attachments.map do |attachment|
|
|
result = Messages::AudioTranscriptionService.new(attachment).perform
|
|
result[:success] ? result[:transcriptions] : ''
|
|
end.join
|
|
end
|
|
end |