Fail to extract content with google groups
-
Hi. Postie is great, we use it for many different projects.
We need to connect postie to an email account that will be fed through a google group. The connection works fine, but the content is not extracted from the message, leaving only the post-title. Further on (and probably an outcome of the fist issue), comments are treated as new posts.
We thought it might be a base64 issue (google groups are sending the content encoded to base64).
Any idea for tackling this issue will be priceless.
Viewing 10 replies - 1 through 10 (of 10 total)
-
Typically this means the “Preferred Text Type” doesn’t match what the sender is sending. Try switching to HTML.
Also look at the “Treat Replies As” setting.
We’ve been trying to change these settings back and forth but it didn’t work. When the exact email is sent directly to the mail box every thing works fine…
I’m a little confused as to what is working and what is not.
Can you describe what you did, what you expected and what actually happened for each scenario?
this what I did:
- connected an email account (i.e blog@example.com) with postie,
- checked that everything is working by sending direct mail to blog@example.com, and everything worked fine.
- connected blog@example.com to a google group
- now only the title is being processed
- update: changing to plain text – makes it work but strips all the design/links from the massage
This is quite strange because the group does send rich text emails.
Can you ensure there is an email waiting then click the debug button in Postie and post the results here?
Starting mail fetch Postie Version: 1.7.32 Wordpress Version: 4.5.1 PHP Version: 7.0.2 OS: Linux ip-172-31-22-222 3.13.0-83-generic #127-Ubuntu SMP Fri Mar 11 00:25:37 UTC 2016 x86_64 POSTIE_DEBUG: On Time: 2016-04-27 19:05:10 GMT Error log: TMP dir: /tmp/ Postie is in /PATH-TO-DIR/wp-content/plugins/postie/ Cron: On Alternate Cron: Off wp_content_path: /PATH-TO-DIR/wp-content Connecting to imap.gmail.com:993 (imap-ssl) imap: using Google INBOX imap: connection string - {imap.gmail.com:993/service=imap/ssl/notls/novalidate-cert}INBOX imap: disabling GSSAPI imap: connected stdClass Object ( [flags] => 31 [messages] => 1 [recent] => 0 [unseen] => 1 [uidnext] => 41 [uidvalidity] => 1 ) imap: deleting message 1 imap: expunge imap: closing connection There are 1 messages to process memory at start of email processing:8685072 Array ( [add_meta] => no [admin_username] => admin [allow_html_in_body] => 1 [allow_html_in_subject] => 1 [allow_subject_in_mail] => 1 [audiotemplate] => <a href="{FILELINK}">{FILENAME}</a> [audiotypes] => Array ( [0] => m4a [1] => mp3 [2] => ogg [3] => wav [4] => mpeg ) [authorized_addresses] => Array ( ) [banned_files_list] => Array ( ) [confirmation_email] => admin [convertnewline] => 1 [converturls] => 1 [custom_image_field] => 0 [default_post_category] => 6 [category_match] => 1 [default_post_tags] => Array ( [0] => sens ) [default_title] => post: [delete_mail_after_processing] => 1 [drop_signature] => 1 [filternewlines] => 0 [forward_rejected_mail] => 1 [icon_set] => silver [icon_size] => 32 [auto_gallery] => 0 [image_new_window] => [image_placeholder] => #img%# [images_append] => 1 [imagetemplate] => <a href="{PAGELINK}"><img src="{MEDIUM}" alt="{CAPTION}" width="{MEDIUMWIDTH}" height="{MEDIUMHEIGHT}" class="alignnone size-medium wp-image-{ID}" /></a> [imagetemplates] => Array ( [simple_link] => <a href="{FILELINK}">{FILENAME}</a> [no_wrappers] => <a href="{IMAGE}"><img src="{THUMBNAIL}" alt="{CAPTION}" title="{CAPTION}" class="attachment" /></a> [wordpress_default] => <a href="{PAGELINK}"><img src="{MEDIUM}" alt="{CAPTION}" width="{MEDIUMWIDTH}" height="{MEDIUMHEIGHT}" class="alignnone size-medium wp-image-{ID}" /></a> [thumbnail_left] => <div style="float:left;margin-right:10px;"><a href="{IMAGE}"><img src="{THUMBNAIL}" alt="{CAPTION}" title="{CAPTION}" class="attachment" /></a></div> [thumbnail_right] => <div style="float:right;margin-left:10px;"><a href="{IMAGE}"><img src="{THUMBNAIL}" alt="{CAPTION}" title="{CAPTION}" class="attachment" /></a></div> [robert_felty] => <div style="margin-right:10px;background:black;color:white;padding:2px; width:{MEDIUMWIDTH};float:left"><a href="{IMAGE}"><img src="{MEDIUM}" alt="{CAPTION}" title="{CAPTION}" class="attachment" /></a><div style="padding:.2em;text-align:left">{CAPTION}</div></div> [postie_legacy] => <div class="postie-image-div"><a href="{IMAGE}"><img src="{THUMBNAIL}" alt="{FILENAME}" title="{FILENAME}" style="border:none" class="postie-image" /></a></div> [custom] => ) [input_protocol] => imap-ssl [interval] => twiceperhour [mail_server] => imap.gmail.com [mail_server_port] => 993 [mail_userid] => blog@example.com [maxemails] => 0 [message_start] => :start [message_end] => :end [message_encoding] => UTF-8 [message_dequote] => 1 [post_status] => publish [prefer_text_type] => html [return_to_sender] => 0 [role_access] => Array ( ) [selected_audiotemplate] => simple_link [selected_imagetemplate] => wordpress_default [selected_video1template] => simple_link [selected_video2template] => simple_link [shortcode] => [sig_pattern_list] => Array ( [0] => --\s?[\r\n]? [1] => --\s [2] => -- [3] => --- ) [smtp] => Array ( ) [start_image_count_at_zero] => 0 [supported_file_types] => Array ( [0] => application ) [turn_authorization_off] => 1 [time_offset] => 0 [video1template] => <a href="{FILELINK}">{FILENAME}</a> [video1types] => Array ( [0] => mp4 [1] => mpeg4 [2] => 3gp [3] => 3gpp [4] => 3gpp2 [5] => 3gp2 [6] => mov [7] => mpeg [8] => quicktime ) [video2template] => <a href="{FILELINK}">{FILENAME}</a> [video2types] => Array ( [0] => x-flv ) [video1templates] => Array ( [simple_link] => <a href="{FILELINK}">{FILENAME}</a> [small] => <object classid="clsid:02BF25D5-8C17-4B23-BC80-D3488ABDDC6B" codebase="http://www.apple.com/qtactivex/qtplugin.cab" width="128"height="96"> <param name="src" value="{FILELINK}" /> <param name="autoplay" value="no" /> <param name="controller" value="true" /> <embed src="{FILELINK}" width="128" height="96"autoplay="no" controller="true" type="video/quicktime" pluginspage="http://www.apple.com/quicktime/download/" width="128" height="110" scale="tofit"></embed> </object> [medium] => <object classid="clsid:02BF25D5-8C17-4B23-BC80-D3488ABDDC6B" codebase="http://www.apple.com/qtactivex/qtplugin.cab" width="320"height="240"> <param name="src" value="{FILELINK}" /> <param name="autoplay" value="no" /> <param name="controller" value="true" /> <embed src="{FILELINK}" width="320" height="240"autoplay="no" controller="true" type="video/quicktime" pluginspage="http://www.apple.com/quicktime/download/" width="320" height="260" scale="tofit"></embed> </object> [medium_widescreen] => <object classid="clsid:02BF25D5-8C17-4B23-BC80-D3488ABDDC6B" codebase="http://www.apple.com/qtactivex/qtplugin.cab" width="480"height="270"> <param name="src" value="{FILELINK}" /> <param name="autoplay" value="no" /> <param name="controller" value="true" /> <embed src="{FILELINK}" width="480" height="270"autoplay="no" controller="true" type="video/quicktime" pluginspage="http://www.apple.com/quicktime/download/" width="480" height="290" scale="tofit"></embed> </object> [large] => <object classid="clsid:02BF25D5-8C17-4B23-BC80-D3488ABDDC6B" codebase="http://www.apple.com/qtactivex/qtplugin.cab" width="480"height="360"> <param name="src" value="{FILELINK}" /> <param name="autoplay" value="no" /> <param name="controller" value="true" /> <embed src="{FILELINK}" width="480" height="360"autoplay="no" controller="true" type="video/quicktime" pluginspage="http://www.apple.com/quicktime/download/" width="480" height="380" scale="tofit"></embed> </object> [large_widescreen] => <object classid="clsid:02BF25D5-8C17-4B23-BC80-D3488ABDDC6B" codebase="http://www.apple.com/qtactivex/qtplugin.cab" width="640"height="360"> <param name="src" value="{FILELINK}" /> <param name="autoplay" value="no" /> <param name="controller" value="true" /> <embed src="{FILELINK}" width="640" height="360"autoplay="no" controller="true" type="video/quicktime" pluginspage="http://www.apple.com/quicktime/download/" width="640" height="380" scale="tofit"></embed> </object> [vshortcode] => [video width="568" height="320" {FILETYPE}="{FILELINK}"][/video] [custom] => ) [video2templates] => Array ( [simple_link] => <a href="{FILELINK}">{FILENAME}</a> [small] => <object classid="clsid:02BF25D5-8C17-4B23-BC80-D3488ABDDC6B" codebase="http://www.apple.com/qtactivex/qtplugin.cab" width="128"height="96"> <param name="src" value="{FILELINK}" /> <param name="autoplay" value="no" /> <param name="controller" value="true" /> <embed src="{FILELINK}" width="128" height="96"autoplay="no" controller="true" type="video/quicktime" pluginspage="http://www.apple.com/quicktime/download/" width="128" height="110"></embed> </object> [medium] => <object classid="clsid:02BF25D5-8C17-4B23-BC80-D3488ABDDC6B" codebase="http://www.apple.com/qtactivex/qtplugin.cab" width="320"height="240"> <param name="src" value="{FILELINK}" /> <param name="autoplay" value="no" /> <param name="controller" value="true" /> <embed src="{FILELINK}" width="320" height="240"autoplay="no" controller="true" type="video/quicktime" pluginspage="http://www.apple.com/quicktime/download/" width="320" height="260"></embed> </object> [medium_widescreen] => <object classid="clsid:02BF25D5-8C17-4B23-BC80-D3488ABDDC6B" codebase="http://www.apple.com/qtactivex/qtplugin.cab" width="480"height="270"> <param name="src" value="{FILELINK}" /> <param name="autoplay" value="no" /> <param name="controller" value="true" /> <embed src="{FILELINK}" width="480" height="270"autoplay="no" controller="true" type="video/quicktime" pluginspage="http://www.apple.com/quicktime/download/" width="480" height="290"></embed> </object> [large] => <object classid="clsid:02BF25D5-8C17-4B23-BC80-D3488ABDDC6B" codebase="http://www.apple.com/qtactivex/qtplugin.cab" width="480"height="360"> <param name="src" value="{FILELINK}" /> <param name="autoplay" value="no" /> <param name="controller" value="true" /> <embed src="{FILELINK}" width="480" height="360"autoplay="no" controller="true" type="video/quicktime" pluginspage="http://www.apple.com/quicktime/download/" width="480" height="380"></embed> </object> [large_widescreen] => <object classid="clsid:02BF25D5-8C17-4B23-BC80-D3488ABDDC6B" codebase="http://www.apple.com/qtactivex/qtplugin.cab" width="640"height="360"> <param name="src" value="{FILELINK}" /> <param name="autoplay" value="no" /> <param name="controller" value="true" /> <embed src="{FILELINK}" width="640" height="360"autoplay="no" controller="true" type="video/quicktime" pluginspage="http://www.apple.com/quicktime/download/" width="640" height="380"></embed> </object> [flv_embed] => [flv:{FILELINK} 480 270] [custom] => ) [wrap_pre] => no [featured_image] => 1 [include_featured_image] => 1 [email_tls] => 0 [post_format] => standard [post_type] => post [generaltemplates] => Array ( [postie_default] => <a href="{FILELINK}">{ICON} {FILENAME}</a> [simple_link] => <a href="{FILELINK}">{FILENAME}</a> [custom] => ) [generaltemplate] => <a href="{FILELINK}">{ICON} {FILENAME}</a> [selected_generaltemplate] => postie_default [generate_thumbnails] => 1 [reply_as_comment] => 1 [force_user_login] => 0 [auto_gallery_link] => Default [ignore_mail_state] => 0 [strip_reply] => 1 [postie_log_error] => 1 [postie_log_debug] => 0 [category_colon] => 1 [category_dash] => 1 [category_bracket] => 1 ) 1: ------------------------------------ RemoveExtraCharactersInEmailAddress: sender@mail.com (1) Array ( [0] => sender name <sender@mail.com> [1] => sender@mail.com ) ValidatePoster: post postie_filter_email sender@mail.com RemoveExtraCharactersInEmailAddress: groupname@googlegroups.com (1) Array ( [0] => "groupname@googlegroups.com" <groupname@googlegroups.com> [1] => groupname@googlegroups.com ) ValidatePoster: post postie_filter_email2 sender@mail.com ValidatePoster: post postie_filter_email3 sender@mail.com Confirming Access For sender@mail.com ValidatePoster: looking up default user admin ValidatePoster: pre postie_author (default) 1 ValidatePoster: post postie_author (default) 1 ValidatePoster: found user '1' tmp post id is 94 Message Id is :<CAN9upigbDgBTM+gbuR-j4BQiSYVx2BPaPAk00sOsUV9jvHZkBw@mail.gmail.com> filter_PreferedText: begin 2 filter_PreferedText: part: 0 text/plain filter_PreferedText: checking prefered type filter_PreferedText: removing: plain filter_PreferedText: part: 1 text/html filter_PreferedText: checking prefered type filter_PreferedText: keeping: html PGRpdiBkaXI9InJ0bCI+PGRpdiBkaXI9Imx0ciI+YW5vdGhlciBlbWFpbDwvZGl2PjxkaXYgZGly PSJsdHIiPjx1bD48bGk+dGVzdGVkPC9saT48bGk+MTwvbGk+PGxpPjI8L2xpPjxsaT4zPC9saT48 L3VsPjxkaXY+PGEgaHJlZj0iaHR0cDovL2V4YW1wbGUuY29tLyI+bGluazwvYT48L2Rpdj48ZGl2 Pjxicj48L2Rpdj48L2Rpdj48L2Rpdj4NCg0KPHA+PC9wPgoKLS0gPGJyIC8+CuKAj9en15nXkdec 16og15DXqiDXlNeU15XXk9ei15Qg15TXlteVINee16TXoNeZINep15DXqteUINeo16nXldedINec 16fXkdeV16bXlCAn16DXmdeh15XXmSDXodeg16EnINep15wg16fXkdeV16bXldeqIEdvb2dsZS48 YnIgLz4K15vXk9eZINec15HXmNecINeQ 1 parts filter_PreferedText: end GetContent: ---- start GetContent: primary= multipart, secondary = alternative GetContent: pre sanitize file name '' GetContent: file name '' GetContent: extension '' GetContent: secondary lookup failed, checking configured extensions GetContent: found no extension GetContent: mimetype multipart/alternative GetContent: multipart: 1 filter_PreferedText: begin 1 filter_PreferedText: part: 0 text/html filter_PreferedText: checking prefered type filter_PreferedText: keeping: html PGRpdiBkaXI9InJ0bCI+PGRpdiBkaXI9Imx0ciI+YW5vdGhlciBlbWFpbDwvZGl2PjxkaXYgZGly PSJsdHIiPjx1bD48bGk+dGVzdGVkPC9saT48bGk+MTwvbGk+PGxpPjI8L2xpPjxsaT4zPC9saT48 L3VsPjxkaXY+PGEgaHJlZj0iaHR0cDovL2V4YW1wbGUuY29tLyI+bGluazwvYT48L2Rpdj48ZGl2 Pjxicj48L2Rpdj48L2Rpdj48L2Rpdj4NCg0KPHA+PC9wPgoKLS0gPGJyIC8+CuKAj9en15nXkdec 16og15DXqiDXlNeU15XXk9ei15Qg15TXlteVINee16TXoNeZINep15DXqteUINeo16nXldedINec 16fXkdeV16bXlCAn16DXmdeh15XXmSDXodeg16EnINep15wg16fXkdeV16bXldeqIEdvb2dsZS48 YnIgLz4K15vXk9eZINec15HXmNecINeQ 1 parts filter_PreferedText: end GetContent: ---- start GetContent: primary= text, secondary = html DecodeBase64Part: base64 detected GetContent: pre sanitize file name '' GetContent: file name '' GetContent: extension '' GetContent: secondary lookup failed, checking configured extensions GetContent: found no extension GetContent: mimetype text/html GetContent: ctype_primary: text GetContent: text charset: UTF-8 before HandleMessageEncoding email charset: utf-8 email encoding: after HandleMessageEncoding GetContent: html filter_CleanHtml: Looking for invalid tags filter_CleanHtml: <div dir="rtl"><div dir="ltr">another email</div><div dir="ltr"><ul><li>tested</li><li>1</li><li>2</li><li>3</li></ul><div><a href="http://example.com/">link</a></div><div><br></div></div></div> <p></p> -- <br /> קיבלת את ההודעה הזו מפני שאתה רשום לקבוצה 'ניסוי סנס' של קבוצות Google.<br /> כדי לבטל את הרישום לקבוצה הזו ולהפסיק לקבל ממנה אימייל, שלח אימייל אל <a href="mailto:groupname+unsubscribe@googlegroups.com">groupname+unsubscribe@googlegroups.com</a>.<br /> כדי לפרסם בקבוצה הזו, שלח אימייל אל <a href="mailto:groupname@googlegroups.com">groupname@googlegroups.com</a>.<br /> כדי להציג את הדיון הזה באתר, היכנס ל-<a href="https://groups.google.com/d/msgid/groupname/CAN9upigbDgBTM%2BgbuR-j4BQiSYVx2BPaPAk00sOsUV9jvHZkBw%40mail.gmail.com?utm_medium=email&utm_source=footer">https://groups.google.com/d/msgid/groupname/CAN9upigbDgBTM%2BgbuR-j4BQiSYVx2BPaPAk00sOsUV9jvHZkBw%40mail.gmail.com</a>.<br /> לאפשרויות נוספות, היכנס ל-<a href="https://groups.google.com/d/optout">https://groups.google.com/d/optout</a>.<br /> GetContent: meta_return: <div dir="rtl"><div dir="ltr">another email</div><div dir="ltr"><ul><li>tested</li><li>1</li><li>2</li><li>3</li></ul><div><a href="http://example.com/">link</a></div><div><br></div></div></div> <p></p> -- <br /> קיבלת את ההודעה הזו מפני שאתה רשום לקבוצה 'ניסוי סנס' של קבוצות Google.<br /> כדי לבטל את הרישום לקבוצה הזו ולהפסיק לקבל ממנה אימייל, שלח אימייל אל <a href="mailto:groupname+u GetContent: ==== end GetContent: meta_return: <div dir="rtl"><div dir="ltr">another email</div><div dir="ltr"><ul><li>tested</li><li>1</li><li>2</li><li>3</li></ul><div><a href="http://example.com/">link</a></div><div><br></div></div></div> <p></p> -- <br /> קיבלת את ההודעה הזו מפני שאתה רשום לקבוצה 'ניסוי סנס' של קבוצות Google.<br /> כדי לבטל את הרישום לקבוצה הזו ולהפסיק לקבל ממנה אימייל, שלח אימייל אל <a href="mailto:groupname+u GetContent: ==== end CreatePost: '<div dir="rtl"><div dir="ltr">another email</div><div dir="ltr"><ul><li>tested</li><li>1</li><li>2</li><li>3</li></ul><div><a href="http://example.com/">link</a></div><div><br></div></div></div> <p></p> -- <br /> קיבלת את ההודעה הזו מפני שאתה רשום לקבוצה 'ניסוי סנס' של קבוצות Google.<br /> כדי לבטל את הרישום לקבוצה הזו ולהפסיק לקבל ממנה אימייל, שלח אימייל אל <a href="mailto:groupname+unsubscribe@googlegroups.com">groupname+unsubscribe@googlegroups.com</a>.<br /> כדי לפרסם בקבוצה הזו, שלח אימייל אל <a href="mailto:groupname@googlegroups.com">groupname@googlegroups.com</a>.<br /> כדי להציג את הדיון הזה באתר, היכנס ל-<a href="https://groups.google.com/d/msgid/groupname/CAN9upigbDgBTM%2BgbuR-j4BQiSYVx2BPaPAk00sOsUV9jvHZkBw%40mail.gmail.com?utm_medium=email&utm_source=footer">https://groups.google.com/d/msgid/groupname/CAN9upigbDgBTM%2BgbuR-j4BQiSYVx2BPaPAk00sOsUV9jvHZkBw%40mail.gmail.com</a>.<br /> לאפשרויות נוספות, היכנס ל-<a href="https://groups.google.com/d/optout">https://groups.google.com/d/optout</a>.<br /> ' Array ( [html] => Array ( ) [cids] => Array ( ) [image_files] => Array ( ) ) Predecoded subject: new beta test email tag_Subject: Looking for subject in email body tag_Subject: No subject found, using default [1] Subject: new beta test email filter_RemoveSignature: pattern: /^(--\s?[\r\n]?|--\s|--|---)\s?$/mi filter_RemoveSignatureWorker: signature found in base, removing Array ( [0] => -- [1] => -- ) filter_RemoveSignatureWorker sig new text: '' post sig: post excerpt: RemoveExtraCharactersInEmailAddress: sender@mail.com (1) Array ( [0] => sender name <sender@mail.com> [1] => sender@mail.com ) post author: date header: Wed, 27 Apr 2016 19:04:10 +0000 before HandleMessageEncoding email charset: email encoding: after HandleMessageEncoding decoded date: Wed, 27 Apr 2016 19:04:10 +0000 tag_Date: not html tag_Delay: post date: 2016-04-27 19:04:10 / 2016-04-27 19:04:10 (gmt) post date: post ubb: post type: found format 'standard' post type: pre category: 6 post postie_category_default 6 tag_Categories: found categories Array ( ) tag_Categories: using default: 6 post category: post tag: post comment: post status: post cid: begin: filter_linkify begin: filter_linkify (html) begin: filter_linkify (email) post linkify: post vodafone: post custom: GetParentPostForReply: Looking for parent 'new beta test email' GetParentPostForReply: No parent found Not a reply post newline: post start: post end: images in post: 0 Array ( ) Auto gallery: none auto adding images to end post body ReplaceImagePlaceHolders: Post postie_post filter Array ( [post_author] => 1 [comment_author] => sender [comment_author_url] => [user_ID] => [email_author] => sender@mail.com [post_date] => 2016-04-27 19:04:10 [post_date_gmt] => 2016-04-27 19:04:10 [post_content] => [post_title] => new beta test email [post_type] => post [ping_status] => open [post_category] => Array ( [0] => 6 ) [tags_input] => Array ( [0] => sens ) [comment_status] => open [post_name] => new-beta-test-email [post_excerpt] => [ID] => 94 [customImages] => Array ( ) [post_status] => publish ) Post Author: 1 Date: 2016-04-27 19:04:10 Category: 6 Ping Status: open Comment Status: open Subject: new beta test email Postname: new-beta-test-email Post Id: 94 Post Type: post MailToRecipients: send mail To: admin@mail.com From: =?utf-8?b?16nXmdeX15Qg15HXodeZ16HXmdeq?= <sender@mail.com> MailToRecipients: Successfully posted to new blog Your post 'new beta test email' has been successfully published to שיחה בסיסית <http://blog.domain.com/2016/04/27/new-beta-test-email/>. Done 1: processed Mail fetch complete, 1 emails memory at end of email processing:9069144Looks like the signature detection is finding a false positive. Try setting “Drop The Signature From Mail” to No.
You were right, it works.
Now how can we get rid of the annoying signature?2 options
1) play around with the signature regular expressions
2) Create a Postie addon http://postieplugin.com/extending/ to parse the text and remove the extra stuffThanks.
Viewing 10 replies - 1 through 10 (of 10 total)
- The topic ‘Fail to extract content with google groups’ is closed to new replies.
