Rectangle 27 0

You can try this to fetch the header information of all the mails in just 1 Go to server.

import imaplib
import email

obj = imaplib.IMAP4_SSL('imap.gmail.com', 993)
obj.login('username', 'password')
obj.select('folder_name')
resp,data = obj.uid('FETCH', '1:*' , '(RFC822.HEADER)')
messages = [data[i][1].strip() + "\r\nSize:" + data[i][0].split()[4] + "\r\nUID:" + data[i][0].split()[2]  for i in xrange(0, len(data), 2)]
for msg in messages:
    msg_str = email.message_from_string(msg)
    message_id = msg_str.get('Message-ID')

python - Fetching multiple IMAP messages at once - Stack Overflow

python imap imaplib
Rectangle 27 0

The following was posted in the IMAP Protocol mailing list earlier this week:

"As of yesterday [Monday October 7th], Gmail now supports reporting flags changes anywhere we would report new/expunged messages (ie, most places its allowed by the protocol, but definitely during IDLE). Its only enabled for gmail.com users at the moment, Google Apps users will follow in a week or so, assuming we don't find any issues."

However, there are apparently issues with the new functionality because today they said it is being rolled back:

"We're seeing several reports of programs not expected extra FETCH responses, which we rolled out on Monday. We're rolling back soon, since older versions of a very popular client are having issues (though not to our knowledge with this part of things)."

python - Gmail IMAP is sometimes returning bad results for fetch - Sta...

python gmail imap gmail-imap
Rectangle 27 0

You don't need to dirtily hack imaplib. You could try using the SocksiPy package, which supports socks4, socks5 and http proxy (connect):

Something like this, obviously you'd want to handle the setproxy options better, via extra arguments to a custom __init__ method, etc. You could probably do similar with the IMAP4_SSL.

from imaplib import IMAP4, IMAP4_PORT
from socks import sockssocket, PROXY_TYPE_SOCKS4, PROXY_TYPE_SOCKS5, PROXY_TYPE_HTTP

class SocksIMAP4(IMAP4):
    def open(self,host,port=IMAP4_PORT):
        self.host = host
        self.port = port
        self.sock = sockssocket()
        self.sock.setproxy(PROXY_TYPE_SOCKS5,'socks.example.com')
        self.sock.connect((host,port))
        self.file = self.sock.makefile('rb')

Nice find MattH, that's exactly the kind of thing I was looking for (didn't know SocksiPy). I'll give it a go. Thanks.

Python - How can I fetch emails via POP or IMAP through a proxy? - Sta...

python proxy imap pop
Rectangle 27 0

Answer to my own question... There's a quick and dirty way to force trafic from a python script to go through a proxy without hassle using Socksipy (thanks MattH for pointing me that way)

import socks
import socket
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS4,proxy_ip,port,True)
socket.socket = socks.socksocket

That global socket override is obviously a bit brutal, but works as a quick fix till I find the time to properly subclass IMAP4 and IMAP4_SSL.

Python - How can I fetch emails via POP or IMAP through a proxy? - Sta...

python proxy imap pop
Rectangle 27 0

Emails that are transfered as BASE64 must set Content-Transfer-Encoding. However you are most likely dealing with a MIME/Multipart message (e.g. both text/plain and HTML in the same message), in which case the transfer encoding is set separately for each part. You can test with is_multipart() or if Content-Type is multipart/alternative. If that is the case you use walk to iterate over the different parts.

EDIT: It is quite normal to send text/plain using quoted-printable and HTML using BASE64.

Content-Type: multipart/alternative; boundary="=_d6644db1a848db3cb25f2a8973539487"
Subject: multipart sample
From: Foo Bar <foo@example.net>
To: Fred Flintstone <fred@example.net>

--=_d6644db1a848db3cb25f2a8973539487
Content-Transfer-Encoding: base64
Content-Type: text/plain; charset=utf-8

SOME BASE64 HERE
--=_d6644db1a848db3cb25f2a8973539487
Content-Transfer-Encoding: base64
Content-Type: text/html; charset=utf-8

AND SOME OTHER BASE64 HERE

I can decode payload now, but it's hard to construct the mail after decoding, any suggestion?

It pretty much depends on what the content is, you can try encoding it as quoted-printable and change the content-transfer-encoding.

python - How to determine if a mail fetch by imap base64 encoded? - St...

python email base64 imap
Rectangle 27 0

Emails that are transfered as BASE64 must set Content-Transfer-Encoding. However you are most likely dealing with a MIME/Multipart message (e.g. both text/plain and HTML in the same message), in which case the transfer encoding is set separately for each part. You can test with is_multipart() or if Content-Type is multipart/alternative. If that is the case you use walk to iterate over the different parts.

EDIT: It is quite normal to send text/plain using quoted-printable and HTML using BASE64.

Content-Type: multipart/alternative; boundary="=_d6644db1a848db3cb25f2a8973539487"
Subject: multipart sample
From: Foo Bar <foo@example.net>
To: Fred Flintstone <fred@example.net>

--=_d6644db1a848db3cb25f2a8973539487
Content-Transfer-Encoding: base64
Content-Type: text/plain; charset=utf-8

SOME BASE64 HERE
--=_d6644db1a848db3cb25f2a8973539487
Content-Transfer-Encoding: base64
Content-Type: text/html; charset=utf-8

AND SOME OTHER BASE64 HERE

I can decode payload now, but it's hard to construct the mail after decoding, any suggestion?

It pretty much depends on what the content is, you can try encoding it as quoted-printable and change the content-transfer-encoding.

python - How to determine if a mail fetch by imap base64 encoded? - St...

python email base64 imap
Rectangle 27 0

Okay. I found out that email.utils can help out a lot here.

It has methods to parse date, emails, and any other information from imap messages.

from email.header import decode_header
value, charset = decode_header(string_to_be_decoded)

python - IMAP Fetch Encoding - Stack Overflow

python character-encoding imap imaplib gmail-imap
Rectangle 27 0

All of those will retrieve the full MIME content of the message, which can be quite large if they include attachments, and will take quite a while.

If you fetch BODYSTRUCTURE, you can parse the format/structure of the message, and get an idea of the size before you download it, and where exactly the main "body" part is.

python - IMAP fetch body too slow - Stack Overflow

python imap imaplib
Rectangle 27 0

RFC 3501 says fetch takes a sequence set, but I didn't see a definition for that and the example uses a range form (2:4 = messages 2, 3, and 4). I figured out that a comma separated list of ids works. In python with imaplib, I've got something like:

status, email_ids = con.search(None, query)
    if status != 'OK':
        raise Exception("Error running imap search for spinvox messages: "
                        "%s" % status)

    fetch_ids = ','.join(email_ids[0].split())
    status, data = con.fetch(fetch_ids, '(RFC822.HEADER BODY.PEEK[1])')
    if status != 'OK':
        raise Exception("Error running imap fetch for spinvox message: "
                        "%s" % status)
    for i in range(len(email_ids[0].split())):
        header_msg = email.message_from_string(data[i * 3 + 0][1])
        subject = header_msg['Subject'],
        date = header_msg['Date'],
        body = data[i * 3 + 1][1] # includes some mime multipart junk

python - Fetching multiple IMAP messages at once - Stack Overflow

python imap imaplib
Rectangle 27 0

ALL isn't a fetch item, it's a macro. A bit a special case in the syntax. There are two others, FAST and FULL. They're like fetch items but with some extra restrictions, one of which you've run into.

I suggest that you just use the equivalent fetch items and you'll get the result you want without having to deal with the special restrictions. From RFC3501:

ALL
     Macro equivalent to: (FLAGS INTERNALDATE RFC822.SIZE ENVELOPE)

  FAST
     Macro equivalent to: (FLAGS INTERNALDATE RFC822.SIZE)

  FULL
     Macro equivalent to: (FLAGS INTERNALDATE RFC822.SIZE ENVELOPE BODY)

imap - Pythons imaplib: fetch command, with or without parentheses, 'I...

python imap fetch imaplib imapclient
Rectangle 27 0

Gmail is sending you unsolicited FLAGS updates (because someone changed the message remotely). These aren't in response to your request, but IMAP allows the server to send you any* information at any time. Many servers will save these unsolicited responses for IDLE or NOOP responses, but apparently Gmail does not wish to wait.

*: There are a few rules about which responses can be sent when to avoid race conditions, but this isn't one of those.

python - Gmail IMAP is sometimes returning bad results for fetch - Sta...

python gmail imap gmail-imap
Rectangle 27 0

If I understand you correctly you're trying to put a square peg in a round hole.

An HTTP Proxy only knows how to "talk" HTTP so can't connect to a POP or IMAP server directly.

If you want to do this you'll need to implement your own server somewhere to talk to the mail servers. It would receive HTTP Requests and then make the appropriate calls to the Mail Server. E.g.:

How practical this would be I don't know since you'd have to convert a stateful protocol into a stateless one.

You're 100% right, I forgot to mention socks capability. I removed HTTP from the title, sorry for that, I'm tired ;p

The 'CONNECT' method of an HTTP proxy would let you run any tcp client->server protocol. IF the HTTP proxy allows it, most default configurations restrict 'CONNECT' requests to port 443.

I understand what you mean but how does thunderbird connect through http proxy then?

Python - How can I fetch emails via POP or IMAP through a proxy? - Sta...

python proxy imap pop