Rectangle 27 4

Based on this an other related questions, I have implemented the following solution, which seems to be quite robust and does not require any changes in all the print statements of my codebase:

# -*- coding: utf-8 -*-

import sys

def set_output_encoding(encoding='utf-8'):
    import sys
    import codecs
    '''When piping to the terminal, python knows the encoding needed, and
       sets it automatically. But when piping to another program (for example,
       | less), python can not check the output encoding. In that case, it 
       is None. What I am doing here is to catch this situation for both 
       stdout and stderr and force the encoding'''
    current = sys.stdout.encoding
    if current is None :
        sys.stdout = codecs.getwriter(encoding)(sys.stdout)
    current = sys.stderr.encoding
    if current is None :
        sys.stderr = codecs.getwriter(encoding)(sys.stderr)

GREEK = u'           '

set_output_encoding()

print GREEK
print >> sys.stderr, GREEK
python ddd.py             # Do not pipe anything
python ddd.py | less      # Pipe stdout, let stderr go to the terminal
python ddd.py 2>&1 | less # Pipe both stdout and stderr to less

All of them produce the expected:


                                                                                                                                                                                                                                                                

That's a nice solution!

unicode - Piping output causes python program to fail - Stack Overflow

python unicode terminal redirect terminal-emulator
Rectangle 27 4

Based on this an other related questions, I have implemented the following solution, which seems to be quite robust and does not require any changes in all the print statements of my codebase:

# -*- coding: utf-8 -*-

import sys

def set_output_encoding(encoding='utf-8'):
    import sys
    import codecs
    '''When piping to the terminal, python knows the encoding needed, and
       sets it automatically. But when piping to another program (for example,
       | less), python can not check the output encoding. In that case, it 
       is None. What I am doing here is to catch this situation for both 
       stdout and stderr and force the encoding'''
    current = sys.stdout.encoding
    if current is None :
        sys.stdout = codecs.getwriter(encoding)(sys.stdout)
    current = sys.stderr.encoding
    if current is None :
        sys.stderr = codecs.getwriter(encoding)(sys.stderr)

GREEK = u'           '

set_output_encoding()

print GREEK
print >> sys.stderr, GREEK
python ddd.py             # Do not pipe anything
python ddd.py | less      # Pipe stdout, let stderr go to the terminal
python ddd.py 2>&1 | less # Pipe both stdout and stderr to less

All of them produce the expected:


                                                                                                                                                                                                                                                                

That's a nice solution!

unicode - Piping output causes python program to fail - Stack Overflow

python unicode terminal redirect terminal-emulator
Rectangle 27 4

Based on this an other related questions, I have implemented the following solution, which seems to be quite robust and does not require any changes in all the print statements of my codebase:

# -*- coding: utf-8 -*-

import sys

def set_output_encoding(encoding='utf-8'):
    import sys
    import codecs
    '''When piping to the terminal, python knows the encoding needed, and
       sets it automatically. But when piping to another program (for example,
       | less), python can not check the output encoding. In that case, it 
       is None. What I am doing here is to catch this situation for both 
       stdout and stderr and force the encoding'''
    current = sys.stdout.encoding
    if current is None :
        sys.stdout = codecs.getwriter(encoding)(sys.stdout)
    current = sys.stderr.encoding
    if current is None :
        sys.stderr = codecs.getwriter(encoding)(sys.stderr)

GREEK = u'           '

set_output_encoding()

print GREEK
print >> sys.stderr, GREEK
python ddd.py             # Do not pipe anything
python ddd.py | less      # Pipe stdout, let stderr go to the terminal
python ddd.py 2>&1 | less # Pipe both stdout and stderr to less

All of them produce the expected:


                                                                                                                                                                                                                                                                

That's a nice solution!

unicode - Piping output causes python program to fail - Stack Overflow

python unicode terminal redirect terminal-emulator
Rectangle 27 131

Get a file-like object from the request and copy it to a file. This will also avoid reading the whole thing into memory at once.

import shutil

import requests

url = 'http://example.com/img.png'
response = requests.get(url, stream=True)
with open('img.png', 'wb') as out_file:
    shutil.copyfileobj(response.raw, out_file)
del response

It's worth noting that few servers are set to GZIP their images because images already have their own compression. It's counterproductive, wastes CPU cycles with little benefit. So while this may be an issue with text content, specifically with images it's not.

.raw

is there any way we can access the original filename

Content-Encoding: gzip
r.raw.decode_content = True
Content-Encoding: deflate
response.raw.read(decode_content=True)
shutil.copyfileobj(response.raw, out_file)
by default, decode compressed responses (with GZIP or deflate)
shutil.copyfileobj
.read()

python - How to download image using requests - Stack Overflow

python urllib2 python-requests
Rectangle 27 112

Not quite. Creating a Request object does not actually send the request, and Request objects have no Read() method. (Also: read() is lowercase.) All you need to do is pass the Request as the first argument to urlopen() and that will give you your response.

import urllib2
request = urllib2.Request("http://www.google.com", headers={"Accept" : "text/html"})
contents = urllib2.urlopen(request).read()

python - How do I send a custom header with urllib2 in a HTTP Request?...

python header urllib2
Rectangle 27 57

import urllib2
request = urllib2.Request('http://localhost:8080')
request.get_method = lambda : 'HEAD'

response = urllib2.urlopen(request)
print response.info()

Tested with quick and dirty HTTPd hacked in python:

Server: BaseHTTP/0.3 Python/2.6.6
Date: Sun, 12 Dec 2010 11:52:33 GMT
Content-type: text/html
X-REQUEST_METHOD: HEAD
Sun Dec 12 12:52:28 2010 Server Starts - localhost:8080
localhost.localdomain - - [12/Dec/2010 12:52:33] "HEAD / HTTP/1.1" 200 -
import httplib2
h = httplib2.Http()
resp = h.request("http://www.google.com", 'HEAD')

Making HTTP HEAD request with urllib2 from Python 2 - Stack Overflow

python python-2.7 urllib2 head
Rectangle 27 57

import urllib2
request = urllib2.Request('http://localhost:8080')
request.get_method = lambda : 'HEAD'

response = urllib2.urlopen(request)
print response.info()

Tested with quick and dirty HTTPd hacked in python:

Server: BaseHTTP/0.3 Python/2.6.6
Date: Sun, 12 Dec 2010 11:52:33 GMT
Content-type: text/html
X-REQUEST_METHOD: HEAD
Sun Dec 12 12:52:28 2010 Server Starts - localhost:8080
localhost.localdomain - - [12/Dec/2010 12:52:33] "HEAD / HTTP/1.1" 200 -
import httplib2
h = httplib2.Http()
resp = h.request("http://www.google.com", 'HEAD')

Making HTTP HEAD request with urllib2 from Python 2 - Stack Overflow

python python-2.7 urllib2 head
Rectangle 27 13

import urllib2
class RequestWithMethod(urllib2.Request):
  def __init__(self, *args, **kwargs):
    self._method = kwargs.pop('method', None)
    urllib2.Request.__init__(self, *args, **kwargs)

  def get_method(self):
    return self._method if self._method else super(RequestWithMethod, self).get_method()
self._method = kwargs.pop('method', None)
urllib2.Request.get_method(self)

python - How to make HTTP DELETE method using urllib2? - Stack Overflo...

python urllib2
Rectangle 27 169

urllib2 provides some extra functionality, namely the urlopen() function can allow you to specify headers (normally you'd have had to use httplib in the past, which is far more verbose.) More importantly though, urllib2 provides the Request class, which allows for a more declarative approach to doing a request:

r = Request(url='http://www.mysite.com')
r.add_header('User-Agent', 'awesome fetcher')
r.add_data(urllib.urlencode({'foo': 'bar'})
response = urlopen(r)

Note that urlencode() is only in urllib, not urllib2.

There are also handlers for implementing more advanced URL support in urllib2. The short answer is, unless you're working with legacy code, you probably want to use the URL opener from urllib2, but you still need to import into urllib for some of the utility functions.

Bonus answer With Google App Engine, you can use any of httplib, urllib or urllib2, but all of them are just wrappers for Google's URL Fetch API. That is, you are still subject to the same limitations such as ports, protocols, and the length of the response allowed. You can use the core of the libraries as you would expect for retrieving HTTP URLs, though.

How does somebody create a url with an encoded query string using urllib2? It's the only reason I'm using urllib and I'd like to make sure I'm doing everything the latest/greatest way.

Like in my above example, you use urlopen() and Request from urllib2, and you use urlencode() from urllib. No real harm in using both libraries, as long as you make sure you use the correct urlopen. The [urllib docs][1] are clear on that using this is acecepted usage. [1]: docs.python.org/library/urllib2.html#urllib2.urlopen

urllib2.urlopen

urllib2 does not support put or delete which is a pain

python - What are the differences between the urllib, urllib2, and req...

python urllib2 urllib python-requests
Rectangle 27 169

urllib2 provides some extra functionality, namely the urlopen() function can allow you to specify headers (normally you'd have had to use httplib in the past, which is far more verbose.) More importantly though, urllib2 provides the Request class, which allows for a more declarative approach to doing a request:

r = Request(url='http://www.mysite.com')
r.add_header('User-Agent', 'awesome fetcher')
r.add_data(urllib.urlencode({'foo': 'bar'})
response = urlopen(r)

Note that urlencode() is only in urllib, not urllib2.

There are also handlers for implementing more advanced URL support in urllib2. The short answer is, unless you're working with legacy code, you probably want to use the URL opener from urllib2, but you still need to import into urllib for some of the utility functions.

Bonus answer With Google App Engine, you can use any of httplib, urllib or urllib2, but all of them are just wrappers for Google's URL Fetch API. That is, you are still subject to the same limitations such as ports, protocols, and the length of the response allowed. You can use the core of the libraries as you would expect for retrieving HTTP URLs, though.

How does somebody create a url with an encoded query string using urllib2? It's the only reason I'm using urllib and I'd like to make sure I'm doing everything the latest/greatest way.

Like in my above example, you use urlopen() and Request from urllib2, and you use urlencode() from urllib. No real harm in using both libraries, as long as you make sure you use the correct urlopen. The [urllib docs][1] are clear on that using this is acecepted usage. [1]: docs.python.org/library/urllib2.html#urllib2.urlopen

urllib2.urlopen

urllib2 does not support put or delete which is a pain

python - What are the differences between the urllib, urllib2, and req...

python urllib2 urllib python-requests
Rectangle 27 169

urllib2 provides some extra functionality, namely the urlopen() function can allow you to specify headers (normally you'd have had to use httplib in the past, which is far more verbose.) More importantly though, urllib2 provides the Request class, which allows for a more declarative approach to doing a request:

r = Request(url='http://www.mysite.com')
r.add_header('User-Agent', 'awesome fetcher')
r.add_data(urllib.urlencode({'foo': 'bar'})
response = urlopen(r)

Note that urlencode() is only in urllib, not urllib2.

There are also handlers for implementing more advanced URL support in urllib2. The short answer is, unless you're working with legacy code, you probably want to use the URL opener from urllib2, but you still need to import into urllib for some of the utility functions.

Bonus answer With Google App Engine, you can use any of httplib, urllib or urllib2, but all of them are just wrappers for Google's URL Fetch API. That is, you are still subject to the same limitations such as ports, protocols, and the length of the response allowed. You can use the core of the libraries as you would expect for retrieving HTTP URLs, though.

How does somebody create a url with an encoded query string using urllib2? It's the only reason I'm using urllib and I'd like to make sure I'm doing everything the latest/greatest way.

Like in my above example, you use urlopen() and Request from urllib2, and you use urlencode() from urllib. No real harm in using both libraries, as long as you make sure you use the correct urlopen. The [urllib docs][1] are clear on that using this is acecepted usage. [1]: docs.python.org/library/urllib2.html#urllib2.urlopen

urllib2.urlopen

urllib2 does not support put or delete which is a pain

python - What are the differences between the urllib, urllib2, and req...

python urllib2 urllib python-requests
Rectangle 27 7

You can subclass the urllib2.Request object and override the method when you instantiate the class.

import urllib2

class RequestWithMethod(urllib2.Request):
  def __init__(self, method, *args, **kwargs):
    self._method = method
    urllib2.Request.__init__(*args, **kwargs)

  def get_method(self):
    return self._method

python - How to make HTTP DELETE method using urllib2? - Stack Overflo...

python urllib2
Rectangle 27 225

import urllib2
opener = urllib2.build_opener(urllib2.HTTPHandler)
request = urllib2.Request('http://example.org', data='your_put_data')
request.add_header('Content-Type', 'your/contenttype')
request.get_method = lambda: 'PUT'
url = opener.open(request)

Looks like a bit of a dirty hack, but it seems to work elegantly and completly

It would be less of a hack if you were to subclass urllib2.Request instead of monkey-patching it.

This answer was brilliant when it was written, but nowadays it's a lot easier to use the 'requests' package instead, see John Carter's answer. 'Requests' is in no way a toy - it is extremely capable.

requests

Thank you, I didn't want to use anything outside of the standard python library and this works perfect. I don't quite understand why the urllib2 is designed to only cater for GET and POST as standard, but this work around is champion. Can the urllib2.build_opener(urllib2.HTTPHandler) be reused in multiple calls?

Is there any way to do HTTP PUT in python - Stack Overflow

python http put
Rectangle 27 87

You can use urllib2

>>> import urllib2
>>> content = urllib2.urlopen(some_url).read()
>>> print content

Also you can use httplib

>>> import httplib
>>> conn = httplib.HTTPConnection("www.python.org")
>>> conn.request("HEAD","/index.html")
>>> res = conn.getresponse()
>>> print res.status, res.reason
200 OK
>>> import requests
>>> r = requests.get('https://api.github.com/user', auth=('user', 'pass'))
>>> r.status_code
200

Ah, a problem I had was that I needed to send the request with httplib. Thanks!

+1 for using requests

How do you send an HTTP Get Web Request in Python? - Stack Overflow

python http streamreader webrequest
Rectangle 27 11

import urllib2

request_headers = {
"Accept-Language": "en-US,en;q=0.5",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Referer": "http://thewebsite.com",
"Connection": "keep-alive" 
}

request = urllib2.Request("http://thewebsite.com", headers=request_headers)
contents = urllib2.urlopen(request).read()
print contents

python - How do I send a custom header with urllib2 in a HTTP Request?...

python header urllib2
Rectangle 27 15

import urllib2
request = urllib2.Request('http://localhost:8080')
request.get_method = lambda : 'HEAD'

response = urllib2.urlopen(request)
response.info().gettype()
import httplib2
h = httplib2.Http()
resp = h.request("http://www.google.com", 'HEAD')
assert resp[0]['status'] == 200
assert resp[0]['content-type'] == 'text/html'
...

Slightly nasty in that you're leaving get_method as an unbound function rather than binding it to request. (Viz, it'll work but it's bad style and if you wanted to use self in it - tough.)

Could you elaborate a bit more about pros and cons of this solution? I'm not an Python expert as you can see, so I could benefit knowing when it can turn bad ;) As fas as I understand the concern is that it's a hack that may or may not work depending on implementation change?

This second version in this code is the only one that worked for me for a URL with a 403 Forbidden. Others were throwing an exception.

How do you send a HEAD HTTP request in Python 2? - Stack Overflow

python python-2.7 http http-headers content-type
Rectangle 27 15

import urllib2
request = urllib2.Request('http://localhost:8080')
request.get_method = lambda : 'HEAD'

response = urllib2.urlopen(request)
response.info().gettype()
import httplib2
h = httplib2.Http()
resp = h.request("http://www.google.com", 'HEAD')
assert resp[0]['status'] == 200
assert resp[0]['content-type'] == 'text/html'
...

Slightly nasty in that you're leaving get_method as an unbound function rather than binding it to request. (Viz, it'll work but it's bad style and if you wanted to use self in it - tough.)

Could you elaborate a bit more about pros and cons of this solution? I'm not an Python expert as you can see, so I could benefit knowing when it can turn bad ;) As fas as I understand the concern is that it's a hack that may or may not work depending on implementation change?

This second version in this code is the only one that worked for me for a URL with a 403 Forbidden. Others were throwing an exception.

How do you send a HEAD HTTP request in Python 2? - Stack Overflow

python python-2.7 http http-headers content-type
Rectangle 27 15

import urllib2
request = urllib2.Request('http://localhost:8080')
request.get_method = lambda : 'HEAD'

response = urllib2.urlopen(request)
response.info().gettype()
import httplib2
h = httplib2.Http()
resp = h.request("http://www.google.com", 'HEAD')
assert resp[0]['status'] == 200
assert resp[0]['content-type'] == 'text/html'
...

Slightly nasty in that you're leaving get_method as an unbound function rather than binding it to request. (Viz, it'll work but it's bad style and if you wanted to use self in it - tough.)

Could you elaborate a bit more about pros and cons of this solution? I'm not an Python expert as you can see, so I could benefit knowing when it can turn bad ;) As fas as I understand the concern is that it's a hack that may or may not work depending on implementation change?

This second version in this code is the only one that worked for me for a URL with a 403 Forbidden. Others were throwing an exception.

How do you send a HEAD HTTP request in Python 2? - Stack Overflow

python python-2.7 http http-headers content-type
Rectangle 27 15

import urllib2
request = urllib2.Request('http://localhost:8080')
request.get_method = lambda : 'HEAD'

response = urllib2.urlopen(request)
response.info().gettype()
import httplib2
h = httplib2.Http()
resp = h.request("http://www.google.com", 'HEAD')
assert resp[0]['status'] == 200
assert resp[0]['content-type'] == 'text/html'
...

Slightly nasty in that you're leaving get_method as an unbound function rather than binding it to request. (Viz, it'll work but it's bad style and if you wanted to use self in it - tough.)

Could you elaborate a bit more about pros and cons of this solution? I'm not an Python expert as you can see, so I could benefit knowing when it can turn bad ;) As fas as I understand the concern is that it's a hack that may or may not work depending on implementation change?

This second version in this code is the only one that worked for me for a URL with a 403 Forbidden. Others were throwing an exception.

How do you send a HEAD HTTP request in Python 2? - Stack Overflow

python python-2.7 http http-headers content-type
Rectangle 27 18

urllib and urllib2 are both Python modules that do URL request related stuff but offer different functionalities.

1) urllib2 can accept a Request object to set the headers for a URL request, urllib accepts only a URL.

2) urllib provides the urlencode method which is used for the generation of GET query strings, urllib2 doesn't have such a function. This is one of the reasons why urllib is often used along with urllib2.

Requests - Requests is a simple, easy-to-use HTTP library written in Python.

1) Python Requests encodes the parameters automatically so you just pass them as simple arguments, unlike in the case of urllib, where you need to use the method urllib.encode() to encode the parameters before passing them.

2) It automatically decoded the response into Unicode.

3) Requests also has far more convenient error handling.If your authentication failed, urllib2 would raise a urllib2.URLError, while Requests would return a normal response object, as expected. All you have to see if the request was successful by boolean response.ok

python - What are the differences between the urllib, urllib2, and req...

python urllib2 urllib python-requests