Rectangle 27 15

from geopy.geocoders import Nominatim
from geopy.exc import GeocoderTimedOut

my_address = '1600 Pennsylvania Avenue NW Washington, DC 20500'

geolocator = Nominatim()
try:
    location = geolocator.geocode(my_address)
    print location.latitude, location.longitude
except GeocoderTimedOut as e:
    print("Error: geocode failed on input %s with message %s"%(my_address, e.msg))

You can also consider increasing the timeout on the geocode call you are making to your geolocator. In my example it would be something like:

location = geolocator.geocode(my_address, timeout=None)

Just a note, the message inside e in the GeocoderTimedOut error seems to be .message now instead of .msg

.msg is not working

python - Geopy: catch timeout error - Stack Overflow

python scrapy geopy
Rectangle 27 1

You may be experiencing this problem because you tried to request this address multiple times and they temporarily blocked you or slowed you down because of their usage policy. It states no more requests than one per second and that you should cache your results. I ran into this problem and you have a couple solutions. If you don't want to change your code much you can get a Google API key that you can use for something like 2500 requests/day for free or you can cache your results. Because I was already using DynamoDB on AWS for my problem I went ahead and just created a table that I cache my results in. Here is the gist of my code.

python - Geopy: catch timeout error - Stack Overflow

python scrapy geopy
Rectangle 27 5

I expect that you exceded usage policy for Nominatim service (http://wiki.openstreetmap.org/wiki/Nominatim_usage_policy). Try to put a sleep of 1 sec between requests and cache the results, probable are a lot of duplicates.

from time import sleep
### your code
row[14] = location.longitude
sleep(1) # after last line in if
coords = {}
if coords.has_key([row[8], row[12] ]):
    row[13] , row[14] = coords[ [ row[8], row[12] ] ]
else:
    #geolocate
import os
from time import sleep
from geopy.geocoders import Nominatim
os.getcwd() #check current working directory
os.chdir("C:\Users\Philip\Documents\HDSDA1\Project\Global Terrorism Database")

#import file as a csv
import csv
gtd=open("gtd_original.csv","r")
csv_f=csv.reader(gtd)
outf=open("r_ready.csv","wb")
writer=csv.writer(outf,dialect='excel')
coords = {}
for row in csv_f:
    if row[13] in ("","NA") or row[14] in ("","NA"):   
        lookup = row[12] + "," + row[8]  # creates a city,country

        if coords.has_key( (row[8], row[12]) ):   ## test if result is already cached
            row[13] , row[14] = coords[ (row[8], row[12]) ]
        else:    
            geolocator = Nominatim()
            location = geolocator.geocode(lookup, timeout = None) #looks up the city/country on maps
            try:
                location.latitude
            except:
                lookup = row[8]
                location = geolocator.geocode(lookup) 
            row[13] = location.latitude
            row[14] = location.longitude
            coords[ (row[8], row[12]) ] = (row[13] , row[14])  # cache the new coords
            sleep(1) # sleep for 1 sec (required by Nominatim usage policy)

    writer.writerow(row)      
gtd.close()
outf.close()

Thanks for the quick answer. Can you explain how to put a sleep between results? And provide an example of how I could implement it? I'm just feeling a little out of my depth

@pitts_brother I assume you already got this but: import time then in your loop - time.sleep(seconds)

Timeout error in Python geopy geocoder - Stack Overflow

python timeout geopy
Rectangle 27 19

What you can do is define an errback in your Request instances:

errback (callable) a function that will be called if any exception was raised while processing the request. This includes pages that failed with 404 HTTP errors and such. It receives a Twisted Failure instance as first parameter.

Here's some sample code (for scrapy 1.0) that you can use:

# -*- coding: utf-8 -*-
# errbacks.py
import scrapy

# from scrapy.contrib.spidermiddleware.httperror import HttpError
from scrapy.spidermiddlewares.httperror import HttpError
from twisted.internet.error import DNSLookupError
from twisted.internet.error import TimeoutError


class ErrbackSpider(scrapy.Spider):
    name = "errbacks"
    start_urls = [
        "http://www.httpbin.org/",              # HTTP 200 expected
        "http://www.httpbin.org/status/404",    # Not found error
        "http://www.httpbin.org/status/500",    # server issue
        "http://www.httpbin.org:12345/",        # non-responding host, timeout expected
        "http://www.httphttpbinbin.org/",       # DNS error expected
    ]

    def start_requests(self):
        for u in self.start_urls:
            yield scrapy.Request(u, callback=self.parse_httpbin,
                                    errback=self.errback_httpbin,
                                    dont_filter=True)

    def parse_httpbin(self, response):
        self.logger.error('Got successful response from {}'.format(response.url))
        # do something useful now

    def errback_httpbin(self, failure):
        # log all errback failures,
        # in case you want to do something special for some errors,
        # you may need the failure's type
        self.logger.error(repr(failure))

        #if isinstance(failure.value, HttpError):
        if failure.check(HttpError):
            # you can get the response
            response = failure.value.response
            self.logger.error('HttpError on %s', response.url)

        #elif isinstance(failure.value, DNSLookupError):
        elif failure.check(DNSLookupError):
            # this is the original request
            request = failure.request
            self.logger.error('DNSLookupError on %s', request.url)

        #elif isinstance(failure.value, TimeoutError):
        elif failure.check(TimeoutError):
            request = failure.request
            self.logger.error('TimeoutError on %s', request.url)

And the output in scrapy shell (only 1 retry and 5s download timeout):

$ scrapy runspider errbacks.py --set DOWNLOAD_TIMEOUT=5 --set RETRY_TIMES=1
2015-06-30 23:45:55 [scrapy] INFO: Scrapy 1.0.0 started (bot: scrapybot)
2015-06-30 23:45:55 [scrapy] INFO: Optional features available: ssl, http11
2015-06-30 23:45:55 [scrapy] INFO: Overridden settings: {'DOWNLOAD_TIMEOUT': '5', 'RETRY_TIMES': '1'}
2015-06-30 23:45:56 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState
2015-06-30 23:45:56 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2015-06-30 23:45:56 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2015-06-30 23:45:56 [scrapy] INFO: Enabled item pipelines: 
2015-06-30 23:45:56 [scrapy] INFO: Spider opened
2015-06-30 23:45:56 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2015-06-30 23:45:56 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2015-06-30 23:45:56 [scrapy] DEBUG: Retrying <GET http://www.httphttpbinbin.org/> (failed 1 times): DNS lookup failed: address 'www.httphttpbinbin.org' not found: [Errno -5] No address associated with hostname.
2015-06-30 23:45:56 [scrapy] DEBUG: Gave up retrying <GET http://www.httphttpbinbin.org/> (failed 2 times): DNS lookup failed: address 'www.httphttpbinbin.org' not found: [Errno -5] No address associated with hostname.
2015-06-30 23:45:56 [errbacks] ERROR: <twisted.python.failure.Failure <class 'twisted.internet.error.DNSLookupError'>>
2015-06-30 23:45:56 [errbacks] ERROR: DNSLookupError on http://www.httphttpbinbin.org/
2015-06-30 23:45:56 [scrapy] DEBUG: Crawled (200) <GET http://www.httpbin.org/> (referer: None)
2015-06-30 23:45:56 [scrapy] DEBUG: Crawled (404) <GET http://www.httpbin.org/status/404> (referer: None)
2015-06-30 23:45:56 [errbacks] ERROR: Got successful response from http://www.httpbin.org/
2015-06-30 23:45:56 [errbacks] ERROR: <twisted.python.failure.Failure <class 'scrapy.spidermiddlewares.httperror.HttpError'>>
2015-06-30 23:45:56 [errbacks] ERROR: HttpError on http://www.httpbin.org/status/404
2015-06-30 23:45:56 [scrapy] DEBUG: Retrying <GET http://www.httpbin.org/status/500> (failed 1 times): 500 Internal Server Error
2015-06-30 23:45:57 [scrapy] DEBUG: Gave up retrying <GET http://www.httpbin.org/status/500> (failed 2 times): 500 Internal Server Error
2015-06-30 23:45:57 [scrapy] DEBUG: Crawled (500) <GET http://www.httpbin.org/status/500> (referer: None)
2015-06-30 23:45:57 [errbacks] ERROR: <twisted.python.failure.Failure <class 'scrapy.spidermiddlewares.httperror.HttpError'>>
2015-06-30 23:45:57 [errbacks] ERROR: HttpError on http://www.httpbin.org/status/500
2015-06-30 23:46:01 [scrapy] DEBUG: Retrying <GET http://www.httpbin.org:12345/> (failed 1 times): User timeout caused connection failure.
2015-06-30 23:46:06 [scrapy] DEBUG: Gave up retrying <GET http://www.httpbin.org:12345/> (failed 2 times): User timeout caused connection failure.
2015-06-30 23:46:06 [errbacks] ERROR: <twisted.python.failure.Failure <class 'twisted.internet.error.TimeoutError'>>
2015-06-30 23:46:06 [errbacks] ERROR: TimeoutError on http://www.httpbin.org:12345/
2015-06-30 23:46:06 [scrapy] INFO: Closing spider (finished)
2015-06-30 23:46:06 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 4,
 'downloader/exception_type_count/twisted.internet.error.DNSLookupError': 2,
 'downloader/exception_type_count/twisted.internet.error.TimeoutError': 2,
 'downloader/request_bytes': 1748,
 'downloader/request_count': 8,
 'downloader/request_method_count/GET': 8,
 'downloader/response_bytes': 12506,
 'downloader/response_count': 4,
 'downloader/response_status_count/200': 1,
 'downloader/response_status_count/404': 1,
 'downloader/response_status_count/500': 2,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2015, 6, 30, 21, 46, 6, 537191),
 'log_count/DEBUG': 10,
 'log_count/ERROR': 9,
 'log_count/INFO': 7,
 'response_received_count': 3,
 'scheduler/dequeued': 8,
 'scheduler/dequeued/memory': 8,
 'scheduler/enqueued': 8,
 'scheduler/enqueued/memory': 8,
 'start_time': datetime.datetime(2015, 6, 30, 21, 45, 56, 322235)}
2015-06-30 23:46:06 [scrapy] INFO: Spider closed (finished)

Notice how scrapy logs the exceptions in its stats:

'downloader/exception_type_count/twisted.internet.error.DNSLookupError': 2,
'downloader/exception_type_count/twisted.internet.error.TimeoutError': 2,

python - How do I catch errors with scrapy so I can do something when ...

python scrapy twisted
Rectangle 27 2

Nominatim has stopped working I guess so I used GoogleV3. This returns less information for the Address but it may still work.

from geopy.geocoders import GoogleV3
def geopy(): 

    loc = raw_input("What location? ")
    geolocator =  GoogleV3()
    location = geolocator.geocode(loc)
    if location != None:
        Address = location.address
        lat_long = location.latitude,location.longitude
        print Address, lat_long

    else:
        print "There is no geographic information to return for the word in input. \n"

I see random timeouts with GoogleV3, supposedly you can do up to 2500 queries in a day. I did hit the limit one day when I didn't setup a cron job correctly.

python - Geopy error and timeout - Stack Overflow

python geopy
Rectangle 27 13

How does the @timeout(timelimit) decorator work?

@timeout(100)
def foo(arg1, kwarg1=None):
    '''time this out!'''
    something_worth_timing_out()

The above is the decorator syntax. The below is exactly equivalent:

def foo(arg1, kwarg1=None):
    '''time this out!'''
    something_worth_timing_out()

foo = timeout(100)(foo)

Note that we name the function that wraps foo, "foo". That's what the decorator syntax means and does.

from functools import wraps
import errno
import os
import signal
class TimeoutError(Exception):
    pass

This is what's called in the line, @timeout(timelimit). These arguments will be locked into the underlying functions, making those functions "closures" so-called because they close-over the data:

This will return a function that takes a function as an argument, which the next line proceeds to define. This function will return a function that wraps the original function. :

def decorator(func):
def _handle_timeout(signum, frame):
            raise TimeoutError(error_message)

And this is the actual wrapper. Before calling the wrapped function, it sets a signal that will interrupt the function if it does not finish in time with an exception:

def wrapper(*args, **kwargs):
            signal.signal(signal.SIGALRM, _handle_timeout)
            signal.alarm(seconds)
            try:
                result = func(*args, **kwargs)
            finally:
                signal.alarm(0)

This will return the result if the function completes:

return result

This returns the wrapper. It makes sure the wrapped function gets the attributes from the original function, like docstrings, name, function signature...

return wraps(func)(wrapper)

and this is where the decorator is returned, from the original call, @timeout(timelimit):

return decorator

The wraps function allows the function that wraps the target function to get the documentation of that function, because foo no longer points at the original function:

>>> help(foo)
Help on function foo in module __main__:

foo(arg1, kwarg1=None)
    time this out!

To further clarify, wraps returns a decorator, and is intended to be used much like this function. It would be better written like this:

def timeout(seconds=100, error_message=os.strerror(errno.ETIME)):
    def decorator(func):
        def _handle_timeout(signum, frame):
            raise TimeoutError(error_message)
        @wraps(func)
        def wrapper(*args, **kwargs):
            signal.signal(signal.SIGALRM, _handle_timeout)
            signal.alarm(seconds)
            try:
                result = func(*args, **kwargs)
            finally:
                signal.alarm(0)
            return result
        return wrapper
    return decorator

1)What does the python interpreter actually do to process this code? specifically, how does it execute the @timeout(timelimit) directive? 2) return wraps(func)(wrapper) is unclear. Why does it have two parenthetical bodies? 3) why does the combination of try and signal.alarm spawn multiple threads, as is necessary to execute this function?

1) see the inner function wrapper. It gets called when you do foo(). 2) see addendum to end of the answer. 3) try has nothing to do with it, finally just stops the timer if it doesn't time-out, and you have to have a thread, otherwise concurrency would be much more difficult.

I guess my fundamental confusion is what the @decorator(args) does. What would equivalent python code be?

Let me clear that up, look at the very beginning of the answer.

python - How does the @timeout(timelimit) decorator work? - Stack Over...

python
Rectangle 27 3

Input with Timeout

Warning: This is intended to work in *nix and OSX as requested but definitely will not work in Windows.

I've used this modification of an ActiveState recipe as a basis for the code below. It's an easy-to-use object that can read input with a timeout. It uses polling to collect characters one at a time and emulate the behavior of raw_input() / input().

Note: apparently the _getch_nix() method below doesn't work for OP but it does for me on OSX 10.9.5. You might have luck calling _getch_osx() instead although it seems to work in 32-bit python only since Carbon doesn't fully support 64-bit.

import sys
import time


class TimeoutInput(object):
    def __init__(self, poll_period=0.05):
        import sys, tty, termios  # apparently timing of import is important if using an IDE
        self.poll_period = poll_period

    def _getch_nix(self):
        import sys, tty, termios
        from select import select
        fd = sys.stdin.fileno()
        old_settings = termios.tcgetattr(fd)
        try:
            tty.setraw(sys.stdin.fileno())
            [i, o, e] = select([sys.stdin.fileno()], [], [], self.poll_period)
            if i:
                ch = sys.stdin.read(1)
            else:
                ch = ''
        finally:
            termios.tcsetattr(fd, termios.TCSADRAIN, old_settings)
        return ch

    def _getch_osx(self):
        # from same discussion on the original ActiveState recipe:
        # http://code.activestate.com/recipes/134892-getch-like-unbuffered-character-reading-from-stdin/#c2
        import Carbon
        if Carbon.Evt.EventAvail(0x0008)[0] == 0:  # 0x0008 is the keyDownMask
            return ''
        else:
            # The event contains the following info:
            # (what,msg,when,where,mod)=Carbon.Evt.GetNextEvent(0x0008)[1]
            #
            # The message (msg) contains the ASCII char which is
            # extracted with the 0x000000FF charCodeMask; this
            # number is converted to an ASCII character with chr() and
            # returned
            (what,msg,when,where,mod)=Carbon.Evt.GetNextEvent(0x0008)[1]
            return chr(msg & 0x000000FF)

    def input(self, prompt=None, timeout=None,
              extend_timeout_with_input=True, require_enter_to_confirm=True):
        """timeout: float seconds or None (blocking)"""
        prompt = prompt or ''
        sys.stdout.write(prompt)  # this avoids a couple of problems with printing
        sys.stdout.flush()  # make sure prompt appears before we start waiting for input
        input_chars = []
        start_time = time.time()
        received_enter = False
        while (time.time() - start_time) < timeout:
            # keep polling for characters
            c = self._getch_osx()  # self.poll_period determines spin speed
            if c in ('\n', '\r'):
                received_enter = True
                break
            elif c:
                input_chars.append(c)
                sys.stdout.write(c)
                sys.stdout.flush()
                if extend_timeout_with_input:
                    start_time = time.time()
        sys.stdout.write('\n')  # just for consistency with other "prints"
        sys.stdout.flush()
        captured_string = ''.join(input_chars)
        if require_enter_to_confirm:
            return_string = captured_string if received_enter else ''
        else:
            return_string = captured_string
        return return_string
# this should work like raw_input() except it will time out
ti = TimeoutInput(poll_period=0.05)
s = ti.input(prompt='wait for timeout:', timeout=5.0,
             extend_timeout_with_input=False, require_enter_to_confirm=False)
print(s)

This implements your original intention as I understand it. I don't see any value to making recursive calls - I think what you want is just to get input repeatedly? Please correct me if that is wrong.

ti = TimeoutInput()
prompt = "Hello is it me you're looking for?"
timeout = 4.0
while True:
    # some_function()
    s = ti.input(prompt, timeout)
    if s.lower() == 'q':
        print "goodbye"
        break

Hi @kobejohn, I tried to implement your solution but an error came up for fileno(). So I think you get what I want... So some_function is initiated and takes how ever long it takes to complete. At the completion of some_function a prompt will come up on screen will you like to continue, if after a specified time say 5 seconds the user did not do anything some_function will perform its duties again and the process continues until I type q when prompted within allocated time. Hope that made sense, sorry if It wasn't clear above. I am happy to give you the bounty but I would like a working sol

@user3374113 Thanks for the information. 1) Please provide the details of the error you get. I have limited ability to test with *nix but I will try. 2) What OS exactly are you using? 3) Based on your description what you want is a loop rather than recursion. The solution above does exactly what you described with a loop. Hopefully we can get it working.

@user3374113 .... maybe you are working in IDLE? If so, please run the script outside of IDLE. While IDLE is has many good points, it tends to mess up unexpected things in the background.

please ignore the previous error i was referring to. The error I am getting is: error: (25, 'Inappropriate ioctl for device') . I am using enthought canopy. The error is pointing to: c = self._getch() and specifically old_settings = termios.tcgetattr(fd) in _getch(self). I'm using OSX Mountain lion btw.

@user3374113 I have no osx systems to test this on but I found some code in the same ActiveState discussion. Can you try the modified code above? I just added a function specific to osx but it is hard coded now to use the osx function. If it works, I'll rewrite it to work transparently.

python - User input with a timeout, in a loop - Stack Overflow

python python-2.7 raw-input keyboardinterrupt
Rectangle 27 130

You may use the signal package if you are running on UNIX:

In [1]: import signal

# Register an handler for the timeout
In [2]: def handler(signum, frame):
   ...:     print "Forever is over!"
   ...:     raise Exception("end of time")
   ...: 

# This function *may* run for an indetermined time...
In [3]: def loop_forever():
   ...:     import time
   ...:     while 1:
   ...:         print "sec"
   ...:         time.sleep(1)
   ...:         
   ...:         

# Register the signal function handler
In [4]: signal.signal(signal.SIGALRM, handler)
Out[4]: 0

# Define a timeout for your function
In [5]: signal.alarm(10)
Out[5]: 0

In [6]: try:
   ...:     loop_forever()
   ...: except Exception, exc: 
   ...:     print exc
   ....: 
sec
sec
sec
sec
sec
sec
sec
sec
Forever is over!
end of time

# Cancel the timer if the function returned before timeout
# (ok, mine won't but yours maybe will :)
In [7]: signal.alarm(0)
Out[7]: 0

10 seconds after the call alarm.alarm(10), the handler is called. This raises an exception that you can intercept from the regular Python code.

This module doesn't play well with threads (but then, who does?)

Note that since we raise an exception when timeout happens, it may end up caught and ignored inside the function, for example of one such function:

def loop_forever():
    while 1:
        print 'sec'
        try:
            time.sleep(10)
        except:
            continue

I use Python 2.5.4. There is such an error: Traceback (most recent call last): File "aa.py", line 85, in func signal.signal(signal.SIGALRM, handler) AttributeError: 'module' object has no attribute 'SIGALRM'

@flypen that's because signal.alarm and the related SIGALRM are not available on Windows platforms.

If there are a lot of processes, and each calls signal.signal --- will they all work properly? Won't each signal.signal call cancel "concurrent" one?

Warning for those wishing to use this with a C extension: The Python signal handler won't be called until the C function returns control to the Python interpreter. For this use case, use ATOzTOA's answer: stackoverflow.com/a/14924210/1286628

I second the warning about threads. signal.alarm only works on main thread. I tried to use this in Django views - immediate fail with verbiage about main thread only.

python - Timeout on a function call - Stack Overflow

python multithreading timeout python-multithreading
Rectangle 27 87

multiprocessing.Process
import multiprocessing
import time

# bar
def bar():
    for i in range(100):
        print "Tick"
        time.sleep(1)

if __name__ == '__main__':
    # Start bar as a process
    p = multiprocessing.Process(target=bar)
    p.start()

    # Wait for 10 seconds or until process finishes
    p.join(10)

    # If thread is still active
    if p.is_alive():
        print "running... let's kill it..."

        # Terminate
        p.terminate()
        p.join()

How can I get the return value of the target method ?

@ATOzTOA what if I am doing all this inside a class function definition? How do I update class variables?

@bad_keypoints See this answer: stackoverflow.com/a/10415215/1384471 Basically, you pass a list along that you put the answer into.

@sudo then remove the join(). that makes your x number of concurrent subprocesses being running untill them finish their work, or amount defined in join(10). Case you have a blocking I/O for 10 processes, using join(10) you have set them to wait all of them max 10 for EACH process that has started. Use daemon flag like this example stackoverflow.com/a/27420072/2480481. Of course u can pass flag daemon=True directly to multiprocessing.Process() function.

python - Timeout on a function call - Stack Overflow

python multithreading timeout python-multithreading
Rectangle 27 178

I don't know much about the low level details; but, given that in python 2.6 the API offers the ability to wait for threads and terminate processes, what about running the process in a separate thread?

import subprocess, threading

class Command(object):
    def __init__(self, cmd):
        self.cmd = cmd
        self.process = None

    def run(self, timeout):
        def target():
            print 'Thread started'
            self.process = subprocess.Popen(self.cmd, shell=True)
            self.process.communicate()
            print 'Thread finished'

        thread = threading.Thread(target=target)
        thread.start()

        thread.join(timeout)
        if thread.is_alive():
            print 'Terminating process'
            self.process.terminate()
            thread.join()
        print self.process.returncode

command = Command("echo 'Process started'; sleep 2; echo 'Process finished'")
command.run(timeout=3)
command.run(timeout=1)

The output of this snippet in my machine is:

Thread started
Process started
Process finished
Thread finished
0
Thread started
Process started
Terminating process
Thread finished
-15

where it can be seen that, in the first execution, the process finished correctly (return code 0), while the in the second one the process was terminated (return code -15).

I haven't tested in windows; but, aside from updating the example command, I think it should work since I haven't found in the documentation anything that says that thread.join or process.terminate is not supported.

+1 For being platform independent. I've run this on both linux and windows 7 (cygwin and plain windows python) -- works as expected in all three cases.

For anybody having the problem @redice was having, this question may help. In short, if you use shell=True, the shell becomes the child process which gets killed, and its command (child of the child process) lives on.

This answer does not provide the same functionality of the original since it doesn't return stdout.

python - Using module 'subprocess' with timeout - Stack Overflow

python multithreading timeout subprocess
Rectangle 27 100

from subprocess import STDOUT, check_output

output = check_output(cmd, stderr=STDOUT, timeout=seconds)

output is a byte string that contains command's merged stdout, stderr data.

This code raises CalledProcessError on non-zero exit status as specified in the question's text unlike proc.communicate() method.

I've removed shell=True because it is often used unnecessarily. You can always add it back if cmd indeed requires it. If you add shell=True i.e., if the child process spawns its own descendants; check_output() can return much later than the timeout indicates, see Subprocess timeout failure.

The timeout feature is available on Python 2.x via the subprocess32 backport of the 3.2+ subprocess module.

Indeed, and subprocess timeout support exists in the subprocess32 backport that I maintain for use on Python 2. pypi.python.org/pypi/subprocess32

@gps Sridhar asked for cross platform solution, while your backport only supports POSIX : when I tried it out, MSVC complained (expected) about missing unistd.h :)

If you don't need the output, you can just use the subprocess.call.

shell=True

python - Using module 'subprocess' with timeout - Stack Overflow

python multithreading timeout subprocess
Rectangle 27 179

The process for timing out an operations is described in the documentation for signal.

The basic idea is to use signal handlers to set an alarm for some time interval and raise an exception once that timer expires.

Note that this will only work on UNIX.

Here's an implementation that creates a decorator (save the following code as timeout.py).

from functools import wraps
import errno
import os
import signal

class TimeoutError(Exception):
    pass

def timeout(seconds=10, error_message=os.strerror(errno.ETIME)):
    def decorator(func):
        def _handle_timeout(signum, frame):
            raise TimeoutError(error_message)

        def wrapper(*args, **kwargs):
            signal.signal(signal.SIGALRM, _handle_timeout)
            signal.alarm(seconds)
            try:
                result = func(*args, **kwargs)
            finally:
                signal.alarm(0)
            return result

        return wraps(func)(wrapper)

    return decorator

This creates a decorator called @timeout that can be applied to any long running functions.

So, in your application code, you can use the decorator like so:

from timeout import timeout

# Timeout a long running function with the default expiry of 10 seconds.
@timeout
def long_running_function1():
    ...

# Timeout after 5 seconds
@timeout(5)
def long_running_function2():
    ...

# Timeout after 30 seconds, with the error "Connection timed out"
@timeout(30, os.strerror(errno.ETIMEDOUT))
def long_running_function3():
    ...

Beware that this is not thread-safe: if you're using multithreading, the signal will get caught by a random thread. For single-threaded programs though, this is the easiest solution.

Nice. Also, it is recommended to decorate the function wrapper with @functools.wraps(func)

FYI, there are missing parens after the first "@timeout". It should read @timeout() def ....

@wim I think it can only be used in main thread, because if you use it in worker thread, it will raise 'ValueError: signal only works in main thread'.

Sign up for our newsletter and get our top new questions delivered to your inbox (see an example).

python - Timeout function if it takes too long to finish - Stack Overf...

python
Rectangle 27 179

The process for timing out an operations is described in the documentation for signal.

The basic idea is to use signal handlers to set an alarm for some time interval and raise an exception once that timer expires.

Note that this will only work on UNIX.

Here's an implementation that creates a decorator (save the following code as timeout.py).

from functools import wraps
import errno
import os
import signal

class TimeoutError(Exception):
    pass

def timeout(seconds=10, error_message=os.strerror(errno.ETIME)):
    def decorator(func):
        def _handle_timeout(signum, frame):
            raise TimeoutError(error_message)

        def wrapper(*args, **kwargs):
            signal.signal(signal.SIGALRM, _handle_timeout)
            signal.alarm(seconds)
            try:
                result = func(*args, **kwargs)
            finally:
                signal.alarm(0)
            return result

        return wraps(func)(wrapper)

    return decorator

This creates a decorator called @timeout that can be applied to any long running functions.

So, in your application code, you can use the decorator like so:

from timeout import timeout

# Timeout a long running function with the default expiry of 10 seconds.
@timeout
def long_running_function1():
    ...

# Timeout after 5 seconds
@timeout(5)
def long_running_function2():
    ...

# Timeout after 30 seconds, with the error "Connection timed out"
@timeout(30, os.strerror(errno.ETIMEDOUT))
def long_running_function3():
    ...

Beware that this is not thread-safe: if you're using multithreading, the signal will get caught by a random thread. For single-threaded programs though, this is the easiest solution.

Nice. Also, it is recommended to decorate the function wrapper with @functools.wraps(func)

FYI, there are missing parens after the first "@timeout". It should read @timeout() def ....

@wim I think it can only be used in main thread, because if you use it in worker thread, it will raise 'ValueError: signal only works in main thread'.

python - Timeout function if it takes too long to finish - Stack Overf...

python
Rectangle 27 178

The process for timing out an operations is described in the documentation for signal.

The basic idea is to use signal handlers to set an alarm for some time interval and raise an exception once that timer expires.

Note that this will only work on UNIX.

Here's an implementation that creates a decorator (save the following code as timeout.py).

from functools import wraps
import errno
import os
import signal

class TimeoutError(Exception):
    pass

def timeout(seconds=10, error_message=os.strerror(errno.ETIME)):
    def decorator(func):
        def _handle_timeout(signum, frame):
            raise TimeoutError(error_message)

        def wrapper(*args, **kwargs):
            signal.signal(signal.SIGALRM, _handle_timeout)
            signal.alarm(seconds)
            try:
                result = func(*args, **kwargs)
            finally:
                signal.alarm(0)
            return result

        return wraps(func)(wrapper)

    return decorator

This creates a decorator called @timeout that can be applied to any long running functions.

So, in your application code, you can use the decorator like so:

from timeout import timeout

# Timeout a long running function with the default expiry of 10 seconds.
@timeout
def long_running_function1():
    ...

# Timeout after 5 seconds
@timeout(5)
def long_running_function2():
    ...

# Timeout after 30 seconds, with the error "Connection timed out"
@timeout(30, os.strerror(errno.ETIMEDOUT))
def long_running_function3():
    ...

Beware that this is not thread-safe: if you're using multithreading, the signal will get caught by a random thread. For single-threaded programs though, this is the easiest solution.

Nice. Also, it is recommended to decorate the function wrapper with @functools.wraps(func)

FYI, there are missing parens after the first "@timeout". It should read @timeout() def ....

@wim I think it can only be used in main thread, because if you use it in worker thread, it will raise 'ValueError: signal only works in main thread'.

python - Timeout function if it takes too long to finish - Stack Overf...

python
Rectangle 27 3

You could import the exception object and modify your except block:

import socket

try:
    soup = BeautifulSoup(urllib2.urlopen(url,timeout=10))   

except urllib2.URLError as e:
    print ("There was an error: %r" % e)
except socket.timeout as e: # <-------- this block here
    print "We timed out"

update: Well, learnt something new - just found a reference to a .reason property:

except urllib2.URLError as e:
    if isinstance(e.reason, socket.timeout):
        pass # ignore this one
    else:
        # do stuff re other errors if you can...
        raise # otherwise propagate the error

python - catching error 60 ( timeout ) with urllib 2 - Stack Overflow

python
Rectangle 27 40

I have a different proposal which is a pure function (with the same API as the threading suggestion) and seems to work fine (based on suggestions on this thread)

def timeout(func, args=(), kwargs={}, timeout_duration=1, default=None):
    import signal

    class TimeoutError(Exception):
        pass

    def handler(signum, frame):
        raise TimeoutError()

    # set the timeout handler
    signal.signal(signal.SIGALRM, handler) 
    signal.alarm(timeout_duration)
    try:
        result = func(*args, **kwargs)
    except TimeoutError as exc:
        result = default
    finally:
        signal.alarm(0)

    return result

One more note: The Unix signal method only works if you are applying it in the main thread. Applying it in a sub-thread throws an exception and will not work.

This is not the best solution because it only works on linux.

You should avoid setting kwargs to an empty dict. A common Python gotcha is that default arguments on functions are mutable. So that dictionary will be shared across all calls to timeout. It is much better to set the default to None and, on the first line of the function, add kwargs = kwargs or {}. Args is okay because tuples are not mutable.

python - Timeout on a function call - Stack Overflow

python multithreading timeout python-multithreading
Rectangle 27 72

import signal
  ...
class Alarm(Exception):
    pass

def alarm_handler(signum, frame):
    raise Alarm

signal.signal(signal.SIGALRM, alarm_handler)
signal.alarm(5*60)  # 5 minutes
try:
    stdoutdata, stderrdata = proc.communicate()
    signal.alarm(0)  # reset the alarm
except Alarm:
    print "Oops, taking too long!"
    # whatever else

Well, I am interested in a cross-platform solution that works at least on win/linux/mac.

I like this unix-based approach. Ideally, one would combine this with a windows-specific approach (using CreateProcess and Jobs) .. but for now, the solution below is simple, easy and works-so-far.

This solution would work only_if signal.signal(signal.SIGALARM, alarm_handler) is called from the main thread. See the documentation for signal

Unfortunately, when running (on linux) in the context of an Apache module (like mod_python, mod_perl, or mod_php), I've found the use of signals and alarms to be disallowed (presumably because they interfere with Apache's own IPC logic). So to achieve the goal of timing out a command I have been forced to write "parent loops" which launch a child process and then sit in a "sleep"y loop watching the clock (and possibly also monitoring output from the child).

python - Using module 'subprocess' with timeout - Stack Overflow

python multithreading timeout subprocess
Rectangle 27 67

import subprocess, shlex
from threading import Timer

def run(cmd, timeout_sec):
  proc = subprocess.Popen(shlex.split(cmd), stdout=subprocess.PIPE, 
    stderr=subprocess.PIPE)
  kill_proc = lambda p: p.kill()
  timer = Timer(timeout_sec, kill_proc, [proc])
  try:
    timer.start()
    stdout,stderr = proc.communicate()
  finally:
    timer.cancel()
lambda
t = Timer(timeout, proc.kill)

+1 This should be the accepted answer, because it doesn't require the way in which the process is launched to be changed.

Why does it require the lambda? Couldn't the bound method p.kill be used without the lambda there?

// , Would you be willing to include an example of the use of this?

python - Using module 'subprocess' with timeout - Stack Overflow

python multithreading timeout subprocess
Rectangle 27 106

with
with timeout(seconds=3):
    time.sleep(4)

The code is still using signal and thus UNIX only:

import signal

class timeout:
    def __init__(self, seconds=1, error_message='Timeout'):
        self.seconds = seconds
        self.error_message = error_message
    def handle_timeout(self, signum, frame):
        raise TimeoutError(self.error_message)
    def __enter__(self):
        signal.signal(signal.SIGALRM, self.handle_timeout)
        signal.alarm(self.seconds)
    def __exit__(self, type, value, traceback):
        signal.alarm(0)

Python < v3 does not have a TimeoutError. But one can very easily write one own class with like explained here: stackoverflow.com/a/1319675/380038

You could easily add in a decorator @timeout.timeout as a static method to this. Then, you could easily choose between a decorator or a with statement.

Interesting to note that if inside the with Timeout(t) context any error is raised, the __exit__ is still called, avoiding, thus, any complication caused by TimeOutError being raised instead of the real error. This is a very lovable solution.

Can someone recommend a viable solution that, like this, that works in threads?

@Nick Some time ago I created version of timeout decorator, that works with floats - stackoverflow.com/questions/11901328/

python - Timeout function if it takes too long to finish - Stack Overf...

python
Rectangle 27 106

with
with timeout(seconds=3):
    time.sleep(4)

The code is still using signal and thus UNIX only:

import signal

class timeout:
    def __init__(self, seconds=1, error_message='Timeout'):
        self.seconds = seconds
        self.error_message = error_message
    def handle_timeout(self, signum, frame):
        raise TimeoutError(self.error_message)
    def __enter__(self):
        signal.signal(signal.SIGALRM, self.handle_timeout)
        signal.alarm(self.seconds)
    def __exit__(self, type, value, traceback):
        signal.alarm(0)

Python < v3 does not have a TimeoutError. But one can very easily write one own class with like explained here: stackoverflow.com/a/1319675/380038

You could easily add in a decorator @timeout.timeout as a static method to this. Then, you could easily choose between a decorator or a with statement.

Interesting to note that if inside the with Timeout(t) context any error is raised, the __exit__ is still called, avoiding, thus, any complication caused by TimeOutError being raised instead of the real error. This is a very lovable solution.

Can someone recommend a viable solution that, like this, that works in threads?

@Nick Some time ago I created version of timeout decorator, that works with floats - stackoverflow.com/questions/11901328/

python - Timeout function if it takes too long to finish - Stack Overf...

python