Rectangle 27 5

Think of it like this. When you need to make a request to say a database or another url to retrieve data you do not want to block your tornado IO. So the @tornado.web.asynchronous will allow the IO to handle other requests while it waits for the content to load (ex. database or url).

@tornado.web.asynchronous

python - what does @tornado.web.asynchronous decorator mean? - Stack O...

python tornado
Rectangle 27 1

So, WSGI protocol (and so Django) uses syncronous model. It means that when your app starts processing a request it takes control and gives it back only when request is finished. That's why it can process single request at once. To allow simultaneous requests one usually launches wsgi application in multithreaded or multiprocessed mode.

The Tornado server on other side uses asynchronous model. The idea here is to have own scheduler instead of OS scheduler that works with threads and processes. So your code runs some logic, then launches some long task (DB call, URL fetch), sets up what to run when task finishes and gives control back to scheduler.

Giving controll back to scheduler is crucial part, it allows async server to work fast because it can start processing new request while previous is waiting for data.

This answer explains sync/async detailed. It focuses on client, but I think you can see the idea.

So whats wrong with your code: Popen does not give control to IOLoop. Python does nothing until your subprocess is finished, and so can not process other requests, even not Django's requests. runserver "works" here because it's multithreaded. So while locking entirely the thread, other threads can still process requests.

For this reason it's usually not recommended to run WSGI apps under async server like tornado. The doc claims it will be less scalable, but you can see the problem on your own code. So if you need both servers (e.g. Tornado for sockets and Django for main site), I'd suggest to run both behind nginx, and use uwsgi or gunicorn to run Django. Or take a look at django-channels app instead of tornado.

Besides, while it works on test environment, I guess it's not a recomended way to do what you try to achieve. It's hard to suggest the solution, as I don't know what do you call with Popen, but it seams to be something long running. Maybe you should take a look at Celery project. It's a package for running long-term background job.

However, back to running sub-processes. In Tornado you can use tornado.process.Subprocess. It's a wrapper over Popen to allow it to work with IOLoop. Unfortunately I don't know if you can use it in wsgi part under tornado. There are some projects I remember, like django futures but it seems to be abandoned.

As another quick and dirty fix - you can run Tornado with several processes. Check this example on how to fork server. But I will not recoment using this in production anyway (fork is OK, running wsgi fallback is not).

  • Run the Popen call in some background queue, like Celery
  • Process such views with Tornado and use tornado.processes module to run subprocess.

And overall, I'd seek for another deployment infrastructure, and would not run Durango under tornado.

Thank you @lgor! The explanation is very clear. Do you have any suggestions about what server I can use that would work well with Django for my purpose? The reason I need popen is that I have some legacy stuff that I have to call subprocess to run.

I would use uwsgi-docs.readthedocs.io/en/latest to run Django app, perhaps behind Nginx. You can either run some amount of workers (threads x processes) and hope that such Popen-blocks are OK (they're OK, but it will handle small load, bigger amount of users can make server stuck). If you need higher load - check Celery. You will run a task, get task-id immediately, and you can return ID to client (browser). Browser will poll when task is ready and can load task results. But you will need to update both django view and template (some ajax polling)

python - Tornado server caused Django unable to handle concurrent requ...

python django web concurrency tornado
Rectangle 27 0

class Handler(tornado.web.RequestHandler):
    def get(self):
        self.write('response')
        self.finish() # Connection is now closed
        foo()

Yes, if foo() does blocking IO. No, if foo() does not. Rule of thumb: if foo() uses IOStream then it's non-blocking. If it uses sockets without IOStream, then it is blocking.

python - Tornado process data in request handler after return - Stack ...

python tornado
Rectangle 27 0

Tornado isn't designed to do authentication and other operations in RequestHandler.__init__. That's why you get an exception when you call self.finish from __init__: the RequestHandler isn't ready to run finish yet.

get_current_user()

In your get_current_user(), don't set self.user and self.private, just return a tuple. Something like this:

def get_current_user(self):
    private = -1
    user = self.get_secure_cookie("user")
    if user:
        private = self.UserModel.get_user_level_by_name(self.user)

    return (user, private) if private == 4 else None

Tornado will handle the rest. In your get() method, the current (user, private) tuple will be set in self.current_user.

python - How does tornado stop current request handler? - Stack Overfl...

python tornado
Rectangle 27 0

In general, the best way to record something like this is to attach it to the RequestHandler instance. For logging, you can override Application.log_request, which has access to the handler. You may need to pass either the request ID or the RequestHandler instance around to other parts of your code if you need to use this in multiple places.

It is possible but tricky to use StackContext as something like a threading.Local if you really need this to be pervasive, but explicitly passing it around is probably best.

python - assign process id to an asynchronous request tornado - Stack ...

python tornado
Rectangle 27 0

Say if you are using jQuery to send this PUT request:

$.ajax({
    type: "PUT",
    url: "/yourURL",
    data: JSON.stringify({'json':'your json here'),
    dataType: 'json'
})

The data should not be like: data: {'json': 'your json here'}, because it will automatically be encoded into query string, which needs to be parsed by parse_qs

Then in Tornado

def put(self, pid):
    d = json.loads(self.request.body)
    print d

python - Tornado RESTful PUT handler method not getting request argume...

python rest tornado
Rectangle 27 0

I would use the delete method just like you'd use anything. Set up handler with the delete method (probably specifying and id) then delete that record if it exist and if user has appropriate permissions. I've never used Tornado but perhaps something like:

class WidgetHandler(RequestHandler):
  def initialize(self, database):
    self.database = database    

  def delete(self, id):
    # if user has permission to delete
        # if record exists
        # delete it

app = Application([
  (r'/widgets/([0-9]+)', WidgetHandler, dict(database=database)),
  ])

If you are using javascript/jquery/ajax on the front to make the request the jquery ajax supports delete:

// get your id somehow
// dynamically build url   
$.ajax({
    url: '/widgets/' + theIdYouGotSomehow,
    type: 'DELETE',
    success: function(result) {
        // Do something with the result
    }
});

For your comment, you can just hit the url with the delete method. /widgets/12

but they [delete, put, etc] are not supported by all browsers.

Some stackoverflow posts suggest all modern browsers support the delete method.

thanks. Two questions on the front-end part: 1. how do I specify the id to be deleted in the JQuery ajax function? 2. is 'DELETE' supported by all the browsers?

python - Delete data using a POST request in Tornado? - Stack Overflow

python tornado
Rectangle 27 0

If you want to match ,,, you need to match the inverse of a-zA-Z0-9, which would be [^a-zA-Z0-9]. Seeing as how you used \w prior, you may aswell use \W which is the same as [^\w].

python - Tornado request handler mapping to international characters -...

python regex unicode tornado
Rectangle 27 0

Tornado isn't designed to do authentication and other operations in RequestHandler.__init__. That's why you get an exception when you call self.finish from __init__: the RequestHandler isn't ready to run finish yet.

get_current_user()

In your get_current_user(), don't set self.user and self.private, just return a tuple. Something like this:

def get_current_user(self):
    private = -1
    user = self.get_secure_cookie("user")
    if user:
        private = self.UserModel.get_user_level_by_name(self.user)

    return (user, private) if private == 4 else None

Tornado will handle the rest. In your get() method, the current (user, private) tuple will be set in self.current_user.

python - How does tornado stop current request handler? - Stack Overfl...

python tornado
Rectangle 27 0

If you're starting the connection from the javascript console, the browser doesn't have a chance to show you the self-signed certificate warning and give you a chance to accept it. If you go to https://localhost:9001 first and accept the certificate there, does it work?

python - Tornado WebSocketHandler won't respond to SSL request - Stack...

python ssl websocket tornado
Rectangle 27 0

I had a similar handler and the self.finish() will trigger the response back to the client. So if you move that line to above your p.apply_async it ought to work as you intend.

python - Processing a long request in Tornado never finishes - Stack O...

python tornado
Rectangle 27 0

AFAIK, on_connection_close is only called only when the client terminates the connection, which may explain your problem. Regarding threading, I don't know what you want to do, but I can't see why you would want to create a thread in a Tornado request as one of the advantages of Tornado is exactly that you don't have to use threading. If I were to add a join to your example I would put it just before self.finish(), however, you can probably just omit it... that will depend on what you want to do with the thread, but remember that Tornado is single-threaded and the whole process will block if the thread is not finished by the time join() comes.

I think you're right re: when on_connection_close called. Been doing some experimentation and it also seems to vary a bit whether ioloop is using epoll or select(). Currently wondering if I should do the join() in a callback I attach via add_callback(). As per threads, I'm using them to perform a bit of work in a 'fire and forget' way. Basically, I'd like to accept the request, send back some ACK and have that work get performed. If I wait until the work is complete before sending back a response, the 'client' will have timed out.

python - Where to join threads created in an asynchronous tornado requ...

python multithreading join tornado
Rectangle 27 0

A single MySQL connection for the whole application is the correct design here. If there is a network error, one request will get an exception, but the MySQL connection will reconnect for the next request and your application will recover. Built-in connection pooling allows your application to have several sockets connected to MySQL in order to await several MySQL operations at once.

If you create a new MySQL connection in each request handler you'll suffer the cost of setting up a TCP socket to MySQL for each HTTP request, as well as several network round-trips to log in to MySQL. Your performance will be much, much worse. If you measure it I expect you'll see that your request latency suffers badly if you create a new MySQL connection in each request handler.

Hi,Jesse,Thank you very much for your help! You are right, and when I review the source code of tornado framework, I find it has realized the automatic connecting mechanism to Mysql.

In a web request handler of python tornado, should a global connection...

python mysql web tornado
Rectangle 27 0

put handler will parse request.body, if request had proper content-type header (application/x-www-form-urlencoded), for example if you are using tornado http client:

headers = HTTPHeaders({'content-type': 'application/x-www-form-urlencoded'})
http_client.fetch(
      HTTPRequest(url, 'PUT', body=urllib.urlencode(body), headers=headers))

python - Tornado RESTful PUT handler method not getting request argume...

python rest tornado
Rectangle 27 0

so; Decorator you can use. create

# decorators.py
def is_ajax(method):

    @wraps(method)
    def wrapper(self, *args, **kwargs):
        if "X-Requested-With" in self.request.headers:
            if self.request.headers['X-Requested-With'] == "XMLHttpRequest":
                return method(self, *args, **kwargs)

        else:                                                                                                                                                                 
            self.redirect("/")                                                     

    return wrapper

and

from tornado.web import RequestHandler
from decorators import is_ajax


class MyHandler(RequestHandler):

    @is_ajax # is_ajax decorators.
    def get(self):
        self.write("ok!")
if self.request.headers.get('X-Requested-With') == "XMLHttpRequest":

python - How to run ajax request handler tornado - Stack Overflow

python tornado
Rectangle 27 0

The difference between a request for an HTTP page and a websocket is the presence of a header. Unfortunately afaik there is no way to tell the tornado router to choose one or the other handler based on a header (other than host).

I can however make a handler that inherits from both my already quite elaborate MyBaseRequestHandler and WebSocketHandler, with some magic. The following code works in python 3.5 and tornado 4.3, your mileage may vary on other versions:

class WebSocketHandlerMixin(tornado.websocket.WebSocketHandler):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        # since my parent doesn't keep calling the super() constructor,
        # I need to do it myself
        bases = type(self).__bases__
        assert WebSocketHandlerMixin in bases
        meindex = bases.index(WebSocketHandlerMixin)
        try:
            nextparent = bases[meindex + 1]
        except IndexError:
            raise Exception("WebSocketHandlerMixin should be followed "
                            "by another parent to make sense")

        # undisallow methods --- t.ws.WebSocketHandler disallows methods,
        # we need to re-enable these methods
        def wrapper(method):
            def undisallow(*args2, **kwargs2):
                getattr(nextparent, method)(self, *args2, **kwargs2)
            return undisallow

        for method in ["write", "redirect", "set_header", "set_cookie",
                       "set_status", "flush", "finish"]:
            setattr(self, method, wrapper(method))
        nextparent.__init__(self, *args, **kwargs)

    async def get(self, *args, **kwargs):
        if self.request.headers.get("Upgrade", "").lower() != 'websocket':
            return await self.http_get(*args, **kwargs)
        # super get is not async
        super().get(*args, **kwargs)


class MyDualUseHandler(WebSocketHandlerMixin, MyBaseHandler):
    def http_get():
        self.write("My HTTP page")

    def open(self, *args, **kwargs):
        self.write_message("Hello WS buddy")

    def on_message(self, message):
        self.write_message("I hear you: %s" % message)

python - How can I serve a HTTP page and a websocket on the same url i...

python websocket tornado
Rectangle 27 0

without more details it's hard to answer. It sounds like an issue using global variables or something like that. A good way to help debug this type of issue is to add logging statements before and after you modify data at each step.

run your tornado app with --logging=debug on the command line to enable DEBUG level logging. Then add logging statements to help track down where the extra data is added.

import logging
...
def on_response(self, response):
    logging.debug(response)
    self.write(response.body)

Started logging some of the data, and found out that, bizarrely, it was using an existing instance of a class I had (part of the RSS handler). Strange! But, thanks to your answer I fixed it. Appreciate it!

python - Tornado: Make Blocking HTTP Request in Async Handler - Stack ...

python asynchronous tornado
Rectangle 27 0

get_cache should return a value that indicates whether it finished the request or not (or it should return the cached data and leave it to the caller to finish the request). I would do one of the following:

@gen.coroutine
def serve_from_cache(self):
  response = yield gen.Task(get_redis)
  if response:
      self.finish(response)
      raise gen.Return(True)
  else:
      raise gen.Return(False)

@gen.coroutine
def get(self):
  if (yield self.serve_from_cache()):
    return
  # do work
  yield self.set_cache(...)
@gen.coroutine
def get_cache(self):
    return yield gen.Task(get_redis)


@gen.coroutine
def get(self):
    resp = yield self.get_cache()
    if resp:
      self.finish(resp)
      return
    # do work...

python - Refactoring a Tornado Request Handler - Stack Overflow

python tornado yield
Rectangle 27 0

As a work-around, I found them in the request.body and parsed the encoded parameters manually. It was kindof annoying, but it works.

new_values = urlparse.parse_qs(self.request.body)

# values show as lists with only one item
for k in new_values:                             
    new_values[k] = new_values[k][0]

python - Tornado RESTful PUT handler method not getting request argume...

python rest tornado
Rectangle 27 0

Like normal RequestHandler instances, WebsocketHandler instances have a HTTPServerRequest object set to the request attribute of the Handler. You can use the HTTPServerRequest.remote_ip attribute to get the IP of the remote connection. For example:

class EchoWebSocket(websocket.WebSocketHandler):
    def initialize(self):
        self._closed = False

    def open(self):
        print(type(self.request))
        print(self.request)
        print(self.request.remote_ip)
<class 'tornado.httputil.HTTPServerRequest'>
HTTPServerRequest(protocol='http', host='localhost:8888', method='GET', uri='/ws', version='HTTP/1.1', remote_ip='::1', headers={'Connection': 'Upgrade', 'Upgrade': 'websocket', 'Accept-Encoding': 'gzip', 'Sec-Websocket-Version': '13', 'Host': 'localhost:8888', 'Sec-Websocket-Key': 'oAJpF4f4kp26b2KRjYmRGw=='})
::1

when I connect from external of my network, i.e. 3g it shows the ip of my modem instead, how could i show the true originating ip please

@Ossama The router is acting like a reverse proxy in this scenario - the client is connecting to your Public IP (which is your router) and the router is forwarding the request on to your local IP (192.168.x.x for example). If the router isn't including the real client IP in the headers its passing to tornado, it may not be possible for you to get the real client IP. You could try using the xheaders option and see if your router includes X-Real-IP/X-Forwarded-For.

how can i get this? my header is as follow,

HTTPServerRequest(protocol='http', host='192.168.0.6:8888', method='GET', uri='/ws', version='HTTP/1.1', remote_ip='192.168.0.3', headers={'Origin': '192.168.0.6';, 'Upgrade': 'websocket', 'Sec-Websocket-Extensions': 'x-webkit-deflate-frame', 'Sec-Websocket-Version': '13', 'Connection': 'Upgrade', 'Sec-Websocket-Key': 'klB25dpHtCvRri6UXqKxEw==', 'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 8_0_2 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12A405 Safari/600.1.4', 'Host': '192.168.0.6:8888', 'Pragma': 'no-cache', 'Cache-Control': 'no-cache'})

python - How do I get the client IP of a Tornado web socket request? -...

python websocket ip tornado