Twisted Mixing

lvh

[email protected]

Introduction

Rackspace

rackspace.svg

Crypto 101

crypto101.png

Twisted

twisted.svg

About this talk

Twisted code ↔ other code

Why Twisted?

Prerequisites

Python, some Twisted probably helps

Introducing Twisted

Reactor

An object that reacts to events

IReactor{}

  • Time: callLater…
  • Process: spawnProcess
  • Threads: call(In|From)Thread, …
  • (TCP|UDP|SSL|Multicast)
  • (UNIX|UNIXDatagram|Socket)
  • FDSet: (add|remove)(Reader|Writer), …

Deferred

An object you get now,

gets you result or failure later

Blocking API

try:
    result = blocking_read()
except SomeError as e:
    on_failure(e)
else:
    on_result(result)

Deferred API

d = async_read()
d.addCallbacks(on_result, on_failure)

Inline callbacks

try:
    result = yield async_read()
except SomeError as e:
    on_failure(e)
else:
    on_result(result)

Twisted and your app

SOA

Service Oriented Architecture

WSGI

Web Server Gateway Interface

twistd web --wsgi=wsgi.app

Demo

  • Flask app, served by t.w.wsgi
  • Real-time chat, with txsockjs

Test run, demo

Blocking code in Twisted

Can’t block reactor

Blocking IO

def _getDataAtURL(url):
    return requests.get(url) # BLOCKS!

Blocking computation

def _compute(n):
    x = 2
    for _ in xrange(n): # BLOCKS!
        x *= x
    send_somewhere(x)

Can’t block reactor

Alternatives:

  1. Don’t block
  2. Block another thread

Don’t block

IO bound? Asynchronous IO!

CPU bound? Cooperate!

Asynchronous I/O!

treq: requests-like, but asynchronous 

def _getDataAtURL(url):
    return treq.get(url)

Cooperation!

t.internet.task.coiterate & friends 

def _compute(n):
    x = 2
    for _ in xrange(n):
        x *= x
        yield # Yields to the reactor :)
    send_somewhere(x)

coiterate(_compute(n))

Don’t block?

Avoiding blocking isn’t always possible

  • Blocking API: DBAPI2, WSGI…
  • Opaque code: scrypt, Pillow…
  • Kernel/syscall level: file IO, …

Block somewhere else

Can’t block reactor thread

→ block a different one!

  • … in the same process: deferToThread
  • … in a child process: spawnProcess…
  • … in a remote process: Ampoule, RPC…

deferToThread

  • Easy automagic deferreds!
  • Shared mutable state :-(

Twisted in blocking code

itamarst/crochet

setup()

@run_in_reactor

  • Run in reactor thread
  • Return EventualResult

EventualResult?

  • Synchronous analog of Deferred
  • wait(timeout=None)

Example #1

from twisted.web.client import getPage
from crochet import setup, run_in_reactor
setup()

@run_in_reactor
def download_page(url):
    return getPage(url)

url = "http://tm.tl/5000"
result = download_page(url)
print result.wait()

Example #2

  • Twisted queries exchange rate every 30s
  • Flask app serves the latest exchange rate

Demo

Twisted part

class ExchangeRate(object):
    # ...

    @run_in_reactor
    def start(self):
        self._lc = LoopingCall(self._download)
        self._lc.start(30, now=True)

    def _download(self):
        d = getPage(url)
        # ...

Flask part

@app.route('/')
def index():
    rate = EURUSD.latest_value()
    if rate is None:
        rate = "unavailable"
    return "EUR/USD rate: {0}.".format(rate)

app.run()

Twisted in gevent

jyio/geventreactor

install()

“Blocking” code

gevent-style automagic suspending

Deferreds ↔ greenlets

r = waitForDeferred(d)

d = waitForGreenlet(g)

Demo

TODO

Recap

Twisted plays well with others

  • Many supported protocols
  • Cooperates with blocking code
  • Cooperates with other event loops

Conclusion

If you want to use Twisted, you probably can.

That doesn’t necessarily mean it’s a good idea.

Although it obviously is ;-)

Questions?