18.5.7. Develop with asyncio — Python 3.4.0 documentation

### Navigation

- index
- modules |
- next |
- previous |
- Python »
- 3.4.0 Documentation »
- The Python Standard Library »
- 18. Interprocess Communication and Networking »
- 18.5. `asyncio` – Asynchronous I/O, event loop, coroutines and tasks »

# 18.5.7. Develop with asyncio

Asynchronous programming is different than classical “sequential” programming. This page lists common traps and explains how to avoid them.

## 18.5.7.1. Concurrency and multithreading

An event loop runs in a thread and executes all callbacks and tasks in the same thread. While a task is running in the event loop, no other task is running in the same thread. But when the task uses `yield from`, the task is suspended and the event loop executes the next task.

To schedule a callback from a different thread, the `BaseEventLoop.call_soon_threadsafe()` method should be used. Example to schedule a coroutine from a different thread:

    loop.call_soon_threadsafe(asyncio.async, coro_func())

Most asyncio objects are not thread safe. You should only worry if you access objects outside the event loop. For example, to cancel a future, don’t call directly its `Future.cancel()` method, but:

    loop.call_soon_threadsafe(fut.cancel)

To handle signals and to execute subprocesses, the event loop must be run in the main thread.

The `BaseEventLoop.run_in_executor()` method can be used with a thread pool executor to execute a callback in different thread to not block the thread of the event loop.

See also

See the Synchronization primitives section to synchronize tasks.

## 18.5.7.2. Handle blocking functions correctly

Blocking functions should not be called directly. For example, if a function blocks for 1 second, other tasks are delayed by 1 second which can have an important impact on reactivity.

For networking and subprocesses, the `asyncio` module provides high-level APIs like protocols.

An executor can be used to run a task in a different thread or even in a different process, to not block the thread of the event loop. See the `BaseEventLoop.run_in_executor()` method.

See also

The Delayed calls section details how the event loop handles time.

## 18.5.7.3. Logging

The `asyncio` module logs information with the `logging` module in the logger `'asyncio'`.

## 18.5.7.4. Detect coroutine objects never scheduled

When a coroutine function is called but not passed to `async()` or to the `Task` constructor, it is not scheduled and it is probably a bug.

To detect such bug, set the environment variable `PYTHONASYNCIODEBUG` to `1`. When the coroutine object is destroyed by the garbage collector, a log will be emitted with the traceback where the coroutine function was called. See the asyncio logger.

The debug flag changes the behaviour of the `coroutine()` decorator. The debug flag value is only used when then coroutine function is defined, not when it is called. Coroutine functions defined before the debug flag is set to `True` will not be tracked. For example, it is not possible to debug coroutines defined in the `asyncio` module, because the module must be imported before the flag value can be changed.

Example with the bug:

    import asyncio

    @asyncio.coroutine
    def test():
        print("never scheduled")

    test()

Output in debug mode:

    Coroutine 'test' defined at test.py:4 was never yielded from

The fix is to call the `async()` function or create a `Task` object with this coroutine object.

## 18.5.7.5. Detect exceptions not consumed

Python usually calls `sys.displayhook()` on unhandled exceptions. If `Future.set_exception()` is called, but the exception is not consumed, `sys.displayhook()` is not called. Instead, a log is emitted when the future is deleted by the garbage collector, with the traceback where the exception was raised. See the asyncio logger.

Example of unhandled exception:

    import asyncio

    @asyncio.coroutine
    def bug():
        raise Exception("not consumed")

    loop = asyncio.get_event_loop()
    asyncio.async(bug())
    loop.run_forever()

Output:

    Future/Task exception was never retrieved:
    Traceback (most recent call last):
      File "/usr/lib/python3.4/asyncio/tasks.py", line 279, in _step
        result = next(coro)
      File "/usr/lib/python3.4/asyncio/tasks.py", line 80, in coro
        res = func(*args, **kw)
      File "test.py", line 5, in bug
        raise Exception("not consumed")
    Exception: not consumed

There are different options to fix this issue. The first option is to chain to coroutine in another coroutine and use classic try/except:

    @asyncio.coroutine
    def handle_exception():
        try:
            yield from bug()
        except Exception:
            print("exception consumed")

    loop = asyncio.get_event_loop()
    asyncio.async(handle_exception())
    loop.run_forever()

Another option is to use the `BaseEventLoop.run_until_complete()` function:

    task = asyncio.async(bug())
    try:
        loop.run_until_complete(task)
    except Exception:
        print("exception consumed")

See also the `Future.exception()` method.

## 18.5.7.6. Chain coroutines correctly

When a coroutine function calls other coroutine functions and tasks, they should be chained explicitly with `yield from`. Otherwise, the execution is not guaranteed to be sequential.

Example with different bugs using `asyncio.sleep()` to simulate slow operations:

    import asyncio

    @asyncio.coroutine
    def create():
        yield from asyncio.sleep(3.0)
        print("(1) create file")

    @asyncio.coroutine
    def write():
        yield from asyncio.sleep(1.0)
        print("(2) write into file")

    @asyncio.coroutine
    def close():
        print("(3) close file")

    @asyncio.coroutine
    def test():
        asyncio.async(create())
        asyncio.async(write())
        asyncio.async(close())
        yield from asyncio.sleep(2.0)
        loop.stop()

    loop = asyncio.get_event_loop()
    asyncio.async(test())
    loop.run_forever()
    print("Pending tasks at exit: %s" % asyncio.Task.all_tasks(loop))
    loop.close()

Expected output:

    (1) create file
    (2) write into file
    (3) close file
    Pending tasks at exit: set()

Actual output:

    (3) close file
    (2) write into file
    Pending tasks at exit: {Task(<create>)<PENDING>}

The loop stopped before the `create()` finished, `close()` has been called before `write()`, whereas coroutine functions were called in this order: `create()`, `write()`, `close()`.

To fix the example, tasks must be marked with `yield from`:

    @asyncio.coroutine
    def test():
        yield from asyncio.async(create())
        yield from asyncio.async(write())
        yield from asyncio.async(close())
        yield from asyncio.sleep(2.0)
        loop.stop()

Or without `asyncio.async()`:

    @asyncio.coroutine
    def test():
        yield from create()
        yield from write()
        yield from close()
        yield from asyncio.sleep(2.0)
        loop.stop()

### Navigation

- index
- modules |
- next |
- previous |
- Python »
- 3.4.0 Documentation »
- The Python Standard Library »
- 18. Interprocess Communication and Networking »
- 18.5. `asyncio` – Asynchronous I/O, event loop, coroutines and tasks »