- Link to AWS big data project
- What is asyncio
- Python standard library for event driven programming in Python 3.4
- Allows multi-tasking with non-blocking I/O
- Python 3.5 adds language keywords to simplify programming
- Queues and Coroutines -- No threads!
- Queues hold units of work that need to be processed
- Coroutines represent tasks that are waiting on something
- While waiting coroutine suspends, allowing other tasks to continue
- Coroutines can wait for work to appear in a Queue
- Together, this paradigm supports a notion of cooperative multitasking,
where tasks voluntarily suspend when doing I/O or when waiting for more work
- As opposed to threads, which is known as pre-emptive multitasking, and means that
you have to be careful because your code could be interrupted as any time
- LDM processing
- Chunks flow through series of queues
- Diagram
- Threads actually used to increase CPU usage...
- ...but only for places doing file/network I/O where the Python GIL is released
- Lessons
- A single, multi-threaded Python process using event driven programming can handle entire
Level 2 Feed
- At times using almost 600% of one core on an 8 core machine.
- Different programming paradigm--good patterns for Python are still emerging
