gh-137026: Add an explainer guide for asyncio #137215
Conversation
anordin95
requested review from
1st1,
asvetlov,
kumaraditya303 and
willingc
as code owners
This comment was marked as resolved.
This comment was marked as resolved.
anordin95
changed the title
anordin95
changed the title
AA-Turner
changed the title
- Start sentences on new lines to minimize disruption of diffs.
…Which concurrency do I want" section.
There was a problem hiding this comment.
This looks good to me. I've left a few grammar fixes as suggestions, but I'm comfortable enough with this to put the first approval on here.
I haven't read too deep into some of the technical discussions about asyncio internals, but I'm personally fine with keeping terms like "queue" in here. I really think we should focus on creating a HOWTO that's clear and easy to understand for beginners, not something that documents the exact inner workings of asyncio in detail.
This is based on the concept of lies-to-children ("a simplified, and often technically incorrect, explanation of technical or complex subjects employed as a teaching method"). This tutorial is targeted towards beginners, so it should be totally OK to use some information that's technically incorrect in the pursuit of initial comprehension.
For example, most schools initially teach you the concept of planetary electron shells, despite the fact that there are more correct (and complicated) models in quantum mechanics. We should not teach the quantum mechanics of asyncio here. Instead, our goal should be to spark curiosity in readers, even if the information is not as technically correct.
Thanks for all your hard work @anordin95!
Co-authored-by: Peter Bierma <[email protected]>
Thanks @ZeroIntensity and @anordin95. HOWTO guides are not strictly targeted to beginners rather they are a deeper explanation of something. The name HOWTO comes from early Linux days as a deep dive into a particular area. OK, we are down to one key decision: the best mental model for an event loop. We've been discussing a) the queue and b) the execution of scheduled work.
From the event loop docs, the above defines work run by an event loop. I'm going to suggest that instead of sticking with "queue" which doesn't sit well with multiple asyncio maintainers that we explore some other terms: "to do list", "work checklist", "collection of work". Working from the conductor metaphor: Like a symphony conductor, the event loop orchestrates multiple performers (tasks). It knows when each section should play, pauses some while others perform, and coordinates the overall harmony of concurrent operations. |
This is a beautiful idea! But, I'm a bit wary of using it, since in a symphony many sections play truly concurrently. The trumpets, the drums & the flutes could all be going at once. I reworked the event loop paragraph a decent bit (see below). The idea of a queue is mentioned only once now and is explicitly called out as a rough analogy twice in the sentence . Otherwise, I refer to the event loop's jobs with a variety of similar but different terms to help reinforce and contextualize the idea. Alongside this change, my idea also entails removing any other references to a queue in the broader article. In more technical terms, the event loop contains a collection of jobs to be run. Some jobs are added directly by you, and some indirectly by |
|
Cool @anordin95! I like the last commit you pushed. 🎉 Can we continue throughout the doc to deemphasize queue? |
Yep! Was just taking a little break. All references to queue besides the one in that first analogy are now gone. |
|
@ZeroIntensity, @willingc: It should be noted, I just added a blurb about cpython/Doc/howto/a-conceptual-overview-of-asyncio.rst Lines 207 to 228 in 9d3828d |
| tree/main>`_ that inspired this HOWTO article, by Alexander Nordin. | ||
| * This in-depth `YouTube tutorial series <https://www.youtube.com/ | ||
| watch?v=Xbl7XjFYsN4&list=PLhNSoGM2ik6SIkVGXWBwerucXjgP1rHmB>`_ on | ||
| ``asyncio`` created by core Python team member, Łukasz Lang. |
There was a problem hiding this comment.
| ``asyncio`` created by core Python team member, Łukasz Lang. | |
| ``asyncio`` created by core Python team member, Łukasz Langa. |
| That example highlights how using only ``await coroutine`` could | ||
| unintentionally hog control from other tasks and effectively stall the event | ||
| loop. | ||
| :func:`asyncio.run` can help you detect such occurences with the ``debug=True`` |
There was a problem hiding this comment.
To clarify, the way I'd expect most developers to use this is not by modifying the code, but by running python in dev mode locally (-X dev or PYTHONDEVMODE=1). A link should take users to: https://docs.python.org/3/library/asyncio-dev.html#asyncio-debug-mode
| Once it pauses or completes, it returns control to the event loop. | ||
| The event loop will then select another job from its pool and invoke it. | ||
| You can *roughly* think of the collection of jobs as a queue: jobs are added and | ||
| then processed one at a time, generally (but not always) in order. |
There was a problem hiding this comment.
generally (but not always) in order.
I still find this bit awkward as it's again suggesting that order of execution is unpredictable. Can we say that the jobs are processed in the order that they were scheduled or something?
| Because the coroutine ``main()`` exits before awaiting the task and no other | ||
| references to the task are made, the task object ``hello_task`` *might* be |
There was a problem hiding this comment.
This may not be the best example, as the application is expected to close at the end of main(). So the task gets cancelled anyway. i.e. Even if I keep a reference, I can get the task to not complete:
>>> async def hello():
... await asyncio.sleep(1)
... print("BAR")
...
>>> async def main():
... global t
... t = asyncio.create_task(hello())
...
>>> asyncio.run(main())
>>> t
<Task cancelled name='Task-10' coro=<hello() done, defined at <stdin>:1>>
Explainer guide for asyncio
gh-137026: HOWTO article for asyncio, and a reference to it from the main page of the asyncio docs.
Hi!
I've used Python's
asyncioa couple times now, but never really felt confident in my mental model of how it fundamentally works and therefore how I can best leverage it. The official docs provide good documentation for each specific function in the package, but, in my opinion, are missing a cohesive overview of the systems design and architecture. Something that could help the user understand the why and how behind the recommended patterns. And a way to help the user make informed decisions about which tool in the asyncio toolkit they ought to grab, or to recognize when asyncio is the entirely wrong toolkit. I thought I'd take a stab at filling that gap and contributing back to a community that's given so much!📚 Documentation preview 📚: https://cpython-previews--137215.org.readthedocs.build/en/137215/howto/a-conceptual-overview-of-asyncio.html