Skip to main content

Python in Parallel: Sub-Interpreters vs. NoGIL vs. Multiprocessing

Track:
Python Internals & Ecosystem
Type:
Talk
Level:
intermediate
Room:
Forum Hall
Start:
14:00 on 12 July 2024
Duration:
30 minutes

Abstract

In the realm of Python development, achieving parallelism and harnessing the full power of modern multi-core processors is challenging. Traditionally constrained by the Global Interpreter Lock (GIL), multi-threading was not useful for true parallelism in Python, hence developers turned into multi-processing. Now this is all changing with upcoming developments.

  • A new per-Interpreter GIL in Python 3.12 helps to facilitate true parallelism by opening up sub interpreters to run Python codes at the same time. In addition, it is going to be easier to use these features in Python 3.13 by having support for sub interpreters in the Stdlib.
  • An upcoming compile option to disable-gil is opening a path to a world with NoGIL where Python threads can truly run in parallel. Which can potentially become the default in the upcoming years. Multiprocessing, sub-interpreters and multi-threading without GIL are all different ways of facilitating multi-core parallelism in Python. Each of these approaches offers a unique pathway to parallel execution, but comes with its own set of trade-offs, complexities, and suitability for different types of problems. During this talk we will explore each of these options and asses their pros and cons.

Whether you’re building CPU-bound high throughput applications, IO-bound services, or simply curious about the future of parallel processing in Python, this talk will help you with the knowledge to make informed decisions and leverage Python’s parallel computing capabilities.


The speaker

Samet Yaslan

Samet Yaslan

Samet has been chairing the Python Standards Group at Optiver since 2021, with the mission of standardizing Python codebases by making best practices known to everyone in the company. He advocates for clean, simple, and well-tested code. He has earned his living using Python for more than 10 years and began participating in the open-source CPython project last year. He continues to contribute with new pull requests occasionally and really enjoys the opportunity to give back!