A bit of advertising: in-person Python course in Madrid, with Securízame
Thanks to Securízame, and together with four colleagues, we are going to teach a complete Python course. It starts with an introduction to the language, goes on with a specific part for pentesters and another for sysadmins, and ends with the advanced part. That last one is mine.
The full syllabus is on the Securízame website:
If you want to get into Python, it would not hurt to have someone give you a hand and guide you while you learn, right?
Have a look at the syllabus. Once you see it I will not need to tell you anything else. You will convince yourself.
Advanced Python course
As I said, I get the advanced part. What is it about? It focuses on four points:
- Structuring and managing projects efficiently.
- Improving and increasing performance.
- Concurrency and load distribution.
- Deploying projects and using Docker with Python.
The full, extended syllabus is here:
https://cursos.securizame.com/python-avanzado/
A small taste
To whet your appetite, here is a tiny preview of what we will cover in the course.
Python and its nonexistent parallelism
I assume you know what threads are. They exist in almost every programming language, and Python, of course, also lets us use threads. Or does it?
What do you mean no, Python has the threading library, and it is there to create threads.
The library exists, yes. But it is one of Python’s big little lies.
Why Python lies to us, what did I ever do to it?
A bit of background.
When Guido van Rossum, the creator of Python, designed the language, he had one thing very clear: it had to be simple. Not only the language, the interpreter too.
With that idea in mind, he built the interpreter without thread support. Over the years, programmers asked for that support more and more.
But adding threads meant complicating the interpreter. A lot. Why? Because parallelism brings, among many other problems to solve:
- Synchronization.
- Race conditions.
- Deadlocks.
- Memory leaks.
Python and its fake threads
Even so, programmers kept asking for threads. They solved it like this:
- They created the GIL (Global Interpreter Lock).
- They created a library that gives the programmer an API to create and manage threads.
OK, and what is this thing?
The GIL
We already said it: adding real threads takes a lot of effort and complicates the interpreter. What the GIL does is simulate thread execution. That is:
- Python creates one process and one thread by default. Any program we write runs in that context.
- If we create more threads, Python and the GIL keep switching which thread runs on the processor, so that only one thread runs at a time. We get concurrency, but not parallelism, which is what intuition tells us.
What a load of rubbish, right? When you find this out for the first time you feel pretty let down. At least I was left thinking: what kind of crap language is this.
Keep in mind what I said above: Python was created to be simple, in every respect. That does not have to be bad. It depends on what we need. For example, I would never write an operating system or the control system of a missile in Python.
Despite the GIL, you can do most of whatever crosses your mind with Python. No need to worry.
Surviving the GIL
OK, we now know Python has no real threads. So what do we do if we need more performance? Well, we have plenty of options:
- Multiprocessing.
- Lightweight threads.
- Coroutines.
- Distributed processing.
- JIT (Just In Time) techniques.
- Compiling all or part of our Python code.
- Using other, “unofficial” Python interpreters.
- Using C for the critical parts, with a Python wrapper.
- Turning input/output into event-driven I/O.
- And so on.
There are options. It all depends on the kind of problem in front of us: heavy network usage, compute load…
In the course we will study the points in bold. Yes, it is intense. But do not worry, explained properly it is not that hard.
The easiest approach, especially when it comes to understanding the concept, is the first one: multiprocessing.
It is as simple as this: instead of creating threads, we create processes. That is it. Processes run on different cores of the processor, and with that we get real parallelism.
Careful, this is no silver bullet. It has plenty of problems and it does not work for everything. In the course we will go over its pros and cons.
Lastly: a curious case
Since threads in Python do not exist the way we understand them in other languages, curious things happen. This one, for instance: when running with threads is slower than running without them.
Take the following example: a producer that generates numbers from 1 to 1,000,000 and one or more consumers that work with those numbers.
The code without threads:
def countdown(n):
while n > 0:
n -= 1
def main():
countdown(100000000)
if __name__ == '__main__':
main()
The code with threads. We split the work across four threads:
from threading import Thread
def countdown(n):
while n > 0:
n -= 1
def main():
COUNT = 100000000
t1 = Thread(target=countdown, args=(COUNT//4, ))
t2 = Thread(target=countdown, args=(COUNT//4, ))
t3 = Thread(target=countdown, args=(COUNT//4, ))
t4 = Thread(target=countdown, args=(COUNT//4, ))
t1.start(); t2.start(); t3.start(); t4.start()
t1.join(); t2.join(); t3.join(); t4.join()
if __name__ == '__main__':
main()
If we run each one, these are the timings:
# python count.py
real 0m6.820s
user 0m6.585s
sys 0m0.079s
# python threads-count.py
real 0m15.583s
user 0m10.851s
sys 0m14.071s
Wow. The run without threads was faster. Why? Without going into much detail (I will leave that for another post), this is what happened:
- Python simulates concurrency, as we already know.
- Python keeps swapping thread execution so that only one runs at a time.
- That context switch, in certain situations, costs more than the work the thread itself is going to do.
- Therefore, the single-threaded solution is faster: it wastes no time on context switches, and the work each thread does costs less than the switch itself.
For the Python experts reading this: I know I am leaving out a lot of details and that this is a very simplified explanation.
Conclusion
Concurrency and parallelism are complicated in any programming language, although also very interesting. In Python, on top of that, they can be a real challenge.
In the next post I will tell you more about the dark side of concurrency in Python.
See you in June at Securízame.
