There are plenty of libraries for measuring how fast a piece of Python runs. They are called profilers, and the standard library ships a few of its own:
timeitprofilecProfile
They work, but they are a bit clumsy to use, and none of them was designed to live inside your production code.
So today I want to show you a tiny library I wrote for timing specific chunks of Python with as little ceremony as possible.
Say we have this code:
# file: bottleneck_functions.py
import time
def bottleneck_1():
r = 0
for x in range(4000):
r += ((x*x) * x) / 1000
def bottleneck_2():
for x in range(5000):
time.sleep(0.02)
def bottleneck_3():
for x in range(10000):
time.sleep(0.001)
def main():
bottleneck_1()
bottleneck_2()
bottleneck_3()
After running it we still have no idea which of the three is slowing the program down.
As I said, you could reach for timeit:
import timeit
if __name__ == '__main__':
print(timeit.timeit("bottleneck_1()", setup="from bottleneck_functions import bottleneck_1"))
print(timeit.timeit("bottleneck_2()", setup="from bottleneck_functions import bottleneck_2"))
print(timeit.timeit("bottleneck_3()", setup="from bottleneck_functions import bottleneck_3"))
It is not intuitive, and in a real application it gets awkward fast.
Now picture a small web application that calls those three functions, and suppose that:
-
We want to measure how long each one takes.
-
We do not want to change the source code depending on the environment the app runs in.
-
We only want the measuring to happen in development and staging.
Got it? Good. This is the web app:
# file: web_app.py
from flask import Flask
from bottleneck_functions import *
app = Flask(__name__)
@app.route("/", methods=["GET"])
def home():
bottleneck_1()
bottleneck_2()
bottleneck_3()
return "Ok!"
app.run()
As it is, there is no way to time each function and spot the bottleneck.
This is where python-performance-tools comes in:
> pip install python-performance-tools
# file: web_app_profiling.py
from flask import Flask
from performance_tools import *
from bottleneck_functions import *
app = Flask(__name__)
@app.route("/", methods=["GET"])
def home():
with catch_time("bottleneck function 1"):
bottleneck_1()
with catch_time("bottleneck function 2"):
bottleneck_2()
with catch_time("bottleneck function 3"):
bottleneck_3()
return "Ok!"
app.run()
Run the app, open http://127.0.0.1:5000 and you will see something like this:
> python web_app_profiling.py
[.. OMITTED ... ]
Time: 0.035312797 :: bottleneck function 1
Time: 5.018828881 :: bottleneck function 2
Time: 12.981881883 :: bottleneck function 3
Now we know how long each function takes.
But remember, we only wanted this in development and staging. catch_time takes an optional second argument: an activation function. Any function that returns a boolean will do, and the timer only runs when it returns True.
# file: web_app_profiling.py
import os
from flask import Flask
from performance_tools import *
from bottleneck_functions import *
app = Flask(__name__)
environment = os.environ.get("ENVIRONMENT", None)
def activate_function():
if environment.lower() in ("development", "staging"):
return True
else:
return False
@app.route("/", methods=["GET"])
def home():
with catch_time("bottleneck function 1", activate_function):
bottleneck_1()
with catch_time("bottleneck function 2", activate_function):
bottleneck_2()
with catch_time("bottleneck function 3", activate_function):
bottleneck_3()
return "Ok!"
app.run()
Let’s check that it works.
Enabling profiling
> export ENVIRONMENT=development
> python web_app_profiling.py
[.. OMITTED ... ]
Time: 0.035312797 :: bottleneck function 1
Time: 5.018828881 :: bottleneck function 2
Time: 12.981881883 :: bottleneck function 3
Disabling profiling
> export ENVIRONMENT=production
> python web_app_profiling.py
[.. OMITTED ... ]
[.. NO PROFILING INFORMATION ... ]
References
-
Python profilers in the standard library: https://docs.python.org/3/library/profile.html
-
Python
timeit: https://docs.python.org/3/library/timeit.html -
Python Performance Tools: https://github.com/cr0hn/python-performance-tools
