Sooner or later you have to serialize data, and Python gives you more than one way to do it out of the box. The question is which one is more efficient.
JSON vs Pickle
The standard library ships with two serializers: JSON and Pickle. Before picking one there are a couple of things worth measuring:
- how long the serialization takes
- how much memory it uses
To answer that I wrote a small benchmark that runs both. It builds 10, 500 and 1000 random elements and measures those two numbers for each run.
I also threw UltraJSON into the comparison, a high performance JSON library for Python.
And for Pickle I tested the two latest protocol versions, 4 and 5.
Results
The raw output looks like this:
Number of elements: 10
-----------------------
Json Size: 0.02715015411376953 MB
Json time: 0.006659951999999997 sec
UJson time: 0.0030970319999999996 sec
Pickle Size Proto 5: 0.02531909942626953 MB
Pickle time Proto 5: 0.003745575000000001 sec
Pickle Size Proto 4: 0.02531909942626953 MB
Pickle time Proto 4: 0.0031195769999999984 sec
Number of elements: 500
------------------------
Json Size: 24.98400592803955 MB
Json time: 1.651241939 sec
UJson time: 0.6243056419999997 sec
Pickle Size Proto 5: 25.035669326782227 MB
Pickle time Proto 5: 0.38723978699999995 sec
Pickle Size Proto 4: 25.035669326782227 MB
Pickle time Proto 4: 0.38475265300000006 sec
Number of elements: 1000
-------------------------
Json Size: 98.13797855377197 MB
Json time: 6.275222810999999 sec
UJson time: 2.5652533450000004 sec
Pickle Size Proto 5: 98.24791717529297 MB
Pickle time Proto 5: 1.3807646590000004 sec
Pickle Size Proto 4: 98.24791717529297 MB
Pickle time Proto 4: 1.3946891620000006 sec
The same numbers as charts:
Comparing memory usage
There is no real difference between JSON, UJson, Pickle v5 and Pickle v4. They all take up pretty much the same space.
Comparing serializing time
Here the gap is obvious. Pickle wins, with UJson not far behind.
