In this post, we will explore how Unix pipes are implemented in Linux by iteratively optimizing a test program that writes and reads data through a pipe.1
We will begin with a simple program with a throughput of around 3.5GiB/s, and improve its performance twentyfold.
The USE Method provides a strategy for performing a complete check of system health, identifying common bottlenecks and errors. For each system resource, metrics for utilization, saturation and errors are identified and checked. Any issues discovered are then investigated using further strategies.
This is an example USE-based metric list for Linux operating systems (eg, Ubuntu, CentOS, Fedora). This is primarily intended for system administrators of the physical systems, who are using command line tools. Some of these metrics can be found in remote monitoring tools.
TL;DR:
- List Resizing : the backing array is grown by approximately 12% (in Java ArrayList grows by 50% when expanded2 and in Ruby, Array grows by 100%). The Python implementation optimizes for memory usage over speed. Another reason to preallocate Python lists when possible.
- Inserting at the beginning of a list takes linear time. Sometimes, better use Deques which trade constant time insert and remove from both ends in exchange for constant time indexing.
By Russel Cohen
Nylas provides a modern developer
platform for email, contacts, and calendar. Stop fighting old protocols,
and start building great products
Disclaimer I am the creator and main developer of Brython. I am aware that this makes me suspect of partiality ! The test conditions are explained in detail below so that anyone can easily reproduce them and compare with the results presented here ; if something is wrong please post a comment and I will…
My name is Stephan, and I'm a scientist on the Climatology team at The Climate Corporation. We make extensive use of Python to build statistical weather models, and sometimes we need our code to be fast. Here's how I choose between Numba and Cython, two of the best options for accelerating numeric Python code. Most…
Python list implementation