This describes tools and techniques that can identify memory leaks in Long running Python programs:

• Is it a Leak?
• Sources of Leaks
• A Bit About (C)Python Memory Management
  • Reference Counts
  • Garbage Collection
  • The Big Picture
  • CPython’s Object Allocator (pymalloc)

Here is a visualisation of memory allocators from top to bottom (from the Python source Objects/obmalloc.c):

    _____   ______   ______       ________
   [ int ] [ dict ] [ list ] ... [ string ]       Python core         |
+3 | <----- Object-specific memory -----> | <-- Non-object memory --> |
    _______________________________       |                           |
   [   Python's object allocator   ]      |                           |
+2 | ####### Object memory ####### | <------ Internal buffers ------> |
    ______________________________________________________________    |
   [          Python's raw memory allocator (PyMem_ API)          ]   |
+1 | <----- Python memory (under PyMem manager's control) ------> |   |
    __________________________________________________________________
   [    Underlying general-purpose allocator (ex: C library malloc)   ]
 0 | <------ Virtual memory allocated for the python process -------> |

   =========================================================================
    _______________________________________________________________________
   [                OS-specific Virtual Memory Manager (VMM)               ]
-1 | <--- Kernel dynamic storage allocation & management (page-based) ---> |
    __________________________________   __________________________________
   [                                  ] [                                  ]
-2 | <-- Physical memory: ROM/RAM --> | | <-- Secondary storage (swap) --> |

He found that the more HTTP client requests he did, the more memory his Node process would consume, but it was really slow.

[...] Then I ran Node with UMEM_DEBUG set to record various important information about the memory allocations

[...] Every hour, it grabbed the output of pmap -x and a core file and stored those in Joyent Manta

[...] In MDB there's a particularly helpful command ::findleaks that will show you the memory addresses and the stack traces for leaked memory, not unlike using valgrind, but without all the performance penalty.

[...] At this point we knew that we were looking for something in v0.10 that called MakeCallback but that didn't first have a HandleScope on the stack. I then worked up this simple DTrace script.

In this post, we will explore how Unix pipes are implemented in Linux by iteratively optimizing a test program that writes and reads data through a pipe.1

We will begin with a simple program with a throughput of around 3.5GiB/s, and improve its performance twentyfold.

The USE Method provides a strategy for performing a complete check of system health, identifying common bottlenecks and errors. For each system resource, metrics for utilization, saturation and errors are identified and checked. Any issues discovered are then investigated using further strategies.
This is an example USE-based metric list for Linux operating systems (eg, Ubuntu, CentOS, Fedora). This is primarily intended for system administrators of the physical systems, who are using command line tools. Some of these metrics can be found in remote monitoring tools.

TL;DR:

• List Resizing : the backing array is grown by approximately 12% (in Java ArrayList grows by 50% when expanded2 and in Ruby, Array grows by 100%). The Python implementation optimizes for memory usage over speed. Another reason to preallocate Python lists when possible.
• Inserting at the beginning of a list takes linear time. Sometimes, better use Deques which trade constant time insert and remove from both ends in exchange for constant time indexing.

By Russel Cohen

Nylas provides a modern developer
platform for email, contacts, and calendar. Stop fighting old protocols,
and start building great products

Disclaimer I am the creator and main developer of Brython. I am aware that this makes me suspect of partiality ! The test conditions are explained in detail below so that anyone can easily reproduce them and compare with the results presented here ; if something is wrong please post a comment and I will…

My name is Stephan, and I'm a scientist on the Climatology team at The Climate Corporation. We make extensive use of Python to build statistical weather models, and sometimes we need our code to be fast. Here's how I choose between Numba and Cython, two of the best options for accelerating numeric Python code. Most…

Python list implementation