This describes tools and techniques that can identify memory leaks in Long running Python programs:
- Is it a Leak?
- Sources of Leaks
- A Bit About (C)Python Memory Management
- Reference Counts
- Garbage Collection
- The Big Picture
- CPython’s Object Allocator (pymalloc)
Here is a visualisation of memory allocators from top to bottom (from the Python source Objects/obmalloc.c):
_____ ______ ______ ________
[ int ] [ dict ] [ list ] ... [ string ] Python core |
+3 | <----- Object-specific memory -----> | <-- Non-object memory --> |
_______________________________ | |
[ Python's object allocator ] | |
+2 | ####### Object memory ####### | <------ Internal buffers ------> |
______________________________________________________________ |
[ Python's raw memory allocator (PyMem_ API) ] |
+1 | <----- Python memory (under PyMem manager's control) ------> | |
__________________________________________________________________
[ Underlying general-purpose allocator (ex: C library malloc) ]
0 | <------ Virtual memory allocated for the python process -------> |
=========================================================================
_______________________________________________________________________
[ OS-specific Virtual Memory Manager (VMM) ]
-1 | <--- Kernel dynamic storage allocation & management (page-based) ---> |
__________________________________ __________________________________
[ ] [ ]
-2 | <-- Physical memory: ROM/RAM --> | | <-- Secondary storage (swap) --> |
He found that the more HTTP client requests he did, the more memory his Node process would consume, but it was really slow.
[...] Then I ran Node with UMEM_DEBUG set to record various important information about the memory allocations
[...] Every hour, it grabbed the output of pmap -x and a core file and stored those in Joyent Manta
[...] In MDB there's a particularly helpful command ::findleaks that will show you the memory addresses and the stack traces for leaked memory, not unlike using valgrind, but without all the performance penalty.
[...] At this point we knew that we were looking for something in v0.10 that called MakeCallback but that didn't first have a HandleScope on the stack. I then worked up this simple DTrace script.
Key point: this isn’t (just) about PGP or SMTP, it’s email as a whole system:
Even after we replace PGP, encrypted email will remain unsafe. Here’s why:
- If messages can be sent in plaintext, they will be sent in plaintext.
- Metadata is as important as content, and email leaks it.
- Every archived message will eventually leak.
- Every long term secret will eventually leak.
Un ensemble de papiers très sales sont en train d’être publiés chez Disclose [...]
on y apprend les détails de la fourniture d’un système de surveillance français au gouvernement Egyptien d’Al-Sissi. [...]
On y apprend que Nexa Technologies, via sa filiale émiratie, Advanced Middle East Systems … Attendez, on s’arrête un instant, Advances Middle East Systems… A.M.E.Sys. Amesys quoi. Ça ne s’invente pas.
On se croirait dans un mauvais film d’espionnage où les méchants [...] sont juste partis là où le climat politique était plus favorable, où ça dérange moins de vendre des outils pour tuer des gens. [...]
on y apprend également que cette vente a été « validée » par le SBDU. [...]
Ce ministre de l’Économie, celui qui porte la responsabilité de la validation de ce dispositif [...] c’est Emmanuel Macron, c’est le Président de la République.