This describes tools and techniques that can identify memory leaks in Long running Python programs:
- Is it a Leak?
- Sources of Leaks
- A Bit About (C)Python Memory Management
- Reference Counts
- Garbage Collection
- The Big Picture
- CPython’s Object Allocator (pymalloc)
Here is a visualisation of memory allocators from top to bottom (from the Python source Objects/obmalloc.c):
_____ ______ ______ ________
[ int ] [ dict ] [ list ] ... [ string ] Python core |
+3 | <----- Object-specific memory -----> | <-- Non-object memory --> |
_______________________________ | |
[ Python's object allocator ] | |
+2 | ####### Object memory ####### | <------ Internal buffers ------> |
______________________________________________________________ |
[ Python's raw memory allocator (PyMem_ API) ] |
+1 | <----- Python memory (under PyMem manager's control) ------> | |
__________________________________________________________________
[ Underlying general-purpose allocator (ex: C library malloc) ]
0 | <------ Virtual memory allocated for the python process -------> |
=========================================================================
_______________________________________________________________________
[ OS-specific Virtual Memory Manager (VMM) ]
-1 | <--- Kernel dynamic storage allocation & management (page-based) ---> |
__________________________________ __________________________________
[ ] [ ]
-2 | <-- Physical memory: ROM/RAM --> | | <-- Secondary storage (swap) --> |He found that the more HTTP client requests he did, the more memory his Node process would consume, but it was really slow.
[...] Then I ran Node with UMEM_DEBUG set to record various important information about the memory allocations
[...] Every hour, it grabbed the output of pmap -x and a core file and stored those in Joyent Manta
[...] In MDB there's a particularly helpful command ::findleaks that will show you the memory addresses and the stack traces for leaked memory, not unlike using valgrind, but without all the performance penalty.
[...] At this point we knew that we were looking for something in v0.10 that called MakeCallback but that didn't first have a HandleScope on the stack. I then worked up this simple DTrace script.
All of the below properties or methods, when requested/called in JavaScript, will trigger the browser to synchronously calculate the style and layout*. This is also called reflow or layout thrashing, and is common performance bottleneck.
In the end, we found that the maximum connections concept was too imprecise to provide the complete answer to the puzzle. In this article, we’ll describe other approaches such as load shedding that we found worked well.
À oui.sncf, je travaille au sein d'une équipe en charge de l'usine logicielle,
qui administre depuis des années une instance Gitlab self-hosted.
Cet article contient quelques-unes de nos recommandations à l'intention des utilisateurs de notre Gitlab, ayant pour but à la fois améliorer les performances de leurs pipelines …
Today, I am happy to announce a new version 2.2.0 of fpdf2 !
https://github.com/alexanderankin/pyfpdf/ 
Doc: https://alexanderankin.github.io/pyfpdf/
During the last few months, I contributed a few improvements to fpdf2,
David Ankin fork of PyFPDF,
the user-friendly Python library to generate PDFs:
from …
Covering :
- Persistent Connections
- Parallelism
- Asynchronicity
- Performances:
Time needed for `serialized' called: 12.12s Time needed for `Session' called: 11.22s Time needed for `FuturesSession w/ 2 workers' called: 5.65s Time needed for `FuturesSession w/ max workers' called: 1.25s Time needed for `aiohttp' called: 1.19s - Streaming
if you’re using Python, Alpine Linux will quite often:
- Make your builds much slower.
- Make your images bigger.
- Waste your time.
- On occassion, introduce obscure runtime bugs.
We faced an issue with DNS on alpine images too at work...
cf. https://wiki.musl-libc.org/functional-differences-from-glibc.html#Name-Resolver/DNS
TL;DR:
- List Resizing : the backing array is grown by approximately 12% (in Java ArrayList grows by 50% when expanded2 and in Ruby, Array grows by 100%). The Python implementation optimizes for memory usage over speed. Another reason to preallocate Python lists when possible.
- Inserting at the beginning of a list takes linear time. Sometimes, better use Deques which trade constant time insert and remove from both ends in exchange for constant time indexing.
By Russel Cohen
Top 2 performers are bjoern (written in C) > meinheld (based on picoev)
I often hear people who are happy because PyPy makes their code 2 times faster or so. Here is a short personal story which shows PyPy can go well beyond that.
In this blog post, I'm going to demonstrate how to reuse WiseMapping HTML+JS rendering engine to easily visualize...
text-based mindmaps
like this one
have many benefits
they are readable as-it-is
they don't require any tool to be edited
they follow the UNIX tenets
For the impatient ones, here is …
En mars dernier, j'ai participé à la Battledev de RegionJobs.
Et c'était super fun.
Pour ceux qui ne connaissent pas, c'est une compétition de programmation en ligne, où l'on doit résoudre 5 questions de difficulté croissante en 2 heures.
J'ai eu l'occasion d'y participer avec quelques collègues, et ça a …
Nylas provides a modern developer
platform for email, contacts, and calendar. Stop fighting old protocols,
and start building great products
Disclaimer I am the creator and main developer of Brython. I am aware that this makes me suspect of partiality ! The test conditions are explained in detail below so that anyone can easily reproduce them and compare with the results presented here ; if something is wrong please post a comment and I will…
My name is Stephan, and I'm a scientist on the Climatology team at The Climate Corporation. We make extensive use of Python to build statistical weather models, and sometimes we need our code to be fast. Here's how I choose between Numba and Cython, two of the best options for accelerating numeric Python code. Most…