Let me tell you about the still-not-defunct real-time log processing pipeline we built at my now-defunct last job. It handled logs from a large number of embedded devices that our ISP operated [...] Eventually our team's log processing system evolved to become the primary monitoring and alerting infrastructure for our ISP.
you mostly get told that you shouldn't be using unstructured logs anyway, you should be using event streams.
That advice is not wrong, but it's incomplete.There's a file called /dev/kmsg which, if you write to it, produces messages into the kernel's buffer. Let's do that! For all our messages!
RAM is even more volatile than disk, and you have to reboot after a kernel panic. So the RAM is gone, right? Well, no. Sort of. Not exactly.
have the client to stream logs to the server. This is possible using HTTP POST and Chunked encoding,
The log uploader uses a backoff timer so that if it's been trying to upload for a while, it uploads less often. However, the backoff timer was limited to no more than the usual inter-upload interval.
Someone probably told you that log messages are too slow, or too big, or too hard to read, or too hard to use, or you should use them while debugging and then delete them. All those people were living in the past and they didn't have a fancy log pipeline. Computers are really, really fast now. Storage is really, really cheap.
How much are you paying for someone to run some bloaty full-text indexer on all your logs, to save a few milliseconds per grep?
Just a reminder to myself, for the next time I need to compile a classic GNU tool :
git clone --recurse-submodules https://git.savannah.gnu.org/git/grep.git
./bootstrap # no need to install gnulib or call autoreconf -vif & cie
./configure && makeI was checking if a bug still existed in latest source code of grep.
It does not in v3.1.43, but the following hangs forever with grep 2.16 under Ubuntu:
env -i LANG=en_US.UTF-8 src/grep -Fq $'\377\302' ../ludochaordic/content/images/2015/05/Strip-SVN-800-final.jpgIt did not git bisect to find out when the bug was solved.
FROM: http://taint.org
Die shell script, DIE !
In this post, I'll show how easy it ease to convert fragile shell scripts to Python scripts, using sh.py. I'll use as an example a simple script to check your HTML code from the command-line, using the W3C validator.
Now you ask me, why the …
I often find myself grep-ing for information in system or application log files. And often, by combining pipes, I end up generating a flow of values that is sometimes difficult to interpret.
In this post I'll show you a quick-and-dirty but handy solution to get basic statistical quantities from …
As a tabletop RPG game master, whenever I need to imagine a universe background for a scenario, I need illustrations to picture myself the atmosphere, and get some inspiration. I usually simply surf the web from blog to blog, or spend some time on inspirational websites like DeviantArt, Tumblr, Unurth …