Why you cannot trust Lennart Poettering / systemd
I've been watching the systemd drama with a skeptical eye, from a distance. But tonight I read something that crystallized it all for me, so now I'm compelled to point out why Lennart Poettering and systemd cannot be trusted.
The gory story begins with their journald discussion, here http://thread.gmane.org/gmane.comp.sysutils.systemd.devel/6922/focus=6950 where they rationalize why they created their own data structure rather than using an existing embedded DB. Aside from their claim that nothing suitable existed being wrong (LMDB belies those claims) the fact is the solution they developed is inherently unreliable https://bugs.freedesktop.org/show_bug.cgi?id=64116 which utterly invalidates their claimed attention to robustness.
Going thru the rationale point by point
- small, embeddable, in pure C - at just 6KLOC, LMDB is the smallest transactional DB engine in the world, written in pure C. http://symas.com/mdb/inmem/#sec1
- stable API, sanely managed, free software - the LMDB API is modeled after the BDB API so it was mature from day 1 and has had no incompatible changes since LMDB's introduction. It is professionally managed by software veterans and released under the OpenLDAP Public license - totally free.
- OOM safe - LMDB does practically no memory allocations of its own - readers require zero mallocs, writers seldom require more than 1. It can never use enough memory to drive a system into swapping; it can never cause an OOM (out of memory) condition.
- typeless - LMDB is typeless, as a key/value store it doesn't care what types of values you store.
- efficient with large and binary data - LMDB is particularly efficient with large data, nothing else comes anywhere close. http://symas.com/mdb/microbench/#sec4
The gory story begins with their journald discussion, here http://thread.gmane.org/gmane.comp.sysutils.systemd.devel/6922/focus=6950 where they rationalize why they created their own data structure rather than using an existing embedded DB. Aside from their claim that nothing suitable existed being wrong (LMDB belies those claims) the fact is the solution they developed is inherently unreliable https://bugs.freedesktop.org/show_bug.cgi?id=64116 which utterly invalidates their claimed attention to robustness.
Going thru the rationale point by point
- small, embeddable, in pure C - at just 6KLOC, LMDB is the smallest transactional DB engine in the world, written in pure C. http://symas.com/mdb/inmem/#sec1
- stable API, sanely managed, free software - the LMDB API is modeled after the BDB API so it was mature from day 1 and has had no incompatible changes since LMDB's introduction. It is professionally managed by software veterans and released under the OpenLDAP Public license - totally free.
- OOM safe - LMDB does practically no memory allocations of its own - readers require zero mallocs, writers seldom require more than 1. It can never use enough memory to drive a system into swapping; it can never cause an OOM (out of memory) condition.
- typeless - LMDB is typeless, as a key/value store it doesn't care what types of values you store.
- efficient with large and binary data - LMDB is particularly efficient with large data, nothing else comes anywhere close. http://symas.com/mdb/microbench/#sec4
- does not require file locks; readers don't block writers, and writers don't block readers.
- robust against failures - LMDB is crash-proof by design, and proven in the real world. https://symas.com/carrier-grade-stability-and-performance/
- robust against failures - LMDB is crash-proof by design, and proven in the real world. https://symas.com/carrier-grade-stability-and-performance/
- needed in-line compression - this is a red herring. LMDB uses 1/3rd as much space as DBs that offer compression. Compressors need to average 3-4x compression ratios to match LMDB's space efficiency, but few can guarantee more than 2x compression on average.
http://symas.com/mdb/inmem/scaling.html (By the way, I've got nothing against compression in general. In fact I was one of the pioneers of open source lossless data compression, nearly 30 years ago.
https://sourceforge.net/p/arc/ It serves a purpose and I know, better than most, where it's useful and appropriate.)
Leaving aside the blatant ignorance and NIH-ism, this thread shines a bright light on the whole hazy question of "Unix design philosophy" that has shrouded the systemd debate. The Unix philosophy is to write small, single-purpose tools that can be composed together (in pipelines, via scripts, etc.) to build larger solutions. The systemd folks are quick to point out that systemd is not monolithic; it in fact currently is comprised of 69 individual binaries. http://0pointer.de/blog/projects/the-biggest-myths.html
This rebuttal from the systemd camp utterly misses the point though. Having 69 binaries that are all intimately tied together, interdependent, and written by the same person is just playing a shell game. The point in writing dedicated tools is to have dedicated authors, subject-matter experts, responsible for each tool. Unless you're an extremely gifted and brilliant individual, you cannot acquire and maintain expertise in all of the subject areas that a complex system requires for its operation. The systemd team has clearly demonstrated that they're incompetent at designing robust multi-process data storage systems. What else in their all-in-one kitchen sink project have they also failed to design competently, leaving implementation issues aside? Which other viable technologies have they ignored, in their quest for in-house homegrown purity?
Leaving aside the blatant ignorance and NIH-ism, this thread shines a bright light on the whole hazy question of "Unix design philosophy" that has shrouded the systemd debate. The Unix philosophy is to write small, single-purpose tools that can be composed together (in pipelines, via scripts, etc.) to build larger solutions. The systemd folks are quick to point out that systemd is not monolithic; it in fact currently is comprised of 69 individual binaries. http://0pointer.de/blog/projects/the-biggest-myths.html
This rebuttal from the systemd camp utterly misses the point though. Having 69 binaries that are all intimately tied together, interdependent, and written by the same person is just playing a shell game. The point in writing dedicated tools is to have dedicated authors, subject-matter experts, responsible for each tool. Unless you're an extremely gifted and brilliant individual, you cannot acquire and maintain expertise in all of the subject areas that a complex system requires for its operation. The systemd team has clearly demonstrated that they're incompetent at designing robust multi-process data storage systems. What else in their all-in-one kitchen sink project have they also failed to design competently, leaving implementation issues aside? Which other viable technologies have they ignored, in their quest for in-house homegrown purity?