Can
+you use ALSA to get ‘bit-perfect’ audio playback on Linux?
+
+
While I’m not convinced that the ‘bit-perfect’ audio movement has
+much merit to it, I think it’s fair to ask that our computers shouldn’t
+fiddle around with sampled audio more than they really have to. If I
+have a high-definition audio file or stream from Qobuz, for example,
+that was mastered with a 24-bit sample size and 96,000 samples per
+second, and I play that file to a top-quality DAC, I’d hop to hear
+something pretty close to the original recording.
+
In the Linux world these days, most mainstream desktop Linux set-ups
+use the Pulse audio framework. Pulse has a bad reputation for, well, for
+all sorts of things, really. Among hifi enthusiasts, though, it has a
+bad reputation for resampling and down-sampling audio. It doesn’t do
+that on a whim – the job that it thinks it has to do requires
+manipulating the audio. So I wouldn’t expect things to be better, or
+easier, with newer audio servers like PipeWire.
+
Now, ‘Bit perfect’ isn’t really a well-defined term. Still, the kinds
+of things that Pulse is accused (rightly or wrongly) of doing probably
+disqualify it from being bit-perfect on any terms. It might not even be
+“bit-kind-of-OK”, which is a far bigger problem.
+
A solution that is often proposed to this problem – if we can’t make
+Pulse, et al., behave themselves – is to have our audio software write
+directly to the ALSA devices in the kernel. Pulse, after all, is just a
+wrapper around ALSA, as are JACK, PipeWire, and all the rest. If we use
+ALSA directly, it’s suggested, we can avoid all the alleged horrors of
+Pulse, and get bit-perfect audio.
+
But can we? In order to answer that question, even approximately, we
+need to take a dive into the murky world of ALSA.
+
ALSA is comprehensible.
+No, really, it is
+
ALSA is a set of audio drivers for the Linux kernel. The drivers make
+themselves visible to the user through a bunch of pseudo-files in the
+directories /dev/snd and /proc/asound. In
+practice, so far as I know, no mainstream audio application uses these
+interfaces directly. In practice, there’s no need to. Audio applications
+use a library to interface to ALSA. For C/C++ programs, the usual
+library is libasound.
+
This library isn’t just a thin wrapper around the kernel devices:
+it’s a complete audio processing framework in its own right. The
+operation of the library is controlled by a bunch of configuration
+files, the most fundamental of which is usually
+/usr/share/alsa/alsa.conf. This turgid, undocumented
+monstrosity defines such things as how to process audio using various
+plug-ins. One such plug-in is dmix, which allows multiple
+applications to write to the same audio device at the same time. You’ll
+appreciate, I’m sure, that this mixing operation is difficult to do
+without interfering with at least one of the audio streams, and maybe
+both.
+
To an application that uses libalsa, an ALSA PCM device
+has a name like this:
+
hw:CARD=digital,DEV=0
+
In a lot of older documentation, you’ll see only numbers used:
+
hw:CARD=0,DEV=0
+
A ‘card’ is a particular peripheral device, attached to the computer.
+Cards have devices and, sometimes, sub-devices. A ‘card’ need not be a
+plug-in card on a motherboard: my external USB DAC is a ‘card’ in ALSA
+terms.
+
Both the names and the numbers of the various cards can be seen by
+looking at the contents of /proc/asound/cards. Here is
+mine:
+
0 [digital ]: USB-Audio - Head Box S2 digital
+ Pro-Ject Head Box S2 digital at usb-0000:00:14.0-9.2, high speed
+ 1 [HDMI ]: HDA-Intel - HDA Intel HDMI
+ HDA Intel HDMI at 0xf1630000 irq 37
+ 2 [PCH ]: HDA-Intel - HDA Intel PCH
+ HDA Intel PCH at 0xf1634000 irq 35
+
Card 0, aka digital is my external USB DAC. This was an
+expensive item, and I expect it to produce top-notch sound when it is
+managed properly (and it does).
+
+
+The Pro-Ject S2 DAC
+
+
Card 1 is a built-in HDMI audio device, that I’ve never used. Card 2
+is the device that handles the built-in headphone/microphone socket.
+
In general, we prefer to name our cards these days, rather than
+number them, because the numbers tend to change. They will certainly
+change if USB devices are plugged or removed. There are some tricks you
+can do in the kernel to get consistent numbering but, these days,
+there’s really no need – just use names.
+
Each card has one or more devices. The devices are generally
+numbered, rather than named, because the numbers are consistent. The
+devices might correspond to specific outputs – analog, optical, coaxial,
+etc.
+
The trickiest part of the PCM name, and the one the is frequently
+explained wrongly, is the hw part. This part of the device
+name describes a protocol. Sometimes you’ll see the term
+‘interface’ instead, but I prefer protocol. Whatever you call it, this
+term describes the way in which the ALSA library will interact with the
+kernel’s audio devices. The same protocol can apply to many different
+cards and devices.
+
You don’t have to specify a protocol. I might have formulated my
+device name like this:
+
default:CARD=0:DEV=0
+
This means ‘use whatever the default protocol is, for the specific
+card and device’. The default will usually be determined by
+configuration in alsa.conf.
+
There are several protocols that are almost universally defined.
+hw is a protocol in which no format conversions of any kind
+are applied. The application will have to produce output in exactly the
+format the hardware driver accepts. In particular, the audio parameters
+have to match in sample rate, bit width, and endianness, among other
+things.
+
The plughw protocol, on the other hand, is more generous
+to applications. This protocol routes audio through plug-ins that can do
+many audio conversions, including resampling. If you really want to
+avoid your samples being tampered with, this is something to avoid.
+
But… you guessed it… the ‘default’ protocol almost certainly includes
+these conversions. It probably also includes the dmix
+plug-in. Why?
+
Looking at device
+capabilities
+
To the best of my knowledge, there’s no way to find the exact
+capabilities of an audio device by looking at files on the filesystem.
+You’ll need some software to do this. Some audio players can do it, but
+a more general-purpose approach is the alsacap utility,
+originally by Volker Schatz, and now available on GitHub. This utility
+may be in your Linux distribution’s repository but, if not, you’ll have
+to build it from source. It’s very old software, and I’ve sometimes had
+to fiddle with it to make it compile on some machines. In any event,
+this is what it says about my USB DAC (Card 0, remember):
+
Card 0, ID `digital', name `Head Box S2 digital'
+ Device 0, ID `USB Audio', name `USB Audio', 1 subdevices (1 available)
+ 2 channels, sampling rate 44100..768000 Hz
+ Sample formats: S32_LE, SPECIAL
+ Buffer size range from 16 to 1536000
+ Period size range from 8 to 768000
+
This device is very flexible in its sampling rate: it will handle
+everything from CD sample rates upwards. But it’s a stereo device – it
+won’t handle mono audio or any surround-sound formats. Most problematic
+though: the only common data format is S32_LE. That’s 32
+bits per sample, in little-endian bit ordering. Why is this an issue?
+Because almost no audio files or streams are in this format.
+
‘hw’
+is a good idea, but we might not be able to use it directly
+
If we’re looking for bit-perfect audio, or something close to it, it
+seems that the ‘hw’ protocol will offer the best results. Unfortunately,
+on my particular set-up – and probably on yours – it won’t work without
+some fiddling.
+
For my DAC, we’re going to have to apply at least a zero-padding
+conversion, and maybe a bit-ordering conversion as well. That is, we’ll
+need to ensure that, whatever the source material, the data written to
+the DAC is 32 bits, little-endian. It really doesn’t matter whether we
+do these conversions in the audio software or in ALSA: there’s only one
+correct way to do them.
+
Don’t get me wrong: even from a ‘bit-perfect’ perspective, these
+conversions are harmless. The number ‘20’, for example, is not a
+different number from ‘020’, and the same is true in binary. And it
+doesn’t matter what order we read the digits out, so long as everybody
+agrees on the order.
+
The problem is that unless we use the ‘hw’ protocol, we lose
+control over what ALSA is actually doing. And if we do use it, the audio
+software has to know how how to cope. Simple utilities like
+aplay don’t have the necessary smarts. If I try this:
Playing WAVE '/usr/share/sounds/alsa/Front_Center.wav'
+: Signed 16 bit Little Endian, Rate 48000 Hz, Mono
+aplay: set_params:1387: Sample format non available
+Available formats:
+- S32_LE
+- SPECIAL
+aplay: set_params:1387: Sample format non available
+Available formats:
+- S32_LE
+- SPECIAL
+
I’m trying to play a 16-bit mono file into a 32-bit, stereo device.
+Although the conversions are straightforward, aplay won’t
+do them, and the hardware can’t do them.
the audio clip plays just fine. That’s because the audio processing
+chain in the ALSA library does whatever conversions are necessary, to
+map the source to the target device. But how do we know whether it’s
+doing ‘harmless’ conversions, like padding, or nasty ones, like
+resampling or mixing?
+
We don’t.
+
We have to check the
+capabilities
+
So if we want bit-perfect audio, or close to it, we need to check the
+capabilities of the device (e.g., using alsacap), and work
+out whether there is a safe transformation between the source and device
+formats.
+
A particular case where there might not be, is where the source is an
+audio CD or CD rip, and the target is a device whose sample rate is
+fixed at 48,000 per second. These devices are pretty common in the
+computer audio world. CD audio is always recorded at 44,100 samples per
+second. Many, perhaps most, audio files and streams from on-line
+suppliers are in this format. Playing this CD audio on a DAC that can
+only do 48,000 samples per second must use a conversion. In
+this case, it actually matters whether the conversion is done in ALSA,
+or in the music player software, because there are good and bad ways of
+doing it. I would guess (and it is just a guess) that ALSA doesn’t use a
+very sophisticated approach. Why? Because anybody who really cares about
+audio quality to this extent will be using a specialist music player
+application that can do the conversion well. And, despite what the hifi
+snobs say, this conversion can be done well. Well enough that
+no human hearing will be able to tell the original from the resampled
+sound, anyway.
+
But it won’t be ‘bit-perfect’, however we do it. Bit-perfect is a
+technical consideration, not a subjective one.
+
We
+need decent audio hardware, regardless of the Linux set-up
+
In order to avoid the kinds of problems I mentioned, we need audio
+hardware that can cope, without conversion, with all the sample rates we
+need to play. It also has to be able to handle the largest sample size
+we will play. In practice, this means 24 bits per sample – it’s rare to
+encounter anything more than this in the wild.
+
It’s not only the audio hardware – the DAC – that has to be able to
+cope with these constraints: the driver in the Linux kernel has to cope
+as well. Oh, and the hardware has to do it without appreciable jitter
+(timing errors). For USB DACs, we don’t have to worry too much about any
+of these things – the USB audio driver built into Linux will handle
+whatever the hardware can handle, and USB is asynchronous and therefore
+immune to jitter. Audio hardware built into motherboards is a different
+matter. The Intel PCH integrated audio in my computer has a 16-bit DAC –
+or so it says. It’s probably honest – 16-bit DACs are commodity items.
+But if a motherboard audio device claimed more than 16-bit
+resolution, I think I’d be a bit sceptical.
+
But 24 and even 32-bit sample sizes are common in serious hifi DACs.
+Some will even use multiple DACs of this size. I feel reasonably
+confident that, if I play music through my USB DAC, using as the ASLA
+device specification hw:CARD=digitial, the DAC will receive
+unmodified audio data. Well, except for the harmless zero-padding,
+anyway.
+
Or am I?
+
We don’t
+really know what the ALSA driver is doing
+
If I use the hw protocol with ALSA, the data is still
+going through the ALSA driver in the kernel. It has to – that’s what the
+driver is for. But does the driver just forward the unmodified bits to
+the hardware? Well, no, as it turns out.
+
I know this because the volume control in alsamixer
+still has an effect on the volume, even when using the hw
+protocol. This volume control is implemented in the Linux kernel, not
+the software I’m using to play audio. Of course, that software might
+also fiddle with the audio stream, but that’s a different
+matter. If I was worried about the software, I could use different
+software. But I can’t really use a different kernel driver.
+
The fact that the volume control works means that the kernel driver
+is manipulating the bit stream mathematically. I’m reasonably confident
+that if I set the volume control to ‘100%’ (or 0dB) then the math will
+be benign. But without looking at the source code for the driver, I
+can’t really be certain.
+
I must point out that I’m talking about my USB DAC here.
+Your sound device and driver might behave differently – there’s only one
+way to find out, and that’s by doing the kinds of experiments I’ve
+described in this article.
+
So…?
+
‘Bit-perfect’ is an objective measure, not a subjective one. To be
+‘bit-perfect’ means that the audio data goes straight from the file or
+stream to the DAC without any alteration. I’m prepared to accept that
+harmless modifications like bit padding and bit ordering don’t take away
+the ‘bit-perfect’ status. But mixing and resampling certainly do.
+
With the tests I’ve described above, I’m reasonably sure that my USB
+DAC, with suitable player software, driven from ALSA using the
+hw protocol, probably qualifies as ‘bit-perfect’.
+At least, it’s as close as makes no difference.
+
But there’s no magic way of doing this: there’s no
+‘use-bitperfect=true’ property that you can set anywhere that
+will ensure you getting bit-perfect, or even good, audio transmission.
+You really have to understand how ALSA works, and have a good grasp of
+digital audio principles.
+
And when we move on to Pulse, PipeWire, etc., the situation becomes
+even more complicated; in addition to the uncertainties I’ve descried in
+this article, we have the additional uncertainties introduced by that
+software.
+
But let’s not get too despondent. You don’t really need
+bit-perfect audio transmission. With modern, high-quality audio DACs,
+and a bit of care, you can still get excellent audio quality. That’s
+true even for systems using Pulse, if you take enough care over
+configuration (and you’re not using a buggy version).
+
I grew up with cassette tape as my main source of recorded music.
+We’ve come a long way since then, in terms of sound quality. We
+shouldn’t get hung up on things that don’t really matter.
+
+
+
+
+
+
+
+
+
+
diff --git a/back_to_alsa.html b/back_to_alsa.html
new file mode 100644
index 0000000..6e4c7a6
--- /dev/null
+++ b/back_to_alsa.html
@@ -0,0 +1,220 @@
+
+
+
+
+ Kevin Boone: Going back to ALSA in 2024
+
+
+
+
+
+
+
+
+
+
+
+
+
ALSA is the low-level audio driver architecture in the Linux kernel.
+It’s been around for twenty years, and I suspect it’s supported by every
+Linux system on Earth that does audio. Still, over the last ten years or
+so we’ve got used to not using ALSA directly. Instead, most mainstream
+Linux distributions provide a sound server that wraps ALSA, and provides
+additional features. The most common of these wrappers are Pulse and
+PipeWire. Pulse seems these days to be mostly associated with
+Ubuntu-like systems; PipeWire is more used by Fedora and its
+derivatives.
+
Both Pulse and PipeWire work reasonably well for most people and,
+frankly, there’s little reason to remove them. Even systems that don’t
+enforce the use of an audio server, like Gentoo, still provide one, and
+I suspect most users install one. However, there are a few reasons why
+we might want to avoid these things, and just use ALSA directly.
+
+
There are bugs in Pulse and PipeWire that can be hard to work
+around.
+
Pulse/PipeWire are significant users of system resources.
+
It’s difficult to prevent Pulse/PipeWire tampering with the audio
+stream, for various reasons. It sometimes isn’t even possible to bypass
+Pulse/PipeWire and configure an application to use ALSA directly,
+because audio servers tend to lock all the ALSA devices for their own
+use. This problem makes it hard to play hifi audio sources at full
+quality.
+
Pulse/Pipewire rely on services like DBus. That’s not usually a
+problem in systems with integrated desktop environments, because DBus is
+more-or-less essential there. In the embedded Linux world, though, we
+might not have have a desktop, or even a graphical display. Reliance on
+audio servers makes the development of embedded systems more
+difficult.
+
+
There are good reasons to persist with modern audio servers, despite
+these limitations.
+
+
First, application support for direct use of ALSA is diminishing.
+Pulse is so ubiquitous that some application maintainers won’t support
+anything else. Mozilla Firefox is a case in point. It’s possible to
+build Firefox with ALSA support, but it’s recognized to be unstable. The
+binary version of Firefox provided by most mainstream distributions
+doesn’t have ALSA support. The Firefox maintainers should hang their
+heads with shame, frankly, for letting ALSA support lapse like this.
+There’s simply no justification for the developers of any audio
+application not to support ALSA because, by supporting ALSA, you’re
+automatically supporting all the other things. Still, this is the way
+things are going.
+
Audio servers like Pulse are actually useful. You might not realize
+just how useful, until you try to do without one.
+
Modern desktop environments like Gnome increasingly assume that an
+audio server is available. Some things that seem absolutely fundamental,
+like being able to control audio volume, won’t work without
+fiddling.
+
It’s hard to get rid of Pulse or PipeWire on systems that have it.
+On Ubuntu you can apt remove pulseaudio but that will take
+a heap of other stuff with it, which you’ll then have to reinstall. And
+then Pulse will keep coming back. It’s a drag, frankly.
+
+
What to expect with ALSA
+alone
+
If you really do want to remove Pulse/PipeWire, and have the
+perseverance to do it, here’s what you can expect.
+
Audio mixing
+does work, despite what everybody says
+
One of the claimed benefits of Pulse, when it was first released, was
+that it would allow multiple applications to play audio at the same
+time. ALSA has had this support built in for at least ten years. What
+Pulse can do, however, and ALSA can’t, is provide separate volume
+controls for different applications.
+
There
+may, or may not, be audio control integration in the desktop
+
I use Xfce4 which does, in principle, have an ALSA mixer control. I
+say ‘in principle’ because, so far as I can see, it doesn’t exist in any
+of the mainstream Linux repositories. You’d have to build it from
+source.
+
Of course, you can always run alsamixer in a terminal –
+it’s just the lack of a graphical control that’s a nuisance.
+
You’ll probably also find that the keyboard volume controls won’t
+work. You should be able to map these keys to alsactl
+commands to change the volume but, if you have multiple ALSA audio
+devices, it won’t always be obvious which one you’re changing.
+
You’ll
+probably have to tell software which ALSA device to use
+
ALSA dates from a time before it was common to have multiple audio
+devices in the same computer. Now it uncommon not to. Most computers
+will have at least one conventional soundcard, and one or more HDMI
+audio channels.
+
You’ll probably have to tell your software which device to use, and
+the way to do that will vary from one application to another. For VLC,
+for example, I do:
+
$ vlc -A ALSA --alsa-audio-device=...
+
Working out which ALSA device to use can be a challenge. Even if the
+software lists the ones it knows about (as the VLC graphical interface
+can) the list almost certainly won’t be complete. You have to know
+something about how ALSA works to choose the right device. I go into
+this subject more in my article on bit-perfect ALSA.
+
For simple cases, I wrote a little Java program jalsaselector
+which changes the default ALSA output card.
+
You certainly won’t be able to change ALSA devices whilst an
+application is running. That’s a capability that really does need an
+audio server.
+
apulse
+can come to the rescue when applications only support Pulse
+
apulse is an
+application wrapper that intercepts API calls to Pulse, and converts
+them to ALSA operations. It works for both input and output. It’s not
+foolproof, because it doesn’t implement the entire Pulse API; but it’s
+sufficient, for example, for making Firefox do Zoom calls.
+
apulse has the same problem that all ALSA applications
+have in the modern world: it doesn’t provide an easy way to select
+between multiple audio devices. At the command line, you can use
+environment variables, like this:
+
$ APULSE_PLAYBACK_DEVICE=bluealsa apulse firefox
+
This way of working won’t give you a way to change output or input
+devices on the fly, once the application is running; but that’s a
+problem with ALSA in general, not with apulse.
+
Headphones
+will be a headache. Well, Bluetooth ones will.
+
Assuming that your Linux installation provides fundamental Bluetooth
+support – so you can list, pair, connect, and disconnect devices – you
+can use bluealsa to
+make the connection between ALSA and the Linux Bluetooth stack.
+
bluealsa runs as a daemon, and interacts with DBus for
+finding device information. Some fiddly configuration at the DBus level
+is necessary for it to work with non-root users.
+
bluealsa provides an ALSA protocol, not a
+card. It won’t appear, for example, in the list of cards you
+get from /proc/asound/cards. This means that it usually
+isn’t seen as a potential target by audio applications, even those with
+good ALSA support. You’ll have to specify the bluealsa
+device on the command line and, again, the way to do that varies between
+applications.
+
Because ALSA doesn’t support dynamic switching of output devices,
+making a Bluetooth device a default is hazardous because, if the
+Bluetooth device is switched off, ALSA will have no output at all.
+
Another problem is that the version of bluealsa in
+mainstream Linux repositories probably won’t support high-quality audio
+profiles like AptX. These are proprietary, and subject to patent
+complications. I believe you can get this support by building
+bluealsa from source, if you have the necessary
+dependencies. I haven’t tried this, so I can’t comment on how well it
+works. Without AptX and the like, the best audio quality you’ll get is
+A2DP. That’s not the worst of the Bluetooth audio profiles in common
+use, but it’s still pretty awful.
+
Finding the necessary documentation to set this all up is quite
+difficult, because Bluetooth support and ALSA support are documented –
+to the extent that they are documented at all – in different places.
+Expect some head-scratching.
+
All in all, while an ALSA-only system will support Bluetooth audio, I
+have to wonder whether it’s worth the effort.
+
The future
+
Of the Linux distributions I use, only Gentoo provides adequate
+documentation for using ALSA alone. Gentoo even has some documentation
+for bluealsa. At this time, nearly all the software you
+need to run an ALSA-only Linux effectively on Fedora or Ubuntu is still
+in these distributions’ binary repositories. How long that will remain
+the case, as the distributions continue to focus their efforts on Pulse
+and PipeWire, remains to be seen. Even now, removing Pulse from Ubuntu,
+or PipeWire from Fedora, is so difficult that it’s hardly worth the
+effort.
+
Gentoo, however, continues to make an ALSA-only system practicable,
+with a bit of effort. Well, a lot of effort, if you want to use
+Bluetooth audio.
In my retrocomputing articles I normally look at computing and
+electronics between 1970 and about 2000. In this article, just for a
+change, I’ll be looking back about 2300 years, to a mechanical computer
+– the Antikythera Mechanism.
+
The story of the discovery of this device in a shipwreck by pearl
+divers is well-known, and I won’t go into it here.
+
Over the last hundred years or so we’ve achieved what we believe is a
+reasonable understanding of how this device worked and, to a lesser
+extent, how it was used. This is a remarkable feat in itself,
+considering that the surviving mechanical parts are gummed together by
+two millennia of seabed accretions.
+
+
+The AntiKythera Mechanism, now in the
+National Archaeological Museum in Athens, Photo by
+zeevveez.
+
+
I’m a little sceptical of the claim that the Antikythera Mechanism
+was used for navigation. Although it’s clearly a computing device, my
+feeling is that was more likely used on the desktop, like an orrery – a
+mechanical model of the motions of the moon and planets. The makers of
+the Mechanism probably had an Earth-centered view of the Solar System,
+which makes computing the movements of the planets even more difficult
+than it is with our modern understanding. The gearing that was used to
+model the apparent motion of the planets – if we have understood it
+correctly – is truly remarkable.
+
In this article, however, I will tackle something a little simpler:
+using a clockwork device to predict eclipses of the Sun and Moon. It’s
+almost certain that the Mechanism was used this way – the clue is the
+gear wheel with 223 teeth that can be seen in the photo above. I’ll
+explain why this number is significant later. It turns out that
+modelling eclipses without celestial mechanics is not conceptually all
+that difficult – provided we have acute observational skill and
+extraordinary patience over decades.
+
How we get eclipses
+
Let’s start by looking at how we get eclipses in the first place.
+
To get a solar eclipse of any kind, the Sun, Moon, and Earth must be
+approximately in a line, with the Moon between the Sun and the Earth
+(see diagram below). To get a lunar eclipse, the Earth must be between
+the Sun and the Moon. The time when the Moon lies between the Sun and
+the Earth we call ‘new moon’. At this time, the Moon is not generally
+visible from the Earth, because there’s nothing to illuminate the face
+that we can normally see. At new moon, we’ll only see the moon at all
+when it happens to be causing an eclipse.
+
If a solar eclipse can only happen at new moon, then a lunar eclipse
+can only happen at full moon. So, very often, we’ll see a solar eclipse
+at the new moon following a lunar eclipse. I’ll explain why this is
+important later.
+
The time between consecutive new moons is called the synodic
+month, and it’s about 29.5 Earth days in duration.
+
The Earth and all the other planets of our solar system orbit the Sun
+in what is very close to a plane. That is, we could put them all on a
+(very large) flat surface. We call this plane the ecliptic. If
+the Moon also orbited the Earth in this plane, then we’d see an eclipse
+every synodic month – a solar eclipse at new moon and a lunar eclipse at
+full moon. But that isn’t the case – the orbit of the Moon around the
+Earth is inclined at about 5 degrees to the ecliptic. So, while it has
+to be new moon to see a solar eclipse, we won’t see one every new moon.
+In nearly all cases, the Moon will be too far from the ecliptic for its
+shadow to fall on the Earth. The drawing below shows this situation,
+where the tilt of the Moon’s orbit at a particular time makes it
+unlikely that there will be any eclipse, even at new moon or full
+moon.
+
+
+The Earth-Moon-Sun system, not at all to
+scale
+
+
The points at which the orbit of the Moon crosses the ecliptic are
+called its orbital nodes. There are two – a rising node, and a
+falling node. The time bateween nodes is called the draconic
+month, and it’s about 27.2 Earth days. That the draconic and
+synodic months are nearly the same is, presumably, just a
+coincidence.
+
+
Note In case you were wondering, draconic is
+indeed related to ‘dragon’. In mythology, a lunar eclipse was envisaged
+as a dragon ‘eating’ the Moon.
+
+
So to see some kind of eclipse, at least the following conditions
+must apply:
+
+
It must be new moon (for a solar eclipse) or full moon (for a
+lunar eclipse).
+
The Moon must lie close to one of its nodes; that is, it must lie
+close to the ecliptic plane. It doesn’t have to be stop-on; how far from
+the ecliptic the Moon can be, and still cast a visible shadow on the
+Earth, depends on a number of other factors.
+
+
One of those other factors is how far from the Earth the Moon is, at
+the time under consideration. For a solar eclipse, the further away the
+Moon is from the Earth, the larger the shadow it will cast, and the more
+extensive the area from which the eclipse can be seen. The Moon’s orbit
+around the Earth is elliptical; the closest point is called
+perigee, the furthest apogee. The time between
+successive apogees is called the anomalistic month, and is
+about 27.6 Earth days. It’s very similar to the draconian month of 27.2
+days. Again, this is probably just a coincidence.
+
So, for the purposes of eclipse prediction, there are three different
+kinds of month that we must consider:
+
+
+
+
+
+
+
+
+
Month name
+
Description
+
Duration
+
+
+
+
+
Synodic month
+
Time between consecutive new moons
+
29.53 Earth days
+
+
+
Draconic month
+
Time between Moon’s consecutive crossing of its rising node
+
27.21 Earth days
+
+
+
Anomalistic month
+
Time between consecutive perigees
+
27.55 Earth days
+
+
+
+
Basic eclipse prediction
+
If I see a solar eclipse of some sort today, at my location,
+then my next strong chance of seeing an eclipse is in six synodic
+months’ time. In that time the moon will have completed approximately
+six draconic months, and be close to the ecliptic plain again (it’s
+actually about 6.4 draconic months, because the draconic month is a
+little shorter than the synodic). You might think that the best chance
+to see an eclipse would be a whole number of draconic months since the
+last eclipse, because the moon’s orbit will be crossing the ecliptic
+again, and in line with the Earth and the Sun. But, no – we only see
+solar eclipses at all on or near new moon, and lunar eclipses
+at full moon. Remember that the Moon must not only be near the ecliptic,
+but it must be in line with the Earth and the Sun as well.
+
But six synodic months later is not the only time we might
+see an eclipse.
+
Suppose that, during the today’s eclipse, the Moon was just below the
+ecliptic. At the next new moon, it will be just above the
+ecliptic. That is, it might still be at a point where it could cast a
+shadow on the Earth, and cause an eclipse somewhere. But two synodic
+months later, the Moon will definitely be too far out of line.
+
Similarly, if six months from now the Moon was just able to produce
+an eclipse, being just at the limit of its position with respect to the
+ecliptic, then conceivably there might be an eclipse a (synodic) month
+earlier – five synodic months from today.
+
It turns out that, in practice, if there’s an eclipse today, the next
+eclipse – if there is one – will probably be six synodic months away.
+Less likely, but still worth considering, are intervals of five and six
+synodic months.
+
So, essentially, mechanical eclipse prediction comes down to timing
+synodic months – the phases of the Moon.
+
There’s a lot of probabilistic qualifications in the preceding
+discussion: what if we want to be sure when the next eclipse
+will be? Is that even possible?
+
The saros cycle
+
The best prediction of an eclipse so far seems to be ‘six synodic
+months from the last one’. Maybe. Can we do better? In modern times we
+certainly can, because we have accurate mathematical models of our solar
+system. But how would the ancients have done it?
+
The key is to find how many synodic months have to pass before a
+whole number – or close to whole number – of draconic months have
+passed. At this point we will again be at new moon, with the moon at or
+close to the ecliptic.
+
As a bonus, it would help if the Earth-Moon distance were the same at
+this point in time as it was at the initial point, because that factor
+dictates the size of the shadow cast on the Earth or the Moon.
+
The synodic, draconic, and anomalistic months are not multiples of
+one another and, because they are so similar in duration, it will take a
+long time (perhaps forever) before the Earth, Moon, and Sun are in the
+exact same alignment again. If we look for an alignment where a
+whole number of synodic, draconic, and anomalistic months all occur
+within a 24-hour period, it turns out that 223 synodic months, which
+is
+6585.32 days, is almost exactly the same as 242 draconic months –
+6585.36 days, and 239 anomalistic months – 6585.54 days. It’s not a
+perfect alignment, but we have to wait a long time to find a better one.
+If we want the alignment to be within a period of a couple of hours,
+we’ll have to wait 3263 synodic months, which is about 263 years.
+
+
Note I don’t know if there’s a way to calculate these
+time periods analytically. I wrote a program to do it.
+
+
+
Note It’s worth bearing in mind that looking for closer
+alignments is probably unprofitable – the figures for the lengths of the
+various lunar months are averages, not exact. There’s probably no way to
+perform a more exact calculation.
+
+
The period of 223 synodic months is called a saros. Although
+this term is derived from ancient sources, it turns out that its use in
+the context of eclipse prediction only dates back to 1686. It was used
+this way by Edmond Halley (of comet fame), but whether the ancients used
+it in the same way is contested. That the ancients could
+predict eclipses is not contested – just the terminology.
+
It turns out that, if there’s an eclipse today, we can be reasonably
+certain that there will be an eclipse with similar characteristics (same
+duration, same area of coverage) in 223 synodic months. Unfortunately,
+223 synodic months is not a whole number of Earth days. It’s about 18
+years, but it’s eight hours away from the nearest whole number of days.
+What that means is that, although the Moon will case the same kind of
+shadow on the Earth (for a solar eclipse), different parts of the Earth
+will be affected.
+
If we’re using as our initial, reference eclipse an event where the
+coverage area on the Earth was large (so the Moon was at its furthest
+from the Earth) then it’s plausible that a particular location will see
+both eclipses – today’s, and the one eighteen years later. After all, a
+total solar eclipse with the Sun at apogee can darken a whole continent.
+But, as an ancient astronomer, I’d do better to look at lunar
+eclipses. I will already have noticed that a solar eclipse that I
+do see was preceded by a lunar eclipse in the same synodic
+month; it would seem reasonable to conclude that most or all lunar
+eclipses are followed by a solar eclipse somewhere on Earth.
+And it turns out that lunar eclipses are much easier to see from Earth
+than are solar eclipses – and not just because different regions of the
+Earth are in shadow. It’s actually very difficult to see a modest
+partial eclipse of the sun, because it’s not safe to look directly at
+it. But there are no such problems with the Moon.
+
So it turns out that the saros interval of about 18 years gives us a
+really good way to predict eclipses. But we already know that there are
+far more eclipses than that. If there’s an eclipse today, there’s
+possibly going to be one in six synodic months’ time – that’s less than
+half a year, and a lot shorter than 18 years.
+
But here’s the crucial point: the saros cycle is quite repetitive. If
+I see an eclipse today, and another a synodic month later, I can be
+almost certain that I will see an eclipse a saros period later, and one
+synodic month after that. Moreover, the type of eclipse will be
+similar. So if I can collect 18 years’-worth of eclipse data, I have
+everything I need to predict eclipses with reasonable accuracy for
+hundreds of years to come.
+
The
+importance of observation on repetitive phenomena
+
We know that the ancient Greeks, and before them the Chaldeans, could
+predict eclipses. We also know that they didn’t have modern computers.
+What they did have was an understanding that natural phenomena repeated,
+and a lot of patience. They almost certainly came upon the saros cycle
+by making astronomical observations for many decades, or even centuries.
+Our ancient forebears were as interested in the skies as we are –
+perhaps even more. After all, the positions of the Sun and Moon had a
+direct impact on agriculture, weather, and tides.
+
How do we do all this with
+clockwork?
+
Most likely none of us has seen a mechanical timepiece with an
+‘eclipse’ dial. What we do sometimes still see, and is instructive to
+consider, is a ‘moon phase’ dial on a mechanical clock or watch. If we
+can display moon phases, a suitable reduction gearing will be able to
+drive a pointer with a saros cycle. So the problem is not different in
+kind, only in engineering complexity.
+
The slowest-moving gear in a conventional watch is the hour gear,
+which usually makes a full rotation every twelve hours. So to display
+the moon phase we need a gear that rotates 29.53 times, for every two
+rotations of the hour gear. Or, alternatively, 59.06 times for each
+rotation of the hour gear.
+
If we accept a small amount of inaccuracy, we can round this to 59
+revolutions of the hour gear for one revolution of the moon phase gear.
+Implementing a gear ratio of 59:1 is easy enough if you have a way to
+make a ‘finger’ gear – essentially a gear with only one tooth. The
+complete gear mechanism is a one-tooth finger gear meshing with a gear
+with 59 teeth. This is, in fact, the way that modern mechanical watches
+display moon phase.
+
This gearing will be accurate to about one part in a thousand. So the
+moon phase display would slip about one day every 2.7 years. A
+mechanical moon phase display would have to be wrong by at least a day
+or so, to be completely useless. So this clock would track moon phase
+reasonably accurately for
+2-3 years, before the moon dial needed to be reset.
+
Could we get better precision with a mechanical gearbox? Certainly:
+we can get a gear ratio of 59.04 using three coupled gear ratios: 20:72,
+into 16:41, into 10:64. The more exact ratio 59.06, but I couldn’t come
+up with a way to implement that, using a modest number of gears with
+modest tooth counts. Still, my three-step gearing would be accurate to
+three parts in ten thousand, so should handle moon phases tolerably well
+for twenty years or so. We could achieve better accuracy, if needed,
+using more gears.
+
To implement a complete saros cycle display requires, in simplistic
+terms, dividing down the moon phase gear by 223:1 (because there are 223
+synodic months in the complete cycle) or, perhaps, dividing down the day
+gear by 6585.32.
+
It seems, however, that this isn’t the way the original Antikythera
+Mechanism worked. Although it probably did have a display of Moon phase,
+it appears that it was derived, not by a simple down-gearing of the
+number of days, but by a subtraction process based on the draconic month
+and the apparent motion of the Sun around the Earth. After all, it is
+this combination of the Earth-Sun and Earth-Moon motions that makes the
+synodic and draconic months.
+
Whether this method of calculation is more accurate, or just more
+expedient, I don’t know (but probably somebody does).
+
Closing remarks
+
The complexity of the Antikythera Mechanism, and the
+single-mindedness that its builders must have needed to collect the
+astronomical data to implement it, has lead some people to propose that
+it was built by visitors from space. Or, alternatively, that it’s a
+modern fake, like the rash of crystal skulls that appeared in the
+1960.
+
For sure, there’s nothing really like the AM – nothing we have so far
+discovered from that time period, anyway. But there are scattered
+ancient writings that allude to complex machinery, and there’s no doubt
+that astronomy was an obsessive interest for many ancient cultures –
+just look at the astronomical alignments embodied in Stonehenge.
+
The reason I don’t feel the same sense of awe and wonder
+when I look back on, for example, the Z80 microprocessor is that I lived
+through most of the technological developments that produced it. Yes,
+I’m old enough to remember when integrated circuits were rejected by
+many engineers as the Devil’s work.
+
When our entire civilization has been submerged by the rising waters
+of the melting ice caps, will our distant descendants find a ZX81 while
+diving for pearls? And, if they do, will they respect and admire our
+ingenuity, or claim that such a thing must have been gifted to us by
+aliens?
+
I wonder.
+
+
+
+
+
+
+
+
+
+
diff --git a/gentoo_mx.html b/gentoo_mx.html
new file mode 100644
index 0000000..d0f0fe3
--- /dev/null
+++ b/gentoo_mx.html
@@ -0,0 +1,294 @@
+
+
+
+
+ Kevin Boone: Gentoo vs. MX Linux on
+old-ish laptops
+
+
+
+
+
+
+
+
+
+
+
+
+
I have a pathological dread of waste. The idea of disposing of
+something that works, just because there’s a newer, shinier alternative,
+makes me sick to my stomach. Maybe that’s a generational thing: I grew
+up in a time and place where nothing was wasted. I find that I
+am drawn to things that were made to last, and can be repaired. Nowhere
+is this more apparent than in my choice of computers: I own no computer
+more recent than 2017. My favourite desktop computer, in fact, runs
+CP/M; I built it myself, with a soldering iron.
+
Linux has long been the go-to operating system for computers that,
+while not necessarily ancient, aren’t based on the latest technology. In
+fact, there may be no alternative, with that kind of hardware. So it
+dismays me to see Linux becoming increasingly inappropriate for this
+kind of use. Nobody much cares about keeping ten-year-old PC hardware
+alive; in fact, I suspect there’s more interest in CP/M and the Z80. A
+ten-year-old laptop has no retro charm: the only reason for continuing
+to use it, is that to do otherwise would be wasteful.
+
Laptop computers present a particular challenge in this area: they
+are usually too poorly cooled to realize the full power of their CPUs.
+Why that’s such an issue –and, with Gentoo, it is – is a point I’ll
+return to later.
+
I’ve been on a bit of a mission lately: to find which Linux
+distributions are most effective on old laptops. For the last three
+months I’ve been migrating my old hardware to Gentoo Linux, but I continue to search for
+a more practical alternative.
+
In this article, I’ll report my experiences with MX Linux, particularly how it stacks up
+alongside Gentoo. Now, it’s true that this is hardly a like-for-like
+comparison: Gentoo isn’t really a Linux distribution in the way that,
+say, Fedora and Ubuntu are. It’s more a collection of tools for building
+a Linux installation according to certain rules. Historically, Gentoo
+users have compiled most of their software from source, although
+recently there have been moves towards a measure of pre-compiled binary
+provision. Nevertheless, my criterion for comparison is simply this: how
+easy do these competing Linux variants make it, to keep old laptops
+alive and useful? So the comparison seems fair to me.
+
Gentoo on old laptops
+
The great strength of Gentoo is that you can, with patience,
+construct a Linux installation that is highly optimized for a particular
+application. If you want a Linux that uses nothing but good-ol’ ALSA for
+audio support, for example, you can have it. You can configure the build
+system (‘Portage’) for ‘ALSA only’, and applications like Firefox and
+VLC will get built without support for anything but ALSA as their audio
+handler. Similarly, if I don’t want to use Wayland, or systemd, or
+Bluetooth, support for these things can be compiled out.
+
The build rules affect not just how software is built, but
+which software is built. That is, the dependency relationships
+between software packages are governed by the same rules that control
+the build process. With care and patience, you can install a Linux that
+contains all of what you want, and little of what you don’t. In
+principle, this is ideal for old hardware, which has no resources to
+spare.
+
But here’s the catch: to use Gentoo to install Linux on old laptops,
+you’re going to need new computers to do all the actual
+compilation. This is because the further your needs are from the
+mainstream, the more software will have to be compiled from source.
+Although Gentoo now has a binary repository, it mostly holds software
+that has been configured for mainstream use. And running on old hardware
+is, sadly, a non-mainstream use of Linux these days. I can build a
+version of Firefox that has ALSA support for audio, but I’ll
+have to build it: I’m unlikely to be able to get a pre-built
+binary package for it. And building Firefox on a 2015 laptop can easily
+take three days. Yes, days.
+
Now, the Gentoo maintainers are aware of this problem, and have
+workarounds, of a sort. Despite what I just said, you can get a
+pre-compiled Firefox package with ALSA support: but it’s a trick. This
+package provides the Pulse Audio variant of Firefox – which is widely
+available as a pre-compiled binary – and a shim library to adapt it to
+ALSA operation. This works, and all the messiness is largely invisible
+to the user but, still, it’s ugly. Firefox will use a bit more memory,
+and load the CPU a tiny bit more, than a pure-ALSA build.
+
The Gentoo binary repository provides, in some selected cases,
+multiple binary variants of the same package, with different features
+enabled. This makes it more likely that you’ll find a binary to suit a
+non-mainstream installation but, in my experience, there isn’t enough
+provision of this kind to make a significant difference to the amount of
+compilation.
+
In short, unless you’re willing to compile software for your old
+computers on a new one, you’re going to have to make compromises: do you
+want to use more binary packages, at the expense of a sub-optimal final
+installation, or live with more compilation? I should point out that
+compiling all the software I use, to get the Linux I really wanted, took
+nearly three weeks of round-the-clock compilation on a 2015 laptop.
+
Laptops are particularly badly hit by this compilation burden,
+because they are usually inadequately cooled. It shouldn’t take
+three weeks to compile a Linux installation on a ten-year-old i7 CPU –
+that it does is a result of the fact that the CPU will have to be
+severely throttled to keep its temperature under control, when running
+at full CPU load with no breaks.
+
All in all, running Gentoo on an old laptop is a job for enthusiasts,
+not for people who just want to use Linux for running applications. To
+work well there will need to be a lot of compiling, and this will
+probably have to be repeated every time there’s a significant update.
+Desktop and server computers are less problematic in this area, because
+they can usually run at full load without throttling. But, all the same,
+the amount of compilation shouldn’t be underestimated, even there.
+
Don’t be fooled by the people who tell you that they’re running
+Gentoo on decades-old Pentium systems and the like. Yes, this can be
+done. But it’s done as an enthusiast’s exercise, like running
+Doom on your refrigerator control panel – it’s possible, but
+it’s not for day-to-day use.
+
I know I’ve said it before, but Gentoo’s unique advantages are
+hardest to exploit where they would be most useful.
+
I also have to point out – for the purposes of the comparison that
+follows – that Gentoo has no automated installer. You’re expected to
+install it using command-line tools. However, the Gentoo documentation
+is excellent, and the installation isn’t difficult; the time it takes to
+read the documentation is a trivial part of the overall installation
+time, if you’re building much of the system from source.
+
MX Linux on old laptops
+
MX positions itself as a ‘midweight, stable, simple’ desktop
+operating system. It’s unashamedly a desktop system: there’s no
+server-class or embedded installation. Like Gentoo, MX does not force
+the use of systemd, and that’s usually a sign of a thoughtful approach
+to software provision. MX is available with various integrated graphical
+desktops, but its most popular variant is Xfce4. I use Xfce4 on Gentoo
+and Fedora already, so I have some basis for comparing the MX
+implementation with these alternatives.
+
MX is based on Debian, and can use software from the same binary
+repositories as Debian and its derivatives, like Ubuntu. However, MX has
+a whole stack of graphical software of its own, for customization and
+administration. I found this software to be well-implemented and
+functional; I sometimes even use it, rather than searching for
+a command-line way to do something, which is my usual approach.
+
MX is more mainstream that Gentoo in that it provides a graphical
+installer, as well as this bunch of graphical tools. This installer does
+allow for a measure of customization, particularly as regards the disk
+layout. It can cope with a disk that has existing partitions that have
+to be preserved – not all Linux installers can, sadly. However, I didn’t
+find the graphical partition manager any easier to use than simply
+running fdisk at the prompt, as I had to with Gentoo.
+Either you understand how Linux works in this respect, or you don’t: the
+available tools won’t explain it to you. The installer offers relatively
+little choice of which software to install – it’s optimized to create a
+general-purpose desktop system. Still, this does make the installation
+fast compared with, for example, Fedora. It’s not fair to compare the
+installation time with that of Gentoo, because Gentoo will have so much
+software to compile from source.
+
MX does not claim to be a minimal Linux and, although its basic
+desktop installation is slimmer than, say, Fedora with Gnome, it’s still
+general-purpose. If you’re trying to make every byte of memory and every
+CPU cycle count, you’re going to have a lot of post-installation work to
+do. For example, I only use ALSA for audio, not just because frameworks
+like Pipewire and Pulse can use a heap of memory, but because it just
+works better for me. The MX installer provides Pipewire, and all the
+applications it installs are set up to use it. It’s not difficult to
+disable Pipewire and go back to using ALSA – and I have – but this
+requires making a lot of changes to the desktop configuration. I have to
+replace the Pulse volume control with an ALSA one, for example. I also
+have to install apulse and change everything that runs
+Firefox to run apulse firefox instead, because the Firefox
+binary has no ALSA support. None of these changes are rocket surgery,
+but they do require a fair amount of knowledge, and some patience. Of
+course, some users of old laptops will be happy to live with the
+overheads of Pipewire, in exchange for the convenience it offers; but
+some of the stock MX installation is nothing but eye-candy, and an enemy
+of resource-efficiency. Again, to be fair to MX, I accept that MX does
+not position itself as a minimal Linux installation, and I’m aware that
+many will justifiably find my objections petty.
+
Both cosmetically and ergonomically, the Xfce4 set-up is slick. It
+looks nice, and the elements of the desktop are conveniently laid out.
+But, as I said, there are places where cosmetic impact has been chosen
+over efficiency. For example, the default install uses Conky to display
+date/time and system status information over the desktop wallpaper. This
+is eye-catching, but uses a non-trivial amount of memory and CPU
+(particularly updates every second) with little practical benefit on a
+low-resource system. Of course, Linux being what it is, you can turn
+this off, but you have to know how.
+
Compared with Ubuntu and Firefox I found it easier with MX to remove
+and disable things that I don’t need, to conserve resources. For
+example, I have no need for the tumblerd thumbnail
+generator and, when I did apt remove tumblerd to get rid of
+it, this didn’t strip out a whole chunk of the desktop with it. That’s
+the kind of thing that tends to happen with Fedora and Ubuntu – all the
+software is so tightly integrated that it’s hard to remove one thing
+without breaking something else. On MX, the Thunar file manager
+complained (just once) when I removed tumblerd, but
+removing it didn’t take Thunar with it.
+
MX’s real competitor, if that’s the right word, is not Gentoo, but
+the Xfce4 ‘spin’ of Fedora. Both are less bloated than the Gnome/KDE
+behemoths that most Linux maintainers favour, but I suspect that MX has
+the edge – just – where low resource usage is crucial. Also, Fedora uses
+systemd, for better or worse, while MX does not have to.
+
I feel I should point out that, although I’m no fan of systemd,
+removing it does not by itself make Linux better. No, really,
+it doesn’t. That’s what sets MX apart from, say, Devuan: MX has a
+character of its own, that goes beyond mere ‘lack of systemd’. Still,
+doing without systemd probably saves a hundred megabytes or so, and that
+might be significant on some hardware.
+
Comparison
+
In terms of RAM usage, Gentoo is the clear winner; in fact, it’s not
+even close. Running Xorg and the basic Xfce4 desktop components requires
+about 1.4Gb for MX, and only about 500Mb for Gentoo. That’s not just
+because there are fewer processes running – although that’s part of it –
+each process on Gentoo uses less memory than the same process on MX,
+even when these are of the same version. So, for example, the Gentoo
+build of Xorg (mostly compiled from source) used about 70Mb on my 2015
+Thinkpad, against 140Mb on MX (a pre-compiled binary, of course).
+
I found these memory differences applied across the board. Even
+Gentoo components that I hadn’t built from source used less
+memory than their MX counterparts. I don’t know if that’s because Gentoo
+binaries are built with better consideration for optimization, or if
+they just include less code, and potentially can do less. Whatever the
+explanation, the differences were evident.
+
Gentoo runs common applications a little faster, too – on the same
+hardware many applications were noticeably snappier than they were on MX
+(or Fedora, or Ubuntu, for that matter). Again, it’s not entirely
+obvious why this should be the case, for applications that
+weren’t built from source.
+
But…
+
There’s no getting away from the fact that Gentoo is a total bear to
+install on an old laptop, and an even larger bear (a grizzly?) when it
+comes to routine maintenance. I can set up MX and install most of the
+applications I use in a couple of hours. Setting up Gentoo took
+weeks on a 2015 laptop. This isn’t practical unless you have
+some way to spread the compilation load. Of course, you can mitigate
+this up-front set-up burden by configuring the system to be able to take
+advantage of more pre-compiled components, but this will have
+implications down the line.
+
I’ve heard of all sorts of tricks to make Gentoo’s compilation burden
+more manageable on a laptop. Apart from the obvious one of doing the
+compilation somewhere else, I’ve heard of people using cooling pads and
+desktop fans, standing their laptops end-on to improve convection,
+installing faster fans, and so on. All these measures are for the
+enthusiast, as it using one computer to compile for another. I just want
+to run applications.
+
I think the typical Gentoo user is like somebody who buys a Ferrari,
+and then spends every weekend fiddling with the engine, so it can do 180
+mph rather than 170. But I just want a Volkswagen.
+
So, stretching this metaphor a little further, I guess MX Linux is
+the Volkswagen Golf of the Linux world. It will be practicable for many
+people, and will work tolerably well with little maintenance. But it’s
+not going to, um… fly on ancient hardware. OK, this is a mixed metaphor,
+as well as a stretched one. But perhaps you get my point.
+
At present, I am persevering with MX Linux on my old laptops, but I
+haven’t ruled out the possibility of going back to Gentoo, should the
+need arise. I have back-ups of the Gentoo installation, of course,
+because the prospect of another three-week set-up time hardly bears
+thinking about.
+
+
+
+
+
+
+
+
+
+
diff --git a/gentoo_one_month.html b/gentoo_one_month.html
new file mode 100644
index 0000000..9e51516
--- /dev/null
+++ b/gentoo_one_month.html
@@ -0,0 +1,192 @@
+
+
+
+
+ Kevin Boone: One month with Gentoo
+
+
+
+
+
+
+
+
+
+
+
+
+
I recently installed Gentoo Linux on the computers I use on a daily
+basis. I had been aware of Gentoo for a long time, but this is the first
+time I have used it seriously.
+
In this article I’ll summarise what I’ve learned about the
+effectiveness of Gentoo in the month or so I’ve been using it: the good,
+the bad, and the ugly.
+
TL;DR – Gentoo has much to recommend it, but perhaps not enough for
+me to adopt it exclusively.
+
The good
+
First, with a bit (well, a lot) of fiddling, Gentoo Linux is really
+efficient. This is particularly noticeable on older computers. Part of
+that efficiency comes from being able to free myself from the bloated
+cruft that most Linux distributions impose on me. Another part, I think,
+comes from being able (sometimes compelled) to compile applications from
+source for the specific machine architecture.
+
I have installed Gentoo with no systemd, no Pulse Audio or similar,
+no heavyweight desktop, just Xfce. I use good old ALSA for audio, even
+for Bluetooth audio (which, I admit, is fiddly).
+
At idle, with nothing but the desktop running, CPU usage is
+essentially zero – as it should be –and memory usage about 400Mb. With
+Ubuntu in its default configuration (with Gnome, Pulse, and systemd) on
+the same hardware, idle CPU usage is at least 10%, and idle memory in
+gigabytes. This means that, with Gentoo, more computing resources are
+available to do real work.
+
Of course, on a modern, over-resourced system, these advantages might
+not be so apparent.
+
Second, the Gentoo maintainers do a great job of making software
+available. The fact that many things have to be built from source is an
+inconvenience, to be sure; but the positive side is that most
+applications work natively, without having to be wrapped up in Snap or
+FlatPak. I was particularly impressed by how easy it was to install the
+Spotify client – most other Linux distributions seem to have given up on
+Spotify, and expect their users to use Snap or FlatPak. And, although
+FreeCAD took three days (really – three whole days) to compile, it works
+without Snap/FlatPak and the like. And it works pretty well, on hardware
+that is far from the latest and greatest.
+
Third, the documentation is excellent. This is extremely rare in the
+open-source world, and this benefit should not be overlooked.
+
The bad
+
The worst feature of Gentoo is the constant compiling. Gentoo doesn’t
+have milestone releases: a new version of something is released every
+day. You don’t have to update every day, or even every week. But the
+longer you wait between updates, the more updates will need to be
+compiled. So, unless you never update, you’re going to have the
+some compilation workload, just distributed differently.
+
After the first month of routine use, I started an update and found I
+was going to get new versions of Thunderbird, Firefox, and LibreOffice,
+in addition to about a hundred smaller packages. There’s no realistic
+way I can rebuild all this stuff monthly – it will take several days. I
+don’t know whether this is a typical update schedule; that is, I don’t
+know how frequently updates are released for behemoth packages like
+Firefox. Perhaps I was just unlucky; but I don’t think so.
+
I have switched to using pre-compiled binary versions of Thunderbird,
+Firefox, and LibreOffice, although this negates some of the benefits of
+being able to build from source. I simply can’t have my computers
+compiling constantly for several days out of every month. I’m not sure
+they would even survive such abuse for long – running non-stop at full
+CPU capacity can’t be good for hardware. It’s also important to realise
+that pre-compiled binaries are only available for selected packages, and
+then only if they match your system configuration.
+
If you have several computers with similar hardware you can, in
+principle, spread the compilation load between them. This isn’t
+particularly easy to set up, but I can see how it might help certain
+people.
+
It’s ironic that the need for repeated compilation is most burdensome
+on low-powered hardware, which has the most to gain from it. Or,
+alternatively, Gentoo works best on systems that have the least need for
+the unique benefits it offers. It is this factor that primarily makes me
+reluctant to adopt Gentoo permanently.
+
The ugly
+
The Portage package management system is very complex, and it doesn’t
+always work. It can be particularly fiddly when different applications
+need different versions of the same dependencies. Python is a particular
+problem: there are various versions in circulation, and some
+applications only work with one of them.
+
To be fair, this is a problem for all Linux distributions, and it’s
+one that many distribution maintainers have given up trying to solve in
+an efficient way. The use of FlatPak, etc., makes it easier to maintain
+multiple dependency versions, at the cost of enormous duplication, and
+consequent waste of storage. Portage does at least try to address the
+problem, and mostly it’s successful. When it isn’t, it’s really
+painful.
+
To be fair, FlatPak and the like are available in Gentoo,
+although I think most users try to avoid them.
+
Dependency management is not an easy problem, but the additional
+complexity of building from source makes it even more difficult, because
+there are both build-time and runtime dependencies to manage. To its
+credit, Portage and the Gentoo maintainers make a pretty good job of it,
+given the complexity. But the Gentoo forums are full of cries for help
+from people who have tried to do an update, and ended up in a tangled
+mess of incompatible dependencies.
+
Is there an alternative?
+
I have a visceral distaste for the bloat and inefficiency of most
+modern Linux distributions. The problem isn’t just systemd, although
+this component seems to attract the most hate. Rather, it’s a general
+attitude to software management, brought about by the current low cost
+of hardware.
+
A computer with an 8-core, 3GHz CPU, 32Gb of RAM, and a couple of
+terabytes of storage doesn’t care how bloated its Linux is. It’s so
+over-resourced that the hardware will absorb the inefficiency. I have no
+problem running Ubuntu or Fedora on such hardware, although Gentoo is
+generally a little faster, at everything.
+
The real battle lies in the middle ground – those computers that are
+capable of doing useful work if treated with care, but not
+over-resourced. I have a very nice, lightweight Yoga laptop, that has
+only 8Gb of soldered, non-upgradeable RAM. It’s got a great screen and
+keyboard, and it’s light enough to carry everywhere. It’s been all
+around the world, stuffed in my backpack.
+
It does run the latest Ubuntu, in its default configuration, but it’s
+swapping to disk when doing nothing but showing the Gnome desktop. A
+couple of applications running, and it’s thrashing badly.
+
The alternative to Gentoo on a machine like this is to install
+Ubuntu, and strip out all the cruft. There’s no easy way to get rid of
+systemd but, apart from the objection on principle, it’s not hugely
+problematic for non-critical use these days. I can get rid of Gnome,
+Pulse, and a heap of other useless stuff, which frees memory and
+storage. It doesn’t free them to the extent that Gentoo does, but a
+useful improvement is possible.
+
I use Xfce rather than Gnome/KDE. It’s not as polished, but the
+Ubuntu implementation of Xfce4 is actually pretty good. It uses the
+Gnome network manager applet for network configuration, but that can be
+forgiven, since there really isn’t an alternative, unless you want to
+use the console interface.
+
Removing Pulse might mean using a wrapper like apulse to
+route Pulse output to ALSA, for those applications which only support
+Pulse (fortunately, there aren’t many).
+
The problem with this approach is that removing Pulse and Gnome also
+takes away a heap of other stuff, some of which I need; so I have to
+reinstall it. It’s almost certain that, when I update Ubuntu, I’ll get
+back all the cruft I removed, and have to trim everything down
+again.
+
So, for me, the choice lies between a pared-to-the-bone Ubuntu, which
+will need repairing with every update, or Gentoo, with its burden of
+compilation. The other alternatives seem to be either using regular
+Ubuntu, which is so memory-hungry as to be almost unusable, or throwing
+away a perfectly serviceable computer.
+
I can’t say I’m happy with any of these alternatives. The future of
+Linux looks increasingly like a descent into a kind of open-source
+Windows.
+
+
+
+
+
+
+
+
+
+
diff --git a/gentoo_pia.html b/gentoo_pia.html
new file mode 100644
index 0000000..3074f2e
--- /dev/null
+++ b/gentoo_pia.html
@@ -0,0 +1,198 @@
+
+
+
+
+ Kevin Boone: Using
+Private Internet Access VPN with Gentoo Linux
+
+
+
+
+
+
+
+
+
+
+
+
+
Using
+Private Internet Access VPN with Gentoo Linux
+
+
This is a short note about how to connect to the PIA VPN from a
+Gentoo Linux system. It’s not that difficult, but there’s no built-in
+support, either from Gentoo or from PIA. In fact, the PIA proprietary
+software installer warns that the host isn’t compatible. Still, the
+proprietary installer does work, with a bit of fiddling.
+
Alternatively, you can just use OpenVPN.
+
Using PIA’s proprietary
+software
+
I should point out that I’m running a systemd-free Gentoo with OpenRC
+as the service manager – I’m not sure whether the PIA installer is more
+friendly to an installation with systemd. Probably it is, frankly – most
+things are these days.
+
PIA supplies an installer for Linux in the form of a self-extracting
+binary with a name like pia-linux-xxx.run. The installer
+won’t allow itself to be run as root: it will expect to be able to run
+sudo to get root permissions. It installs in
+/opt, and it seems generally well-behaved; that is, it
+doesn’t appear to spray files all over the disk as some binary
+installers do.
+
What it does do, however, is attempt to install start-up scripts in
+/etc/init.d. These won’t work, because they refer to the
+wrong installation locations. It’s not difficult to create an init
+script for OpenRC, however.
+
The PIA software has a client-server architecture. There’s a service
+daemon, pia-daemon which would normally get started at boot
+time. Then there are a couple of clients that talk to that daemon. The
+graphical client – the part you see if you hit the ‘PIA’ icon in your
+Linux desktop – is an application called pia-client.
+
I found that neither part would start immediately after installation.
+It’s worth running both the daemon and the client from a console, so you
+can see the error messages. The obvious one was:
+
Error while loading shared libraries: libgssapi_krb5.so.2...:
+
The library it’s complaining about wasn’t part of my Gentoo install,
+and I get the impression that it usually won’t be. It’s easy enough to
+install, though:
+
# emerge app-crypt/mit-krb5
+
With that done, I was able to start pia-daemon and
+pia-client in a console, and connect to the VPN using the
+graphical application.
+
Of course, I’d rather have the daemon start at boot time, although it
+does take a few seconds to start, and then just use the graphical client
+to manage my VPN connection. That means creating an init script for
+OpenRC, although first I had to remove the defective
+/etc/init.d/pia and
+/etc/init.d/pia-daemon.
+
My OpenRC init script is below – this just goes in
+/etc/init.d/pia. I’m not expert on OpenRC, so it’s possible
+I’ve got something wrong; but this configuration seems to work for
+me.
+
#!/sbin/openrc-run
+
+name="PIA daemon"
+description="PIA daemon"
+command=/opt/piavpn/bin/pia-daemon
+command_background=yes
+pidfile=/run/pia-daemon.pid
+
+depend()
+ {
+ need net
+ }
+
I’ve put need net as a dependency to ensure that the
+network is basically up before starting the daemon. Again, I’m not sure
+that’s the textbook approach, but it works for me.
+
To make the daemon start at boot:
+
# rc-update add pia default
+
The PIA installer created a .desktop file for the
+client, so it shows up as a menu item in the desktop menu. or you can
+just run pia-client from a command line.
+
Using OpenVPN
+
The use of OpenVPN is described in the Gentoo
+documentation, but I think there’s an easier way. Whichever way you
+do it, you’ll need to install OpenVPN:
+
emerge --ask openvpn
+
Doing this creates a service called openvpn that you can
+start and stop using rc-service. However, you’ll need at
+least one configuration file. By default, OpenVPN reads
+/etc/openvpn/openvpn.conf.
+
The Gentoo documentation explains how to create this file manually.
+However, the OpenVPN website can create it for you. If you log into your
+account, and hunt around a bit (it’s not obvious where to look), you’ll
+see a link to “OpenVPN configuration generator”. Currently, it’s right
+at the bottom of the ‘Downloads’ page in small print. From here you’ll
+be able to select the VPN endpoint you want to connect to, the OpenVPN
+version, and the client platform. You must also select the TLS
+parameters to use. So far as I know they will all work; they just have
+different security implications. There’s nothing on the website to help
+you choose between them – PIA presumes, I guess, that if you’re setting
+up OpenVPN yourself, you’ll know what will suit your needs best.
+
The configuration page will generate a .ovpn file, that
+you can use as openvpn.conf. You can have as many of these
+files as you want, if you need to connect to different endpoints.
+However, you’ll need to come up with some way to switch between
+them.
+
I found that I had problems with IPv6 routing with the auto-generated
+configuration. Since I have little need for IPv6 at present, I turned
+IPv6 support off in openvpn.conf, rather than spending a
+day troubleshooting. These are the additions I made:
This will prompt you for your PIA credentials. If you prefer, you can
+put your credentials in a file, and link them using
+auth-user-pass in openvpn.conf. You’ll need to
+be a bit careful with this approach, of course, for the obvious
+reasons.
+
Although you can have the OpenVPN service start at boot time, there’s
+perhaps no need to do so, unless you actually want the VPN active all
+the time. I find it easier just to start and stop from a command
+line.
+
There is a bit of a gotcha here: I find that the OpenVPN service
+always starts with a warning message:
+
OpenVPN: service started, but is inactive
+
I see that warning whether the connection is working or not. When
+everything is set up correctly, the connection does get made, despite
+the warning. In the early stages of set-up, it’s certainly worth
+checking that you really do have a VPN connection. If it’s working,
+ifconfig will show a tun0 interface, with an
+IP number. If it doesn’t connect, you’ll have to look in the system log
+to see why.
+
It’s worth bearing in mind that setting up OpenVPN for PIA is not in
+any way Gentoo-specific. Although OpenVPN is pretty complicated, there’s
+a huge amount of information available about it.
+
Which to use?
+
In my experience, the PIA proprietary software outperforms OpenVPN in
+almost all respects. Of course, if you have an ideological objection to
+proprietary software, you won’t want to use it, however good it is.
+Although PIA doesn’t support Gentoo, their proprietary software isn’t
+too difficult to install. The fact that the client lets you choose your
+VPN endpoint from a clickable list, without fiddling about with
+configuration files and certificates, is a bonus. If you’re running
+Linux without a graphical desktop, PIA provides a command-line client as
+well.
+
OpenVPN is not a particularly easy thing to set up, but it’s a
+transferable skill. It’s compatible with most VPN services so, provided
+you can get the connection parameters and certificates, you’ll be able
+to change VPN providers easily, and even use multiple providers on the
+same computer. In addition, OpenVPN doesn’t require anything to be
+started at boot time (although it’s possible). To be fair, the PIA
+proprietary software doesn’t have to be started at boot time, either,
+but it usually is.
+
+
+
+
+
+
+
+
+
+
diff --git a/headphonejack.html b/headphonejack.html
new file mode 100644
index 0000000..e3f20d8
--- /dev/null
+++ b/headphonejack.html
@@ -0,0 +1,180 @@
+
+
+
+
+ Kevin Boone: They
+don’t make ’em like that any more: the 3.5mm headphone jack socket
+
+
+
+
+
+
+
+
+
+
+
+
+
They
+don’t make ’em like that any more: the 3.5mm headphone jack socket
+
+
+
“If it ain’t broke, replace it with something that is.”
+
+
About five years ago I was suddenly, unexpectedly taken ill. Not just
+‘that’s a bit nasty’ ill, but ‘prepare for the worst’ ill. One thing
+that kept my spirits up in hospital, in the long watches of the night,
+was listening to comedy shows and audiobooks. I used my smartphone for
+this, since I had it with me, and a pair of old, wired headphones that I
+just had time to grab on my way to the ambulance.
+
I survived, of course, as evidenced by my continued ramblings on this
+site. But it was an unpleasant experience, made just a little better by
+a simple piece of technology: the 3.5mm headphone jack.
+
Now, of course, I do own wireless headphones and earbuds – I think
+almost everybody does. I also own several of those irritating USB
+dongles, that provide a 3.5mm port for devices that don’t have one. But
+here’s the problem: I can’t use my Bluetooth earbuds while they’re
+charging. And I can’t easily charge my phone whilst it’s connected to
+the USB dongle. In a critical-care facility, it’s hard enough to find
+one free mains socket to connect a charger to, let alone two.
+
In the debate about the benefits of wired and wireless headphones, I
+doubt anybody is thinking “What if I get sick?” It certainly wasn’t
+something I was thinking, either, until I actually did. Still, that
+experience made me think about all the advantages of having a headphone
+jack, some of which I’d thought about before, and some I hadn’t.
+
+
Sound quality is better. Maybe that won’t always be the case, but it
+is now. All decent headphones have an analog jack.
+
Almost any headphone or earphone will work with almost anything with
+a standard jack socket. That isn’t the case for USB dongles. Maybe that
+won’t always be the case, but it is now.
+
I can use the headphone jack to connect my phone or tablet to an
+amplifier, should the need arise. For certain headphones, that need
+does arise.
+
I can charge my phone whilst listening to music, which I do all
+day.
+
I don’t have to carry around a stupid dongle, which I will
+invariably lose.
+
Ordinary wired earbuds can be small enough to wear whilst sleeping.
+I’ve never seen Bluetooth earbuds that are.
+
+
The headphone jack is a “just works” kind of technology: there’s
+nothing complicated about it, and its not encumbered by patents, so
+anybody can make compatible equipment.
+
So, although “What if I get sick?” probably isn’t at the top of the
+list of questions that will guide your buying decision, we have to
+wonder what other things we lose, by replacing a well-established,
+robust technology with a complicated, flaky one.
+
On the other hand, the advantages of doing away with the
+headphone jack are:
+
+
Your cellphone is about ten pence cheaper.
+
Er… that’s it.
+
+
What makes the loss of the headphone jack so hard to bear is that
+it wasn’t done for the consumer’s benefit. To be sure,
+manufacturers made certain claims about the alleged benefits of losing
+the jack, but few of them stand up to much logical scrutiny.
+
The first manufacturer to make a point of dropping the headphone jack
+(I believe) was not Apple – as is commonly believed – but Oppo, and back
+in 2014. Their reason for doing so was at least a credible technical
+one: they said it made their phones about half a millimetre thinner.
+Maybe that was a selling point, maybe it wasn’t. But Apple couldn’t fall
+back even on this claim, because people found ways to fit a 3.5mm jack
+socket into the iPhones that lacked one, and even posted videos on
+Youtube showing how they did it. It wasn’t easy, but it was clearly
+possible. If Apple genuinely thought that omitting the jack
+would leave more room for other features, they didn’t actually provide
+any.
+
It’s notable that a number of companies mocked Apple’s decision to
+drop the headphone jack, before quietly doing the same themselves:
+Samsung and Google in particular. Samsung even cynically tried to
+withdraw all the advertising in which they had mocked Apple. Of course,
+nothing is ever really gone on the Internet, so we can continue to
+marvel at Samsung’s barefaced duplicity.
+
Some manufacturers claimed that the presence of the headphone jack
+made it difficult to keep their phones waterproof; but there’s a whole
+range of good-quality phones from around 2019-2020 that are waterproof
+to a reasonable degree, without sacrificing the jack.
+
No. All of these weak excuses are simply distractions from the
+real reason Apple, Samsung, and Google dropped the headphone
+jack: they all have a substantial investment in the manufacture of
+wireless headphones.
+
Apple owns Beats – this was a multi-billion dollar investment, in a
+company that manufactures Bluetooth headphones.
+
Samsung owns Harmon, which is know for the Harmon-Kardon and JBL
+audio brands. Again, these were multi-billion dollar acquisitions for
+Samsung, in companies with a strong interest in wireless audio.
+
Google owns Synaptics, Dysonics, RevX, and Tempow, all of which are
+active in the development of wireless audio. Google also hired Peter Liu
+from Bose, who was one of the original developers of the Bluetooth Low
+Energy specification.
+
It is very much in the interest of companies like Apple, Samsung, and
+Google to encourage their customers to buy into wireless Bluetooth
+headphones. Or, better yet, to force them to do so, by taking away the
+means to do anything else. After all, their executives have to justify
+the billions of dollars they’ve spent, acquiring suppliers and
+developers of wireless audio equipment.
+
The 3.5mm headphone jack has almost nothing to recommend it,
+technically speaking. It was criticized by hi-fi enthusiasts almost from
+its inception. The only things it has going for it are its simplicity,
+and the fact that everybody uses it. Well, everybody outside the world
+of mobile gadgets, anyway. This simplicity and ubiquity is a great
+benefit to consumers, which is why Apple, et al., don’t want to provide
+it – they have nothing to gain if consumers spend less money on their
+products.
+
The loss of the headphone jack would have hurt even if there were
+good technical reasons for it. As it turns out, there are none – it’s
+just another cynical way for big businesses to gouge consumers. A side
+effect of their strategy is that wired headphones themselves are
+becoming less available, as cellphone users were until recently major
+purchasers of these devices. It’s not difficult to get top-quality
+headphones with a jack – they all have one; but there are fewer and
+fewer mid-priced wired earbuds on the market.
+
The way to discourage companies behaving this way is for us all to
+take our business elsewhere. It’s still possible to get decent
+cellphones from Motorola, Asus, and Sony that retain the 3.5mm jack. I
+don’t believe that anything recent from Apple, Samsung, or Google has
+one, for reasons I explained earlier. Still, there are older cellphone
+models from these suppliers that do have a jack, and which are still
+good phones. I own the Samsung S10+, S10e, and Note 9, for example.
+They’re a few years old, but they still do everything I want and
+more.
+
If you’re a fan of the 3.5mm jack, it’s time to vote with your
+wallet.
+
+
+
+
+
+
+
+
+
+
diff --git a/install_lineageos.html b/install_lineageos.html
new file mode 100644
index 0000000..79e30be
--- /dev/null
+++ b/install_lineageos.html
@@ -0,0 +1,314 @@
+
+
+
+
+ Kevin Boone: Some
+notes on installing Lineage OS on modern-ish Samsung Android
+devices
+
+
+
+
+
+
+
+
+
+
+
+
+
Some
+notes on installing Lineage OS on modern-ish Samsung Android
+devices
+
+
This article provides some supplementary notes about installing
+Lineage OS on modern-ish Samsung devices, and rooting them (in that
+order). At the time of writing, the most modern device that Lineage
+supports is the S20 FE; the oldest is probably the S8. Unofficial
+versions exist for some other devices.
+
I’m not trying to supplant the official Lineage
+instructions, which are already quite thorough. All I seek to do is
+to add my own experiences, in the hope that I can save somebody some
+time and frustration.
+
+
Note
+I install Lineage OS on my cellphones to de-Google them. It’s certainly
+possible to install Google stuff under Lineage OS, and also to install
+stubs of Google services to fool applications into thinking that these
+services are available. However, I’m specifically trying to avoid Google
+services, so I won’t be describing how to do any of that in this article
+(because I don’t know).
+
+
Please be aware that there’s a lot more I don’t know about the
+process of installing Lineage OS. I have successfully installed it on
+the S10, S10+, S10e, Note 9, and S5e (Wifi). The last of these is a
+tablet; the rest are smartphones. The installation process is the same
+for all of them once you get the custom recovery software installed (see
+below). It’s possible that you can install the custom recovery the same
+way on all devices, too – but the Lineage OS instructions are different
+in this regard for different devices, and I’ve generally followed
+them.
+
In particular, some devices are documented as using the Odin tool to
+install the custom recovery, others use Heimdall. More on this point
+later.
+
If you’re unsure about whether installing Lineage OS is right for
+you, please see at least the ‘Things to watch out for’ section at the
+end of this article. It might help you make up your mind.
+
Basic principles
+
Assuming that you’re starting from a stock, up-to-date Samsung
+device, installation and rooting has four main parts.
+
+
Unlock the bootloader. This is essentially irreversible, and will
+delete all data. Make a backup first.
+
Install a custom recovery image. The purpose of the custom recovery
+is to write the operating system image (and perhaps other things) to the
+device’s flash memory. Over the years there have been many custom
+recovery applications; the most common is probably TWRP. Lineage OS has
+its own custom recovery, and that’s all I’ve used with Samsung devices.
+Samsung devices have a built-in recovery mode, but it will only install
+Samsung software. This – installing the custom recovery – is the part of
+the process that requires Odin or Heimdall.
+
Use the custom recovery image to install Lineage OS. This part of
+the process uses the adb sideload command.
+
Install Magisk to provide root access, if required.
+
+
I should point out that installing Lineage OS does not require a
+rooted device, and it will not root the device itself. Lineage OS used
+to have its own mechanism for enabling root access, but the maintainers
+now recommend Magisk.
+
Prerequisites
+
You’ll need:
+
+
A computer with the adb utility. This is released by
+Google as part of the Android ‘platform tools’ package, and is widely
+available for most platforms. This is a command-line utility, and I’m
+assuming you know how to run adb at a prompt on your
+computer.
+
Either Odin or Heimdall, depending on the device. It may be
+possible to use these interchangeably, but I have not been brave enough
+to try. Odin is available as a graphical utility for Windows and Mac,
+and a command-line utility for Linux. The Linux version is called
+odin4 and is available, at the time of writing, from here.
+However, you might need to do some web searching, as these tools can be
+a bit elusive.
+
A USB C cable, and a working USB port on the device. The USB ports
+on Samsung devices are precarious, and data transfer tends to fail after
+a while, even if charging still works. I don’t know any way to install
+Lineage OS without a working USB port.
+
+
I would strongly recommend that, before going any further, you learn
+and practice the button-press combinations needed to be able to get into
+the various “special modes”. You’ll need to be able to do these fluently
+and at speed, or you’ll end up repeating various parts of the
+installation, perhaps several times.
+
How to get into special
+modes
+
Installing Lineage OS on a Samsung device requires being able to
+start the device in particular special modes: “download” mode, and
+“recovery” mode.
+
For full details you’ll need to check the specific instructions for
+your device on the Lineage OS website. However, I’ve found that the
+following work for all my Samsung cellphones.
+
If you’ve already installed Lineage OS – you’re trying to install
+Magisk for rooting, for example – the easiest way to get into the
+special modes is to do so directly from the Lineage user interface. To
+do this, enable the ‘Advanced restart’ feature from the Settings app.
+Then the power-off menu will have an additional ‘Restart’ button that
+allows direct entry into download mode or recovery mode.
+
If you don’t have Lineage OS installed yet – and you probably don’t,
+if you’re reading this – then you’ll need to learn and practice the
+finger-breaking button combinations below.
+
Download mode
+
This is the mode you need for installing a custom recovery image
+using the Heimdall or Odin utilities. With the device powered off, press
+and hold the volume-down and bixby buttons, then plug in the USB cable.
+Unless you have three hands, it takes some practice to complete these
+steps.
+
Recovery mode
+
This is the mode you’ll need for installing Lineage OS using the
+custom recovery menu. With the device powered off, press and hold
+volume-up, bixby, and power, for a long time, until the recovery screen
+appears.
+
If you’re currently running Android or Lineage OS, and your device is
+connected to a computer, you might be able to get into recovery mode by
+running adb reboot recovery. If this works, it might put
+the recovery utility directly into ‘sideload’ mode (see below), which
+will save a couple fo button-presses. But I’ve found it doesn’t always
+work for me.
+
Hard power-off
+
To power off when there seems to be no other way to do so, press and
+hold the volume-down and power buttons together, and hold them for at
+least ten seconds, or until the screen switches off.
+
This isn’t just for emergencies – there are steps in the official
+documentation that require a hard power-off as part of the
+installation.
+
From download mode to
+recovery mode
+
This is the most fiddly mode switch of all, and you have to be able
+to get it right. If you fail, you’ll probably have to install the custom
+recovery from scratch and try again.
+
With the device in download mode, and having finished downloading the
+custom recovery, do a hard power-off followed immediately by
+recovery mode. That is, hold volume-down and power until the screen
+switches off and then, within half a second or so, press and hold the
+volume-up, bixby, and power buttons together.
+
This process is very fiddly, and I get it wrong as often as right,
+even after practice.
+
Bootloader unlock issues
+
The first step in the installation of Lineage OS, or any custom ROM,
+is to unlock the bootloader.
+
The Lineage OS instructions say to enable developer mode on the
+device, and then find the ‘Enable OEM Unlocking’ control in the Settings
+app. What they don’t say is that you won’t even see this control if
+there is no WiFi connection. I have no idea why this is the case, and
+I’m not sure whether anybody else does. I’ve been caught out by this
+many times, because installing Lineage is usually the very first thing I
+do on a cellphone. So I usually rush through all the initial set-up
+steps, to get as quickly as possible to the Settings app. But if you
+don’t enable WiFi, you won’t be able to enable the bootloader unlock
+mode when you get there.
+
There are many reports of how certain devices don’t even have the
+‘Enable OEM Unlocking’ control. These seem to be devices that are locked
+to some specific cellular carrier. I don’t think it’s possible to
+proceed without completing this step successfully.
+
+
Note
+Although it’s stated in the Lineage installation instructions, it bears
+repeating: unlocking the bootloader will erase all data on the device
+and, even if you proceed no further, certain features of the device will
+be lost, particularly those concerned with media DRM. There’s really no
+going back, beyond this point.
+
+
Installing the custom
+recovery software
+
Depending on the device, the instructions say to use Odin or
+Heimdall. Although these utilities have many functions, only one is
+needed to flash the recovery utility. The second-trickiest part of the
+process is getting into ‘download’ mode to begin it (see above).
+Thereafter it’s just a matter of running the appropriate Odin or
+Heimdall commands, which are well-documented.
+
The trickiest part is getting from download mode to recovery mode
+without actually starting Android. The finger-breaker for this is also
+described above and, although I know I’m repeating myself, it’s good to
+practice it until you can do it without getting your fingers in a
+tangle.
+
The problem is that if you leave download mode without
+entering recovery mode, the device will reboot normally, and restart
+Samsung’s Android. Doing this will overwrite the custom recovery with
+the Samsung stock recovery, which is no use to us. So it’s crucial to go
+straight from download mode to recovery mode.
+
Using the
+custom recovery to install Lineage OS
+
When you boot into recovery mode, either now or after installing
+Lineage, you should see the Lineage OS custom recovery. This recovery
+utility has large, white text on a purple background and is
+touch-sensitive. If you see lots of tiny text, and the menus
+are not touch sensitive, then you’re not running the Lineage recovery,
+but the stock Samsung one. This is no use for installing anything except
+Samsung software.
+
So if you’re in this situation, you’ll need to retry the installation
+of the custom recovery.
+
Installing Lineage OS from the custom recovery is easy, and
+well-documented. You’ll need to use the recovery menus to perform a
+factory data erase, and then ‘Apply update’. You can then use the
+adb sideload command to send the appropriate Lineage OS
+image to the device. This will take a few minutes, and then you’ll be
+back at the recovery menu.
+
It’s possible to install Google stuff at this point and, according to
+the documentation, you need to do so before starting Lineage OS for the
+first time. I’ve never done this, so I can’t comment further. I just
+reboot here, and start Lineage OS.
+
Rooting using Magisk
+
This is now the approved way to root Lineage OS. Magisk has two parts
+to it – some firmware which is executed at boot time, and a regular
+Android app. I believe there are various ways to install these
+components; I’m just describing the method that works for me.
Enter recovery mode, and install the APK using
+adb sideload, as you did to install Lineage OS itself. It’s
+normal to get a ‘Signature verification error’ message at this
+point.
+
Reboot into Lineage OS.
+
You’ll see a new app, entitled ‘Magisk’, with a generic Android
+logo. This is just a stub to download the rest of Magisk. Run this
+app.
+
When prompted, tell the downloader stub to download the full
+app.
+
When the full app has installed, run it. You’ll be told that
+additional steps are required.
+
When prompted, choose ‘Direct installation’. This will install some
+software and reboot the device.
+
+
To test that root access is now possible, I usually install a
+terminal emulator like Termux. Then I run su in the
+terminal. This should cause Magisk to pop up a confirmation box and, if
+you grant permission for root access, the prompt in the terminal will
+change from $ to #. Other software that
+requires root access should also work, although you’ll need to approve
+their access when Magisk prompts.
+
Things to watch out for
+
+
If you get error messages about permissions from adb,
+bear in mind that the missing permissions are probably on the host
+computer, not the Android device. You might need to run adb
+as an administrative user.
+
If you’re installing Lineage without Google services, you’ll have to
+find some way to install apps – if you need any – without using the
+Google Play Store. A good starting point is to install the F-Droid app,
+which you can get as an APK from the
+F-Droid website using the built-in web browser. You can get apps
+from other places in APK format, but you shouldn’t expect too much –
+most proprietary apps assume the availability of Google services.
+
On some Samsung devices, you can expect to get a warning message
+about the risks of using unofficial software every time you reboot. I
+don’t know why this applies to some devices and not others.
+
Lineage OS can be upgraded in place between minor versions, e.g.,
+from 20.0 to 20.1. However, there is no simple upgrade between major
+versions – you’ll have to start from scratch, and be prepared to
+reinstall all your apps and data.
+
Unlocking the bootloader will trip the ‘Knox fuse’ on the main
+board. Even if you can find some way to reinstall the original Samsung
+firmware – which is difficult – the Knox fuse is a physical device that
+cannot be reset in software. You might struggle to get a cellphone
+repaired under Samsung’s warranty terms, if you’ve tripped the Knox
+fuse. There really is no going back.
This article is about container-awareness in the Java virtual
+machine, particularly in Java 17. Container-awareness is a method for
+the JVM to detect the resources it has available, when running in a
+container, and adjust itself accordingly. In the article I will discuss
+only memory resources. Everything here applies equally well to
+CPU resources; I’m using memory simply because it’s easier to monitor
+than CPU.
+
The ultimate goal of this article is to show how OpenShift resource
+limits and requests affect the JVM. However, to get to that point, we
+need to start by looking at simpler scenarios.
+
The JVM heap and its sizing
+
The Java virtual machine uses a flexible heap with garbage collection
+for its object management. Broadly speaking, the JVM allocates
+newly-created objects in the heap, where they remain until they are
+elligible to be removed by the garbage collector. The heap can expand
+between its initial and maximum values, according to the number of
+objects in play at a particular time.
+
It’s always been possible for the application designer to set the
+initial and maximum java heap size using the -Xms and
+-Xmx command-line switches. For example:
+
$ java -Xms512m -Xmx2048m ...
+
This is a reasonable way to control the heap size in a traditional
+operating system, that is, one that is not based on container
+technology. The operating system has a specific, well-known amount of
+memory available to it, and it’s unlikely to change much from day to
+day. Changing the JVM’s memory allocation is likely to amount to hacking
+on some configuration file or script.
+
Container technology, however, provides for flexible memory
+management. Administrators of container-based systems like Kubernetes or
+OpenShift expect to be able to tune an application’s memory usage by
+tuning the amount of memory assigned to its container. Moreover, tuning
+memory usage at the application might be complicated, if the
+configuration is buried somewhere inside a container image.
+
Later versions of Java have a container awareness feature,
+which allows the JVM to learn (or, at least, guess) the amount of memory
+present in the container. By ‘later’ I mean, perhaps confusingly, later
+releases of Java 1.8, Java 11, and Java 17. That is, the changes were
+back-ported from Java 17 to the later releases of earlier versions. If
+you see what I mean. The point I’m making is that container awareness is
+available on all Java platforms today, even as far back as Java 8.
+
+
Note Despite all this, there are subtle differences
+between Java versions and, in this article I use Java 17
+exclusively.
+
+
As a result, we generally advise against using fixed heap settings
+(-Xmx and -Xms) when running Java in a container
+platform.
+
But what happens if the designer sets no heap limits at all? The JVM
+has to allocate some heap range, but how? And how will the
+JVM’s behaviour depend on the container’s memory settings?
+
In general, if no specific heap values are given, the Java JVM will
+apply defaults. Historically, Java had a rather complicated algorithm
+for working out which heap defaults to use; it depended on whether we
+were running in ‘server’ or ‘client’ mode, the amount of platform
+virtual memory, whether we were running a 32-bit or 64-bit platform, the
+garbage collection algorithm, and so on. The defaults changed from one
+Java version to another and, although they were documented, it wasn’t
+always easy to find the documentation. It made sense, in the
+pre-container days, to use at least the -Xmx switch for all
+non-trivial applications.
+
These days, for most reasonable platform configurations, the
+defaults are much simpler: the maximum heap is 1/4 of the platform’s
+reported memory size, and the minimum heap is 1/64, up to a limit of
+512Mb. There are complications related to set-ups with very large, and
+very small, memory sizes; but these won’t affect most installations.
+
Using a
+program to report memory and heap allocations
+
In this article, I’ll be using a simple Java program to report on the
+platform memory and allocated heap sizes. It will get this information
+from JMX MBeans. I won’t present the source code here – it’s very simple
+and, as always, it’s available from my GitHub
+repository.
+
As well as the Java source, the application bundle contains scripts
+and a Dockerfile, to build the code into a container. I’ll
+demonstrate its use in a Podman container (although Docker will work
+just the same) and also in an OpenShift pod.
+
When I run the program on my workstation, here is the output:
The ‘native’ memory size is what JMX reports to the program as the
+system memory. My workstation has 32Gb of RAM, so that value looks about
+right. The maximum and initial heap values are 1/4 and 1/64 of that
+figure, more or less. So that’s as expected.
+
Just to prove that the program works, I can run it with specific heap
+settings:
You’ll see that the heap values now reflect the -Xmx and
+-Xms settings, so I’m reasonably sure that the simple
+program works properly.
+
Heap mamangement in a
+container
+
Let’s try this application in a container. I’m using Podman, but
+Docker will behave the same.
+
In the application bundle is a Dockerfile for building
+the container. It looks like this:
+
FROM openjdk:17-alpine
+ADD target/java_docker_test-0.0.1-jar-with-dependencies.jar .
+ADD ./run.sh .
+ENTRYPOINT ["/run.sh"]
+
The FROM line specifies that my image will be based on
+the Alpine Java 17 base image. Alpine is a lightweight Linux version,
+popular in containers.
+
To the base images I add the Java application’s JAR file, and a
+script run.sh. This script just runs the Java application,
+exactly as I did on my workstation. Here’s what it looks like:
You see that the results are exactly the same as running outside the
+container. There’s no reason they should be different:
+podman will not constrain the container’s memory, unless we
+ask it to.
+
So let’s do that – let’s fix the values of RAM and swap.
Note
+Be aware that ‘–memory-swap’ really means ‘RAM plus swap’. This
+configuration allocates 1Gb RAM and 2Gb swap.
+
+
The program reports its memory size as 1000Mb (which matches the
+--memory argument) and, again, the heap sizes are 1/4 and
+1/64, as always.
+
How does the JVM know how much memory its container has? We can
+experiment with this by logging into the container, by setting a shell
+as the --entrypoint:
Let’s look at the system memory within the container:
+
/ # free
+ total used free shared buff/cache available
+Mem: 32224352 4529920 2732716 5085020 24961716 22134684
+Swap: 8388604 170248 8218356
+
These Mem and Swap figures match my workstation totals, not the
+container’s. The container only has 1Gb available, and somehow the JVM
+has worked this out.
+
Linux containers using a technology called ‘cgroups’ (control groups)
+for managing resources at the per-process level. Somewhat irritatingly
+for our purposes, there are two different versions of cgroups in
+circulation: v1 and v2. Most modern Linux kernels support both, but
+characteristics of the underlying platform dictate which to use.
+podman running on my workstation uses cgroups v2 but, as
+we’ll see, containers on my OpenShift installation uses v1.
+
Both versions do much the same thing; the irritation is that the
+filenames we’ll use to report cgroups metrics are different.
+Unfortunately, if you do a lot of work with cgroups, you really have to
+be familiar with both versions, and we’ll see both in use in this
+article.
+
There is an article on
+cgroups (v1) elsewhere on my website, which might be of interest to
+readers who want to see it in action.
+
From within the running container, we can get the memory and swap
+limits from cgroups like this:
You’ll note that these agree with the limits I applied to the
+podman command.
+
It’s worth noting at this point that, so far as the JVM is concerned,
+the swap value is irrelevant. By default, when the JVM queries
+its memory allocation to calculate heap sizes, it uses ‘RAM’, not swap.
+Of course, it’s not real RAM, it’s a container simulation of RAM.
+
So that’s how the heap allocation works in a Podman container: the
+JVM uses cgroups to determine the amount of allocated RAM, and sets the
+heap limits to 1/4 and 1/64 of that figure.
+
It’s worth bearing in mind that cgroups enforces limits on
+the JVM. The JVM only reads the cgroups metrics to set the heap sizes
+appropriately. You could set -Xmx2000m in a container with
+only 1Gb allocated, and the JVM would try to create a heap that large –
+but it wouldn’t be allowed.
+
Heap
+management with OpenShift resource requests and limits
+
Now let’s try the same thing on OpenShift.
+
There are many ways to deploy a Java application in an OpenShift pod,
+but I want to make the test conditions as close as possible to the
+Podman/Docker case. To that end, I’ll do a Docker deployment on
+OpenShift, using the same Dockerfile and Java code as we
+used with Podman. Here’s how to do that.
This will take a little while, particularly the first time. All being
+well, you’ll see the same Docker build steps in the output as we saw
+when building for a local Podman installation.
+
+
Note
+This step, oc start-build, is what you’ll need to repeat,
+if you change the Java program or scripts.
+
+
All being well, you’ll see that a build pod was created, and it
+completed. It should have created a new OpenShift container image called
+java-docker-test.
+
To get this image into a running pod, we can create a deployment from
+the image.
+
$ oc new-app java-docker-test
+
Use oc get pods to see what new pods have been created;
+you should see a pod with a name of the form
+java-docker-test-XXXXXXXXXX-XXXXX.
+
To see the application’s output, have a look at the logs from this
+pod:
The native memory size is reported as 6.6Gb. The heap sizes are, as
+always, 1/4 and 1/64 of this. But why 6.6Gb? Almost certainly the
+OpenShift node I’m running this pod on has much more memory than this. I
+didn’t apply any limits to the pod (yet). So why this figure?
+
It’s not just a platform default – this is a calculated value. But
+I’ll come back to how the calculation is done later, as it’s a bit of a
+distraction.
+
Let’s apply a resource request and limit to the deployment. The pod
+will restart, and we can look at the logs again.
Because we’re using cgroups v1 here, the name of the file containing
+the limit is not memory.max, as it was earlier – it’s
+memory.limit_in_bytes. The value is 1Gb, which is as we set
+on the command line.
+
The mystery I deferred is how the JVM arrived at a memory size of
+6.6Gb when we didn’t set any limit. According the OpenShift
+documentation, a pod with no resource limit has unbounded resources.
+That is, the JVM should be able to allocate memory until the OpenShift
+node runs out of RAM. But, clearly, the JVM has arrived at some
+heap sizing. But how?
+
Log into the pod again, and look at the cgroups limit in this
+scenario:
That’s a truly colossal number – scarcely a limit at all. It’s as
+close to ‘unbounded’ as makes no practical difference. To see where the
+JVM’s memory limit comes from, let’s ask it. Log into the running pod
+and run:
+
$ java -Xlog:os+container=trace -version
+...
+[0.031s][trace][os,container] Memory Limit is: 9223372036854771712
+[0.031s][trace][os,container] Non-Hierarchical Memory Limit is: Unlimited
+[0.031s][trace][os,container] Path to /memory.stat is /sys/fs/cgroup/memory/memory.stat
+[0.031s][trace][os,container] Hierarchical Memory Limit is: 7125319680
+...
+
You can see that the JVM has determined that its memory allocation is
+unlimited. So it’s parsed the file memory.stat, and found
+the value of the “hierarchical memory limit”.
+
The what now? This isn’t a concept that I really have space to
+explain in detail here. In essence, OpenShift containers run as
+processes in a hierarchy of resource constraints. The ‘hierarchical
+limit’ is obtained by taking the limit on the container’s parent
+process, and subtracting the memory used by other child processes (other
+pods) with the same parent.
+
Doing this math gives us about 71 trillion bytes, or 6.6Gb.
+
If you think this is an odd way to work out a memory limit, I’d be
+inclined to agree. But what else can the JVM do? Java has no notion of
+an unconstrained heap: it has to have some maximum. The JVM could just
+use the platform RAM as its base figure, as it does outside a container.
+But it isn’t safe for a JVM to assume that it has that much RAM
+available – it could be enormous, and there are too many other
+containers competing for it. I guess the ‘hierarchical limit’ is the
+best compromise that that JVM maintainers could think of.
+
In practice, I think we should assume that a JVM running in a pod
+with no limit will get what amounts to a random limit. In
+almost all circumstances it will make sense to apply a limit.
+
Resources limits and
+requests
+
In the previous command I set a resource limit and a
+resource request, both to the same value. My experience is that
+Java developers working on OpenShift often don’t understand the
+difference between these values.
+
I’ve noticed a vague expectation that the ‘limit’ figure should
+constrain the maximum heap, and the ‘request’ figure the minimum. It
+seems plausible that this should be the case but, as we’ve seen, that
+isn’t the case. In the previous example I set both request and limit to
+1Gb, but the minimum heap value remained at 16Mb – the usual 1/64 of the
+total memory.
+
In fact, the JVM only sees the ‘limit’ figure. If there is, in fact,
+a way for a JVM to find the ‘request’ value from the platform, I don’t
+know of one. The request figure has no effect on the JVM heap size,
+maximum or minimum.
+
However, it does have implications. The request figure is used by the
+Kubernetes scheduler to decide which node to run the pod on. If
+we set a low memory request, then the scheduler will allow the pod to
+run on a node with low memory availability. The JVM will still get the
+heap allocation determined by the limit figure, however large that is.
+But if the pod is running on a node with low memory availability, it’s
+possible that the pod will fail at runtime, because there just isn’t
+enough memory to satisfy the limit.
+
Setting a low resource request is a ‘neighbourly’ thing for an
+application installer to do. It allows for fair distribution of pods
+between nodes. However, it’s probably not in the application’s best
+interests to have ‘limit’ and ‘request’ values that are hugely
+different. If we set a limit of, say, 1Gb, we’re doing so in the
+expectation that the pod might, sooner or later, use 1Gb. If the request
+value is, say, 128Mb we’re saying that the pod will be satisfied with
+128Mb; if you’ve set a limit of 1Gb, that’s likely not to be the
+case.
+
It takes care and experience to determine good resource requests and
+limits for an application, and probably some testing. I usually
+recommend that installers set the same values for limit and request if
+they can. In practice, though, we often can’t, because if every
+application component does that, resources could be badly
+over-committed.
+
Percentage heap allocations
+
My experience is that the larger the memory allocated to a pod, the
+less sense it makes to allow the JVM to assign only a quarter of it for
+its heap. To be sure, most Java applications will use memory outside the
+heap. A common consumer of out-of-heap memory is thread stacks, and that
+can be very significant on heavily-loaded, concurrent applications. Many
+Java libraries allocate memory outside the heap – Netty is a good
+example of this. The Java classes that expand zipfiles also use non-heap
+memory, although this might not be obvious.
+
All things considered, though, if you’re allocating 32Gb of RAM to a
+Java pod, you’re probably expecting at least, say, 28Gb of it to be
+available to the JVM heap. Of course, we can give effect to that
+expectation by using the command-line argument -Xmx28000m,
+but that’s something we discourage, as I described earlier.
+
We would like the administrator to be able to tune the amount of
+memory available to a Java application by tuning the pod’s allocation,
+rather than by rebuilding the image with new code (even if it’s only a
+new start script).
+
In scenarios like this, it can make sense to allocate a specific
+fraction of the memory to the heap, rather than an amount. For
+example, if I want the heap to use between 20% and 70% of the total
+memory, I can run Java like this:
Recent Java versions have a bunch of additional command-line
+arguments for more subtle control of the heap in a container
+environment.
+
Conclusions
+
So what have we learned from all this?
+
+
In the container environment, the JVM uses cgroups to work out how
+much memory it has available and, if the installer does not specify any
+heaps sizes, it sets the maximum and initial values to 1/4 and 1/64 of
+the available memory limit
+
The container’s memory limit will nearly always be less than the
+platform’s RAM
+
The JVM only uses the container’s RAM size in its calculations, not
+swap However, both RAM and swap are a simulation – platform swap can be
+used to provide the container’s RAM if circumstances dictate
+
On OpenShift, the cgroups memory limit is derived from the resource
+limit values in the deployment, not the resource
+request value
+
Request values are used to determine which node to run the
+pod on
+
In practice, it’s rarely sensible to run a Java application on
+OpenShift without specifying at least the resource limit, if
+not the request
+
In a container environment, it often makes sense to specify heap
+sizes as a fraction of the available memory, rather than as specific
+values.
+
+
This has been a long article, to explain something which is actually
+quite simple: when deploying a Java application on OpenShift, you should
+nearly always set resource request and limit values, and you should
+consider sizing the heap using fractions of the available memory.
+
+
+
+
+
+
+
+
+
+
diff --git a/kafka_streams_hello.html b/kafka_streams_hello.html
new file mode 100644
index 0000000..286f3b9
--- /dev/null
+++ b/kafka_streams_hello.html
@@ -0,0 +1,399 @@
+
+
+
+
+ Kevin Boone: Getting started with Kafka
+Streams
+
+
+
+
+
+
+
+
+
+
+
+
+
Kafka Streams is a Java library and framework for creating
+applications that consume, process, and return Apache Kafka messages.
+It’s conceptually similar to Apache Camel, but tightly coupled to Kafka,
+and optimized for the kinds of operations that Kakfa clients typically
+do. Kafka Streams is therefore less flexible than Camel, but probably
+makes it easier to do Kafka-related things. Camel has a Kafka client of
+its own, so an alternative to Kafka Streams would be to write a Camel
+application with Kafka producers and consumers.
+
+
Note
+There’s no need to understand Camel to follow this article. However,
+because Camel and Kafka Streams have broadly similar applications, I’ll
+point out similarities and differences from time to time.
+
+
In this article I describe step-by-step how to deploy and run a
+trivial Kafka Streams application, from the ground up. I’ll use nothing
+but command-line tools and a text editor so, I hope, it’s clear what’s
+going on.
+
The article is a somewhat expanded version of the first step of the
+official Kafka Streams tutorial, and begins with the same
+Pipe.java applcation. It then expands this application to
+do some simple data processing. All the Pipe application does is copy
+from one Kafka topic to another. This is shown conceptually in the
+diagram below.
+
+
+A conceptual diagram of the ‘Hello World’
+of Kafka Streams applications
+
+
I’ll explain what a ‘topology’ is shortly.
+
Anatomy of a
+simple Kafka Streams application
+
+
Note
+With a few exceptions, which I hope are obvious, when I don’t give
+package names for Java classes, they are in the package
+org.apache.kafka.streams.
+
+
Running a basic Kafka Streams applications amounts to instantiating
+KafkaStreams object, and calling start() on it. The
+start method will not exit until something in the
+application causes it to stop.
+
The sequence of processing steps that the application performs is
+called, in Kafka Streams jargon, a ‘topology’. A Streams topology is
+broadly similar to a Camel ‘context’ with a single ‘route’.
+
So if we have defined a toplogy, coding a basic Streams application
+looks like this:
+
Topology topology = ...
+Properties props = new Properties();
+props.put(StreamsConfig.APPLICATION_ID_CONFIG, "streams-pipe");
+props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
+// Set other properties...
+
+KafkaStreams streams = new KafkaStreams (topology, props);
+streams.start();
+
There are many properties that might need to be set, but in almost
+all cases you’ll need to set the application ID and the bootstrap server
+– this, of course, identifies the Kafka broker(s) the application will
+make first contact with.
+
The interesting part of programming Kafka Streams is defining the
+toplogy. More on that later, once we’ve set everything up.
+
Prerequisites
+
In this article I assume that the reader has a working Kafka
+installation. It doesn’t matter whether it’s a simple, local
+installation with one broker, or a complex deployment with multiple
+brokers in the cloud. However, I’m assuming that you have a way to run
+the usual command-line utilities like
+kafka-console-producer.sh. If you’re running a Linux
+installation, you probably just need to
+
cd /opt/kafka
+
or wherever the installation is. In a Kubernetes environment you’ll
+probably have to log into a running Kafka broker pod to get to the
+necessary binaries, or install Kafka locally as well.
+
In any case, in the commands in this article, I assume that Kafka is
+running on my development workstation and, where I need a Kafka
+bootstrap address, it will always be localhost:9092.
+
To follow this example (and any of the official Kafka tutorials)
+you’ll need Maven, or an IDE tool that can understand Maven
+pom.xml files. Of course you’ll need a Java compiler –
+anything from Java 1.8 onwards should work fine.
+
Installing the sample
+application
+
Kafka provides a Maven archetype for constructing a trivial Kafka
+Streams application. Use the archetype like this:
I’ve used my own names here; of course you can use any names you
+like. In this example, the application will be installed into the
+directory kstreams_test.
+
The generated code will include a Maven pom.xml, some
+logging configuration, and a few Java sources. We will be working with
+Pipe.java which, if you ran the Maven archetype as I did,
+will be in
Of course, this package has many sub-dependencies, including the
+ordinary Kafka Java client.
+
+
Note
+The version of the Kafka Streams library need not be the same as that of
+the Kafka installation itself. Of course, there are advantages to
+assuming that you use the latest versions of both.
+
+
Setting up Kafka
+
The documentation
+is clear that Kafka topics used by a streams application must be created
+administratively. The explanation given is that topics might be shared
+between multiple applications, which might make different assumptions
+about their structure. This is true of non-Streams applications as well
+but, whether the explanation is convincing, a Kafka Streams application
+will not auto-create topics even if Kafka is set up to allow it.
+
So we’ll need to create the topics that the sample application
+uses:
This is an example of what has become known as the “fluent builder”
+pattern – each method call on the ‘builder’ object returns another
+builder object which can have other methods called on it. So the
+specification of the topology amounts to a chain of method calls. This
+will become clearer, if it is not already, with more complex
+examples.
+
+
Note Apache Camel also provides a ‘fluent builder’
+method for defining Camel routes. Unlike Camel, however, a Streams
+topology can’t have multiple to(...) elements. Streams can
+send messages to multiple targets, but not like that.
+
+
First run
+
We’ll need a Kafka producer and consumer to see the application doing
+anything. We’ll send messages to the topic
+streams-plaintext-input, and consume them from
+streams-pipe-output. The application should pipe messages
+from the one topic to the other.
+
A simple way to send and receive messages is to run the simple Kafka
+console tools in two different sessions:
With the application running, anything typed into the message
+producer utility should appear at the message consumer. If it does, well
+done, the Streams application is working.
+
Adding a simple
+transformation
+
Let’s extend the trivial Streams topology to performa a simple
+transformation – converting textual input to lower case.
We’ve inserted a call to mapValues() between the message
+consumptin and production.
+
mapValues is a method that is defined to take one
+parameter, which implements the ValueMapper interface. This
+interface has one method, that takes one argument and returns one return
+value. The lamda expression
+
value->((String)value).toLowerCase()
+
satisfies this interface, because it implicitly defines a method with
+one parameter and one return value.
+
+
Note
+mapValues() is broadly similar to Camel’s
+process(). However, Kafka Streams also has a
+process() method with a slightly different purpose.
+
+
In short, mapValues() provides a Kafka message body to
+an expression, and sets the new message body to the result of the
+expression.
+
+
Note
+The Kafka Streams API relies heavily on generic types and lambda
+expressions. If you’re not familiar with how these work in Java, it
+might be worth brushing up before digging into Kafka Streams beyond the
+‘Hello, World’ level.
+
+
When you run this example, and messages sent to
+streams-plaintext-input should appear on
+streams-pipe-output in only lower case.
+
+
Note
+Kafka messages form key-value pairs. This example does not transform, or
+even see, message keys. There is a map() method that
+operates on keys, as well as message bodies.
+
+
Adding some simple logging
+
Perhaps we want to trace the way that a message is transformed in the
+execution of a topology. To do that, we’ll need to ‘see’ the message
+body, but not change it. We can use the peek() method for
+that.
The peek() method takes a paramter of type
+ForeachAction, which is an interface that specifies a
+single method that takes two values, and returns nothing
+(void). To satisfy this interface, our lambda expression
+must take two arguments:
+
.peek ((key,value)->...)
+
and call a method which returns void (as
+println() does).
+
In this case, I’m only displaying the message body, so I don’t need
+the key argument; but it has to be present, even if only as
+a placeholder, or the lambda expression won’t match the interface.
+
Because the interface method is defined to return void,
+the right-hand side of the lambda’s -> must be a method
+call that is defined to return void as well. So the
+peek() method does not allow an expression that modifies
+the key or the value.
+
When you run this example, you should see some logging output from
+the application, something like this:
+
before: Hello, World!
+after: hello, world!
+
Duplicating and
+filtering the message stream
+
In this section I’ll describe some simple message processing
+operations that could be added to the basic Streams application.
+
+
Note
+I’m not giving explicit instructions on running these examples: the
+changes are in the same part of the code as all those above and, I hope,
+the results will be reasonably clear.
+
+
If we want to duplicate a message to multiple topics, we can’t do
+this:
The to() method is terminal in the fluent builder – it
+doesn’t return anything we can call further methods on. This is,
+perhaps, a little disconcerting to Camel programmers, because the Camel
+builder does allow it.
+
We can, however, do this, which has the same effect:
This topology defintion will take the input from
+streams-plaintext-input and send it to both
+streams-pipe-output and
+streams-pipe-output-long. The reason for the
+-long naming should become clear shortly.
+
+
Note
+It won’t always matter, but it’s worth bearing in mind that this method
+of splitting the stream duplicates the message keys as well as the
+values.
+
+
Rather than a straight duplication, we can filter the messages, so
+that some go to one topic, and some to others. In the example below,
+messages whose bodies are ten characters in length or shorter go to
+streams-pipe-output-long, while others go to
+streams-pipe-output.
The filter() method takes a lambda expression that
+evaluates to a boolean value – true to forward the message,
+false to drop it.
+
When it comes to dynamic routing, that is, routing messages to topics
+whose names are determined at runtime, it’s easy to make the same
+mistake that Camel developers often make.
+
Suppose I want to route messages to a topic whose name depends on
+(say) the current date. I have a method getDateString()
+that returns the date in a suitable format. It’s tempting to write
+something like this:
This fails, for reasons that are not obvious. The reason it fails is
+that the fluent builder pattern creates a topology only once. It may
+well executes for the life of the application, but all the data it needs
+to execute has to be provided at start-up time. The
+getDateString() method will be executed only once and,
+though the date changes, the argument to the to() was
+evaluated only once.
+
Kafka Streams has a way around this problem (as Camel does). Rather
+than initializing the to() method with a string, we
+initialize it with a TopicExtractor. In practice, it seems
+to be common to use a lambda expression to do this, rather than writing
+a class that implements the TopicExtractor interface
+explicitly. So we could do this:
It’s probably not a surprise that this article only scratches the
+surface of the Kafka Streams API, which is extensive. on stateful
+operations, stream merging, partition and key management, error
+handling, etc., etc. place to start. In Part 2 I’ll tackle the first of
+these – the stateful operations of counting and aggregation.
+
+
+
+
+
+
+
+
+
+
diff --git a/kafka_streams_hello_2.html b/kafka_streams_hello_2.html
new file mode 100644
index 0000000..97d6c40
--- /dev/null
+++ b/kafka_streams_hello_2.html
@@ -0,0 +1,302 @@
+
+
+
+
+ Kevin Boone: Getting started with
+Kafka Streams, part 2
+
+
+
+
+
+
+
+
+
+
+
+
+
This article follows directly from Part 1, and uses the same set-up and
+basic code. It covers the rudiments of stateful operations,
+time-windowed operations, and aggregation. That might sound like an
+awful lot of material but, in fact, all three topics can be demonstrated
+in a simple application. If you haven’t read the first article, however,
+I would recommend doing so – I’m not sure this one will make any sense,
+otherwise.
+
All the examples in Part 1 were stateless; that is, they
+worked on one record at a time, and nothing depended on anything that
+happened in the past. Aggregation and counting are, however, inherently
+stateful – they depend critically on the history of the messages that
+have been processed up to a particular point.
+
In this example, I will implement a simple count of records with the
+same key. This could be genuinely useful. Suppose, for example, that you
+have a transactional web-based system for trading certain items. Each
+time a trade happens, the system writes a record to Kafka whose key is
+the ID of the user who made it. Or perhaps (or, in addition) the system
+writes a record indicating the commodity that was traded, and how
+much.
+
We can split the message stream from the relevant topics by key, then
+use a counter to build a table of totals for each key. Then we can,
+perhaps, feed back the totals into another Kafka topic, or write it to a
+database table.
+
+
Note
+In this article, I will treat counting as a special kind of aggregation.
+In both cases, we build a two-dimensional structure from a
+one-dimensional message flow. The difference is that, with aggregation
+there will usually be some further processing; with a counting
+procedure, the counts are generally the final result. Both procedures,
+counting and aggregation, use essentially the same structure in Kafka
+Streams.
+
+
Aggregation is generally by key in Kafka Streams. That is,
+we assume that the message key means something. It might be a
+user ID, or a product ID, or something else, but it has to be something.
+If the messages have no key, then usually the first step in aggregation
+will be to derive one.
+
KStreams and KTables
+
To understand counting and aggregation, it’s crucial to be clear on
+the distinction between a KStream and a
+KTable. Frankly, though, I can do little better to
+illustrate this than to recommend the graphical demonstration of the
+duality between tables and streams in the Confluent documentation.
+
In brief, A KTable is a two-dimensional, key-value
+structure, which is generated by treating individual messages as entries
+in a change log. That is, each row in the table has a key that matches
+the message key, and the value is the latest value received from the
+stream. If you’ve used Debezium to capture database changes into Kafka,
+you’ll already be familiar with the idea that a stream of messages can
+capture the state of a database table as it changes over time. It’s less
+obvious, however, that the reverse is also true – if we have a log of
+changes to a table, we can always ‘replay’ the changes to reconstruct
+the original table.
+
So Kafka Streams assumes a kind of equivalence between message
+streams and tables – it’s always possible to construct one from the
+other. However, some Streams APIs are best expressed in terms of tables,
+rather than streams.
+
+
Note
+The equivalence between KTable and KStream is
+not sufficiently close that they can be derived from a common base
+class. Unfortunately. These classes have some methods in common, but not
+all.
+
+
The sample application
+
I’ll describe how to code and test the application first, and then
+explain how it works.
+
Go back to the original Pipe.java application, created
+from the kafka-streams-quickstart Maven archetype. Replace
+the definition of the topology with the following:
To test the application we will need to send messages to the
+streams-plaintext-input Kafka topic. There is no output
+topic – in this case output is only to the console.
+
I’m assuming that you still have Kafka set up, and the input topic
+exists. So run the application using Maven, as before:
Of course, use your own package name if you didn’t copy the steps in
+Part 1 exactly, and chose your own naming.
+
To exercise the application, you’ll need to send messages with
+specific keys. It’s a little fiddly to do this with
+kafka-console-producer.sh, but possible. You need to
+specify a key-value separator, like this (I have chosen the colon, but
+any character will be fine):
Now enter some key-value pairs at the prompt, for example:
+
dog:asdasd
+cat:xxxx
+dog:yyy
+budgie:asdasdasd
+
The actual message body is irrelevant in this example, because the
+application does not process or even display the bodies – it just counts
+messages with specific keys.
+
As the Streams application runs it will eventually produce output of
+this form:
+
dog:2
+cat:1
+budgie:1
+
That is, there have (so far) been two messages with key
+dog, and one each with cat and
+budgie. If you send more messages, you’ll see updates with
+the new totals.
+
You might notice that updates are not immediate. It might take up to
+30 seconds. This is the time interval at which Kafka Streams commits
+open streams. The other time at which you might see some output is when
+the aggregation cache is full; but, since the default size is 10Mb,
+that’s unlikely to happen here. Both the commit interval and the cache
+size can be changed when the KafkaStreams application
+instance is created.
+
Analysing the application
+
The sample application is mostly trivial, but there’s a lot going on.
+Let’s look at it step by step.
+
The method call
+
builder.stream("streams-plaintext-input")
+
creates an instance of KStream that models the raw
+message stream from the Kafka topic. Calling groupByKey()
+on this object creates an instance of KGroupedStream. This
+models a message stream that can be differentiated by its keys.
+
There are essentially three things that can be done with a
+KGroupedStream:
+
+
Create a counter on it, or
+
create an aggregator on it, or
+
create a new grouped stream with specific time-windowing
+properties.
+
+
In the simple example, I just call count() to create a
+new counter. This method returns a KTable. This is a
+two-dimensional representation of the data stream, where the first
+column is the message key, and the second (in this case) is the
+count.
+
For the purposes of this exercise, it would be nice if there were a
+method KTable.peek() that extracts the table’s contents, as
+KStream.peek() does for streams. So far as I know, however,
+there is no such method. What we can do, however, is convert the
+KTable back to a KStream, and call
+peek() on that.
+
Looking
+at the operation of the application in more detail
+
If the Kafka Streams application is running, try stopping it and
+starting it again. Then send some more messages to
+streams-plaintext-input, with the same keys as before. You
+should notice that the totals include values from the previous run. By
+default, the aggregation is not merely stateful, it is persistent as
+well.
+
It’s reasonable to wonder where this persistent state is stored. By
+default it is stored on disk in a RocksDB database. The location of the
+store is, by default,
+
/tmp/kafka-streams/streams_pipe/0_0
+
streams_pipe, you may recall, is the application name,
+set in the APPLICATION_ID_CONFIG configuration property.
+The numbers in the directory name reflect the fact that the topology
+might include multiple persistent elements, and they need their own data
+stores.
+
Storing data this way is adequate when there is only one instance of
+the client. But in a clustered set-up, where the application may run on
+multiple hosts, what happens if one of the application instances
+crashes, or is shut down, in the middle of an aggregation? The local
+file store is not available to other instances.
+
The solution is to back up the aggregation store on the Kafka broker
+itself. Look at the topics on the broker, after running the
+application:
The aggregation store is kept in a Kafka topic in the form of a
+change log. If necessary, the local aggregation stored can be
+reconstructed from this change log.
+
It is this Kafka-based back-up of the replication store that makes it
+safe to keep the local store in memory, rather than a RocksDB database.
+The WordCount example application shows a way to do
+that.
+
Despite the persistence, you might find that if you leave the
+application running long enough, the aggregation totals get reset.
+That’s because the aggregation process has a time window of one hour by
+default. In practice, applications will usually seek to total over a
+specific time window. The Streams API provides classes like
+TimeWindowedKStream to do this.
+
A technical digression
+
There’s one more thing to demonstrate in this article, and it’s a
+rather technical one, but important once the application gets beyond the
+“Hello, World” stage.
+
In the application we used toStream() to convert the
+table of counts back to a stream, and then peek() to print
+it. It should be possible to send the converted stream to another Kafka
+topic and, indeed, it is – but the results might be surprising. Try
+adding a to() call to the application:
If you send some more data to streams-plaintext-input,
+you should see some output from the consumer. But you’ll probably find
+that the totals are missing. Why?
+
I’ve said that the count() method returns a
+KTable, but KTable is a template class, as
+KStream is: both are templatized by key and value. That is,
+a KTable isn’t just a table of objects; it’s a table of
+specific key and value classes. In most of the previous examples I’ve
+fudged over this fact, and often it’s not important. In this case,
+however, it is.
+
So the count() method doesn’t really return a
+KTable – it returns (in this case) a
+KTable<?, Long>. The type of the key is, in this
+case, unspecified (because the code doesn’t say what it should be). At
+runtime the key class will be String because that’s the way
+the serialization is set up. But the ‘value’ column of the count table
+is parameterized as a Long, whatever the type of the stream
+it was derived from.
+
Then, when we call toStream() on the output of
+count() what we actually get is an instance of
+KStream<?, Long>.
+
This makes perfect sense – a count can only be a number. But when we
+write to the output stream, we’re writing messages whose payload is a
+Long, and kafka-console-consumer.sh assumes by
+default that the values are strings.
+
So to see the counts, we need to run the consumer like this:
If you want to store the counts in the Kafka topic as strings, you’ll
+need to make a conversion in code.
+
Closing remarks
+
In this article I demonstrated the rudiments of stateful operations
+(counts and aggregations) in Kafka Streams. There is, of course, a lot
+more to this topic. In particular, the API provides powerful control of
+time windowing, and a configurable notion of what a ‘timestamp’ amounts
+to in the application. The difference between, for example, the time at
+which an event was detected, and the time it was ingested into Kafka,
+might only amount to microseconds; but with the throughputs that Kafka
+is designed for, that difference might be important.
+
+
+
+
+
+
+
+
+
+
diff --git a/lineageos-degoogled.html b/lineageos-degoogled.html
new file mode 100644
index 0000000..bc7175e
--- /dev/null
+++ b/lineageos-degoogled.html
@@ -0,0 +1,281 @@
+
+
+
+
+ Kevin Boone: How de-Googled is Lineage OS?
+
+
+
+
+
+
+
+
+
+
+
+
+
In an earlier article I wrote about my
+attempts to remove all trace of Google from my life. Part of that
+process, which is still ongoing, was to install Lineage OS on all my
+Android cellphones and tablets, replacing the original, vendor firmware.
+Doing this removes the egregious Google Play Services although, of
+course, this severely limits my ability to run Android apps. That’s a
+sacrifice I’m willing to make, although not without some regrets.
+
I’ve subsequently learned that hard-core de-Googlers eschew Lineage
+OS, because it remains too close to the stock configuration of the
+Android Open-Source Project (AOSP) on which it is based. There are
+certainly smartphone ROMs, like GrapheneOS, that are even more
+Google-free.
+
But I’ve grown to like Lineage. I don’t know what kind of future it
+has, but it works well for me, and it’s easy – as easy as can be
+expected – to install on all the devices I own. Installing and setting
+up Lineage is fiddly enough; I don’t want to make my life even more
+complicated, if I don’t have to.
+
Those of us who are divorcing Google worry most, I think, about
+Google’s intrusive data collection. Of course, Google is by no means the
+only business that engages in such practices – “surveillance capitalism”
+is big business. But Google presents a unique challenge because, not
+only does it collect a lot of data, it has a lot of clever ways to
+process it, and find connections between disparate data elements. Before
+my Google separation, it always amazed me how Google seemed to know
+where I was all the time, even with location services disabled on my
+smartphone. And Google’s advertisers seem to know what I’ve been
+shopping for, even when I’ve been doing my shopping in person at retail
+outlets. How Google does this, I don’t know; but I do want to reduce
+their opportunities to do so.
+
So I need to know what information my cellphone is sending to Google,
+even having removed all proprietary Google stuff.
+
I have to point out that I’m not talking about additional, 3rd-party
+apps that I might have installed on a Lineage OS device – all
+apps have the potential to create privacy problems, but I’m free not to
+use them. Here I’m just thinking about the platform itself.
+
+
Note
+I run Lineage with no Google apps or services of any kind. If you do run
+Google services, you have to accept that absolutely everything you do
+with an Android device will be known to Google. There’s simply no point
+worrying about the trivial privacy breaches in this article – that would
+be like taking a cyanide pill and then worrying about your ingrown
+toenail.
+
+
In this article I’ll be describing various data leaks of which
+Lineage OS has frequently been accused, reporting which ones seem still
+to be present, and suggesting (well, guessing) how serious they might
+be.
+
The captive portal test
+
“Captive portals” are often found in hotels and entertainment venues.
+In a captive portal, all Internet traffic gets directed to the venue’s
+network filter, which ensures that the user has paid for a service or,
+at least, consented to some usage agreement.
+
Android performs a captive portal test every time the device enables
+a network connection. This test is a simple HTTP or HTTPS request on
+some publicly-accessible webserver. The request is expected to return a
+success (2XX) code if the server is reachable. In a captive portal, the
+service-providing organization will capture the HTTP(S) request, and
+return a redirection code to its own webserver. This server will provide
+a web page with further instructions.
+
By default Lineage OS uses Google’s webservers for the captive portal
+test. This means that Google knows every time a device raises a network
+connection.
+
Is this a problem? Google doesn’t get to find out anything except the
+IP number of the device, some limited information about the type of
+device, and the time of day. I’ve looked at the source code, and I don’t
+see any information other than this being sent – the code just uses the
+standard Java HTTP support to make the request. It’s plausible that,
+with a wide-area connection, the carrier might add additional
+information to the request, and Google might be able to infer your
+location from the IP number.
+
If you consider this to be too much of a risk, you can change the
+captive portal connectivity checker. Lineage provides no simple
+interface for this, but you can do it at the command line (e.g., by
+running a terminal app, or adb shell). You don’t need to
+root the phone to do this.
+
$ settings put global captive_portal_http_url http://my_server
+$ settings put global captive_portal_https_url https://my_server
+
Unless you want to disable the captive portal check completely,
+you’ll need to identify a public webserver that can provide the
+appropriate response. There are many such servers; some Android
+replacements that focus more on de-Googling, like GrapheneOS, default to
+using one of these rather than Google. Even then, they usually have
+Google’s servers as a fall-back, because an outage of the conectivity
+check server could otherwise cause serious disruption.
+
On the whole, I regard this (captive portal check) a relatively
+harmless breach of privacy. It isn’t telling Google anything they’re not
+going to find out about in other ways.
+
DNS
+
Every time you use a hostname to identify a remote server, there’s
+going to be a DNS lookup. This lookup translates the hostname into a
+numeric ID for use with the TCP/IP protocol.
+
Internet service providers and mobile carriers operate DNS servers,
+but so does Google. DNS is potentially a privacy problem because the DNS
+server gets to learn every site you visit. It won’t see the actual URL
+of a web request – just the hostname. Still, that’s enough information
+to be concerned about. But it’s worth thinking about who the “you” is in
+“every site you visit”. To track you, personally, as an individual, the
+DNS server needs a way to relate your IP number to something that
+identifies you. There’s no definitive way for Google (or anybody) to do
+that; but there are statistical methods that can be very
+effective. They are particularly effective if you happen to use Google’s
+other services, because these will link a small number of personal
+Google accounts to an IP number.
+
Is this a problem for Lineage OS? While it might have been in the
+past, I don’t think Lineage now uses Google’s DNS, except perhaps as a
+fallback. Both WiFi and carrier Internet connections are initiated using
+protocols that can supply a DNS server. On my Lineage devices, I’m sure
+that these are the DNS servers that are being used. Still, there are
+references to Google’s DNS server – 8.8.8.8 – in the AOSP source code.
+So I can’t prove that Google’s DNS will never be used.
+
If you want, you can supply your own DNS server in the network
+configuration in the Settings app. But, unless you run your own DNS in
+the public Internet, you’ll be putting your trust in one
+mega-corporation or another. I suspect most are less worrying than
+Google, but perhaps not by much.
+
By the way – Lineage OS supports encrypted DNS. While that will
+prevent third-parties from snooping on your DNS traffic – including your
+mobile carrier or ISP – this won’t protect you from snooping at the DNS
+server itself. So encrypted DNS is no protection against Google, if
+you’re using Google’s DNS.
+
Assisted GPS
+
It takes a long time for a mobile device to get a robust fix on GPS
+satellites – a minute in good conditions, or several minutes in a weak
+signal area. Assisted GPS (A-GPS) primes the satellite fix using
+environmental data. This data might including a coarse location from a
+cellular network. With A-GPS, a satellite fix might take only a few
+seconds.
+
A-GPS data is processed by a remote server, that has the storage
+capacity to handle the large amounts of data involved. The main operator
+of such servers is, again, Google.
+
What can Google learn about a device using Assisted GPS? As in any
+Internet operation, it will find the device’s IP number, and it might
+find the coarse location. The Internet traffic associated with A-GPS can
+be encrypted but this, again, won’t protect it from Google. To determine
+the location of a specific individual, Google has to be able to relate
+the IP number to the individual. As discussed above, that can be done
+with a reasonable degree of confidence.
+
On recent Lineage versions, A-GPS is disabled by default. If enabled,
+it uses Google’s servers – so far as I know there are no
+widely-available alternatives. I just keep it disabled, and live with
+the disadvantage of longer GPS start-up times.
+
Time synchronization, NTP
+
At one time, Lineage OS used Googles’ time servers to set the time on
+the device. So far as I know, this is no longer the case – a general
+pool of NTP servers is used. Even if that were not the case, I can’t
+worry too much about leaking time synchronizing data.
+
WebView
+
I believe that WebView is the most troubling source of privacy
+concerns for Lineage OS, and the one whose ramifications are the least
+well-understood.
+
WebView is a component of Android that renders web pages. Of course,
+a web browser will do this, but many Android apps and services have a
+need to render pages without actually being a browser. The ‘captive
+portal’ support I described above is an example: the device needs to
+render a page for user to log in or purchase Internet access, even if no
+web browser is installed.
+
Lineage OS uses the WebView implementation from the AOSP, which is
+based on Chromium. Chromium is Google Chrome without the proprietary
+Google stuff, and it’s undoubtedly less of a privacy concern than Chrome
+would be. But Chromium, even though it’s open-source, is still primarily
+a Google product.
+
There are many known instances where Chromium will provide some user
+data to Google servers. For example, we know that Chromium downloads
+lists of ‘unsafe’ websites to support its ‘safe browsing’ feature. This
+will happen however Chromium is used. When used as a regular web
+browser, Chromium might send data to Google for its ‘hot word’
+detection, for example.
+
When Chromium is only used to provide a WebView
+implementation, I’m not convinced that these minor privacy breaches are
+significant. It’s worth bearing in mind that the Jelly browser that is
+shipped with Lineage OS is just a wrapper around the Chromium WebView –
+if you use this browser, you’ll have the same privacy concerns as if you
+use Chromium itself.
+
There are a number of Google-free WebView implementations, like
+Chromite. GrapheneOS uses a WebView implementation called Vanadium,
+which is essentially a de-Googled Chromium. Installing one of these
+implementations on Lineage OS is not straightforward, or so it seems to
+me.
+
I don’t use Jelly or Chromium itself as a web browser – I install a
+browser that is not based on Google code, like Firefox. This limits my
+exposure to Chromium to occasions where WebView is used other
+than as a browser. In my normal usage, I don’t think there are many
+of those occasions, so I’m not too worried about WebView.
+
Nevertheless, it remains a slight concern and, if I could replace it
+without a lot of effort, I would.
+
Are we in tinfoil hat
+territory now?
+
I don’t like Google knowing so much about me, but I don’t believe
+Google’s data collection is directly harmful to me. My
+disapproval of Google’s activities (and I know Google is not the only
+culprit) is mainly one of principle. I don’t want to be a source of
+revenue for Google, or to legitimize their behaviour by my own inaction.
+I don’t want Google to make the Internet more of a hellscape that it
+currently is.
+
But I’m not paranoid. I don’t think Google is out to get me, or is in
+league with people who are. My rejection of Google falls short of doing
+things that will make my life hugely more difficult.
+
I am aware, all the same, that I have one foot in tinfoil hat
+country.
+
I know a few people – some in my own family – who eschew smartphones
+because they create time-wasting distractions. I certainly know people
+who don’t give smartphones to their kids, because of the well-known
+risks that social media poses to their mental health. But almost nobody
+avoids Google because they believe, as I do, that the surveillance
+economy is detrimental to society in the long term. Even those few who
+do believe this are mostly not willing to take action, because they
+believe (or convince themselves) that the benefits of a connected world
+outweigh the costs of a total lack of privacy. For me that’s like
+understanding the risks of climate change, and yet choosing to run two
+or three gas-guzzling cars because it’s a half-mile walk to the
+shops.
+
The few people who do believe as I do, and are willing to act on
+their beliefs, tend to be people who also believe that they’re being
+monitored by the CIA, or that Covid vaccines are implanting mind-control
+receivers. That’s not a gang that I want to run with.
+
On the whole, I’m satisfied that Lineage OS, as I use it, is
+preventing nearly all of Google’s data collection. I don’t install or
+use any Google services, I don’t enable A-GPS, I don’t use Chromium or
+the built-in browser. I could eliminate more arcane aspects of data
+collection – like the Internet connectivity check – if I wanted to take
+the trouble.
+
I don’t think that taking reasonable precautions to avoid becoming
+part of Google’s data collection economy makes me a tinfoil-hatter.
+Nevertheless, I would probably use GrapheneOS instead, if I had devices
+that supported it. Ironically, if I wanted to use GrapheneOS, I’d have
+to buy Google-branded mobile devices, which is an irony that really
+stings.
+
+
+
+
+
+
+
+
+
+
diff --git a/linux_offsite_storage.html b/linux_offsite_storage.html
new file mode 100644
index 0000000..10fab1c
--- /dev/null
+++ b/linux_offsite_storage.html
@@ -0,0 +1,389 @@
+
+
+
+
+ Kevin Boone: Comparing
+Dropbox, pCloud, and rsync.net for off-site storage for Linux users
+
+
+
+
+
+
+
+
+
+
+
+
+
Comparing
+Dropbox, pCloud, and rsync.net for off-site storage for Linux users
+
+
Long-term data security is the elephant in the sitting room of the IT
+industry. We can still read Akkadian stone tablets from nearly four
+thousand years ago, but I can’t read documents I stored on DAT tape only
+thirty years ago. I have 70,000+ photos and five thousand hours of
+video, and that’s just from domestic activities. Of course, I have this
+stuff backed up on my home NAS, but what if there’s a fire? Or a
+burglary?
+
I imagine that the problem is even more acute for businesses, which
+have legal obligations to store documents for a period of time, quite
+apart from a need to protect their creative output. In this article,
+though, I focus on the domestic situation, where simplicity and low cost
+are primary factors.
+
I will consider three off-site backup and synchronization services
+that have some claim to be Linux-friendly: Dropbox, pCloud, and
+rsync.net. By ‘Linux-friendly’ I mean that they either provide specific
+software for Linux (Dropbox and pCloud), or use protocols that are
+widely used in the Linux world (rsync.net). I’m specifically not
+including Google Drive, because it offers no official support for Linux
+at all. There are third-party Linux tools that provide some measure of
+integration with Google Drive, and Linux users can still use its web
+interface. Nevertheless, I’m focusing on services that specifically
+target Linux – if only in a half-hearted way – rather than those that
+actively repudiate it.
+
The three services I describe in this article are ones that I have
+used extensively, over the last ten years or so. I have most experience
+with Dropbox so, if I seem particular hard on that service, it’s
+probably because I’ve had longer to get familiar with its faults.
+
Service overviews
+
Dropbox is the grand old man of consumer file sync services. It’s
+been around since 2008 and, for Windows users at least, it’s a stable,
+well-supported platform. The service has benefited from constant
+development, and is now quite sophisticated, offering a huge range of
+features beyond basic file sharing and backup. Whether any of those
+features are useful to a Linux user is debatable. Almost no experience
+is needed, to make effective use of the Dropbox service – even on
+Linux.
+
Dropbox provides a proprietary client which attempts to keep a
+single, local directory (typically $HOME/Dropbox) in sync
+with the server, and with other computers using the same account.
+There’s also a web interface, and a well-documented API for extensions,
+which is used by a range of 3rd-party applications. There’s no support
+for traditional file transfer protocols like FTP and, if you want to
+transfer files without syncing, you’ll need additional,
+3rd-party tools (or you can write your own, as I did).
+
rsync.net is a much more focused service and, although it will work
+fine with platforms other than Linux, it’s really intended for people
+with significant Linux/Unix experience. rsync.net just provides a Unix
+filesystem, accessible over SSH. What you do with it is limited mostly
+by your ingenuity. You might choose to synchronize files with the
+server, as Dropbox does, or just upload and download using the command
+line. rsync.net also has a long history – it’s been around since 2001
+although, in my view, it’s only in the last few years that it’s become
+(barely) affordable for small-scale users.
+
pCloud is a relatively new offering, introduced in 2013. It’s a
+similar kind of service to the one Dropbox offers, in that it uses a
+proprietary client. The client provides both synchronization and
+non-synchronized file transfers. Like Dropbox, there’s no support for
+traditional protocols like FTP. And, also like Dropbox, pCloud requires
+little experience to use, even on Linux. However, pCloud offers a more
+flexible synchronization process than Dropbox: you can select abitrary
+directories on the local machine to synchronize with any directories on
+the server.
+
In fact, pCloud exposes the entire shared storage as a directory,
+typically $HOME/pCloudDrive. This is in addition to its
+capabilities for synchronization, and has the benefit that you can use
+simple command-line tools to copy files to and from the server. So you
+don’t need any additional tools to transfer files without
+synchronizing them, which is an advantage over Dropbox, at least for
+Linux users.
+
Pricing
+
It’s almost impossible to compare the costs of these competing
+services effectively, because their pricing models are so different. So,
+in an attempt to make some kind of comparison, I’m looking at the cost
+of 2Tb of storage, paid annually, with no extra features, and a small
+number of concurrent users.
+
For this scenario, Dropbox offers the best value for money, at UK£96
+per year. pCloud comes a close second, at UK£100. Trailing the pack, by
+a long distance, is rsync.net at US$288 (UK£220 at the current exchange
+rate).
+
If you need less that 2Tb of storage, however, then Dropbox is less
+appealing, because it offers nothing smaller. pCloud offers a minimum
+500Gb for UK£50, while rsync.net offers its minimum of 800Gb for $103
+(UK£78).
+
Whatever the storage size, rsync.net offers the least value for money
+for the basic service. However, there’s not a colossal difference
+between the prices of these services, and the prices change almost
+daily, according to the market and the current exchange rates. Moreover,
+all the services have useful optional features that increase the price.
+All have longer-term and lifetime pricing, but it’s hardly worth
+comparing these offerings, because their prices change so enormously, so
+frequently.
+
Both pCloud and Dropbox have a completely free tier, offering quite
+small storage (typically 5-10Gb). All the services have frequent special
+offers. For example, at one time Dropbox provided one year of 2Tb
+storage to everybody who bought a Samsung phone. At the time of writing,
+buying a one-year subscription to the Private Internet Access VPN gets
+you a free year of 500Gb pCloud. It’s worth watching out for these
+offers, because they make it easy and cheap to evaluate the services in
+realistic usage patterns. Sadly, at this time rsync.net has no free
+tier, nor even a free trial.
+
Linux compatibility
+
The most important point to note is that if you want an
+install-and-forget, background file synchronization method, then
+none of these services issatisfactory. rsync.net
+won’t be satisfactory because it simply isn’t designed to work that way.
+Dropbox and pCloud won’t be satisfactory because they just don’t work
+very well on Linux.
+
+
Note
+Please note that I’m referring here specifically to unattended,
+background synchronization – there are other ways to use all these
+services, that work perfectly well. However, both pCloud and Dropbox are
+mostly aimed at providing background synchronization services, and the
+fact that they don’t work very well on Linux is a mark against both of
+them.
+
+
I feel I should also point out that the background synchronization
+works tolerably well with Dropbox for small amounts of synchronized data
+(much less than 10Gb). pCloud doesn’t work very well even with small
+amounts of data.
+
What goes wrong? Here are some things I’ve noticed.
+
+
Synchronization just stops completely, without warning. I notice
+this particularly when resuming a computer from standby, or when
+starting and stopping a VPN service. Both Dropbox and pCloud have this
+problem, quite often.
+
Extended periods of 100% CPU usage (both).
+
Complete failure of synchronization, requiring a full
+re-synchronization (both).
+
Limited Linux filesystem support (Dropbox only supports ext4).
+
Files being renamed as conflicted (see below) for no obvious reason
+(pCloud)
+
Misleading, unhelpful error messages (pCloud)
+
+
My attempts to keep hundreds of gigabytes of data in sync using
+Dropbox were a complete failure. I found Dropbox often using 30%+ CPU
+even when it seemed to be idle, for long periods of time. pCloud
+couldn’t even manage 10Gb reliably.
+
It’s important to understand that background synchronization is a
+tricky problem, when multiple client computers are involved. One
+particularly nasty situation arises when several clients are editing the
+same file at the same time. Both Dropbox and pCloud keep copies of the
+affected file, with ‘conflicted’ in the filename. If you’re working on
+the same file from multiple computers, it’s not unusual to end up with
+many copies of the file on every computer, all with a version number and
+‘conflicted’ appended. Working out which is the authoritative copy can
+be tricky.
+
This isn’t a problem that is specific pCloud or Dropbox – it’s a
+general problem of distributed file management. However, at the time of
+writing, pCloud’s implementation of conflict management is simply buggy.
+I’ve noticed that a file I’m editing will sometimes disappear
+completely, and be replaced with a new file with a ‘conflicted’ name,
+even when no other client is editing it. This is a particular
+problem for files that I save often. The pCloud application often emits
+error messages about files that could not be synchronized even when
+they have been. That’s not a huge problem in itself, but its hard
+to trust the error reporting from the pCloud application.
+
But what about non-synchronized file transfers? By this I mean files
+that are transferred to and from the server using specific
+administrative actions. Here all the services work fine except that, for
+Dropbox, you’ll have to find 3rd-party tools that use the Dropbox API to
+do the transfer – nothing is supplied. For rsync.net you can just use
+rsync, or any conventional file transfer tool. For pCloud
+you can just copy files to and from the pCloudDrive virtual
+directory. However, a plain rsync operation is simply much
+faster than any of the proprietary transfer methods.
+
rsync.net offers the best Linux compatibility of the three services,
+because it’s so simple. There’s no proprietary client – you’ll just be
+using tools that have existed in Linux for decades. In addition, the
+rsync.net technical staff actually understand Linux, which is a huge
+advantage over the competition. For both Dropbox and pCloud it’s hard to
+talk to anybody who knows anything at all about Linux, and it’s clear
+that Linux support is not a priority. For example, the pCloud Linux
+client produces error messages that refer to web pages which describe
+only Windows.
+
Web interface
+
Both Dropbox and pCloud offer a web interface for uploading and
+downloading files, which make these services useable on computers that
+don’t have the proprietary client software. rsync.net does not offer
+similar web features but, to be fair, it doesn’t require any proprietary
+software. rsync.net works with any computer that can use standard
+Linux/Unix file transfer tools.
+
rsync.net does have a web interface, but it’s generally for
+high-level administration: billing, managing snapshots, that kind of
+thing. So far as I can see, you can’t use it for managing files.
+
Support for FTP, etc.
+
rsync.net supports SFTP, and you can also run administrative and
+transfer operations directly over SSH. You can use graphical rsync and
+SFTP clients if you wish, or just command-line tools. None of the other
+services support traditional file transfer protocols.
+
Mobile device support
+
Dropbox and pCloud both offer apps for Android and iPhone. I’ve not
+used any of these extensively, so it’s hard for me to comment. The
+Dropbox Android app only supports selected file transfers, not
+background synchronization, so it’s of little more use on an Android
+handset than the web interface. I don’t know if the iPhone app is more
+useful.
+
To use rsync.net with a mobile device, you’ll need an app that
+supports rsync or SFTP and, again, you’ll need to select specific files
+and directories to upload and download. Obtaining compatible apps is not
+difficult and, because I’m a geek, I just run rsync from a
+terminal on my Android devices.
+
Since this is an article about Linux, not Android, I don’t want to go
+into too much more detail here. In any event, I’m not really in a
+position to, since I never found the mobile apps very useful, and don’t
+have much experience with them.
+
Security and encryption
+
All the services described in this article encrypt data on the wire,
+using methods that ought to suffice for non-military scenarios. But
+that’s only part of the problem – we also have to consider encryption
+‘at rest’, that is, encryption of the data on the provider’s servers. So
+far as I know, only pCloud offers true end-to-end encryption, with the
+stored data only decryptable by the client. And this comes at
+substantial additional cost.
+
So far as I know, Dropbox offers no end-to-end encryption, so the
+confidentiality of your data turns on the integrity of the Dropbox
+staff. So far as I know, there’s nothing to suggest that they can’t be
+trusted, but this is an issue that users should be aware of.
+
As in most things, rsync.net leaves the encryption in the hands of
+the user. If you just rsync files to the server, then they
+will be stored on the server in exactly the same format as you send
+them, with or without your encryption. If, instead, you use a backup
+application like Borg, then you can apply encryption if you like.
+rsync.net offers the greatest flexibility in this regard, as in most
+others; but you need to have the skills to take advantage of it.
+
Teamwork and file sharing
+
Dropbox is the undisputed champion in this area (since I’m excluding
+Google Drive). You can share files with selected permissions with any
+Dropbox user, and these become available for background synchronization.
+You can even share with non-users, who can interact with the web
+console. In fact, Dropbox has a huge range of collaboration tools; I
+can’t really comment on how well they work for Linux users, since I’ve
+rarely used them.
+
pCloud also offers some collaboration tools, but they don’t seem to
+be as well developed as the Dropbox offering. Again, it’s not easy for
+me to comment, because I don’t often use these tools.
+
rsync.net offers nothing like this – it’s hard to see how it could,
+given its basic architecture as a remote filesystem.
+
File administration
+
Background file synchronization is intended to be
+administration-free: the client computers and the server end up storing
+the same files. Filesystem administration is therefore done on the
+clients.
+
Problems arise for both pCloud and Dropbox when using them for
+unsynchronized transfers, that is, for explicit file uploads and
+downloads. What do you do, for example, if you upload a huge bunch of
+files, accidentally including some types of file that you don’t want to
+store at all (temporary files, compiler output files…)? Fixing this
+situation is problematic on Dropbox, because you have to explore the
+files using the web interface, and select the unwanted ones for deletion
+manually.
+
It’s easier with pCloud, because the Linux client exposes the entire
+storage as a virtual directory. So you can use ordinary Linux commands
+to manage files (find, rm, grep,
+etc). But because these commands run on the client, and the files are
+stored remotely, these procedures can be painfully slow.
+
rsync.net allows you to run arbitrary shell commands
+(with some restrictions) directly on the server. So if I want
+to remove all the .o files from my source code directory, I
+can run find source -name *.o -exec rm {} \; on the server.
+I can run this command on the local virtual directory with pCloud, but
+it could take all day.
+
With rsync.net you can even mount the entire remote filesystem using
+sshfs, and then use it in the same way as pCloud’s virtual
+directory. But it’s much quicker to administer the server directly, and
+none of the other services provide a way to do this.
+
Data recovery and snapshots
+
All the services I’m considering here have tools for data recovery,
+that is, restoring accidental changes to the storage. These tools are
+presented in different ways but, essentially, they all amount to
+filesystem snapshots. All the services often some number of snapshots
+free of charge, but long-lasting snapshots are usually typical of a more
+premium service. rsync.net is a bit different in this regard – you can
+have as many snapshots as you like, and keep them as long as you like,
+but the storage counts against your overall quota. Since only
+differences are stored, this might not amount to a huge amount of
+storage.
+
With Dropbox and pCloud, changes made by data recovery tools are
+rolled out to the synchronized storage on clients. So you might find
+that files appear on the clients, or disappear. rsync.net just stores
+snapshots in a hidden directory on the main filesystem.
+
Summary
+
pCloud and Dropbox are, to some extent, directly comparable, because
+they both specialize in background, unattended file synchronization. At
+this time, I’ve found the Dropbox client to work better under Linux than
+the pCloud one. However, the pCloud client has the advantage that it
+creates a virtual directory that maps the entire server, so you can use
+the same software for both background synchronization and explicit
+upload/download operations. For me, this advantage isn’t sufficient to
+overcome the poor reliability of the pCloud client on Linux, nor is the
+slightly lower cost of pCloud. Arguably, if I needed to use end-to-end
+encryption, and didn’t mind the slight additional cost, that might make
+me favour pCloud. Still, the unreliability of the client simply rules it
+out for me, regardless of cost or features. To be fair, the pCloud Linux
+client has improved quite a lot since I first tried it about five years
+ago. It’s almost useable now and, since there are regular
+updates, I remain optimistic about it.
+
The Dropbox client is just about reliable on Linux, for small amounts
+of data (< 10Gb). For data I need to store, but not synchronize, I wrote my own command-line
+client. The Dropbox proprietary client uses extended file attributes
+as part of its operation, and only supports ext4 filesystems – that will
+be a limitation for many users.
+
In the end I found that I was mostly using Dropbox for explicit
+upload and download operations – I didn’t really use the background
+synchronization, because I just had too much data for the client to cope
+with. If I only had 10Gb of data then I imagine Dropbox would work fine
+– but then I wouldn’t want to pay for 2Tb. I will reconsider if Dropbox
+starts to offer a per-Gb pricing model, as rsync.net does.
+
It’s also worth bearing in mind that, for straightforward file
+uploads and downloads – particularly uploads – rsync.net is much, much
+faster than Dropbox. It’s at least ten times faster, but probably more.
+I guess partly this is because simple protocols like rsync
+and SFTP are just faster than the proprietary ones. But also I think
+that Dropbox deliberately throttles some kinds of transfer.
+
For a Linux user, rsync.net offers much more flexibility that any of
+the alternatives I’ve tried. If you want transparent synchronization you
+can have it – but you have to implement it yourself, using scripts or
+similar. You’ll probably need more self-discipline to use rsync.net than
+the other offerings, because it doesn’t hold your hand at all.
+
In the end, there is no clear winner for me. All these services have
+advantages and disadvantages. pCloud would be a definite
+winner, if only its Linux client were more reliable. At present, I’m
+using rsync.net, although it offers the least value for money, mostly
+because it’s so fast. Still, it’s galling that rsync.net offers the
+least storage, and fewest features, for my money. I guess that’s because
+it’s less a mass-market offering than the competition. Still, it’s nice
+to use a service for which Linux users are not second-class
+customers.
+
The other great advantage of rsync.net for me – and this probably
+won’t apply to everybody – is that I interact with it exactly as I do
+with my home NAS. I already have most of the tools I need, to use it
+effectively.
+
+
+
+
+
+
+
+
+
+
diff --git a/no_nvme.html b/no_nvme.html
new file mode 100644
index 0000000..de86dec
--- /dev/null
+++ b/no_nvme.html
@@ -0,0 +1,232 @@
+
+
+
+
+ Kevin Boone: Fitting
+an NVMe disk in a computer with no NVMe support
+
+
+
+
+
+
+
+
+
+
+
+
+
Fitting
+an NVMe disk in a computer with no NVMe support
+
+
I am the proud owner of a Lenovo ThinkStation P710. This was a
+top-of-the-market workstation back in 2015, and it still has some great
+features. Mine has two 40-core Xeon CPUs with 32Gb RAM each, and it’s…
+well, rather slow, to be frank. Things have moved on since 2015 and,
+these days, we’ve become used to the huge speed improvement of NVMe
+solid-state drives (SSDs), compared to the earlier SATA storage. It’s
+all very well having 80 CPU cores, but you’ve still got to feed them
+with work; and all that work is coming from SATA-III disks, which do not
+impress, by contemporary standards.
+
+
Note
+To be fair, the P710 is not slow at all, when it comes to processing
+tasks that can be done in parallel. Compiling many C source files
+concurrently, for example, is astonishingly fast. But for ordinary,
+desktop operations, the modest disk throughput really holds everything
+up.
+
+
So I began looking for ways to install a modern NVMe SSD. Of course,
+the P710 pre-dates NVMe, so there’s no point looking for an M.2 slot on
+the motherboard. Lenovo does (or did) market an M.2 adapter as an
+aftermarket accessory, but it seems that it only supports SATA drives in
+M.2 format, and I specifically wanted NVMe.
+
+
Note
+I’m mostly interested in running Linux from an NVMe disk. I didn’t have
+much success with Windows, but that might just be due to my lack of
+Windows knowledge and experience.
+
+
PCIe M.2 adapters
+
It’s worth bearing in mind that “NVMe” is little more than a
+connection schedule for the PCIe bus. We’re probably most familiar with
+NVMe SSDs in the form of M.2 modules, that plug into an M.2 slot. But,
+in fact, connecting an NVMe M.2 module to the PCIe bus is really only a
+matter of wiring – all the cleverness is in the storage device
+itself.
+
This means that plug-in PCIe M.2 adapters can be inexpensive – I’ve
+seen them for as little as £5. You have to be a bit careful though, and
+consider at least four factors:
+
+
The ‘key’ type – M.2 adapters are available for ‘M key’ and ‘B key’
+modules. NVMe SSDs are usually ‘M key’ devices.
+
The physical size of the PCIe slot the adapter fits. This is a
+function of the number of PCIe data lanes it supports.
+
The PCIe bus version the adapter supports. Recent devices support
+the (essentially) latest, PCIe 4.0, standard. However, my P710 only
+supports PCIe 3.0, and backward compatibility is (I think) not
+guaranteed.
+
The size of M.2 module the adapter can accommodate. In practice,
+this isn’t really a complication for adapters that take a single M.2
+module – they usually support all sizes. Multi-module adapters can be a
+bit more fussy.
+
+
So I needed an M-key M.2 adapter, that supports PCIe 3.0. I needed it
+to support the (physically) largest M.2 NVMe modules available. The PCIe
+slot size is potentially an issue, but the P710 has various slots of
+different sizes, so this wasn’t really a constraint. And, since I’m only
+fitting one NVMe module, the physical capacity of the adapter wasn’t an
+issue, either.
+
There are various adapters that meet my modest needs, with prices in
+the £10 region. They can be had with or without heat sinks; I bought one
+without a heat sink, which was a mistake – but more on that later.
+
Does the
+operating system support the adapter?
+
My P710 has no NVMe adapter support in its BIOS. So, if the adapter
+is to be supported at all, it will have to be supported by the operating
+system. The Linux kernel recognized the device immediately – drivers are
+built in. I never did get it to work with Windows 10; conceivably some
+additional drivers are needed. I don’t use Windows often enough to worry
+about it, so I never did find out. I have no idea whether Windows 11
+offers better support.
+
Once I had the adapter and NVMe module installed, I was able to run
+some storage benchmarks. I found that raw disk reads and writes were
+about four times faster with the NVMe SSD than the best SATA SSD I had
+available, so this is a useful improvement. However, raw disk
+performance is only a limiting factor in a narrow range of applications
+– in most cases, caching has some influence. Benchmarks that exercised
+the cache showed a much smaller improvement with the NVMe SSD. Still,
+even a small improvement is an improvement, and NVMe SSDs aren’t any
+more expensive than SATA SSD these days. There are circumstances in
+which NVMe might be the right choice, even if the storage was
+no faster.
+
What about booting from
+the NVMe drive?
+
This is where the problems start.
+
In order to boot from a particular kind of drive, there must be
+support in the BIOS or the bootloader. Once the operating system kernel
+has been loaded into memory, then the presence of NVMe support in the
+kernel is significant but, until that point, the fact that the Linux
+kernel supports NVMe adapters is irrelevant.
+
The Grub bootloader – the default for almost all mainstream Linux
+installations – does not support NVMe itself. So Grub will only load a
+kernel from an NVMe drive if there is support in BIOS – and, in my P710,
+there is not. I understand that the Grub maintainers have no plans to
+support NVMe, and I’m not aware of better support in any other Linux
+bootloader.
+
Sadly, there’s no complete solution to this problem. What I’ve done
+is to install the Linux kernel on a SATA drive – the BIOS understands
+SATA – and the Linux root filesystem on the NVMe drive. Grub loads the
+kernel from the SATA drive using support in BIOS, and the kernel command
+line has a root=UUID=XXXX-XXXX argument to identify the
+NVMe drive by its UUID as the root filesystem.
+
Setting up this dual-disk system is a bit of a nuisance. No regular,
+point-and-click Linux installer will help with this. What I did was to
+install a minimal Linux on the SATA drive, boot it, and then manually
+install Linux on the NVMe drive using command-line tools. Then I
+configured the Grub bootloader on the SATA drive to load the Linux
+kernel with the appropriate root= argument for the NVMe
+drive. It’s a tedious, fiddly process, and one that I won’t describe in
+detail.
+
In the end, I can almost boot from the NVMe disk – the SATA
+disk supplies only the kernel. Boot time is substantially better with
+the root filesystem on the NVMe disk, booting the same Linux; but both
+boot quickly enough compared to the P710 power-on test process, which
+takes ages.
+
Because my bootloader (Grub) configuration is so unconventional,
+automated update tools invariably break my configuration when updating
+the kernel. So ongoing maintenance is also fiddly.
+
The problem of heat
+
Installing an NVMe drive in a machine that has no support for it
+turns out to be not that difficult – with Linux, anyway. It’s also
+possible – but fiddly – to have Linux almost boot from the
+NVMe, as I described earlier.
+
After installation, the main problem lies in controlling the heat
+dissipated by the SSD. My P710 only supports PCIe 3.0, and controlling
+the heat is very difficult even there. I can only image what a nightmare
+it must be, to control the heat in a PCIe 4.0 system with twice the bus
+speed.
+
Modern laptops and workstation motherboards, which have M.2 slots
+built in, typically have some kind of cooling for the M.2 modules. Even
+if they don’t, there will be logic in the motherboard to monitor the
+temperature, and take some kind of action if it exceeds a safe value.
+That action might be throttling the CPU, or increasing fan speed, or
+something else.
+
Using an M.2 PCIe adapter provides none of this. The motherboard’s
+fan control won’t respond to an overheating NVMe module, or even be
+aware of it. If you want a heat sink – and you probably do – you’ll have
+to fit it yourself.
+
I bought my NVMe SSD from Crucial, who claims that most users don’t
+need a heat sink. Their reasoning is that the SSD has built-in logic to
+protect against overheating by throttling drive operations. However,
+I’ve found that under sustained full load – and that might be nothing
+more than copying large files from one directory to another – the
+threshold for throttling is reached in only about thirty seconds. At the
+point, some parts of the module reach (measured) temperatures in the
+region of 80°C.
+
This scorching temperature might be within the design range of the
+SSD module, but it makes me uncomfortable. In any event, the reason I’m
+adding an NVMe drive is to improve storage speed; having it throttled
+after thirty seconds is not helpful in this regard.
+
Fortunately, NVMe heat sinks are not expensive. Some M.2 adapters
+have one built in; some (at the top end of the market) even have their
+own fans. I’ve found that an inexpensive heat sink increases the time it
+takes under full load for the SSD to reach throttling temperature from
+thirty seconds to many minutes. It reduces the SSD temperature under
+what I consider normal application load from about 50°C to about
+40°C.
+
Nevertheless, high loads will still drive the SSD into throttling
+eventually, and temperatures can generally be rather high, even in
+day-to-day usage. I can improve the cooling a little by increasing the
+case fan speed, but the P710 doesn’t provide any way to do that using
+software – it’s a boot-time setting.
+
It’s worth bearing in mind that, although some very
+impressive-looking M.2 coolers are available, there might not be room to
+install one when the NVMe SSD is installed in a plug-in adapter board,
+rather than the motherboard.
+
So – is it worth it?
+
Installing an MVMe SSD in my old P710 has made it noticeably faster
+in most day-to-day operations. However, the overall performance of the
+machine hasn’t increased in line with the storage benchmarks – I guess
+many other factors are in play.
+
Moreover, controlling the heat is a problem and, if you want good
+storage performance, you’ll have to tackle it somehow. In the future, I
+will probably have to invest in a more upmarket M.2 adapter, that has a
+built-in fan.
+
There’s no way – so far as I can tell – to boot a Linux kernel from
+an NVMe drive, in a machine that has no BIOS support for this. You can
+come quite close to this goal, but the set-up is rather fiddly.
+
+
+
+
+
+
+
+
+
+
diff --git a/no_smartphone.html b/no_smartphone.html
new file mode 100644
index 0000000..77f605f
--- /dev/null
+++ b/no_smartphone.html
@@ -0,0 +1,414 @@
+
+
+
+
+ Kevin Boone: My
+stumbling journey towards a smartphone-free future
+
+
+
+
+
+
+
+
+
+
+
+
+
My
+stumbling journey towards a smartphone-free future
+
+
Last year I divorced Google from my life
+by installing LineageOS with no Google apps support on my Android phone.
+This lost me the ability to install any apps that were not open source
+or, at least, available as APK files to side-load. I take this inability
+to install apps to be a gain rather than a loss – this step alone has
+removed a lot of pointless apps from my life. Still, I continue to find
+new ways to waste time on my phone: scrolling through news feeds,
+playing stupid puzzle games, and so on.
+
I now regard this move away from Google services as the first step on
+my journey to a completely smartphone-free life. However it may appear,
+I should point out that I’m not a luddite: I work in the computing
+industry, and I have a stack of computer equipment, some of which I
+designed and built myself. I’ve developed Android apps myself. Moreover,
+I have no objection to mobile telephony in general: on the contrary, I
+think it’s one of the most significant, beneficial inventions in human
+history. What I don’t like are the time-wasting opportunities that
+smartphones make almost inevitable, and I really don’t want to be
+part of the surveillance economy.
+
Why is there a problem?
+
It should be as simple as this: if you want a smartphone, have one;
+if you don’t, don’t. Why do we not have this simple choice any more?
+
The modern smartphone is a general-purpose device: apart from
+telephony it provides Internet access, camera, media, email, navigation
+and many other things. Replacing a smartphone requires additional
+devices which, taken together, are considerably more expensive, as well
+as being less convenient. Either that, or we must learn to live without
+certain useful features that we have come to rely on.
+
Nevertheless, I spent my first forty years without a smartphone, and
+I can’t remember my life being impoverished as a result. So I have to
+question how much I need these all these technological capabilities, and
+what I might be losing by concentrating them into one device.
+
Smartphones are not yet ubiquitous: I’m told that about 20% of people
+in the UK don’t have one. Many of these people are elderly, and find
+this kind of technology unfamiliar. But I know people in their 80s who
+are welded to their handsets, and teenagers who won’t touch them with a
+stick: it’s not a simple young/old split.
+
There a problem because, as the uptake of smartphones has increased,
+the number of businesses and services that are willing to cater to the
+smartphone-deficient has decreased. It’s clearly cheaper for a business
+to provide a fully-automated smartphone interface than human contacts.
+Still, that many businesses and services now offer smartphone apps is
+fine; that so many off nothing else is not. Even people who are happy,
+in general, with using smartphones have complained to me about the sheer
+number of different apps they need, and how unreliable they often
+are.
+
+
For the purposes of full disclosure, I have to say that I really
+can’t live without a portable music player, so I have repurposed a smartphone as a media
+player. This device has no web browser, and I have removed its ability
+to install apps. So it’s really only useful as a media player. Still, I
+can’t claim not to have a smartphone at all – I’ve just chosen to
+disable all its communications features.
+
+
The Web is full of articles written by people who gave up their
+smartphones for a month, or two months, or whatever. They generally
+attest to the positivity of the experience. But even two months isn’t
+nearly long enough to discover all the things we rely on our smartphones
+for.
+
In this article, I am collecting details of things I have struggled
+with, in attempting to ditch my smartphone. Some will be obvious –
+mapping, for example. Others, perhaps, relate to smartphone features
+that I use only occasionally, and whose absense only because problematic
+after a period of months.
+
I’m writing this for other people who are considering embarking on
+the same adventure: a future without a smartphone. If you’re in that
+position, it’s as well to be aware of what you’ll be losing, so you can
+make alternative arrangements or, at least, being the grieving process.
+I have to point out from the outset that some of the features I rely
+on in my smartphone have no practicalalternative.
+
+
The issue of smartphone use, particularly social media use, among
+children is a complex one, and something I won’t address at all. There
+are many on-line resources for parents who are worried about this issue.
+See, for example, Kids for
+Now and Smartphone-Free
+Childhood. I suspect that adults who want to get rid of their
+smartphones, have different concerns from those that affect
+children.
+
+
Mapping
+
This is one application of a smartphone that I truly miss.
+
I have a satnav in my car, and it works fine. But I used to use my
+smartphone for mapping whilst hiking, especially in the hills. I used
+ViewRanger (now Outdoor Active) for planning and guiding hikes. I
+sometimes used it when I simply got lost in a city centre, and had no
+printed map.
+
I can’t do any of this without a smartphone. However, I do have a
+stand-alone GPS unit specially for hiking. Unlike a smartphone it has a
+daylight-viewable screen, 50-hour battery life (and it takes common
+batteries, which my phone does not), and it floats if dropped in a
+river. It’s altogether a better outdoor navigation companion that Google
+Maps ever could be. I also have a smart watch (which is less useful that
+it once was, without its companion app – see below) which can give me a
+grid reference.
+
I can plot a route using the ViewRanger website, and copy it to my
+GPS as a GPX file. This works fine, but it’s a few more manual steps
+than simply syncing the ViewRanger Android app wth the website.
+
In the end, though, trying to be rid of my smartphone has made me
+aware of the risks I’ve been taking, setting off into the hills with
+nothing but an electronic device for navigation. I can’t imagine this is
+something the mountain rescue folks would recommend. So, while I do have
+a GPS unit, and I will be using it, I will be taking map and compass on
+the trail in future, as we did in the pre-GPS days.
+
For urban navigation, I really have solution, except to print or draw
+a map in advance. It’s still possible to get commercial printed map
+booklets for major cities; for smaller localities you may have to rely
+on your memory or your printer.
+
Fitness tracking
+
I have a Garmin fitness watch; I use it every day. I have certain
+training targets that I try to meet, and the watch keeps track of the
+time I spend exercising in particular heart-rate zones.
+
It’s difficult to use these watches without a smartphone app. The app
+isn’t where the magic happens: it’s just a gateway to the device
+vendor’s services. But something has to feed data from the watch to the
+service, and I know of no way to do that, except using the app.
+
Without an app, my Garmin watch still records the training data, and
+I can download it to my computer, and use software to process it. And
+this is, in fact, what I now do. Having become familiar with the
+process, I prefer it, and it raises no privacy concerns.
+
However, I needed an Android app to set the watch up in the first
+place. It needs to know things like my height and weight, to estimate
+heart rate targets, etc. The device itself does not seem to have a user
+interface where I can enter these parameters. If I bought a new one now,
+I’m not sure I could set it up at all.
+
On reflection, though, I have come to understand that I mostly use
+the watch because I’m just not well-organized enough to log my exercise
+activities. I could simply keep this information in a spreadsheet, or
+even in my diary. It’s been easier just to let the watch work it out. In
+the long term, I’m not sure whether I’ll continue using the Garmin watch
+when I ditch my smartphone for good: to some extent it just provides a
+reason for me to avoid tackling my laziness and lack of
+organization.
+
Two-factor authentication
+support
+
Many businesses and institutions use a smartphone app to control
+access to their services. The app generates an authentication code that
+has to be entered into a web-based service, probably along with other
+credentials. So far, all the businesses I deal with have been able to
+work around my smartphone deficiency. They can, for example, send an
+authentication code by SMS. One bank won’t do this but, on my
+insistence, sent me a hardware authentication card to use instead of an
+app. This works fine for me. But I had to demand this, and threaten to
+take my business elsewhere.
+
So far as possible, I will not in the future do business with any
+organization that requires me to use a smartphone app. There might come
+a time when this becomes impossible. When that happens, I’m not sure
+what I’ll do.
+
Camera
+
I don’t usually take photos with my phone. If I’m taking photos, I
+care how they come out. I use a real camera with proper lenses, that
+I’ve spent decades learning how to use properly.
+
If you need to take photos at unpredictable times, and send them away
+to somebody immediately, I suspect you won’t be able to ditch your
+smartphone. But even the old flip phones have a camera of some sort –
+it’s just less convenient to copy the photos onto a computer to send
+them to somebody.
+
Such a camera will be is good enough for recording the licence-plate
+number of the car that’s just reversed into you in the supermarket car
+park. If you’re the kind of person who likes to take photos of you meals
+and share them with friends, you’re probably out of luck.
+
Weather forecasts
+
In the UK, you won’t usually go far wrong if you assume that it’s
+going to rain later; but unexpected heavy snow has caught out
+unfortunate hikers.
+
However, getting an up-to-date forecast requires Internet access,
+which I don’t usually have in the hills – by their very nature, the
+places I like to hike have no mobile coverage. So I can’t really think
+of an occasion when I will really need a weather forecast, and will
+actually be able to get one, with a smartphone or without.
+
I appreciate, of course, that this is a bigger deal for some people.
+And there’s no question that ready access to weather information is
+convenient in some situations.
+
Tide forecasts
+
Look it up before you go: tide times aren’t going to change from the
+forecast over a period of a few days.
+
Taxis, etc
+
I don’t use taxi companies that require a smartphone app. There are
+still plenty of taxi operators in my neighbourhood which I can hire with
+a phone call, or just by flagging down a cab in the street. The app-only
+firms are a bit cheaper, to be fair. If you’re taking cabs all the time,
+and spending a lot of money, lack of access to app-based services could
+be a nuisance.
+
Parking
+
This is becoming a significant problem. In the UK, many parking
+operators are moving away from cash payment, to smartphone apps. Some
+provide an alternative way to make payment, perhaps by a telephone call,
+but some do not. It’s a particular problem in the outer London
+regions.
+
Parking operators say they do this to save money, but I don’t see any
+of those alleged savings being passed on to consumers.
+
I’m not really sure how I feel about this. So far, I’ve rarely had a
+problem finding somewhere to park. There’s usually some place to park
+that is either free of charge, or takes coins, even if it’s further from
+where I want to be. It’s likely that, when this situation gets worse, I
+will use my bicycle more: I should do that, anyway.
+
In the long term, though – I just don’t know. I’m hoping that these
+changes will be rolled back when there are enough complaints, but I’m
+not all that confident. People with mobility problems are really
+starting to struggle.
+
Alarms
+
I used to rely on my smartphone for alarms. Old-fashioned phones can
+usually raise alarms, but usually not with the same flexibility. Being
+unable to set an alarm for 7.30PM every second Tuesday is a disadvantage
+but, for me, not a showstopper. And I have an alarm clock in the
+bedroom.
+
Calendar, contacts, etc.
+
I’ve gotten used to having my calendar and contact information on my
+smartphone, although I still use an on-line calendar service from my
+desktop computer (not Google’s). Without a smartphone I have to carry a
+paper notebook and a pen, and I have to transfer the data to my computer
+manually.
+
I’m investigating other ways to handle this kind of data, that don’t
+involve a smartphone. To be honest, though, I wouldn’t mind using a
+paper diary exclusively – that’s what I did for decades, before
+computers became portable.
+
What my paper diary won’t do is to alert me when I have an
+appointment coming up. I have been, again, using a smartphone as a
+substitute for proper planning. I can’t deny, however, that lack of a
+smartphone is a significant disadvantage here.
+
Entrance to venues
+
This is increasingly problem in many parts of the UK. During the
+Covid-19 pandemic, many bars, restaurants, and concert venues stopped
+taking money face-to-face, in favour of remote payment. Generally, the
+customer would get a barcode or QR code that could be displayed on a
+smartphone screen, to get access or collect an order.
+
Sometimes these smartphone codes arrive by email, and can be printed.
+This is somewhat inconvenient, particularly if you don’t have a printer,
+but fine for me. Sometimes, though, there’s only an app.
+
There aren’t many places I want to go to badly enough to break my
+resolution to avoid the smartphone. I like to hope that businesses will
+eventually realize that they are losing custom, and behave more
+inclusively.
+
Airline travel
+
I recently flew from Bergen in Norway to Amsterdam. This was the
+first time I had encountered an airport that seemed to be completely
+unstaffed. The baggage drop and check-in were completely automated, and
+driven by bar-codes. I had checked in on-line the day before, and I had
+printed my bar-code, so it was fine. But if I had not had a printer
+available, and didn’t have a smartphone to display the bar-code, I could
+have been stuck.
+
There must – surely – have been a staffed check-in desk somewhere,
+but I didn’t see one. This wasn’t a problem I had at Heathrow: although
+there’s little to be said in favour of Heathrow Airport, it was at least
+staffed.
+
I don’t fly often, and I imagine it will be even less often when
+there is no way to travel without a smartphone.
+
Social media
+
Some of the popular social media sites are accessible using a desktop
+computer rather than a smartphone; most are not. In general, I don’t use
+social media – I consider the use of computer technology to be primarily
+a business activity, not a leisure one. But a number of charitable and
+social organizations I’ve been involved with in the past now organize
+themselves entirely using WhatsApp, which excludes me.
+
Oddly, this even includes a choir with which I used to sing, that
+specializes in renaissance music. The average age of members of this
+choir is about fifty, and about half do not have smartphones. The
+advantages of using social media for organization are obvious. The
+disadvantage is that you could lose a large part of your membership,
+which is more-or-less what happened to the choir.
+
Because I’ve never been much of a social media fan, being unable to
+use it has not really changed anything. However, the move to social
+media by voluntary and charitable organizations is a lamentable one,
+because there’s never enough volunteers, and we really don’t want to
+discourage them.
+
I sometimes wonder if these kinds of organizations emphasize social
+media in a deliberate attempt to recruit ‘the youth’? If it is, it
+probably won’t work: among the people who have decided to ditch their
+smartphones, there are as many under 30s as over 50s.
+
For all that, if you’re a significant social media consumer, the lack
+of a smartphone is likely to be a crushing disadvantage.
+
News and feeds
+
One of the first things I’ve generally done each day is to look at
+news feeds on my smartphone. I can do this just as well on a desktop
+computer – but I typically don’t because, when I’m sitting at my
+computer, I’m working. In the pre-smartphone days, I would have read a
+print newspaper over breakfast, but it’s cheaper and less intrusive to
+read a screen.
+
This is one of those applications of a smartphone that seem to offer
+a modicum of convenience, with few negative connotations. But, at the
+same time, the convenience really is a modest one. I have many other
+sources of news.
+
Email and messaging
+
Email has conventionally been the most effective way to contact me,
+whatever the time of day, or wherever I was. My smartphone handles email
+pretty well, even after degoogling. I’ve gotten used to checking my
+email regularly, or whenever my smartphone nagged me, which was all the
+time.
+
Without a smartphone, the best way to contact me urgently is by SMS
+message. I still use email, of course, but not when I’m away from my
+computer. The problem with SMS is that it’s fiddly to send anything
+other than text.
+
Many people use ‘instant’ messaging services, that are often
+associated with social media sites. With outa smartphone, these services
+will not be usable. But social media messaging services are no more
+‘instant’ than email, nor any easier to use. Still, I can’t control the
+way other people choose to communicate, and my preferences definitely
+make me less contactable. I am unclear, at present, whether that will
+turn out to be a good thing or a bad one in the long term.
+
Card scanners
+
Some people I know use card-scanning apps to store business cards,
+membership cards, and similar items on their smartphones. As a person
+who tends to be a little forgetful, I can see why that would be useful.
+I don’t think I’d want to store scans of my credit cards on a
+smartphone, but it would be useful to store membership cards.
+
I can’t think of an alternative to an app for this, except to be
+better organized in general.
+
Games
+
The ability to play games on a smartphone is, to me, a wholly
+negative thing. I do not at all miss being unable to do so. But if
+you’re a gamer, you’ll need to make alternative arrangements.
+
Music
+
If there is an inexpensive, portable music player that handles both
+locally-stored files (in modern formats) and remote streaming services,
+I haven’t found one. My Astell and Kern player can do both these things
+well, but it’s hardly portable, and certainly not inexpensive.
+
This is why I have repurposed a small smartphone as a media player.
+It’s a shame nobody seems to make a decent, affordable portable media
+player these days. It’s hardly surprising, given the ubiquity of
+smartphones, but it’s still disappointing.
+
If you only want to play locally-stored audio files (rips of CDs, for
+example), you’re not completely out of luck. Old-style “MP3 players” are
+still available at reasonable prices, both new and used. Often these
+only support wired headphones; that isn’t a problem for me, although I
+accept that it will be a nuisance for many.
+
If you want to play streaming music services using Bluetooth
+headphones, I suspect it’s going to be hard to avoid a smartphone.
+
E-books
+
I have, from time to time, read novels on my smartphone. It’s not an
+ideal platform for this, but sometimes my smartphone is all I’ve had
+with me.
+
I have a Kobo e-reader, and it’s superior to a smartphone for
+reading, in just about all respects. But it’s an additional expense, and
+I have to remember to take it with me.
+
Health monitoring devices
+
In the last five years or so we’ve seen increased availability of
+home health monitoring devices: ECG recorders, oxygen saturation
+recorders, 24-hour blood pressure monitors, and so on.
+
Not all of these devices have a user interface or display, or provide
+a way to transfer data to a computer for analysis. Instead, consumer
+devices usually have to synchronize with a smartphone app – even if you
+want to use some other application to process the data. I don’t really
+see a viable alternative to a smartphone, if you want to use these
+consumer medical devices.
+
Closing remarks
+
As I’ve moved away from using a smartphone, I’ve realized that many
+of my reasons for using one are merely compensation for my laziness and
+lack of planning. There are, undoubtedly, some really useful smartphone
+applications – mapping and calendar management are particularly
+significant for me. It would be good if we could have these positive
+things without all the negative ones but, so far, this seems to be
+impossible. If you use a smartphone, you’re buying into the whole
+package – the good things and the bad.
+
Particularly worrying is the fact that essential services might one
+day be unavailable to anybody who doesn’t want to use a smartphone. I
+can only hope that, by that time, Governments will have legislated to
+restrict the use of covert surveillance by smartphone vendors and their
+partners in crime.
+
+
+
+
+
+
+
+
+
+
diff --git a/nuvi300.html b/nuvi300.html
new file mode 100644
index 0000000..739a008
--- /dev/null
+++ b/nuvi300.html
@@ -0,0 +1,170 @@
+
+
+
+
+ Kevin Boone: They
+don’t make them like that any more: Garmin Nuvi 300
+
+
+
+
+
+
+
+
+
+
+
+
+
They
+don’t make them like that any more: Garmin Nuvi 300
+
In the days
+before an automotive satnav unit was just an extension of a smartphone,
+the Garmin Nuvi 300 was the pinnacle of this technology. Garmin’s
+competitors – of which there were, and remain, few – might not have
+agreed with that assessment, but the Nuvi 300 was feature-packed, and
+undeniably practical. The automotive satnav market has been in
+decline ever since.
+
+
+The Nuvi 300, complete with adjustable,
+flip-up antenna at the rear
+
+
The Nuvi 300 was released in 2005, and cost about £79. That’s roughly
+£140 in 2024 money. It was cheaper than the competition, but had
+features that no competitor had then, and most don’t have now. Here’s
+just a few.
+
+
It could play music and audiobooks. When playing audio, it would
+pause the playback while speaking directions, so you didn’t miss
+anything. It even supported Audible’s proprietary audiobook format.
+
It had a real audio output jack (bear in mind that the Nuvi 300
+predates Bluetooth). I could plug the Nuvi’s audio output into my car
+stereo, rather than relying on the built-in speaker.
+
Apart from the usual postcode input, the Nuvi could give directions
+to a GB OS grid reference. This is a huge benefit when navigating to a
+tiny car park in the middle of a forest – no postcode.
+
The Nuvi could route around traffic obstructions whose locations
+were broadcast by local radio stations (using RDS/TMC). No smartphone
+app, no subscription. It worked, and was free to use, forever.
+Admittedly, you needed an additional antenna for this.
+
+
You could zoom the map in and out, and rotate it, using finger
+gestures. The built-in satnav in my Volkswagen can’t do that, twenty
+years later.
+
The power supply connection was on the mounting cradle, not the
+satnav itself. It’s hard to overstate how big a deal this was in 2005.
+At that time, it was common for cars to be broken into to just steal a
+satnav – some were that expensive. There was nothing to disconnect on
+the Nuvi – it just unclipped from its cradle.
+
The Nuvi had a bicycle mode, which changed the routing algorithm to
+favour minor roads, but still avoid footpaths. Weirdly, although Garmin
+advertised this as a feature, they never supplied a bicycle or
+motorcycle mount, so far as I know, and the Nuvi wasn’t waterproof.
+Although a niche feature, I did actually use my Nuvi 300 for cycling
+from time to time.
+
+
It’s interesting to compare these features with contemporary
+automotive satnav devices. In general, modern devices fall short in
+important areas, despite twenty years of technological development.
+
Modern stand-alone satnav devices generally don’t have audio outputs,
+although you might be able to pair them with a Bluetooth-equipped car
+radio.
+
That’s not as important as it used to be, though, because they
+usually don’t play music or audiobooks. Garmin removed this feature
+because they claimed it to be a safety hazard. So now I have to play
+music from my smartphone, which is a safety hazard, not to
+mention illegal, if I’m moving. So I don’t find this argument
+compelling.
+
To remove a modern device from the car, you have to fiddle about with
+a power connector on the unit. And leave a dangling cable on the
+dashboard.
+
Worst of all, I can’t find a modern satnav that supports OS grid
+references. To navigate to that tiny car park in the hills, I have to
+convert the grid reference to latitude and longitude. What a drag.
+
Many modern devices don’t support RDS/TMC routing. Some can get
+traffic information from a paired smartphone. But this requires a
+specific app, which could be doing anything it likes with your location
+data; and, of course, it requires you to be carrying a smartphone. And
+to be within signal range.
+
Of course, there’s a lot that a modern device can do, that would have
+been impractical twenty years ago. To update the maps on my Nuvi 300, I
+had to copy them to an SD card, using proprietary Garmin software.
+Modern devices can download maps using Wifi or 4G. Having said that, I
+have to admin that to update the maps on my car’s built-in satnav, I
+have to take it to a Volkswagen dealer. Frankly, I don’t find the modern
+developments in this area to be particularly useful, compared to what
+has been lost.
+
I wonder sometimes: why is it that so many consumer electronics
+manufacturers discard useful features as technology develops? Why don’t
+mobile phones have a headphone jack any more? Why do we need a stack of
+obscure tools to replace the hard disk in a modern laptop, when my W520
+just has a sliding lever that pops the disk out? Why does my
+stupidly-expensive Astell and Kern media player not have FM radio
+support, when my 2005 Cowon does?
+
Why, in short, does technological advance not make consumer devices
+better?
+
Part of the problem, I guess, is that pricing is so competitive, that
+every feature, however cheap, has to justify itself on cost grounds.
+Nobody wants to pay much for anything. In fact, nobody wants to pay
+anything, which is the modern world is so choked with intrusive
+advertising.
+
Only a fraction of satnav owners want to navigate to a grid
+reference, or play audiobooks, and it costs money to maintain these
+features. Similarly, supplying a powered mounting cradle requires
+specific design and tooling, compared to a passive cradle and a dangling
+cable. Maybe this feature would only add fifty pence to the sale price
+of the unit but, in a market where price is everything, that’s still
+fifty pence too much for many people.
+
The problem with this argument was that, back in 2005, the Nuvi 300
+was cheaper than anything else on the market, as well as being
+superior. Conceivably Garmin deliberately under-priced the unit, to
+encourage people away from its main competitor (Tomtom, in the UK).
+
Whatever the explanation, the Garmin Nuvi 300 was the first
+automotive satnav I owned and, at one time, I owned three of them. I
+still have one on my quad-bike. They all still work, although it’s hard
+– but not impossible – to update the maps. The only reason I’m not still
+using them in my car is that cars have satnav built in these days. The
+built-in satnav is rubbish, but not so bad that I feel the need to
+install another one.
+
And, of course, many people – perhaps most people – use smartphone
+navigation apps these days. Smartphones are completely inappropriate for
+vehicular navigation, not to mention catastrophic for personal privacy,
+but they come with the advantage that you probably already have one. So,
+in a choice between using something inappropriate, and paying for
+something better, many people will opt for the inappropriate.
+
+
+
+
+
+
+
+
+
+
diff --git a/phone_media_player.html b/phone_media_player.html
new file mode 100644
index 0000000..bab0ea3
--- /dev/null
+++ b/phone_media_player.html
@@ -0,0 +1,209 @@
+
+
+
+
+ Kevin Boone: Some
+thoughts on using a small cellphone as a media player
+
+
+
+
+
+
+
+
+
+
+
+
+
Some
+thoughts on using a small cellphone as a media player
+
+
Of course, it’s easy to use any modern cellphone as a media player –
+just use any media player app. Job done.
+
This post, however, is not about using your day-to-day phone as a
+media player – it’s about customizing a cellphone so that its primary
+use is as a media player, and not using it for telephony or
+communications at all.
+
For most of us, there’s really no good reason to do that. If you’re
+already carrying a cellphone around, why would you want a second one as
+a media player? The only answer I can think of is that, like me, you
+don’t want to use a smartphone at all – but still want a
+high-tech media player.
+
Last year I divorced Google from my life.
+I’m wondering now – do I really need a smartphone at all? Or
+could I manage with a Nokia brick, or its modern equivalent? There are
+certain advantages to this:
+
+
No creepy tracking by apps
+
Respectable battery life – weeks rather than hours
+
Fewer distractions
+
+
I would really only miss two things if I dumped my smartphone: Google
+Maps (and I’m already at stage six of the seven stages of grief over
+that), and the media player. I use my phone for music and audiobooks all
+the time, and occasionally for video.
+
At one time there was a huge range of portable media players to
+choose from – I should know: I’ve bought most of them. These days,
+portable media players are a speciality item – they’re expensive, and
+not very portable. And most run Android anyway, which makes them
+essentially a smartphone, but without the antenna.
+
So why not repurpose a small cellphone as a media player?
+
If I dedicate a phone to be only a media player, I get certain
+advantages over playing media on my main phone:
+
+
With a bit of effort, I can remove unnecessary bloat and increase
+battery life. I can even remove distractions like the web browser, if I
+really want to.
+
I can use a really small phone: my choice is the Samsung S10e which,
+to be frank, is too small to use as a regular phone. But it’s a great
+size for a pocket media player.
+
I don’t always have to carry it with me. In practice, I
+usually do; but I don’t have to.
+
I don’t need to leave it switched on when I’m not using it.
+
I can disable the cellular antenna by leaving it in ‘airplane mode’,
+further increasing the battery life.
+
+
If you remove the SIM card from a cellphone, doesn’t that by itself
+essentially convert it to a WiFi-only device, suitable for little more
+than media? Well, yes and no. ‘Yes’, because that prevents its being
+used for telephony or roaming Internet access. This in turn removes much
+of the motivation for installing distracting apps on it. But also ‘no’,
+because it’s still full of vendor bloatware, creepy spyware, and other
+power-draining rubbish. Moreover, so long as I’ve still got access to
+the Google Play store, I’ve got a way to reinstall any of the
+time-wasting apps that I want to remove from my life.
+
So my first step in dedicating a phone to be a media player is to
+install a non-commercial operating system on it. Currently I favour
+LineageOS, but there are others. All these platforms are more-or-less
+free of bloat, and this alone radically increases battery life.
+
Even better, LineageOS starts from a shut-down in about 25 seconds on
+my S10e, compared with a minute with the stock Samsung firmware. So it’s
+practicable to turn it off when I’m not using it for media. Even if I
+don’t turn it off when I’m not using it, I still get six days of light
+use – the stock firmware requires me to charge every day, whether I’m
+using the phone or not.
+
Because there’s no SIM card, and no telephony, I can leave the device
+in ‘airplane mode’ all the time. Although this disables the cellular
+antenna, and the power drain that accompanies it, it doesn’t
+prevent me using Bluetooth and WiFi if I want. Moreover, the ‘airplane
+mode’ setting is persistent across restarts, so I don’t have to fiddle
+with it.
+
For a really distraction-free experience, I needed to remove the web
+browser. That’s not easy – I can disable it in various ways, but
+reversing the disablement is too easy. To remove the browser completely,
+I need to root the phone, and use a root-only package remover. Rooting
+is easy enough using a tool like Magisk, but it’s very easy to break the
+device completely, messing about with system packages.
+
But having made the decision to root the phone and do
+potentially-destructive things to it, I can remove other unnecessary
+time-wasters while I’m about it (although there are few of these in
+LineageOS). If I want, I can remove the telephony subsystem completely.
+This saves a bit of memory, but it takes away the ability even to make
+emergency calls. I’m not entirely convinced that I want to go that far,
+although I’ve removed all the SMS messaging stuff.
+
There are a few other things that can be done, to improve battery
+life. The S10e has an OLED screen, so the screen’s power usage is
+directly proportional to the amount of colour displayed. I’ve enabled
+the ‘dark’ theme generally, and I’m using a plain black image as the
+wallpaper. I’ve set the screen timeout to five seconds. I’ve turned off
+all the things that vibrate. I usually leave WiFi and Bluetooth off,
+unless I’m actually using them. Same for location services – I rarely
+need GPS on a media player (but it’s occasionally useful; I wouldn’t
+remove it completely).
+
+
+There’s a thing you don’t often see: two
+weeks’ battery remaining
+
+
With these changes I get about six days of light usage between
+charges. The standby time can be measured in weeks. But I find that I
+don’t need even that much battery endurance, because I only switch the
+device on when I’m using it (which, to be fair, is quite a lot).
+
I have thousands of music albums and audiobooks in FLAC format, and
+playing them is no problem. I can use VLC (which is open source, and
+available from F-Droid), or the built-in media player. But what about
+streaming services?
+
This is where things get a bit sketchy. Companies like Spotify do not
+distribute their apps in APK format, suitable for side-loading onto a
+device. They aren’t on F-Droid, either – they wouldn’t be accepted.
+There’s no reason why the vendors
+shouldn’t supply APK files – their revenue comes from subscriptions, not
+app sales. Nevertheless, if you want to use Spotify, Qobuz, et al.,
+you’re going to be fishing for an APK in murky waters. I wouldn’t do
+that on a phone that I relied on, or that contained any personal data.
+However, I’m not storing anything personal on my media player. I haven’t
+even enabled a lock screen.
+
Still, it irks me that I can’t use USB Audio Player Pro – my
+favourite music player – even though I’ve paid for it. Without Google
+Play services, there’s no way to use this app. And I’m sure there are
+many other apps and services that I wouldn’t be able to use – but I
+don’t know what they are, because I don’t use them.
+
So is it worth it?
+
I’ve already de-googled my main phone, which means that I was already
+getting many of the advantages I’ve suggested in this article. But I
+still got the constant distractions and time-wasting that the smartphone
+provided, with or without Google and its ready access to apps. I found I
+was constantly checking email, looking a websites, and all the usual
+stuff. And I still had something that had to be charged every day or two
+at most – and it had to be kept charged, because people called me and
+messaged me. And it was comparatively heavy, and would have been
+expensive to lose.
+
The de-googling process is complicated enough but, if you want to go
+further by removing the web browser, that’s an additional complexity –
+and a risky one. But an S10e can be had second-hand for about fifty
+quid, so the risk is a comparatively small one, even if you break
+it.
+
The trend with smartphones is to make them larger and larger. I’m not
+sure there is much, if anything, currently on the market as small as
+Samsung’s S10e. Of course you can dedicate a large phone as a media
+player, but the modest size of the S10e is part of what makes it
+appealing in such a role.
+
I find that the de-googled, de-bloated, SIM-less S10e makes a pretty
+good media player. It’s not much bigger than, say, the 7th-generation
+iPod Touch, which was very popular in its day. Start-up time does not
+compare well with portable media players of ten years ago, but it’s
+actually faster to start than my Astell & Kern player – and a lot
+more portable. Sound quality is… OK. It’s better with an external DAC,
+but then it’s no longer a pocket device.
+
In the end the question isn’t really whether a small smartphone makes
+a decent media player – clearly it does. The important question is
+whether you actually need one, and the answer will only be ‘yes’ if you
+don’t intend to use a smartphone as a regular phone. I am so far only
+experimenting with this self-deprivation, and I’m not sure how it will
+work out in the long term.
+
+
+
+
+
+
+
+
+
+
diff --git a/quad-306.html b/quad-306.html
new file mode 100644
index 0000000..cbcaddb
--- /dev/null
+++ b/quad-306.html
@@ -0,0 +1,185 @@
+
+
+
+
+ Kevin Boone: They
+don’t make them like that any more: the Quad 306 amplifier
+
+
+
+
+
+
+
+
+
+
+
+
+
They
+don’t make them like that any more: the Quad 306 amplifier
+
+
British electronics company Quad was known mostly for its signature
+electrostatic loudspeakers – monstrous, window-sized slabs with brass
+mesh grills – but it also did a good business in amplifiers and other
+hifi equipment. The 306 was more-or-less the last product released by
+the Company under the auspices of its founder, Peter Walker, and
+probably the best.
+
The 306 is a power amplifier. It has no controls except an on/off
+switch, and no connections except input, speakers, and mains. When it
+was released in 1987 you’d use it with a preamplifier, which had the
+actual controls, and a bunch of separate audio sources. These days you’d
+probably do the same, but it’s perfectly possible to connect the Quad
+306 directly to the output of a computer’s soundcard, and it will work
+fine.
+
Walker notoriously said that the perfect power amplifier is “a wire
+with gain”. That is, the amplifier should make the electrical signal
+larger, but not modify it in any other way. Such a goal is difficult to
+achieve, and there’s been much discussion about whether it’s even worth
+attempting. Still, the design goal of the 306, and it’s predecessor the
+405 range (yes, Quad product naming was bizarre) was for the amplifier
+to be as transparent as possible. Any modification to the audio
+properties would be done elsewhere.
+
Whether that was a laudable goal in 1987 is debatable, but it’s
+certainly a laudable goal now: with digital signal processing
+technology we can now make pretty-much anything sound like anything
+else. There’s little need for an amplifier to do more than enlarge.
+
+
+Industrial-grade simplicity, and no
+attempt to disguise the heat sink
+
+
The 306 could, in principle, produce a constant 50 watts into two
+8-ohm speakers. Music being what it is, that meant in practice about ten
+watts, with the extra power reserved for peaks. It was, and remains,
+plenty loud in a domestic setting. The case was comparatively small for
+the time, and there were no cooling fans. So heat management was by a
+substantial heat sink, mounted unconventionally on the front. The whole
+unit was a dull, industrial grey, at a time when brushed aluminium was
+all the rage. Certainly it looked businesslike.
+
Mounting the heat sink on the front makes sense, from a technical
+perspective, because there was little else on the front panel. The rear
+had all the connections, so a rear heat sink could not have been as
+large, and the wiring would have been complicated. The internal layout
+of the 306 is, in fact, very simple.
+
+
+The interior of the 306 shows its simple
+design
+
+
All the electronics are mounted on a single, single-sided printed
+circuit board. It’s relatively easy to dismantle the whole thing for
+servicing and modifications. And there was plenty of modification, from
+simple upgrades to the speaker sockets, to radical changes to the
+electronics.
+
A notable factor about the design is the small number of electrolytic
+capacitors. There are only four – the massive power smoothing
+capacitors. They have to be this large to store the energy needed to
+deliver powerful transients – the thump of a bass drum, for example.
+Minimizing the number of electrolytic capacitors matters because these
+devices change their electrical properties as they age.
+
Quad made much of the fact that the electronic design was
+non-critical in its component choices. If you replaced, for example, a
+95-ohm resistor with a 110-ohm resistor it wouldn’t do any harm. Because
+resistors were not made to tight tolerances in the 1980s – cheap ones
+still are not – this lack of fussiness was important for
+maintenance.
+
You might also notice that there are no internal adjustments
+of any kind. Either the 306 works perfectly straight from the assembly
+line, or it doesn’t work at all. The basic electronic design principle
+was called by Quad ‘current dumping’. It’s a principle that lends itself
+to unfussy component selection and layout.
+
A hifi power amplifier was a relatively specialist piece of equipment
+in the 80s, as it is now. Then, as now, only enthusiasts with plenty of
+funds would assemble a hifi system from separate components; everybody
+else had some sort of ‘music centre’ that integrated everything
+together. ‘Everything’ in 1987 might have included a CD player but, for
+most people, good-quality music sources were FM radio and vinyl records.
+Plebs like me played cassette tapes, but real enthusiasts scorned such
+things, just as they scorned the ‘music centres’, which generally were
+cheap and nasty, and sounded awful.
+
So anybody who wanted a separate power amplifier in the 80s would
+expect to have to pay in hard coin: the 306 cost about £1500 in 2024
+money. Looking inside one, it’s pretty clear that it doesn’t contain
+£1500’s worth of components. Presumably some of that money went into the
+design process, but it’s hard to avoid the conclusion that some of it
+went into the name: Quad was a prestige brand in the 70s and 80s.
+Everybody with an interest in hifi knew about Quad, just as everybody
+with an interest in cars knew about Jaguar.
+
If you spend such a lot of money on an amplifier, you’re probably
+going to find that it sounds good. But the one I bought for £40 ten
+years ago sounded pretty good as well. I sold it because it, and the
+additional equipment that went with it, was too large for the shelf in
+my office. The Denon compact system I replaced it with was a lot more
+convenient, and a space-saver, but it didn’t sound so good.
+
Of course, the 306 wasn’t perfect. An obvious, irritating problem was
+that it was all too easy to trip the over-current protection by playing
+it too loud. The protective device did not auto-reset – you had to grope
+around behind the stack of hifi equipment to find the tiny reset button
+on the back of the 306. But probably the worst fault was that the
+enormous mains transformer wasn’t mounted on the chassis, but on the
+printed circuit board. If you dropped the amplifier, there was a good
+chance that you’d crack the circuit board. This was repairable, but not
+easily.
+
Given how good the 306 was, it’s not obvious why it wasn’t more of a
+success. In fact, by 1987, Quad’s amplifier business was already in
+decline. According to Ken Kessler’s book Quad: The Closest
+Approach, the 405-2 – the predecessor of the 306 – sold about
+100,000 units. The 306 and all the amplifiers that came after it did not
+sell that many together.
+
I suspect that part of the reason for the decline was the increased
+popularity of CD players. One of the things that CDs taught us was that
+an amplifier probably wasn’t the limiting factor in sound quality. If
+you had half-decent speakers and a CD player, even an inexpensive
+amplifier could sound pretty good. Not as good, perhaps, as a top-flight
+unit, but good enough for most people.
+
At the same time, the quality of integrated hifi systems started to
+improve. It was, and remains, possible to get bettter sound from
+carefully chosen, separate components; but the difference gets smaller
+every year. My Denon compact system included a CD player and DAB/FM/AM
+radio along with the amplifier and speakers, and it still cost only a
+quarter as much as a Quad 306.
+
If you want a Quad 306 today – and I strongly recommend it – you can
+pick up a reasonable one from an on-line auction sites for about £200.
+You should probably replace the huge capacitors if the previous owner
+did not, because capacitors of that size don’t age well. You should also
+check the PCB for cracks at the same time. With these simple things
+taken care of, the 306 will probably outlive you, and still be sounding
+great when it does.
+
+
+
+
+
+
+
+
+
+
diff --git a/rex.html b/rex.html
new file mode 100644
index 0000000..d4011a9
--- /dev/null
+++ b/rex.html
@@ -0,0 +1,152 @@
+
+
+
+
+ Kevin Boone: They
+don’t make them like that any more: the Xircom REX 6000 PDA
+
+
+
+
+
+
+
+
+
+
+
+
+
They
+don’t make them like that any more: the Xircom REX 6000 PDA
+
+
I imagine that anybody interested in the history of technology has
+heard of the Palm Pilot PDAs, that were wildly popular at the
+beginning of the millennium. These iconic devices were the first, truly
+successful pocket computers, although they were not the first consumer
+PDAs – the Psion Organizer range was already well-established
+by that time.
+
The Palms could manage calendar and contact information, and store
+short documents. They had a graphical display, unlike the early Psion
+devices, with a handwriting recognition pad. They were a trendy and
+productive alternative to a paper pocket diary, which was the only
+realistic alternative at the time, if you didn’t want the bulk of a
+Psion. The Palm III could synchronize to a PC, using a desktop docking
+station and a serial cable – a real, RS232 serial cable in those
+days.
+
The problem with the Palm and Psion devices was their size – or so it
+seemed at the time. In a weird reversal, of the kind that often happens
+in consumer electronics, many people are now carrying around smartphones
+with huge screens, that are bigger and heavier than a Palm III. Still,
+we tended to consider the size as a disadvantage twenty-or-so years
+ago.
+
And so the REX. Even if you’re interested in the history of
+technology, you may well not have heard of this device. It actually
+appeared on the market around 1997, produced by the Franklin company,
+which was better-known for its pocket dictionaries and translators. But
+it didn’t enjoy its brief flare of popularity until Franklin sold the
+rights to REX to Xircom, which was itself purchased by Intel in
+2001.
+
What made the REX 6000 special was its miniscule size. It did the
+same kinds of things that the Palm and Psion products did, but it did
+them in a package little bigger than a credit card. When I first saw one
+– in one of the many electronics shops that used to populate London’s
+Tottenham Court Road – my eyes almost popped: I’d never seen anything
+like it, and I knew immediately that I had to have one.
+
Almost the whole surface of the REX device was the LCD screen, as you
+can see in the photo:
+
+
What’s less clear from the photo is that this device is, in fact, a
+PCMCIA card. At the time of the REX, and the Palm III, almost every
+laptop computer had a PCMCIA slot. Typically these slots were used for
+connecting external disks and network cards. REX’s innovation was that
+the entire device was a self-contained PCMCIA card.
+
So, in principle, keeping your REX synchronized just meant plugging
+it from time to time into the PCMCIA slot on your laptop. If you didn’t
+have a laptop, there was a PCMCIA-to-serial adapter for a desktop
+computer.
+
Unlike the Palm devices, REX didn’t have handwriting recognition. You
+had to enter text using a tiny virtual keyboard and a stylus (which was
+included). This was less a problem than it might seem because, in
+practice, users did most of the actual data entry on a PC. The REX was
+supposed to synchronize to Microsoft Outlook, although it came with its
+own, PC-based PDA application as well. There was also an online service,
+rex.net, that provided news articles and weather forecasts that you
+could transfer to the REX.
+
The REX was a technological marvel, and ought to have been a success.
+With the REX I could keep all my contact details, appointments, and
+to-do lists in my wallet, along with a couple of ebooks (I had better
+eyesight in those days). It wasn’t all that expensive – about £100 in
+2001, which is about £200 in today’s money.
+
The REX did not fail because it was too expensive, or because some
+better product displaced it. It failed because Intel, its new owner, was
+not interested in it. Intel most likely did not buy Xircom to get its
+hands on the REX: probably Intel was interested in Xircom’s PC network
+adapters, which were very popular. The REX would have seemed like an
+out-of-place oddity to Intel. So, within weeks of Intel’s taking control
+of Xircom, they closed down the rex.net service. The bugs in the REX
+Windows software – and there were many – were never fixed. And so users
+drifted away, and potential customers continued to use their Palms or
+Psions until, presumably, the rise of the smartphone.
+
During its short life, the REX had an enthusiastic user and developer
+community, although probably not as vigorous as the Palm community.
+Still, enthusiasts managed to produce a number of useful add-on
+applications, despite the total lack of support from Intel. The REX used
+a Z80-compatible CPU, so there was no shortage of tools that could
+produce code for it. The real problem was the lack of documentation on
+the device’s internals. It was also difficult to use it in any
+productive way under Linux. To be fair, the Windows software was so
+buggy that there wasn’t really a productive way to use it under Windows,
+either – REX never really got past the proof-of-concept stage under
+Xircom, and Intel’s lack of interest prevented it going further. It was
+the lack of Linux support that led to me selling my REX – only a few
+months after buying it so rapturously – something I now regret.
+
No other portable computing device had the form factor of the REX in
+2001, and none has now. To be fair, there’s no demand for a
+PCMCIA-shaped pocket computing device, because nobody uses PCMCIA any
+more. But there’s potentially a demand for a credit-card computer, when
+we consider what could be packed into a device this size with modern
+technology. But these days we’re so welded to our smartphones that any
+REX-like device faces insurmountable competition.
+
If you want a REX 6000 in your collection, you can still find them on
+on-line auction sites. But you’ll have to be patient, and be willing to
+pay more than you might expect. You probably won’t able to do anything
+useful with it – unless you’re willing to run Windows 98 and put up with
+the terrible, buggy software. But the hardware will probably work fine,
+because these devices were indestructible. And they used
+widely-available, user-replaceable batteries: something that modern
+manufacturers would do well to emulate.
The Internet is full of rants about systemd, and I don’t want this
+post to be another one. Many of the complaints people make about it
+don’t stand up to much scrutiny, even the technical ones; and many
+complaints are not even technical. My particular interest in Linux is
+primarily for embedded applications; and there, I suggest, systemd is
+creating a potential (technical) problem. In this article I will try to
+articulate what the problem is; but I have no solution to offer.
+
Recapping the last ten years
+
systemd is a set of integrated applications concerned with system
+management. It replaces not only the traditional init
+process that brings up long-lived processes, but also much of the other
+system infrastructure: user session management, device management,
+logging, timing, and an increasing number of other functions.
+
The majority of Linux users are uninterested in the pros and cons of
+systemd. A small number are violently opposed to it, and a small number
+are violently opposed to those who are opposed to it. Nevertheless, most
+mainstream Linux distributions have adopted it after a shorter (Fedora)
+or longer (Debian) period of argument.
+
I think there’s little argument that the main target for systemd is a
+general-purpose computer, with a modern, integrated graphical desktop
+(Gnome, KDE). systemd does well in systems like this because it can
+handle process initialization on demand and in parallel. This
+potentially makes boot times faster, and keeps resource usage down,
+because it isn’t necessary to start a lot of services that are used only
+occasionally. I don’t think these advantages are the main reasons for
+systemd’s widespread adoption (see below), but they’re certainly
+important.
+
Embedded Linux: the problem
+
Unfortunately, what makes systemd good for general-purpose desktop
+applications potentially makes it unsatisfactory for embedded Linux
+systems. As an illustration, I’ll show some memory figures from the
+Raspberry Pi 3B that’s currently on my workbench. The board is running
+the DietPi Linux distribution – probably the best fully-maintained Pi
+distribution, if you want a minimal system. Although DietPi uses systemd
+(it has little alternative, as I’ll explain later) it doesn’t
+necessarily use the full set of components. In fact, a minimal
+installation of DietPi, for console operation, installs only the systemd
+service manager (the ‘init’ process), the log daemon, and the
+udev daemon.
+
This is the resource usage, as reported by top, for the
+systemd init process (alone) on the Raspberry Pi 3B.
+
PID USER PR NI VIRT RES SHR S %CPU %MEM
+ 1 root 20 0 168144 11404 8592 S 0.0 0.3
+
systemd init is not a particularly greedy user of memory by
+contemporary standards – its virtual address space is 168Mb, but only
+~8Mb is currently mapped to RAM. That’s about 0.3% of the Pi 3’s 4Gb of
+RAM. But here’s the same figures for SystemV init, on
+exactly the same hardware:
+
PID USER PR NI VIRT RES SHR S %CPU %MEM
+ 1 root 20 0 2744 1532 1436 S 0.0 0.2
+
It’s much smaller. Just to be clear – I got the systemd
+implementation and the SystemV init implementation from the same Debian
+ARM binary repository. I haven’t used fiddly compiler optimizations or
+anything like that, to bias the resource usage figures.
+
Now let’s look at the systemd logging daemon,
+systemd-journald.
+
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
+ 147 root 20 0 41508 7744 6652 S 0.0 0.2 0:12.05 systemd-jour+
+
Again, it’s a small, but noticeable, user of the 4Gb RAM. And, for
+comparison, these are the figures from my own syslogd-lite,
+which I wrote specifically for embedded applications.
+
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
+ 117 root 20 0 1976 72 0 S 0.0 0.0 0:00.00 syslogd
+
Note that the memory and CPU usage of this logging daemon are both
+essentially zero. This is an unfair comparison, in a way, because I
+wrote syslogd-lite specifically to minimize resource usage, and it has
+few features. But it shows that it’s plausible to write utilities that
+target embedded systems specifically, and sometimes it’s necessary.
+systemd does not do this, and was never intended to. Running a
+general-purpose software set like systemd on minimal, embedded hardware
+can’t be expected to be effective.
+
With care, though, a minimal installation of systemd does
+run on a low-resource ARM board like the Pi 3. In fact, it will run on a
+board with 1Gb RAM, perhaps even lower. But, as the RAM decreases, the
+proportion of it occupied by systemd increases.
+
What’s less obvious is the effect on boot time of the use of systemd
+init. Proponents of systemd argue – correctly, I think – that it
+decreases boot time in the kinds of system for which it was designed.
+But on my Pi 3 it increases boot time quite noticeably. That is, the
+systemd ‘init’ process takes an extra half-second or so to start,
+compared to SystemV init. Half a second isn’t much, but in an embedded
+application I actually care about boot time. I couldn’t care less on my
+desktop computers, because they spend most of their lives suspended. I
+rarely actually reboot them.
+
The extra start-up time of the init process is the result, I guess,
+of the additional size and complexity of the systemd executable. It’s
+about 200kB in size itself, and dynamically links 26 other libraries.
+SystemV init, on the same hardware, is 40kB, and links only the standard
+C library. The additional complexity of systemd is not wasted: it’s
+needed for the additional functionality that systemd offers, in its
+intended environment. But this functionality is mostly not needed in an
+embedded application, so the additional complexity of systemd is a cost
+without a benefit.
+
I’ve found that most of the services that systemd replaces have an
+alternative that is smaller, and faster to start, in an embedded
+environment. Unfortunately, some of these services don’t have
+an alternative any more.
+
So what?
+
I’m not obliged to run systemd on my Raspberry Pi systems and, in
+fact, usually I do not. I build my own Linux installation, using
+binaries that I cherry-pick from the Debian repositories, and code I
+write myself. Most of the binaries work without systemd. Some complain
+about not finding systemd, but work anyway. Some things don’t work
+without systemd: the Gnome display manager, for example, as it is built
+for the standard Raspberry Pi, won’t work. It can be made to work, but
+you have to build it from source. How long it will continue to work,
+even if built from source, is open to question. But I’m not going to be
+running Gnome on an embedded Linux board, so I don’t see this as a
+problem for my applications.
+
The more fundamental problem is that the people who most like
+systemd are distribution managers. Sure, there are administrators
+who like it, and defend it vigorously; but most end users and
+administrators don’t really care. But for the maintainers of mainstream
+Linux distributions, systemd is like Christmas. systemd works reasonably
+well for a whole range of usage scenarios and, best of all, it all comes
+in one bundle. So a distribution manager doesn’t have to maintain,
+integrate, and support a whole bunch of system utilities from different
+sources – systemd provides everything in one huge build.
+
There are just a few Linux distributions that don’t use systemd, and
+they are not widely used. Maintaining them is difficult, with just a
+handful of volunteers. All the large, commercial Linux providers have
+moved to systemd.
+
Only Gentoo and its derivatives (so far as I know) make systemd
+optional, with fully-supported alternatives. And even Gentoo can’t be
+made 100% free of systemd – not in practice.
+
[Edited (November ’24) to add: thank you to everyone who wrote in to
+tell me about Artix, Devuan, Void, Calculate, MX, and others. It’s good
+to see that there are more distributions that continue to work without
+systemd than I previously thought.]
+
Is it even
+practical to avoid systemd any more?
+
Take, for example, the udev daemon. This service
+monitors the kernel, and makes configuration changes when devices are
+added and removed. It’s not an essential part of Linux, but it removes
+the need for a whole heap of manual configuration.
+
At the time of writing there is no fully-featured
+udev implementation outside of systemd. The original
+udev code was absorbed into the systemd project about ten
+years ago. The Gentoo alternative eudev is no longer fully
+maintained. At present, if your Linux distribution requires
+udev, you’re forced to use the version from systemd. This
+version can be used without the rest of systemd, but it nevertheless
+does systemd operations. In particular, it tries to communicate with
+systemd over DBus. This communication (so far) fails gracefully if
+systemd is not present, but it’s not clear how long this will continue
+to be the case.
+
systemd is a tightly-coupled set of services. I don’t think a design
+goal of the systemd maintainers is to make its components modular.
+udev doesn’t need to talk to systemd but, clearly, there’s
+some benefit to its doing so in a systemd installation. I understand
+that there was a kind of ‘gentlemen’s agreement’ between the systemd
+maintainers and the Gentoo maintainers, to keep udev
+independent of the rest of systemd. I think we can see that this
+agreement has broken down a little already; I suspect it will break down
+more, if the systemd folks think that tighter integration will make
+systemd work better.
+
Many parts of systemd continue – for now – to have non-systemd
+alternatives. For example, systemd has a clock synchronizer
+systemd-timesyncd. The systemd maintainers are perfectly
+honest that this software lacks features that exist in alternatives like
+Chrony and OpenNTPD, and it’s less accurate.
+systemd-timesyncd is included because it’s fast to start,
+and satisfactory for applications where exact time synchronization is
+not critical.
+
At present, Chrony remains widely used, even in some distributions
+(like Fedora) that use systemd. But with systemd becoming ubiquitous,
+what motivation will there be to maintain non-systemd alternatives?
+These alternatives could fall into disrepair, even though some are
+superior to the systemd utilities – and even the systemd maintainers
+admit this.
+
Similarly, systemd has a DHCP client. It isn’t the only DHCP client
+that is available, but my concern is that one day it might be. In my
+tests, I’ve found that the systemd components are larger, and slower to
+start, than the traditional alternatives (where they still exist).
+Again, this isn’t a criticism of systemd itself – I’m testing them in an
+environment they were not designed for.
+
So where does that
+leave embedded Linux?
+
I’ve found that many systemd components are less effective in an
+embedded environment than the traditional alternatives. I’ve shown some
+illustrative examples in this article, but I really don’t think there’s
+much controversy here: this simply isn’t the environment that systemd
+was designed for. But it’s getting increasingly difficult to find a
+mainstream Linux distribution that doesn’t use systemd – even Raspberry
+Pi distributions use it. As systemd absorbs more functionality into
+itself, there’s going to be little motivation to maintain alternatives.
+After all, if everybody uses systemd, what motivation is there to
+support anything else? My concern is that we’re moving towards a future
+where Linux is inconceivable without systemd. That will be a problem for
+those environments where systemd really doesn’t shine.
+
I wish I knew the solution to this problem. There’s no point
+complaining about systemd, because distribution maintainers have grown
+to like it too much. And, in any event, well-reasoned, technical
+concerns are drowned out by all the ranting and conspiracy theories. All
+we can do – if we care – is to continue to use and support Linux
+distributions that don’t insist on systemd, and stand ready to develop
+or maintain alternatives to those bits of Linux that it absorbs.
+
+
+
+
+
+
+
+
+
+
diff --git a/tpwin.html b/tpwin.html
new file mode 100644
index 0000000..9cc70da
--- /dev/null
+++ b/tpwin.html
@@ -0,0 +1,213 @@
+
+
+
+
+ Kevin Boone: They
+don’t make ’em like that any more: Borland Turbo Pascal 7
+
+
+
+
+
+
+
+
+
+
+
+
+
They
+don’t make ’em like that any more: Borland Turbo Pascal 7
+
+
Borland’s Turbo Pascal was one of the most successful
+programming tools of all time. It introduced development techniques,
+such as integrated source-level debugging, that we take for granted
+today; but which were radical at the time.
+
TP 7, also known as Turbo Pascal for Windows 1.5, was the
+pinnacle of the Turbo Pascal line, and its end. The name ‘Turbo’ was
+associated with what were regarded as hobbyist products – ‘Borland
+Pascal’ was a different, much more expensive, product. But Turbo Pascal
+was enough for some quite substantial applications, and was very
+affordable. It was also ‘book licensed’, which I think was a new thing
+in the software industry. You could buy one copy of TP, and any number
+of people could use it – but only one at a time. Even when it was
+distributed on floppy disks, there was no copy protection. (Turbo)
+Pascal eventually gave way to Delphi which, despite its technical
+merits, never had the same impact.
+
I used TP 7 to write medical imaging software back in the 90s, to run
+under Windows 3.1 – which was a great operating system in its day. Like
+previous releases, Windows 3.1 was still, essentially, an extension to
+MSDOS, but with clever tricks to allow it to multitask (after a fashion)
+and address more memory. It had a simple, consistent user interface
+model – both for the user and the programmer. It had, at that time, none
+of the wasteful and irritating anti-features that have come to make
+modern Windows so depressing to use. So Windows 3.1 was a platform we
+all wanted our applications to support – not that there were many
+alternatives at that time.
+
Like many other programmers, I learned the Pascal programming
+language specifically to use TP 7 – I was essentially a C programmer in
+those days, but programming for Windows in C was a horrible job.
+Frankly, it’s scarcely better today. It was, and is, necessary to
+combine a whole stack of command-line tools that take incomprehensible
+text files as input. TP 7 automated and concealed all that
+nastiness.
+
It wasn’t just the tools that were a problem: the user interface
+model was, essentially, object-oriented; but C did not make it easy to
+program in an object-oriented way. TP 5 had introduced basic
+object-oriented concepts to the Pascal language, and these were were to
+prove essential. The Windows-specific features were provided by
+something called Object Windows Library: a relatively thick,
+object-oriented wrapper around the basic Windows C API. My recollection
+is that Borland introduced an object-oriented programming library for
+Microsoft Windows before Microsoft did – and Borland’s was better.
+
+
+Turbo Pascal running in Windows
+3.1
+
+
OWL made it easy to create nice-looking user interfaces, with a bit
+of eye candy. The familiar ‘green tick button’ was an OWL enhancement,
+which graced so many Windows applications that it eventually became an
+irritating cliche.
+
All, in all, It was much easier to program for Windows using
+Turbo Pascal 7 than with anything else. Not only did it provide a
+programming model that matched the way the Windows user interface
+worked, the application itself had a Windows graphical interface – many
+Windows programming tools at that time actually ran under MSDOS, and
+were entirely text-based. TP 7 also had fully-graphical tools for
+designing the user interface elements, like menus and icons. Laying out
+a menu using a definition file with an obscure format, using Windows
+Notepad, was never an agreeable experience. Microsoft did produce
+graphical tools for this kind of operation, but Turbo Pascal combined
+them into a seamless IDE. All I had to do to build and run my programs
+was to hit the F7 key. I could even set breakpoints for the debugger,
+just by clicking a line of code. As I said, common enough today, but
+revolutionary for early Windows programming.
+
But what really made TP 7 special was its CP/M heritage. Turbo Pascal
+came to Windows by way of CP/M and MSDOS, which meant it had a compiler
+core that was designed to run with tight memory and a comparatively slow
+CPU. In the days when most compilers had to write a stack of
+intermediate files on disk, TP just constructed everything in memory. It
+did this using highly-optimized code, written in assembly language, by
+very skilled people. It doesn’t matter much today if your compiler spits
+out a heap of temporary files, because disk storage is fast and cheap.
+But in the days of the floppy disk, you had to minimize disk access.
+Unfortunately, you also had to minimize memory use, because
+there wasn’t much of that, either. Programming for CP/M took real skill
+and dedication.
+
So when TP came to Windows machines, with their 16-bit CPUs and
+megabytes of RAM, it was like an early Christmas for developers. A
+compiler designed to be adequate with 32kB of RAM and a single floppy
+drive will positively fly when given megabytes and a hard disk.
+
All this meant that TP 7 was a joy to use. It was like a modern IDE,
+but without any of the bloat. Developers who would never have tackled
+Windows programming with Microsoft tools were able to move from MSDOS to
+Windows without too much difficulty – even if they first had to learn
+Pascal, as I did. The compiled code was pretty speedy but, if it wasn’t
+brisk enough, you could write assembly language directly in the Pascal
+source files.
+
I don’t think it’s too much of an exaggeration to claim that Turbo
+Pascal contributed to the widespread uptake, and eventual domination, of
+Microsoft Windows on desktop PCs. If there had been a ‘Turbo Pascal for
+Linux’ in the early 90s, then we might all be running Linux on
+our desktops today.
+
So what happened to Turbo Pascal? Three things, at least, contributed
+to its eventual downfall.
+
First, there was increasing competition from the C programming
+language. Borland did produce a Turbo C, and later a Turbo C++, but they
+weren’t as influential as Pascal. It’s easy to forget that PCs weren’t
+the first general-purpose computing devices. Most scientific and
+educational establishments, and many businesses, used graphical
+workstations and a central minicomputer. C was already well-established
+in these environments before desktop computers started to become
+prominent. And before that we had mainframes and terminals – but that
+was a different world, where Fortran and COBOL ruled.
+
Most ‘serious’ PC applications were written in assembly language in
+the early days, as they had been for CP/M – the low resources almost
+necessitated this. Minicomputers were not so limited, so compilers were
+more practicable, even in the 1980s. So when compilers started to become
+of interest to microcomputer developers, there was almost no reason to
+favour one programming language over another. Pascal could have
+become the standard programming language for the PC. That it did not, I
+think, reflects the fact that many software companies that had been
+active in industrial and scientific computing saw an opportunity in PCs,
+and tried to move into that market. These companies already had
+substantial experience with C, and they brought their preferences with
+them. So while I was happy to learn a new programming language to
+develop for Windows, a software business with a thousand staff might not
+have wanted to take that step.
+
The second factor, I think, was the increasing palatability of
+Microsoft’s C tools. Microsoft did eventually produce an IDE of its own
+and, of course, Microsoft controlled the Windows platform. I don’t think
+Microsoft’s C IDEs ever became as easy to use as Borland’s, but
+they probably didn’t have to, given Microsoft’s position.
+
Third, Microsoft’s Visual BASIC started to attact small-scale and
+bespoke developers. BASIC was already a staple of the microcomputing
+world, because it was easy to learn, and did not require a separate
+compilation step. Attaching bits of BASIC code to Windows user interface
+elements was an efficient way to produce relatively simple, small-volume
+applications.
+
I loathed Visual BASIC in the 90s, and I loathed it until the day VB6
+drew its last laboured, wheezing breath in 2005. Still, I can’t deny its
+popularity, and I’m sure that many of the same people who were drawn to
+Turbo Pascal were later drawn to Visual BASIC, for much the same
+reasons. VB lowered the barriers to entry into Windows programming in
+the same way that TP had previously done. Arguably it lowered them too
+much, but let’s not go there today.
+
Early versions of Turbo Pascal are now freely, and legally,
+available. Retrocomputing enthusiasts with an interest in early PCs and
+MSDOS are well catered for. So far as I know, however, TP 7 remains
+somebody’s intellectual property, although it isn’t entirely clear
+whose. It probably comes into the category of ‘abandonware’ but, to be
+frank, it’s not all that important: it can’t produce code that runs on
+modern Windows systems, and few people have much interest in early
+Windows – it’s just not retro enough.
+
So that’s Turbo Pascal: a colossus of the microcomputing world, which
+faded and died largely unlamented. It always was a creature of its time
+– it prospered for the brief period when microcomputers were limited
+enough to reward skilled development, but widespread enough to create a
+substantial market for programmers. By the time it started to become
+difficult to find enough highly-skilled programmers to satisfy the
+market needs, it didn’t matter: computers were getting powerful enough
+to make good our limitations as programmers. That almost nobody has the
+skills these days to implement something like TP is something we could
+eventually come to regret – or, rather, our grandchildren could.
+Software development is becoming a semi-skilled industry, and I doubt
+that the rise of AI will entirely compensate for this.
+
But who needs a Turbo Pascal, when you have 128Gb of RAM attached to
+an 5GHz CPU, for a hastily-written, bloated Java program to burn?
+
+
+
+
+
+
+
+
+
+
diff --git a/vim1.html b/vim1.html
new file mode 100644
index 0000000..d16e69c
--- /dev/null
+++ b/vim1.html
@@ -0,0 +1,318 @@
+
+
+
+
+ Kevin Boone: A few interesting
+features of Vim, part 1
+
+
+
+
+
+
+
+
+
+
+
+
+
Almost every Linux installation includes the Vim editor. It’s far
+from the only editor available for Linux, but it’s very ubiquity makes
+it important. Vim is an extension of the traditional Unix
+vi editor, but with many, many more features. Vim is my
+preferred editor for programming, not just because I can rely on it
+being available, but because it’s actually rather good. Of course, it
+takes a bit of effort to be able to use it effectively, but that’s true
+of most powerful tools.
+
I doubt that anybody who uses Vim needs every feature it offers.
+However, there are some features that don’t seem to be widely known, and
+which can be quite useful. In this article I’ll begin to uncover some of
+them.
+
+
Please note that I won’t be describing anything that needs particular
+plug-ins or extensions. There are many, very powerful plug-ins for Vim,
+and it’s certainly worth seeking them out if you can’t make Vim do
+something you want. Nevertheless, in this article I’ll only be
+discussing things that every installation of Vim should be able to
+do.
+
+
These suggestions are in no particular order, but some of the later
+ones do rely on earlier ones, in order to make sense.
+
Using horizontal splits
+
Personally, I don’t find it helpful to run Vim with many buffers and
+splits, although it’s possible. Still, it’s useful to understand the
+basics of splits, because so many other features rely on them.
+
If you’ve opened a file full-screen in Vim, you can open another file
+using
+
:n(ew) {filename}
+
By default this will split the screen horizontally, putting the new
+file into a screen region above the existing one.
+
You can select which split is active using various
+ctrl-w keystrokes, of which Vim has a profusion. Since I
+use relatively few splits, I’ve found the easiest keystrokes to remember
+are ctrl-w followed by an arrow key. This activates the
+split in the direction of the arrow (and so works with both horizontal
+and vertical splits).
+
Another useful key combination is ctrl-w, o which shows
+the current split region full-screen. You can restore the split using
+:sp, which will show the same file in both split regions,
+followed by ctrl-^, which loads the previous buffer into
+the split. These kinds of operations get increasingly fiddly when you
+have more than one split.
+
Filtering lines in log files
+
Suppose you want to see all the lines in a log file that contain the
+text WARN. This is a good application for the ‘global’ (g)
+command. This command applies a regular expression to each line in the
+buffer, and executes a (Vim) command on each matching line.
+
With no command, g just feeds the matches to a pager.
+So, for example:
+
:g/WARN
+
selects all the lines matching WARN and passes them to a pager. You
+can scroll the pager, but you can’t edit the results, or switch
+back-and-forth between the original and the filtered version.
+
To apply the change more persistently, you can run g
+with the d(elete) command. However, in this case, we want
+to delete non-matching lines. We can use either
+
:g!/WARN/d
+
or
+
:v/WARN/d
+
depending on which form is easier to remember – they work
+identically.
+
These commands delete all lines that do not match WARN, and leaves
+the remainder for regular editing. You can restore the missing lines
+just with the undo (u) operation. What you can’t
+easily do, however, is to make edits to the lines that contain WARN, and
+then restore the missing lines. The undo command will just undo your
+edits.
+
An alternative method for managing complexity, particularly in a
+structured document like source code, is to use folding, as
+described below.
+
Compilation and the quickfix
+list
+
Vim has specific support for running compilation commands, and
+jumping to files and lines that contain errors. This support makes use
+of the quickfix list, although there are other applications
+of the quickfix list, unrelated to compilation, that I’ll touch on
+later.
+
By default the command
+
:make
+
runs the ordinary make command, parses the results, and
+puts matching error lines into the quickfix buffer. If there are errors,
+you can run
+
:copen
+
to show the quickfix buffer. By default :copen splits
+the window horizontally, with the quickfix buffer at the bottom. You can
+use the ctrl-w commands, as described above, to cycle
+between the split regions. More importantly, you can just select any
+line in the quickfix buffer that begins with a filename and a line
+number, and hit Enter. This will select the relevant file and line in
+the source code buffer above the quickfix window.
+
If you don’t want to have the quickfix list open on screen, you can
+still select error lines using commands like :cnext.
+However, I’ve not found this very helpful, because I find that I need to
+see all the nested error lines to understand the error.
+
You can change what command :make runs using ‘:set
+makeprg=…’. However, whatever command you use must produce output in a
+format that Vim can parse. I use Maven for compiling Java and, so far as
+I know, :make can’t parse Maven’s output without additional
+support. There are various plug-ins for this.
+
Searching for patterns
+in a source tree
+
When I’m writing code, I often leave notes for myself like this:
+
// TODO: add error checks
+
Suppose you want to find all remarks like this, in all the source
+files in your application, and edit the corresponding files. You could,
+of course, use the ordinary grep utility in the console. A
+nicer way is to use the grep support built into Vim. Its
+format is similar to the command-line version, but you can’t (so far as
+I know) pass arguments. To search for TODO in every source
+file under the current directory You could do this:
+
:grep TODO **/*.c
+
:grep also uses the quickfix list. So, after running the
+command, you can use :copen to see the results, and jump to
+the relevant places in the files where Vim found the pattern.
+
Like the ordinary command-line grep, :grep
+searches for regular expressions, not just specific text. There’s an
+alternative :Vimgrep that uses Vim’s regular expression
+extensions (which need an article all to themselves).
+
Using an external code
+formatter
+
Vim has a built-in code formatter. To use it, all you have to do is
+to select a block of text, and hit the = key. This formats
+according to various built-in formatting parameters which you can
+configure, to some extent. To format the entire file, you can hit
+gg=G.
+
I find that I can’t get the ‘Whitesmiths’ formatting that I prefer,
+using only Vim. I can get close, but not close enough. In any event,
+external formatting tools will generally give better results than Vim’s
+built-in method, particularly when optimal formatting requires an
+understanding of the structure of the code, rather than just matching
+patterns. The differences are quite subtle, but sometimes
+preferable.
+
Moreover, sometimes I have to use other code styles than the one I
+prefer, particularly if I’ve working on somebody else’s project. Again,
+I find it easier to use an external formatting tool, particularly one
+that just takes the name of a style on the command line.
+
My preferred code formatting tool is clang-format. It
+has built-in style names that I can just pass on the command line (e.g.,
+‘--style=gnu’), but it can also take a style file, which
+has several hundred parameters for controlling the format in fine
+detail. With clang-format I can get exactly the
+code layout I like.
+
Vim has a method to apply an external command to a region of text.
+The basic syntax is simply
+
:!command
+
So, to run clang-format on the entire file you can
+run:
+
:1,$!clang-format --style=gnu
+
Here, 1,$ is a region specification, stretching from the
+first line to the end of the file. Or, to use your own style file:
+
:1,$!clang-format --style=file
+
You can also select a region of text, and just run the colon command
+on the selection, without needing to specify a region.
+
Vim expects the external utility to take text from standard input,
+and write the result to standard output. You can wrap up the command in
+a script, if it works in a different way. It’s sometimes helpful to use
+a script anyway, if the command is long, or has fiddly arguments.
+
This technique can also be used for decoding text in place, but this
+requires care. For example, I find in my work that I often have to look
+at configuration files where values are base64-encoded. For example:
+
text=SGVsbG8sIFdvcmxkCg==
+
You might think that you could just select the encoded text, and then
+run :!base64 -d. That doesn’t work, because Vim will always
+output whole lines to the external utility. So, to use this
+technique, you’ll have to first split out the encoded text onto its own
+line.
+
Using marks to navigate a
+large file
+
I often find that I’m looking at a particular part of a source file,
+then I have to go and look at some other place in the same file. How can
+I get back to the original point?
+
One way to do this is simply to add some marker characters to the
+file at the original point and then, when you need to get back, just
+search for those characters. Of course, you have to use characters that
+don’t otherwise appear in the file and, of course, you have to delete
+them afterwards.
+
A less invasive method is just to use marks. There are some
+complications concerning the use of multiple buffers but, for simple
+applications, these need not concern us.
+
With that in mind, Vim supports 26 marks per buffer. They are denoted
+by the characters a to z. So, to set mark ‘a’
+on the current line, just enter ma. Then, to jump back to
+that point, just enter 'a (that’s an apostrophe followed by
+a). Of course, you can use the other letters to set other
+marks; but I find it difficult to remember what they mean, when I use
+more than two or three.
+
Entering special
+characters using digraphs
+
+
Your operating system or graphical desktop might provide other ways
+to get special characters, but the Vim method is
+platform-independent.
+
+
Digraphs are introduced using ctrl-k, followed by two
+characters. You’ll see the character being built up on the status line.
+Vim digraphs are kind-of logical. For example, to get é (‘e’ with an
+acute accent) enter ctrl-k e '. The apostrophe has the same
+general shape as the acute accent. To get è (‘e’ with a grave accent)
+use ctrl-k e‘. The back-tick character, again, looks a bit
+like the accent. An umlaut character (as in ’ö’) is signalled by the
+colon (:) character, while the circumflex (as in ‘ô’) is signalled by
+‘>’. Both these characters look a little like rotated versions of the
+accents they stand for.
+
Not all digraphs are logical, though; and the ones that are may not
+always be easy to remember. You can get a full list of digraphs by
+executing :dig.
+
Managing complexity using
+folding
+
Folding is the process of temporarily hiding sections of a file. This
+can be particularly effective with source code, where there are clear
+markers of the beginning and end of a section.
+
For example, consider the long and complicated function definition
+zip_read_header_from_cd below.
+
+
+Before folding the function
+
+
To hide the whole of this function, just leaving the definition in
+place, select the opening brace character, and hit zf%.
+zf means ‘create a fold’; the % specifies how
+to move the selection point. This action will replace the whole function
+body with a fold marker, as here:
+
+
+After folding the function
+
+
The text is still present in the file, it’s just not visible. There
+are many other ways to define the boundaries of a fold; you could, for
+example, just select a text block and then do zf to fold
+away the selection. Folds can even by nested, one inside the other,
+although I don’t find nested folds easy to navigate in Vim.
+
To reveal the folded text, just move to the fold marker, and enter
+zo (‘o’ for ‘open’). Interestingly, Vim remembers where the
+folds are, even if you open them. So if you enter zc (‘c’
+for ‘close’) in a region of text that is within a fold, the whole fold
+closes again. Of course, you can’t actually see the fold boundaries, so
+this only works in specific circumstances.
+
To open all folds, just hit zR.
+
Using abbreviations to
+speed things up
+
Vim supports abbreviations: when you type certain keystrokes,
+followed by a space, the keystrokes are replaced immediately by some
+other text. The replacement can be quite sophisticated: it might contain
+function calls, for example.
+
In practice, I use this feature for correcting words that I
+consistently mistype. One such word is ‘paramters’. To create an
+abbreviation spelling correction, I execute:
+
:ab paramters parameters
+
(I had to turn this abbreviation off, in order to type the misspelled
+word into this page – otherwise Vim just corrects it).
+
Of course, commands like this are best placed in a start-up file, so
+they are always available.
+
It’s worth reading the Vim manual if you want to use abbreviations,
+because there are various subtleties about how they work in various
+modes.
+
Closing remarks
+
This article has outlined some powerful features of Vim, but there’s
+much, much more available. Features like folding and marks are rather
+more complicated than this article suggests, and do merit a close
+reading of the documentation.
Do you need to
+wipe that hard drive? And can you?
+
+
TL;DR: yes, and maybe. If the drive contains state defence secrets,
+it’s probably better to crush it.
+
The problem
+
I’ve got a couple of laptops that I’d like to sell. They contain hard
+drives (SSDs, in this case) on which is data I’d not like to fall into
+anybody else’s hands. I’ve also got a bunch of high-capacity magnetic
+disks that I’ve retired from a RAID array.
+
Can I sell these things safely? It’s worth bearing in mind that the
+financial gains – for me – are minimal. If you’re a computer dealer, or
+a corporate IT department, then things might be different, both in the
+sums of money involved, and the sensitivity of the data.
+
Because individual used drives don’t command high prices on the
+second-hand market, there are people or organizations buying them up,
+just to see what data they contain. Most recovered data will be garbage
+to anything but the previous owner; but if even one in a thousand
+second-hand drives contains a dump of credit card details, that’s a big
+win for the villains. And if even one in a million turns out to contain
+state secrets, that’s a jackpot.
+
My second-hand drive may prove to contain an email password buried in
+a swapfile or temporary file somewhere, or something of a similar level
+of sensitivity. I encrypt all the sensitive data on my drives, but I
+can’t control where the operating system might buffer it. Now, to be
+sure, the chances of anybody who buys my old drives having the skills to
+unpack that data, and its being worth their while doing so, are minimal.
+I’m not very concerned. Nevertheless, if I can easily reduce that risk
+to near-zero (and it’s never going to be zero), then I probably
+should.
+
The corporate IT department certainly should.
+
So, yes, I probably want to wipe all the data from a hard drive that
+I plan to sell or re-use. The alternative, of course, is to smash the
+drive to pieces with a club-hammer; and that may be appropriate in some
+cases.
+
What
+about disk encryption as an alternative to wiping?
+
If you use whole-disk encryption, doesn’t the problem go away? After
+all, if the data on the disk is robustly encrypted, perhaps you don’t
+even need to wipe it?
+
The problem here is that, if the data is very valuable, it’s been
+known for people to be coerced or blackmailed into handing over their
+encryption passwords. It’s better, I think, for there to be no data to
+recover in the first place.
+
So even for my domestic data, I’m not going to sell a laptop or a
+hard drive without taking reasonable steps to ensure that my personal
+data is not recoverable. What ‘reasonable’ amounts to, of course,
+depends on the circumstances.
+
Possible solutions
+
Here are some methods that I might use to prepare a drive for
+sale.
+
+
Writing a new filesystem on top of the old one (‘formatting’ is the
+Windows term)
+
Using erasure utilities built into PC firmware (BIOS)
+
Invoking a drive’s built-in erasure procedure
+
Writing zeros, or random data, to each disk sector, perhaps
+repeatedly
+
+
Unreliable approaches
+
The first of these methods, writing a new filesystem, we can dispose
+of quickly. This won’t defeat anything but the most casual intrusion.
+Nearly all the original data will still be present, and there are many
+widely-available utilities for recovering data in this situation. This
+is simply not a safe procedure to use on drives that contain any kind of
+sensitive data.
+
If your computer has firmware (BIOS) with a built-in disk-wiping
+feature then, of course, you could consider using it for convenience.
+Such an approach will be easier than taking the drive out and wiping it
+using a different machine. It’s hard to dismantle a modern compact
+laptop, and you probably won’t be able to wipe the operating system
+drive, when the operating system is running.
+
However, these firmware facilities really are just for convenience –
+they generally do one of the other things I’ve listed above. If you’re
+lucky, you may get to choose which one.
+
Because firmware methods don’t offer anything in principle
+that other methods don’t, I won’t consider them further. If you want to
+wipe a laptop drive without removing it, you could always boot the
+laptop from a USB stick or external drive.
+
Using built-in disk
+erasure features
+
So, turning to proprietary methods: modern SATA drives almost always
+have a built-in erasure method. This can be invoked by sending the drive
+a specific command. On a Linux system you can do this easily using the
+hdparm utility:
+
# hdparm --security-erase ...
+
or
+
# hdparm --security-erase-enhanced ...
+
Again, if it’s the operating system disk you want to erase, you’ll
+have to boot from an external operating system drive to do this.
+
What these proprietary procedures actually do depend entirely on the
+drive manufacturer. On magnetic drives, they probably overwrite the
+individual sectors, which is something that can be done better using
+utilities specifically designed for this purpose. On SSDs, it’s even
+less clear what they do. SSD technology does, in principle, allow for
+some quite rapid erasure techniques. If the SSD is self-encrypting, for
+example, then all the erasure might do is to overwrite the encryption
+password in the drive firmware with a new one. This is a shockingly fast
+way to erase an SSD, if it’s supported. There are other fast-erasure
+methods that a drive might implement.
+
Unfortunately, I’ve seen reports that drives from a number of major
+manufacturers have weaknesses in their secure erasure methods. I don’t
+know how reliable these reports are (the Internet being what it is) but,
+for example, if the encryption passwords generated by the drive firmware
+are guessable (perhaps they use weak random number generation), then any
+change of the password will be useless. It’s certainly
+plausible that vendors could use weak methods, and keep quiet
+about it.
+
Before using a SATA drive’s built-in erasure method, I would want to
+satisfy myself that it’s reliable. This might not be easy – the Internet
+is full of misinformation, and drive vendors are not particularly
+forthcoming with information in this area. As ever, you have to weigh up
+the risks and benefits.
+
There is another reported problem with using the built-in
+erasure method in SATA drives. Again, it’s not a problem I’ve had, and I
+don’t know how common it is. If it’s an external drive with a USB
+interface, then (apparently) the USB protocol can time out in the middle
+of the erase operation. If this happens then (apparently) you’re left
+with a dead drive. I guess if this does happen, it’s more likely to
+happen with magnetic disks, where erasure tends to be slow. And if the
+alternative to erasing the disk is to smash it with a club-hammer, it
+might be worth the risk. As I say, I’ve not encountered the problem, and
+I don’t know how likely it is.
+
Some drive vendors supply diagnostic software that has an erasure
+feature, in addition to the drive’s responding to the SATA erase
+command. In the end, though, I prefer not to trust vendor’s erasure
+methods. Not because I don’t think they work, but because I can’t be
+sure how they work. I prefer to have these things under my
+control, so I can assess the risks and benefits.
+
Drive-wiping applications
+
This brings us to brute-force erasure methods.
+
For magnetic disks, the ‘gold standard’ method of erasure is to write
+each sector repeatedly with random numbers. If the intruder is very
+determined, it seems that a single overwrite is not enough. Each change
+in magnetic state leaves traces that can be recovered with specialized
+hardware.
+
Now, to be sure, this isn’t the kind of hardware that a bloke in the
+pub will have access to. To need an approach to erasure like this, you’d
+have to be very paranoid, or be storing national defence plans.
+
For most personal data, my feeling is that simply overwriting the
+sectors with a single pass of random numbers will be sufficient, on any
+type of drive. I do this often enough that I wrote my own software to
+do it. Why would I do that? There are many existing utilities that
+can do this kind of disk erasure, but mine combines the erase with a
+read-back test, so I can be sure that the disk remains fully functional
+after being wiped.
+
It’s worth bearing in mind that applying even a single pass of
+writing every sector will take a long, long time on a large magnetic
+disk – hours to days. It will be quicker on an SSD, but you might not
+want to do it on an SSD.
+
Can we wipe SSDs?
+
Is there an issue in this area with SSDs? There’s plenty of
+information (or misinformation) on the Internet to suggest that there
+might be. There seem to be two main (potential) problem areas.
+
First, SSDs have limited write lifetimes. They are limited both in
+the number of times a specific sector can be written, and in the total
+amount of data that can be written. This much, at least, is not
+contentious. There’s no question that writing every sector will have
+some impact on the drive’s long-term service life. But how
+much?
+
Modern SSDs have claimed sector-write lives of perhaps 100,000
+cycles. Total write lifetimes are claimed to be of the order of hundreds
+of terabytes. It seems to me that a single pass of writing every sector
+isn’t going to have any practical impact on drive life. Given the way
+that SSDs work, I see no evidence that writing more than a single pass
+will increase the effectiveness of the erasure anyway.
+
The second problem is more subtle. Modern SSDs are
+overprovisioned; that is, they contain more storage than they
+advertise. The ‘spare’ storage is used to replace sectors that have
+failed in use. A failed sector may still contain sensitive data – you
+just won’t be able to read or write it using any common tools. Disk
+vendors sometimes provide proprietary tools for inspecting this kind of
+data. Such tools, if available, could probably get data from failed
+sectors that are inaccessible to an operating system.
+
Another potential problem with SSDs is that they don’t necessarily
+store data in predictable ways – predictable to the user, that is. For
+example, because so many sectors end up containing only zeros, a drive
+might have an ‘all zero’ flag that it writes, rather than writing the
+actual zeros. This is an optimization technique. So if you overwrite a
+sector with zeros, it might not change the actual stored data in any
+predictable way. Proprietary tools may be able to recover the underlying
+data, which was not changed by the write. Again, an ordinary operating
+system will not see this data – the drive’s data management is
+completely transparent. You’d need very specialized tools to get to this
+data.
+
Then there’s the issue of wear leveling. Because of the
+potential risks of over-writing sectors, SSDs don’t necessarily store
+the same logical sector in the same physical location.
+So overwriting a particular sector might not actually change the
+contents of the physical sector – the drive might write the new data in
+a different physical location. However, my feeling is that if you write
+every sector, wear leveling won’t be a problem, except if it’s
+combined with extensive overprovisioning.
+
My own feeling is that overwriting each sector of an SSD with random
+data is both safe in practice, and reliable enough for ordinary domestic
+data. As ever, though, your plans for world domination might need more
+careful handling. And the club-hammer is always on option.
+
Summary
+
Despite extensive research, I remain conflicted. I’m fairly sure that
+my data is of limited use to anybody but me. In addition, I’m fairly
+sure that I can erase it well enough that nobody but the most determined
+intruder will be able to get at it, and even then only with great
+difficulty.
+
And yet… and yet. A private individual selling old laptop PCs and NAS
+drives isn’t going to make a whole lot of money. Is it worth even the
+tiny risk? I guess that depends on how much I need the money.
+
+
+
+
+
In the days
+before an automotive satnav unit was just an extension of a smartphone,
+the Garmin Nuvi 300 was the pinnacle of this technology. Garmin’s
+competitors – of which there were, and remain, few – might not have
+agreed with that assessment, but the Nuvi 300 was feature-packed, and
+undeniably practical. The automotive satnav market has been in
+decline ever since.
+
+
+
+
+
+