Even seasoned Linux developers speak of Gentoo in hushed tones. It's not for the faint-hearted, that's for sure. But how stout-hearted must you be, for Gentoo to be appropriate?
Desktop Linux will take off next year -- or so people have been saying for years. Do desktop containerization technologies like Flatpak make this more, or less, likely?
Even seasoned Linux developers speak of Gentoo in hushed tones. It's not for the faint-hearted, that's for sure. But how stout-hearted must you be, for Gentoo to be appropriate?
diff --git a/eclipse.html b/eclipse.html
index 7f3abcd..bd18925 100644
--- a/eclipse.html
+++ b/eclipse.html
@@ -2,8 +2,7 @@
- Kevin Boone: Predicting eclipses with
-clockwork
+ Kevin Boone: Predicting eclipses with clockwork
@@ -41,111 +40,36 @@
-
Predicting eclipses with
-clockwork
+
Predicting eclipses with clockwork
-
In my retrocomputing articles I normally look at computing and
-electronics between 1970 and about 2000. In this article, just for a
-change, I’ll be looking back about 2300 years, to a mechanical computer
-– the Antikythera Mechanism.
-
The story of the discovery of this device in a shipwreck by pearl
-divers is well-known, and I won’t go into it here.
-
Over the last hundred years or so we’ve achieved what we believe is a
-reasonable understanding of how this device worked and, to a lesser
-extent, how it was used. This is a remarkable feat in itself,
-considering that the surviving mechanical parts are gummed together by
-two millennia of seabed accretions.
+
In my retrocomputing articles I normally look at computing and electronics between 1970 and about 2000. In this article, just for a change, I’ll be looking back about 2300 years, to a mechanical computer – the Antikythera Mechanism.
+
The story of the discovery of this device in a shipwreck by pearl divers is well-known, and I won’t go into it here.
+
Over the last hundred years or so we’ve achieved what we believe is a reasonable understanding of how this device worked and, to a lesser extent, how it was used. This is a remarkable feat in itself, considering that the surviving mechanical parts are gummed together by two millennia of seabed accretions.
-
-The AntiKythera Mechanism, now in the
-National Archaeological Museum in Athens, Photo by
-zeevveez.
+The AntiKythera Mechanism, now in the National Archaeological Museum in Athens, Photo by zeevveez.
-
I’m a little sceptical of the claim that the Antikythera Mechanism
-was used for navigation. Although it’s clearly a computing device, my
-feeling is that was more likely used on the desktop, like an orrery – a
-mechanical model of the motions of the moon and planets. The makers of
-the Mechanism probably had an Earth-centered view of the Solar System,
-which makes computing the movements of the planets even more difficult
-than it is with our modern understanding. The gearing that was used to
-model the apparent motion of the planets – if we have understood it
-correctly – is truly remarkable.
-
In this article, however, I will tackle something a little simpler:
-using a clockwork device to predict eclipses of the Sun and Moon. It’s
-almost certain that the Mechanism was used this way – the clue is the
-gear wheel with 223 teeth that can be seen in the photo above. I’ll
-explain why this number is significant later. It turns out that
-modelling eclipses without celestial mechanics is not conceptually all
-that difficult – provided we have acute observational skill and
-extraordinary patience over decades.
+
I’m a little sceptical of the claim that the Antikythera Mechanism was used for navigation. Although it’s clearly a computing device, my feeling is that was more likely used on the desktop, like an orrery – a mechanical model of the motions of the moon and planets. The makers of the Mechanism probably had an Earth-centered view of the Solar System, which makes computing the movements of the planets even more difficult than it is with our modern understanding. The gearing that was used to model the apparent motion of the planets – if we have understood it correctly – is truly remarkable.
+
In this article, however, I will tackle something a little simpler: using a clockwork device to predict eclipses of the Sun and Moon. It’s almost certain that the Mechanism was used this way – the clue is the gear wheel with 223 teeth that can be seen in the photo above. I’ll explain why this number is significant later. It turns out that modelling eclipses without celestial mechanics is not conceptually all that difficult – provided we have acute observational skill and extraordinary patience over decades.
How we get eclipses
Let’s start by looking at how we get eclipses in the first place.
-
To get a solar eclipse of any kind, the Sun, Moon, and Earth must be
-approximately in a line, with the Moon between the Sun and the Earth
-(see diagram below). To get a lunar eclipse, the Earth must be between
-the Sun and the Moon. The time when the Moon lies between the Sun and
-the Earth we call ‘new moon’. At this time, the Moon is not generally
-visible from the Earth, because there’s nothing to illuminate the face
-that we can normally see. At new moon, we’ll only see the moon at all
-when it happens to be causing an eclipse.
-
If a solar eclipse can only happen at new moon, then a lunar eclipse
-can only happen at full moon. So, very often, we’ll see a solar eclipse
-at the new moon following a lunar eclipse. I’ll explain why this is
-important later.
-
The time between consecutive new moons is called the synodic
-month, and it’s about 29.5 Earth days in duration.
-
The Earth and all the other planets of our solar system orbit the Sun
-in what is very close to a plane. That is, we could put them all on a
-(very large) flat surface. We call this plane the ecliptic. If
-the Moon also orbited the Earth in this plane, then we’d see an eclipse
-every synodic month – a solar eclipse at new moon and a lunar eclipse at
-full moon. But that isn’t the case – the orbit of the Moon around the
-Earth is inclined at about 5 degrees to the ecliptic. So, while it has
-to be new moon to see a solar eclipse, we won’t see one every new moon.
-In nearly all cases, the Moon will be too far from the ecliptic for its
-shadow to fall on the Earth. The drawing below shows this situation,
-where the tilt of the Moon’s orbit at a particular time makes it
-unlikely that there will be any eclipse, even at new moon or full
-moon.
+
To get a solar eclipse of any kind, the Sun, Moon, and Earth must be approximately in a line, with the Moon between the Sun and the Earth (see diagram below). To get a lunar eclipse, the Earth must be between the Sun and the Moon. The time when the Moon lies between the Sun and the Earth we call ‘new moon’. At this time, the Moon is not generally visible from the Earth, because there’s nothing to illuminate the face that we can normally see. At new moon, we’ll only see the moon at all when it happens to be causing an eclipse.
+
If a solar eclipse can only happen at new moon, then a lunar eclipse can only happen at full moon. So, very often, we’ll see a solar eclipse at the new moon following a lunar eclipse. I’ll explain why this is important later.
+
The time between consecutive new moons is called the synodic month, and it’s about 29.5 Earth days in duration.
+
The Earth and all the other planets of our solar system orbit the Sun in what is very close to a plane. That is, we could put them all on a (very large) flat surface. We call this plane the ecliptic. If the Moon also orbited the Earth in this plane, then we’d see an eclipse every synodic month – a solar eclipse at new moon and a lunar eclipse at full moon. But that isn’t the case – the orbit of the Moon around the Earth is inclined at about 5 degrees to the ecliptic. So, while it has to be new moon to see a solar eclipse, we won’t see one every new moon. In nearly all cases, the Moon will be too far from the ecliptic for its shadow to fall on the Earth. The drawing below shows this situation, where the tilt of the Moon’s orbit at a particular time makes it unlikely that there will be any eclipse, even at new moon or full moon.
-
-The Earth-Moon-Sun system, not at all to
-scale
+The Earth-Moon-Sun system, not at all to scale
-
The points at which the orbit of the Moon crosses the ecliptic are
-called its orbital nodes. There are two – a rising node, and a
-falling node. The time bateween nodes is called the draconic
-month, and it’s about 27.2 Earth days. That the draconic and
-synodic months are nearly the same is, presumably, just a
-coincidence.
+
The points at which the orbit of the Moon crosses the ecliptic are called its orbital nodes. There are two – a rising node, and a falling node. The time bateween nodes is called the draconic month, and it’s about 27.2 Earth days. That the draconic and synodic months are nearly the same is, presumably, just a coincidence.
-
Note In case you were wondering, draconic is
-indeed related to ‘dragon’. In mythology, a lunar eclipse was envisaged
-as a dragon ‘eating’ the Moon.
+
Note In case you were wondering, draconic is indeed related to ‘dragon’. In mythology, a lunar eclipse was envisaged as a dragon ‘eating’ the Moon.
-
So to see some kind of eclipse, at least the following conditions
-must apply:
+
So to see some kind of eclipse, at least the following conditions must apply:
-
It must be new moon (for a solar eclipse) or full moon (for a
-lunar eclipse).
-
The Moon must lie close to one of its nodes; that is, it must lie
-close to the ecliptic plane. It doesn’t have to be stop-on; how far from
-the ecliptic the Moon can be, and still cast a visible shadow on the
-Earth, depends on a number of other factors.
+
It must be new moon (for a solar eclipse) or full moon (for a lunar eclipse).
+
The Moon must lie close to one of its nodes; that is, it must lie close to the ecliptic plane. It doesn’t have to be stop-on; how far from the ecliptic the Moon can be, and still cast a visible shadow on the Earth, depends on a number of other factors.
-
One of those other factors is how far from the Earth the Moon is, at
-the time under consideration. For a solar eclipse, the further away the
-Moon is from the Earth, the larger the shadow it will cast, and the more
-extensive the area from which the eclipse can be seen. The Moon’s orbit
-around the Earth is elliptical; the closest point is called
-perigee, the furthest apogee. The time between
-successive apogees is called the anomalistic month, and is
-about 27.6 Earth days. It’s very similar to the draconian month of 27.2
-days. Again, this is probably just a coincidence.
-
So, for the purposes of eclipse prediction, there are three different
-kinds of month that we must consider:
+
One of those other factors is how far from the Earth the Moon is, at the time under consideration. For a solar eclipse, the further away the Moon is from the Earth, the larger the shadow it will cast, and the more extensive the area from which the eclipse can be seen. The Moon’s orbit around the Earth is elliptical; the closest point is called perigee, the furthest apogee. The time between successive apogees is called the anomalistic month, and is about 27.6 Earth days. It’s very similar to the draconian month of 27.2 days. Again, this is probably just a coincidence.
+
So, for the purposes of eclipse prediction, there are three different kinds of month that we must consider:
@@ -178,192 +102,47 @@
How we get eclipses
Basic eclipse prediction
-
If I see a solar eclipse of some sort today, at my location,
-then my next strong chance of seeing an eclipse is in six synodic
-months’ time. In that time the moon will have completed approximately
-six draconic months, and be close to the ecliptic plain again (it’s
-actually about 6.4 draconic months, because the draconic month is a
-little shorter than the synodic). You might think that the best chance
-to see an eclipse would be a whole number of draconic months since the
-last eclipse, because the moon’s orbit will be crossing the ecliptic
-again, and in line with the Earth and the Sun. But, no – we only see
-solar eclipses at all on or near new moon, and lunar eclipses
-at full moon. Remember that the Moon must not only be near the ecliptic,
-but it must be in line with the Earth and the Sun as well.
-
But six synodic months later is not the only time we might
-see an eclipse.
-
Suppose that, during the today’s eclipse, the Moon was just below the
-ecliptic. At the next new moon, it will be just above the
-ecliptic. That is, it might still be at a point where it could cast a
-shadow on the Earth, and cause an eclipse somewhere. But two synodic
-months later, the Moon will definitely be too far out of line.
-
Similarly, if six months from now the Moon was just able to produce
-an eclipse, being just at the limit of its position with respect to the
-ecliptic, then conceivably there might be an eclipse a (synodic) month
-earlier – five synodic months from today.
-
It turns out that, in practice, if there’s an eclipse today, the next
-eclipse – if there is one – will probably be six synodic months away.
-Less likely, but still worth considering, are intervals of five and six
-synodic months.
-
So, essentially, mechanical eclipse prediction comes down to timing
-synodic months – the phases of the Moon.
-
There’s a lot of probabilistic qualifications in the preceding
-discussion: what if we want to be sure when the next eclipse
-will be? Is that even possible?
+
If I see a solar eclipse of some sort today, at my location, then my next strong chance of seeing an eclipse is in six synodic months’ time. In that time the moon will have completed approximately six draconic months, and be close to the ecliptic plain again (it’s actually about 6.4 draconic months, because the draconic month is a little shorter than the synodic). You might think that the best chance to see an eclipse would be a whole number of draconic months since the last eclipse, because the moon’s orbit will be crossing the ecliptic again, and in line with the Earth and the Sun. But, no – we only see solar eclipses at all on or near new moon, and lunar eclipses at full moon. Remember that the Moon must not only be near the ecliptic, but it must be in line with the Earth and the Sun as well.
+
But six synodic months later is not the only time we might see an eclipse.
+
Suppose that, during the today’s eclipse, the Moon was just below the ecliptic. At the next new moon, it will be just above the ecliptic. That is, it might still be at a point where it could cast a shadow on the Earth, and cause an eclipse somewhere. But two synodic months later, the Moon will definitely be too far out of line.
+
Similarly, if six months from now the Moon was just able to produce an eclipse, being just at the limit of its position with respect to the ecliptic, then conceivably there might be an eclipse a (synodic) month earlier – five synodic months from today.
+
It turns out that, in practice, if there’s an eclipse today, the next eclipse – if there is one – will probably be six synodic months away. Less likely, but still worth considering, are intervals of five and six synodic months.
+
So, essentially, mechanical eclipse prediction comes down to timing synodic months – the phases of the Moon.
+
There’s a lot of probabilistic qualifications in the preceding discussion: what if we want to be sure when the next eclipse will be? Is that even possible?
The saros cycle
-
The best prediction of an eclipse so far seems to be ‘six synodic
-months from the last one’. Maybe. Can we do better? In modern times we
-certainly can, because we have accurate mathematical models of our solar
-system. But how would the ancients have done it?
-
The key is to find how many synodic months have to pass before a
-whole number – or close to whole number – of draconic months have
-passed. At this point we will again be at new moon, with the moon at or
-close to the ecliptic.
-
As a bonus, it would help if the Earth-Moon distance were the same at
-this point in time as it was at the initial point, because that factor
-dictates the size of the shadow cast on the Earth or the Moon.
-
The synodic, draconic, and anomalistic months are not multiples of
-one another and, because they are so similar in duration, it will take a
-long time (perhaps forever) before the Earth, Moon, and Sun are in the
-exact same alignment again. If we look for an alignment where a
-whole number of synodic, draconic, and anomalistic months all occur
-within a 24-hour period, it turns out that 223 synodic months, which
-is
-6585.32 days, is almost exactly the same as 242 draconic months –
-6585.36 days, and 239 anomalistic months – 6585.54 days. It’s not a
-perfect alignment, but we have to wait a long time to find a better one.
-If we want the alignment to be within a period of a couple of hours,
-we’ll have to wait 3263 synodic months, which is about 263 years.
+
The best prediction of an eclipse so far seems to be ‘six synodic months from the last one’. Maybe. Can we do better? In modern times we certainly can, because we have accurate mathematical models of our solar system. But how would the ancients have done it?
+
The key is to find how many synodic months have to pass before a whole number – or close to whole number – of draconic months have passed. At this point we will again be at new moon, with the moon at or close to the ecliptic.
+
As a bonus, it would help if the Earth-Moon distance were the same at this point in time as it was at the initial point, because that factor dictates the size of the shadow cast on the Earth or the Moon.
+
The synodic, draconic, and anomalistic months are not multiples of one another and, because they are so similar in duration, it will take a long time (perhaps forever) before the Earth, Moon, and Sun are in the exact same alignment again. If we look for an alignment where a whole number of synodic, draconic, and anomalistic months all occur within a 24-hour period, it turns out that 223 synodic months, which is
+6585.32 days, is almost exactly the same as 242 draconic months – 6585.36 days, and 239 anomalistic months – 6585.54 days. It’s not a perfect alignment, but we have to wait a long time to find a better one. If we want the alignment to be within a period of a couple of hours, we’ll have to wait 3263 synodic months, which is about 263 years.
-
Note I don’t know if there’s a way to calculate these
-time periods analytically. I wrote a program to do it.
+
Note I don’t know if there’s a way to calculate these time periods analytically. I wrote a program to do it.
-
Note It’s worth bearing in mind that looking for closer
-alignments is probably unprofitable – the figures for the lengths of the
-various lunar months are averages, not exact. There’s probably no way to
-perform a more exact calculation.
+
Note It’s worth bearing in mind that looking for closer alignments is probably unprofitable – the figures for the lengths of the various lunar months are averages, not exact. There’s probably no way to perform a more exact calculation.
-
The period of 223 synodic months is called a saros. Although
-this term is derived from ancient sources, it turns out that its use in
-the context of eclipse prediction only dates back to 1686. It was used
-this way by Edmond Halley (of comet fame), but whether the ancients used
-it in the same way is contested. That the ancients could
-predict eclipses is not contested – just the terminology.
-
It turns out that, if there’s an eclipse today, we can be reasonably
-certain that there will be an eclipse with similar characteristics (same
-duration, same area of coverage) in 223 synodic months. Unfortunately,
-223 synodic months is not a whole number of Earth days. It’s about 18
-years, but it’s eight hours away from the nearest whole number of days.
-What that means is that, although the Moon will case the same kind of
-shadow on the Earth (for a solar eclipse), different parts of the Earth
-will be affected.
-
If we’re using as our initial, reference eclipse an event where the
-coverage area on the Earth was large (so the Moon was at its furthest
-from the Earth) then it’s plausible that a particular location will see
-both eclipses – today’s, and the one eighteen years later. After all, a
-total solar eclipse with the Sun at apogee can darken a whole continent.
-But, as an ancient astronomer, I’d do better to look at lunar
-eclipses. I will already have noticed that a solar eclipse that I
-do see was preceded by a lunar eclipse in the same synodic
-month; it would seem reasonable to conclude that most or all lunar
-eclipses are followed by a solar eclipse somewhere on Earth.
-And it turns out that lunar eclipses are much easier to see from Earth
-than are solar eclipses – and not just because different regions of the
-Earth are in shadow. It’s actually very difficult to see a modest
-partial eclipse of the sun, because it’s not safe to look directly at
-it. But there are no such problems with the Moon.
-
So it turns out that the saros interval of about 18 years gives us a
-really good way to predict eclipses. But we already know that there are
-far more eclipses than that. If there’s an eclipse today, there’s
-possibly going to be one in six synodic months’ time – that’s less than
-half a year, and a lot shorter than 18 years.
-
But here’s the crucial point: the saros cycle is quite repetitive. If
-I see an eclipse today, and another a synodic month later, I can be
-almost certain that I will see an eclipse a saros period later, and one
-synodic month after that. Moreover, the type of eclipse will be
-similar. So if I can collect 18 years’-worth of eclipse data, I have
-everything I need to predict eclipses with reasonable accuracy for
-hundreds of years to come.
-
The
-importance of observation on repetitive phenomena
-
We know that the ancient Greeks, and before them the Chaldeans, could
-predict eclipses. We also know that they didn’t have modern computers.
-What they did have was an understanding that natural phenomena repeated,
-and a lot of patience. They almost certainly came upon the saros cycle
-by making astronomical observations for many decades, or even centuries.
-Our ancient forebears were as interested in the skies as we are –
-perhaps even more. After all, the positions of the Sun and Moon had a
-direct impact on agriculture, weather, and tides.
-
How do we do all this with
-clockwork?
-
Most likely none of us has seen a mechanical timepiece with an
-‘eclipse’ dial. What we do sometimes still see, and is instructive to
-consider, is a ‘moon phase’ dial on a mechanical clock or watch. If we
-can display moon phases, a suitable reduction gearing will be able to
-drive a pointer with a saros cycle. So the problem is not different in
-kind, only in engineering complexity.
-
The slowest-moving gear in a conventional watch is the hour gear,
-which usually makes a full rotation every twelve hours. So to display
-the moon phase we need a gear that rotates 29.53 times, for every two
-rotations of the hour gear. Or, alternatively, 59.06 times for each
-rotation of the hour gear.
-
If we accept a small amount of inaccuracy, we can round this to 59
-revolutions of the hour gear for one revolution of the moon phase gear.
-Implementing a gear ratio of 59:1 is easy enough if you have a way to
-make a ‘finger’ gear – essentially a gear with only one tooth. The
-complete gear mechanism is a one-tooth finger gear meshing with a gear
-with 59 teeth. This is, in fact, the way that modern mechanical watches
-display moon phase.
-
This gearing will be accurate to about one part in a thousand. So the
-moon phase display would slip about one day every 2.7 years. A
-mechanical moon phase display would have to be wrong by at least a day
-or so, to be completely useless. So this clock would track moon phase
-reasonably accurately for
+
The period of 223 synodic months is called a saros. Although this term is derived from ancient sources, it turns out that its use in the context of eclipse prediction only dates back to 1686. It was used this way by Edmond Halley (of comet fame), but whether the ancients used it in the same way is contested. That the ancients could predict eclipses is not contested – just the terminology.
+
It turns out that, if there’s an eclipse today, we can be reasonably certain that there will be an eclipse with similar characteristics (same duration, same area of coverage) in 223 synodic months. Unfortunately, 223 synodic months is not a whole number of Earth days. It’s about 18 years, but it’s eight hours away from the nearest whole number of days. What that means is that, although the Moon will case the same kind of shadow on the Earth (for a solar eclipse), different parts of the Earth will be affected.
+
If we’re using as our initial, reference eclipse an event where the coverage area on the Earth was large (so the Moon was at its furthest from the Earth) then it’s plausible that a particular location will see both eclipses – today’s, and the one eighteen years later. After all, a total solar eclipse with the Sun at apogee can darken a whole continent. But, as an ancient astronomer, I’d do better to look at lunar eclipses. I will already have noticed that a solar eclipse that I do see was preceded by a lunar eclipse in the same synodic month; it would seem reasonable to conclude that most or all lunar eclipses are followed by a solar eclipse somewhere on Earth. And it turns out that lunar eclipses are much easier to see from Earth than are solar eclipses – and not just because different regions of the Earth are in shadow. It’s actually very difficult to see a modest partial eclipse of the sun, because it’s not safe to look directly at it. But there are no such problems with the Moon.
+
So it turns out that the saros interval of about 18 years gives us a really good way to predict eclipses. But we already know that there are far more eclipses than that. If there’s an eclipse today, there’s possibly going to be one in six synodic months’ time – that’s less than half a year, and a lot shorter than 18 years.
+
But here’s the crucial point: the saros cycle is quite repetitive. If I see an eclipse today, and another a synodic month later, I can be almost certain that I will see an eclipse a saros period later, and one synodic month after that. Moreover, the type of eclipse will be similar. So if I can collect 18 years’-worth of eclipse data, I have everything I need to predict eclipses with reasonable accuracy for hundreds of years to come.
+
The importance of observation on repetitive phenomena
+
We know that the ancient Greeks, and before them the Chaldeans, could predict eclipses. We also know that they didn’t have modern computers. What they did have was an understanding that natural phenomena repeated, and a lot of patience. They almost certainly came upon the saros cycle by making astronomical observations for many decades, or even centuries. Our ancient forebears were as interested in the skies as we are – perhaps even more. After all, the positions of the Sun and Moon had a direct impact on agriculture, weather, and tides.
+
How do we do all this with clockwork?
+
Most likely none of us has seen a mechanical timepiece with an ‘eclipse’ dial. What we do sometimes still see, and is instructive to consider, is a ‘moon phase’ dial on a mechanical clock or watch. If we can display moon phases, a suitable reduction gearing will be able to drive a pointer with a saros cycle. So the problem is not different in kind, only in engineering complexity.
+
The slowest-moving gear in a conventional watch is the hour gear, which usually makes a full rotation every twelve hours. So to display the moon phase we need a gear that rotates 29.53 times, for every two rotations of the hour gear. Or, alternatively, 59.06 times for each rotation of the hour gear.
+
If we accept a small amount of inaccuracy, we can round this to 59 revolutions of the hour gear for one revolution of the moon phase gear. Implementing a gear ratio of 59:1 is easy enough if you have a way to make a ‘finger’ gear – essentially a gear with only one tooth. The complete gear mechanism is a one-tooth finger gear meshing with a gear with 59 teeth. This is, in fact, the way that modern mechanical watches display moon phase.
+
This gearing will be accurate to about one part in a thousand. So the moon phase display would slip about one day every 2.7 years. A mechanical moon phase display would have to be wrong by at least a day or so, to be completely useless. So this clock would track moon phase reasonably accurately for
2-3 years, before the moon dial needed to be reset.
-
Could we get better precision with a mechanical gearbox? Certainly:
-we can get a gear ratio of 59.04 using three coupled gear ratios: 20:72,
-into 16:41, into 10:64. The more exact ratio 59.06, but I couldn’t come
-up with a way to implement that, using a modest number of gears with
-modest tooth counts. Still, my three-step gearing would be accurate to
-three parts in ten thousand, so should handle moon phases tolerably well
-for twenty years or so. We could achieve better accuracy, if needed,
-using more gears.
-
To implement a complete saros cycle display requires, in simplistic
-terms, dividing down the moon phase gear by 223:1 (because there are 223
-synodic months in the complete cycle) or, perhaps, dividing down the day
-gear by 6585.32.
-
It seems, however, that this isn’t the way the original Antikythera
-Mechanism worked. Although it probably did have a display of Moon phase,
-it appears that it was derived, not by a simple down-gearing of the
-number of days, but by a subtraction process based on the draconic month
-and the apparent motion of the Sun around the Earth. After all, it is
-this combination of the Earth-Sun and Earth-Moon motions that makes the
-synodic and draconic months.
-
Whether this method of calculation is more accurate, or just more
-expedient, I don’t know (but probably somebody does).
+
Could we get better precision with a mechanical gearbox? Certainly: we can get a gear ratio of 59.04 using three coupled gear ratios: 20:72, into 16:41, into 10:64. The more exact ratio 59.06, but I couldn’t come up with a way to implement that, using a modest number of gears with modest tooth counts. Still, my three-step gearing would be accurate to three parts in ten thousand, so should handle moon phases tolerably well for twenty years or so. We could achieve better accuracy, if needed, using more gears.
+
To implement a complete saros cycle display requires, in simplistic terms, dividing down the moon phase gear by 223:1 (because there are 223 synodic months in the complete cycle) or, perhaps, dividing down the day gear by 6585.32.
+
It seems, however, that this isn’t the way the original Antikythera Mechanism worked. Although it probably did have a display of Moon phase, it appears that it was derived, not by a simple down-gearing of the number of days, but by a subtraction process based on the draconic month and the apparent motion of the Sun around the Earth. After all, it is this combination of the Earth-Sun and Earth-Moon motions that makes the synodic and draconic months.
+
Whether this method of calculation is more accurate, or just more expedient, I don’t know (but probably somebody does).
Closing remarks
-
The complexity of the Antikythera Mechanism, and the
-single-mindedness that its builders must have needed to collect the
-astronomical data to implement it, has lead some people to propose that
-it was built by visitors from space. Or, alternatively, that it’s a
-modern fake, like the rash of crystal skulls that appeared in the
-1960.
-
For sure, there’s nothing really like the AM – nothing we have so far
-discovered from that time period, anyway. But there are scattered
-ancient writings that allude to complex machinery, and there’s no doubt
-that astronomy was an obsessive interest for many ancient cultures –
-just look at the astronomical alignments embodied in Stonehenge.
-
The reason I don’t feel the same sense of awe and wonder
-when I look back on, for example, the Z80 microprocessor is that I lived
-through most of the technological developments that produced it. Yes,
-I’m old enough to remember when integrated circuits were rejected by
-many engineers as the Devil’s work.
-
When our entire civilization has been submerged by the rising waters
-of the melting ice caps, will our distant descendants find a ZX81 while
-diving for pearls? And, if they do, will they respect and admire our
-ingenuity, or claim that such a thing must have been gifted to us by
-aliens?
+
The complexity of the Antikythera Mechanism, and the single-mindedness that its builders must have needed to collect the astronomical data to implement it, has lead some people to propose that it was built by visitors from space. Or, alternatively, that it’s a modern fake, like the rash of crystal skulls that appeared in the 1960.
+
For sure, there’s nothing really like the AM – nothing we have so far discovered from that time period, anyway. But there are scattered ancient writings that allude to complex machinery, and there’s no doubt that astronomy was an obsessive interest for many ancient cultures – just look at the astronomical alignments embodied in Stonehenge.
+
The reason I don’t feel the same sense of awe and wonder when I look back on, for example, the Z80 microprocessor is that I lived through most of the technological developments that produced it. Yes, I’m old enough to remember when integrated circuits were rejected by many engineers as the Devil’s work.
+
When our entire civilization has been submerged by the rising waters of the melting ice caps, will our distant descendants find a ZX81 while diving for pearls? And, if they do, will they respect and admire our ingenuity, or claim that such a thing must have been gifted to us by aliens?
diff --git a/feed.xml b/feed.xml
index f40d771..9071f9d 100644
--- a/feed.xml
+++ b/feed.xml
@@ -9,8 +9,17 @@
Kevin Boone's website
https://kevinboone.me
https://kevinboone.me/img/favicon.ico
-<<<<<<< HEAD
-Wed, June 12 2024
+Sat, June 22 2024
+
+Using Private Internet Access VPN with Gentoo Linux
+https://kevinboone.me/gentoo_pia.html
+https://kevinboone.me/gentoo_pia.html
+It's not that difficult, but there's no built-in support.
+
+
+Sat, June 22 2024
+
+
Gentoo Linux -- is this the solution to the Winux problem?
https://kevinboone.me/gentoo_winux.html
@@ -32,10 +41,6 @@
-=======
-Mon, May 20 2024
-
->>>>>>> 4d301249128091a279ac0cd104d999b24b777030
Do you need to wipe that hard drive? And can you?
https://kevinboone.me/wipe_disk.html
https://kevinboone.me/wipe_disk.html
diff --git a/general_computing-groupindex.html b/general_computing-groupindex.html
index aea8aec..a052fff 100644
--- a/general_computing-groupindex.html
+++ b/general_computing-groupindex.html
@@ -112,11 +112,7 @@
General computing
diff --git a/gentoo_pia.html b/gentoo_pia.html
new file mode 100644
index 0000000..d9b5fdc
--- /dev/null
+++ b/gentoo_pia.html
@@ -0,0 +1,111 @@
+
+
+
+
+ Kevin Boone: Using Private Internet Access VPN with Gentoo Linux
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Using Private Internet Access VPN with Gentoo Linux
+
+
This is a short note about how to connect to the PIA VPN from a Gentoo Linux system. It’s not that difficult, but there’s no built-in support, either from Gentoo or from PIA. In fact, the PIA proprietary software installer warns that the host isn’t compatible. Still, the proprietary installer does work, with a bit of fiddling.
+
Alternatively, you can just use OpenVPN.
+
Using PIA’s proprietary software
+
I should point out that I’m running a systemd-free Gentoo with OpenRC as the service manager – I’m not sure whether the PIA installer is more friendly to an installation with systemd. Probably it is, frankly – most things are these days.
+
PIA supplies an installer for Linux in the form of a self-extracting binary with a name like pia-linux-xxx.run. The installer won’t allow itself to be run as root: it will expect to be able to run sudo to get root permissions. It installs in /opt, and it seems generally well-behaved; that is, it doesn’t appear to spray files all over the disk as some binary installers do.
+
What it does do, however, is attempt to install start-up scripts in /etc/init.d. These won’t work, because they refer to the wrong installation locations. It’s not difficult to create an init script for OpenRC, however.
+
The PIA software has a client-server architecture. There’s a service daemon, pia-daemon which would normally get started at boot time. Then there are a couple of clients that talk to that daemon. The graphical client – the part you see if you hit the ‘PIA’ icon in your Linux desktop – is an application called pia-client.
+
I found that neither part would start immediately after installation. It’s worth running both the daemon and the client from a console, so you can see the error messages. The obvious one was:
+
Error while loading shared libraries: libgssapi_krb5.so.2...:
+
The library it’s complaining about wasn’t part of my Gentoo install, and I get the impression that it usually won’t be. It’s easy enough to install, though:
+
# emerge app-crypt/mit-krb5
+
With that done, I was able to start pia-daemon and pia-client in a console, and connect to the VPN using the graphical application.
+
Of course, I’d rather have the daemon start at boot time, although it does take a few seconds to start, and then just use the graphical client to manage my VPN connection. That means creating an init script for OpenRC, although first I had to remove the defective /etc/init.d/pia and /etc/init.d/pia-daemon.
+
My OpenRC init script is below – this just goes in /etc/init.d/pia. I’m not expert on OpenRC, so it’s possible I’ve got something wrong; but this configuration seems to work for me.
+
#!/sbin/openrc-run
+
+name="PIA daemon"
+description="PIA daemon"
+command=/opt/piavpn/bin/pia-daemon
+command_background=yes
+pidfile=/run/pia-daemon.pid
+
+depend()
+ {
+ need net
+ }
+
I’ve put need net as a dependency to ensure that the network is basically up before starting the daemon. Again, I’m not sure that’s the textbook approach, but it works for me.
+
To make the daemon start at boot:
+
# rc-update add pia default
+
The PIA installer created a .desktop file for the client, so it shows up as a menu item in the desktop menu. or you can just run pia-client from a command line.
+
Using OpenVPN
+
The use of OpenVPN is described in the Gentoo documentation, but I think there’s an easier way. Whichever way you do it, you’ll need to install OpenVPN:
+
emerge --ask openvpn
+
Doing this creates a service called openvpn that you can start and stop using rc-service. However, you’ll need at least one configuration file. By default, OpenVPN reads /etc/openvpn/openvpn.conf.
+
The Gentoo documentation explains how to create this file manually. However, the OpenVPN website can create it for you. If you log into your account, and hunt around a bit (it’s not obvious where to look), you’ll see a link to “OpenVPN configuration generator”. Currently, it’s right at the bottom of the ‘Downloads’ page in small print. From here you’ll be able to select the VPN endpoint you want to connect to, the OpenVPN version, and the client platform. You must also select the TLS parameters to use. So far as I know they will all work; they just have different security implications. There’s nothing on the website to help you choose between them – PIA presumes, I guess, that if you’re setting up OpenVPN yourself, you’ll know what will suit your needs best.
+
The configuration page will generate a .ovpn file, that you can use as openvpn.conf. You can have as many of these files as you want, if you need to connect to different endpoints. However, you’ll need to come up with some way to switch between them.
+
I found that I had problems with IPv6 routing with the auto-generated configuration. Since I have little need for IPv6 at present, I turned IPv6 support off in openvpn.conf, rather than spending a day troubleshooting. These are the additions I made:
This will prompt you for your PIA credentials. If you prefer, you can put your credentials in a file, and link them using auth-user-pass in openvpn.conf. You’ll need to be a bit careful with this approach, of course, for the obvious reasons.
+
Although you can have the OpenVPN service start at boot time, there’s perhaps no need to do so, unless you actually want the VPN active all the time. I find it easier just to start and stop from a command line.
+
There is a bit of a gotcha here: I find that the OpenVPN service always starts with a warning message:
+
OpenVPN: service started, but is inactive
+
I see that warning whether the connection is working or not. When everything is set up correctly, the connection does get made, despite the warning. In the early stages of set-up, it’s certainly worth checking that you really do have a VPN connection. If it’s working, ifconfig will show a tun0 interface, with an IP number. If it doesn’t connect, you’ll have to look in the system log to see why.
+
It’s worth bearing in mind that setting up OpenVPN for PIA is not in any way Gentoo-specific. Although OpenVPN is pretty complicated, there’s a huge amount of information available about it.
+
Which to use?
+
In my experience, the PIA proprietary software outperforms OpenVPN in almost all respects. Of course, if you have an ideological objection to proprietary software, you won’t want to use it, however good it is. Although PIA doesn’t support Gentoo, their proprietary software isn’t too difficult to install. The fact that the client lets you choose your VPN endpoint from a clickable list, without fiddling about with configuration files and certificates, is a bonus. If you’re running Linux without a graphical desktop, PIA provides a command-line client as well.
+
OpenVPN is not a particularly easy thing to set up, but it’s a transferable skill. It’s compatible with most VPN services so, provided you can get the connection parameters and certificates, you’ll be able to change VPN providers easily, and even use multiple providers on the same computer. In addition, OpenVPN doesn’t require anything to be started at boot time (although it’s possible). To be fair, the PIA proprietary software doesn’t have to be started at boot time, either, but it usually is.
diff --git a/img/argleton0.png b/img/argleton0.png
old mode 100755
new mode 100644
diff --git a/img/argleton1.png b/img/argleton1.png
old mode 100755
new mode 100644
diff --git a/img/argleton2.png b/img/argleton2.png
old mode 100755
new mode 100644
diff --git a/img/argleton3.png b/img/argleton3.png
old mode 100755
new mode 100644
diff --git a/img/argleton_thumbnail.png b/img/argleton_thumbnail.png
old mode 100755
new mode 100644
diff --git a/img/brachistochrone1.png b/img/brachistochrone1.png
old mode 100755
new mode 100644
diff --git a/img/brachistochrone2.png b/img/brachistochrone2.png
old mode 100755
new mode 100644
diff --git a/img/brachistochrone_b_15.png b/img/brachistochrone_b_15.png
old mode 100755
new mode 100644
diff --git a/img/brachistochrone_b_4.png b/img/brachistochrone_b_4.png
old mode 100755
new mode 100644
diff --git a/img/electricity-img1.png b/img/electricity-img1.png
old mode 100755
new mode 100644
diff --git a/img/electricity-img10.png b/img/electricity-img10.png
old mode 100755
new mode 100644
diff --git a/img/electricity-img11.png b/img/electricity-img11.png
old mode 100755
new mode 100644
diff --git a/img/electricity-img12.png b/img/electricity-img12.png
old mode 100755
new mode 100644
diff --git a/img/electricity-img13.png b/img/electricity-img13.png
old mode 100755
new mode 100644
diff --git a/img/electricity-img14.png b/img/electricity-img14.png
old mode 100755
new mode 100644
diff --git a/img/electricity-img15.png b/img/electricity-img15.png
old mode 100755
new mode 100644
diff --git a/img/electricity-img2.png b/img/electricity-img2.png
old mode 100755
new mode 100644
diff --git a/img/electricity-img3.png b/img/electricity-img3.png
old mode 100755
new mode 100644
diff --git a/img/electricity-img4.png b/img/electricity-img4.png
old mode 100755
new mode 100644
diff --git a/img/electricity-img5.png b/img/electricity-img5.png
old mode 100755
new mode 100644
diff --git a/img/electricity-img6.png b/img/electricity-img6.png
old mode 100755
new mode 100644
diff --git a/img/electricity-img7.png b/img/electricity-img7.png
old mode 100755
new mode 100644
diff --git a/img/electricity-img8.png b/img/electricity-img8.png
old mode 100755
new mode 100644
diff --git a/img/electricity-img9.png b/img/electricity-img9.png
old mode 100755
new mode 100644
diff --git a/img/email.png b/img/email.png
old mode 100755
new mode 100644
diff --git a/img/favicon.ico b/img/favicon.ico
old mode 100755
new mode 100644
diff --git a/img/frotz_bz.png b/img/frotz_bz.png
old mode 100755
new mode 100644
diff --git a/img/gateaux1.png b/img/gateaux1.png
old mode 100755
new mode 100644
diff --git a/img/gateaux2.png b/img/gateaux2.png
old mode 100755
new mode 100644
diff --git a/img/gateaux3.png b/img/gateaux3.png
old mode 100755
new mode 100644
diff --git a/img/gateaux4.png b/img/gateaux4.png
old mode 100755
new mode 100644
diff --git a/img/gateaux5.png b/img/gateaux5.png
old mode 100755
new mode 100644
diff --git a/img/gateaux6.png b/img/gateaux6.png
old mode 100755
new mode 100644
diff --git a/img/gateaux7.png b/img/gateaux7.png
old mode 100755
new mode 100644
diff --git a/img/gtk-logo.png b/img/gtk-logo.png
old mode 100755
new mode 100644
diff --git a/img/horse.png b/img/horse.png
old mode 100755
new mode 100644
diff --git a/img/lego_sample.avi b/img/lego_sample.avi
old mode 100755
new mode 100644
diff --git a/img/living_cofd.png b/img/living_cofd.png
old mode 100755
new mode 100644
diff --git a/img/living_cofd2.png b/img/living_cofd2.png
old mode 100755
new mode 100644
diff --git a/img/living_cofd3.png b/img/living_cofd3.png
old mode 100755
new mode 100644
diff --git a/img/living_cofd4.png b/img/living_cofd4.png
old mode 100755
new mode 100644
diff --git a/img/living_cofd5.png b/img/living_cofd5.png
old mode 100755
new mode 100644
diff --git a/img/movieclip.jpg b/img/movieclip.jpg
old mode 100755
new mode 100644
diff --git a/img/notes.gif b/img/notes.gif
old mode 100755
new mode 100644
diff --git a/img/notes.png b/img/notes.png
old mode 100755
new mode 100644
diff --git a/img/phone_media_player_1.png b/img/phone_media_player_1.png
old mode 100755
new mode 100644
diff --git a/img/pi.png b/img/pi.png
old mode 100755
new mode 100644
diff --git a/img/pia_logo.png b/img/pia_logo.png
new file mode 100644
index 0000000..d548859
Binary files /dev/null and b/img/pia_logo.png differ
diff --git a/img/plug.png b/img/plug.png
old mode 100755
new mode 100644
diff --git a/img/report.png b/img/report.png
old mode 100755
new mode 100644
diff --git a/img/steam101_bigwheel.jpg b/img/steam101_bigwheel.jpg
old mode 100755
new mode 100644
diff --git a/img/steam101_connrod.jpg b/img/steam101_connrod.jpg
old mode 100755
new mode 100644
diff --git a/img/steam101_dora.jpg b/img/steam101_dora.jpg
old mode 100755
new mode 100644
diff --git a/img/steam101_driver.gif b/img/steam101_driver.gif
old mode 100755
new mode 100644
diff --git a/img/steam101_elf.jpg b/img/steam101_elf.jpg
old mode 100755
new mode 100644
diff --git a/img/steam101_fireman.gif b/img/steam101_fireman.gif
old mode 100755
new mode 100644
diff --git a/img/steam101_flow.jpg b/img/steam101_flow.jpg
old mode 100755
new mode 100644
diff --git a/img/steam101_parts.jpg b/img/steam101_parts.jpg
old mode 100755
new mode 100644
diff --git a/img/steam101_steamchest.jpg b/img/steam101_steamchest.jpg
old mode 100755
new mode 100644
diff --git a/img/steam101_tank.jpg b/img/steam101_tank.jpg
old mode 100755
new mode 100644
diff --git a/img/steam101_valvegear.jpg b/img/steam101_valvegear.jpg
old mode 100755
new mode 100644
diff --git a/img/thomas_crapper.jpg b/img/thomas_crapper.jpg
old mode 100755
new mode 100644
diff --git a/img/tree.png b/img/tree.png
old mode 100755
new mode 100644
diff --git a/img/tux_disk.png b/img/tux_disk.png
old mode 100755
new mode 100644
diff --git a/img/zom.png b/img/zom.png
old mode 100755
new mode 100644
diff --git a/img/zom_fig_1.png b/img/zom_fig_1.png
old mode 100755
new mode 100644
diff --git a/img/zom_fig_2.png b/img/zom_fig_2.png
old mode 100755
new mode 100644
diff --git a/img/zom_fig_3.png b/img/zom_fig_3.png
old mode 100755
new mode 100644
diff --git a/img/zom_fig_4.png b/img/zom_fig_4.png
old mode 100755
new mode 100644
diff --git a/img/zom_fig_5.png b/img/zom_fig_5.png
old mode 100755
new mode 100644
diff --git a/img/zom_fig_6.png b/img/zom_fig_6.png
old mode 100755
new mode 100644
diff --git a/img/zom_fig_7.png b/img/zom_fig_7.png
old mode 100755
new mode 100644
diff --git a/java_container_awareness.html b/java_container_awareness.html
index 2f61731..1dfc56c 100644
--- a/java_container_awareness.html
+++ b/java_container_awareness.html
@@ -2,8 +2,7 @@
- Kevin Boone: Exploring Java
-17’s container-awareness features
+ Kevin Boone: Exploring Java 17’s container-awareness features
@@ -41,139 +40,60 @@
-
Exploring Java
-17’s container-awareness features
+
Exploring Java 17’s container-awareness features
-
This article is about container-awareness in the Java virtual
-machine, particularly in Java 17. Container-awareness is a method for
-the JVM to detect the resources it has available, when running in a
-container, and adjust itself accordingly. In the article I will discuss
-only memory resources. Everything here applies equally well to
-CPU resources; I’m using memory simply because it’s easier to monitor
-than CPU.
-
The ultimate goal of this article is to show how OpenShift resource
-limits and requests affect the JVM. However, to get to that point, we
-need to start by looking at simpler scenarios.
+
This article is about container-awareness in the Java virtual machine, particularly in Java 17. Container-awareness is a method for the JVM to detect the resources it has available, when running in a container, and adjust itself accordingly. In the article I will discuss only memory resources. Everything here applies equally well to CPU resources; I’m using memory simply because it’s easier to monitor than CPU.
+
The ultimate goal of this article is to show how OpenShift resource limits and requests affect the JVM. However, to get to that point, we need to start by looking at simpler scenarios.
The JVM heap and its sizing
-
The Java virtual machine uses a flexible heap with garbage collection
-for its object management. Broadly speaking, the JVM allocates
-newly-created objects in the heap, where they remain until they are
-elligible to be removed by the garbage collector. The heap can expand
-between its initial and maximum values, according to the number of
-objects in play at a particular time.
-
It’s always been possible for the application designer to set the
-initial and maximum java heap size using the -Xms and
--Xmx command-line switches. For example:
+
The Java virtual machine uses a flexible heap with garbage collection for its object management. Broadly speaking, the JVM allocates newly-created objects in the heap, where they remain until they are elligible to be removed by the garbage collector. The heap can expand between its initial and maximum values, according to the number of objects in play at a particular time.
+
It’s always been possible for the application designer to set the initial and maximum java heap size using the -Xms and -Xmx command-line switches. For example:
$ java -Xms512m -Xmx2048m ...
-
This is a reasonable way to control the heap size in a traditional
-operating system, that is, one that is not based on container
-technology. The operating system has a specific, well-known amount of
-memory available to it, and it’s unlikely to change much from day to
-day. Changing the JVM’s memory allocation is likely to amount to hacking
-on some configuration file or script.
-
Container technology, however, provides for flexible memory
-management. Administrators of container-based systems like Kubernetes or
-OpenShift expect to be able to tune an application’s memory usage by
-tuning the amount of memory assigned to its container. Moreover, tuning
-memory usage at the application might be complicated, if the
-configuration is buried somewhere inside a container image.
-
Later versions of Java have a container awareness feature,
-which allows the JVM to learn (or, at least, guess) the amount of memory
-present in the container. By ‘later’ I mean, perhaps confusingly, later
-releases of Java 1.8, Java 11, and Java 17. That is, the changes were
-back-ported from Java 17 to the later releases of earlier versions. If
-you see what I mean. The point I’m making is that container awareness is
-available on all Java platforms today, even as far back as Java 8.
+
This is a reasonable way to control the heap size in a traditional operating system, that is, one that is not based on container technology. The operating system has a specific, well-known amount of memory available to it, and it’s unlikely to change much from day to day. Changing the JVM’s memory allocation is likely to amount to hacking on some configuration file or script.
+
Container technology, however, provides for flexible memory management. Administrators of container-based systems like Kubernetes or OpenShift expect to be able to tune an application’s memory usage by tuning the amount of memory assigned to its container. Moreover, tuning memory usage at the application might be complicated, if the configuration is buried somewhere inside a container image.
+
Later versions of Java have a container awareness feature, which allows the JVM to learn (or, at least, guess) the amount of memory present in the container. By ‘later’ I mean, perhaps confusingly, later releases of Java 1.8, Java 11, and Java 17. That is, the changes were back-ported from Java 17 to the later releases of earlier versions. If you see what I mean. The point I’m making is that container awareness is available on all Java platforms today, even as far back as Java 8.
-
Note Despite all this, there are subtle differences
-between Java versions and, in this article I use Java 17
-exclusively.
+
Note Despite all this, there are subtle differences between Java versions and, in this article I use Java 17 exclusively.
-
As a result, we generally advise against using fixed heap settings
-(-Xmx and -Xms) when running Java in a container
-platform.
-
But what happens if the designer sets no heap limits at all? The JVM
-has to allocate some heap range, but how? And how will the
-JVM’s behaviour depend on the container’s memory settings?
-
In general, if no specific heap values are given, the Java JVM will
-apply defaults. Historically, Java had a rather complicated algorithm
-for working out which heap defaults to use; it depended on whether we
-were running in ‘server’ or ‘client’ mode, the amount of platform
-virtual memory, whether we were running a 32-bit or 64-bit platform, the
-garbage collection algorithm, and so on. The defaults changed from one
-Java version to another and, although they were documented, it wasn’t
-always easy to find the documentation. It made sense, in the
-pre-container days, to use at least the -Xmx switch for all
-non-trivial applications.
-
These days, for most reasonable platform configurations, the
-defaults are much simpler: the maximum heap is 1/4 of the platform’s
-reported memory size, and the minimum heap is 1/64, up to a limit of
-512Mb. There are complications related to set-ups with very large, and
-very small, memory sizes; but these won’t affect most installations.
-
Using a
-program to report memory and heap allocations
-
In this article, I’ll be using a simple Java program to report on the
-platform memory and allocated heap sizes. It will get this information
-from JMX MBeans. I won’t present the source code here – it’s very simple
-and, as always, it’s available from my GitHub
-repository.
-
As well as the Java source, the application bundle contains scripts
-and a Dockerfile, to build the code into a container. I’ll
-demonstrate its use in a Podman container (although Docker will work
-just the same) and also in an OpenShift pod.
+
As a result, we generally advise against using fixed heap settings (-Xmx and -Xms) when running Java in a container platform.
+
But what happens if the designer sets no heap limits at all? The JVM has to allocate some heap range, but how? And how will the JVM’s behaviour depend on the container’s memory settings?
+
In general, if no specific heap values are given, the Java JVM will apply defaults. Historically, Java had a rather complicated algorithm for working out which heap defaults to use; it depended on whether we were running in ‘server’ or ‘client’ mode, the amount of platform virtual memory, whether we were running a 32-bit or 64-bit platform, the garbage collection algorithm, and so on. The defaults changed from one Java version to another and, although they were documented, it wasn’t always easy to find the documentation. It made sense, in the pre-container days, to use at least the -Xmx switch for all non-trivial applications.
+
These days, for most reasonable platform configurations, the defaults are much simpler: the maximum heap is 1/4 of the platform’s reported memory size, and the minimum heap is 1/64, up to a limit of 512Mb. There are complications related to set-ups with very large, and very small, memory sizes; but these won’t affect most installations.
+
Using a program to report memory and heap allocations
+
In this article, I’ll be using a simple Java program to report on the platform memory and allocated heap sizes. It will get this information from JMX MBeans. I won’t present the source code here – it’s very simple and, as always, it’s available from my GitHub repository.
+
As well as the Java source, the application bundle contains scripts and a Dockerfile, to build the code into a container. I’ll demonstrate its use in a Podman container (although Docker will work just the same) and also in an OpenShift pod.
When I run the program on my workstation, here is the output:
The ‘native’ memory size is what JMX reports to the program as the
-system memory. My workstation has 32Gb of RAM, so that value looks about
-right. The maximum and initial heap values are 1/4 and 1/64 of that
-figure, more or less. So that’s as expected.
-
Just to prove that the program works, I can run it with specific heap
-settings:
+
The ‘native’ memory size is what JMX reports to the program as the system memory. My workstation has 32Gb of RAM, so that value looks about right. The maximum and initial heap values are 1/4 and 1/64 of that figure, more or less. So that’s as expected.
+
Just to prove that the program works, I can run it with specific heap settings:
You’ll see that the heap values now reflect the -Xmx and
--Xms settings, so I’m reasonably sure that the simple
-program works properly.
-
Heap mamangement in a
-container
-
Let’s try this application in a container. I’m using Podman, but
-Docker will behave the same.
-
In the application bundle is a Dockerfile for building
-the container. It looks like this:
+
You’ll see that the heap values now reflect the -Xmx and -Xms settings, so I’m reasonably sure that the simple program works properly.
+
Heap mamangement in a container
+
Let’s try this application in a container. I’m using Podman, but Docker will behave the same.
+
In the application bundle is a Dockerfile for building the container. It looks like this:
FROM openjdk:17-alpine
ADD target/java_docker_test-0.0.1-jar-with-dependencies.jar .
ADD ./run.sh .
ENTRYPOINT ["/run.sh"]
-
The FROM line specifies that my image will be based on
-the Alpine Java 17 base image. Alpine is a lightweight Linux version,
-popular in containers.
-
To the base images I add the Java application’s JAR file, and a
-script run.sh. This script just runs the Java application,
-exactly as I did on my workstation. Here’s what it looks like:
+
The FROM line specifies that my image will be based on the Alpine Java 17 base image. Alpine is a lightweight Linux version, popular in containers.
+
To the base images I add the Java application’s JAR file, and a script run.sh. This script just runs the Java application, exactly as I did on my workstation. Here’s what it looks like:
You see that the results are exactly the same as running outside the
-container. There’s no reason they should be different:
-podman will not constrain the container’s memory, unless we
-ask it to.
+
You see that the results are exactly the same as running outside the container. There’s no reason they should be different: podman will not constrain the container’s memory, unless we ask it to.
So let’s do that – let’s fix the values of RAM and swap.
Heap mamangement in a
Init heap size (Xms): 16.0 MB
Note
-Be aware that ‘–memory-swap’ really means ‘RAM plus swap’. This
-configuration allocates 1Gb RAM and 2Gb swap.
+Be aware that ‘–memory-swap’ really means ‘RAM plus swap’. This configuration allocates 1Gb RAM and 2Gb swap.
-
The program reports its memory size as 1000Mb (which matches the
---memory argument) and, again, the heap sizes are 1/4 and
-1/64, as always.
-
How does the JVM know how much memory its container has? We can
-experiment with this by logging into the container, by setting a shell
-as the --entrypoint:
+
The program reports its memory size as 1000Mb (which matches the --memory argument) and, again, the heap sizes are 1/4 and 1/64, as always.
+
How does the JVM know how much memory its container has? We can experiment with this by logging into the container, by setting a shell as the --entrypoint:
Let’s look at the system memory within the container:
/ # free
total used free shared buff/cache available
Mem: 32224352 4529920 2732716 5085020 24961716 22134684
Swap: 8388604 170248 8218356
-
These Mem and Swap figures match my workstation totals, not the
-container’s. The container only has 1Gb available, and somehow the JVM
-has worked this out.
-
Linux containers using a technology called ‘cgroups’ (control groups)
-for managing resources at the per-process level. Somewhat irritatingly
-for our purposes, there are two different versions of cgroups in
-circulation: v1 and v2. Most modern Linux kernels support both, but
-characteristics of the underlying platform dictate which to use.
-podman running on my workstation uses cgroups v2 but, as
-we’ll see, containers on my OpenShift installation uses v1.
-
Both versions do much the same thing; the irritation is that the
-filenames we’ll use to report cgroups metrics are different.
-Unfortunately, if you do a lot of work with cgroups, you really have to
-be familiar with both versions, and we’ll see both in use in this
-article.
-
There is an article on
-cgroups (v1) elsewhere on my website, which might be of interest to
-readers who want to see it in action.
-
From within the running container, we can get the memory and swap
-limits from cgroups like this:
+
These Mem and Swap figures match my workstation totals, not the container’s. The container only has 1Gb available, and somehow the JVM has worked this out.
+
Linux containers using a technology called ‘cgroups’ (control groups) for managing resources at the per-process level. Somewhat irritatingly for our purposes, there are two different versions of cgroups in circulation: v1 and v2. Most modern Linux kernels support both, but characteristics of the underlying platform dictate which to use. podman running on my workstation uses cgroups v2 but, as we’ll see, containers on my OpenShift installation uses v1.
+
Both versions do much the same thing; the irritation is that the filenames we’ll use to report cgroups metrics are different. Unfortunately, if you do a lot of work with cgroups, you really have to be familiar with both versions, and we’ll see both in use in this article.
+
There is an article on cgroups (v1) elsewhere on my website, which might be of interest to readers who want to see it in action.
+
From within the running container, we can get the memory and swap limits from cgroups like this:
You’ll note that these agree with the limits I applied to the
-podman command.
-
It’s worth noting at this point that, so far as the JVM is concerned,
-the swap value is irrelevant. By default, when the JVM queries
-its memory allocation to calculate heap sizes, it uses ‘RAM’, not swap.
-Of course, it’s not real RAM, it’s a container simulation of RAM.
-
So that’s how the heap allocation works in a Podman container: the
-JVM uses cgroups to determine the amount of allocated RAM, and sets the
-heap limits to 1/4 and 1/64 of that figure.
-
It’s worth bearing in mind that cgroups enforces limits on
-the JVM. The JVM only reads the cgroups metrics to set the heap sizes
-appropriately. You could set -Xmx2000m in a container with
-only 1Gb allocated, and the JVM would try to create a heap that large –
-but it wouldn’t be allowed.
-
Heap
-management with OpenShift resource requests and limits
+
You’ll note that these agree with the limits I applied to the podman command.
+
It’s worth noting at this point that, so far as the JVM is concerned, the swap value is irrelevant. By default, when the JVM queries its memory allocation to calculate heap sizes, it uses ‘RAM’, not swap. Of course, it’s not real RAM, it’s a container simulation of RAM.
+
So that’s how the heap allocation works in a Podman container: the JVM uses cgroups to determine the amount of allocated RAM, and sets the heap limits to 1/4 and 1/64 of that figure.
+
It’s worth bearing in mind that cgroups enforces limits on the JVM. The JVM only reads the cgroups metrics to set the heap sizes appropriately. You could set -Xmx2000m in a container with only 1Gb allocated, and the JVM would try to create a heap that large – but it wouldn’t be allowed.
+
Heap management with OpenShift resource requests and limits
Now let’s try the same thing on OpenShift.
-
There are many ways to deploy a Java application in an OpenShift pod,
-but I want to make the test conditions as close as possible to the
-Podman/Docker case. To that end, I’ll do a Docker deployment on
-OpenShift, using the same Dockerfile and Java code as we
-used with Podman. Here’s how to do that.
+
There are many ways to deploy a Java application in an OpenShift pod, but I want to make the test conditions as close as possible to the Podman/Docker case. To that end, I’ll do a Docker deployment on OpenShift, using the same Dockerfile and Java code as we used with Podman. Here’s how to do that.
This will take a little while, particularly the first time. All being
-well, you’ll see the same Docker build steps in the output as we saw
-when building for a local Podman installation.
+
This will take a little while, particularly the first time. All being well, you’ll see the same Docker build steps in the output as we saw when building for a local Podman installation.
Note
-This step, oc start-build, is what you’ll need to repeat,
-if you change the Java program or scripts.
+This step, oc start-build, is what you’ll need to repeat, if you change the Java program or scripts.
-
All being well, you’ll see that a build pod was created, and it
-completed. It should have created a new OpenShift container image called
-java-docker-test.
-
To get this image into a running pod, we can create a deployment from
-the image.
+
All being well, you’ll see that a build pod was created, and it completed. It should have created a new OpenShift container image called java-docker-test.
+
To get this image into a running pod, we can create a deployment from the image.
$ oc new-app java-docker-test
-
Use oc get pods to see what new pods have been created;
-you should see a pod with a name of the form
-java-docker-test-XXXXXXXXXX-XXXXX.
-
To see the application’s output, have a look at the logs from this
-pod:
+
Use oc get pods to see what new pods have been created; you should see a pod with a name of the form java-docker-test-XXXXXXXXXX-XXXXX.
+
To see the application’s output, have a look at the logs from this pod:
The native memory size is reported as 6.6Gb. The heap sizes are, as
-always, 1/4 and 1/64 of this. But why 6.6Gb? Almost certainly the
-OpenShift node I’m running this pod on has much more memory than this. I
-didn’t apply any limits to the pod (yet). So why this figure?
-
It’s not just a platform default – this is a calculated value. But
-I’ll come back to how the calculation is done later, as it’s a bit of a
-distraction.
-
Let’s apply a resource request and limit to the deployment. The pod
-will restart, and we can look at the logs again.
+
The native memory size is reported as 6.6Gb. The heap sizes are, as always, 1/4 and 1/64 of this. But why 6.6Gb? Almost certainly the OpenShift node I’m running this pod on has much more memory than this. I didn’t apply any limits to the pod (yet). So why this figure?
+
It’s not just a platform default – this is a calculated value. But I’ll come back to how the calculation is done later, as it’s a bit of a distraction.
+
Let’s apply a resource request and limit to the deployment. The pod will restart, and we can look at the logs again.
Because we’re using cgroups v1 here, the name of the file containing
-the limit is not memory.max, as it was earlier – it’s
-memory.limit_in_bytes. The value is 1Gb, which is as we set
-on the command line.
-
The mystery I deferred is how the JVM arrived at a memory size of
-6.6Gb when we didn’t set any limit. According the OpenShift
-documentation, a pod with no resource limit has unbounded resources.
-That is, the JVM should be able to allocate memory until the OpenShift
-node runs out of RAM. But, clearly, the JVM has arrived at some
-heap sizing. But how?
-
Log into the pod again, and look at the cgroups limit in this
-scenario:
+
Because we’re using cgroups v1 here, the name of the file containing the limit is not memory.max, as it was earlier – it’s memory.limit_in_bytes. The value is 1Gb, which is as we set on the command line.
+
The mystery I deferred is how the JVM arrived at a memory size of 6.6Gb when we didn’t set any limit. According the OpenShift documentation, a pod with no resource limit has unbounded resources. That is, the JVM should be able to allocate memory until the OpenShift node runs out of RAM. But, clearly, the JVM has arrived at some heap sizing. But how?
+
Log into the pod again, and look at the cgroups limit in this scenario:
That’s a truly colossal number – scarcely a limit at all. It’s as
-close to ‘unbounded’ as makes no practical difference. To see where the
-JVM’s memory limit comes from, let’s ask it. Log into the running pod
-and run:
+
That’s a truly colossal number – scarcely a limit at all. It’s as close to ‘unbounded’ as makes no practical difference. To see where the JVM’s memory limit comes from, let’s ask it. Log into the running pod and run:
Heap mamangement in a
[0.031s][trace][os,container] Path to /memory.stat is /sys/fs/cgroup/memory/memory.stat
[0.031s][trace][os,container] Hierarchical Memory Limit is: 7125319680
...
-
You can see that the JVM has determined that its memory allocation is
-unlimited. So it’s parsed the file memory.stat, and found
-the value of the “hierarchical memory limit”.
-
The what now? This isn’t a concept that I really have space to
-explain in detail here. In essence, OpenShift containers run as
-processes in a hierarchy of resource constraints. The ‘hierarchical
-limit’ is obtained by taking the limit on the container’s parent
-process, and subtracting the memory used by other child processes (other
-pods) with the same parent.
+
You can see that the JVM has determined that its memory allocation is unlimited. So it’s parsed the file memory.stat, and found the value of the “hierarchical memory limit”.
+
The what now? This isn’t a concept that I really have space to explain in detail here. In essence, OpenShift containers run as processes in a hierarchy of resource constraints. The ‘hierarchical limit’ is obtained by taking the limit on the container’s parent process, and subtracting the memory used by other child processes (other pods) with the same parent.
Doing this math gives us about 71 trillion bytes, or 6.6Gb.
-
If you think this is an odd way to work out a memory limit, I’d be
-inclined to agree. But what else can the JVM do? Java has no notion of
-an unconstrained heap: it has to have some maximum. The JVM could just
-use the platform RAM as its base figure, as it does outside a container.
-But it isn’t safe for a JVM to assume that it has that much RAM
-available – it could be enormous, and there are too many other
-containers competing for it. I guess the ‘hierarchical limit’ is the
-best compromise that that JVM maintainers could think of.
-
In practice, I think we should assume that a JVM running in a pod
-with no limit will get what amounts to a random limit. In
-almost all circumstances it will make sense to apply a limit.
-
Resources limits and
-requests
-
In the previous command I set a resource limit and a
-resource request, both to the same value. My experience is that
-Java developers working on OpenShift often don’t understand the
-difference between these values.
-
I’ve noticed a vague expectation that the ‘limit’ figure should
-constrain the maximum heap, and the ‘request’ figure the minimum. It
-seems plausible that this should be the case but, as we’ve seen, that
-isn’t the case. In the previous example I set both request and limit to
-1Gb, but the minimum heap value remained at 16Mb – the usual 1/64 of the
-total memory.
-
In fact, the JVM only sees the ‘limit’ figure. If there is, in fact,
-a way for a JVM to find the ‘request’ value from the platform, I don’t
-know of one. The request figure has no effect on the JVM heap size,
-maximum or minimum.
-
However, it does have implications. The request figure is used by the
-Kubernetes scheduler to decide which node to run the pod on. If
-we set a low memory request, then the scheduler will allow the pod to
-run on a node with low memory availability. The JVM will still get the
-heap allocation determined by the limit figure, however large that is.
-But if the pod is running on a node with low memory availability, it’s
-possible that the pod will fail at runtime, because there just isn’t
-enough memory to satisfy the limit.
-
Setting a low resource request is a ‘neighbourly’ thing for an
-application installer to do. It allows for fair distribution of pods
-between nodes. However, it’s probably not in the application’s best
-interests to have ‘limit’ and ‘request’ values that are hugely
-different. If we set a limit of, say, 1Gb, we’re doing so in the
-expectation that the pod might, sooner or later, use 1Gb. If the request
-value is, say, 128Mb we’re saying that the pod will be satisfied with
-128Mb; if you’ve set a limit of 1Gb, that’s likely not to be the
-case.
-
It takes care and experience to determine good resource requests and
-limits for an application, and probably some testing. I usually
-recommend that installers set the same values for limit and request if
-they can. In practice, though, we often can’t, because if every
-application component does that, resources could be badly
-over-committed.
+
If you think this is an odd way to work out a memory limit, I’d be inclined to agree. But what else can the JVM do? Java has no notion of an unconstrained heap: it has to have some maximum. The JVM could just use the platform RAM as its base figure, as it does outside a container. But it isn’t safe for a JVM to assume that it has that much RAM available – it could be enormous, and there are too many other containers competing for it. I guess the ‘hierarchical limit’ is the best compromise that that JVM maintainers could think of.
+
In practice, I think we should assume that a JVM running in a pod with no limit will get what amounts to a random limit. In almost all circumstances it will make sense to apply a limit.
+
Resources limits and requests
+
In the previous command I set a resource limit and a resource request, both to the same value. My experience is that Java developers working on OpenShift often don’t understand the difference between these values.
+
I’ve noticed a vague expectation that the ‘limit’ figure should constrain the maximum heap, and the ‘request’ figure the minimum. It seems plausible that this should be the case but, as we’ve seen, that isn’t the case. In the previous example I set both request and limit to 1Gb, but the minimum heap value remained at 16Mb – the usual 1/64 of the total memory.
+
In fact, the JVM only sees the ‘limit’ figure. If there is, in fact, a way for a JVM to find the ‘request’ value from the platform, I don’t know of one. The request figure has no effect on the JVM heap size, maximum or minimum.
+
However, it does have implications. The request figure is used by the Kubernetes scheduler to decide which node to run the pod on. If we set a low memory request, then the scheduler will allow the pod to run on a node with low memory availability. The JVM will still get the heap allocation determined by the limit figure, however large that is. But if the pod is running on a node with low memory availability, it’s possible that the pod will fail at runtime, because there just isn’t enough memory to satisfy the limit.
+
Setting a low resource request is a ‘neighbourly’ thing for an application installer to do. It allows for fair distribution of pods between nodes. However, it’s probably not in the application’s best interests to have ‘limit’ and ‘request’ values that are hugely different. If we set a limit of, say, 1Gb, we’re doing so in the expectation that the pod might, sooner or later, use 1Gb. If the request value is, say, 128Mb we’re saying that the pod will be satisfied with 128Mb; if you’ve set a limit of 1Gb, that’s likely not to be the case.
+
It takes care and experience to determine good resource requests and limits for an application, and probably some testing. I usually recommend that installers set the same values for limit and request if they can. In practice, though, we often can’t, because if every application component does that, resources could be badly over-committed.
Percentage heap allocations
-
My experience is that the larger the memory allocated to a pod, the
-less sense it makes to allow the JVM to assign only a quarter of it for
-its heap. To be sure, most Java applications will use memory outside the
-heap. A common consumer of out-of-heap memory is thread stacks, and that
-can be very significant on heavily-loaded, concurrent applications. Many
-Java libraries allocate memory outside the heap – Netty is a good
-example of this. The Java classes that expand zipfiles also use non-heap
-memory, although this might not be obvious.
-
All things considered, though, if you’re allocating 32Gb of RAM to a
-Java pod, you’re probably expecting at least, say, 28Gb of it to be
-available to the JVM heap. Of course, we can give effect to that
-expectation by using the command-line argument -Xmx28000m,
-but that’s something we discourage, as I described earlier.
-
We would like the administrator to be able to tune the amount of
-memory available to a Java application by tuning the pod’s allocation,
-rather than by rebuilding the image with new code (even if it’s only a
-new start script).
-
In scenarios like this, it can make sense to allocate a specific
-fraction of the memory to the heap, rather than an amount. For
-example, if I want the heap to use between 20% and 70% of the total
-memory, I can run Java like this:
+
My experience is that the larger the memory allocated to a pod, the less sense it makes to allow the JVM to assign only a quarter of it for its heap. To be sure, most Java applications will use memory outside the heap. A common consumer of out-of-heap memory is thread stacks, and that can be very significant on heavily-loaded, concurrent applications. Many Java libraries allocate memory outside the heap – Netty is a good example of this. The Java classes that expand zipfiles also use non-heap memory, although this might not be obvious.
+
All things considered, though, if you’re allocating 32Gb of RAM to a Java pod, you’re probably expecting at least, say, 28Gb of it to be available to the JVM heap. Of course, we can give effect to that expectation by using the command-line argument -Xmx28000m, but that’s something we discourage, as I described earlier.
+
We would like the administrator to be able to tune the amount of memory available to a Java application by tuning the pod’s allocation, rather than by rebuilding the image with new code (even if it’s only a new start script).
+
In scenarios like this, it can make sense to allocate a specific fraction of the memory to the heap, rather than an amount. For example, if I want the heap to use between 20% and 70% of the total memory, I can run Java like this:
Recent Java versions have a bunch of additional command-line
-arguments for more subtle control of the heap in a container
-environment.
+
Recent Java versions have a bunch of additional command-line arguments for more subtle control of the heap in a container environment.
Conclusions
So what have we learned from all this?
-
In the container environment, the JVM uses cgroups to work out how
-much memory it has available and, if the installer does not specify any
-heaps sizes, it sets the maximum and initial values to 1/4 and 1/64 of
-the available memory limit
-
The container’s memory limit will nearly always be less than the
-platform’s RAM
-
The JVM only uses the container’s RAM size in its calculations, not
-swap However, both RAM and swap are a simulation – platform swap can be
-used to provide the container’s RAM if circumstances dictate
-
On OpenShift, the cgroups memory limit is derived from the resource
-limit values in the deployment, not the resource
-request value
-
Request values are used to determine which node to run the
-pod on
-
In practice, it’s rarely sensible to run a Java application on
-OpenShift without specifying at least the resource limit, if
-not the request
-
In a container environment, it often makes sense to specify heap
-sizes as a fraction of the available memory, rather than as specific
-values.
+
In the container environment, the JVM uses cgroups to work out how much memory it has available and, if the installer does not specify any heaps sizes, it sets the maximum and initial values to 1/4 and 1/64 of the available memory limit
+
The container’s memory limit will nearly always be less than the platform’s RAM
+
The JVM only uses the container’s RAM size in its calculations, not swap However, both RAM and swap are a simulation – platform swap can be used to provide the container’s RAM if circumstances dictate
+
On OpenShift, the cgroups memory limit is derived from the resource limit values in the deployment, not the resource request value
+
Request values are used to determine which node to run the pod on
+
In practice, it’s rarely sensible to run a Java application on OpenShift without specifying at least the resource limit, if not the request
+
In a container environment, it often makes sense to specify heap sizes as a fraction of the available memory, rather than as specific values.
-
This has been a long article, to explain something which is actually
-quite simple: when deploying a Java application on OpenShift, you should
-nearly always set resource request and limit values, and you should
-consider sizing the heap using fractions of the available memory.
+
This has been a long article, to explain something which is actually quite simple: when deploying a Java application on OpenShift, you should nearly always set resource request and limit values, and you should consider sizing the heap using fractions of the available memory.
diff --git a/kafka_streams_hello.html b/kafka_streams_hello.html
index ead766c..7543f71 100644
--- a/kafka_streams_hello.html
+++ b/kafka_streams_hello.html
@@ -2,8 +2,7 @@
- Kevin Boone: Getting started with Kafka
-Streams
+ Kevin Boone: Getting started with Kafka Streams
@@ -41,59 +40,27 @@
-
Getting started with Kafka
-Streams
+
Getting started with Kafka Streams
-
Kafka Streams is a Java library and framework for creating
-applications that consume, process, and return Apache Kafka messages.
-It’s conceptually similar to Apache Camel, but tightly coupled to Kafka,
-and optimized for the kinds of operations that Kakfa clients typically
-do. Kafka Streams is therefore less flexible than Camel, but probably
-makes it easier to do Kafka-related things. Camel has a Kafka client of
-its own, so an alternative to Kafka Streams would be to write a Camel
-application with Kafka producers and consumers.
+
Kafka Streams is a Java library and framework for creating applications that consume, process, and return Apache Kafka messages. It’s conceptually similar to Apache Camel, but tightly coupled to Kafka, and optimized for the kinds of operations that Kakfa clients typically do. Kafka Streams is therefore less flexible than Camel, but probably makes it easier to do Kafka-related things. Camel has a Kafka client of its own, so an alternative to Kafka Streams would be to write a Camel application with Kafka producers and consumers.
Note
-There’s no need to understand Camel to follow this article. However,
-because Camel and Kafka Streams have broadly similar applications, I’ll
-point out similarities and differences from time to time.
+There’s no need to understand Camel to follow this article. However, because Camel and Kafka Streams have broadly similar applications, I’ll point out similarities and differences from time to time.
-
In this article I describe step-by-step how to deploy and run a
-trivial Kafka Streams application, from the ground up. I’ll use nothing
-but command-line tools and a text editor so, I hope, it’s clear what’s
-going on.
-
The article is a somewhat expanded version of the first step of the
-official Kafka Streams tutorial, and begins with the same
-Pipe.java applcation. It then expands this application to
-do some simple data processing. All the Pipe application does is copy
-from one Kafka topic to another. This is shown conceptually in the
-diagram below.
+
In this article I describe step-by-step how to deploy and run a trivial Kafka Streams application, from the ground up. I’ll use nothing but command-line tools and a text editor so, I hope, it’s clear what’s going on.
+
The article is a somewhat expanded version of the first step of the official Kafka Streams tutorial, and begins with the same Pipe.java applcation. It then expands this application to do some simple data processing. All the Pipe application does is copy from one Kafka topic to another. This is shown conceptually in the diagram below.
-
-A conceptual diagram of the ‘Hello World’
-of Kafka Streams applications
+A conceptual diagram of the ‘Hello World’ of Kafka Streams applications
I’ll explain what a ‘topology’ is shortly.
-
Anatomy of a
-simple Kafka Streams application
+
Anatomy of a simple Kafka Streams application
Note
-With a few exceptions, which I hope are obvious, when I don’t give
-package names for Java classes, they are in the package
-org.apache.kafka.streams.
+With a few exceptions, which I hope are obvious, when I don’t give package names for Java classes, they are in the package org.apache.kafka.streams.
-
Running a basic Kafka Streams applications amounts to instantiating
-KafkaStreams object, and calling start() on it. The
-start method will not exit until something in the
-application causes it to stop.
-
The sequence of processing steps that the application performs is
-called, in Kafka Streams jargon, a ‘topology’. A Streams topology is
-broadly similar to a Camel ‘context’ with a single ‘route’.
-
So if we have defined a toplogy, coding a basic Streams application
-looks like this:
+
Running a basic Kafka Streams applications amounts to instantiating KafkaStreams object, and calling start() on it. The start method will not exit until something in the application causes it to stop.
+
The sequence of processing steps that the application performs is called, in Kafka Streams jargon, a ‘topology’. A Streams topology is broadly similar to a Camel ‘context’ with a single ‘route’.
+
So if we have defined a toplogy, coding a basic Streams application looks like this:
Anatomy of a
KafkaStreams streams = new KafkaStreams (topology, props);
streams.start();
-
There are many properties that might need to be set, but in almost
-all cases you’ll need to set the application ID and the bootstrap server
-– this, of course, identifies the Kafka broker(s) the application will
-make first contact with.
-
The interesting part of programming Kafka Streams is defining the
-toplogy. More on that later, once we’ve set everything up.
+
There are many properties that might need to be set, but in almost all cases you’ll need to set the application ID and the bootstrap server – this, of course, identifies the Kafka broker(s) the application will make first contact with.
+
The interesting part of programming Kafka Streams is defining the toplogy. More on that later, once we’ve set everything up.
Prerequisites
-
In this article I assume that the reader has a working Kafka
-installation. It doesn’t matter whether it’s a simple, local
-installation with one broker, or a complex deployment with multiple
-brokers in the cloud. However, I’m assuming that you have a way to run
-the usual command-line utilities like
-kafka-console-producer.sh. If you’re running a Linux
-installation, you probably just need to
+
In this article I assume that the reader has a working Kafka installation. It doesn’t matter whether it’s a simple, local installation with one broker, or a complex deployment with multiple brokers in the cloud. However, I’m assuming that you have a way to run the usual command-line utilities like kafka-console-producer.sh. If you’re running a Linux installation, you probably just need to
cd /opt/kafka
-
or wherever the installation is. In a Kubernetes environment you’ll
-probably have to log into a running Kafka broker pod to get to the
-necessary binaries, or install Kafka locally as well.
-
In any case, in the commands in this article, I assume that Kafka is
-running on my development workstation and, where I need a Kafka
-bootstrap address, it will always be localhost:9092.
-
To follow this example (and any of the official Kafka tutorials)
-you’ll need Maven, or an IDE tool that can understand Maven
-pom.xml files. Of course you’ll need a Java compiler –
-anything from Java 1.8 onwards should work fine.
-
Installing the sample
-application
-
Kafka provides a Maven archetype for constructing a trivial Kafka
-Streams application. Use the archetype like this:
+
or wherever the installation is. In a Kubernetes environment you’ll probably have to log into a running Kafka broker pod to get to the necessary binaries, or install Kafka locally as well.
+
In any case, in the commands in this article, I assume that Kafka is running on my development workstation and, where I need a Kafka bootstrap address, it will always be localhost:9092.
+
To follow this example (and any of the official Kafka tutorials) you’ll need Maven, or an IDE tool that can understand Maven pom.xml files. Of course you’ll need a Java compiler – anything from Java 1.8 onwards should work fine.
+
Installing the sample application
+
Kafka provides a Maven archetype for constructing a trivial Kafka Streams application. Use the archetype like this:
Installing the sample
-DartifactId=kstreams_test \
-Dversion=0.1 \
-Dpackage=me.kevinboone.apacheintegration.kstreams_test
-
I’ve used my own names here; of course you can use any names you
-like. In this example, the application will be installed into the
-directory kstreams_test.
-
The generated code will include a Maven pom.xml, some
-logging configuration, and a few Java sources. We will be working with
-Pipe.java which, if you ran the Maven archetype as I did,
-will be in
+
I’ve used my own names here; of course you can use any names you like. In this example, the application will be installed into the directory kstreams_test.
+
The generated code will include a Maven pom.xml, some logging configuration, and a few Java sources. We will be working with Pipe.java which, if you ran the Maven archetype as I did, will be in
Of course, this package has many sub-dependencies, including the
-ordinary Kafka Java client.
+
Of course, this package has many sub-dependencies, including the ordinary Kafka Java client.
Note
-The version of the Kafka Streams library need not be the same as that of
-the Kafka installation itself. Of course, there are advantages to
-assuming that you use the latest versions of both.
+The version of the Kafka Streams library need not be the same as that of the Kafka installation itself. Of course, there are advantages to assuming that you use the latest versions of both.
Setting up Kafka
-
The documentation
-is clear that Kafka topics used by a streams application must be created
-administratively. The explanation given is that topics might be shared
-between multiple applications, which might make different assumptions
-about their structure. This is true of non-Streams applications as well
-but, whether the explanation is convincing, a Kafka Streams application
-will not auto-create topics even if Kafka is set up to allow it.
-
So we’ll need to create the topics that the sample application
-uses:
+
The documentation is clear that Kafka topics used by a streams application must be created administratively. The explanation given is that topics might be shared between multiple applications, which might make different assumptions about their structure. This is true of non-Streams applications as well but, whether the explanation is convincing, a Kafka Streams application will not auto-create topics even if Kafka is set up to allow it.
+
So we’ll need to create the topics that the sample application uses:
This is an example of what has become known as the “fluent builder”
-pattern – each method call on the ‘builder’ object returns another
-builder object which can have other methods called on it. So the
-specification of the topology amounts to a chain of method calls. This
-will become clearer, if it is not already, with more complex
-examples.
+
This is an example of what has become known as the “fluent builder” pattern – each method call on the ‘builder’ object returns another builder object which can have other methods called on it. So the specification of the topology amounts to a chain of method calls. This will become clearer, if it is not already, with more complex examples.
-
Note Apache Camel also provides a ‘fluent builder’
-method for defining Camel routes. Unlike Camel, however, a Streams
-topology can’t have multiple to(...) elements. Streams can
-send messages to multiple targets, but not like that.
+
Note Apache Camel also provides a ‘fluent builder’ method for defining Camel routes. Unlike Camel, however, a Streams topology can’t have multiple to(...) elements. Streams can send messages to multiple targets, but not like that.
First run
-
We’ll need a Kafka producer and consumer to see the application doing
-anything. We’ll send messages to the topic
-streams-plaintext-input, and consume them from
-streams-pipe-output. The application should pipe messages
-from the one topic to the other.
-
A simple way to send and receive messages is to run the simple Kafka
-console tools in two different sessions:
+
We’ll need a Kafka producer and consumer to see the application doing anything. We’ll send messages to the topic streams-plaintext-input, and consume them from streams-pipe-output. The application should pipe messages from the one topic to the other.
+
A simple way to send and receive messages is to run the simple Kafka console tools in two different sessions:
With the application running, anything typed into the message
-producer utility should appear at the message consumer. If it does, well
-done, the Streams application is working.
-
Adding a simple
-transformation
-
Let’s extend the trivial Streams topology to performa a simple
-transformation – converting textual input to lower case.
+
With the application running, anything typed into the message producer utility should appear at the message consumer. If it does, well done, the Streams application is working.
+
Adding a simple transformation
+
Let’s extend the trivial Streams topology to performa a simple transformation – converting textual input to lower case.
We’ve inserted a call to mapValues() between the message
-consumptin and production.
-
mapValues is a method that is defined to take one
-parameter, which implements the ValueMapper interface. This
-interface has one method, that takes one argument and returns one return
-value. The lamda expression
+
We’ve inserted a call to mapValues() between the message consumptin and production.
+
mapValues is a method that is defined to take one parameter, which implements the ValueMapper interface. This interface has one method, that takes one argument and returns one return value. The lamda expression
value->((String)value).toLowerCase()
-
satisfies this interface, because it implicitly defines a method with
-one parameter and one return value.
+
satisfies this interface, because it implicitly defines a method with one parameter and one return value.
Note
-mapValues() is broadly similar to Camel’s
-process(). However, Kafka Streams also has a
-process() method with a slightly different purpose.
+mapValues() is broadly similar to Camel’s process(). However, Kafka Streams also has a process() method with a slightly different purpose.
-
In short, mapValues() provides a Kafka message body to
-an expression, and sets the new message body to the result of the
-expression.
+
In short, mapValues() provides a Kafka message body to an expression, and sets the new message body to the result of the expression.
Note
-The Kafka Streams API relies heavily on generic types and lambda
-expressions. If you’re not familiar with how these work in Java, it
-might be worth brushing up before digging into Kafka Streams beyond the
-‘Hello, World’ level.
+The Kafka Streams API relies heavily on generic types and lambda expressions. If you’re not familiar with how these work in Java, it might be worth brushing up before digging into Kafka Streams beyond the ‘Hello, World’ level.
-
When you run this example, and messages sent to
-streams-plaintext-input should appear on
-streams-pipe-output in only lower case.
+
When you run this example, and messages sent to streams-plaintext-input should appear on streams-pipe-output in only lower case.
Note
-Kafka messages form key-value pairs. This example does not transform, or
-even see, message keys. There is a map() method that
-operates on keys, as well as message bodies.
+Kafka messages form key-value pairs. This example does not transform, or even see, message keys. There is a map() method that operates on keys, as well as message bodies.
Adding some simple logging
-
Perhaps we want to trace the way that a message is transformed in the
-execution of a topology. To do that, we’ll need to ‘see’ the message
-body, but not change it. We can use the peek() method for
-that.
+
Perhaps we want to trace the way that a message is transformed in the execution of a topology. To do that, we’ll need to ‘see’ the message body, but not change it. We can use the peek() method for that.
The peek() method takes a paramter of type
-ForeachAction, which is an interface that specifies a
-single method that takes two values, and returns nothing
-(void). To satisfy this interface, our lambda expression
-must take two arguments:
+
The peek() method takes a paramter of type ForeachAction, which is an interface that specifies a single method that takes two values, and returns nothing (void). To satisfy this interface, our lambda expression must take two arguments:
.peek ((key,value)->...)
-
and call a method which returns void (as
-println() does).
-
In this case, I’m only displaying the message body, so I don’t need
-the key argument; but it has to be present, even if only as
-a placeholder, or the lambda expression won’t match the interface.
-
Because the interface method is defined to return void,
-the right-hand side of the lambda’s -> must be a method
-call that is defined to return void as well. So the
-peek() method does not allow an expression that modifies
-the key or the value.
-
When you run this example, you should see some logging output from
-the application, something like this:
+
and call a method which returns void (as println() does).
+
In this case, I’m only displaying the message body, so I don’t need the key argument; but it has to be present, even if only as a placeholder, or the lambda expression won’t match the interface.
+
Because the interface method is defined to return void, the right-hand side of the lambda’s -> must be a method call that is defined to return void as well. So the peek() method does not allow an expression that modifies the key or the value.
+
When you run this example, you should see some logging output from the application, something like this:
before: Hello, World!
after: hello, world!
-
Duplicating and
-filtering the message stream
-
In this section I’ll describe some simple message processing
-operations that could be added to the basic Streams application.
+
Duplicating and filtering the message stream
+
In this section I’ll describe some simple message processing operations that could be added to the basic Streams application.
Note
-I’m not giving explicit instructions on running these examples: the
-changes are in the same part of the code as all those above and, I hope,
-the results will be reasonably clear.
+I’m not giving explicit instructions on running these examples: the changes are in the same part of the code as all those above and, I hope, the results will be reasonably clear.
-
If we want to duplicate a message to multiple topics, we can’t do
-this:
+
If we want to duplicate a message to multiple topics, we can’t do this:
The to() method is terminal in the fluent builder – it
-doesn’t return anything we can call further methods on. This is,
-perhaps, a little disconcerting to Camel programmers, because the Camel
-builder does allow it.
+
The to() method is terminal in the fluent builder – it doesn’t return anything we can call further methods on. This is, perhaps, a little disconcerting to Camel programmers, because the Camel builder does allow it.
We can, however, do this, which has the same effect:
This topology defintion will take the input from
-streams-plaintext-input and send it to both
-streams-pipe-output and
-streams-pipe-output-long. The reason for the
--long naming should become clear shortly.
+
This topology defintion will take the input from streams-plaintext-input and send it to both streams-pipe-output and streams-pipe-output-long. The reason for the -long naming should become clear shortly.
Note
-It won’t always matter, but it’s worth bearing in mind that this method
-of splitting the stream duplicates the message keys as well as the
-values.
+It won’t always matter, but it’s worth bearing in mind that this method of splitting the stream duplicates the message keys as well as the values.
-
Rather than a straight duplication, we can filter the messages, so
-that some go to one topic, and some to others. In the example below,
-messages whose bodies are ten characters in length or shorter go to
-streams-pipe-output-long, while others go to
-streams-pipe-output.
+
Rather than a straight duplication, we can filter the messages, so that some go to one topic, and some to others. In the example below, messages whose bodies are ten characters in length or shorter go to streams-pipe-output-long, while others go to streams-pipe-output.
The filter() method takes a lambda expression that
-evaluates to a boolean value – true to forward the message,
-false to drop it.
-
When it comes to dynamic routing, that is, routing messages to topics
-whose names are determined at runtime, it’s easy to make the same
-mistake that Camel developers often make.
-
Suppose I want to route messages to a topic whose name depends on
-(say) the current date. I have a method getDateString()
-that returns the date in a suitable format. It’s tempting to write
-something like this:
+
The filter() method takes a lambda expression that evaluates to a boolean value – true to forward the message, false to drop it.
+
When it comes to dynamic routing, that is, routing messages to topics whose names are determined at runtime, it’s easy to make the same mistake that Camel developers often make.
+
Suppose I want to route messages to a topic whose name depends on (say) the current date. I have a method getDateString() that returns the date in a suitable format. It’s tempting to write something like this:
This fails, for reasons that are not obvious. The reason it fails is
-that the fluent builder pattern creates a topology only once. It may
-well executes for the life of the application, but all the data it needs
-to execute has to be provided at start-up time. The
-getDateString() method will be executed only once and,
-though the date changes, the argument to the to() was
-evaluated only once.
-
Kafka Streams has a way around this problem (as Camel does). Rather
-than initializing the to() method with a string, we
-initialize it with a TopicExtractor. In practice, it seems
-to be common to use a lambda expression to do this, rather than writing
-a class that implements the TopicExtractor interface
-explicitly. So we could do this:
+
This fails, for reasons that are not obvious. The reason it fails is that the fluent builder pattern creates a topology only once. It may well executes for the life of the application, but all the data it needs to execute has to be provided at start-up time. The getDateString() method will be executed only once and, though the date changes, the argument to the to() was evaluated only once.
+
Kafka Streams has a way around this problem (as Camel does). Rather than initializing the to() method with a string, we initialize it with a TopicExtractor. In practice, it seems to be common to use a lambda expression to do this, rather than writing a class that implements the TopicExtractor interface explicitly. So we could do this:
Duplicating and
Branched.withConsumer
(stream->stream.to ("streams-pipe-output")) );
Closing remarks
-
It’s probably not a surprise that this article only scratches the
-surface of the Kafka Streams API, which is extensive. on stateful
-operations, stream merging, partition and key management, error
-handling, etc., etc. place to start. In Part 2 I’ll tackle the first of
-these – the stateful operations of counting and aggregation.
+
It’s probably not a surprise that this article only scratches the surface of the Kafka Streams API, which is extensive. on stateful operations, stream merging, partition and key management, error handling, etc., etc. place to start. In Part 2 I’ll tackle the first of these – the stateful operations of counting and aggregation.
diff --git a/kafka_streams_hello_2.html b/kafka_streams_hello_2.html
index c4e48d2..0d8c201 100644
--- a/kafka_streams_hello_2.html
+++ b/kafka_streams_hello_2.html
@@ -2,8 +2,7 @@
- Kevin Boone: Getting started with
-Kafka Streams, part 2
+ Kevin Boone: Getting started with Kafka Streams, part 2
@@ -41,99 +40,36 @@
-
Getting started with
-Kafka Streams, part 2
+
Getting started with Kafka Streams, part 2
-
This article follows directly from Part 1, and uses the same set-up and
-basic code. It covers the rudiments of stateful operations,
-time-windowed operations, and aggregation. That might sound like an
-awful lot of material but, in fact, all three topics can be demonstrated
-in a simple application. If you haven’t read the first article, however,
-I would recommend doing so – I’m not sure this one will make any sense,
-otherwise.
-
All the examples in Part 1 were stateless; that is, they
-worked on one record at a time, and nothing depended on anything that
-happened in the past. Aggregation and counting are, however, inherently
-stateful – they depend critically on the history of the messages that
-have been processed up to a particular point.
-
In this example, I will implement a simple count of records with the
-same key. This could be genuinely useful. Suppose, for example, that you
-have a transactional web-based system for trading certain items. Each
-time a trade happens, the system writes a record to Kafka whose key is
-the ID of the user who made it. Or perhaps (or, in addition) the system
-writes a record indicating the commodity that was traded, and how
-much.
-
We can split the message stream from the relevant topics by key, then
-use a counter to build a table of totals for each key. Then we can,
-perhaps, feed back the totals into another Kafka topic, or write it to a
-database table.
+
This article follows directly from Part 1, and uses the same set-up and basic code. It covers the rudiments of stateful operations, time-windowed operations, and aggregation. That might sound like an awful lot of material but, in fact, all three topics can be demonstrated in a simple application. If you haven’t read the first article, however, I would recommend doing so – I’m not sure this one will make any sense, otherwise.
+
All the examples in Part 1 were stateless; that is, they worked on one record at a time, and nothing depended on anything that happened in the past. Aggregation and counting are, however, inherently stateful – they depend critically on the history of the messages that have been processed up to a particular point.
+
In this example, I will implement a simple count of records with the same key. This could be genuinely useful. Suppose, for example, that you have a transactional web-based system for trading certain items. Each time a trade happens, the system writes a record to Kafka whose key is the ID of the user who made it. Or perhaps (or, in addition) the system writes a record indicating the commodity that was traded, and how much.
+
We can split the message stream from the relevant topics by key, then use a counter to build a table of totals for each key. Then we can, perhaps, feed back the totals into another Kafka topic, or write it to a database table.
Note
-In this article, I will treat counting as a special kind of aggregation.
-In both cases, we build a two-dimensional structure from a
-one-dimensional message flow. The difference is that, with aggregation
-there will usually be some further processing; with a counting
-procedure, the counts are generally the final result. Both procedures,
-counting and aggregation, use essentially the same structure in Kafka
-Streams.
+In this article, I will treat counting as a special kind of aggregation. In both cases, we build a two-dimensional structure from a one-dimensional message flow. The difference is that, with aggregation there will usually be some further processing; with a counting procedure, the counts are generally the final result. Both procedures, counting and aggregation, use essentially the same structure in Kafka Streams.
-
Aggregation is generally by key in Kafka Streams. That is,
-we assume that the message key means something. It might be a
-user ID, or a product ID, or something else, but it has to be something.
-If the messages have no key, then usually the first step in aggregation
-will be to derive one.
+
Aggregation is generally by key in Kafka Streams. That is, we assume that the message key means something. It might be a user ID, or a product ID, or something else, but it has to be something. If the messages have no key, then usually the first step in aggregation will be to derive one.
KStreams and KTables
-
To understand counting and aggregation, it’s crucial to be clear on
-the distinction between a KStream and a
-KTable. Frankly, though, I can do little better to
-illustrate this than to recommend the graphical demonstration of the
-duality between tables and streams in the Confluent documentation.
-
In brief, A KTable is a two-dimensional, key-value
-structure, which is generated by treating individual messages as entries
-in a change log. That is, each row in the table has a key that matches
-the message key, and the value is the latest value received from the
-stream. If you’ve used Debezium to capture database changes into Kafka,
-you’ll already be familiar with the idea that a stream of messages can
-capture the state of a database table as it changes over time. It’s less
-obvious, however, that the reverse is also true – if we have a log of
-changes to a table, we can always ‘replay’ the changes to reconstruct
-the original table.
-
So Kafka Streams assumes a kind of equivalence between message
-streams and tables – it’s always possible to construct one from the
-other. However, some Streams APIs are best expressed in terms of tables,
-rather than streams.
+
To understand counting and aggregation, it’s crucial to be clear on the distinction between a KStream and a KTable. Frankly, though, I can do little better to illustrate this than to recommend the graphical demonstration of the duality between tables and streams in the Confluent documentation.
+
In brief, A KTable is a two-dimensional, key-value structure, which is generated by treating individual messages as entries in a change log. That is, each row in the table has a key that matches the message key, and the value is the latest value received from the stream. If you’ve used Debezium to capture database changes into Kafka, you’ll already be familiar with the idea that a stream of messages can capture the state of a database table as it changes over time. It’s less obvious, however, that the reverse is also true – if we have a log of changes to a table, we can always ‘replay’ the changes to reconstruct the original table.
+
So Kafka Streams assumes a kind of equivalence between message streams and tables – it’s always possible to construct one from the other. However, some Streams APIs are best expressed in terms of tables, rather than streams.
Note
-The equivalence between KTable and KStream is
-not sufficiently close that they can be derived from a common base
-class. Unfortunately. These classes have some methods in common, but not
-all.
+The equivalence between KTable and KStream is not sufficiently close that they can be derived from a common base class. Unfortunately. These classes have some methods in common, but not all.
The sample application
-
I’ll describe how to code and test the application first, and then
-explain how it works.
-
Go back to the original Pipe.java application, created
-from the kafka-streams-quickstart Maven archetype. Replace
-the definition of the topology with the following:
+
I’ll describe how to code and test the application first, and then explain how it works.
+
Go back to the original Pipe.java application, created from the kafka-streams-quickstart Maven archetype. Replace the definition of the topology with the following:
To test the application we will need to send messages to the
-streams-plaintext-input Kafka topic. There is no output
-topic – in this case output is only to the console.
-
I’m assuming that you still have Kafka set up, and the input topic
-exists. So run the application using Maven, as before:
+
To test the application we will need to send messages to the streams-plaintext-input Kafka topic. There is no output topic – in this case output is only to the console.
+
I’m assuming that you still have Kafka set up, and the input topic exists. So run the application using Maven, as before:
Of course, use your own package name if you didn’t copy the steps in
-Part 1 exactly, and chose your own naming.
-
To exercise the application, you’ll need to send messages with
-specific keys. It’s a little fiddly to do this with
-kafka-console-producer.sh, but possible. You need to
-specify a key-value separator, like this (I have chosen the colon, but
-any character will be fine):
+
Of course, use your own package name if you didn’t copy the steps in Part 1 exactly, and chose your own naming.
+
To exercise the application, you’ll need to send messages with specific keys. It’s a little fiddly to do this with kafka-console-producer.sh, but possible. You need to specify a key-value separator, like this (I have chosen the colon, but any character will be fine):
The actual message body is irrelevant in this example, because the
-application does not process or even display the bodies – it just counts
-messages with specific keys.
-
As the Streams application runs it will eventually produce output of
-this form:
+
The actual message body is irrelevant in this example, because the application does not process or even display the bodies – it just counts messages with specific keys.
+
As the Streams application runs it will eventually produce output of this form:
dog:2
cat:1
budgie:1
-
That is, there have (so far) been two messages with key
-dog, and one each with cat and
-budgie. If you send more messages, you’ll see updates with
-the new totals.
-
You might notice that updates are not immediate. It might take up to
-30 seconds. This is the time interval at which Kafka Streams commits
-open streams. The other time at which you might see some output is when
-the aggregation cache is full; but, since the default size is 10Mb,
-that’s unlikely to happen here. Both the commit interval and the cache
-size can be changed when the KafkaStreams application
-instance is created.
+
That is, there have (so far) been two messages with key dog, and one each with cat and budgie. If you send more messages, you’ll see updates with the new totals.
+
You might notice that updates are not immediate. It might take up to 30 seconds. This is the time interval at which Kafka Streams commits open streams. The other time at which you might see some output is when the aggregation cache is full; but, since the default size is 10Mb, that’s unlikely to happen here. Both the commit interval and the cache size can be changed when the KafkaStreams application instance is created.
Analysing the application
-
The sample application is mostly trivial, but there’s a lot going on.
-Let’s look at it step by step.
+
The sample application is mostly trivial, but there’s a lot going on. Let’s look at it step by step.
The method call
builder.stream("streams-plaintext-input")
-
creates an instance of KStream that models the raw
-message stream from the Kafka topic. Calling groupByKey()
-on this object creates an instance of KGroupedStream. This
-models a message stream that can be differentiated by its keys.
-
There are essentially three things that can be done with a
-KGroupedStream:
+
creates an instance of KStream that models the raw message stream from the Kafka topic. Calling groupByKey() on this object creates an instance of KGroupedStream. This models a message stream that can be differentiated by its keys.
+
There are essentially three things that can be done with a KGroupedStream:
Create a counter on it, or
create an aggregator on it, or
-
create a new grouped stream with specific time-windowing
-properties.
+
create a new grouped stream with specific time-windowing properties.
-
In the simple example, I just call count() to create a
-new counter. This method returns a KTable. This is a
-two-dimensional representation of the data stream, where the first
-column is the message key, and the second (in this case) is the
-count.
-
For the purposes of this exercise, it would be nice if there were a
-method KTable.peek() that extracts the table’s contents, as
-KStream.peek() does for streams. So far as I know, however,
-there is no such method. What we can do, however, is convert the
-KTable back to a KStream, and call
-peek() on that.
-
Looking
-at the operation of the application in more detail
-
If the Kafka Streams application is running, try stopping it and
-starting it again. Then send some more messages to
-streams-plaintext-input, with the same keys as before. You
-should notice that the totals include values from the previous run. By
-default, the aggregation is not merely stateful, it is persistent as
-well.
-
It’s reasonable to wonder where this persistent state is stored. By
-default it is stored on disk in a RocksDB database. The location of the
-store is, by default,
+
In the simple example, I just call count() to create a new counter. This method returns a KTable. This is a two-dimensional representation of the data stream, where the first column is the message key, and the second (in this case) is the count.
+
For the purposes of this exercise, it would be nice if there were a method KTable.peek() that extracts the table’s contents, as KStream.peek() does for streams. So far as I know, however, there is no such method. What we can do, however, is convert the KTable back to a KStream, and call peek() on that.
+
Looking at the operation of the application in more detail
+
If the Kafka Streams application is running, try stopping it and starting it again. Then send some more messages to streams-plaintext-input, with the same keys as before. You should notice that the totals include values from the previous run. By default, the aggregation is not merely stateful, it is persistent as well.
+
It’s reasonable to wonder where this persistent state is stored. By default it is stored on disk in a RocksDB database. The location of the store is, by default,
/tmp/kafka-streams/streams_pipe/0_0
-
streams_pipe, you may recall, is the application name,
-set in the APPLICATION_ID_CONFIG configuration property.
-The numbers in the directory name reflect the fact that the topology
-might include multiple persistent elements, and they need their own data
-stores.
-
Storing data this way is adequate when there is only one instance of
-the client. But in a clustered set-up, where the application may run on
-multiple hosts, what happens if one of the application instances
-crashes, or is shut down, in the middle of an aggregation? The local
-file store is not available to other instances.
-
The solution is to back up the aggregation store on the Kafka broker
-itself. Look at the topics on the broker, after running the
-application:
+
streams_pipe, you may recall, is the application name, set in the APPLICATION_ID_CONFIG configuration property. The numbers in the directory name reflect the fact that the topology might include multiple persistent elements, and they need their own data stores.
+
Storing data this way is adequate when there is only one instance of the client. But in a clustered set-up, where the application may run on multiple hosts, what happens if one of the application instances crashes, or is shut down, in the middle of an aggregation? The local file store is not available to other instances.
+
The solution is to back up the aggregation store on the Kafka broker itself. Look at the topics on the broker, after running the application:
The aggregation store is kept in a Kafka topic in the form of a
-change log. If necessary, the local aggregation stored can be
-reconstructed from this change log.
-
It is this Kafka-based back-up of the replication store that makes it
-safe to keep the local store in memory, rather than a RocksDB database.
-The WordCount example application shows a way to do
-that.
-
Despite the persistence, you might find that if you leave the
-application running long enough, the aggregation totals get reset.
-That’s because the aggregation process has a time window of one hour by
-default. In practice, applications will usually seek to total over a
-specific time window. The Streams API provides classes like
-TimeWindowedKStream to do this.
+
The aggregation store is kept in a Kafka topic in the form of a change log. If necessary, the local aggregation stored can be reconstructed from this change log.
+
It is this Kafka-based back-up of the replication store that makes it safe to keep the local store in memory, rather than a RocksDB database. The WordCount example application shows a way to do that.
+
Despite the persistence, you might find that if you leave the application running long enough, the aggregation totals get reset. That’s because the aggregation process has a time window of one hour by default. In practice, applications will usually seek to total over a specific time window. The Streams API provides classes like TimeWindowedKStream to do this.
A technical digression
-
There’s one more thing to demonstrate in this article, and it’s a
-rather technical one, but important once the application gets beyond the
-“Hello, World” stage.
-
In the application we used toStream() to convert the
-table of counts back to a stream, and then peek() to print
-it. It should be possible to send the converted stream to another Kafka
-topic and, indeed, it is – but the results might be surprising. Try
-adding a to() call to the application:
+
There’s one more thing to demonstrate in this article, and it’s a rather technical one, but important once the application gets beyond the “Hello, World” stage.
+
In the application we used toStream() to convert the table of counts back to a stream, and then peek() to print it. It should be possible to send the converted stream to another Kafka topic and, indeed, it is – but the results might be surprising. Try adding a to() call to the application:
If you send some more data to streams-plaintext-input,
-you should see some output from the consumer. But you’ll probably find
-that the totals are missing. Why?
-
I’ve said that the count() method returns a
-KTable, but KTable is a template class, as
-KStream is: both are templatized by key and value. That is,
-a KTable isn’t just a table of objects; it’s a table of
-specific key and value classes. In most of the previous examples I’ve
-fudged over this fact, and often it’s not important. In this case,
-however, it is.
-
So the count() method doesn’t really return a
-KTable – it returns (in this case) a
-KTable<?, Long>. The type of the key is, in this
-case, unspecified (because the code doesn’t say what it should be). At
-runtime the key class will be String because that’s the way
-the serialization is set up. But the ‘value’ column of the count table
-is parameterized as a Long, whatever the type of the stream
-it was derived from.
-
Then, when we call toStream() on the output of
-count() what we actually get is an instance of
-KStream<?, Long>.
-
This makes perfect sense – a count can only be a number. But when we
-write to the output stream, we’re writing messages whose payload is a
-Long, and kafka-console-consumer.sh assumes by
-default that the values are strings.
+
If you send some more data to streams-plaintext-input, you should see some output from the consumer. But you’ll probably find that the totals are missing. Why?
+
I’ve said that the count() method returns a KTable, but KTable is a template class, as KStream is: both are templatized by key and value. That is, a KTable isn’t just a table of objects; it’s a table of specific key and value classes. In most of the previous examples I’ve fudged over this fact, and often it’s not important. In this case, however, it is.
+
So the count() method doesn’t really return a KTable – it returns (in this case) a KTable<?, Long>. The type of the key is, in this case, unspecified (because the code doesn’t say what it should be). At runtime the key class will be String because that’s the way the serialization is set up. But the ‘value’ column of the count table is parameterized as a Long, whatever the type of the stream it was derived from.
+
Then, when we call toStream() on the output of count() what we actually get is an instance of KStream<?, Long>.
+
This makes perfect sense – a count can only be a number. But when we write to the output stream, we’re writing messages whose payload is a Long, and kafka-console-consumer.sh assumes by default that the values are strings.
So to see the counts, we need to run the consumer like this:
If you want to store the counts in the Kafka topic as strings, you’ll
-need to make a conversion in code.
+
If you want to store the counts in the Kafka topic as strings, you’ll need to make a conversion in code.
Closing remarks
-
In this article I demonstrated the rudiments of stateful operations
-(counts and aggregations) in Kafka Streams. There is, of course, a lot
-more to this topic. In particular, the API provides powerful control of
-time windowing, and a configurable notion of what a ‘timestamp’ amounts
-to in the application. The difference between, for example, the time at
-which an event was detected, and the time it was ingested into Kafka,
-might only amount to microseconds; but with the throughputs that Kafka
-is designed for, that difference might be important.
+
In this article I demonstrated the rudiments of stateful operations (counts and aggregations) in Kafka Streams. There is, of course, a lot more to this topic. In particular, the API provides powerful control of time windowing, and a configurable notion of what a ‘timestamp’ amounts to in the application. The difference between, for example, the time at which an event was detected, and the time it was ingested into Kafka, might only amount to microseconds; but with the throughputs that Kafka is designed for, that difference might be important.
diff --git a/no_smartphone.html b/no_smartphone.html
index 4210014..733454e 100644
--- a/no_smartphone.html
+++ b/no_smartphone.html
@@ -2,8 +2,7 @@
- Kevin Boone: My journey towards
-a smartphone-free future
+ Kevin Boone: My journey towards a smartphone-free future
@@ -41,342 +40,99 @@
-
My journey towards
-a smartphone-free future
+
My journey towards a smartphone-free future
-
Last year I divorced Google from my life
-by installing LineageOS with no Google apps support on my Android phone.
-This lost me the ability to install any apps that were not open source
-or, at least, available as APK files to side-load. In fact, I take this
-inability to install apps to be a gain rather than a loss – this step
-alone has removed a lot of pointless apps from my life. Still, I
-continue to find new ways to waste time on my phone: scrolling through
-news feeds, playing stupid puzzle games, and so on.
-
So I now regard this move away from Google services as the first step
-on my journey to a completely smartphone-free life. However it may
-appear, I should point out that I’m not a luddite: I work in the
-computing industry, and I have a stack of computer equipment, some of
-which I designed and built myself. I’ve even written apps for Android
-myself. Moreover, I have no objection to mobile telephony in general: on
-the contrary, I think it’s one of the most significant, beneficial
-inventions in human history. What I don’t like are the time-wasting
-opportunities that smartphones make almost inevitable, and I
-really don’t want to be part of the surveillance economy.
+
Last year I divorced Google from my life by installing LineageOS with no Google apps support on my Android phone. This lost me the ability to install any apps that were not open source or, at least, available as APK files to side-load. In fact, I take this inability to install apps to be a gain rather than a loss – this step alone has removed a lot of pointless apps from my life. Still, I continue to find new ways to waste time on my phone: scrolling through news feeds, playing stupid puzzle games, and so on.
+
So I now regard this move away from Google services as the first step on my journey to a completely smartphone-free life. However it may appear, I should point out that I’m not a luddite: I work in the computing industry, and I have a stack of computer equipment, some of which I designed and built myself. I’ve even written apps for Android myself. Moreover, I have no objection to mobile telephony in general: on the contrary, I think it’s one of the most significant, beneficial inventions in human history. What I don’t like are the time-wasting opportunities that smartphones make almost inevitable, and I really don’t want to be part of the surveillance economy.
Why is there a problem?
-
It should be as simple as this: if you want a smartphone, have one;
-if you don’t, don’t. Why do we not have this simple choice any more?
-
The modern smartphone is a general-purpose device: apart from
-telephony it provides Internet access, camera, media, email, navigation
-and many other things. Replacing a smartphone requires many additional
-devices, which together are considerably more expensive, as well as
-inconvenient. Either that, or we must learn to live without certain
-convenience features that we have come to accept.
-
Nevertheless, I spent my first forty years without a smartphone, and
-I can’t remember my life being impoverished as a result. So I have to
-question how much I need these all these capabilities, and what I might
-be losing by concentrating them into one device.
-
Smartphones are not yet ubiquitous: I’m told that about 20% of people
-in the UK don’t have one. Many of these people are elderly, and find
-this kind of technology unfamiliar. But I know people in their 80s who
-are welded to their handsets, and teenagers who won’t touch them: it’s
-not simply a young/old split.
-
The problem is that, as the uptake of smartphones has increased, the
-number of businesses and services that can cater to the phone-challenged
-has decreased. It’s clearly cheaper for a business to use a
-fully-automated interface than to provide human contacts. That many
-businesses and services now offer smartphone apps is fine; that so many
-insist on them is not. Even people who are happy, in general, with using
-smartphones have complained to me about the sheer number of different
-apps they need. The situation is not helped by the fact that the apps
-often don’t work reliably.
+
It should be as simple as this: if you want a smartphone, have one; if you don’t, don’t. Why do we not have this simple choice any more?
+
The modern smartphone is a general-purpose device: apart from telephony it provides Internet access, camera, media, email, navigation and many other things. Replacing a smartphone requires many additional devices, which together are considerably more expensive, as well as inconvenient. Either that, or we must learn to live without certain convenience features that we have come to accept.
+
Nevertheless, I spent my first forty years without a smartphone, and I can’t remember my life being impoverished as a result. So I have to question how much I need these all these capabilities, and what I might be losing by concentrating them into one device.
+
Smartphones are not yet ubiquitous: I’m told that about 20% of people in the UK don’t have one. Many of these people are elderly, and find this kind of technology unfamiliar. But I know people in their 80s who are welded to their handsets, and teenagers who won’t touch them: it’s not simply a young/old split.
+
The problem is that, as the uptake of smartphones has increased, the number of businesses and services that can cater to the phone-challenged has decreased. It’s clearly cheaper for a business to use a fully-automated interface than to provide human contacts. That many businesses and services now offer smartphone apps is fine; that so many insist on them is not. Even people who are happy, in general, with using smartphones have complained to me about the sheer number of different apps they need. The situation is not helped by the fact that the apps often don’t work reliably.
-
For the purposes of full disclosure, I have to say that I really
-can’t manage without a portable music player, so I have repurposed a smartphone as a media
-player. This device has no web browser, and I have removed its ability
-to install new apps. So it’s really only useful as a media player.
-Still, I can’t claim not to have a smartphone – I’ve just chosen not to
-use it as such.
+
For the purposes of full disclosure, I have to say that I really can’t manage without a portable music player, so I have repurposed a smartphone as a media player. This device has no web browser, and I have removed its ability to install new apps. So it’s really only useful as a media player. Still, I can’t claim not to have a smartphone – I’ve just chosen not to use it as such.
-
The Web is full of articles written by people who gave up their
-smartphones for a month, or two months, or whatever. They generally
-attest to the positivity of the experience. But even two months isn’t
-nearly long enough to discover all the things we rely on our smartphones
-for, and did not even realize.
-
In this article, I am collecting the things I have struggled with, in
-my attempts to ditch my smartphone. Some will be obvious; others,
-perhaps, you’ll only notice when they’re gone.
-
I’m writing this for other people who are considering embarking on
-the same adventure: a future without a smartphone. If you’re in that
-position, it’s as well to be aware of what you’ll be losing. I feel I
-must point out from the outset that some of the features I rely on in my
-smartphone have no practical alternative.
+
The Web is full of articles written by people who gave up their smartphones for a month, or two months, or whatever. They generally attest to the positivity of the experience. But even two months isn’t nearly long enough to discover all the things we rely on our smartphones for, and did not even realize.
+
In this article, I am collecting the things I have struggled with, in my attempts to ditch my smartphone. Some will be obvious; others, perhaps, you’ll only notice when they’re gone.
+
I’m writing this for other people who are considering embarking on the same adventure: a future without a smartphone. If you’re in that position, it’s as well to be aware of what you’ll be losing. I feel I must point out from the outset that some of the features I rely on in my smartphone have no practical alternative.
Mapping
-
This is one application of a smartphone that I really, truly
-miss.
-
I have a satnav in my car, and it works fine. I used to use my
-smartphone for mapping whilst hiking, not driving, especially in the
-hills. I used ViewRanger (now Outdoor Active) for planning and guiding
-hikes. I sometimes used it when I simply got lost in a city centre, and
-had no printed map.
-
I can’t do any of this without a smartphone. However, I do have a
-stand-alone GPS unit specially for hiking. Unlike a smartphone it has a
-daylight-viewable screen, 50-hour battery life (and it takes common
-batteries, which my phone does not), and it floats if dropped in a
-river. It’s altogether a better outdoor navigation companion that Google
-Maps ever could be. I also have a smart watch (which is less useful that
-it was, without its companion smartphone app – see below) which can give
-me a grid reference.
-
I can plot a route using the ViewRanger website, and copy it to my
-GPS as a GPX file. This works fine, but it’s a few more manual steps
-than simply syncing the ViewRanger app to the website.
-
In the end, though, trying to be rid of my smartphone has made me
-aware of the risks I’ve been running, taking to the hills with nothing
-but a electronic device for navigation. I can’t imagine this is
-something the mountain rescue folks would recommend. So, while I do have
-a GPS unit, and I will be using it, I will be taking map and compass on
-the trail in future, as we did in the pre-GPS days.
+
This is one application of a smartphone that I really, truly miss.
+
I have a satnav in my car, and it works fine. I used to use my smartphone for mapping whilst hiking, not driving, especially in the hills. I used ViewRanger (now Outdoor Active) for planning and guiding hikes. I sometimes used it when I simply got lost in a city centre, and had no printed map.
+
I can’t do any of this without a smartphone. However, I do have a stand-alone GPS unit specially for hiking. Unlike a smartphone it has a daylight-viewable screen, 50-hour battery life (and it takes common batteries, which my phone does not), and it floats if dropped in a river. It’s altogether a better outdoor navigation companion that Google Maps ever could be. I also have a smart watch (which is less useful that it was, without its companion smartphone app – see below) which can give me a grid reference.
+
I can plot a route using the ViewRanger website, and copy it to my GPS as a GPX file. This works fine, but it’s a few more manual steps than simply syncing the ViewRanger app to the website.
+
In the end, though, trying to be rid of my smartphone has made me aware of the risks I’ve been running, taking to the hills with nothing but a electronic device for navigation. I can’t imagine this is something the mountain rescue folks would recommend. So, while I do have a GPS unit, and I will be using it, I will be taking map and compass on the trail in future, as we did in the pre-GPS days.
Fitness tracking
-
I have a Garmin fitness watch; I use it every day. I have certain
-training targets that I try to meet, and the watch keeps track of the
-time I spend exercising in particular heart-rate zones.
-
It’s difficult to use these watches without a smartphone app. The
-problem is that the app, even if you have it, isn’t where the magic
-happens – the app is just a gateway to Garmin’s services. I’m not sure
-how much I trust Garmin to keep my personal data safe. I used a
-throw-away email address to register, and I’ve falsified as much
-personal information I can, without invalidating the computations their
-system does.
-
Even if I trust Garmin, I can’t be sure that the Company won’t get
-bought out, along with all my personal data. That’s what happened to
-FitBit, after all. Google now holds all the fitness data that FitBit
-collected. I’m not sure how I feel about that, except relief that I was
-never a FitBit user.
-
So, without a smartphone app, my Garmin watch still records the
-training data, but I needed an app to set it up in the first place. I
-don’t know whether I would have been able to configure it at all without
-an app. The watch (or the remote service it’s connected to) needs to
-know things like my height and weight, to estimate heart rate targets,
-etc. The device itself does not seem to have a user interface where I
-can enter these parameters. Now that it’s been set up, I can use a Linux
-utility to read out the data from my watch onto a computer. This is
-better, in a way, than using Garmin’s app, because I can process the
-data as I like. However, I suspect few people will undertake such a task
-– it’s not hugely convenient.
-
On reflection, though, I have come to understand that I mostly use
-the watch because I’m just not well-organized enough to log my exercise
-activities. I could simply keep this information in a spreadsheet, or
-even in my diary. It’s easier just to let the watch work it out. In the
-long term, I’m not sure whether I’ll continue using the Garmin watch
-when I ditch my smartphone for good: to some extent it just provides a
-reason for me to avoid tacking my laziness and lack of organization.
-
Two-factor authentication
-support
-
Many businesses and institutions use a smartphone app to control
-access to their services. The app generates an authentication code that
-has to be entered into a web-based service, probably along with other
-credentials. So far, all the businesses I deal with have been able to
-work around my smartphone deficiency. They can, for example, send an
-authentication code by SMS. One bank won’t do this but, on my
-insistence, sent me a hardware authentication card to use instead of an
-app. This works fine for me.
-
So far as possible, I will not in the future do business with any
-organization that requires me to use a smartphone app. There might come
-a time when this becomes impossible. When that happens, I’m not sure
-what I’ll do.
+
I have a Garmin fitness watch; I use it every day. I have certain training targets that I try to meet, and the watch keeps track of the time I spend exercising in particular heart-rate zones.
+
It’s difficult to use these watches without a smartphone app. The problem is that the app, even if you have it, isn’t where the magic happens – the app is just a gateway to Garmin’s services. I’m not sure how much I trust Garmin to keep my personal data safe. I used a throw-away email address to register, and I’ve falsified as much personal information I can, without invalidating the computations their system does.
+
Even if I trust Garmin, I can’t be sure that the Company won’t get bought out, along with all my personal data. That’s what happened to FitBit, after all. Google now holds all the fitness data that FitBit collected. I’m not sure how I feel about that, except relief that I was never a FitBit user.
+
So, without a smartphone app, my Garmin watch still records the training data, but I needed an app to set it up in the first place. I don’t know whether I would have been able to configure it at all without an app. The watch (or the remote service it’s connected to) needs to know things like my height and weight, to estimate heart rate targets, etc. The device itself does not seem to have a user interface where I can enter these parameters. Now that it’s been set up, I can use a Linux utility to read out the data from my watch onto a computer. This is better, in a way, than using Garmin’s app, because I can process the data as I like. However, I suspect few people will undertake such a task – it’s not hugely convenient.
+
On reflection, though, I have come to understand that I mostly use the watch because I’m just not well-organized enough to log my exercise activities. I could simply keep this information in a spreadsheet, or even in my diary. It’s easier just to let the watch work it out. In the long term, I’m not sure whether I’ll continue using the Garmin watch when I ditch my smartphone for good: to some extent it just provides a reason for me to avoid tacking my laziness and lack of organization.
+
Two-factor authentication support
+
Many businesses and institutions use a smartphone app to control access to their services. The app generates an authentication code that has to be entered into a web-based service, probably along with other credentials. So far, all the businesses I deal with have been able to work around my smartphone deficiency. They can, for example, send an authentication code by SMS. One bank won’t do this but, on my insistence, sent me a hardware authentication card to use instead of an app. This works fine for me.
+
So far as possible, I will not in the future do business with any organization that requires me to use a smartphone app. There might come a time when this becomes impossible. When that happens, I’m not sure what I’ll do.
Camera
-
I don’t usually take photos with my phone. If I’m taking photos, I
-care how they come out. I use a real camera with proper lenses, that
-I’ve spent decades learning how to use properly.
-
If you need to take photos at unpredictable times, and send them away
-to somebody immediately, I suspect you won’t be able to ditch your
-smartphone. But even the old flip phones have a camera of some sort –
-it’s just less convenient, to copy the photos onto a computer to send
-them to somebody.
+
I don’t usually take photos with my phone. If I’m taking photos, I care how they come out. I use a real camera with proper lenses, that I’ve spent decades learning how to use properly.
+
If you need to take photos at unpredictable times, and send them away to somebody immediately, I suspect you won’t be able to ditch your smartphone. But even the old flip phones have a camera of some sort – it’s just less convenient, to copy the photos onto a computer to send them to somebody.
Weather forecasts
-
In the UK, you won’t usually go far wrong if you assume that it’s
-going to rain later; but unexpected heavy snow has caught out
-unfortunate hikers.
-
However, getting an up-to-date forecast requires Internet access,
-which I don’t usually have in the hills – by their very nature, the
-places I like to hike have no mobile coverage.
-
So I can’t really think of an occasion when I will really need a
-weather forecast, and actually be able to get one, with a smartphone or
-without.
-
I appreciate, of course, that this is a bigger deal for some people.
-And there’s no question that ready access to weather information is
-convenient in some situations.
+
In the UK, you won’t usually go far wrong if you assume that it’s going to rain later; but unexpected heavy snow has caught out unfortunate hikers.
+
However, getting an up-to-date forecast requires Internet access, which I don’t usually have in the hills – by their very nature, the places I like to hike have no mobile coverage.
+
So I can’t really think of an occasion when I will really need a weather forecast, and actually be able to get one, with a smartphone or without.
+
I appreciate, of course, that this is a bigger deal for some people. And there’s no question that ready access to weather information is convenient in some situations.
Tide forecasts
-
Look it up before you go! Tide times aren’t going to change from the
-forecast over a period of a few days.
+
Look it up before you go! Tide times aren’t going to change from the forecast over a period of a few days.
Taxis, etc
-
I don’t use taxi companies that require a smartphone app. There are
-still plenty of taxi operators in my neighbourhood which I can hire with
-a phone call, or just flagging a cab in the street. The app-only firms
-are a bit cheaper, to be fair. If you’re taking cabs all the time, and
-spending a lot of money, lack of access to app-based services could be a
-nuisance.
+
I don’t use taxi companies that require a smartphone app. There are still plenty of taxi operators in my neighbourhood which I can hire with a phone call, or just flagging a cab in the street. The app-only firms are a bit cheaper, to be fair. If you’re taking cabs all the time, and spending a lot of money, lack of access to app-based services could be a nuisance.
Parking
-
This is becoming an increasing problem. In the UK, many parking
-operators are moving away from cash payment, to smartphone apps. Some
-provide an alternative way to make payment, perhaps by a telephone call,
-but some do not. It’s a particular problem in London.
-
Parking operators say they do this to save money, but I don’t see any
-of those savings being passed on to consumers.
-
I’m not really sure how I feel about this. So far, I’ve rarely had a
-problem finding somewhere to park. There’s usually some place to park
-that is either free of charge, or takes coins, even if it’s further from
-where I want to be. It’s likely that, when this situation gets worse, I
-will use my bicycle more: I should do that, anyway.
-
In the long term, though – I just don’t know. I’m hoping that these
-changes will be rolled back when there are enough complaints, but I’m
-not all that confident.
+
This is becoming an increasing problem. In the UK, many parking operators are moving away from cash payment, to smartphone apps. Some provide an alternative way to make payment, perhaps by a telephone call, but some do not. It’s a particular problem in London.
+
Parking operators say they do this to save money, but I don’t see any of those savings being passed on to consumers.
+
I’m not really sure how I feel about this. So far, I’ve rarely had a problem finding somewhere to park. There’s usually some place to park that is either free of charge, or takes coins, even if it’s further from where I want to be. It’s likely that, when this situation gets worse, I will use my bicycle more: I should do that, anyway.
+
In the long term, though – I just don’t know. I’m hoping that these changes will be rolled back when there are enough complaints, but I’m not all that confident.
Alarms
-
I usually to rely on my smartphone for alarms. Old-fashioned phones
-can usually raise alarms, but usually not with the same flexibility.
-Being unable to set an alarm for 7.30PM every second Tuesday is a
-disadvantage but, for me, not a showstopper. And I have an alarm clock
-in the bedroom.
+
I usually to rely on my smartphone for alarms. Old-fashioned phones can usually raise alarms, but usually not with the same flexibility. Being unable to set an alarm for 7.30PM every second Tuesday is a disadvantage but, for me, not a showstopper. And I have an alarm clock in the bedroom.
Calendar, contacts, etc.
-
I’ve gotten used to having my calendar and contact information on my
-smartphone. I still use a on-line calendar service from my desktop
-computer (not Google’s). Now I have to carry a paper notebook and a pen,
-and I have to transfer the data to my computer manually.
-
I’m investigating other ways to handle this kind of data, that don’t
-involve a smartphone. To be honest, though, I wouldn’t mind using a
-paper diary exclusively – that’s what I did for decades, before
-computers became portable.
-
What my paper diary won’t do is to alert me when I have an
-appointment coming up. I have been, again, using a smartphone as a
-substitute for proper planning.
+
I’ve gotten used to having my calendar and contact information on my smartphone. I still use a on-line calendar service from my desktop computer (not Google’s). Now I have to carry a paper notebook and a pen, and I have to transfer the data to my computer manually.
+
I’m investigating other ways to handle this kind of data, that don’t involve a smartphone. To be honest, though, I wouldn’t mind using a paper diary exclusively – that’s what I did for decades, before computers became portable.
+
What my paper diary won’t do is to alert me when I have an appointment coming up. I have been, again, using a smartphone as a substitute for proper planning.
Entrance to venues
-
This is, again, an increasing problem. During the Covid-19 pandemic,
-many bars, restaurants, and concert venues stopped taking money
-face-to-face, in favour of remote payment. Generally, the customer would
-get a barcode or QR code that could be displayed on a smartphone screen,
-to get access or collect an order.
-
Sometimes these smartphone codes arrive by email, and can be printed.
-This is somewhat inconvenient, particularly if you don’t have a printer,
-but fine for me. Sometimes, though, there’s only an app.
-
There aren’t many places I want to go to badly enough to break my
-resolution to avoid the smartphone. I like to hope that businesses will
-eventually realize that they are losing custom, and behave more
-inclusively.
+
This is, again, an increasing problem. During the Covid-19 pandemic, many bars, restaurants, and concert venues stopped taking money face-to-face, in favour of remote payment. Generally, the customer would get a barcode or QR code that could be displayed on a smartphone screen, to get access or collect an order.
+
Sometimes these smartphone codes arrive by email, and can be printed. This is somewhat inconvenient, particularly if you don’t have a printer, but fine for me. Sometimes, though, there’s only an app.
+
There aren’t many places I want to go to badly enough to break my resolution to avoid the smartphone. I like to hope that businesses will eventually realize that they are losing custom, and behave more inclusively.
Airline travel
-
I recently flew from Bergen in Norway to Amsterdam. This was the
-first time I had encountered an airpoint that seemed to be completely
-unstaffed. The baggage drop and check-in were completely automated, and
-driven by bar-codes. I had checked in on-line the day before, and I had
-printed my bar-code, so it was fine. But if I had not had a printer
-available, and didn’t have a smartphone to display the bar-code, I could
-have been stuck.
-
There must – surely – have been a staffed check-in desk somwhere, but
-I didn’t see one. This wasn’t a problem I had at Heathrow: although
-there’s little to be said in favour of Heathrow Airport, it was at least
-staffed.
-
I don’t fly often, and I imagine it will be even less often when
-there is no way to travel without a smartphone.
+
I recently flew from Bergen in Norway to Amsterdam. This was the first time I had encountered an airpoint that seemed to be completely unstaffed. The baggage drop and check-in were completely automated, and driven by bar-codes. I had checked in on-line the day before, and I had printed my bar-code, so it was fine. But if I had not had a printer available, and didn’t have a smartphone to display the bar-code, I could have been stuck.
+
There must – surely – have been a staffed check-in desk somwhere, but I didn’t see one. This wasn’t a problem I had at Heathrow: although there’s little to be said in favour of Heathrow Airport, it was at least staffed.
+
I don’t fly often, and I imagine it will be even less often when there is no way to travel without a smartphone.
Social media
-
Some of the popular social media sites are accessible using a desktop
-computer rather than a smartphone; some are not. In general, I don’t use
-social media – I’m old enough to consider the use of computer technology
-as a business activity, not a social one. But a number of charitable and
-social organizations I’ve been involved with in the past now organize
-themselves entirely using WhatsApp, and that more-or-less excludes
-me.
-
Oddly, this even includes a choir that specializes in renaissance
-music. The average age of members of this choir was about fifty, and
-about half the singers did not have smartphones. The advantages of using
-social media for organization are obvious. The disadvantage is that you
-could lose a large part of your membership, which is more-or-less what
-happened to the choir.
-
Because I’ve never been much of a social media fan, being unable to
-use it has been mostly not had a significant impact. However, the move
-to social media by voluntary and charitable organizations is a
-lamentable one, because there’s never enough volunteers, and we really
-don’t want to discourage them.
-
I sometimes wonder if these kinds of organizations emphasize social
-media in a deliberate attempt to recruit ‘the youth’? If it is, it
-probably won’t work: among the people who have decided to ditch their
-smartphones, there are as many under 30s as over 50s.
-
For all that, if you’re a significant social media consumer, the lack
-of a smartphone is likely to be a disadvantage.
+
Some of the popular social media sites are accessible using a desktop computer rather than a smartphone; some are not. In general, I don’t use social media – I’m old enough to consider the use of computer technology as a business activity, not a social one. But a number of charitable and social organizations I’ve been involved with in the past now organize themselves entirely using WhatsApp, and that more-or-less excludes me.
+
Oddly, this even includes a choir that specializes in renaissance music. The average age of members of this choir was about fifty, and about half the singers did not have smartphones. The advantages of using social media for organization are obvious. The disadvantage is that you could lose a large part of your membership, which is more-or-less what happened to the choir.
+
Because I’ve never been much of a social media fan, being unable to use it has been mostly not had a significant impact. However, the move to social media by voluntary and charitable organizations is a lamentable one, because there’s never enough volunteers, and we really don’t want to discourage them.
+
I sometimes wonder if these kinds of organizations emphasize social media in a deliberate attempt to recruit ‘the youth’? If it is, it probably won’t work: among the people who have decided to ditch their smartphones, there are as many under 30s as over 50s.
+
For all that, if you’re a significant social media consumer, the lack of a smartphone is likely to be a disadvantage.
News and feeds
-
One of the first things I’ve generally done each day is to look at
-news feeds on my smartphone. I can do this just as well on a desktop
-computer – but I typically don’t because, when I’m sitting at my
-computer, I’m working.
-
This is one of those applications of a smartphone that seem to offer
-a modicum of convenience, with few negative connotations. But, at the
-same time, the convenience really is a modest one. I have many other
-sources of news.
+
One of the first things I’ve generally done each day is to look at news feeds on my smartphone. I can do this just as well on a desktop computer – but I typically don’t because, when I’m sitting at my computer, I’m working.
+
This is one of those applications of a smartphone that seem to offer a modicum of convenience, with few negative connotations. But, at the same time, the convenience really is a modest one. I have many other sources of news.
Email and messaging
-
Email has conventionally been the most effective way to contact me,
-whatever the time of day, or wherever I was. My smartphone handles email
-pretty well – even now it’s degoogled. I’ve gotten used to checking my
-email regularly, or whenever my smartphone nags me, which is all the
-time.
-
Without a smartphone, the best way to contact will be by SMS message.
-I’ll still be using email, of course, but not when I’m away from home.
-The problem with SMS is that it’s fiddly to send anything other than
-text.
-
Many people use ‘instant’ messaging services, that are often
-associated with social media sites. With a smartphone, these services
-will not be usable. But social media messaging services are no more
-‘instant’ than email, nor any easier to use. Still, I can’t control the
-way other people choose to communicate, and my preferences definitely
-make me less contactable. I am unclear, at present, whether that will
-turn out to be a good thing or a bad one in the long term.
+
Email has conventionally been the most effective way to contact me, whatever the time of day, or wherever I was. My smartphone handles email pretty well – even now it’s degoogled. I’ve gotten used to checking my email regularly, or whenever my smartphone nags me, which is all the time.
+
Without a smartphone, the best way to contact will be by SMS message. I’ll still be using email, of course, but not when I’m away from home. The problem with SMS is that it’s fiddly to send anything other than text.
+
Many people use ‘instant’ messaging services, that are often associated with social media sites. With a smartphone, these services will not be usable. But social media messaging services are no more ‘instant’ than email, nor any easier to use. Still, I can’t control the way other people choose to communicate, and my preferences definitely make me less contactable. I am unclear, at present, whether that will turn out to be a good thing or a bad one in the long term.
Card scanners
-
Some people I know use card-scanning apps to store business cards,
-membership cards, and similar items on their smartphones. As a person
-who tends to be a little forgetful, I can see why that would be useful.
-I don’t think I’d want to store scans of my credit cards on a
-smartphone, but it would be useful to store membership cards.
-
I can’t think of an alternative to an app for this, except to be
-better organized.
+
Some people I know use card-scanning apps to store business cards, membership cards, and similar items on their smartphones. As a person who tends to be a little forgetful, I can see why that would be useful. I don’t think I’d want to store scans of my credit cards on a smartphone, but it would be useful to store membership cards.
+
I can’t think of an alternative to an app for this, except to be better organized.
Games
-
The ability to play games on a smartphone is, to me, a wholly
-negative thing. I do not at all miss being unable to do so. But if
-you’re a gamer, you’ll need to make alternative arrangements.
+
The ability to play games on a smartphone is, to me, a wholly negative thing. I do not at all miss being unable to do so. But if you’re a gamer, you’ll need to make alternative arrangements.
Music
-
If there is an inexpensive, portable music player that handles both
-local files (in modern formats) and remote streaming services, I haven’t
-found one. My Astell and Kern player can do all these things, but it’s
-hardly portable, and certainly not inexpensive.
-
This is why I have repurposed a small smartphone as a media player.
-It’s a shame nobody seems to make a decent portable media player these
-days. It’s hardly surprising, given the ubiquity of smartphones, but
-still disappointing.
+
If there is an inexpensive, portable music player that handles both local files (in modern formats) and remote streaming services, I haven’t found one. My Astell and Kern player can do all these things, but it’s hardly portable, and certainly not inexpensive.
+
This is why I have repurposed a small smartphone as a media player. It’s a shame nobody seems to make a decent portable media player these days. It’s hardly surprising, given the ubiquity of smartphones, but still disappointing.
Ebooks
-
I have, from time to time, read novels on my smartphone. It’s not an
-ideal platform for this, but I’ve sometimes found that I have my
-smartphone, and don’t have anything else.
-
I have a Kobo e-reader, and it’s superior to a smartphone for
-reading, in just about all respects. But it’s an additional expense, and
-I have to remember to take it with me.
+
I have, from time to time, read novels on my smartphone. It’s not an ideal platform for this, but I’ve sometimes found that I have my smartphone, and don’t have anything else.
+
I have a Kobo e-reader, and it’s superior to a smartphone for reading, in just about all respects. But it’s an additional expense, and I have to remember to take it with me.
Health monitoring devices
-
In the last five years or so we’ve seen increased availability of
-home health monitoring devices: ECG recorders, oxygen saturation
-recorders, 24-hour blood pressure monitors, and so on.
-
Some of these devices have user interfaces and displays, and can be
-used in a stand-alone way; some do not. Even if there isn’t a display,
-you might be able to transfer the data using a USB connection or a
-memory stick, and view it using a PC application. On the whole, though,
-only the more expensive, hospital-grade devices can do this.
-
Consumer devices usually have to synchronize with a smartphone app –
-even if you want to use some other application to process the data. I
-don’t really see a viable alternative to a smartphone if you want to use
-these consumer medical devices.
+
In the last five years or so we’ve seen increased availability of home health monitoring devices: ECG recorders, oxygen saturation recorders, 24-hour blood pressure monitors, and so on.
+
Some of these devices have user interfaces and displays, and can be used in a stand-alone way; some do not. Even if there isn’t a display, you might be able to transfer the data using a USB connection or a memory stick, and view it using a PC application. On the whole, though, only the more expensive, hospital-grade devices can do this.
+
Consumer devices usually have to synchronize with a smartphone app – even if you want to use some other application to process the data. I don’t really see a viable alternative to a smartphone if you want to use these consumer medical devices.
Closing remarks
-
As I’ve moved away from using a smartphone, I’ve become aware that
-many of the reasons I used it were compensation for my laziness and lack
-of planning. There are, undoubtedly, some really useful smartphone
-applications – mapping and calendar management are particularly
-significant for me. It would be good if we could have these positive
-things without all the negative ones.
-
At present, though, I don’t think we can. If you use a smartphone,
-you’re buying into the whole package – the good things and the bad.
+
As I’ve moved away from using a smartphone, I’ve become aware that many of the reasons I used it were compensation for my laziness and lack of planning. There are, undoubtedly, some really useful smartphone applications – mapping and calendar management are particularly significant for me. It would be good if we could have these positive things without all the negative ones.
+
At present, though, I don’t think we can. If you use a smartphone, you’re buying into the whole package – the good things and the bad.
diff --git a/nuvi300.html b/nuvi300.html
index f557533..f987522 100644
--- a/nuvi300.html
+++ b/nuvi300.html
@@ -2,8 +2,7 @@
- Kevin Boone: They
-don’t make them like that any more: Garmin Nuvi 300
+ Kevin Boone: They don’t make them like that any more: Garmin Nuvi 300
@@ -41,124 +40,36 @@
-
They
-don’t make them like that any more: Garmin Nuvi 300
-
In the days
-before an automotive satnav unit was just an extension of a smartphone,
-the Garmin Nuvi 300 was the pinnacle of this technology. Garmin’s
-competitors – of which there were, and remain, few – might not have
-agreed with that assessment, but the Nuvi 300 was feature-packed, and
-undeniably practical. The automotive satnav market has been in
-decline ever since.
+
They don’t make them like that any more: Garmin Nuvi 300
+
In the days before an automotive satnav unit was just an extension of a smartphone, the Garmin Nuvi 300 was the pinnacle of this technology. Garmin’s competitors – of which there were, and remain, few – might not have agreed with that assessment, but the Nuvi 300 was feature-packed, and undeniably practical. The automotive satnav market has been in decline ever since.
-
-The Nuvi 300, complete with adjustable,
-flip-up antenna at the rear
+The Nuvi 300, complete with adjustable, flip-up antenna at the rear
-
The Nuvi 300 was released in 2005, and cost about £79. That’s roughly
-£140 in 2024 money. It was cheaper than the competition, but had
-features that no competitor had then, and most don’t have now. Here’s
-just a few.
+
The Nuvi 300 was released in 2005, and cost about £79. That’s roughly £140 in 2024 money. It was cheaper than the competition, but had features that no competitor had then, and most don’t have now. Here’s just a few.
-
It could play music and audiobooks. When playing audio, it would
-pause the playback while speaking directions, so you didn’t miss
-anything. It even supported Audible’s proprietary audiobook format.
-
It had a real audio output jack (bear in mind that the Nuvi 300
-predates Bluetooth). I could plug the Nuvi’s audio output into my car
-stereo, rather than relying on the built-in speaker.
-
Apart from the usual postcode input, the Nuvi could give directions
-to a GB OS grid reference. This is a huge benefit when navigating to a
-tiny car park in the middle of a forest – no postcode.
-
The Nuvi could route around traffic obstructions whose locations
-were broadcast by local radio stations (using RDS/TMC). No smartphone
-app, no subscription. It worked, and was free to use, forever.
-Admittedly, you needed an additional antenna for this.
+
It could play music and audiobooks. When playing audio, it would pause the playback while speaking directions, so you didn’t miss anything. It even supported Audible’s proprietary audiobook format.
+
It had a real audio output jack (bear in mind that the Nuvi 300 predates Bluetooth). I could plug the Nuvi’s audio output into my car stereo, rather than relying on the built-in speaker.
+
Apart from the usual postcode input, the Nuvi could give directions to a GB OS grid reference. This is a huge benefit when navigating to a tiny car park in the middle of a forest – no postcode.
+
The Nuvi could route around traffic obstructions whose locations were broadcast by local radio stations (using RDS/TMC). No smartphone app, no subscription. It worked, and was free to use, forever. Admittedly, you needed an additional antenna for this.
-
You could zoom the map in and out, and rotate it, using finger
-gestures. The built-in satnav in my Volkswagen can’t do that, twenty
-years later.
-
The power supply connection was on the mounting cradle, not the
-satnav itself. It’s hard to overstate how big a deal this was in 2005.
-At that time, it was common for cars to be broken into to just steal a
-satnav – some were that expensive. There was nothing to disconnect on
-the Nuvi – it just unclipped from its cradle.
-
The Nuvi had a bicycle mode, which changed the routing algorithm to
-favour minor roads, but still avoid footpaths. Weirdly, although Garmin
-advertised this as a feature, they never supplied a bicycle or
-motorcycle mount, so far as I know, and the Nuvi wasn’t waterproof.
-Although a niche feature, I did actually use my Nuvi 300 for cycling
-from time to time.
+
You could zoom the map in and out, and rotate it, using finger gestures. The built-in satnav in my Volkswagen can’t do that, twenty years later.
+
The power supply connection was on the mounting cradle, not the satnav itself. It’s hard to overstate how big a deal this was in 2005. At that time, it was common for cars to be broken into to just steal a satnav – some were that expensive. There was nothing to disconnect on the Nuvi – it just unclipped from its cradle.
+
The Nuvi had a bicycle mode, which changed the routing algorithm to favour minor roads, but still avoid footpaths. Weirdly, although Garmin advertised this as a feature, they never supplied a bicycle or motorcycle mount, so far as I know, and the Nuvi wasn’t waterproof. Although a niche feature, I did actually use my Nuvi 300 for cycling from time to time.
-
It’s interesting to compare these features with contemporary
-automotive satnav devices. In general, modern devices fall short in
-important areas, despite twenty years of technological development.
-
Modern stand-alone satnav devices generally don’t have audio outputs,
-although you might be able to pair them with a Bluetooth-equipped car
-radio.
-
That’s not as important as it used to be, though, because they
-usually don’t play music or audiobooks. Garmin removed this feature
-because they claimed it to be a safety hazard. So now I have to play
-music from my smartphone, which is a safety hazard, not to
-mention illegal, if I’m moving. So I don’t find this argument
-compelling.
-
To remove a modern device from the car, you have to fiddle about with
-a power connector on the unit. And leave a dangling cable on the
-dashboard.
-
Worst of all, I can’t find a modern satnav that supports OS grid
-references. To navigate to that tiny car park in the hills, I have to
-convert the grid reference to latitude and longitude. What a drag.
-
Many modern devices don’t support RDS/TMC routing. Some can get
-traffic information from a paired smartphone. But this requires a
-specific app, which could be doing anything it likes with your location
-data; and, of course, it requires you to be carrying a smartphone. And
-to be within signal range.
-
Of course, there’s a lot that a modern device can do, that would have
-been impractical twenty years ago. To update the maps on my Nuvi 300, I
-had to copy them to an SD card, using proprietary Garmin software.
-Modern devices can download maps using Wifi or 4G. Having said that, I
-have to admin that to update the maps on my car’s built-in satnav, I
-have to take it to a Volkswagen dealer. Frankly, I don’t find the modern
-developments in this area to be particularly useful, compared to what
-has been lost.
-
I wonder sometimes: why is it that so many consumer electronics
-manufacturers discard useful features as technology develops? Why don’t
-mobile phones have a headphone jack any more? Why do we need a stack of
-obscure tools to replace the hard disk in a modern laptop, when my W520
-just has a sliding lever that pops the disk out? Why does my
-stupidly-expensive Astell and Kern media player not have FM radio
-support, when my 2005 Cowon does?
-
Why, in short, does technological advance not make consumer devices
-better?
-
Part of the problem, I guess, is that pricing is so competitive, that
-every feature, however cheap, has to justify itself on cost grounds.
-Nobody wants to pay much for anything. In fact, nobody wants to pay
-anything, which is the modern world is so choked with intrusive
-advertising.
-
Only a fraction of satnav owners want to navigate to a grid
-reference, or play audiobooks, and it costs money to maintain these
-features. Similarly, supplying a powered mounting cradle requires
-specific design and tooling, compared to a passive cradle and a dangling
-cable. Maybe this feature would only add fifty pence to the sale price
-of the unit but, in a market where price is everything, that’s still
-fifty pence too much for many people.
-
The problem with this argument was that, back in 2005, the Nuvi 300
-was cheaper than anything else on the market, as well as being
-superior. Conceivably Garmin deliberately under-priced the unit, to
-encourage people away from its main competitor (Tomtom, in the UK).
-
Whatever the explanation, the Garmin Nuvi 300 was the first
-automotive satnav I owned and, at one time, I owned three of them. I
-still have one on my quad-bike. They all still work, although it’s hard
-– but not impossible – to update the maps. The only reason I’m not still
-using them in my car is that cars have satnav built in these days. The
-built-in satnav is rubbish, but not so bad that I feel the need to
-install another one.
-
And, of course, many people – perhaps most people – use smartphone
-navigation apps these days. Smartphones are completely inappropriate for
-vehicular navigation, not to mention catastrophic for personal privacy,
-but they come with the advantage that you probably already have one. So,
-in a choice between using something inappropriate, and paying for
-something better, many people will opt for the inappropriate.
+
It’s interesting to compare these features with contemporary automotive satnav devices. In general, modern devices fall short in important areas, despite twenty years of technological development.
+
Modern stand-alone satnav devices generally don’t have audio outputs, although you might be able to pair them with a Bluetooth-equipped car radio.
+
That’s not as important as it used to be, though, because they usually don’t play music or audiobooks. Garmin removed this feature because they claimed it to be a safety hazard. So now I have to play music from my smartphone, which is a safety hazard, not to mention illegal, if I’m moving. So I don’t find this argument compelling.
+
To remove a modern device from the car, you have to fiddle about with a power connector on the unit. And leave a dangling cable on the dashboard.
+
Worst of all, I can’t find a modern satnav that supports OS grid references. To navigate to that tiny car park in the hills, I have to convert the grid reference to latitude and longitude. What a drag.
+
Many modern devices don’t support RDS/TMC routing. Some can get traffic information from a paired smartphone. But this requires a specific app, which could be doing anything it likes with your location data; and, of course, it requires you to be carrying a smartphone. And to be within signal range.
+
Of course, there’s a lot that a modern device can do, that would have been impractical twenty years ago. To update the maps on my Nuvi 300, I had to copy them to an SD card, using proprietary Garmin software. Modern devices can download maps using Wifi or 4G. Having said that, I have to admin that to update the maps on my car’s built-in satnav, I have to take it to a Volkswagen dealer. Frankly, I don’t find the modern developments in this area to be particularly useful, compared to what has been lost.
+
I wonder sometimes: why is it that so many consumer electronics manufacturers discard useful features as technology develops? Why don’t mobile phones have a headphone jack any more? Why do we need a stack of obscure tools to replace the hard disk in a modern laptop, when my W520 just has a sliding lever that pops the disk out? Why does my stupidly-expensive Astell and Kern media player not have FM radio support, when my 2005 Cowon does?
+
Why, in short, does technological advance not make consumer devices better?
+
Part of the problem, I guess, is that pricing is so competitive, that every feature, however cheap, has to justify itself on cost grounds. Nobody wants to pay much for anything. In fact, nobody wants to pay anything, which is the modern world is so choked with intrusive advertising.
+
Only a fraction of satnav owners want to navigate to a grid reference, or play audiobooks, and it costs money to maintain these features. Similarly, supplying a powered mounting cradle requires specific design and tooling, compared to a passive cradle and a dangling cable. Maybe this feature would only add fifty pence to the sale price of the unit but, in a market where price is everything, that’s still fifty pence too much for many people.
+
The problem with this argument was that, back in 2005, the Nuvi 300 was cheaper than anything else on the market, as well as being superior. Conceivably Garmin deliberately under-priced the unit, to encourage people away from its main competitor (Tomtom, in the UK).
+
Whatever the explanation, the Garmin Nuvi 300 was the first automotive satnav I owned and, at one time, I owned three of them. I still have one on my quad-bike. They all still work, although it’s hard – but not impossible – to update the maps. The only reason I’m not still using them in my car is that cars have satnav built in these days. The built-in satnav is rubbish, but not so bad that I feel the need to install another one.
+
And, of course, many people – perhaps most people – use smartphone navigation apps these days. Smartphones are completely inappropriate for vehicular navigation, not to mention catastrophic for personal privacy, but they come with the advantage that you probably already have one. So, in a choice between using something inappropriate, and paying for something better, many people will opt for the inappropriate.
diff --git a/phone_media_player.html b/phone_media_player.html
index b27a57b..eda6725 100644
--- a/phone_media_player.html
+++ b/phone_media_player.html
@@ -2,8 +2,7 @@
- Kevin Boone: Some
-thoughts on using a small cellphone as a media player
+ Kevin Boone: Some thoughts on using a small cellphone as a media player
@@ -41,163 +40,49 @@
-
Some
-thoughts on using a small cellphone as a media player
+
Some thoughts on using a small cellphone as a media player
-
Of course, it’s easy to use any modern cellphone as a media player –
-just use any media player app. Job done.
-
This post, however, is not about using your day-to-day phone as a
-media player – it’s about customizing a cellphone so that its primary
-use is as a media player, and not using it for telephony or
-communications at all.
-
For most of us, there’s really no good reason to do that. If you’re
-already carrying a cellphone around, why would you want a second one as
-a media player? The only answer I can think of is that, like me, you
-don’t want to use a smartphone at all – but still want a
-high-tech media player.
-
Last year I divorced Google from my life.
-I’m wondering now – do I really need a smartphone at all? Or
-could I manage with a Nokia brick, or its modern equivalent? There are
-certain advantages to this:
+
Of course, it’s easy to use any modern cellphone as a media player – just use any media player app. Job done.
+
This post, however, is not about using your day-to-day phone as a media player – it’s about customizing a cellphone so that its primary use is as a media player, and not using it for telephony or communications at all.
+
For most of us, there’s really no good reason to do that. If you’re already carrying a cellphone around, why would you want a second one as a media player? The only answer I can think of is that, like me, you don’t want to use a smartphone at all – but still want a high-tech media player.
+
Last year I divorced Google from my life. I’m wondering now – do I really need a smartphone at all? Or could I manage with a Nokia brick, or its modern equivalent? There are certain advantages to this:
No creepy tracking by apps
Respectable battery life – weeks rather than hours
Fewer distractions
-
I would really only miss two things if I dumped my smartphone: Google
-Maps (and I’m already at stage six of the seven stages of grief over
-that), and the media player. I use my phone for music and audiobooks all
-the time, and occasionally for video.
-
At one time there was a huge range of portable media players to
-choose from – I should know: I’ve bought most of them. These days,
-portable media players are a speciality item – they’re expensive, and
-not very portable. And most run Android anyway, which makes them
-essentially a smartphone, but without the antenna.
+
I would really only miss two things if I dumped my smartphone: Google Maps (and I’m already at stage six of the seven stages of grief over that), and the media player. I use my phone for music and audiobooks all the time, and occasionally for video.
+
At one time there was a huge range of portable media players to choose from – I should know: I’ve bought most of them. These days, portable media players are a speciality item – they’re expensive, and not very portable. And most run Android anyway, which makes them essentially a smartphone, but without the antenna.
So why not repurpose a small cellphone as a media player?
-
If I dedicate a phone to be only a media player, I get certain
-advantages over playing media on my main phone:
+
If I dedicate a phone to be only a media player, I get certain advantages over playing media on my main phone:
-
With a bit of effort, I can remove unnecessary bloat and increase
-battery life. I can even remove distractions like the web browser, if I
-really want to.
-
I can use a really small phone: my choice is the Samsung S10e which,
-to be frank, is too small to use as a regular phone. But it’s a great
-size for a pocket media player.
-
I don’t always have to carry it with me. In practice, I
-usually do; but I don’t have to.
+
With a bit of effort, I can remove unnecessary bloat and increase battery life. I can even remove distractions like the web browser, if I really want to.
+
I can use a really small phone: my choice is the Samsung S10e which, to be frank, is too small to use as a regular phone. But it’s a great size for a pocket media player.
+
I don’t always have to carry it with me. In practice, I usually do; but I don’t have to.
I don’t need to leave it switched on when I’m not using it.
-
I can disable the cellular antenna by leaving it in ‘airplane mode’,
-further increasing the battery life.
+
I can disable the cellular antenna by leaving it in ‘airplane mode’, further increasing the battery life.
-
If you remove the SIM card from a cellphone, doesn’t that by itself
-essentially convert it to a WiFi-only device, suitable for little more
-than media? Well, yes and no. ‘Yes’, because that prevents its being
-used for telephony or roaming Internet access. This in turn removes much
-of the motivation for installing distracting apps on it. But also ‘no’,
-because it’s still full of vendor bloatware, creepy spyware, and other
-power-draining rubbish. Moreover, so long as I’ve still got access to
-the Google Play store, I’ve got a way to reinstall any of the
-time-wasting apps that I want to remove from my life.
-
So my first step in dedicating a phone to be a media player is to
-install a non-commercial operating system on it. Currently I favour
-LineageOS, but there are others. All these platforms are more-or-less
-free of bloat, and this alone radically increases battery life.
-
Even better, LineageOS starts from a shut-down in about 25 seconds on
-my S10e, compared with a minute with the stock Samsung firmware. So it’s
-practicable to turn it off when I’m not using it for media. Even if I
-don’t turn it off when I’m not using it, I still get six days of light
-use – the stock firmware requires me to charge every day, whether I’m
-using the phone or not.
-
Because there’s no SIM card, and no telephony, I can leave the device
-in ‘airplane mode’ all the time. Although this disables the cellular
-antenna, and the power drain that accompanies it, it doesn’t
-prevent me using Bluetooth and WiFi if I want. Moreover, the ‘airplane
-mode’ setting is persistent across restarts, so I don’t have to fiddle
-with it.
-
For a really distraction-free experience, I needed to remove the web
-browser. That’s not easy – I can disable it in various ways, but
-reversing the disablement is too easy. To remove the browser completely,
-I need to root the phone, and use a root-only package remover. Rooting
-is easy enough using a tool like Magisk, but it’s very easy to break the
-device completely, messing about with system packages.
-
But having made the decision to root the phone and do
-potentially-destructive things to it, I can remove other unnecessary
-time-wasters while I’m about it (although there are few of these in
-LineageOS). If I want, I can remove the telephony subsystem completely.
-This saves a bit of memory, but it takes away the ability even to make
-emergency calls. I’m not entirely convinced that I want to go that far,
-although I’ve removed all the SMS messaging stuff.
-
There are a few other things that can be done, to improve battery
-life. The S10e has an OLED screen, so the screen’s power usage is
-directly proportional to the amount of colour displayed. I’ve enabled
-the ‘dark’ theme generally, and I’m using a plain black image as the
-wallpaper. I’ve set the screen timeout to five seconds. I’ve turned off
-all the things that vibrate. I usually leave WiFi and Bluetooth off,
-unless I’m actually using them. Same for location services – I rarely
-need GPS on a media player (but it’s occasionally useful; I wouldn’t
-remove it completely).
+
If you remove the SIM card from a cellphone, doesn’t that by itself essentially convert it to a WiFi-only device, suitable for little more than media? Well, yes and no. ‘Yes’, because that prevents its being used for telephony or roaming Internet access. This in turn removes much of the motivation for installing distracting apps on it. But also ‘no’, because it’s still full of vendor bloatware, creepy spyware, and other power-draining rubbish. Moreover, so long as I’ve still got access to the Google Play store, I’ve got a way to reinstall any of the time-wasting apps that I want to remove from my life.
+
So my first step in dedicating a phone to be a media player is to install a non-commercial operating system on it. Currently I favour LineageOS, but there are others. All these platforms are more-or-less free of bloat, and this alone radically increases battery life.
+
Even better, LineageOS starts from a shut-down in about 25 seconds on my S10e, compared with a minute with the stock Samsung firmware. So it’s practicable to turn it off when I’m not using it for media. Even if I don’t turn it off when I’m not using it, I still get six days of light use – the stock firmware requires me to charge every day, whether I’m using the phone or not.
+
Because there’s no SIM card, and no telephony, I can leave the device in ‘airplane mode’ all the time. Although this disables the cellular antenna, and the power drain that accompanies it, it doesn’t prevent me using Bluetooth and WiFi if I want. Moreover, the ‘airplane mode’ setting is persistent across restarts, so I don’t have to fiddle with it.
+
For a really distraction-free experience, I needed to remove the web browser. That’s not easy – I can disable it in various ways, but reversing the disablement is too easy. To remove the browser completely, I need to root the phone, and use a root-only package remover. Rooting is easy enough using a tool like Magisk, but it’s very easy to break the device completely, messing about with system packages.
+
But having made the decision to root the phone and do potentially-destructive things to it, I can remove other unnecessary time-wasters while I’m about it (although there are few of these in LineageOS). If I want, I can remove the telephony subsystem completely. This saves a bit of memory, but it takes away the ability even to make emergency calls. I’m not entirely convinced that I want to go that far, although I’ve removed all the SMS messaging stuff.
+
There are a few other things that can be done, to improve battery life. The S10e has an OLED screen, so the screen’s power usage is directly proportional to the amount of colour displayed. I’ve enabled the ‘dark’ theme generally, and I’m using a plain black image as the wallpaper. I’ve set the screen timeout to five seconds. I’ve turned off all the things that vibrate. I usually leave WiFi and Bluetooth off, unless I’m actually using them. Same for location services – I rarely need GPS on a media player (but it’s occasionally useful; I wouldn’t remove it completely).
-
-There’s a thing you don’t often see: two
-weeks’ battery remaining
+There’s a thing you don’t often see: two weeks’ battery remaining
-
With these changes I get about six days of light usage between
-charges. The standby time can be measured in weeks. But I find that I
-don’t need even that much battery endurance, because I only switch the
-device on when I’m using it (which, to be fair, is quite a lot).
-
I have thousands of music albums and audiobooks in FLAC format, and
-playing them is no problem. I can use VLC (which is open source, and
-available from F-Droid), or the built-in media player. But what about
-streaming services?
-
This is where things get a bit sketchy. Companies like Spotify do not
-distribute their apps in APK format, suitable for side-loading onto a
-device. They aren’t on F-Droid, either – they wouldn’t be accepted.
-There’s no reason why the vendors
-shouldn’t supply APK files – their revenue comes from subscriptions, not
-app sales. Nevertheless, if you want to use Spotify, Qobuz, et al.,
-you’re going to be fishing for an APK in murky waters. I wouldn’t do
-that on a phone that I relied on, or that contained any personal data.
-However, I’m not storing anything personal on my media player. I haven’t
-even enabled a lock screen.
-
Still, it irks me that I can’t use USB Audio Player Pro – my
-favourite music player – even though I’ve paid for it. Without Google
-Play services, there’s no way to use this app. And I’m sure there are
-many other apps and services that I wouldn’t be able to use – but I
-don’t know what they are, because I don’t use them.
+
With these changes I get about six days of light usage between charges. The standby time can be measured in weeks. But I find that I don’t need even that much battery endurance, because I only switch the device on when I’m using it (which, to be fair, is quite a lot).
+
I have thousands of music albums and audiobooks in FLAC format, and playing them is no problem. I can use VLC (which is open source, and available from F-Droid), or the built-in media player. But what about streaming services?
+
This is where things get a bit sketchy. Companies like Spotify do not distribute their apps in APK format, suitable for side-loading onto a device. They aren’t on F-Droid, either – they wouldn’t be accepted. There’s no reason why the vendors
+shouldn’t supply APK files – their revenue comes from subscriptions, not app sales. Nevertheless, if you want to use Spotify, Qobuz, et al., you’re going to be fishing for an APK in murky waters. I wouldn’t do that on a phone that I relied on, or that contained any personal data. However, I’m not storing anything personal on my media player. I haven’t even enabled a lock screen.
+
Still, it irks me that I can’t use USB Audio Player Pro – my favourite music player – even though I’ve paid for it. Without Google Play services, there’s no way to use this app. And I’m sure there are many other apps and services that I wouldn’t be able to use – but I don’t know what they are, because I don’t use them.
So is it worth it?
-
I’ve already de-googled my main phone, which means that I was already
-getting many of the advantages I’ve suggested in this article. But I
-still got the constant distractions and time-wasting that the smartphone
-provided, with or without Google and its ready access to apps. I found I
-was constantly checking email, looking a websites, and all the usual
-stuff. And I still had something that had to be charged every day or two
-at most – and it had to be kept charged, because people called me and
-messaged me. And it was comparatively heavy, and would have been
-expensive to lose.
-
The de-googling process is complicated enough but, if you want to go
-further by removing the web browser, that’s an additional complexity –
-and a risky one. But an S10e can be had second-hand for about fifty
-quid, so the risk is a comparatively small one, even if you break
-it.
-
The trend with smartphones is to make them larger and larger. I’m not
-sure there is much, if anything, currently on the market as small as
-Samsung’s S10e. Of course you can dedicate a large phone as a media
-player, but the modest size of the S10e is part of what makes it
-appealing in such a role.
-
I find that the de-googled, de-bloated, SIM-less S10e makes a pretty
-good media player. It’s not much bigger than, say, the 7th-generation
-iPod Touch, which was very popular in its day. Start-up time does not
-compare well with portable media players of ten years ago, but it’s
-actually faster to start than my Astell & Kern player – and a lot
-more portable. Sound quality is… OK. It’s better with an external DAC,
-but then it’s no longer a pocket device.
-
In the end the question isn’t really whether a small smartphone makes
-a decent media player – clearly it does. The important question is
-whether you actually need one, and the answer will only be ‘yes’ if you
-don’t intend to use a smartphone as a regular phone. I am so far only
-experimenting with this self-deprivation, and I’m not sure how it will
-work out in the long term.
+
I’ve already de-googled my main phone, which means that I was already getting many of the advantages I’ve suggested in this article. But I still got the constant distractions and time-wasting that the smartphone provided, with or without Google and its ready access to apps. I found I was constantly checking email, looking a websites, and all the usual stuff. And I still had something that had to be charged every day or two at most – and it had to be kept charged, because people called me and messaged me. And it was comparatively heavy, and would have been expensive to lose.
+
The de-googling process is complicated enough but, if you want to go further by removing the web browser, that’s an additional complexity – and a risky one. But an S10e can be had second-hand for about fifty quid, so the risk is a comparatively small one, even if you break it.
+
The trend with smartphones is to make them larger and larger. I’m not sure there is much, if anything, currently on the market as small as Samsung’s S10e. Of course you can dedicate a large phone as a media player, but the modest size of the S10e is part of what makes it appealing in such a role.
+
I find that the de-googled, de-bloated, SIM-less S10e makes a pretty good media player. It’s not much bigger than, say, the 7th-generation iPod Touch, which was very popular in its day. Start-up time does not compare well with portable media players of ten years ago, but it’s actually faster to start than my Astell & Kern player – and a lot more portable. Sound quality is… OK. It’s better with an external DAC, but then it’s no longer a pocket device.
+
In the end the question isn’t really whether a small smartphone makes a decent media player – clearly it does. The important question is whether you actually need one, and the answer will only be ‘yes’ if you don’t intend to use a smartphone as a regular phone. I am so far only experimenting with this self-deprivation, and I’m not sure how it will work out in the long term.
There's been a recent scare that Unicode reading direction characters could be used to conceal malicious code in open-source projects. This is undoubtedly true, but that fact doesn't make it significantly harder to ensure the security of open-source code than it already is.
Juice-jacking is the alleged practice of getting unauthorized access to the contents of a cellphone by subverting public USB charging points. It doesn't happen, and probably never has; so why has there been a recent increase in scare stories?
diff --git a/tpwin.html b/tpwin.html
index 483cbfb..7c3a129 100644
--- a/tpwin.html
+++ b/tpwin.html
@@ -2,8 +2,7 @@
- Kevin Boone: They
-don’t make ’em like that any more: Borland Turbo Pascal 7
+ Kevin Boone: They don’t make ’em like that any more: Borland Turbo Pascal 7
@@ -41,167 +40,31 @@
-
They
-don’t make ’em like that any more: Borland Turbo Pascal 7
+
They don’t make ’em like that any more: Borland Turbo Pascal 7
-
Borland’s Turbo Pascal was one of the most successful
-programming tools of all time. It introduced development techniques,
-such as integrated source-level debugging, that we take for granted
-today; but which were radical at the time.
-
TP 7, also known as Turbo Pascal for Windows 1.5, was the
-pinnacle of the Turbo Pascal line, and its end. The name ‘Turbo’ was
-associated with what were regarded as hobbyist products – ‘Borland
-Pascal’ was a different, much more expensive, product. But Turbo Pascal
-was enough for some quite substantial applications, and was very
-affordable. It was also ‘book licensed’, which I think was a new thing
-in the software industry. You could buy one copy of TP, and any number
-of people could use it – but only one at a time. Even when it was
-distributed on floppy disks, there was no copy protection. (Turbo)
-Pascal eventually gave way to Delphi which, despite its technical
-merits, never had the same impact.
-
I used TP 7 to write medical imaging software back in the 90s, to run
-under Windows 3.1 – which was a great operating system in its day. Like
-previous releases, Windows 3.1 was still, essentially, an extension to
-MSDOS, but with clever tricks to allow it to multitask (after a fashion)
-and address more memory. It had a simple, consistent user interface
-model – both for the user and the programmer. It had, at that time, none
-of the wasteful and irritating anti-features that have come to make
-modern Windows so depressing to use. So Windows 3.1 was a platform we
-all wanted our applications to support – not that there were many
-alternatives at that time.
-
Like many other programmers, I learned the Pascal programming
-language specifically to use TP 7 – I was essentially a C programmer in
-those days, but programming for Windows in C was a horrible job.
-Frankly, it’s scarcely better today. It was, and is, necessary to
-combine a whole stack of command-line tools that take incomprehensible
-text files as input. TP 7 automated and concealed all that
-nastiness.
-
It wasn’t just the tools that were a problem: the user interface
-model was, essentially, object-oriented; but C did not make it easy to
-program in an object-oriented way. TP 5 had introduced basic
-object-oriented concepts to the Pascal language, and these were were to
-prove essential. The Windows-specific features were provided by
-something called Object Windows Library: a relatively thick,
-object-oriented wrapper around the basic Windows C API. My recollection
-is that Borland introduced an object-oriented programming library for
-Microsoft Windows before Microsoft did – and Borland’s was better.
+
Borland’s Turbo Pascal was one of the most successful programming tools of all time. It introduced development techniques, such as integrated source-level debugging, that we take for granted today; but which were radical at the time.
+
TP 7, also known as Turbo Pascal for Windows 1.5, was the pinnacle of the Turbo Pascal line, and its end. The name ‘Turbo’ was associated with what were regarded as hobbyist products – ‘Borland Pascal’ was a different, much more expensive, product. But Turbo Pascal was enough for some quite substantial applications, and was very affordable. It was also ‘book licensed’, which I think was a new thing in the software industry. You could buy one copy of TP, and any number of people could use it – but only one at a time. Even when it was distributed on floppy disks, there was no copy protection. (Turbo) Pascal eventually gave way to Delphi which, despite its technical merits, never had the same impact.
+
I used TP 7 to write medical imaging software back in the 90s, to run under Windows 3.1 – which was a great operating system in its day. Like previous releases, Windows 3.1 was still, essentially, an extension to MSDOS, but with clever tricks to allow it to multitask (after a fashion) and address more memory. It had a simple, consistent user interface model – both for the user and the programmer. It had, at that time, none of the wasteful and irritating anti-features that have come to make modern Windows so depressing to use. So Windows 3.1 was a platform we all wanted our applications to support – not that there were many alternatives at that time.
+
Like many other programmers, I learned the Pascal programming language specifically to use TP 7 – I was essentially a C programmer in those days, but programming for Windows in C was a horrible job. Frankly, it’s scarcely better today. It was, and is, necessary to combine a whole stack of command-line tools that take incomprehensible text files as input. TP 7 automated and concealed all that nastiness.
+
It wasn’t just the tools that were a problem: the user interface model was, essentially, object-oriented; but C did not make it easy to program in an object-oriented way. TP 5 had introduced basic object-oriented concepts to the Pascal language, and these were were to prove essential. The Windows-specific features were provided by something called Object Windows Library: a relatively thick, object-oriented wrapper around the basic Windows C API. My recollection is that Borland introduced an object-oriented programming library for Microsoft Windows before Microsoft did – and Borland’s was better.
-
-Turbo Pascal running in Windows
-3.1
+Turbo Pascal running in Windows 3.1
-
OWL made it easy to create nice-looking user interfaces, with a bit
-of eye candy. The familiar ‘green tick button’ was an OWL enhancement,
-which graced so many Windows applications that it eventually became an
-irritating cliche.
-
All, in all, It was much easier to program for Windows using
-Turbo Pascal 7 than with anything else. Not only did it provide a
-programming model that matched the way the Windows user interface
-worked, the application itself had a Windows graphical interface – many
-Windows programming tools at that time actually ran under MSDOS, and
-were entirely text-based. TP 7 also had fully-graphical tools for
-designing the user interface elements, like menus and icons. Laying out
-a menu using a definition file with an obscure format, using Windows
-Notepad, was never an agreeable experience. Microsoft did produce
-graphical tools for this kind of operation, but Turbo Pascal combined
-them into a seamless IDE. All I had to do to build and run my programs
-was to hit the F7 key. I could even set breakpoints for the debugger,
-just by clicking a line of code. As I said, common enough today, but
-revolutionary for early Windows programming.
-
But what really made TP 7 special was its CP/M heritage. Turbo Pascal
-came to Windows by way of CP/M and MSDOS, which meant it had a compiler
-core that was designed to run with tight memory and a comparatively slow
-CPU. In the days when most compilers had to write a stack of
-intermediate files on disk, TP just constructed everything in memory. It
-did this using highly-optimized code, written in assembly language, by
-very skilled people. It doesn’t matter much today if your compiler spits
-out a heap of temporary files, because disk storage is fast and cheap.
-But in the days of the floppy disk, you had to minimize disk access.
-Unfortunately, you also had to minimize memory use, because
-there wasn’t much of that, either. Programming for CP/M took real skill
-and dedication.
-
So when TP came to Windows machines, with their 16-bit CPUs and
-megabytes of RAM, it was like an early Christmas for developers. A
-compiler designed to be adequate with 32kB of RAM and a single floppy
-drive will positively fly when given megabytes and a hard disk.
-
All this meant that TP 7 was a joy to use. It was like a modern IDE,
-but without any of the bloat. Developers who would never have tackled
-Windows programming with Microsoft tools were able to move from MSDOS to
-Windows without too much difficulty – even if they first had to learn
-Pascal, as I did. The compiled code was pretty speedy but, if it wasn’t
-brisk enough, you could write assembly language directly in the Pascal
-source files.
-
I don’t think it’s too much of an exaggeration to claim that Turbo
-Pascal contributed to the widespread uptake, and eventual domination, of
-Microsoft Windows on desktop PCs. If there had been a ‘Turbo Pascal for
-Linux’ in the early 90s, then we might all be running Linux on
-our desktops today.
-
So what happened to Turbo Pascal? Three things, at least, contributed
-to its eventual downfall.
-
First, there was increasing competition from the C programming
-language. Borland did produce a Turbo C, and later a Turbo C++, but they
-weren’t as influential as Pascal. It’s easy to forget that PCs weren’t
-the first general-purpose computing devices. Most scientific and
-educational establishments, and many businesses, used graphical
-workstations and a central minicomputer. C was already well-established
-in these environments before desktop computers started to become
-prominent. And before that we had mainframes and terminals – but that
-was a different world, where Fortran and COBOL ruled.
-
Most ‘serious’ PC applications were written in assembly language in
-the early days, as they had been for CP/M – the low resources almost
-necessitated this. Minicomputers were not so limited, so compilers were
-more practicable, even in the 1980s. So when compilers started to become
-of interest to microcomputer developers, there was almost no reason to
-favour one programming language over another. Pascal could have
-become the standard programming language for the PC. That it did not, I
-think, reflects the fact that many software companies that had been
-active in industrial and scientific computing saw an opportunity in PCs,
-and tried to move into that market. These companies already had
-substantial experience with C, and they brought their preferences with
-them. So while I was happy to learn a new programming language to
-develop for Windows, a software business with a thousand staff might not
-have wanted to take that step.
-
The second factor, I think, was the increasing palatability of
-Microsoft’s C tools. Microsoft did eventually produce an IDE of its own
-and, of course, Microsoft controlled the Windows platform. I don’t think
-Microsoft’s C IDEs ever became as easy to use as Borland’s, but
-they probably didn’t have to, given Microsoft’s position.
-
Third, Microsoft’s Visual BASIC started to attact small-scale and
-bespoke developers. BASIC was already a staple of the microcomputing
-world, because it was easy to learn, and did not require a separate
-compilation step. Attaching bits of BASIC code to Windows user interface
-elements was an efficient way to produce relatively simple, small-volume
-applications.
-
I loathed Visual BASIC in the 90s, and I loathed it until the day VB6
-drew its last laboured, wheezing breath in 2005. Still, I can’t deny its
-popularity, and I’m sure that many of the same people who were drawn to
-Turbo Pascal were later drawn to Visual BASIC, for much the same
-reasons. VB lowered the barriers to entry into Windows programming in
-the same way that TP had previously done. Arguably it lowered them too
-much, but let’s not go there today.
-
Early versions of Turbo Pascal are now freely, and legally,
-available. Retrocomputing enthusiasts with an interest in early PCs and
-MSDOS are well catered for. So far as I know, however, TP 7 remains
-somebody’s intellectual property, although it isn’t entirely clear
-whose. It probably comes into the category of ‘abandonware’ but, to be
-frank, it’s not all that important: it can’t produce code that runs on
-modern Windows systems, and few people have much interest in early
-Windows – it’s just not retro enough.
-
So that’s Turbo Pascal: a colossus of the microcomputing world, which
-faded and died largely unlamented. It always was a creature of its time
-– it prospered for the brief period when microcomputers were limited
-enough to reward skilled development, but widespread enough to create a
-substantial market for programmers. By the time it started to become
-difficult to find enough highly-skilled programmers to satisfy the
-market needs, it didn’t matter: computers were getting powerful enough
-to make good our limitations as programmers. That almost nobody has the
-skills these days to implement something like TP is something we could
-eventually come to regret – or, rather, our grandchildren could.
-Software development is becoming a semi-skilled industry, and I doubt
-that the rise of AI will entirely compensate for this.
-
But who needs a Turbo Pascal, when you have 128Gb of RAM attached to
-an 5GHz CPU, for a hastily-written, bloated Java program to burn?
+
OWL made it easy to create nice-looking user interfaces, with a bit of eye candy. The familiar ‘green tick button’ was an OWL enhancement, which graced so many Windows applications that it eventually became an irritating cliche.
+
All, in all, It was much easier to program for Windows using Turbo Pascal 7 than with anything else. Not only did it provide a programming model that matched the way the Windows user interface worked, the application itself had a Windows graphical interface – many Windows programming tools at that time actually ran under MSDOS, and were entirely text-based. TP 7 also had fully-graphical tools for designing the user interface elements, like menus and icons. Laying out a menu using a definition file with an obscure format, using Windows Notepad, was never an agreeable experience. Microsoft did produce graphical tools for this kind of operation, but Turbo Pascal combined them into a seamless IDE. All I had to do to build and run my programs was to hit the F7 key. I could even set breakpoints for the debugger, just by clicking a line of code. As I said, common enough today, but revolutionary for early Windows programming.
+
But what really made TP 7 special was its CP/M heritage. Turbo Pascal came to Windows by way of CP/M and MSDOS, which meant it had a compiler core that was designed to run with tight memory and a comparatively slow CPU. In the days when most compilers had to write a stack of intermediate files on disk, TP just constructed everything in memory. It did this using highly-optimized code, written in assembly language, by very skilled people. It doesn’t matter much today if your compiler spits out a heap of temporary files, because disk storage is fast and cheap. But in the days of the floppy disk, you had to minimize disk access. Unfortunately, you also had to minimize memory use, because there wasn’t much of that, either. Programming for CP/M took real skill and dedication.
+
So when TP came to Windows machines, with their 16-bit CPUs and megabytes of RAM, it was like an early Christmas for developers. A compiler designed to be adequate with 32kB of RAM and a single floppy drive will positively fly when given megabytes and a hard disk.
+
All this meant that TP 7 was a joy to use. It was like a modern IDE, but without any of the bloat. Developers who would never have tackled Windows programming with Microsoft tools were able to move from MSDOS to Windows without too much difficulty – even if they first had to learn Pascal, as I did. The compiled code was pretty speedy but, if it wasn’t brisk enough, you could write assembly language directly in the Pascal source files.
+
I don’t think it’s too much of an exaggeration to claim that Turbo Pascal contributed to the widespread uptake, and eventual domination, of Microsoft Windows on desktop PCs. If there had been a ‘Turbo Pascal for Linux’ in the early 90s, then we might all be running Linux on our desktops today.
+
So what happened to Turbo Pascal? Three things, at least, contributed to its eventual downfall.
+
First, there was increasing competition from the C programming language. Borland did produce a Turbo C, and later a Turbo C++, but they weren’t as influential as Pascal. It’s easy to forget that PCs weren’t the first general-purpose computing devices. Most scientific and educational establishments, and many businesses, used graphical workstations and a central minicomputer. C was already well-established in these environments before desktop computers started to become prominent. And before that we had mainframes and terminals – but that was a different world, where Fortran and COBOL ruled.
+
Most ‘serious’ PC applications were written in assembly language in the early days, as they had been for CP/M – the low resources almost necessitated this. Minicomputers were not so limited, so compilers were more practicable, even in the 1980s. So when compilers started to become of interest to microcomputer developers, there was almost no reason to favour one programming language over another. Pascal could have become the standard programming language for the PC. That it did not, I think, reflects the fact that many software companies that had been active in industrial and scientific computing saw an opportunity in PCs, and tried to move into that market. These companies already had substantial experience with C, and they brought their preferences with them. So while I was happy to learn a new programming language to develop for Windows, a software business with a thousand staff might not have wanted to take that step.
+
The second factor, I think, was the increasing palatability of Microsoft’s C tools. Microsoft did eventually produce an IDE of its own and, of course, Microsoft controlled the Windows platform. I don’t think Microsoft’s C IDEs ever became as easy to use as Borland’s, but they probably didn’t have to, given Microsoft’s position.
+
Third, Microsoft’s Visual BASIC started to attact small-scale and bespoke developers. BASIC was already a staple of the microcomputing world, because it was easy to learn, and did not require a separate compilation step. Attaching bits of BASIC code to Windows user interface elements was an efficient way to produce relatively simple, small-volume applications.
+
I loathed Visual BASIC in the 90s, and I loathed it until the day VB6 drew its last laboured, wheezing breath in 2005. Still, I can’t deny its popularity, and I’m sure that many of the same people who were drawn to Turbo Pascal were later drawn to Visual BASIC, for much the same reasons. VB lowered the barriers to entry into Windows programming in the same way that TP had previously done. Arguably it lowered them too much, but let’s not go there today.
+
Early versions of Turbo Pascal are now freely, and legally, available. Retrocomputing enthusiasts with an interest in early PCs and MSDOS are well catered for. So far as I know, however, TP 7 remains somebody’s intellectual property, although it isn’t entirely clear whose. It probably comes into the category of ‘abandonware’ but, to be frank, it’s not all that important: it can’t produce code that runs on modern Windows systems, and few people have much interest in early Windows – it’s just not retro enough.
+
So that’s Turbo Pascal: a colossus of the microcomputing world, which faded and died largely unlamented. It always was a creature of its time – it prospered for the brief period when microcomputers were limited enough to reward skilled development, but widespread enough to create a substantial market for programmers. By the time it started to become difficult to find enough highly-skilled programmers to satisfy the market needs, it didn’t matter: computers were getting powerful enough to make good our limitations as programmers. That almost nobody has the skills these days to implement something like TP is something we could eventually come to regret – or, rather, our grandchildren could. Software development is becoming a semi-skilled industry, and I doubt that the rise of AI will entirely compensate for this.
+
But who needs a Turbo Pascal, when you have 128Gb of RAM attached to an 5GHz CPU, for a hastily-written, bloated Java program to burn?
Do you need to
-wipe that hard drive? And can you?
+
Do you need to wipe that hard drive? And can you?
-
TL;DR: yes, and maybe. If the drive contains state defence secrets,
-it’s probably better to crush it.
+
TL;DR: yes, and maybe. If the drive contains state defence secrets, it’s probably better to crush it.
The problem
-
I’ve got a couple of laptops that I’d like to sell. They contain hard
-drives (SSDs, in this case) on which is data I’d not like to fall into
-anybody else’s hands. I’ve also got a bunch of high-capacity magnetic
-disks that I’ve retired from a RAID array.
-
Can I sell these things safely? It’s worth bearing in mind that the
-financial gains – for me – are minimal. If you’re a computer dealer, or
-a corporate IT department, then things might be different, both in the
-sums of money involved, and the sensitivity of the data.
-
Because individual used drives don’t command high prices on the
-second-hand market, there are people or organizations buying them up,
-just to see what data they contain. Most recovered data will be garbage
-to anything but the previous owner; but if even one in a thousand
-second-hand drives contains a dump of credit card details, that’s a big
-win for the villains. And if even one in a million turns out to contain
-state secrets, that’s a jackpot.
-
My second-hand drive may prove to contain an email password buried in
-a swapfile or temporary file somewhere, or something of a similar level
-of sensitivity. I encrypt all the sensitive data on my drives, but I
-can’t control where the operating system might buffer it. Now, to be
-sure, the chances of anybody who buys my old drives having the skills to
-unpack that data, and its being worth their while doing so, are minimal.
-I’m not very concerned. Nevertheless, if I can easily reduce that risk
-to near-zero (and it’s never going to be zero), then I probably
-should.
+
I’ve got a couple of laptops that I’d like to sell. They contain hard drives (SSDs, in this case) on which is data I’d not like to fall into anybody else’s hands. I’ve also got a bunch of high-capacity magnetic disks that I’ve retired from a RAID array.
+
Can I sell these things safely? It’s worth bearing in mind that the financial gains – for me – are minimal. If you’re a computer dealer, or a corporate IT department, then things might be different, both in the sums of money involved, and the sensitivity of the data.
+
Because individual used drives don’t command high prices on the second-hand market, there are people or organizations buying them up, just to see what data they contain. Most recovered data will be garbage to anything but the previous owner; but if even one in a thousand second-hand drives contains a dump of credit card details, that’s a big win for the villains. And if even one in a million turns out to contain state secrets, that’s a jackpot.
+
My second-hand drive may prove to contain an email password buried in a swapfile or temporary file somewhere, or something of a similar level of sensitivity. I encrypt all the sensitive data on my drives, but I can’t control where the operating system might buffer it. Now, to be sure, the chances of anybody who buys my old drives having the skills to unpack that data, and its being worth their while doing so, are minimal. I’m not very concerned. Nevertheless, if I can easily reduce that risk to near-zero (and it’s never going to be zero), then I probably should.
The corporate IT department certainly should.
-
So, yes, I probably want to wipe all the data from a hard drive that
-I plan to sell or re-use. The alternative, of course, is to smash the
-drive to pieces with a club-hammer; and that may be appropriate in some
-cases.
-
What
-about disk encryption as an alternative to wiping?
-
If you use whole-disk encryption, doesn’t the problem go away? After
-all, if the data on the disk is robustly encrypted, perhaps you don’t
-even need to wipe it?
-
The problem here is that, if the data is very valuable, it’s been
-known for people to be coerced or blackmailed into handing over their
-encryption passwords. It’s better, I think, for there to be no data to
-recover in the first place.
-
So even for my domestic data, I’m not going to sell a laptop or a
-hard drive without taking reasonable steps to ensure that my personal
-data is not recoverable. What ‘reasonable’ amounts to, of course,
-depends on the circumstances.
+
So, yes, I probably want to wipe all the data from a hard drive that I plan to sell or re-use. The alternative, of course, is to smash the drive to pieces with a club-hammer; and that may be appropriate in some cases.
+
What about disk encryption as an alternative to wiping?
+
If you use whole-disk encryption, doesn’t the problem go away? After all, if the data on the disk is robustly encrypted, perhaps you don’t even need to wipe it?
+
The problem here is that, if the data is very valuable, it’s been known for people to be coerced or blackmailed into handing over their encryption passwords. It’s better, I think, for there to be no data to recover in the first place.
+
So even for my domestic data, I’m not going to sell a laptop or a hard drive without taking reasonable steps to ensure that my personal data is not recoverable. What ‘reasonable’ amounts to, of course, depends on the circumstances.
Possible solutions
-
Here are some methods that I might use to prepare a drive for
-sale.
+
Here are some methods that I might use to prepare a drive for sale.
-
Writing a new filesystem on top of the old one (‘formatting’ is the
-Windows term)
+
Writing a new filesystem on top of the old one (‘formatting’ is the Windows term)
Using erasure utilities built into PC firmware (BIOS)
Invoking a drive’s built-in erasure procedure
-
Writing zeros, or random data, to each disk sector, perhaps
-repeatedly
+
Writing zeros, or random data, to each disk sector, perhaps repeatedly
Unreliable approaches
-
The first of these methods, writing a new filesystem, we can dispose
-of quickly. This won’t defeat anything but the most casual intrusion.
-Nearly all the original data will still be present, and there are many
-widely-available utilities for recovering data in this situation. This
-is simply not a safe procedure to use on drives that contain any kind of
-sensitive data.
-
If your computer has firmware (BIOS) with a built-in disk-wiping
-feature then, of course, you could consider using it for convenience.
-Such an approach will be easier than taking the drive out and wiping it
-using a different machine. It’s hard to dismantle a modern compact
-laptop, and you probably won’t be able to wipe the operating system
-drive, when the operating system is running.
-
However, these firmware facilities really are just for convenience –
-they generally do one of the other things I’ve listed above. If you’re
-lucky, you may get to choose which one.
-
Because firmware methods don’t offer anything in principle
-that other methods don’t, I won’t consider them further. If you want to
-wipe a laptop drive without removing it, you could always boot the
-laptop from a USB stick or external drive.
-
Using built-in disk
-erasure features
-
So, turning to proprietary methods: modern SATA drives almost always
-have a built-in erasure method. This can be invoked by sending the drive
-a specific command. On a Linux system you can do this easily using the
-hdparm utility:
+
The first of these methods, writing a new filesystem, we can dispose of quickly. This won’t defeat anything but the most casual intrusion. Nearly all the original data will still be present, and there are many widely-available utilities for recovering data in this situation. This is simply not a safe procedure to use on drives that contain any kind of sensitive data.
+
If your computer has firmware (BIOS) with a built-in disk-wiping feature then, of course, you could consider using it for convenience. Such an approach will be easier than taking the drive out and wiping it using a different machine. It’s hard to dismantle a modern compact laptop, and you probably won’t be able to wipe the operating system drive, when the operating system is running.
+
However, these firmware facilities really are just for convenience – they generally do one of the other things I’ve listed above. If you’re lucky, you may get to choose which one.
+
Because firmware methods don’t offer anything in principle that other methods don’t, I won’t consider them further. If you want to wipe a laptop drive without removing it, you could always boot the laptop from a USB stick or external drive.
+
Using built-in disk erasure features
+
So, turning to proprietary methods: modern SATA drives almost always have a built-in erasure method. This can be invoked by sending the drive a specific command. On a Linux system you can do this easily using the hdparm utility:
# hdparm --security-erase ...
or
# hdparm --security-erase-enhanced ...
-
Again, if it’s the operating system disk you want to erase, you’ll
-have to boot from an external operating system drive to do this.
-
What these proprietary procedures actually do depend entirely on the
-drive manufacturer. On magnetic drives, they probably overwrite the
-individual sectors, which is something that can be done better using
-utilities specifically designed for this purpose. On SSDs, it’s even
-less clear what they do. SSD technology does, in principle, allow for
-some quite rapid erasure techniques. If the SSD is self-encrypting, for
-example, then all the erasure might do is to overwrite the encryption
-password in the drive firmware with a new one. This is a shockingly fast
-way to erase an SSD, if it’s supported. There are other fast-erasure
-methods that a drive might implement.
-
Unfortunately, I’ve seen reports that drives from a number of major
-manufacturers have weaknesses in their secure erasure methods. I don’t
-know how reliable these reports are (the Internet being what it is) but,
-for example, if the encryption passwords generated by the drive firmware
-are guessable (perhaps they use weak random number generation), then any
-change of the password will be useless. It’s certainly
-plausible that vendors could use weak methods, and keep quiet
-about it.
-
Before using a SATA drive’s built-in erasure method, I would want to
-satisfy myself that it’s reliable. This might not be easy – the Internet
-is full of misinformation, and drive vendors are not particularly
-forthcoming with information in this area. As ever, you have to weigh up
-the risks and benefits.
-
There is another reported problem with using the built-in
-erasure method in SATA drives. Again, it’s not a problem I’ve had, and I
-don’t know how common it is. If it’s an external drive with a USB
-interface, then (apparently) the USB protocol can time out in the middle
-of the erase operation. If this happens then (apparently) you’re left
-with a dead drive. I guess if this does happen, it’s more likely to
-happen with magnetic disks, where erasure tends to be slow. And if the
-alternative to erasing the disk is to smash it with a club-hammer, it
-might be worth the risk. As I say, I’ve not encountered the problem, and
-I don’t know how likely it is.
-
Some drive vendors supply diagnostic software that has an erasure
-feature, in addition to the drive’s responding to the SATA erase
-command. In the end, though, I prefer not to trust vendor’s erasure
-methods. Not because I don’t think they work, but because I can’t be
-sure how they work. I prefer to have these things under my
-control, so I can assess the risks and benefits.
+
Again, if it’s the operating system disk you want to erase, you’ll have to boot from an external operating system drive to do this.
+
What these proprietary procedures actually do depend entirely on the drive manufacturer. On magnetic drives, they probably overwrite the individual sectors, which is something that can be done better using utilities specifically designed for this purpose. On SSDs, it’s even less clear what they do. SSD technology does, in principle, allow for some quite rapid erasure techniques. If the SSD is self-encrypting, for example, then all the erasure might do is to overwrite the encryption password in the drive firmware with a new one. This is a shockingly fast way to erase an SSD, if it’s supported. There are other fast-erasure methods that a drive might implement.
+
Unfortunately, I’ve seen reports that drives from a number of major manufacturers have weaknesses in their secure erasure methods. I don’t know how reliable these reports are (the Internet being what it is) but, for example, if the encryption passwords generated by the drive firmware are guessable (perhaps they use weak random number generation), then any change of the password will be useless. It’s certainly plausible that vendors could use weak methods, and keep quiet about it.
+
Before using a SATA drive’s built-in erasure method, I would want to satisfy myself that it’s reliable. This might not be easy – the Internet is full of misinformation, and drive vendors are not particularly forthcoming with information in this area. As ever, you have to weigh up the risks and benefits.
+
There is another reported problem with using the built-in erasure method in SATA drives. Again, it’s not a problem I’ve had, and I don’t know how common it is. If it’s an external drive with a USB interface, then (apparently) the USB protocol can time out in the middle of the erase operation. If this happens then (apparently) you’re left with a dead drive. I guess if this does happen, it’s more likely to happen with magnetic disks, where erasure tends to be slow. And if the alternative to erasing the disk is to smash it with a club-hammer, it might be worth the risk. As I say, I’ve not encountered the problem, and I don’t know how likely it is.
+
Some drive vendors supply diagnostic software that has an erasure feature, in addition to the drive’s responding to the SATA erase command. In the end, though, I prefer not to trust vendor’s erasure methods. Not because I don’t think they work, but because I can’t be sure how they work. I prefer to have these things under my control, so I can assess the risks and benefits.
Drive-wiping applications
This brings us to brute-force erasure methods.
-
For magnetic disks, the ‘gold standard’ method of erasure is to write
-each sector repeatedly with random numbers. If the intruder is very
-determined, it seems that a single overwrite is not enough. Each change
-in magnetic state leaves traces that can be recovered with specialized
-hardware.
-
Now, to be sure, this isn’t the kind of hardware that a bloke in the
-pub will have access to. To need an approach to erasure like this, you’d
-have to be very paranoid, or be storing national defence plans.
-
For most personal data, my feeling is that simply overwriting the
-sectors with a single pass of random numbers will be sufficient, on any
-type of drive. I do this often enough that I wrote my own software to
-do it. Why would I do that? There are many existing utilities that
-can do this kind of disk erasure, but mine combines the erase with a
-read-back test, so I can be sure that the disk remains fully functional
-after being wiped.
-
It’s worth bearing in mind that applying even a single pass of
-writing every sector will take a long, long time on a large magnetic
-disk – hours to days. It will be quicker on an SSD, but you might not
-want to do it on an SSD.
+
For magnetic disks, the ‘gold standard’ method of erasure is to write each sector repeatedly with random numbers. If the intruder is very determined, it seems that a single overwrite is not enough. Each change in magnetic state leaves traces that can be recovered with specialized hardware.
+
Now, to be sure, this isn’t the kind of hardware that a bloke in the pub will have access to. To need an approach to erasure like this, you’d have to be very paranoid, or be storing national defence plans.
+
For most personal data, my feeling is that simply overwriting the sectors with a single pass of random numbers will be sufficient, on any type of drive. I do this often enough that I wrote my own software to do it. Why would I do that? There are many existing utilities that can do this kind of disk erasure, but mine combines the erase with a read-back test, so I can be sure that the disk remains fully functional after being wiped.
+
It’s worth bearing in mind that applying even a single pass of writing every sector will take a long, long time on a large magnetic disk – hours to days. It will be quicker on an SSD, but you might not want to do it on an SSD.
Can we wipe SSDs?
-
Is there an issue in this area with SSDs? There’s plenty of
-information (or misinformation) on the Internet to suggest that there
-might be. There seem to be two main (potential) problem areas.
-
First, SSDs have limited write lifetimes. They are limited both in
-the number of times a specific sector can be written, and in the total
-amount of data that can be written. This much, at least, is not
-contentious. There’s no question that writing every sector will have
-some impact on the drive’s long-term service life. But how
-much?
-
Modern SSDs have claimed sector-write lives of perhaps 100,000
-cycles. Total write lifetimes are claimed to be of the order of hundreds
-of terabytes. It seems to me that a single pass of writing every sector
-isn’t going to have any practical impact on drive life. Given the way
-that SSDs work, I see no evidence that writing more than a single pass
-will increase the effectiveness of the erasure anyway.
-
The second problem is more subtle. Modern SSDs are
-overprovisioned; that is, they contain more storage than they
-advertise. The ‘spare’ storage is used to replace sectors that have
-failed in use. A failed sector may still contain sensitive data – you
-just won’t be able to read or write it using any common tools. Disk
-vendors sometimes provide proprietary tools for inspecting this kind of
-data. Such tools, if available, could probably get data from failed
-sectors that are inaccessible to an operating system.
-
Another potential problem with SSDs is that they don’t necessarily
-store data in predictable ways – predictable to the user, that is. For
-example, because so many sectors end up containing only zeros, a drive
-might have an ‘all zero’ flag that it writes, rather than writing the
-actual zeros. This is an optimization technique. So if you overwrite a
-sector with zeros, it might not change the actual stored data in any
-predictable way. Proprietary tools may be able to recover the underlying
-data, which was not changed by the write. Again, an ordinary operating
-system will not see this data – the drive’s data management is
-completely transparent. You’d need very specialized tools to get to this
-data.
-
Then there’s the issue of wear leveling. Because of the
-potential risks of over-writing sectors, SSDs don’t necessarily store
-the same logical sector in the same physical location.
-So overwriting a particular sector might not actually change the
-contents of the physical sector – the drive might write the new data in
-a different physical location. However, my feeling is that if you write
-every sector, wear leveling won’t be a problem, except if it’s
-combined with extensive overprovisioning.
-
My own feeling is that overwriting each sector of an SSD with random
-data is both safe in practice, and reliable enough for ordinary domestic
-data. As ever, though, your plans for world domination might need more
-careful handling. And the club-hammer is always on option.
+
Is there an issue in this area with SSDs? There’s plenty of information (or misinformation) on the Internet to suggest that there might be. There seem to be two main (potential) problem areas.
+
First, SSDs have limited write lifetimes. They are limited both in the number of times a specific sector can be written, and in the total amount of data that can be written. This much, at least, is not contentious. There’s no question that writing every sector will have some impact on the drive’s long-term service life. But how much?
+
Modern SSDs have claimed sector-write lives of perhaps 100,000 cycles. Total write lifetimes are claimed to be of the order of hundreds of terabytes. It seems to me that a single pass of writing every sector isn’t going to have any practical impact on drive life. Given the way that SSDs work, I see no evidence that writing more than a single pass will increase the effectiveness of the erasure anyway.
+
The second problem is more subtle. Modern SSDs are overprovisioned; that is, they contain more storage than they advertise. The ‘spare’ storage is used to replace sectors that have failed in use. A failed sector may still contain sensitive data – you just won’t be able to read or write it using any common tools. Disk vendors sometimes provide proprietary tools for inspecting this kind of data. Such tools, if available, could probably get data from failed sectors that are inaccessible to an operating system.
+
Another potential problem with SSDs is that they don’t necessarily store data in predictable ways – predictable to the user, that is. For example, because so many sectors end up containing only zeros, a drive might have an ‘all zero’ flag that it writes, rather than writing the actual zeros. This is an optimization technique. So if you overwrite a sector with zeros, it might not change the actual stored data in any predictable way. Proprietary tools may be able to recover the underlying data, which was not changed by the write. Again, an ordinary operating system will not see this data – the drive’s data management is completely transparent. You’d need very specialized tools to get to this data.
+
Then there’s the issue of wear leveling. Because of the potential risks of over-writing sectors, SSDs don’t necessarily store the same logical sector in the same physical location. So overwriting a particular sector might not actually change the contents of the physical sector – the drive might write the new data in a different physical location. However, my feeling is that if you write every sector, wear leveling won’t be a problem, except if it’s combined with extensive overprovisioning.
+
My own feeling is that overwriting each sector of an SSD with random data is both safe in practice, and reliable enough for ordinary domestic data. As ever, though, your plans for world domination might need more careful handling. And the club-hammer is always on option.
Summary
-
Despite extensive research, I remain conflicted. I’m fairly sure that
-my data is of limited use to anybody but me. In addition, I’m fairly
-sure that I can erase it well enough that nobody but the most determined
-intruder will be able to get at it, and even then only with great
-difficulty.
-
And yet… and yet. A private individual selling old laptop PCs and NAS
-drives isn’t going to make a whole lot of money. Is it worth even the
-tiny risk? I guess that depends on how much I need the money.
+
Despite extensive research, I remain conflicted. I’m fairly sure that my data is of limited use to anybody but me. In addition, I’m fairly sure that I can erase it well enough that nobody but the most determined intruder will be able to get at it, and even then only with great difficulty.
+
And yet… and yet. A private individual selling old laptop PCs and NAS drives isn’t going to make a whole lot of money. Is it worth even the tiny risk? I guess that depends on how much I need the money.
-
-
-
In the days
-before an automotive satnav unit was just an extension of a smartphone,
-the Garmin Nuvi 300 was the pinnacle of this technology. Garmin’s
-competitors – of which there were, and remain, few – might not have
-agreed with that assessment, but the Nuvi 300 was feature-packed, and
-undeniably practical. The automotive satnav market has been in
-decline ever since.
-
-
-