-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Decipher a new device and write a decoder
This README
- documents the protocol used by the Digitech XC-0324 temperature sensor ,and then
- continues with a "tutorial for my future self" about how the
xc0324_callbackhandler was developed.
The tutorial was
written "by a newbie, for a newbie" so may well be too verbose for
people with experience, either as a developer or with SDR. But it might
be useful as starting point for someone who has just come across rtl_433
and is interested in learning how to reverse engineer a new device and
develop a decoder for it.
In hindsight the XC-0324 message protocol is quite trivial (only 4 fields - a sync byte, a sensor id byte, temperature encoded in a byte and half (3 nibbles), and a checksum byte). So the code for actually decoding the transmissions is really quite short.
Much of the device handler code revolves around trace outputs, that show the types of output and information that was available at various stages in the reverse engineering process. Both the strategies :
- for deciphering the transmissions, and
- for developing the decoder itself,
are iterative and proceed incrementally. After each incremental step, the interim debug outputs are opened in a spreadsheet package and examined to assist with the reverse engineering process. These linked strategies (using debug mode for reverse engineering; then enhancing the prototype decoder as information is discovered) are the focus of the tutorial. Specifically it covers:
- how to batch process sets of captured test data in debug mode to produce useful csv files, and
- the strategy I followed to reverse engineer the XC-0324 protocol, or in other words why I wanted csv files and how I used them.
- Along the way it provides a gentle introduction to just a few
of the many useful functions available in the
rtl_433codebase.
It is written as if I knew what I was doing - but of course that is far from the truth, so I have included a section which contains a selection of some of the references that I found especially useful, ie it contains pointers to :
- things I wish I had found, read and understood before I started.
PS The tutorial assumes that you :
- know the basics of git,
- have successfully compiled a copy of rtl_433 - see Building rtl_433,
- are comfortable with the basics of coding in some language, and can "read" C code, and
- are comfortable with hex, binary and the idea of how integers of various lengths, characters etc are coded into binary values (aka sequences of bits - 0's and 1's),
or if not, are happy enough to learn through trial, error and internet searches.
Each XC-0324 sensor transmits a pulse every 60 seconds, on frequency of (approx) 433.900 MHz.
The transmitted temperatures can be seen on a paired XC-0322 monitor. A single monitor can display the temperatures from up to 3 XC-0324 sensors.
The radio transmissions are OOK (OnOffKeying) and individual bits are encoded using pulse position modulation (ie the gap widths contain the modulation information about each bit's value)
- Each pulse is about 400 us
- A short gap is (approx) 520 us, and represents a 0 bit
- A long gap is (approx) 1000 us, and represents a 1 bit.
The protocol was deciphered using captured pulses from two sensors. See the test data in rtl_433_tests/tests/XC-0324.
My older sensor transmits a "clean" pulse (ie a captured pulse, examined using Audacity, has pretty stable I and Q values, ie little phase wandering):
My newer transmitter seems less stable (ie within a single pulse, the I and
Q components of the pulse signal appear to "rotate" through several cycles).
The rtl_433 -A -vv output correctly guessed the older transmitter modulation
and gap parameters, but mistook the newer transmitter as pulse width
modulation with "far too many" very short pulses.
A package is 148 bits (plus or minus one or two due to demodulation or transmission errors).
Each package contains 3 repeats of the basic 48 bit message, with 2 zero bits separating each repetition.
A 48 bit message consists of :
- byte 0 = preamble (for synchronisation),
0x5F - byte 1 = device id
- byte 2 and the first nibble of byte 3 encode the temperature
- as a 12 bit integer,
- transmitted in least significant bit first order
- in tenths of degree Celsius
- offset from -40.0 degrees C (minimum temperature specified for the device)
- byte 4 is constant (in all my data)
0x80- maybe a battery status ?
- byte 5 is a check byte (the XOR of bytes 0-4 inclusive)
- each bit is effectively a parity bit for the correspondingly positioned bits in the real message.
This is explanation is pretty basic provided you've done it before, but may be helpful if, like I was, you are new to this.
After setting up your own local clone of the rtl_433 repository, and
compiling rtl_433 from source
(see rtl_433 : Building.md),
the gameplan is:
-
Find the transmission frequency
-
Capture some test files and (crucial) record the corresponding values from the monitor for each file as it is captured.
-
Discover the radio encoding protocol that has been used to transmit a sequence of bits as a radio signal.
-
Create a basic prototype callback handler, then
-
Iteratively:
- Batch process the test files to produce csv files, using preliminary versions of the device handler,
- Open in a spreadsheet package and edit (or cut and paste) the recorded "true" values into the csv file,
- Sort into temperature order,
- Look for recognisable patterns in the data bits
- Form a hypothesis about how (part of) the message might be encoded
- Code that into the prototype callback handler,
- recompile, and
- iterate to decide whether the hypothesis stands up.
-
Then, if you like, submit your test data and your new device handler for inclusion in the official rtl_433 repositories.
Each of these steps is covered in detail below, but first ...
Since these are in the nature of "things I wish I had known before ..." I'll include them here. It also means I can refer to them if necessary, rather than repeating information which has already been covered by others more knowledgeable than me.
They are presented in the order in which they started to become useful and make sense to me.
1 - "The Hobbyists Guide to RTL-SDR" is a book I bought and read to get
started when I bought my first RTL-SDR dongle. In particular it helped
me get the right drivers installed, and get SDR# downloaded and
installed so I could "see" what the dongle was receiving. (A range of
other SDR packages are covered as well)
2 - The airspy download page for SDR# - a software defined radio program.
(There are many other good SDR programs, this just happens to be the one I used)
3 - A page from the rtl_433 wiki, which is where my current adventure began :-)
4 - A blog page (in French) mentioned on the rtl_433 wiki page. The wiki says "a good helper". I would describe it as an excellent helper (once I had used an internet translation service to cope with my appallingly limited French).
NB As at the time of writing (Oct 2018) my browser warns me that the website's certificate is out of date :-( ... but the information and advice certainly isn't.
5 - The Audacity project website. Audacity is great software for looking at the waveform of individual saved ".cu8" test files. As explained (with pictures) in the French blog :
- import the .cu8 file as a raw sample,
- nominate "no-endianess",
- set the correct sampling frequency (250000 Hz) to get a valid timescale, then
- zoom in and see the individual pulses
- (and even the I and Q components of your signal as the left and right stereo channels).
6 - If (like me) you aren't sure what I and Q components of a signal means, there are lots of helpful websites, once you know what to search for. Some that I particularly liked (the first one has some neat interactive graphics) are :
- I/Q Data for Dummies
- National Instruments: What's I Q data
- Understanding I Q signals and quadrature modulation
7 - This sequence of 3 blog posts from RaysHobby is a nice practical worked example of how to collect useful samples and then reverse engineer / decipher the signals from a temperature sensor. Initially he used an Arduino, but a Raspberry Pi version is also covered now.
8 - Last, but far from least, there is a lot that can be learnt from browsing the source code of rtl_433 itself. (There are some weblinks below, but they are also there in your local cloned copy of the rtl_433 repository.)
- Why not begin with an example that provides a template for a new
device handler and includes lots of helpful comments about
different options you can use? I certainly wish I'd come across
this earlier while I was learning about the
rtl_433code base. - I found this following device handler very helpful - perhaps because it was so close in concept to the XC-0324 device I was working on (simple protocol, reporting temperature with nothing else to confuse things)
- And the definitive document about what radio demodulation schemes rtl_433 supports, and the meaning of the key parameters you'll need to discover, if your sensor happens to use that particular protocol
- The
rtl_433codebase includes a suite of utility functions, which are really useful for adding informative trace messages to your outputs during the iterative deciphering phase - In fact it is well worth browsing all the header files, just to get
an idea of the types of functions that
rtl_433already provides. Don't worry if they don't make much sense to you to begin with. I certainly couldn't follow them at first - there's a lot of clever stuff in there. But later on, when I hit a particular deciphering problem, I "remembered" that I'd seen something that "sounded" a bit like that, and investigated further.
It's obvious, but the first step is to become familiar with how to run
rtl_433, and the options it already provides.
From a (bash) shell, execute
<path_to_compiled_executable>/rtl_433 <one or more options>Options to play with include
-
-h: help - provides pointers other more specific helps as well -
-R: lists the available decoders already known tortl_433, -
-R 61: (for example) invokes the 61st decoder in the list -
-G: invokes (ie tries out) all known decoders -
-fto specify a frequency, or-rto specify a test file : specifies what signal the decoders run on -
-v -v: (or equivalently-vv) tells a decoder to emit verbose output -
-vvv: decoder emits more verbose (debug level) output -
-vvvv: decoder emits very verbose (trace level) debug output -
-F csv:<filename>.csvor-F json:<filename>.json: specifies what type of output file(s) you want, and what to call them -
-K FILE: includes an identifyingtagfield in the output file(s), so, when using the-roption to decode a set of test files, you can tell which testfile corresponds to which line of output -
-M bits: tell the decoder debug utility functions to output messages in bit format as well as hex format
There are many other options as well. Much of rtl_433 is
self-documented, so do read through the various -h details,
and the topic specific sub-help entries it points towards. Since rtl_433
is under active development, by the time you are reading this the options
available may well have been extended and improved.
That was pretty easy for my example (it's written on the back of the
XC-0324). I also checked it using the "waterfall" display from SDR#
(a Software Defined Radio program that I had installed from earlier
experiments with my RTL-SDR dongle). The "waterfall" display also
confirmed that the two sensors each transmitted new readings every 60
seconds.
It is best to offset the frequency specified to rtl_433 (using the - f
argument) very slightly, say by about 20 KHz. (I think this slight
"detuning" is to prevent rtl_433 from becoming too sensitive to very
slight changes in a very strongly received signal,
but that's just a guess on my part).
For the XC-0324, where the nominal transmission frequency is
433.9 MHz, a value of -f 433920000 works well.
The mechanics of this are easy as well - mkdir and then cd to your
chosen test data sub-directory and run <path>/rtl_433 -S all -f <frequency>.
(You'll need to specify the <path> to where your compiled copy of
the rtl_433 executable lives, of course. For brevity I'll omit the <path>
for the rest of the tutorial, and just write ./rtl_433).
Use ./rtl_433 -h to see other arguments you might like to use (and of
course, try ./rtl_433 -G -f <frequency> first - you might well be lucky
and find your device has already has a callback handler).
It is crucial to record exactly what value appears on the monitor
when rtl_433 reports that it is saving a test sample (and which test
file that value belongs to!). I spent a happy hour or two (spread over
3 or 4 experimental sessions) watching files being saved every 30 seconds
or so, and recording the corresponding values showing on the XC-0322 monitor.
I could have been much quicker if I'd known what I needed to get from
each experiment :-)
A "good" set of test data should have:
- some low values (eg put the sensor in the freezer, then take it out and record its transmissions as it warms up)
- some high values (eg leave the sensor in full sun, or warm it with a hair dryer)
- a run of consecutive values (eg if the monitor reports changes in 0.1 degrees C steps, several transmission values separated by only 0.1 degrees. NB They don't have to be recorded consecutively, just appear somewhere in the collection of test data)
- a few "repeat values", and
- if possible, transmissions from more than one sensor (or try removing the batteries, and reinserting them - that sometimes makes the sensor "choose" a new id).
The real fun starts here :-)
There are numerous ways of encoding and transmitting digital data via
radio waves, and rtl_433 has already done the hard work involved in
decoding most of them back from radio waves into bit patterns. All you
have to do is figure out which transmission protocol the device uses,
and with what parameters.
In fact rtl_433 even does it's best to guess the transmission
protocol for you. You can use the -a option, or even better (in my case)
the -A option. These options can be used interactively, while capturing
test data (but you should be pretty busy writing down the values corresponding
to each saved file, so I recommend running your saved test files
through these option later on). Use the -r option to specify the test file
(or files - rtl_433 will accept wildcard names like g*.cu8 as the
final parameter on the command line).
But to understand that output, you have to know at least the basics of how the digital to radio encoding works.
Tip : Now is a good time to read up on I and Q signals, and quadrature modulation.
Anyway, (with suitable caveats - I'm at the opposite end of the scale to expert, so this is probably subtly wrong in many places, though hopefully not too misleading) here goes.
There are two broad ways of superimposing a stream of binary data onto a radio signal:
- FSK (Frequency Shift Keying) - think FM radio stations, and
- OOK (On Off Keying) - think AM radio stations.
"Keying" means "something" changes to signify a bit - think morse code. With FSK a frequency changes, with OOK an amplitude changes (usually a "pulse" starts or stops or continues or ...).
This makes more sense if you can "look" at the radio signal - there are
several programs that do exactly that - I used Audacity, and,
following the instructions in the (translated) French blog entry,
I imported several of the saved .cu8 test data files into Audacity.
Looking at the waveform from the XC-0324 transmissions it pretty clearly uses OOK. A transmission consists of a sequence of pulses (ie signal ON), all of pretty much the same amplitude, separated by gaps (signal OFF).
Tip : Now's a good time to read through include/pulse_demod.h to learn
about the wide variety of modulations rtl_433 supports "out of the box",
and what the parameters mean for each modulation type.
Within OOK, there are (of course) numerous different ways of transmitting bits :
- the width of the pulse could signal a bit (eg short pulse = 0, long pulse = 1, or vice versa). This is known as Pulse Width Modulation or PWM.
- the width of the gap could signal the bit (eg short gap = 0, long gap = 1, or vice versa). This is known as distance modulation, or Pulse Position Modulation or PPM.
- the time until a pulse changes (starts or stops) could signal a bit. This is known (I think) as Manchester modulation.
There are other schemes as well, but luckily for me, the XC-0324 uses a
simple PPM approach (as can be seen by looking at the waveforms in
Audacity and noting that the pulses are more or less the same duration,
but some gaps are long and some gaps are short). Of course you have to
tell rtl_433 how short is short, and how long is long. Duration is
measured in us (microseconds) and can be read (with care) from the
Audacity output (provided you've set the sampling rate parameter
correctly). Or, now you know what it's talking about, you can also get
these parameters from the rtl_433 -A Pulse Analyser output. To prevent
confusion I also add -R 0, which prevents existing decoders from being
loaded and trying to decode the signals as well.
Registered 0 out of 120 device decoding protocols [ ]
Test mode active. Reading samples from file: ../../../rtl_433_tests/tests/XC-0324/02/g001_433.917M_250k.cu8
Detected OOK package @0.046908s
Analyzing pulses...
Total count: 150, width: 175.24 ms (43809 S)
Pulse width distribution:
[ 0] count: 148, width: 452 us [432;468] ( 113 S)
[ 1] count: 2, width: 340 us [340;344] ( 85 S)
Gap width distribution:
[ 0] count: 89, width: 520 us [512;628] ( 130 S)
[ 1] count: 60, width: 1008 us [1000;1020] ( 252 S)
Pulse period distribution:
[ 0] count: 89, width: 976 us [872;1080] ( 244 S)
[ 1] count: 60, width: 1464 us [1456;1472] ( 366 S)
Level estimates [high, low]: 15939, 77
RSSI: -0.1 dB SNR: 23.1 dB Noise: -23.3 dB
Frequency offsets [F1, F2]: 16460, 0 (+62.8 kHz, +0.0 kHz)
Guessing modulation: No clue...
Even though the Pulse Analyser says it has no clue about the modulation (not sure why!) it is easy to see that:
- almost all (148) of the pulses have similar widths (average 452 us long)
- the gap widths are split between (averages of) 520 and 1008 us, and hence
- the pulse period (gap between successive pulses) is split between 976 us (~= 452 + 520) and 1464 us (~= 452 + 1008).
The gap is what varies and hence must carry the information about bits, so it certainly looks like PPM to me! And the parameters are easy enough to extract now.
If you are interested in the relationship between microseconds (us) and numbers of samples, it helps to know that the signal was sampled at 250KHz (the default sample rate), so one sample unit lasts (1/ 250000) seconds = 4 us (microseconds), so, for example, a pulse of length 113 sampled units lasts for 113 * 4 us = 452 us (as reported in the Pulse Analyser output :-) ).
Reminder If you haven't already, you can read more about the assorted demodulation
options that rtl_433 supports,
and how to pass this radio demodulation information to rtl_433 by
browsing through the comments in include/pulse_demod.h.
NB. In the interest of not confusing anyone who is meticulous enough to
check this -A output using the test samples in
rtl_433_tests/tests/XC-0324,
I'll mention that I selected an output from my "good" sensor to
show above. The I and Q signal from my "bad" sensor is less stable (frequency
wise), and an earlier version of the -A Pulse Analyser output
used to be quite wrong and misleading for those signals (it detected
lots of very short pulses). But between
my initial drafting of this entry and now, the Pulse Analyser has been
improved :-) Nonetheless, there was a useful lesson from the old output, namely
the fact that the number of detected bits varied wildly between samples was a
good hint that something was wrong with the (attempted) demodulation and
analysis. Viewing the waveforms using Audacity helped me determine
what was "going wrong" with the signals from my "bad" sensor.
rtl_433 includes a special flex-ible decoder, invoked using
rtl_433 -X followed by appropriate parameters which specify what type
of decoding you want to apply. Running ./rtl_433 -X help sends
explanatory help text about the flex decoder to your screen.
From that output you find
./rtl_433: option requires an argument -- 'X'
Use -X <spec> to add a flexible general purpose decoder.
<spec> is "key=value[,key=value...]"
Common keys are:
name=<name> (or: n=<name>)
modulation=<modulation> (or: m=<modulation>)
short=<short> (or: s=<short>)
long=<long> (or: l=<long>)
...
with each of those components further explained on the screen.
For the digitech XC-0324, with information as found above, I ran :
./rtl_433 -R 0 -X "n=MYFIRSTTEST,m=OOK_PPM,s=520,l=1000,r=3000" -r <path_to_test_data>/g*.cu8I could equally well have had a look a live transmissions, by running
./rtl_433 -R 0 -X "n=LIVETEST,m=OOK_PPM,s=520,l=1000,r=3000" -f 433920000By default, output comes out in standard screen format, and looks something like this.
Disabling all device decoders.
Registered 1 out of 120 device decoding protocols [ ]
Test mode active. Reading samples from file: ../../../rtl_433_tests/tests/XC-0324/01/gfile001.cu8
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Tag : gfile001.cu8
time : @0.046920s
model : MYFIRSTTEST
count : 1
num_rows : 1
rows :
len : 149
data : 2fd177c040090bf45df0100242fd177c040090
codes : {149}2fd177c040090bf45df0100242fd177c040090
This is a pretty quick and neat way of testing out the modulation parameters, and getting a first look (in hex format) at the transmissions from your device.
If you get output :-), it means you have the correct demodulation parameters and can move on to the next step.
If not :-( ... it means your transmissions didn't trigger the flex decoder (ie it didn't recognise your transmissions), so nothing was output. You'll have to experiment some more with the demodulation type and parameters.
There's (lots) more that the flex decoder can do, including reading the
specifications from a conf(igure) file - for further information
see the help output.
Now that you know how to demodulate the transmissions, it's time to start writing a decoder for them.
As explained in the (translated) French blog (and no doubt elsewhere as well)
To add a device to rtl_433 the procedure is simple:
- create a
<protocolname>.cfile insrc/devices,- declare the new decoder in
include/rtl_433_devices.h,- add this file to
src/CMakeLists.txtandMakefile.am.The
src/devices/<protocolname>.cfile will contain two main things:
- the decoding function itself (we will come back to this), and
- a r_device structure configuring the device and indicating, among other things, the modulation (and therefore the demodulator to use) and a 1st level of encoding.
Walking through these steps in order :
There are 2 essential elements in a protocol decoder, namely :
- specify the modulation so
rtl_433can demodulate the transmission (i.e create anr_devicewhich tellsrtl_433how to convert the transmission into a collection of bits, which it puts in abitbuffer_tstructure), and - provide a
decode_fn(it used to be described as a callback function) which will receive thebitbuffer_tand perform further processing on it. Ther_deviceneeds to know the address of your decode function, in order to invoke it once it has demodulated the transmission, so that address is part of ther_devicedefinition.
You can use src/devices/newtemplate.c (changing the filename
appropriately of course) to get you started. Or browse through the code
for other devices for ideas.
Alternatively, follow the XC-0324 device code. The final version of
the C code can be found in src/devices/digitech_xc0324.c.
The first version was far simpler (after all, I just
wanted to confirm that I had the demodulation parameters correct and
that something (anything!) was happening when I ran rtl_433 with my
prototype device handler over my test data). So my initial
decode_fn (xc0324_callback) was nothing more than
#include "decoder.h"
static int xc3024_callback(r_device *decoder, bitbuffer_t *bitbuffer)
{
/*
* Early debugging aid to see demodulated bits in buffer and
* to determine if your width settings are matched and firing
* this callback.
*/
fprintf(stderr,"xc3024 callback was triggered :-) \n");
bitbuffer_print(bitbuffer);
return 1;
}The src/devices/newtemplates.c includes a r_device structure appropriate to its
own example device. For the XC-0324, based on the information above,
the r_device structure looks like:
r_device digitech_xc0324 = {
.name = "Digitech XC-0324 temperature sensor",
.modulation = OOK_PULSE_PPM,
.short_width = 520, // = 130 * 4
.long_width = 1000, // = 250 * 4
.reset_limit = 3000,
.decode_fn = &xc0324_callback,
.disabled = 1, // stop debug output from spamming unsuspecting users
};(Note : there is a slight inconsistency in the modulation naming,
the -X approach accepted OOK_PPM, but the .r_device
requires OOK_PULSE_PPM)
It was straightforward to add the definition of the new device handler, I just inserted an extra line in the file at the obvious place at the end of the list (don't forget to add a "continuation" "\" after the previous last entry)
...
DECL(bresser_5in1) \
DECL(digitech_xc0324)
...
Amending these two files was equally simple (note these lists are kept in alphabetical order) :
...
devices/danfoss.c
devices/digitech_xc0324.c
devices/dish_remote_6_3.c
...
and
...
devices/danfoss.c \
devices/digitech_xc0324.c \
devices/dish_remote_6_3.c \
...
First, fix any compiler warnings or errors :-( that occur.
I'm forever forgetting ; s, mismatching
pairs of { }, and many other mistakes that competent C programmers
would never make - carefully reading the warning and error messages,
plus a bit (well OK, a lot) of internet searching usually helps me
sort out the problems, and teaches me something more about C along
the way as well.
Once compiled and invoked on my test data sets,
./rtl_433 -R 120 -K -r <path_to_test_data>/g*.cu8this very basic device and decode function let me see
that my handler was being invoked :-) , and
what bitbuffer_print did.
Not surprisingly, bitbuffer_print just prints out the
bitbuffer, which looks remarkably similar to the strings of hex
characters produced by the -X option flex decoder.
You can find out more about the bitbuffer_t structure by reading
/include/bitbuffer.h
- essentially it is an array of rows; each row containing an array(column) of bits that comprise part of the demodulated radio transmission.
The XC-0324 device handler grew from there as I learnt more about the XC-0324
protocol, and about what various parts of the rtl_433 codebase already
do for you.
Once you get this far, it's almost time to see what your prototype decoder makes of your test data so you can start reverse engineering it :-) But first it is useful to know a bit about the way decoder functions can send information to the rtl_433 output files.
How a device decoder sends field entries to rtl_433 output files in production mode is explained later.
If you experimented with rtl_433 earlier, you'll know that
there are quite a range of output types - csv, json, and syslog
(network message style) outputs are all possible. A user specifies the output
file type, at run time, using options after the -F argument. rtl_433 looks
after the production of each different type of output - the device decoder
functions are isolated from that effort.
It is possible to iteratively reverse engineer a device and develop a
device decoder using basic C functions (printf and the like), and indeed
that's what I did to begin with.
But there are a number of specialised decoder_output_... functions,
which make life much simpler. They are declared in
/include/decoder_utils.h,
and are extremely useful whilst reverse engineering a new device.
They allow a decoder function to include "trace" style messages in the
output (csv, json or whatever),
accompanied by a hex representation of the whole bitbuffer, or
part of the bitbuffer (typically a single row, or an extract from
a single row). If you execute rtl_433 with the -M bits option, a binary
representation is produced as well (I find this so useful that I recommend
always using the -M bits option during development!)
The easiest way to see what each one does is to try them out. My personal favourites are :
-
decoder_output_bitbufferf: produces a hex (and optional binary) representation of the bitbuffer, plus aprintfstyle formatted "trace" message, and -
decoder_output_bitrowf: produces a hex (and optional binary) representation of part (eg one row) of the bitbuffer, plus aprintfstyle formatted "trace" message.
These development style output functions are very easy to use, and very flexible, hence easy to change as reverse engineering progresses.
For technical reasons the trace message is limited to 60 characters long, but that is quite sufficient for the iterative approach used in this tutorial.
Now that all the machinery has been introduced, it is time, finally, to start the reverse engineering in earnest.
Everyone has their own way of solving crossword puzzles, Sudoku and reverse engineering a new device.
What I really did was jump in to the middle, looking for the bytes in the bitbuffer that seemed to change when temperature changed, partially decode that for a few of my (at that stage) 240 tests sample files, and work outwards from there, trying to figure out what the surrounding bytes did.
But in retrospect I could have made the job easier with a bit of thought beforehand about what to look for, and in what order. So that's what is reported below :-)
First a bit of thought about what I would be looking for. Any sensor I am likely to be using will have to be cheap, and therefore simple. If there is an obvious, or easy way of doing something, that's probably what will have been done.
Keeping power consumption low to extend the battery life will be important for the transmitter (and the receiver as well), so the transmitter will want to send a message, then go back to sleep till it is time for the next reading. So it is almost certain to use a broadcast approach (think UDP if you know about LAN network protocols), rather than one with a lot of handshaking / conversation back and forth between the monitor and the sensor (ie not like TCP/IP).
If the sensor sleeps between transmissions, it will need to send a wake-up / synchronisation call to the receiver when it starts a new transmission or message. So there is probably a preamble or sync byte or something similar in the message, and the simplest place to put it is at the start! So byte 0 (and perhaps others) will be a constant sync byte / message type identifier.
Radio transmissions can suffer from interference, so some form of error detection will be essential. It might be byte by byte - eg 1 bit in every 8 bit byte could be a parity bit, or it could encompass the whole message - eg the last byte - or last few bytes - could be a checksum of some sort. Or in more complex cases some form of CRC (cyclic redundancy check). Anyway, expect the message to end with at least one byte for error detection purposes.
What to do if an error is detected (since the receiver can't ask for a repeat transmission)? Simplest approach seems to be just send the reading several times while the sensor is awake and transmitting, and hope one gets through uncorrupted. So expect to see several repeats of the actual temperature reading message in the overall package.
Getting more specific, I know the XC-0322 monitor can listen to up to 3 separate XC-0324 sensors. So some form of sensor id must be in the message - expect a byte (or at least a nibble) for that.
The XC-0324 says it can measure from -40 to +65 degrees Centigrade. And it appears to report in 0.1 degree steps. That amounts to 105*10 different temperature values to report, which is order of magnitude 2^10. Whatever encoding is used (and my initial guess would be some form of integer - simplest solution!) 2^10 must require more than 8 bits, so expect the temperature field to use 2 bytes (or maybe 1.5 bytes = 3 nibbles).
Putting all that together, expect a XC-0324 message to look like:
- a sync or preamble field (allow 1 byte, probably at the start)
- a sensor id (allow say 1 byte)
- 3 or 4 nibbles of temperature, say 2 bytes
- an error check field (allow 1 byte, probably at the end)
That makes a message of about 5 bytes = 40 bits.
The (good) transmissions in my test data seem to have between 148 and 152 bits, which suggests 3 repeats. But with quite a few "spare bits" left over. In fact it turned out there is an extra byte in the message (always constant in my observed data so far - perhaps a battery status indicator?) making 3 message repeats (at 48 bits per message) plus 4 bits left over (a perfect packet turns out to be 148 bits) - so allow 2 bits gap between each repeat message.
I could have made my life much easier by planning the order to search for features in the transmissions as well.
As confessed earlier, in reality I jumped in and started looking for the temperature field, so I could try to decode it. A sad case of "more haste, less speed" :-( A small number of my test transmissions were totally corrupt (probably random electrical noise from household devices); many more had small glitches (a spurious leading zero at the start, extra bits added into the middle of the transmission, occasional bits missing, ...). Looking at the whole lot in one go was quite confusing.
A more sensible approach would have been :
- to start by synchronising the transmissions, so that comparable bytes and bits line up, then
- to try and identify repeats of the basic message (are they all on one row, or does the protocol involve multiple repeated rows), and only then, once you can isolate individual messages
- to start to search for individual fields and try to decode them
OK, now lets look at some data.
The debugging / deciphering strategy (copied from several helpful blog posts) involved repeatedly running an 'under development' device handler on the set of test files, creating csv files, opening and manipulating them in a spreadsheet package, and incrementally adding new features and trace messages until I figured everything out.
The first step is to find the sync byte (should be easy, we expect it to be constant and at the start of the message). There might be a longer preamble - but it turns out not to be the case for the XC-0324.
For this I made a small amendment to the protoype decoder function,
so it used one of the decode_output_ functions
static int xc0324_callback(r_device *decoder, bitbuffer_t *bitbuffer)
{
decoder_output_bitbufferf(decoder, bitbuffer, "xc0324 Look for sync byte");
return 1;
}then recompiled and ran
./rtl_433 -R 120 -F csv:temp.it1.csv -K FILE -M bits -r <path_to_test_data>/g*.cu8-
120is the identity number my under development decoder was allocated byrtl_433 -
-K FILEtells rtl_433 to include a column in the spreadsheet "tagging" which test file it processed to produce each line
You can get a similar output by running the final Digitech XC-0324 decoder
in -vvv (very verbose) mode over test files from the subdirectories in
rtl_433_tests/tests/XC-0324.
(Usually, when a decoder is completed, all the debug and trace messages
are removed, to improve the run time efficiency and long term maintainability
of rtl_433. However, they were left in the Digitech XC-0324 decoder so that
keen readers of this tutorial could simulate what might be seen during the early
stages of deciphering a new device, and developing a new decoder)
./rtl_433 -vvv -R 120 -F csv:temp.vvv.csv -K FILE -M bits -r <path_to_test_data>/g*.cu8I opened the "temp.it1.csv" file in a spreadsheet package, looked at a few lines and
the hex value 0x5F in byte 0 jumped out as a likely candidate.
In the test files that made it to the
rtl_433_test repository, 0x2F looks more likely - but luckily, when I was
working with my full set of over 240 test files, 0X5F appeared much
more frequently. If you are good at binary to hex you can see that 0x2F
is what you would get if you prepend a 0 bit to 0x5F, ie
shifted it right by 1 place in the bitbuffer - all the 0x2F values come
from rows that are 149 or slightly more bits long. The 148 bit rows
usually begin with 0x5F.
I plugged 0x5F in as the first version (and it turned out to be right)
of the sync part of the message, and moved on.
I also took a guess at 48 for the message length (it seemed to be about right for packages of 148 (sometimes plus a few) bits; it suggested 3 repeats and a small number of spacer bits) and luckily that also was right. If you stare very hard at the hex output for a row that has exactly 148 bits,
5f 64 00 40 80 fb 17 d9 00 10 20 3e c5 f6 40 04 08 0f b
you may spot that 5F 64 00 ... repeats again about two thirds of the way
through the transmission (look for c5 f6 40 .... There is actually
another repeat as well, but since it is offset by only 2 bits, it is
pretty hard to see in the hex representation.
The next version of the callback function looked like :
#define XC0324_DEVICE_BITLEN 148
#define XC0324_MESSAGE_BITLEN 48
#define XC0324_MESSAGE_BYTELEN (XC0324_MESSAGE_BITLEN + 7) / 8
#define XC0324_DEVICE_STARTBYTE 0x5F
#define XC0324_DEVICE_MINREPEATS 3
static const uint8_t preamble_pattern[1] = {XC0324_DEVICE_STARTBYTE};
static int xc0324_callback(r_device *decoder, bitbuffer_t *bitbuffer)
{
int r; // a row index
uint8_t b[XC0324_MESSAGE_BYTELEN];
uint16_t bitpos;
int events = 0;
//A clean XC0324 transmission contains 3 repeats of a message in a single row.
//But in case of transmission or demodulation glitches,
//loop over all rows and check for salvageable messages.
for (r = 0; r < bitbuffer->num_rows; ++r) {
if (bitbuffer->bits_per_row[r] < XC0324_MESSAGE_BITLEN) {
// bail out of this row early
continue; // to the next row
}
// We have enough bits so search for a message preamble followed by
// enough bits that it could be a complete message.
bitpos = 0;
while ((bitpos = bitbuffer_search(bitbuffer, r, bitpos,
(const uint8_t *)&preamble_pattern, 8))
+ XC0324_MESSAGE_BITLEN <= bitbuffer->bits_per_row[r]) {
// DO SOMETHING - at this stage just say here I am :-)
bitbuffer_extract_bytes(bitbuffer, r, bitpos, b, XC0324_MESSAGE_BITLEN);
decoder_output_bitrowf(decoder, b, XC0324_MESSAGE_BITLEN,
"XC0324: Found a message at bitpos %03d", bitpos);
events++;
bitpos += XC0324_MESSAGE_BITLEN;
}
}
return events;
}This introduces some more of the machinery that rtl_433 provides. It :
- loops over the rows in
bitbuffer(there arebitbuffer->num_rowsof them) - checks that there are at least 48 bits in row
r(bitbuffer->bits_per_row[r]) - if so, uses
bitbuffer_searchto try and find the bit position (bitpos) of the start of the preamble byte, followed by another 48 bits (ie the likely start of a message), - does "something*, then
- jumps forward 48 bits and keeps looking for another "preamble + 48 bit message", until there are less than 48 bits left (ie not enough for a message).
The do something as shown here was "have a look at what bitbuffer_search
has found", using a couple more of the functions that rtl_433 provides,
namely :
-
bitbuffer_extract_byteswhich does what its name suggests, and -
decoder_output_bitrowfwhich produces a binary representation of the extracted message, plus aprintfstyle "trace" message saying where the message was found.
You can see the type of output this generated by running:
./rtl_433 -vv -R 120 -Mbits -F csv:temp.vv.csv -K -r <path_to_test_data>/g*.cu8Open the "temp.vv.csv" file as a spreadsheet, and examine the hex and bit patterns.
You can see that for most of the test data, three messages are being detected and hence appear in the output csv file. If I'd guessed too large for the message length, this step would have found only 1 repeat and many unused bytes at the end of the package; too small a guess and I'd have found the 3 repeats OK, but been left with a few unused bytes after all 3 repeats (which would have been a strong hint to trial increasing the message length!).
The 3 repeats were far from obvious before, because the 2 bit gap between the 1st and 2nd repeat of the message made the hex values in the middle of the message look quite different from those at the beginning (1st repeat) and end (3rd repeat) of the transmission.
In this simulated output I have "cheated" for you, and included a message containing
the decoded temperature value. In real life, at this stage of the process,
I didn't know how to do that, so instead I had to manually edit (or for later
cycles, cut and paste) in the observed temperature value that I had written
down for each file name as it was saved. Reminder : The -K FILE (or just
-K) run time option puts the file name in the spreadsheet's tag column for you.
(You did write down exactly what the value showing on the Digitech XC-0322
monitor was for each test data file as it was saved didn't you! I learnt
that lesson the hard way).
Since I have data from 2 sensors, the sensor id field should have (only) two values - and looking at the spreadsheet, byte 1 is an obvious candidate. (The test data repository contains some files saved by another contributor, so there are three different sensor ids there).
The prototype decoder is easily modified to examine this, by making the following alterations to the do something loop:
char id [4] = {0};
...
bitbuffer_extract_bytes(bitbuffer, r, bitpos, b, XC0324_MESSAGE_BITLEN);
// Extract the id as hex string
snprintf(id, 3, "%02X", b[1]);
decoder_output_bitrow_debugf(decoder, b, XC0324_MESSAGE_BITLEN,
"XC0324: bitpos %03d : id %s", bitpos, id);Recompiling and rerunning confirms that (as was already evident) only 2 values for this field appear in (almost) all of the messages from all the transmissions. In the case of the data I collected myself, those values were "56" and "64", with a rare "32" appearing once or twice. Thinking ahead, it is worth taking note of messages where an "unusual" value ("32) appears, since they may have a transmission error - which hopefully the error checking bytes (once I find them) will detect.
Now for a bigger challenge, deciphering the temperature field.
Finding the field itself wasn't too hard. The temperature field should change a lot (because there are lots of different observed temperatures in the test data). byte 2 is an obvious candidate. (And with the logic of hindsight, as explained above, probably part of byte 3 as well).
The tricky question was "how to extract the temperature from these bytes?" This was where editing (or pasting) the true observed temperatures into the rows of the spreadsheet, then sorting by observed temperature became invaluable.
If you ran earlier,
./rtl_433 -vv -R 120 -F csv:temp.vv.csv -K -r <path_to_test_data>/g*.cu8you will recall that the debug message it produces cheats, and includes the final "true" temperature in the "trace" message. This simulates editing (or cut and pasting) the true values into the spreadsheet so you can to try out this reverse engineering step yourself, should you want to. Simply sort the data in the spreadsheet by the "trace" message.
I had numerous (failed) attempts before I finally spotted the pattern. For each failed (exploratory) attempt I coded what I hoped was the right decode, output it via a "trace" message, and compared it with the true observed value which I had manually entered into the spreadsheet each time. Here (compressed into a single decode function, in reality I had multiple separate attempts) are a small sample of them.
double temperature;
uint16_t wrong_temp1 = (uint16_t) b[2];
int16_t wrong_temp2 = (int16_t) b[2];
uint16_t wrong_temp3 = (uint16_t) reverse8(b[2]);
uint16_t temp = ((uint16_t)(reverse8(b[3]) & 0x0f) << 8) | reverse8(b[2]) ;
temperature = (temp / 10.0) - 40.0 ;
decoder_output_bitrowf(decoder, b, XC0324_MESSAGE_BITLEN,
"Temps \"OK\" %05.1f C :bad %04d : %04d : %04d :bitpos %03d",
temperature, wrong_temp1, wrong_temp2, wrong_temp3, bitpos);I have also included the correct decode of temperature (labelled "OK" in the output) for comparison.
Tip : Sort the csv file by observed temperature value. It makes sequential patterns in the data fields much easier to spot.
It probably looks as if the correct decode for the temperature bytes appeared out of nowhere.
But if you :
- sort the output by observed temperature value, and
- focus on those test data rows where there is a run of consecutive temperatures (look around 11.2 or 14.1 degrees), and
- stare at the bit patterns for long enough,
you can spot that the first bit in byte 2 changed (by 1) every time the temperature increased by 0.1 degrees C.
Eventually I realised that this suggested something being transmitted least significant bit first (ie the familiar - once I recognised it - issue of "endianess").
A bit of searching revealed the reverse8
function in the rtl_433 codebase designed exactly for reversing
the order of bits in a byte. This lead to wrong_temp3 which
moves by the right amount and in the right direction every time the
temperature incremements by 0.1 degrees C, albeit offset by a constant(ish)
amount from the true value. A bit more experimentation and
thought, and finally realising that part of byte 3 was also involved
and the temperature field encoding was solved.
Once the sync byte has been found and the messages are all nicely aligned to start from the same bit, it is clear that byte 4 is constant in almost all my observations. Not much I can deduce about it (except to speculate it might be battery status?).
byte 5 is left, at the end of each message, just where I might expect to find the error check. (Aside, luckily the XC-0324 doesn't seem to use byte level parity checks, otherwise it would have taken me much longer to spot the reversed / "least significant bit first" pattern for the temperature field!).
Initially I thought it might be some form of CRC (Cyclic Redundancy Check)
check on the message.
And rtl_433 provides a number of functions for calculating CRC's.
See include/util.h
for details.
So I perused other device decoders, did a quick tally to find out which CRC
methods and algorithms seemed to be popular, and tried them out on
the XC-0324 message. All to no avail!
Finally I thought to look at the pattern of byte 5 from a single sensor (ie where the sensor id was constant!) as the temperature changed by 0.1 C (so only 1 or 2 bits in the whole message had changed.) And noticed that there was a sequential pattern (but with jumps) in the values of byte 5.
Since my understanding is that CRC are designed to NOT produce sequential check values like that, I started looking at checksum type algorithms.
Byte 5 tended to have runs where it decreased as the temperature
increased by 0.1 C (ie 1 or 2 bit change in the temperature field) - but
then it would jump by 8 or 16. Simple addition of bytes 0 to 4 didn't
work, but after some more gazing at bit patterns, I realised that
"binary addition WITHOUT carry" (aka XORing bytes 0 to 4) generated
(most of) my byte 5 values. And when the XORing didn't work, often
the calculated temperature did not equal the observed one.
Hence the checksum was doing its job, flagging a corrupted message.
rtl_433 provides a utility function for calculating this type
of checksum, so the decode is easily specified as :
uint8_t chksum;
chksum = xor_bytes(b, 5) // this should == b[5] for a good message
// or even better
chksum = xor_bytes(b, 6) // should == Ox00 for a good messageHaving easy access to the three repeats of a single message stacked one above the other in the spreadsheet made finding "bad" checksums easier - if all 3 agreed, the message was probably good! If 2 agreed and 1 was different, the different one was (probably) a corrupted message. That's why, in the temperature decoding "trace" messages, I labelled the values as "OK" not as "correct" - some of them (where there had been a transmission glitch) were not actually correct. But usually, over the 3 repeats of the message, there was at least one true value.
Once all the fields had been deciphered, all that remained was to adjust the decoder to output them in production mode.
Although rtl_433 takes care of the actual outputting of information,
in whatever format(s) the user specifies at run time, decoder functions
do have to tell rtl_433 what information to put into
output files. They do that using generic functions from the rtl_433 codebase.
The main functions used in production systems are:
-
data_make: creates adata_tstructure (essentially a list of field names and values that are to be output - the person writing the decoder specifies the names and values in the call todata_make) -
data_appendanddata_prepend: used to a much lesser extent, these add additional entries into thedata_tlist, and -
decoder_output_data: passes the current instance of adata_tstructure back tortl_433for output processing.
You can find more about these functions by reading the documentation / comments in the header files :
-
/include/data.hfor thedata_trelated functions, and -
/include/decoder_util.hfor the output functions.
At the time of writing, rtl_433 is under active development,
so there is a slight possibility that the locations might change.
If these file names are out of date, just browse the contents of the
/include/ directory.
data_make can seem a bit overwhelming at first (it is very flexible),
but looking at simple examples helped me figure it out. To show a
typical usage of data_make and decoder_output_data
here are what appears in an almost final version of xc0324_callback:
data_t *data = NULL;
...
data = data_make(
"model", "Device Type", DATA_STRING, "Digitech XC0324",
"id", "ID", DATA_STRING, id,
"temperature_C", "Temperature C", DATA_FORMAT, "%.1f", DATA_DOUBLE, temperature,
"flags", "Constant ?", DATA_INT, flags,
"checksum_status","Checksum status", DATA_STRING, chksum ? "Corrupted" : "OK",
"mic", "Integrity", DATA_STRING, "CHECKSUM",
NULL);
...
decoder_output_data(decoder, data);
return events;On each line, commas separate each entry. The :
- first entry is the name to be used in the
rtl_433output file, - second entry is a pretty title for that field
- "third" entry tells
rtl_433what type of field to expect, and - "fourth" entry tells
rtl_433which C variable from your device decoder
contains the value of the output field. - "third" and "fourth" are in quotes, because other optional entries
can occur after the second entry, and before the "third" entry, eg
DATA_FORMAT, "<a printf style data format specification>",. So really "third" means "second last", and "fourth" means "last".
The final line is a NULL, which is what rtl_433 uses to terminate
the data_t structure (essentially a linked list) it creates for you.
With that included in the decoder function, rtl_433 is
ready to produce any output format (key/value, csv, json,...)
that the user specifies at run time.
Well, almost any output type. csv output is a bit special. For reasons I
won't explain (not complicated, just irrelevant) developers also have
to tell rtl_433 which of the fields should appear in what order in any
csv output file. This is done by creating a NULL terminated list of
field names, and mentioning that in the r_device definition
as follows :
// List of fields to appear in the `-F csv` output
static char *output_fields[] = {
"model",
"id",
"temperature_C",
"flags",
"checksum_status",
"mic",
NULL
};
r_device digitech_xc0324 = {
.name = "Digitech XC-0324 temperature sensor",
...
.fields = output_fields,
};You may have noticed a return events; statement slipped in after the
decoder_output_data call. A decoder function (eg xc0324_callback)
is supposed to return the number of messages it has successfully
decoded. Since rtl_433 could be be trying upwards of 100 different
decoders to see if any of them can recognise the transmission, it is pretty
important to return 0; (ie message not recognised) as soon as that can
be determined, or return n, where n is greater than zero if
the decoder does recognise the message.
I put all that together into my device callback handler, and the job was done :-)
My final device decoder (callback handler) can be see in
rtl_433/src/devices/digitech_xc0324.c
It looks a bit more complex than the explanation developed above - but that
is mostly because of the debug and trace messages that have been left in
the code so you can simulate different stages of the deciphering process,
as described earlier. The trace messages are wrapped in if blocks
similar to :
if (decoder->verbose == 1) {
// Output a -vv level message
decoder_output_bitrowf(decoder, b, XC0324_MESSAGE_BITLEN,
"checksum = 0x%02X not 0x00 <- XC0324:vv row %d bit %d",
chksum, row, bitpos);
}decoder -> verbose is a variable which is controlled by the -v run time
flags :
-
-vvresults indecoder -> verbosehaving value1, -
-vvvresults indecoder -> verbosehaving value2, and -
-vvvvresults indecoder -> verbosehaving value3
There are definitely cleverer ways of finding the fields and deciphering their encoding (for example I've seen reference to using genome sequencing software to decipher where the fields are - sounds really neat and certainly something that would be fun to research later). And more complex messages (eg more fields, perhaps security and even encryption issue to consider).
But hopefully this rather long exposition is enough to get you started on a career as an rtl_433 device reverse engineer.
Finally, I must finish with a BIG thank you to the very helpful and
patient folk responsible for rtl_433.
My initial version of a decoder was a messy and rather "hackish" mix of C code
with "raw" printf statements and shell scripts, with redirections and filters
to create the csv output files that I used while deciphering the XC-0324 message
protocol.
@zuckschwerdt patiently (and very responsively) coached and tutored me, tidied up
my code and concepts enormously, and even developed and added extra utility
functions to the rtl_433 codebase so that now it is possible to follow the
reverse-engineering approach described herein using only the inbuilt features
of rtl_433.
I am truly appreciative of the help I received (and it goes without saying that any mistakes that remain are my responsibility alone)