p1_listener

Smart meters in compliance with DSMR or ESMR standards are fitted with a P1-port. Using a serial converter cable this port can be used to listen to the broadcasts initiated by the meter. The data can then be processed by (in this case) Python for further handling. The Python script in this repository is currently set up to forward specific data as an IP-telegram and/or a WebSocket message, and can write to a file or a database using RRD-tool.

Requirements

• Raspberry Pi
  Any Raspberry Pi should do, I used models zero, 2B, 3B and 4.
• Connector cable P1 – USB (I used the P1 Converter Cable v2 from http://www.smartmeterdashboard.nl/webshop)
• Python3.6+ with modules as described below

Setting up the Raspberry Pi

Make sure the latest updates are installed:

sudo apt update
sudo apt full-upgrade -y
sudo apt clean
sudo reboot

Set timezone to UTC

Set the timezone to UTC to avoid missing data or gaps in data when changing to or from Daylight Saving time.

sudo raspi-config

• Select ‘Localisation Options’
  (use the up/down cursor to select a row, then the right cursor to highlight ‘select’, then press <enter>)
  Select ‘Timezone’
  Select ‘None of the above’
  Select ‘UTC’
  Exit with ‘OK’ and ‘Finish’.

Create directories for organizing files

The directory graphs is used with RRDtool, websocketdata is only used when WebSockets are used.

mkdir dsmr graphs local_lib logs websocketdata

Configure for Smart Meter data reception

• Install the serial tool ‘cu minicom’:

sudo apt install cu minicom -y

• Verify that the communication is successful:
• For meters up to and including version 3:

cu -l /dev/ttyUSB0 -s 9600 --parity=even

• For meter versions 4 and 5:

cu -l /dev/ttyUSB0 -s 115200 --parity=none

• To exit type: <enter>~.
  (so: the enter key, then the tilde followed by the dot)
  Note: there is a slight delay in disconnecting after hitting this key combination.
• The result should be a repeating series of data sets similar to this:

  /XMX5LGBBLA4402915485
  1-3:0.2.8(50)
  0-0:1.0.0(200310123610W)
  0-0:96.1.1(4530303435303034303436393339353137)
  1-0:1.8.1(001164.722kWh)
  1-0:1.8.2(000965.026kWh)
  1-0:2.8.1(000252.398kWh)
  1-0:2.8.2(000589.121kWh)
  0-0:96.14.0(0002)
  1-0:1.7.0(00.087kW)
  1-0:2.7.0(00.000kW)
  0-0:96.7.21(00002)
  0-0:96.7.9(00000)
  1-0:99.97.0(0)(0-0:96.7.19)
  1-0:32.32.0(00001)
  1-0:32.36.0(00000)
  0-0:96.13.0()
  1-0:32.7.0(231.0V)
  1-0:31.7.0(000A)
  1-0:21.7.0(00.087kW)
  1-0:22.7.0(00.000kW)
  0-1:24.1.0(003)
  0-1:96.1.0(4730303332353635353438333236323137)
  0-1:24.2.1(200310123504W)(01621.615*m3)
  !750D

• Add the library to read the serial port using Python:

sudo apt install python3-serial

Install PIP (if necessary)

If you installed Raspberry Pi OS Lite then it lacks the pip tool. Install it now:

sudo apt install python3-pip -y

Install the python timezone library

pip3 install pytz

Install the colorama library

pip3 install colorama

Optional: Install and configure the round robin database

Install the RRD tool and create a database with the suitable round robin archives.

sudo apt install python3-dev librrd-dev -y

sudo apt install rrdtool -y

sudo pip3 install rrdtool

Build the database(s). Easiest way to create a database is to build and run a script. Sample scripts are rrdcreatescript_elec.sh and rrdcreatescript_gas_xx.sh ('xx' stands for '5m' or '1h').
Scripts to create a gas database are included for capturing data every 5 minutes (DSMRv5 meters) and every hour (DSMR v4 and older meters). The 'create' scripts are similar in setup. The sample script to create the electricity database can be explained as follows:

• Line 1 creates database 'electricity.rrd' in folder '/home/pi/data/'.
  The step size is defined as 300 seconds, so the database expects a value at least every 5 minutes.
• Lines 2-5 define what data needs to be sent to the database. In this case the database expects the four counter values (the meter readings) of the high and low rates for power imported and exported.
• The remaining lines define the data which is to be stored in the database: Minimum, Average and Maximum rates. Minimum and maximum are stored at a granularity of 1 hour, 6 hours and 1 day for respectively 1 year, 10 years and 25 years.

The script to create the gas database follows a similar path.
Obviously you can modify the script creation settings as you see fit.
More details on this site.
Sending counter data to the database from within Python looks like this:

from rrdtool import update as rrd_update;

db = '/home/pi/data/electricity.rrd';

ret = rrd_update(db, 'N:%s:%s:%s:%s' %(High_In, Low_In, High_Out, Low_Out));

After sending data for at least two step size intervals you can check the content of the database using this command from the command line:

rrdtool fetch /home/pi/data/electricity.rrd AVERAGE --start -1h --end now

The result would look like this:

               HighIn            LowIn          HighOut           LowOut
1621452300: 1.6989894335e-01 0.0000000000e+00 0.0000000000e+00 0.0000000000e+00
1621452600: 1.6179079386e-01 0.0000000000e+00 0.0000000000e+00 0.0000000000e+00
1621452900: 1.6096484887e-01 0.0000000000e+00 0.0000000000e+00 0.0000000000e+00
1621453200: nan nan nan nan

The first entry on each line is the timestamp. 'nan' means 'not a number', meaning that for instance when creating a graph no data will be shown for that datapoint.
Note that the data in the database does not equal the counter values - the data is converted to a rate for the given period. In the example above, there was no change in counter value for counter values 2-4, which is why the related values in the database show zero rates.

Optional: Install Apache Web Server, Websockets (Autobahn, Twisted)

This section will be added later.

Set up a symlink to enable testing of the library modules

When testing a ‘local_lib’ library module from another directory, the module under certain circumstances requires a symlink to point to the correct folder location. Set up this symlink:

ln -s /home/pi/local_lib/ /home/pi/local_lib/

Create the helper files

The p1_listener script relies on a telegram definition file. This file can be created in two steps, using two scripts:

• Script p1_telegram_discovery.py tries to read the USB0 serial port and capture a single telegram. This telegram is written to a file: /home/pi/dsmr/standard_telegram.txt.
• Script dsmr_definition.py is used to generate the defition of each line of the telegram, so that the p1_listener script 'knows' where to find the information of each OBIS (= Object Identification System) code. Comment out or uncomment the object_list lines in the this script so that only the lines with the OBIS codes that are needed by the p1-listener script will be handled.

Install the local library files

p1_listener uses local libraries that can also be used by other scripts. These library files are expected to be in directory /home/pi/local_lib/.
Library file 'custom_functions.py' contains required libraries. Depending on which output methods are used, one or more of the following library files will also be needed:

• IP_transmit.py
• update_rrd.py
• ws_transmit.py

Install the primary script and it's configuration file

Install the p1_listener.py and p1_listener.config files in the /home/pi directory. Edit the config file to fit the requirements of the installation.
As an editing example, if data is to be sent to a server using an IP telegram, the 'host' and 'timer' entries in the '[IP telegrams]' section must be uncommented as well as at least one obis code.

Run the script

If all went well, you can now run the script. For testing purposes you can start the script with an arbitrary argument. This will run the script in debug mode. For instance, entering on the command line 'python3 p1_listener.py test' will run the script and display all the chosen settings on-screen as well as display the data being sent for the methods selected.
What the script will do:

• Read the configuration file.
• Read the object definitions from the .txt file prepared in the section Create the helper files above.
• Read the DSMR version of the smart meter from the standard telegram .txt file, and set the COM port configurations accordingly.
• Start the infinite loop listening to the p1 telegrams.
• Send data using the selected methods.

Data is transmitted only if the value of the particular variable has changed.
kWh counter values (so: the meter readings) are sent every 5 minutes.
Gas counter value (if used) is sent every time when the meter sends a gas counter update (once per hour for DSMR 4 or lower, and once every 5 minutes for DSMR 5).
If selected, a daily log is written of the counter values at midnight local time.