Smart meter part 1: Getting the meter data
Tag along while I show you how I got access to my realtime electricity usage connecting to my smart meter's HAN port.
| Published: | Mon, July 22, 2019, 10:30 |
| Category: |
Software development
|
| Tag: |
Smart meter
|
By now all electricity consumers in Norway are supposed to have got their electricity meters replaced with new smart meters. (The smart meters are often referred to as AMS (Advanced Metering System)). The new smart meters register electricity consumption at least every hour and automatically sends it to the power company. There's no more use for manually reporting the usage.
What's interesting is that the new smart meters all come with a so-called HAN port (short for Home Area Network). Using that port it's possible to get full access to your own electricity usage realtime. While I'm sure great services (and APIs) for using this data will be provided by both my energy company and third party vendors I didn't want to sit back and wait. (You can today get access to a bit of delayed hourly usage if you log in to https://plugin.elhub.no/. They also have some nice Ajax calls which are easy to understand and tweak.)
Step 1 - opening the HAN port 🔗
By default the physical HAN ports of the smart meters are closed off and not sending any data. All you need to do is to contact customer support at your power company and they'll quickly open it remotely.
My power company - Norgesnett - used almost a month to open it up as they said the newly installed meter had to first be registered in some computer system. Of course I also noticed a pretty bad security vulnerability while at it.
Step 2 - getting the hardware 🔗
The smart meters use the M-Bus standard for the physical data transfer. So to read the data stream you need to get some kind of M-Bus converter. The smart meters act as a so-called master and the receiver must be a slave. The master gives enough power to run a slave.
Hardware - first try 🔗
From what I read from the forums a lot of people are using - successfully - this (or similar) M-Bus to USB master/slave found on AliExpress, but for me and others it didn't work. I received shorter packages than expected and only the first part of it was readable for me.
Not being an electrical engineer and not knowing how to debug or resolve this I just threw money at the problem and bought another converter.
Hardware - second try 🔗
Another commonly used M-Bus to USB master/slave from eBay (the exact same product from the same seller no longer exists, but it looks like this and can be found doing a search) did the trick for me. I connected it to my good old Raspberry Pi (Model B Rev 2). The HAN port in the smart meters has a RJ-45 connector with the signal being transmitted on pin 1 + 2. So I just used an old network cable to connect the smart meter and converter.
Step 3 - reading raw data 🔗
Python is not my mother tongue, but it's a language I really like and enjoy writing. It's almost always available on whatever system you're on, and the standard library is pretty extensive. Do a simple pip install pyserial and you're ready to read data from the USB port.
The serial port settings for the data stream for me was 2400 baud, parity bit none and byte size 8 bits.
So doing something similar to this I got the raw data stream (code works in both Python version 2.7 and 3.4):
import serial
import codecs
import sys
ser = serial.Serial(
port='/dev/ttyUSB0',
baudrate=2400,
parity=serial.PARITY_NONE,
stopbits=serial.STOPBITS_ONE,
bytesize=serial.EIGHTBITS,
timeout=4)
print("Connected to: " + ser.portstr)
while True:
bytes = ser.read(1024)
if bytes:
print('Got %d bytes:' % len(bytes))
bytes = ('%02x' % int(codecs.encode(bytes, 'hex'), 16)).upper()
bytes = ' '.join(bytes[i:i+2] for i in range(0, len(bytes), 2))
print(bytes)
else:
print('Got nothing')
It would output something similar to this (I have anonymized the data a bit):
Connected to: /dev/ttyUSB0 Got 228 bytes: 7E A0 E2 2B 21 13 23 9A E6 E7 00 0F 00 00 00 00 0C 07 E3 06 12 02 14 2F 32 FF 80 00 80 02 19 0A 0E 4B 61 6D 73 74 72 75 70 5F 56 30 30 30 31 09 06 01 01 00 00 05 FF 0A 10 32 32 30 30 35 36 37 32 32 33 31 39 37 37 31 34 09 06 01 01 60 01 01 FF 0A 12 36 38 34 31 31 33 31 42 4E 32 34 33 31 30 31 30 34 30 09 06 01 01 01 07 00 FF 06 00 00 06 A7 09 06 01 01 02 07 00 FF 06 00 00 00 00 09 06 01 01 03 07 00 FF 06 00 00 00 00 09 06 01 01 04 07 00 FF 06 00 00 01 E0 09 06 01 01 1F 07 00 FF 06 00 00 00 88 09 06 01 01 33 07 00 FF 06 00 00 02 36 09 06 01 01 47 07 00 FF 06 00 00 00 6D 09 06 01 01 20 07 00 FF 12 00 EB 09 06 01 01 34 07 00 FF 12 00 EB 09 06 01 01 48 07 00 FF 12 00 EB 83 77 7E
Step 4 - understanding OBIS 🔗
So what are those bytes coming from the HAN port? They are following the DLMS (Device Language Message Specification) protocol and are sent inside HDLC frames and contains OBIS (Object Identification System) codes that describes the electricity usage. Everything is part of IEC 62056 which is a set of standards for electricity metering data exchange.
How often the messages arrives varies from one meter vendor to another. The same goes for the actual format of the messages. I don't know if there are any other vendors, but at least Aidon, Kaifa and Kamstrup have made smart meters for the Norwegian market, and they all provide documentation for their own OBIS messages.
On my Kamstrup meter I get the current power used every 10 second + the total kWh usage every hour.
To really understand the HDLC and OBIS codes you need dig into different sources around the Internet, but the Norwegian forums at hjemmeautomasjon.no is a great source of information. There are so many knowledgeable people sharing their work and helping each other out.
Realtime usage - every 10 second 🔗
I assume I haven't got everything figured out and there are likely some errors, but this is my interpretation of the message:
Header: 7E <-- Frame start flag A <-- 4 bits, A = 0b1010 = frame format type 3 0E2 <-- 1 bit, segmentation bit + 11 bits, frame length sub-field, 0xE2 = 226 bytes (excluding opening and closing frame flags) 2B <-- Destination address, 1 bit, 0b1 = unicast + 6 bit, node address, 0b010101 = 21 + 1 bit, address size, 0b1 = 1 byte 21 <-- Source address, 1 bit, 0b1 = unicast + 6bit, node address, 0b010000 = 16 + 1 bit, address size, 0b1 = 1 byte 13 <-- Control field 23 9A <-- Header check sequence (HCS) field, CRC-16/X-25 Information: E6 <-- Destination LSAP E7 <-- Source LSAP, LSB = 0b1 = command 00 <-- LLC Quality 0F <-- LLC Service Data Unit 00 00 00 00 <-- "Long-Invoke-Id-And-Priority"? 0C <-- string length?, 0C = 12 07 E3 <-- Full year, 0x07E3 = 2019 06 <-- Month, June 12 <-- Day of month, 0x12 = 18 02 <-- Day of week, Tuesday 14 <-- Hour of day, 0x14 = 20 2F <-- Minute of hour, 0x2F = 47 32 <-- Second of minute, 0x32 = 50 FF <-- Hundredths of second, 0xFF = not specified 80 00 <-- Deviation (offset from UTC), 0x8000 = not specified 80 <-- Clock status, 0x80 = 0b10000000, MSB 1 = summer time 02 <-- struct 19 <-- 0x19 = 25 elements 0A <-- visible-string 0E <-- string length 0x0E = 14 bytes 4B 61 6D 73 74 72 75 70 5F 56 30 30 30 31 <-- OBIS List Version Identifier, Kamstrup_V0001 09 <-- octet-string 06 <-- string length, 0x06 = 6 bytes 01 01 00 00 05 FF <-- OBIS for Meter ID, 1.1.0.0.5.255 0A <-- visible-string 10 <-- string length, 10 = 16 bytes 32 32 30 30 35 36 37 32 32 33 31 39 37 37 31 34 <-- Meter ID, altered 09 <-- octet-string 06 <-- string length, 0x06 = 6 bytes 01 01 60 01 01 FF <-- OBIS for meter type, 1.1.96.1.1.255 0A <-- visible-string 12 <-- string lenth, 0x12 = 18 bytes 36 38 34 31 31 33 31 42 4E 32 34 33 31 30 31 30 34 30 <-- Meter type, 6841131BN243101040) 09 <-- octet-string 06 <-- string length, 0x06 = 6 bytes 01 01 01 07 00 FF <-- OBIS for Active Power +, 1.1.1.7.0.255 06 <-- unsigned, 4 bytes 00 00 06 A7 <-- 0x06A7 = 1703 Watt 09 <-- octet-string 06 <-- string length, 0x06 = 6 bytes 01 01 02 07 00 FF <-- OBIS for Active Power -, 1.1.2.7.0.255 06 <-- unsigned, 4 bytes 00 00 00 00 <-- 0 Watt 09 <-- octet-string 06 <-- string length, 0x06 = 6 bytes 01 01 03 07 00 FF <-- OBIS for Reactive Power +, 1.1.3.7.0.255 06 <-- unsigned, 4 bytes 00 00 00 00 <-- 0 Watt 09 <-- octet-string 06 <-- string length, 0x06 = 6 bytes 01 01 04 07 00 FF <-- OBIS for Reactive Power -, 1.1.4.7.0.255 06 <-- unsigned, 4 bytes 00 00 01 E0 <-- 0x01E0 = 480 Watt 09 <-- octet-string 06 <-- string length, 0x06 = 6 bytes 01 01 1F 07 00 FF <-- OBIS for L1 Current, 1.1.31.7.0.255 06 <-- unsigned, 4 bytes 00 00 00 88 <-- 1.36 Ampere 09 <-- octet-string 06 <-- string length, 0x06 = 6 bytes 01 01 33 07 00 FF <-- OBIS for L2 Current, 1.1.51.7.0.255 06 <-- unsigned, 4 bytes 00 00 02 36 <-- 5.66 Ampere 09 <-- octet-string 06 <-- string length, 0x06 = 6 bytes 01 01 47 07 00 FF <-- OBIS for L3 Current, 1.1.71.7.0.255 06 <-- unsigned, 4 bytes 00 00 00 6D <-- 1.09 Ampere 09 <-- octet-string 06 <-- string length, 0x06 = 6 bytes 01 01 20 07 00 FF <-- OBIS for L1 Voltage, 1.1.32.7.0.255 12 <-- unsigned, 2 bytes 00 EB <-- 235 Volt 09 <-- octet-string 06 <-- string length, 0x06 = 6 bytes 01 01 34 07 00 FF <-- OBIS for L2 Voltage, 1.1.52.7.0.255 12 <-- unsigned, 2 bytes 00 EB <-- 235 Volt 09 <-- octet-string 06 <-- string length, 0x06 = 6 bytes 01 01 48 07 00 FF <-- OBIS for L3 Voltage, 1.1.72.7.0.255 12 <-- unsigned, 2 bytes 00 EB <-- 235 Volt End: 83 77 <-- Frame check sequence (FCS) field, CRC-16/X-25, altered 7E <-- Frame end flag
For me the most interesting part of this message is the OBIS for Active Power + (1.1.1.7.0.255) which tells how much power - in Watt - that is currently being used. If you have a house that produces electricity and exports it to the grid (e.g. if you have solar cells) the exported power would appear as the OBIS for Active Power - (1.1.2.7.0.255).
Total usage - every hour 🔗
The message appearing hourly is similar to the one that comes every 10 second, but contains a bit more information:
Connected to: /dev/ttyUSB0 Got 302 bytes: 7E A1 2C 2B 21 13 FC 04 E6 E7 00 0F 00 00 00 00 0C 07 E3 07 09 02 14 00 05 FF 80 00 80 02 23 0A 0E 4B 61 6D 73 74 72 75 70 5F 56 30 30 30 31 09 06 01 01 00 00 05 FF 0A 10 32 32 30 30 35 36 37 32 32 33 31 39 37 37 31 34 09 06 01 01 60 01 01 FF 0A 12 36 38 34 31 31 33 31 42 4E 32 34 33 31 30 31 30 34 30 09 06 01 01 01 07 00 FF 06 00 00 01 6C 09 06 01 01 02 07 00 FF 06 00 00 00 00 09 06 01 01 03 07 00 FF 06 00 00 00 00 09 06 01 01 04 07 00 FF 06 00 00 01 42 09 06 01 01 1F 07 00 FF 06 00 00 00 85 09 06 01 01 33 07 00 FF 06 00 00 00 5C 09 06 01 01 47 07 00 FF 06 00 00 00 3F 09 06 01 01 20 07 00 FF 12 00 EB 09 06 01 01 34 07 00 FF 12 00 EB 09 06 01 01 48 07 00 FF 12 00 EB 09 06 00 01 01 00 00 FF 09 0C 07 E3 07 09 02 14 00 05 FF 80 00 80 09 06 01 01 01 08 00 FF 06 00 38 DE 2A 09 06 01 01 02 08 00 FF 06 00 00 00 00 09 06 01 01 03 08 00 FF 06 00 00 00 1F 09 06 01 01 04 08 00 FF 06 00 09 00 85 83 77 7E
I've left out the identical parts of the message:
Header:
[...same as first message...]
Information:
[...same as first message...]
02 <-- struct
23 <-- 0x23 = 35 elements
[...]
4B 61 6D 73 74 72 75 70 5F 56 30 30 30 31 <-- OBIS List Version Identifier, Kamstrup_V0001
[...]
01 01 00 00 05 FF <-- OBIS for Meter ID, 1.1.0.0.5.255
[...]
01 01 60 01 01 FF <-- OBIS for meter type, 1.1.96.1.1.255
[...]
01 01 01 07 00 FF <-- OBIS for Active Power +, 1.1.1.7.0.255
[...]
01 01 02 07 00 FF <-- OBIS for Active Power -, 1.1.2.7.0.255
[...]
01 01 03 07 00 FF <-- OBIS for Reactive Power +, 1.1.3.7.0.255
[...]
01 01 04 07 00 FF <-- OBIS for Reactive Power -, 1.1.4.7.0.255
[...]
01 01 1F 07 00 FF <-- OBIS for L1 Current, 1.1.31.7.0.255
[...]
01 01 33 07 00 FF <-- OBIS for L2 Current, 1.1.51.7.0.255
[...]
01 01 47 07 00 FF <-- OBIS for L3 Current, 1.1.71.7.0.255
[...]
01 01 20 07 00 FF <-- OBIS for L1 Voltage, 1.1.32.7.0.255
[...]
01 01 34 07 00 FF <-- OBIS for L2 Voltage, 1.1.52.7.0.255
[...]
01 01 48 07 00 FF <-- OBIS for L3 Voltage, 1.1.72.7.0.255
[...]
09 <-- octet-string
06 <-- string length, 0x06 = 6 bytes
00 01 01 00 00 FF <-- OBIS for Real Time Clock (RTC), 0.1.1.0.0.255
09 <-- octet-string
0C <-- string length, 0x0C = 12 bytes
07 E3 <-- Full year, 0x07E3 = 2019
07 <-- Month, July
09 <-- Day of month, 9
02 <-- Day of week, Tuesday
14 <-- Hour of day, 0x14 = 20
00 <-- Minute of hour, 0
05 <-- Second of minute, 5
FF <-- Hundredths of second, 0xFF = not specified
80 00 <-- Deviation (offset from UTC), 0x8000 = not specified
80 <-- Clock status, 0x80 = 0b10000000, MSB 1 = summer time
09 <-- octet-string
06 <-- string length, 0x06 = 6 bytes
01 01 01 08 00 FF <-- OBIS for Active energy A+, 1.1.1.8.0.255
06 <-- unsigned, 4 bytes
00 38 DE 2A <-- 0x38DE2A = 3,726,890 = 37,268.90 kWh
09 <-- octet-string
06 <-- string length, 0x06 = 6 bytes
01 01 02 08 00 FF <-- OBIS for Active energy A-, 1.1.2.8.0.255
06 <-- unsigned, 4 bytes
00 00 00 00 <-- 0 kWh
09 <-- octet-string
06 <-- string length, 0x06 = 6 bytes
01 01 03 08 00 FF <-- OBIS for Reactive energy R+, 1.1.3.8.0.255
06 <-- unsigned, 4 bytes
00 00 00 1F <-- 0x1F = 31 = 0.31 kWh?
09 <-- octet-string
06 <-- string length, 0x06 = 6 bytes
01 01 04 08 00 FF <-- OBIS for Reactive energy R-, 1.1.4.8.0.255
06 <-- unsigned, 4 bytes
00 09 00 85 <-- 0x090085 = 589957 = 5,899.57 kWh?
End:
[...same as first message...]
The only part that I really care about is the Active energy A+ (OBIS code 1.1.1.8.0.255), which is total power usage - in kilowatt hour (kWh) - since the installation of the smart meter. Keeping track of this value one knows the hourly power consumption. This is the value you have to pay for. If you produce and exports power it would appear as Active energy A- (1.1.2.8.0.255).
Error detection 🔗
Error detection is supported through cyclic redundancy check (CRC) in both the header and footer of the frame. In the start there is a header check sequence (HCS), and in the end there is a frame check sequence (FCS). The checksum algorithm used is the CRC-16/X-25. There are libraries for all kinds of programming languages implementing all sorts of checksum calculations. I have used the Python library crccheck which provides the class CrcX25 which takes care of this.
Links to more information 🔗
Here are some of the sources of information I've used for deciphering the messages. I'll leave them here for anyone wanting to dive deeper:
- https://www.dlms.com/files/Green-Book-Ed-83-Excerpt.pdf
- https://www.dlms.com/files/Blue-Book-Ed-122-Excerpt.pdf
- https://github.com/roarfred/AmsToMqttBridge/blob/master/Samples/Kaifa/readme.md
- https://github.com/roarfred/AmsToMqttBridge/blob/master/Samples/Kamstrup/obisdata.md
- http://www.interfacebus.com/HDLC_Protocol_Description.html
- https://www.hjemmeautomasjon.no/forums/topic/390-ny-strømmåler-med-han-interface/?page=4
- https://www.nek.no/wp-content/uploads/2018/10/Kamstrup-HAN-NVE-interface-description_rev_3_1.pdf
- https://byggebolig.no/imageoriginals/88b3d1774ecb41e6a3fe067ae9e6a893.pdf
Rounding it up 🔗
That's it. In this post I've shown how I connected to the HAN port of my smart meter, how I read the data and how to transform the byte arrays into meaningful information. In the next post I'll discuss how I store the data and calculate the price of the electricity usage.