-
-
Notifications
You must be signed in to change notification settings - Fork 14
/
payload.html
366 lines (344 loc) · 19.9 KB
/
payload.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="generator" content="rustdoc">
<title>CBOR Payload Formatter for The Things Network</title>
<!-- Begin scripts/articles/*-header.html: Article Header for Custom Markdown files processed by rustdoc, like chip8.md -->
<meta property="og:title"
content="CBOR Payload Formatter for The Things Network"
data-rh="true">
<meta property="og:description"
content="How we decode CBOR Sensor Data inside The Things Network... With a CBOR Payload Formatter"
data-rh="true">
<meta property="og:image"
content="https://lupyuen.github.io/images/payload-title.jpg">
<meta property="og:type"
content="article" data-rh="true">
<link rel="canonical" href="https://lupyuen.org/articles/payload.html" />
<!-- End scripts/articles/*-header.html -->
<!-- Begin scripts/rustdoc-header.html: Header for Custom Markdown files processed by rustdoc, like chip8.md -->
<link rel="alternate" type="application/rss+xml" title="RSS Feed for lupyuen" href="/rss.xml" />
<link rel="stylesheet" type="text/css" href="../normalize.css">
<link rel="stylesheet" type="text/css" href="../rustdoc.css" id="mainThemeStyle">
<link rel="stylesheet" type="text/css" href="../dark.css">
<link rel="stylesheet" type="text/css" href="../light.css" id="themeStyle">
<link rel="stylesheet" type="text/css" href="../prism.css">
<script src="../storage.js"></script><noscript>
<link rel="stylesheet" href="../noscript.css"></noscript>
<link rel="shortcut icon" href="../favicon.ico">
<style type="text/css">
#crate-search {
background-image: url("../down-arrow.svg");
}
</style>
<!-- End scripts/rustdoc-header.html -->
</head>
<body class="rustdoc">
<!--[if lte IE 8]>
<div class="warning">
This old browser is unsupported and will most likely display funky
things.
</div>
<![endif]-->
<!-- Begin scripts/rustdoc-before.html: Pre-HTML for Custom Markdown files processed by rustdoc, like chip8.md -->
<!-- Begin Theme Picker -->
<div class="theme-picker" style="left: 0"><button id="theme-picker" aria-label="Pick another theme!"><img src="../brush.svg"
width="18" alt="Pick another theme!"></button>
<div id="theme-choices"></div>
</div>
<!-- Theme Picker -->
<!-- End scripts/rustdoc-before.html -->
<h1 class="title">CBOR Payload Formatter for The Things Network</h1>
<nav id="rustdoc"><ul>
<li><a href="#whats-a-payload-formatter" title="What’s a Payload Formatter?">1 What’s a Payload Formatter?</a><ul></ul></li>
<li><a href="#cbor-payload-formatter" title="CBOR Payload Formatter">2 CBOR Payload Formatter</a><ul></ul></li>
<li><a href="#configure-payload-formatter" title="Configure Payload Formatter">3 Configure Payload Formatter</a><ul></ul></li>
<li><a href="#run-payload-formatter" title="Run Payload Formatter">4 Run Payload Formatter</a><ul></ul></li>
<li><a href="#whats-next" title="What’s Next">5 What’s Next</a><ul></ul></li>
<li><a href="#notes" title="Notes">6 Notes</a><ul></ul></li></ul></nav><p>📝 <em>18 Oct 2021</em></p>
<p>Suppose we have an <strong>IoT Sensor Device</strong> (like <a href="https://lupyuen.github.io/articles/pinedio2"><strong>PineDio Stack BL604</strong></a>) connected to <strong>The Things Network</strong> (via LoRaWAN)…</p>
<ul>
<li><a href="https://lupyuen.github.io/articles/ttn"><strong>“The Things Network on PineDio Stack BL604 RISC-V Board”</strong></a></li>
</ul>
<p>And our device <strong>transmits Sensor Data</strong> to The Things Network in <strong>CBOR Format</strong> (because it requires fewer bytes than JSON)…</p>
<ul>
<li><a href="https://lupyuen.github.io/articles/cbor"><strong>“Encode Sensor Data with CBOR on BL602”</strong></a></li>
</ul>
<blockquote>
<p><img src="https://lupyuen.github.io/images/grafana-flow3.jpg" alt="IoT Sensor Device transmits CBOR Sensor Data to The Things Network" /></p>
</blockquote>
<p><em>How shall we process the CBOR Sensor Data transmitted by our device?</em></p>
<p>We could let <strong>each Application fetch and decode</strong> the CBOR Sensor Data from The Things Network…</p>
<p><img src="https://lupyuen.github.io/images/payload-flow3.jpg" alt="Each Application fetches and decodes the CBOR Sensor Data from The Things Network" /></p>
<p>Each Application would <strong>decode CBOR into JSON</strong> before processing, like we’ve done for Grafana and Roblox…</p>
<ul>
<li>
<p><a href="https://lupyuen.github.io/articles/grafana"><strong>“Grafana Data Source for The Things Network”</strong></a></p>
</li>
<li>
<p><a href="https://lupyuen.github.io/articles/roblox"><strong>“IoT Digital Twin with Roblox and The Things Network”</strong></a></p>
</li>
</ul>
<p>(Assuming that JSON is supported natively by each Application)</p>
<p><em>Erm this solution doesn’t scale well if we have many Applications…</em></p>
<p>Exactly! For every Application that we add (like Prometheus), we would need to <strong>decode the CBOR Sensor Data into JSON again and again</strong>.</p>
<p>(And some Applications might need code changes to support CBOR)</p>
<p><em>What’s the right solution then?</em></p>
<p>The proper solution is to configure a <strong>Payload Formatter</strong> at The Things Network that will <strong>decode our CBOR Sensor Data into JSON once</strong>…</p>
<p><img src="https://lupyuen.github.io/images/payload-title.jpg" alt="CBOR Payload Formatter for The Things Network" /></p>
<p>And <strong>distribute the Decoded Sensor Data</strong> (as JSON) to all Applications.</p>
<p>Read on to learn how we created a <strong>CBOR Payload Formatter</strong> for The Things Network.</p>
<p><img src="https://lupyuen.github.io/images/payload-formatter.jpg" alt="Payload Formatter" /></p>
<p><a href="https://www.thethingsindustries.com/docs/integrations/payload-formatters/javascript/">(Source)</a></p>
<h1 id="whats-a-payload-formatter"><a class="doc-anchor" href="#whats-a-payload-formatter">§</a>1 What’s a Payload Formatter?</h1>
<p>A <strong>Payload Formatter is JavaScript Code</strong> that runs on the <strong>servers at The Things Network</strong> to decode our LoRaWAN Message Payload. (Which contains our CBOR-Encoded Sensor Data)</p>
<p>Inside the Payload Formatter we need to provide a JavaScript Function named <strong>decodeUplink</strong> that will decode our LoRaWAN Message Payload…</p>
<div class="example-wrap"><pre class="language-javascript"><code>// Decode the Payload in the Uplink Message
function decodeUplink(input) {
// `input.fPort` contains the LoRaWAN Port Number (like 2)
// `input.bytes` contains the LoRaWAN Message Payload bytes like...
// [ 0xa2, 0x61, 0x74, 0x19, 0x12, 0x3d, 0x61, 0x6c, 0x19, 0x0f, 0xa0 ]
// TODO: Data and warnings to be returned to The Things Network
var data = { "t": 4669, "l": 4000 };
var warnings = [];
// Return the decoded data and warnings
return {
data: data,
warnings: warnings
};
}</code></pre></div>
<p>In the above example we return the <strong>Decoded Data</strong> as…</p>
<div class="example-wrap"><pre class="language-json"><code>{
"t": 4669,
"l": 4000
}</code></pre></div>
<p>(Let’s pretend that’s the Sensor Data for our Temperature Sensor and Light Sensor)</p>
<p><em>Can we run any kind of JavaScript in a Payload Formatter?</em></p>
<p>Nope, here are the Rules for Squid Game <em>(oops)</em> <strong>Payload Formatter</strong>…</p>
<ol>
<li>
<p>Only <a href="https://www.ecma-international.org/ecma-262/5.1/"><strong>JavaScript (ECMAScript) 5.1</strong></a> is supported</p>
<p>(a.k.a “Plain Old JavaScript”)</p>
</li>
<li>
<p>Which means we don’t allow <strong>let</strong>, <strong>const</strong> and <strong>Arrow Functions</strong></p>
<p>(Like <code>x => {...}</code>)</p>
</li>
<li>
<p><strong>JavaScript Modules</strong> are not supported</p>
<p>(Like <strong>require</strong> and <strong>import</strong>)</p>
</li>
<li>
<p><strong>Input / Output</strong> are not supported</p>
<p>(<strong>console.log</strong> will fail!)</p>
</li>
</ol>
<p><a href="https://www.thethingsindustries.com/docs/integrations/payload-formatters/javascript/">(More about Payload Formatters)</a></p>
<p><a href="https://docs.helium.com/use-the-network/console/functions/">(Helium has a similar Payload Decoder)</a></p>
<p><img src="https://lupyuen.github.io/images/payload-formatter2.jpg" alt="decodeUplink Function" /></p>
<p><a href="https://www.thethingsindustries.com/docs/integrations/payload-formatters/javascript/#decode-uplink-example-the-things-node">(Source)</a></p>
<h1 id="cbor-payload-formatter"><a class="doc-anchor" href="#cbor-payload-formatter">§</a>2 CBOR Payload Formatter</h1>
<p>Let’s study the JavaScript for our <strong>CBOR Payload Formatter</strong>: <a href="https://github.com/lupyuen/cbor-the-things-network/blob/master/cbor.js#L410-L441"><code>cbor.js</code></a></p>
<div class="example-wrap"><pre class="language-javascript"><code>// The Things Network Payload Formatter for CBOR
// Based on https://github.com/paroga/cbor-js/blob/master/cbor.js
(function(global, undefined) {
function decode(...) { ... }
...
})(this);</code></pre></div>
<p>The script begins by including the entire contents of this <strong>JavaScript Decoder for CBOR</strong>…</p>
<ul>
<li><a href="https://github.com/paroga/cbor-js/blob/master/cbor.js"><strong>paroga/cbor-js</strong></a></li>
</ul>
<blockquote>
<p><img src="https://lupyuen.github.io/images/payload-code3.png" alt="CBOR Payload Formatter" /></p>
</blockquote>
<p>This defines the CBOR Decoder Function <strong>CBOR.decode</strong>, which we’ll call in a while.</p>
<p>(Yep, this decoder is all Plain Old JavaScript)</p>
<p>Next we define the <strong>decodeUplink</strong> function that will be called by The Things Network…</p>
<div class="example-wrap"><pre class="language-javascript"><code>// Decode the CBOR Payload in the Uplink Message
function decodeUplink(input) {
// Data and warnings to be returned to The Things Network
var data = {};
var warnings = [];</code></pre></div>
<p>Soon we shall compose the <strong>Decoded Data and Decoder Warnings</strong> that will be returned to The Things Network.</p>
<p><strong>input.bytes</strong> contains a byte array of CBOR-Encoded Sensor Data…</p>
<div class="example-wrap"><pre class="language-javascript"><code>[ 0xa2, 0x61, 0x74, 0x19, 0x12, 0x3d, 0x61, 0x6c, 0x19, 0x0f, 0xa0 ]</code></pre></div>
<p>We convert it to an <strong>ArrayBuffer</strong>…</p>
<div class="example-wrap"><pre class="language-javascript"><code> // Catch any exceptions and return them as warnings.
try {
// Convert payload bytes to ArrayBuffer.
// `input.bytes` contains CBOR bytes like...
// [ 0xa2, 0x61, 0x74, 0x19, 0x12, 0x3d, 0x61, 0x6c, 0x19, 0x0f, 0xa0 ]
var array = new Uint8Array(input.bytes);
var buf = array.buffer;</code></pre></div>
<p>And we call our <strong>CBOR Decoder</strong> to decode the ArrayBuffer…</p>
<div class="example-wrap"><pre class="language-javascript"><code> // Decode the ArrayBuffer
data = CBOR.decode(buf);
// `data` contains Key-Value Pairs like...
// { "l": 4000, "t": 4669 }</code></pre></div>
<p>In case of errors, we catch them and <strong>return as warnings</strong>…</p>
<div class="example-wrap"><pre class="language-javascript"><code> } catch (error) {
// Catch any exceptions and return them as warnings.
// The Things Network will drop the message if we return errors.
warnings.push(error);
}</code></pre></div>
<p>Finally we return the <strong>Decoded Data and Decoder Warnings</strong> to The Things Network…</p>
<div class="example-wrap"><pre class="language-javascript"><code> // Return the decoded data and decoder warnings
return {
data: data,
warnings: warnings
};
}</code></pre></div>
<p>The <strong>Decoded Data</strong> will look like this…</p>
<div class="example-wrap"><pre class="language-json"><code>{
"t": 4669,
"l": 4000
}</code></pre></div>
<p>And that’s how we decode CBOR Sensor Data in our Payload Formatter!</p>
<p><img src="https://lupyuen.github.io/images/payload-code4.png" alt="decodeUplink Function" /></p>
<h1 id="configure-payload-formatter"><a class="doc-anchor" href="#configure-payload-formatter">§</a>3 Configure Payload Formatter</h1>
<p>Now we <strong>configure The Things Network</strong> with our Payload Formatter…</p>
<ol>
<li>
<p>Log on to The Things Network Console</p>
</li>
<li>
<p>Click <strong>Applications → (Your Application) → Payload Formatters → Uplink</strong></p>
<p><img src="https://lupyuen.github.io/images/payload-config2.png" alt="Configure Payload Formatter" /></p>
</li>
<li>
<p>Set the <strong>Formatter Type</strong> to <strong>JavaScript</strong></p>
</li>
<li>
<p>Copy and paste the contents of <a href="https://github.com/lupyuen/cbor-the-things-network/blob/master/cbor.js"><strong>cbor.js</strong></a>
into the <strong>Formatter Parameter</strong> box</p>
</li>
<li>
<p>Click <strong>Save Changes</strong></p>
</li>
</ol>
<p>Let’s test our CBOR Payload Formatter!</p>
<p><img src="https://lupyuen.github.io/images/ttn-title.jpg" alt="PineDio Stack BL604 RISC-V Board (foreground) talking to The Things Network via RAKWireless RAK7248 LoRaWAN Gateway (background)" /></p>
<h1 id="run-payload-formatter"><a class="doc-anchor" href="#run-payload-formatter">§</a>4 Run Payload Formatter</h1>
<p>To <strong>test our CBOR Payload Formatter</strong>, we need a LoRaWAN Device that will transmit CBOR Payloads to The Things Network.</p>
<p>Today we shall use <a href="https://lupyuen.github.io/articles/pinedio2"><strong>PineDio Stack BL604</strong></a> (pic above)</p>
<ol>
<li>
<p>Follow the instructions below to <strong>build, flash and run</strong> the LoRaWAN Firmware for PineDio Stack…</p>
<p><a href="https://lupyuen.github.io/articles/tsen#appendix-build-and-run-lorawan-firmware"><strong>“Build and Run LoRaWAN Firmware”</strong></a></p>
</li>
<li>
<p>Enter the command to <strong>transmit Temperature Sensor Data</strong> to The Things Network, encoded with CBOR…</p>
<p><a href="https://lupyuen.github.io/articles/tsen#run-the-lorawan-firmware"><strong>“Run the LoRaWAN Firmware”</strong></a></p>
</li>
<li>
<p>Log on to The Things Network Console</p>
</li>
<li>
<p>Click <strong>Applications → (Your Application) → Live Data</strong></p>
</li>
<li>
<p>Our <strong>Decoded Sensor Data</strong> should appear in the Live Data Table…</p>
<div class="example-wrap"><pre class="language-json"><code>{ "l": 4000, "t": 4669 }</code></pre></div>
<p><img src="https://lupyuen.github.io/images/payload-ttn3.png" alt="Decoded Sensor Data in the Live Data Table" /></p>
</li>
<li>
<p>Click on a message in the Live Data Table.</p>
<p>We should see the <strong>decoded_payload</strong> field with our Decoded Sensor Data…</p>
<div class="example-wrap"><pre class="language-json"><code>"decoded_payload": {
"l": 4000,
"t": 4656
}</code></pre></div>
<p>(Our decoder warnings will also appear here)</p>
</li>
<li>
<p><strong>decoded_payload</strong> will be visible to Applications that are connected to The Things Network via…</p>
<ul>
<li>
<p><a href="https://github.com/lupyuen/cbor-the-things-network#mqtt-log"><strong>MQTT</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/cbor-the-things-network#storage-log"><strong>Storage API</strong></a> (over HTTPS)</p>
</li>
<li>
<p>And <a href="https://www.thethingsnetwork.org/docs/applications-and-integrations/"><strong>other supported protocols</strong></a></p>
</li>
</ul>
<p><img src="https://lupyuen.github.io/images/payload-ttn4.png" alt="decoded_payload Field" /></p>
<p><a href="https://github.com/lupyuen/cbor-the-things-network#mqtt-log">(Source)</a></p>
</li>
</ol>
<p>Congratulations, we have successfully decoded our CBOR Message Payload in The Things Network… Ready to be consumed by multiple Applications!</p>
<p>(Like <strong>Prometheus</strong>, the open source Time Series Database for IoT Sensor Data)</p>
<p><img src="https://lupyuen.github.io/images/grafana-flow2.jpg" alt="Storing The Things Network Sensor Data with Prometheus" /></p>
<p><a href="https://lupyuen.github.io/articles/prometheus">(Source)</a></p>
<h1 id="whats-next"><a class="doc-anchor" href="#whats-next">§</a>5 What’s Next</h1>
<p>Now that we can decode CBOR Sensor Data in The Things Network, it becomes a lot easier to <strong>ingest the Sensor Data into Prometheus</strong> (the open source Time Series Database).</p>
<p>In the next article we shall build an <strong>IoT Monitoring System</strong> that stores the <strong>Sensor Data with Prometheus</strong> and visualises the data in a <strong>Grafana Dashboard</strong>…</p>
<ul>
<li><a href="https://lupyuen.github.io/articles/prometheus"><strong>“Monitor IoT Devices in The Things Network with Prometheus and Grafana”</strong></a></li>
</ul>
<p><img src="https://lupyuen.github.io/images/prometheus-grafana4.png" alt="Monitoring Devices on The Things Network with Prometheus and Grafana" /></p>
<p>Many Thanks to my <a href="https://lupyuen.github.io/articles/sponsor"><strong>GitHub Sponsors</strong></a> for supporting my work! This article wouldn’t have been possible without your support.</p>
<ul>
<li>
<p><a href="https://lupyuen.github.io/articles/sponsor">Sponsor me a coffee</a></p>
</li>
<li>
<p><a href="https://www.reddit.com/r/TheThingsNetwork/comments/qafzu4/cbor_payload_formatter_for_the_things_network/?utm_source=share&utm_medium=web2x&context=3">Discuss this article on Reddit</a></p>
</li>
<li>
<p><a href="https://lupyuen.github.io/articles/book">Read “The RISC-V BL602 / BL604 Book”</a></p>
</li>
<li>
<p><a href="https://lupyuen.github.io">Check out my articles</a></p>
</li>
<li>
<p><a href="https://lupyuen.github.io/rss.xml">RSS Feed</a></p>
</li>
</ul>
<p><em>Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…</em></p>
<p><a href="https://github.com/lupyuen/lupyuen.github.io/blob/master/src/payload.md"><code>lupyuen.github.io/src/payload.md</code></a></p>
<h1 id="notes"><a class="doc-anchor" href="#notes">§</a>6 Notes</h1>
<ol>
<li>
<p>This article is the expanded version of <a href="https://twitter.com/MisterTechBlog/status/1448846003608567809">this Twitter Thread</a></p>
</li>
<li>
<p><strong>Be careful when decoding CBOR</strong> with our Payload Formatter!</p>
<p>Some Data Types in CBOR don’t exist in JSON, and might throw errors.</p>
<p><a href="https://json.nlohmann.me/features/binary_formats/cbor/#deserialization">(See this mapping of CBOR to JSON types)</a></p>
<p>When we’re encoding Sensor Data with CBOR, use Data Types that are supported in JSON: Integers, Floats, Strings and Maps.</p>
</li>
<li>
<p>Here’s a tip for <strong>debugging Payload Formatters</strong>…</p>
<p>We can’t use <strong>console.log</strong> for logging Debug Messages, but we can return Debug Messages as <strong>Decoder Warnings</strong>…</p>
<div class="example-wrap"><pre class="language-javascript"><code>// Decode the payload in the Uplink Message
function decodeUplink(input) {
// Data and warnings to be returned to The Things Network
var data = {};
var warnings = [];
// Omitted: Decode the payload
...
// For Debugging: Show the contents of
// `input.bytes` in the decoder warnings
warnings.push(input.bytes);
// Return the decoded data and decoder warnings
return {
data: data,
warnings: warnings
};
}</code></pre></div>
<p>The Decoder Warnings (and our Debug Messages) will be shown when we click on a message in <strong>Applications → (Your Application) → Live Data</strong>.</p>
</li>
</ol>
<!-- Begin scripts/rustdoc-after.html: Post-HTML for Custom Markdown files processed by rustdoc, like chip8.md -->
<!-- Begin Theme Picker and Prism Theme -->
<script src="../theme.js"></script>
<script src="../prism.js"></script>
<!-- Theme Picker and Prism Theme -->
<!-- End scripts/rustdoc-after.html -->
</body>
</html>