-
-
Notifications
You must be signed in to change notification settings - Fork 14
/
nuttx2.html
907 lines (867 loc) · 54 KB
/
nuttx2.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="generator" content="rustdoc">
<title>Apache NuttX RTOS on RISC-V: Star64 JH7110 SBC</title>
<!-- Begin scripts/articles/*-header.html: Article Header for Custom Markdown files processed by rustdoc, like chip8.md -->
<meta property="og:title"
content="Apache NuttX RTOS on RISC-V: Star64 JH7110 SBC"
data-rh="true">
<meta property="og:description"
content="(Partially) Booting Apache NuttX Real-Time Operating System on Pine64's Star64 64-bit RISC-V Single-Board Computer, based on StarFive JH7110 SoC"
data-rh="true">
<meta name="description"
content="(Partially) Booting Apache NuttX Real-Time Operating System on Pine64's Star64 64-bit RISC-V Single-Board Computer, based on StarFive JH7110 SoC">
<meta property="og:image"
content="https://lupyuen.github.io/images/nuttx2-title.jpg">
<meta property="og:type"
content="article" data-rh="true">
<link rel="canonical"
href="https://lupyuen.org/articles/nuttx2.html" />
<!-- End scripts/articles/*-header.html -->
<!-- Begin scripts/rustdoc-header.html: Header for Custom Markdown files processed by rustdoc, like chip8.md -->
<link rel="alternate" type="application/rss+xml" title="RSS Feed for lupyuen" href="/rss.xml" />
<link rel="stylesheet" type="text/css" href="../normalize.css">
<link rel="stylesheet" type="text/css" href="../rustdoc.css" id="mainThemeStyle">
<link rel="stylesheet" type="text/css" href="../dark.css">
<link rel="stylesheet" type="text/css" href="../light.css" id="themeStyle">
<link rel="stylesheet" type="text/css" href="../prism.css">
<script src="../storage.js"></script><noscript>
<link rel="stylesheet" href="../noscript.css"></noscript>
<link rel="shortcut icon" href="../favicon.ico">
<style type="text/css">
#crate-search {
background-image: url("../down-arrow.svg");
}
</style>
<!-- End scripts/rustdoc-header.html -->
</head>
<body class="rustdoc">
<!--[if lte IE 8]>
<div class="warning">
This old browser is unsupported and will most likely display funky
things.
</div>
<![endif]-->
<!-- Begin scripts/rustdoc-before.html: Pre-HTML for Custom Markdown files processed by rustdoc, like chip8.md -->
<!-- Begin Theme Picker -->
<div class="theme-picker" style="left: 0"><button id="theme-picker" aria-label="Pick another theme!"><img src="../brush.svg"
width="18" alt="Pick another theme!"></button>
<div id="theme-choices"></div>
</div>
<!-- Theme Picker -->
<!-- End scripts/rustdoc-before.html -->
<h1 class="title">Apache NuttX RTOS on RISC-V: Star64 JH7110 SBC</h1>
<nav id="rustdoc"><ul>
<li><a href="#start-with-qemu-emulator" title="Start with QEMU Emulator">1 Start with QEMU Emulator</a><ul></ul></li>
<li><a href="#print-to-qemu-console" title="Print to QEMU Console">2 Print to QEMU Console</a><ul></ul></li>
<li><a href="#uart-controller-on-star64" title="UART Controller on Star64">3 UART Controller on Star64</a><ul></ul></li>
<li><a href="#risc-v-linux-kernel-header" title="RISC-V Linux Kernel Header">4 RISC-V Linux Kernel Header</a><ul></ul></li>
<li><a href="#start-address-of-nuttx-kernel" title="Start Address of NuttX Kernel">5 Start Address of NuttX Kernel</a><ul></ul></li>
<li><a href="#boot-nuttx-on-star64" title="Boot NuttX on Star64">6 Boot NuttX on Star64</a><ul></ul></li>
<li><a href="#nuttx-fails-to-get-hart-id" title="NuttX Fails To Get Hart ID">7 NuttX Fails To Get Hart ID</a><ul></ul></li>
<li><a href="#risc-v-privilege-levels" title="RISC-V Privilege Levels">8 RISC-V Privilege Levels</a><ul></ul></li>
<li><a href="#downgrade-nuttx-to-supervisor-mode" title="Downgrade NuttX to Supervisor Mode">9 Downgrade NuttX to Supervisor Mode</a><ul></ul></li>
<li><a href="#fix-the-nuttx-boot-code" title="Fix the NuttX Boot Code">10 Fix the NuttX Boot Code</a><ul></ul></li>
<li><a href="#whats-next" title="What’s Next">11 What’s Next</a><ul></ul></li>
<li><a href="#appendix-hart-id-from-opensbi" title="Appendix: Hart ID from OpenSBI">12 Appendix: Hart ID from OpenSBI</a><ul></ul></li>
<li><a href="#appendix-nuttx-in-supervisor-mode" title="Appendix: NuttX in Supervisor Mode">13 Appendix: NuttX in Supervisor Mode</a><ul></ul></li>
<li><a href="#appendix-nuttx-start-address" title="Appendix: NuttX Start Address">14 Appendix: NuttX Start Address</a><ul></ul></li>
<li><a href="#appendix-nuttx-crash-log" title="Appendix: NuttX Crash Log">15 Appendix: NuttX Crash Log</a><ul></ul></li></ul></nav><p>📝 <em>9 Jul 2023</em></p>
<p><img src="https://lupyuen.github.io/images/nuttx2-title.jpg" alt="Pine64 Star64 64-bit RISC-V SBC" /></p>
<p>In this article we’ll boot a tiny bit of <a href="https://nuttx.apache.org/docs/latest/index.html"><strong>Apache NuttX RTOS</strong></a> on the <a href="https://wiki.pine64.org/wiki/STAR64"><strong>Pine64 Star64</strong></a> 64-bit RISC-V Single-Board Computer.</p>
<p>(Based on <a href="https://doc-en.rvspace.org/Doc_Center/jh7110.html"><strong>StarFive JH7110</strong></a>, the same SoC in VisionFive2)</p>
<p><em>What’s NuttX?</em></p>
<p><a href="https://nuttx.apache.org/docs/latest/index.html"><strong>Apache NuttX</strong></a> is a <strong>Real-Time Operating System (RTOS)</strong> that runs on many kinds of devices, from 8-bit to 64-bit.</p>
<p><em>NuttX supports Star64?</em></p>
<p>Nope NuttX won’t run on Star64 yet, we’ll hit some interesting (and highly educational) RISC-V challenges.</p>
<p>But the things that we learn today will be super helpful for <a href="https://lupyuen.github.io/articles/riscv#jump-to-start"><strong>porting NuttX to Star64</strong></a>.</p>
<p>Please join me as we whip up a tiny tasty treat of NuttX on Star64…</p>
<ul>
<li>
<p>Migrate NuttX from <strong>QEMU Emulator</strong> to Real Hardware</p>
</li>
<li>
<p>Log to the <strong>Serial Console</strong> in RISC-V Assembly</p>
</li>
<li>
<p>Trick <strong>U-Boot Bootloader</strong> into thinking we’re Linux</p>
</li>
<li>
<p>Downgrade from Machine to <strong>Supervisor Privilege Level</strong></p>
</li>
<li>
<p>With a little help from <strong>OpenSBI Supervisor Interface</strong></p>
</li>
</ul>
<p><img src="https://lupyuen.github.io/images/riscv-title.png" alt="Apache NuttX RTOS on 64-bit QEMU RISC-V Emulator" /></p>
<h1 id="start-with-qemu-emulator"><a class="doc-anchor" href="#start-with-qemu-emulator">§</a>1 Start with QEMU Emulator</h1>
<p>Earlier we successfully tested <strong>NuttX RTOS on QEMU Emulator</strong> for 64-bit RISC-V (pic above)…</p>
<ul>
<li><a href="https://lupyuen.github.io/articles/riscv"><strong>“64-bit RISC-V with Apache NuttX Real-Time Operating System”</strong></a></li>
</ul>
<p>Let’s run this on Star64! Starting with the <strong>NuttX Boot Code</strong> (in RISC-V Assembly)…</p>
<ul>
<li><a href="https://lupyuen.github.io/articles/riscv#risc-v-boot-code-in-nuttx"><strong>“RISC-V Boot Code in NuttX”</strong></a></li>
</ul>
<p><em>Surely we’ll run into problems?</em></p>
<p>Fortunately we have a <a href="https://lupyuen.github.io/articles/linux#serial-console-on-star64"><strong>Serial Debug Console</strong></a> connected to Star64. (Pic below)</p>
<p>We’ll print some <strong>Debug Logs</strong> as NuttX boots on Star64. Here’s our plan…</p>
<ul>
<li>
<p>Check the <strong>Serial Console on QEMU Emulator</strong>, how it’s wired up</p>
</li>
<li>
<p><strong>Test our Debug Log</strong> on QEMU Emulator</p>
</li>
<li>
<p><strong>Port our Debug Log</strong> to Star64</p>
</li>
</ul>
<p>Our cooking begins…</p>
<p><img src="https://lupyuen.github.io/images/linux-title.jpg" alt="Star64 SBC with Woodpecker USB Serial Adapter" /></p>
<p><a href="https://lupyuen.github.io/articles/linux"><em>Star64 with Woodpecker USB Serial Adapter</em></a></p>
<h1 id="print-to-qemu-console"><a class="doc-anchor" href="#print-to-qemu-console">§</a>2 Print to QEMU Console</h1>
<p><em>We’re printing to the Serial Console on QEMU Emulator…</em></p>
<p><em>What’s the UART Controller in QEMU?</em></p>
<p>We check the <strong>NuttX Build Configuration</strong> for QEMU: <a href="https://github.com/apache/nuttx/blob/master/boards/risc-v/qemu-rv/rv-virt/configs/nsh64/defconfig#L10-L16">nsh64/defconfig</a></p>
<div class="example-wrap"><pre class="language-text"><code>CONFIG_16550_ADDRWIDTH=0
CONFIG_16550_UART0=y
CONFIG_16550_UART0_BASE=0x10000000
CONFIG_16550_UART0_CLOCK=3686400
CONFIG_16550_UART0_IRQ=37
CONFIG_16550_UART0_SERIAL_CONSOLE=y
CONFIG_16550_UART=y</code></pre></div>
<p>This says that QEMU emulates a <a href="https://en.wikipedia.org/wiki/16550_UART"><strong>16550 UART Controller</strong></a>.</p>
<p>And the <strong>Base Address</strong> of QEMU’s UART Controller is <strong><code>0x1000</code> <code>0000</code></strong>.</p>
<p><em>How to print to the 16550 UART Port?</em></p>
<p>We check the <strong>NuttX Driver</strong> for 16550 UART: <a href="https://github.com/apache/nuttx/blob/master/drivers/serial/uart_16550.c#L1539-L1553">uart_16550.c</a></p>
<div class="example-wrap"><pre class="language-c"><code>// Send one byte to 16550 UART
static void u16550_send(struct uart_dev_s *dev, int ch) {
// Fetch the 16550 Struct
FAR struct u16550_s *priv = (FAR struct u16550_s *)dev->priv;
// Print to 16550 UART...
u16550_serialout(
priv, // 16550 Struct
UART_THR_OFFSET, // Offset of Transmit Holding Register
(uart_datawidth_t) ch // Character to print
);
}</code></pre></div>
<p><a href="https://github.com/apache/nuttx/blob/master/drivers/serial/uart_16550.c#L610-L624">(<strong>u16550_serialout</strong> is defined here)</a></p>
<p>To print a character, the driver writes to the UART Base Address <strong><code>0x1000</code> <code>0000</code></strong> at Offset <strong>UART_THR_OFFSET</strong>.</p>
<p>And we discover that <strong>UART_THR_OFFSET</strong> is 0: <a href="https://github.com/apache/nuttx/blob/master/include/nuttx/serial/uart_16550.h#L172-L200">uart_16550.h</a></p>
<div class="example-wrap"><pre class="language-c"><code>#define UART_THR_INCR 0 /* Transmit Holding Register (when DLAB is 0) */
#define UART_THR_OFFSET (CONFIG_16550_REGINCR*UART_THR_INCR)</code></pre></div>
<p>Which means that we can print to QEMU Console by writing to <strong><code>0x1000</code> <code>0000</code></strong>. How convenient!</p>
<div class="example-wrap"><pre class="language-c"><code>// Print `1` to QEMU Console
*(volatile uint8_t *) 0x10000000 = '1';</code></pre></div>
<p><em>What about RISC-V Assembly?</em></p>
<p>This is how we print to QEMU Console in <strong>RISC-V Assembly</strong> (to debug our NuttX Boot Code)…</p>
<div class="example-wrap"><pre class="language-text"><code>/* Load UART Base Address to Register t0 */
li t0, 0x10000000
/* Load `1` to Register t1 */
li t1, 0x31
/* Store byte from Register t1 to UART Base Address, Offset 0 */
sb t1, 0(t0)
/* Load `2` to Register t1 */
li t1, 0x32
/* Store byte from Register t1 to UART Base Address, Offset 0 */
sb t1, 0(t0)
/* Load `3` to Register t1 */
li t1, 0x33
/* Store byte from Register t1 to UART Base Address, Offset 0 */
sb t1, 0(t0)</code></pre></div>
<p><a href="https://github.com/lupyuen2/wip-nuttx/blob/star64/arch/risc-v/src/qemu-rv/qemu_rv_head.S#L71-L93">(Previously here)</a></p>
<p><a href="https://lupyuen.github.io/articles/riscv#other-instructions">(<strong><code>li</code></strong> loads a Value into a Register)</a></p>
<p><a href="https://five-embeddev.com/quickref/instructions.html#-rv32--load-and-store-instructions">(<strong><code>sb</code></strong> stores a byte from a Register into an Address)</a></p>
<p>When we start QEMU Emulator, our RISC-V Assembly prints “<strong><code>123</code></strong>” to the QEMU Console (pic below)…</p>
<div class="example-wrap"><pre class="language-bash"><code>## Remove `-bios none` for newer versions of NuttX
$ qemu-system-riscv64 \
-semihosting \
-M virt,aclint=on \
-cpu rv64 \
-bios none \
-kernel nuttx \
-nographic
123123123123123123112323
NuttShell (NSH) NuttX-12.0.3
nsh> </code></pre></div>
<p>“<strong><code>123</code></strong>” is printed 8 times because QEMU is running with 8 CPUs.</p>
<p>Now we sprinkle some Debug Logs on Star64…</p>
<p><img src="https://lupyuen.github.io/images/riscv-print.png" alt="NuttX prints to QEMU Console" /></p>
<h1 id="uart-controller-on-star64"><a class="doc-anchor" href="#uart-controller-on-star64">§</a>3 UART Controller on Star64</h1>
<p><em>What’s the UART Controller in Star64?</em></p>
<p>Star64 JH7110 uses the <strong>8250 UART Controller</strong>, according to…</p>
<ul>
<li><a href="https://doc-en.rvspace.org/VisionFive2/DG_UART/JH7110_SDK/function_layer.html"><strong>JH7110 UART Developing Guide</strong></a></li>
</ul>
<p>Which is <a href="https://en.wikipedia.org/wiki/16550_UART"><strong>compatible with the 16550 UART Controller</strong></a> used by QEMU.</p>
<p><em>But what’s the UART Base Address for Star64?</em></p>
<p>UART0 Base Address is at <strong><code>0x1000</code> <code>0000</code></strong>, according to…</p>
<ul>
<li>
<p><a href="https://doc-en.rvspace.org/JH7110/TRM/JH7110_TRM/system_memory_map.html"><strong>JH7110 System Memory Map</strong></a></p>
</li>
<li>
<p><a href="https://doc-en.rvspace.org/VisionFive2/DG_UART/JH7110_SDK/general_uart_controller.html"><strong>JH7110 UART Device Tree</strong></a></p>
</li>
<li>
<p><a href="https://doc-en.rvspace.org/JH7110/Datasheet/JH7110_DS/uart.html"><strong>JH7110 UART Datasheet</strong></a></p>
</li>
</ul>
<p><em>Isn’t that the same UART Base Address as QEMU?</em></p>
<p>Yep! Earlier we saw the <strong>UART Base Address</strong> for NuttX QEMU: <a href="https://github.com/apache/nuttx/blob/master/boards/risc-v/qemu-rv/rv-virt/configs/nsh64/defconfig#L10-L16">nsh64/defconfig</a></p>
<div class="example-wrap"><pre class="language-text"><code>CONFIG_16550_UART0_BASE=0x10000000</code></pre></div>
<p><strong><code>0x1000</code> <code>0000</code></strong> is the exact same UART Base Address for QEMU AND Star64…</p>
<p>So no changes needed, our Debug Log will work on Star64!</p>
<p><a href="https://github.com/lupyuen/nuttx-star64#hang-in-uart-transmit">(Star64 uses a different <strong>CONFIG_16550_REGINCR</strong>)</a></p>
<p>To boot NuttX on Star64, it needs a special ingredient…</p>
<p><img src="https://lupyuen.github.io/images/star64-kernel.png" alt="Armbian Kernel Image" /></p>
<p><a href="https://lupyuen.github.io/articles/star64#inside-the-kernel-image"><em>Kernel Header for RISC-V Armbian Linux</em></a></p>
<h1 id="risc-v-linux-kernel-header"><a class="doc-anchor" href="#risc-v-linux-kernel-header">§</a>4 RISC-V Linux Kernel Header</h1>
<p><em>How will Star64 boot NuttX?</em></p>
<p>Star64’s <a href="https://lupyuen.github.io/articles/linux#u-boot-bootloader-for-star64"><strong>U-Boot Bootloader</strong></a> will load NuttX Kernel into RAM and run it.</p>
<p>But we need to embed the <strong>RISC-V Linux Kernel Header</strong> (and pretend we’re Linux)…</p>
<ul>
<li>
<p><a href="https://lupyuen.github.io/articles/star64#inside-the-kernel-image"><strong>“Inside the Kernel Image”</strong></a></p>
</li>
<li>
<p><a href="https://lupyuen.github.io/articles/star64#appendix-decode-the-risc-v-linux-header"><strong>“Decode the RISC-V Linux Header”</strong></a></p>
</li>
</ul>
<p>Thus we cooked up this Assembly Code for our <strong>RISC-V Linux Header</strong>: <a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/jh7110/jh7110_head.S#L42-L75">jh7110_head.S</a></p>
<div class="example-wrap"><pre class="language-text"><code>c.li s4, -13 /* Magic Signature "MZ" (2 bytes) */
.long 0 /* Executable Code padded to 8 bytes */
.quad 0x200000 /* Image load offset from start of RAM */
.quad _ebss - __start /* Effective size of kernel image */
.quad 0 /* Kernel flags, little-endian */
.long 2 /* Version of this header */
.long 0 /* Reserved */
.quad 0 /* Reserved */
.ascii "RISCV\x00\x00\x00" /* Magic number, "RISCV" (8 bytes) */
.ascii "RSC\x05" /* Magic number 2, "RSC\x05" (4 bytes) */
.long 0 /* Reserved for PE COFF offset */
real_start:
/* Actual Boot Code starts here... */</code></pre></div>
<p><a href="https://lupyuen.github.io/articles/star64#decompile-the-kernel-with-ghidra">(<strong><code>c.li</code></strong> emits the <strong>Magic Signature “MZ”</strong>)</a></p>
<p>Note that <strong>Image Load Offset</strong> must be <strong><code>0x20</code> <code>0000</code></strong>…</p>
<div class="example-wrap"><pre class="language-text"><code>.quad 0x200000 /* Image load offset from start of RAM */</code></pre></div>
<p>That’s because our NuttX Kernel will start at RAM Address <strong><code>0x4020</code> <code>0000</code></strong>. Chew on it for a bit…</p>
<h1 id="start-address-of-nuttx-kernel"><a class="doc-anchor" href="#start-address-of-nuttx-kernel">§</a>5 Start Address of NuttX Kernel</h1>
<p><em>What’s this magical address <code>0x4020</code> <code>0000</code>?</em></p>
<p>From previous articles, we saw that Star64’s U-Boot Bootloader will load Linux Kernels into RAM at Address <strong><code>0x4020</code> <code>0000</code></strong>…</p>
<ul>
<li>
<p><a href="https://lupyuen.github.io/articles/star64#armbian-image-for-star64"><strong>“Armbian Image for Star64”</strong></a></p>
</li>
<li>
<p><a href="https://lupyuen.github.io/articles/star64#yocto-image-for-star64"><strong>“Yocto Image for Star64”</strong></a></p>
</li>
</ul>
<p>Thus we do the same for NuttX on Star64.</p>
<p>This is how we set the Start Address to <strong><code>0x4020</code> <code>0000</code></strong> in our <strong>NuttX Build Configuration</strong>: <a href="https://github.com/apache/nuttx/blob/master/boards/risc-v/jh7110/star64/configs/nsh/defconfig#L79-L80">nsh/defconfig</a></p>
<div class="example-wrap"><pre class="language-text"><code>// TODO: Fix CONFIG_RAM_SIZE
CONFIG_RAM_SIZE=33554432
CONFIG_RAM_START=0x40200000</code></pre></div>
<p>And our <strong>NuttX Linker Script</strong>: <a href="https://github.com/apache/nuttx/blob/master/boards/risc-v/jh7110/star64/scripts/ld.script#L20-L53">ld.script</a></p>
<div class="example-wrap"><pre class="language-text"><code>MEMORY
{
/* Previously 0x80000000 */
kflash (rx) : ORIGIN = 0x40200000, LENGTH = 2048K /* w/ cache */
}
...
SECTIONS
{
/* Previously 0x80000000 */
. = 0x40200000;
.text :</code></pre></div>
<p><a href="https://lupyuen.github.io/articles/nuttx2#appendix-nuttx-start-address">(Also <strong>knsh64/defconfig</strong> and <strong>ld-kernel64.script</strong>)</a></p>
<p><em>We’re sure this is correct?</em></p>
<p>We check the <strong>RISC-V Disassembly</strong> of our NuttX Kernel: <a href="https://github.com/lupyuen2/wip-nuttx/releases/download/star64-0.0.1/nuttx.S">nuttx.S</a></p>
<div class="example-wrap"><pre class="language-text"><code>0000000040200000 <__start>:
li s4, -0xd /* Magic Signature "MZ" (2 bytes) */
40200000: 5a4d li s4,-13
j real_start /* Jump to Kernel Start (2 bytes) */
40200002: a83d j 40200040 <real_start></code></pre></div>
<p>The Start Address is indeed <strong><code>0x4020</code> <code>0000</code></strong>.</p>
<p>Yep Looks Good To Us (YLGTU), we’re ready to serve it on Star64!</p>
<p><img src="https://lupyuen.github.io/images/star64-nuttx.png" alt="Boot NuttX on Star64" /></p>
<h1 id="boot-nuttx-on-star64"><a class="doc-anchor" href="#boot-nuttx-on-star64">§</a>6 Boot NuttX on Star64</h1>
<p>We’re finally ready to <strong>boot NuttX on Star64</strong>! We compile <strong>NuttX for RISC-V QEMU</strong> with these steps…</p>
<ul>
<li><a href="https://lupyuen.github.io/articles/riscv#appendix-build-apache-nuttx-rtos-for-64-bit-risc-v-qemu"><strong>“Build Apache NuttX RTOS for 64-bit RISC-V QEMU”</strong></a></li>
</ul>
<p>Then we tweak it to <strong>boot on Star64</strong> (and rebuild)…</p>
<ol>
<li>
<p>Print the <a href="https://lupyuen.github.io/articles/nuttx2#print-to-qemu-console"><strong>Debug Logs</strong></a> in RISC-V Assembly</p>
</li>
<li>
<p>Check the <a href="https://lupyuen.github.io/articles/nuttx2#uart-controller-on-star64"><strong>UART Base Address</strong></a></p>
</li>
<li>
<p>Embed the <a href="https://lupyuen.github.io/articles/nuttx2#risc-v-linux-kernel-header"><strong>RISC-V Kernel Header</strong></a></p>
</li>
<li>
<p>Set the <a href="https://lupyuen.github.io/articles/nuttx2#start-address-of-nuttx-kernel"><strong>Start Address</strong></a> of NuttX Kernel</p>
</li>
</ol>
<p>This produces the <strong>NuttX ELF Image</strong> for Star64…</p>
<ul>
<li>
<p><a href="https://github.com/lupyuen2/wip-nuttx/releases/download/star64-0.0.1/nuttx"><strong>nuttx: NuttX ELF Image</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen2/wip-nuttx/releases/tag/star64-0.0.1">See the <strong>Build Steps</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen2/wip-nuttx/pull/31/files">See the <strong>Modified Files</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen2/wip-nuttx/releases/tag/star64-0.0.1">See the <strong>Build Outputs</strong></a></p>
</li>
</ul>
<p><em>How to boot NuttX on microSD?</em></p>
<p>For the microSD Image, we start with this <a href="https://www.armbian.com/star64/"><strong>Armbian Image for Star64</strong></a>…</p>
<ul>
<li>
<p><a href="https://github.com/armbianro/os/releases/download/23.8.0-trunk.69/Armbian_23.8.0-trunk.69_Star64_lunar_edge_5.15.0_minimal.img.xz"><strong>Armbian 23.8 Lunar for Star64 (Minimal)</strong></a></p>
<p><a href="https://github.com/armbianro/os/releases/">(Or pick the latest <strong>Star64 Minimal Image</strong>)</a></p>
</li>
</ul>
<p>Uncompress the <strong>.xz</strong> file. Write the <strong>.img</strong> file to a microSD Card with <a href="https://www.balena.io/etcher/"><strong>Balena Etcher</strong></a> or <a href="https://wiki.gnome.org/Apps/Disks"><strong>GNOME Disks</strong></a>.</p>
<p>The <a href="https://lupyuen.github.io/articles/star64#armbian-image-for-star64"><strong>Device Tree</strong></a> is missing, so we fix it…</p>
<div class="example-wrap"><pre class="language-bash"><code>## Fix the Missing Device Tree
sudo chmod go+w /run/media/$USER/armbi_root/boot
sudo chmod go+w /run/media/$USER/armbi_root/boot/dtb/starfive
cp \
/run/media/$USER/armbi_root/boot/dtb/starfive/jh7110-visionfive-v2.dtb \
/run/media/$USER/armbi_root/boot/dtb/starfive/jh7110-star64-pine64.dtb</code></pre></div>
<p>Then we overwrite the Linux Kernel Image by our <strong>NuttX Binary Image</strong>…</p>
<div class="example-wrap"><pre class="language-bash"><code>## We assume that `nuttx` contains the NuttX ELF Image.
## Export the NuttX Binary Image to `nuttx.bin`
riscv64-unknown-elf-objcopy \
-O binary \
nuttx \
nuttx.bin
## Delete Linux Kernel `/boot/Image`
rm /run/media/$USER/armbi_root/boot/Image
## Copy `nuttx.bin` to Linux Kernel `/boot/Image`
cp nuttx.bin /run/media/$USER/armbi_root/boot/Image</code></pre></div>
<p>Insert the microSD Card into Star64 and power up.</p>
<p><a href="https://lupyuen.github.io/articles/tftp">(Or boot <strong>NuttX over the Network</strong>)</a></p>
<p>When it boots on Star64, NuttX prints “<strong><code>123</code></strong>” yay! (Pic above)</p>
<div class="example-wrap"><pre class="language-text"><code>Starting kernel ...
clk u5_dw_i2c_clk_core already disabled
clk u5_dw_i2c_clk_apb already disabled
123</code></pre></div>
<p><a href="https://lupyuen.github.io/articles/nuttx2#print-to-qemu-console">(Printed by our <strong>Boot Code</strong>)</a></p>
<p>But NuttX crashes with a <strong>RISC-V Illegal Instruction Exception</strong>…</p>
<div class="example-wrap"><pre class="language-text"><code>Unhandled exception: Illegal instruction
EPC: 000000004020005c RA: 00000000fff471c6 TVAL: 00000000f1402573
EPC: ffffffff804ba05c RA: 00000000402011c6 reloc adjusted
SP: 00000000ff733630 GP: 00000000ff735e00 TP: 0000000000000001
T0: 0000000010000000 T1: 0000000000000033 T2: 7869662e6b637366
S0: 0000000000000400 S1: 00000000ffff1428 A0: 0000000000000001
A1: 0000000046000000 A2: 0000000000000600 A3: 0000000000004000</code></pre></div>
<p><a href="https://lupyuen.github.io/articles/nuttx2#appendix-nuttx-crash-log">(See the <strong>Complete Log</strong>)</a></p>
<p>(<strong>EPC</strong> is the Program Counter for the Exception: <strong><code>0x4020</code> <code>005C</code></strong>)</p>
<p>And Star64 (OpenSBI) shows the offending <strong>RISC-V Machine Code</strong> (in brackets)…</p>
<div class="example-wrap"><pre class="language-text"><code>Code:
0313 0320 8023 0062 0313 0330 8023 0062
(2573 f140)
resetting ...
reset not supported yet
### ERROR ### Please RESET the board ###</code></pre></div>
<p>Why did NuttX crash at <strong><code>0x4020</code> <code>005C</code></strong>? We percolate our code…</p>
<p><img src="https://lupyuen.github.io/images/star64-exception.jpg" alt="Cody AI Assistant tries to explain our RISC-V Exception" /></p>
<p><em>Cody AI Assistant tries to explain our RISC-V Exception</em></p>
<h1 id="nuttx-fails-to-get-hart-id"><a class="doc-anchor" href="#nuttx-fails-to-get-hart-id">§</a>7 NuttX Fails To Get Hart ID</h1>
<p><em>What’s at <code>0x4020</code> <code>005C</code>?</em></p>
<p><em>Why did it crash NuttX?</em></p>
<p>We look up our <strong>NuttX RISC-V Disassembly nuttx.S</strong> and see this in our Boot Code: <a href="https://github.com/lupyuen2/wip-nuttx/blob/ed09c34532ee7c51ac2da816cd6cf0adcce336e6/arch/risc-v/src/qemu-rv/qemu_rv_head.S#L92-L103">qemu_rv_head.S</a></p>
<div class="example-wrap"><pre class="language-text"><code>nuttx/arch/risc-v/src/chip/qemu_rv_head.S:95
/* Load the Hart ID (CPU ID) */
csrr a0, mhartid
4020005c: f1402573 csrr a0, mhartid</code></pre></div>
<p>Breaking it down…</p>
<div class="example-wrap"><pre class="language-text"><code>/* Load the Hart ID (CPU ID) */
csrr a0, mhartid</code></pre></div>
<ul>
<li>
<p><strong><code>csrr</code></strong> is the RISC-V Instruction that reads the <a href="https://five-embeddev.com/quickref/instructions.html#-csr--csr-instructions"><strong>Control and Status Register</strong></a></p>
<p>(Which contains the CPU ID)</p>
</li>
<li>
<p><strong><code>a0</code></strong> is the RISC-V Register that will be loaded with the CPU ID</p>
</li>
<li>
<p><strong><code>mhartid</code></strong> says that we’ll read from the <a href="https://five-embeddev.com/riscv-isa-manual/latest/machine.html#hart-id-register-mhartid"><strong>Hart ID Register</strong></a>, containing the ID of the Hardware Thread (“Hart”) that’s running our code.</p>
<p>(Equivalent to CPU ID)</p>
</li>
</ul>
<p>So the above code will load the <strong>Hart ID</strong> (or CPU ID) into Register A0.</p>
<p><a href="https://lupyuen.github.io/articles/riscv#get-cpu-id">(As explained here)</a></p>
<p><em>But it worked perfectly on QEMU! Why did it fail?</em></p>
<p>Ah that’s because something super spicy has changed on Star64: Our Privilege Level…</p>
<p><img src="https://lupyuen.github.io/images/nuttx2-privilege.jpg" alt="RISC-V Privilege Levels" /></p>
<h1 id="risc-v-privilege-levels"><a class="doc-anchor" href="#risc-v-privilege-levels">§</a>8 RISC-V Privilege Levels</h1>
<p><em>What’s this Privilege Level?</em></p>
<p>RISC-V Machine Code runs at three <strong>Privilege Levels</strong>…</p>
<ul>
<li>
<p><strong>M: Machine Mode</strong> (Most powerful)</p>
</li>
<li>
<p><strong>S: Supervisor Mode</strong> (Less powerful)</p>
</li>
<li>
<p><strong>U: User Mode</strong> (Least powerful)</p>
</li>
</ul>
<p>NuttX on Star64 runs in <strong>Supervisor Mode</strong>. Which doesn’t allow access to <a href="https://five-embeddev.com/riscv-isa-manual/latest/machine.html"><strong>Machine-Mode CSR Registers</strong></a>. (Pic above)</p>
<p>Remember this?</p>
<div class="example-wrap"><pre class="language-text"><code>/* Load the Hart ID (CPU ID) */
csrr a0, mhartid</code></pre></div>
<p>The <strong>“<code>m</code>”</strong> in <a href="https://five-embeddev.com/riscv-isa-manual/latest/machine.html#hart-id-register-mhartid"><strong><code>mhartid</code></strong></a> signifies that it’s a <strong>Machine-Mode Register</strong>.</p>
<p>That’s why NuttX fails to read the Hart ID!</p>
<p><em>What runs in Machine Mode?</em></p>
<p><a href="https://lupyuen.github.io/articles/linux#opensbi-supervisor-binary-interface"><strong>OpenSBI (Supervisor Binary Interface)</strong></a> is the first thing that boots on Star64.</p>
<p>It runs in <strong>Machine Mode</strong> and starts the U-Boot Bootloader.</p>
<p><a href="https://lupyuen.github.io/articles/linux#opensbi-supervisor-binary-interface">(More about <strong>OpenSBI</strong>)</a></p>
<p><em>What about U-Boot Bootloader?</em></p>
<p><a href="https://lupyuen.github.io/articles/linux#u-boot-bootloader-for-star64"><strong>U-Boot Bootloader</strong></a> runs in <strong>Supervisor Mode</strong>. And starts NuttX, also in Supervisor Mode.</p>
<p>Thus <strong>OpenSBI is the only thing</strong> that runs in Machine Mode. And can access the Machine-Mode Registers. (Pic above)</p>
<p><a href="https://lupyuen.github.io/articles/linux#u-boot-bootloader-for-star64">(More about <strong>U-Boot</strong>)</a></p>
<p><em>QEMU doesn’t have this problem?</em></p>
<p>Because QEMU runs NuttX in (super-powerful) <strong>Machine Mode</strong>!</p>
<p><img src="https://lupyuen.github.io/images/nuttx2-privilege2.jpg" alt="NuttX QEMU runs in Machine Mode" /></p>
<p>NuttX needs to fetch the Hart ID with a different recipe…</p>
<p><img src="https://lupyuen.github.io/images/star64-opensbi.jpg" alt="OpenSBI starts U-Boot Bootloader on Star64" /></p>
<p><a href="https://lupyuen.github.io/articles/linux#opensbi-supervisor-binary-interface"><em>OpenSBI starts U-Boot Bootloader on Star64</em></a></p>
<h1 id="downgrade-nuttx-to-supervisor-mode"><a class="doc-anchor" href="#downgrade-nuttx-to-supervisor-mode">§</a>9 Downgrade NuttX to Supervisor Mode</h1>
<p><em>OpenSBI runs in Machine Mode and reads the Hart ID (CPU ID)…</em></p>
<p><em>How will NuttX get the Hart ID from OpenSBI?</em></p>
<p>Thankfully OpenSBI will pass the Hart ID to NuttX through <a href="https://lupyuen.github.io/articles/nuttx2#appendix-hart-id-from-opensbi"><strong>Register A0</strong></a>.</p>
<p>So this (overly-powerful) line in our <a href="https://github.com/lupyuen2/wip-nuttx/blob/ed09c34532ee7c51ac2da816cd6cf0adcce336e6/arch/risc-v/src/qemu-rv/qemu_rv_head.S#L92-L103"><strong>NuttX Boot Code</strong></a>…</p>
<div class="example-wrap"><pre class="language-text"><code>/* Load the Hart ID (CPU ID) */
csrr a0, mhartid</code></pre></div>
<p>Gets demoted to this: <a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/jh7110/jh7110_head.S#L68-L76">jh7110_head.S</a></p>
<div class="example-wrap"><pre class="language-text"><code>/* We assume that OpenSBI has passed Hart ID (value 1) in Register A0. */
/* But NuttX expects Hart ID to start at 0, so we subtract 1. */
addi a0, a0, -1</code></pre></div>
<p><a href="https://lupyuen.github.io/articles/nuttx2#appendix-hart-id-from-opensbi">(OpenSBI passes <strong>Hart ID as 1</strong>, instead of 0)</a></p>
<p><a href="https://five-embeddev.com/quickref/instructions.html#-rv32--integer-register-immediate-instructions">(<strong><code>addi</code></strong> adds an Immediate Value to a Register)</a></p>
<p><em>What about other CSR Instructions in our NuttX Boot Code?</em></p>
<p>Easy! We change the Machine-Mode <strong><code>m</code></strong> Registers to Supervisor-Mode <strong><code>s</code></strong> Registers…</p>
<ul>
<li>
<p><strong>To Disable Interrupts:</strong> Change <a href="https://lupyuen.github.io/articles/riscv#disable-interrupts"><strong><code>mie</code></strong></a> to <a href="https://five-embeddev.com/riscv-isa-manual/latest/supervisor.html#supervisor-interrupt-registers-sip-and-sie"><strong><code>sie</code></strong></a></p>
<div class="example-wrap"><pre class="language-text"><code>/* Disable all interrupts (i.e. timer, external) */
csrw sie, zero
/* Previously `mie` */</code></pre></div>
<p><a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/jh7110/jh7110_head.S#L125-L131">(Source)</a></p>
</li>
<li>
<p><strong>To Load Trap Vector Table:</strong> Change <a href="https://lupyuen.github.io/articles/riscv#load-interrupt-vector"><strong><code>mtvec</code></strong></a> to <a href="https://five-embeddev.com/riscv-isa-manual/latest/supervisor.html#supervisor-trap-vector-base-address-register-stvec"><strong><code>stvec</code></strong></a></p>
<div class="example-wrap"><pre class="language-text"><code>/* Load address of Trap Vector Table */
csrw stvec, t0
/* Previously `mtvec` */</code></pre></div>
<p><a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/jh7110/jh7110_head.S#L131-L134">(Source)</a></p>
</li>
</ul>
<p>Time to taste this…</p>
<p><img src="https://lupyuen.github.io/images/nuttx2-boot.png" alt="After fixing NuttX for Supervisor Mode" /></p>
<h1 id="fix-the-nuttx-boot-code"><a class="doc-anchor" href="#fix-the-nuttx-boot-code">§</a>10 Fix the NuttX Boot Code</h1>
<p>From the previous section, we identified these fixes to run our NuttX Boot Code in <strong>Supervisor Mode</strong>…</p>
<ol>
<li>
<p>Remove <strong><code>mhartid</code></strong> because OpenSBI will pass <strong>Hart ID</strong> in Register A0</p>
</li>
<li>
<p>Subtract 1 from <strong>Register A0</strong> because NuttX expects Hart ID to start with 0</p>
</li>
<li>
<p>To Disable Interrupts: Change <a href="https://lupyuen.github.io/articles/riscv#disable-interrupts"><strong><code>mie</code></strong></a> to <a href="https://five-embeddev.com/riscv-isa-manual/latest/supervisor.html#supervisor-interrupt-registers-sip-and-sie"><strong><code>sie</code></strong></a></p>
</li>
<li>
<p>To Load Trap Vector Table: Change <a href="https://lupyuen.github.io/articles/riscv#load-interrupt-vector"><strong><code>mtvec</code></strong></a> to <a href="https://five-embeddev.com/riscv-isa-manual/latest/supervisor.html#supervisor-trap-vector-base-address-register-stvec"><strong><code>stvec</code></strong></a></p>
</li>
</ol>
<p>Here’s the updated NuttX Boot Code and our analysis…</p>
<ul>
<li>
<p><a href="https://lupyuen.github.io/articles/nuttx2#appendix-nuttx-in-supervisor-mode"><strong>NuttX in Supervisor Mode</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen2/wip-nuttx/releases/tag/star64-0.0.1">See the <strong>Build Steps</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen2/wip-nuttx/pull/31/files">See the <strong>Modified Files</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen2/wip-nuttx/releases/tag/star64-0.0.1">See the <strong>Build Outputs</strong></a></p>
</li>
</ul>
<p><em>What happens when we run this?</em></p>
<p>When we boot the modified NuttX on Star64, we see this (pic above)…</p>
<div class="example-wrap"><pre class="language-text"><code>Starting kernel ...
clk u5_dw_i2c_clk_core already disabled
clk u5_dw_i2c_clk_apb already disabled
123067</code></pre></div>
<p><a href="https://github.com/lupyuen2/wip-nuttx/releases/tag/star64-0.0.1">(Source)</a></p>
<p>Now we’re smokin’ hot…</p>
<ul>
<li>
<p>No more <a href="https://lupyuen.github.io/articles/nuttx2#appendix-nuttx-crash-log"><strong>Crash Dump</strong></a>!</p>
</li>
<li>
<p>“<strong><code>0</code></strong>” is the Adjusted Hart ID passed by OpenSBI to NuttX</p>
<p><a href="https://lupyuen.github.io/articles/nuttx2#appendix-hart-id-from-opensbi">(From here)</a></p>
</li>
<li>
<p>“<strong><code>7</code></strong>” is the last thing that’s printed by our NuttX Boot Code…</p>
<div class="example-wrap"><pre class="language-text"><code>/* Print `7` */
li t0, 0x10000000
li t1, 0x37
sb t1, 0(t0)</code></pre></div>
<p><a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/qemu-rv/qemu_rv_head.S#L163-L199">(Previously here)</a></p>
<p>Before jumping to the Start Code…</p>
<div class="example-wrap"><pre class="language-text"><code>/* Jump to jh7110_start */
jal x1, jh7110_start</code></pre></div>
<p><a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/jh7110/jh7110_head.S#L134-L138">(Source)</a></p>
</li>
<li>
<p>Which means that our <strong>NuttX Boot Code has completed</strong> execution yay!</p>
</li>
<li>
<p>But NuttX hangs in the C Function <a href="https://lupyuen.github.io/articles/riscv#jump-to-start"><strong>jh7110_start</strong></a></p>
</li>
</ul>
<p>Why? Stay tuned for more tantalising treats in the next article!</p>
<ul>
<li><a href="https://lupyuen.github.io/articles/privilege"><strong>“Star64 JH7110 + NuttX RTOS: RISC-V Privilege Levels and UART Registers”</strong></a></li>
</ul>
<p><img src="https://lupyuen.github.io/images/nuttx2-star64.jpg" alt="Pine64 Star64 64-bit RISC-V SBC" /></p>
<h1 id="whats-next"><a class="doc-anchor" href="#whats-next">§</a>11 What’s Next</h1>
<p>I hope this has been an Educational Experience on booting a fresh new OS for a 64-bit RISC-V SBC…</p>
<ul>
<li>
<p>We migrated NuttX RTOS from <strong>QEMU Emulator</strong> to Star64 JH7110</p>
</li>
<li>
<p>And printed to the <strong>Serial Console</strong> in RISC-V Assembly</p>
</li>
<li>
<p>We fooled <strong>U-Boot Bootloader</strong> into thinking we’re Linux</p>
</li>
<li>
<p>But we demoted NuttX to <strong>Supervisor Privilege Level</strong></p>
</li>
<li>
<p>Helped by <strong>OpenSBI Supervisor Interface</strong></p>
</li>
</ul>
<p>This is the first in a series of (yummy) articles on porting NuttX to Star64, please join me next time…</p>
<ul>
<li><a href="https://lupyuen.github.io/articles/privilege"><strong>“Star64 JH7110 + NuttX RTOS: RISC-V Privilege Levels and UART Registers”</strong></a></li>
</ul>
<p>Many Thanks to my <a href="https://lupyuen.github.io/articles/sponsor"><strong>GitHub Sponsors</strong></a> for supporting my work! This article wouldn’t have been possible without your support.</p>
<ul>
<li>
<p><a href="https://lupyuen.github.io/articles/sponsor"><strong>Sponsor me a coffee</strong></a></p>
</li>
<li>
<p><a href="https://news.ycombinator.com/item?id=36649714"><strong>Discuss this article on Hacker News</strong></a></p>
</li>
<li>
<p><a href="https://forum.pine64.org/showthread.php?tid=18469"><strong>Discuss this article on Pine64 Forum</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/nuttx-ox64"><strong>My Current Project: “Apache NuttX RTOS for Ox64 BL808”</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/nuttx-star64"><strong>My Other Project: “NuttX for Star64 JH7110”</strong></a></p>
</li>
<li>
<p><a href="https://github.com/lupyuen/pinephone-nuttx"><strong>Older Project: “NuttX for PinePhone”</strong></a></p>
</li>
<li>
<p><a href="https://lupyuen.github.io"><strong>Check out my articles</strong></a></p>
</li>
<li>
<p><a href="https://lupyuen.github.io/rss.xml"><strong>RSS Feed</strong></a></p>
</li>
</ul>
<p><em>Got a question, comment or suggestion? Create an Issue or submit a Pull Request here…</em></p>
<p><a href="https://github.com/lupyuen/lupyuen.github.io/blob/master/src/nuttx2.md"><strong>lupyuen.github.io/src/nuttx2.md</strong></a></p>
<p><img src="https://lupyuen.github.io/images/nuttx2-privilege.jpg" alt="RISC-V Privilege Levels" /></p>
<h1 id="appendix-hart-id-from-opensbi"><a class="doc-anchor" href="#appendix-hart-id-from-opensbi">§</a>12 Appendix: Hart ID from OpenSBI</h1>
<p><em>NuttX can’t read the <code>mhartid</code> CSR Register in Supervisor Mode…</em></p>
<p><em>How to get the Hart ID from OpenSBI?</em></p>
<p>NuttX failed to read the <strong><code>mhartid</code></strong> CSR Register because it’s a <strong>Machine-Mode Register</strong> (pic above)…</p>
<ul>
<li>
<p><a href="https://lupyuen.github.io/articles/nuttx2#nuttx-fails-to-get-hart-id"><strong>“NuttX Fails To Get Hart ID”</strong></a></p>
</li>
<li>
<p><a href="https://lupyuen.github.io/articles/nuttx2#risc-v-privilege-levels"><strong>“RISC-V Privilege Levels”</strong></a></p>
</li>
</ul>
<p>Let’s figure out how Linux gets the Hart ID from OpenSBI. We refer to the <strong>Linux Boot Code</strong>: <a href="https://github.com/torvalds/linux/blob/master/arch/riscv/kernel/head.S">linux/arch/riscv/kernel/head.S</a></p>
<p>(Tip: CONFIG_RISCV_M_MODE is False and CONFIG_EFI is True)</p>
<div class="example-wrap"><pre class="language-c"><code>/* Save hart ID and DTB physical address */
mv s0, a0
mv s1, a1</code></pre></div>
<p>Here we see that U-Boot <a href="https://github.com/riscv-non-isa/riscv-sbi-doc/blob/master/riscv-sbi.adoc#function-hart-start-fid-0">(or OpenSBI)</a> will pass 2 arguments when it starts the kernel…</p>
<ul>
<li>
<p><strong>Register A0:</strong> Hart ID</p>
</li>
<li>
<p><strong>Register A1:</strong> RAM Address of Device Tree</p>
</li>
</ul>
<p>So we’ll simply read the Hart ID from Register A0. (And ignore A1)</p>
<p><em>What are the actual values of Registers A0 and A1?</em></p>
<p>Thanks to our <a href="https://lupyuen.github.io/articles/nuttx2#appendix-nuttx-crash-log"><strong>Crash Dump</strong></a>, we know the actual values of A0 and A1!</p>
<div class="example-wrap"><pre class="language-text"><code>SP: 00000000ff733630 GP: 00000000ff735e00 TP: 0000000000000001
T0: 0000000010000000 T1: 0000000000000033 T2: 7869662e6b637366
S0: 0000000000000400 S1: 00000000ffff1428 A0: 0000000000000001
A1: 0000000046000000 A2: 0000000000000600 A3: 0000000000004000</code></pre></div>
<p>This says that…</p>
<ul>
<li>
<p><strong>Hart ID</strong> is 1 (Register A0)</p>
</li>
<li>
<p><strong>RAM Address of Device Tree</strong> is <code>0x4600</code> <code>0000</code> (Register A1)</p>
</li>
</ul>
<p>Yep looks correct! But we’ll subtract 1 from Register A0 because NuttX expects Hart ID to start with 0.</p>
<p>We’ll see this implementation in our modified NuttX Boot Code. (Next section)</p>
<p><img src="https://lupyuen.github.io/images/plic-title.jpg" alt="Platform-Level Interrupt Controller in JH7110 (U74) SoC" /></p>
<p><em>Why does OpenSBI return Hart ID 1? (Instead of 0)</em></p>
<p>According to the <a href="https://starfivetech.com/uploads/u74mc_core_complex_manual_21G1.pdf"><strong>SiFive U74 Manual</strong></a> (Page 96), there are 5 RISC-V Cores in JH7110 (pic above)…</p>
<ul>
<li>
<p><strong>Hart 0:</strong> S7 Monitor Core (RV64IMACB)</p>
</li>
<li>
<p><strong>Harts 1 to 4:</strong> U74 Application Cores (RV64GCB)</p>
</li>
</ul>
<p>OpenSBI boots on the <strong>First Application Core</strong>…</p>
<p>Which is why OpenSBI returns <strong>Hart ID 1</strong>. (Instead of 0)</p>
<p>(Though we pass the Hart ID to NuttX as Hart 0, since NuttX expects <a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/jh7110/jh7110_head.S#L76-L83"><strong>Hart ID to start at 0</strong></a>)</p>
<p><em>The Linux Boot Code looks confusing. What are CSR_IE and CSR_IP?</em></p>
<div class="example-wrap"><pre class="language-text"><code>/* Mask all interrupts */
csrw CSR_IE, zero
csrw CSR_IP, zero</code></pre></div>
<p><a href="https://github.com/torvalds/linux/blob/master/arch/riscv/kernel/head.S#L195-L200">(Source)</a></p>
<p>That’s because the Linux Boot Code will work for Machine Mode AND Supervisor Mode! Here’s how <strong><code>CSR_IE</code></strong> and <strong><code>CSR_IP</code></strong> are mapped to the <strong><code>m</code></strong> and <strong><code>s</code></strong> CSR Registers…</p>
<p>(Remember: CONFIG_RISCV_M_MODE is false for NuttX)</p>
<div class="example-wrap"><pre class="language-text"><code>#ifdef CONFIG_RISCV_M_MODE
/* Use Machine-Mode CSR Registers */
# define CSR_IE CSR_MIE
# define CSR_IP CSR_MIP
...
#else
/* Use Supervisor-Mode CSR Registers */
# define CSR_IE CSR_SIE
# define CSR_IP CSR_SIP
...
#endif /* !CONFIG_RISCV_M_MODE */</code></pre></div>
<p><a href="https://github.com/torvalds/linux/blob/master/arch/riscv/include/asm/csr.h#L393-L446">(Source)</a></p>
<p><img src="https://lupyuen.github.io/images/nuttx2-boot.png" alt="After fixing NuttX for Supervisor Mode" /></p>
<h1 id="appendix-nuttx-in-supervisor-mode"><a class="doc-anchor" href="#appendix-nuttx-in-supervisor-mode">§</a>13 Appendix: NuttX in Supervisor Mode</h1>
<p>Earlier we identified these fixes for the <a href="https://lupyuen.github.io/articles/nuttx2#downgrade-nuttx-to-supervisor-mode"><strong>NuttX Boot Code</strong></a> to run in Supervisor Mode…</p>
<ol>
<li>
<p>Remove <strong><code>mhartid</code></strong> because OpenSBI will pass <strong>Hart ID</strong> in Register A0</p>
</li>
<li>
<p>Subtract 1 from <strong>Register A0</strong> because NuttX expects Hart ID to start with 0</p>
</li>
<li>
<p>To Disable Interrupts: Change <a href="https://lupyuen.github.io/articles/riscv#disable-interrupts"><strong><code>mie</code></strong></a> to <a href="https://five-embeddev.com/riscv-isa-manual/latest/supervisor.html#supervisor-interrupt-registers-sip-and-sie"><strong><code>sie</code></strong></a></p>
</li>
<li>
<p>To Load Trap Vector Table: Change <a href="https://lupyuen.github.io/articles/riscv#load-interrupt-vector"><strong><code>mtvec</code></strong></a> to <a href="https://five-embeddev.com/riscv-isa-manual/latest/supervisor.html#supervisor-trap-vector-base-address-register-stvec"><strong><code>stvec</code></strong></a></p>
</li>
</ol>
<p>Below is the updated <strong>NuttX Boot Code for Supervisor Mode</strong>, and our analysis: <a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/jh7110/jh7110_head.S">jh7110_head.S</a></p>
<p><strong>For All Hart IDs:</strong></p>
<p>We receive the Hart ID from OpenSBI, subtract 1 and print it…</p>
<div class="example-wrap"><pre class="language-text"><code>real_start:
...
/* Load mhartid (cpuid) */
/* Previously: csrr a0, mhartid */
/* We assume that OpenSBI has passed Hart ID (value 1) in Register a0. */
/* But NuttX expects Hart ID to start at 0, so we subtract 1. */
addi a0, a0, -1</code></pre></div>
<p><a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/jh7110/jh7110_head.S#L68-L76">(Source)</a></p>
<p><a href="https://lupyuen.github.io/articles/riscv#risc-v-boot-code-in-nuttx">(<strong>RISC-V Instructions</strong> explained)</a></p>
<p><strong>If Hart ID is 0 (First CPU):</strong></p>
<p>Set Stack Pointer to the Idle Thread Stack…</p>
<div class="example-wrap"><pre class="language-text"><code> /* Set stack pointer to the idle thread stack */
bnez a0, 1f
la sp, QEMU_RV_IDLESTACK_TOP
j 2f</code></pre></div>
<p><a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/jh7110/jh7110_head.S#L76-L83">(Source)</a></p>
<p><strong>If Hart ID is 1, 2, 3, …</strong></p>
<ul>
<li>Validate the Hart ID (Must be less than Number of CPUs)</li>
<li>Compute the Stack Base Address based on <code>g_cpu_basestack</code> and Hart ID</li>
<li>Set the Stack Pointer to the computed Stack Base Address</li>
</ul>
<div class="example-wrap"><pre class="language-text"><code>1:
/* Load the number of CPUs that the kernel supports */
#ifdef CONFIG_SMP
li t1, CONFIG_SMP_NCPUS
#else
li t1, 1
#endif
/* If a0 (mhartid) >= t1 (the number of CPUs), stop here */
blt a0, t1, 3f
csrw sie, zero
/* Previously: csrw mie, zero */
wfi
3:
/* To get g_cpu_basestack[mhartid], must get g_cpu_basestack first */
la t0, g_cpu_basestack
/* Offset = pointer width * hart id */
#ifdef CONFIG_ARCH_RV32
slli t1, a0, 2
#else
slli t1, a0, 3
#endif
add t0, t0, t1
/* Load idle stack base to sp */
REGLOAD sp, 0(t0)
/*
* sp (stack top) = sp + idle stack size - XCPTCONTEXT_SIZE
*
* Note: Reserve some space used by up_initial_state since we are already
* running and using the per CPU idle stack.
*/
li t0, STACK_ALIGN_UP(CONFIG_IDLETHREAD_STACKSIZE - XCPTCONTEXT_SIZE)
add sp, sp, t0</code></pre></div>
<p><a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/jh7110/jh7110_head.S#L83-L127">(Source)</a></p>
<p><strong>For All Hart IDs:</strong></p>
<ul>
<li>Disable Interrupts</li>
<li>Load the Trap Vector Table</li>
<li>Jump to <a href="https://lupyuen.github.io/articles/riscv#jump-to-start"><strong>Start Code jh7110_start</strong></a></li>
</ul>
<div class="example-wrap"><pre class="language-text"><code>2:
/* Disable all interrupts (i.e. timer, external) in mie */
csrw sie, zero
/* Previously: csrw mie, zero */
/* Load the Trap Vector Table */
la t0, __trap_vec
csrw stvec, t0
/* Previously: csrw mtvec, t0 */
/* Jump to jh7110_start */
jal x1, jh7110_start
/* We shouldn't return from _start */</code></pre></div>
<p><a href="https://github.com/apache/nuttx/blob/master/arch/risc-v/src/jh7110/jh7110_head.S#L127-L149">(Source)</a></p>
<p>Note that we don’t load the Trap Vector Table, because we’ll use OpenSBI for Crash Logging.</p>
<p><a href="https://lupyuen.github.io/articles/nuttx2#appendix-nuttx-crash-log">(Like when we hit Exceptions with <strong>Machine-Mode Instructions</strong>)</a></p>
<p><img src="https://lupyuen.github.io/images/riscv-build.png" alt="Building Apache NuttX RTOS in 4 minutes" /></p>
<h1 id="appendix-nuttx-start-address"><a class="doc-anchor" href="#appendix-nuttx-start-address">§</a>14 Appendix: NuttX Start Address</h1>
<p>Previously we changed the NuttX Start Address in <a href="https://github.com/apache/nuttx/blob/master/boards/risc-v/jh7110/star64/configs/nsh/defconfig#L79-L80"><strong>nsh/defconfig</strong></a> and <a href="https://github.com/apache/nuttx/blob/master/boards/risc-v/jh7110/star64/scripts/ld.script#L20-L53"><strong>ld.script</strong></a>…</p>
<ul>
<li><a href="https://lupyuen.github.io/articles/nuttx2#start-address-of-nuttx-kernel"><strong>“Start Address of NuttX Kernel”</strong></a></li>
</ul>
<p>Remember to change this Linker Script since we’re building for <strong>NuttX Kernel Mode</strong>: <a href="https://github.com/apache/nuttx/blob/master/boards/risc-v/jh7110/star64/scripts/ld.script#L20-L53">ld.script</a></p>
<div class="example-wrap"><pre class="language-text"><code>MEMORY
{
/* Previously 0x80000000 */
kflash (rx) : ORIGIN = 0x40200000, LENGTH = 2048K /* w/ cache */
/* Previously 0x80200000 */
ksram (rwx) : ORIGIN = 0x40400000, LENGTH = 2048K /* w/ cache */
/* Previously 0x80400000 */
pgram (rwx) : ORIGIN = 0x40600000, LENGTH = 4096K /* w/ cache */
}
...
SECTIONS
{
/* Previously 0x80000000 */
. = 0x40200000;
.text :</code></pre></div>
<p>Which should match the <strong>NuttX Build Configuration</strong> for Kernel Mode: <a href="https://github.com/apache/nuttx/blob/master/boards/risc-v/jh7110/star64/configs/nsh/defconfig">nsh/defconfig</a></p>
<div class="example-wrap"><pre class="language-text"><code>CONFIG_ARCH_PGPOOL_PBASE=0x40600000
CONFIG_ARCH_PGPOOL_VBASE=0x40600000
// TODO: Fix CONFIG_RAM_SIZE
CONFIG_RAM_SIZE=1048576
CONFIG_RAM_START=0x40200000</code></pre></div>
<p><em>Why will we use NuttX Kernel Mode?</em></p>
<div class="example-wrap"><pre class="language-bash"><code>## Configure NuttX Build for Kernel Mode
tools/configure.sh rv-virt:knsh64</code></pre></div>
<p><a href="https://github.com/lupyuen2/wip-nuttx/releases/tag/star64-0.0.1">(Source)</a></p>
<p>We use <code>rv-virt:knsh64</code> (NuttX Kernel Mode) instead of <code>rv-virt:nsh64</code> (NuttX Flat Mode) so that NuttX will run in <strong>RISC-V Supervisor Mode</strong>. (Instead of RISC-V Machine Mode)</p>
<p>More about this in the next article.</p>
<p><img src="https://lupyuen.github.io/images/star64-nuttx.png" alt="Boot NuttX on Star64" /></p>
<h1 id="appendix-nuttx-crash-log"><a class="doc-anchor" href="#appendix-nuttx-crash-log">§</a>15 Appendix: NuttX Crash Log</h1>
<p>Earlier we ran NuttX QEMU on Star64 (before fixing for <a href="https://lupyuen.github.io/articles/nuttx2#appendix-nuttx-in-supervisor-mode"><strong>Supervisor Mode</strong></a>) and it crashed…</p>
<ul>
<li>
<p><a href="https://lupyuen.github.io/articles/nuttx2#boot-nuttx-on-star64"><strong>“Boot NuttX on Star64”</strong></a></p>
</li>
<li>
<p><a href="https://lupyuen.github.io/articles/nuttx2#nuttx-fails-to-get-hart-id"><strong>“NuttX Fails To Get Hart ID”</strong></a></p>
</li>
</ul>
<p>Here’s the Crash Log dumped by OpenSBI…</p>
<div class="example-wrap"><pre class="language-text"><code>Retrieving file: /boot/extlinux/extlinux.conf
383 bytes read in 7 ms (52.7 KiB/s)
1:[6CArmbian
Retrieving file: /boot/uInitrd
10911538 bytes read in 466 ms (22.3 MiB/s)
Retrieving file: /boot/Image
163201 bytes read in 14 ms (11.1 MiB/s)
append: root=UUID=99f62df4-be35-475c-99ef-2ba3f74fe6b5 console=ttyS0,115200n8 console=tty0 earlycon=sbi rootflags=data=writeback stmmaceth=chain_mode:1 rw rw no_console_suspend consoleblank=0 fsck.fix=yes fsck.repair=yes net.ifnames=0 splash plymouth.ignore-serial-consoles
Retrieving file: /boot/dtb/starfive/jh7110-star64-pine64.dtb
50235 bytes read in 14 ms (3.4 MiB/s)
## Loading init Ramdisk from Legacy Image at 46100000 ...
Image Name: uInitrd
Image Type: RISC-V Linux RAMDisk Image (gzip compressed)
Data Size: 10911474 Bytes = 10.4 MiB
Load Address: 00000000
Entry Point: 00000000
Verifying Checksum ... OK
## Flattened Device Tree blob at 46000000
Booting using the fdt blob at 0x46000000
Using Device Tree in place at 0000000046000000, end 000000004600f43a
Starting kernel ...
clk u5_dw_i2c_clk_core already disabled
clk u5_dw_i2c_clk_apb already disabled
123Unhandled exception: Illegal instruction
EPC: 000000004020005c RA: 00000000fff471c6 TVAL: 00000000f1402573
EPC: ffffffff804ba05c RA: 00000000402011c6 reloc adjusted
SP: 00000000ff733630 GP: 00000000ff735e00 TP: 0000000000000001
T0: 0000000010000000 T1: 0000000000000033 T2: 7869662e6b637366
S0: 0000000000000400 S1: 00000000ffff1428 A0: 0000000000000001
A1: 0000000046000000 A2: 0000000000000600 A3: 0000000000004000
A4: 0000000000000000 A5: 0000000040200000 A6: 00000000fffd5708
A7: 0000000000000000 S2: 00000000fff47194 S3: 0000000000000003
S4: fffffffffffffff3 S5: 00000000fffdbb50 S6: 0000000000000000
S7: 0000000000000000 S8: 00000000fff47194 S9: 0000000000000002
S10: 0000000000000000 S11: 0000000000000000 T3: 0000000000000023
T4: 000000004600b5cc T5: 000000000000ff00 T6: 000000004600b5cc
Code: 0313 0320 8023 0062 0313 0330 8023 0062 (2573 f140)
resetting ...
reset not supported yet
### ERROR ### Please RESET the board ###</code></pre></div>
<!-- Begin scripts/rustdoc-after.html: Post-HTML for Custom Markdown files processed by rustdoc, like chip8.md -->
<!-- Begin Theme Picker and Prism Theme -->
<script src="../theme.js"></script>
<script src="../prism.js"></script>
<!-- Theme Picker and Prism Theme -->
<!-- End scripts/rustdoc-after.html -->
</body>
</html>