https://github.com/lupyuen/pinephone-nuttx
Apache NuttX RTOS for PinePhone
https://github.com/lupyuen/pinephone-nuttx
allwinner-a64 arm64 cortex-a53 de2 mipi-dsi nuttx osdev pinephone zig
Last synced: 5 days ago
JSON representation
Apache NuttX RTOS for PinePhone
- Host: GitHub
- URL: https://github.com/lupyuen/pinephone-nuttx
- Owner: lupyuen
- License: apache-2.0
- Created: 2022-08-21T08:03:52.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2026-05-23T00:51:38.000Z (13 days ago)
- Last Synced: 2026-05-23T02:34:25.321Z (13 days ago)
- Topics: allwinner-a64, arm64, cortex-a53, de2, mipi-dsi, nuttx, osdev, pinephone, zig
- Language: Zig
- Homepage: https://lupyuen.org/articles/what.html
- Size: 651 KB
- Stars: 99
- Watchers: 2
- Forks: 7
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Funding: .github/FUNDING.yml
- License: LICENSE
Awesome Lists containing this project
- awesome-zig - lupyuen/pinephone-nuttx - Apache NuttX RTOS for PinePhone. Apache NuttX is a lightweight Real-Time Operating System (RTOS) that runs on PINE64 PinePhone. [lupyuen.github.io/articles/what](https://lupyuen.github.io/articles/what). (Systems Programming / Embedded Development)
README
# Apache NuttX RTOS for PinePhone
[](https://github.com/lupyuen/pinephone-nuttx/actions/workflows/pinephone.yml)
Apache NuttX is a lightweight __Real-Time Operating System (RTOS)__ that runs on PINE64 PinePhone. Read the articles...
- ["NuttX RTOS for PinePhone: What is it?"](https://lupyuen.github.io/articles/what)
[(Watch the Demo on YouTube)](https://youtu.be/kGI_0yK1vws)
- ["NuttX RTOS for PinePhone: The First Year"](https://lupyuen.github.io/articles/pinephone2)
[(Watch the Presentation on Google Drive)](https://drive.google.com/file/d/1WL-6HVjhtqktHRmZiDbPCOs6934fQlEQ/view?usp=drive_link)
__PinePhone Display__
- ["NuttX RTOS for PinePhone: MIPI Display Serial Interface"](https://lupyuen.github.io/articles/dsi3)
- ["NuttX RTOS for PinePhone: Display Engine"](https://lupyuen.github.io/articles/de3)
__LCD Touch Panel__
- ["NuttX RTOS for PinePhone: LCD Panel"](https://lupyuen.github.io/articles/lcd)
- ["NuttX RTOS for PinePhone: Framebuffer"](https://lupyuen.github.io/articles/fb)
- ["NuttX RTOS for PinePhone: Touch Panel"](https://lupyuen.github.io/articles/touch2)
__LVGL Graphics__
- ["NuttX RTOS for PinePhone: Boot to LVGL"](https://lupyuen.github.io/articles/lvgl2)
- ["NuttX RTOS for PinePhone: LVGL Terminal for NSH Shell"](https://lupyuen.github.io/articles/terminal)
- ["(Possibly) LVGL in WebAssembly with Zig Compiler"](https://lupyuen.github.io/articles/lvgl3)
- ["NuttX RTOS for PinePhone: Feature Phone UI in LVGL, Zig and WebAssembly"](https://lupyuen.github.io/articles/lvgl4)
__Accelerometer and Gyroscope__
- ["Inside a Smartphone Accelerometer: PinePhone with NuttX RTOS"](https://www.hackster.io/lupyuen/inside-a-smartphone-accelerometer-pinephone-with-nuttx-rtos-b92b58)
__4G LTE Modem__
- ["NuttX RTOS for PinePhone: 4G LTE Modem"](https://lupyuen.github.io/articles/lte)
- ["NuttX RTOS for PinePhone: Phone Calls and Text Messages"](https://lupyuen.github.io/articles/lte2)
__GPIO__
- ["NuttX RTOS for PinePhone: Blinking the LEDs"](https://lupyuen.github.io/articles/pio)
__UART__
- ["NuttX RTOS for PinePhone: UART Driver"](https://lupyuen.github.io/articles/serial)
__USB__
- ["NuttX RTOS for PinePhone: Simpler USB with EHCI (Enhanced Host Controller Interface)"](https://lupyuen.github.io/articles/usb3)
- ["NuttX RTOS for PinePhone: Exploring USB"](https://lupyuen.github.io/articles/usb2)
__Power Management Integrated Circuit__
- ["Power Management Integrated Circuit (PMIC)"](https://lupyuen.github.io/articles/lcd#power-on-lcd-panel)
__Interrupts__
- ["NuttX RTOS for PinePhone: Fixing the Interrupts"](https://lupyuen.github.io/articles/interrupt)
__Boot Sequence__
- ["PinePhone Boots NuttX"](https://lupyuen.github.io/articles/unicorn2#pinephone-boots-nuttx)
- ["PinePhone Continues Booting NuttX"](https://lupyuen.github.io/articles/unicorn2#pinephone-continues-booting-nuttx)
__Arm64 Emulation__
- ["(Possibly) Emulate PinePhone with Unicorn Emulator"](https://lupyuen.github.io/articles/unicorn)
- ["(Clickable) Call Graph for Apache NuttX Real-Time Operating System"](https://lupyuen.github.io/articles/unicorn2)
__Build and Boot NuttX__
- ["Build Apache NuttX RTOS for PinePhone"](https://lupyuen.github.io/articles/lvgl2#appendix-build-apache-nuttx-rtos-for-pinephone)
- ["Boot Apache NuttX RTOS on PinePhone"](https://lupyuen.github.io/articles/lvgl2#appendix-boot-apache-nuttx-rtos-on-pinephone)
__Download NuttX Binaries__
- [Apache NuttX RTOS for PinePhone (Boot to LVGL Terminal)](https://github.com/lupyuen2/wip-pinephone-nuttx/releases/tag/nuttx-12.0.1)
- [Apache NuttX RTOS for PinePhone (Boot to LVGL Demo)](https://github.com/lupyuen2/wip-pinephone-nuttx/releases/tag/nuttx-12.0.0)
__How It Started__
- ["Apache NuttX RTOS on Arm Cortex-A53: How it might run on PinePhone"](https://lupyuen.github.io/articles/arm)
- ["PinePhone boots Apache NuttX RTOS"](https://lupyuen.github.io/articles/uboot)
- ["Preparing a Pull Request for Apache NuttX RTOS"](https://lupyuen.github.io/articles/pr)
__Older Articles__
- ["Understanding PinePhone's Display (MIPI DSI)"](https://lupyuen.github.io/articles/dsi)
- ["NuttX RTOS for PinePhone: Display Driver in Zig"](https://lupyuen.github.io/articles/dsi2)
- ["Rendering PinePhone's Display (DE and TCON0)"](https://lupyuen.github.io/articles/de)
- ["NuttX RTOS for PinePhone: Render Graphics in Zig"](https://lupyuen.github.io/articles/de2)
__Can AI Help?__
- ["Apache NuttX RTOS trips ChatGPT"](https://lupyuen.github.io/articles/chatgpt)
_What's NuttX? Why run it on PinePhone?_
If we're new to NuttX, here's a gentle intro...
- ["NuttX RTOS for PinePhone: What is it?"](https://lupyuen.github.io/articles/what)
The following is a journal that documents the porting of NuttX to PinePhone. It looks super messy and unstructured, please read the articles (at the top of this page) instead.
**Table of Contents**
- [NuttX Automated Daily Build for PinePhone](#nuttx-automated-daily-build-for-pinephone)
- [NuttX on QEMU](#nuttx-on-qemu)
- [Download NuttX](#download-nuttx)
- [Download Toolchain](#download-toolchain)
- [Download QEMU](#download-qemu)
- [Build NuttX: Single Core](#build-nuttx-single-core)
- [Test NuttX with QEMU: Single Core](#test-nuttx-with-qemu-single-core)
- [Build NuttX: Multi Core](#build-nuttx-multi-core)
- [Test NuttX with QEMU: Multi Core](#test-nuttx-with-qemu-multi-core)
- [Inside NuttX for Cortex-A53](#inside-nuttx-for-cortex-a53)
- [NuttX Image](#nuttx-image)
- [PinePhone Image](#pinephone-image)
- [Will NuttX Boot On PinePhone?](#will-nuttx-boot-on-pinephone)
- [UART Driver for NuttX](#uart-driver-for-nuttx)
- [Garbled Console Output](#garbled-console-output)
- [Configure UART Port](#configure-uart-port)
- [Test UART3 Port](#test-uart3-port)
- [Boot NuttX on PinePhone](#boot-nuttx-on-pinephone)
- [PinePhone U-Boot Log](#pinephone-u-boot-log)
- [NuttX Boots On PinePhone](#nuttx-boots-on-pinephone)
- [NuttX Boot Log](#nuttx-boot-log)
- [Interrupt Controller](#interrupt-controller)
- [Multi Core SMP](#multi-core-smp)
- [System Timer](#system-timer)
- [Timer Interrupt Isn't Handled](#timer-interrupt-isnt-handled)
- [Arm64 Vector Table Is Wrong](#arm64-vector-table-is-wrong)
- [Test PinePhone GIC with QEMU](#test-pinephone-gic-with-qemu)
- [Handling Interrupts](#handling-interrupts)
- [Dump Interrupt Vector Table](#dump-interrupt-vector-table)
- [Interrupt Debugging](#interrupt-debugging)
- [Memory Map](#memory-map)
- [Boot Sequence](#boot-sequence)
- [Boot Debugging](#boot-debugging)
- [UART Interrupts](#uart-interrupts)
- [Backlight and LEDs](#backlight-and-leds)
- [BASIC Blinks The LEDs](#basic-blinks-the-leds)
- [NuttX Drivers for Allwinner A64 PIO and PinePhone LEDs](#nuttx-drivers-for-allwinner-a64-pio-and-pinephone-leds)
- [PinePhone Device Tree](#pinephone-device-tree)
- [LCD Controller (TCON0)](#lcd-controller-tcon0)
- [MIPI DSI Interface](#mipi-dsi-interface)
- [Display PHY](#display-phy)
- [Backlight PWM](#backlight-pwm)
- [LED](#led)
- [Framebuffer](#framebuffer)
- [Display Engine](#display-engine)
- [Touch Panel](#touch-panel)
- [Video Codec](#video-codec)
- [GPU](#gpu)
- [Deinterlace](#deinterlace)
- [Zig on PinePhone](#zig-on-pinephone)
- [PinePhone Display Driver](#pinephone-display-driver)
- [Zig Driver for PinePhone MIPI DSI](#zig-driver-for-pinephone-mipi-dsi)
- [Compose MIPI DSI Long Packet in Zig](#compose-mipi-dsi-long-packet-in-zig)
- [Compose MIPI DSI Short Packet in Zig](#compose-mipi-dsi-short-packet-in-zig)
- [Compute Error Correction Code in Zig](#compute-error-correction-code-in-zig)
- [Compute Cyclic Redundancy Check in Zig](#compute-cyclic-redundancy-check-in-zig)
- [Test PinePhone MIPI DSI Driver with QEMU](#test-pinephone-mipi-dsi-driver-with-qemu)
- [Test Case for PinePhone MIPI DSI Driver](#test-case-for-pinephone-mipi-dsi-driver)
- [Initialise ST7703 LCD Controller in Zig](#initialise-st7703-lcd-controller-in-zig)
- [Test Zig Display Driver for PinePhone](#test-zig-display-driver-for-pinephone)
- [Display Engine in Allwinner A64](#display-engine-in-allwinner-a64)
- [Display Engine Mixers](#display-engine-mixers)
- [Render Colours](#render-colours)
- [Render Mandelbrot Set](#render-mandelbrot-set)
- [Animate Madelbrot Set](#animate-madelbrot-set)
- [Render Square Overlay](#render-square-overlay)
- [Render Circle Overlay](#render-circle-overlay)
- [Test PinePhone Display Engine](#test-pinephone-display-engine)
- [Display Engine Usage](#display-engine-usage)
- [Other Display Engine Features](#other-display-engine-features)
- [Timing Controller in Allwinner A64](#timing-controller-in-allwinner-a64)
- [Zig Driver for PinePhone Display Engine](#zig-driver-for-pinephone-display-engine)
- [Test Zig Driver for PinePhone Display Engine](#test-zig-driver-for-pinephone-display-engine)
- [Complete PinePhone Display Driver in Zig](#complete-pinephone-display-driver-in-zig)
- [Add MIPI DSI to NuttX Kernel](#add-mipi-dsi-to-nuttx-kernel)
- [Test MIPI DSI for NuttX Kernel](#test-mipi-dsi-for-nuttx-kernel)
- [Test Timing Controller TCON0 Driver for NuttX Kernel](#test-timing-controller-tcon0-driver-for-nuttx-kernel)
- [Test Display Engine Driver for NuttX Kernel](#test-display-engine-driver-for-nuttx-kernel)
- [Missing Pixels in PinePhone Image](#missing-pixels-in-pinephone-image)
- [Fix Missing Pixels in PinePhone Image](#fix-missing-pixels-in-pinephone-image)
- [Merge PinePhone into NuttX Mainline](#merge-pinephone-into-nuttx-mainline)
- [LVGL on NuttX on PinePhone](#lvgl-on-nuttx-on-pinephone)
- [LVGL Settings for PinePhone](#lvgl-settings-for-pinephone)
- [LVGL Demos on PinePhone](#lvgl-demos-on-pinephone)
- [Boot to LVGL on PinePhone](#boot-to-lvgl-on-pinephone)
- [PinePhone Touch Panel](#pinephone-touch-panel)
- [Handle Interrupts from Touch Panel](#handle-interrupts-from-touch-panel)
- [NuttX Touch Panel Driver for PinePhone](#nuttx-touch-panel-driver-for-pinephone)
- [LVGL Terminal for NuttX](#lvgl-terminal-for-nuttx)
- [Pipe a Command to NuttX NSH Shell](#pipe-a-command-to-nuttx-nsh-shell)
- [Poll for NSH Output](#poll-for-nsh-output)
- [Timer for LVGL Terminal](#timer-for-lvgl-terminal)
- [Poll for NSH Output in LVGL Timer](#poll-for-nsh-output-in-lvgl-timer)
- [Render Terminal with LVGL Widgets](#render-terminal-with-lvgl-widgets)
- [Set LVGL Terminal Font to Monospace](#set-lvgl-terminal-font-to-monospace)
- [Handle Input from LVGL Keyboard](#handle-input-from-lvgl-keyboard)
- [Handle Output from NSH Shell](#handle-output-from-nsh-shell)
- [PinePhone on NuttX becomes a Feature Phone](#pinephone-on-nuttx-becomes-a-feature-phone)
- [USB Driver and LTE Modem Driver for PinePhone](#usb-driver-and-lte-modem-driver-for-pinephone)
- [4G LTE Modem](#4g-lte-modem)
- [Outgoing Phone Call](#outgoing-phone-call)
- [Send SMS in Text Mode](#send-sms-in-text-mode)
- [Send SMS in PDU Mode](#send-sms-in-pdu-mode)
- [SMS PDU Format](#sms-pdu-format)
- [SMS Text Mode vs PDU Mode](#sms-text-mode-vs-pdu-mode)
- [PinePhone Accelerometer and Gyroscope](#pinephone-accelerometer-and-gyroscope)
- [Compile NuttX on Android with Termux](#compile-nuttx-on-android-with-termux)
- [Emulate PinePhone with Unicorn Emulator](#emulate-pinephone-with-unicorn-emulator)
- [Simulate PinePhone UI with Zig, LVGL and WebAssembly](#simulate-pinephone-ui-with-zig-lvgl-and-webassembly)
- [Apache NuttX RTOS for PinePhone Pro](#apache-nuttx-rtos-for-pinephone-pro)
- [Test Logs](#test-logs)
- [USB Devices on PinePhone](#usb-devices-on-pinephone)
- [Testing Zig Backlight Driver on PinePhone](#testing-zig-backlight-driver-on-pinephone)
- [Testing Zig Display Engine Driver on PinePhone](#testing-zig-display-engine-driver-on-pinephone)
- [Testing Zig Display Engine Driver on QEMU](#testing-zig-display-engine-driver-on-qemu)
- [Testing p-boot Display Engine on PinePhone](#testing-p-boot-display-engine-on-pinephone)
- [Testing NuttX Zig Driver for MIPI DSI on PinePhone](#testing-nuttx-zig-driver-for-mipi-dsi-on-pinephone)
- [Testing NuttX Zig Driver for MIPI DSI on QEMU](#testing-nuttx-zig-driver-for-mipi-dsi-on-qemu)
- [Testing p-boot Driver for MIPI DSI (with logging)](#testing-p-boot-driver-for-mipi-dsi-with-logging)
- [Testing p-boot Driver for MIPI DSI (without logging)](#testing-p-boot-driver-for-mipi-dsi-without-logging)
- [Testing Zig on PinePhone](#testing-zig-on-pinephone)
- [Testing GIC Version 2 on PinePhone](#testing-gic-version-2-on-pinephone)
- [Testing GIC Version 2 on QEMU](#testing-gic-version-2-on-qemu)
- [Boot Files for Manjaro Phosh on PinePhone](#boot-files-for-manjaro-phosh-on-pinephone)
- [GIC Register Dump](#gic-register-dump)
# NuttX Automated Daily Build for PinePhone
NuttX for PinePhone is now built automatically every day via GitHub Actions.
The Daily Releases are available here...
- [pinephone-nuttx/releases](https://github.com/lupyuen/pinephone-nuttx/releases)
[nuttx.hash](https://github.com/lupyuen/pinephone-nuttx/releases/download/pinephone-nuttx-2023-08-12/nuttx.hash) contains the Commit Hash of the NuttX Kernel and NuttX Apps repos...
```text
NuttX Source: https://github.com/apache/nuttx/tree/4bb30ab0c136dc36182cf43995337fc1d44f0a37
NuttX Apps: https://github.com/apache/nuttx-apps/tree/5a4e9e4389264522d02dae78bcb3f9813743300f
```
The GitHub Actions Workflow is here...
- [pinephone.yml](https://github.com/lupyuen/pinephone-nuttx/blob/main/.github/workflows/pinephone.yml)
# NuttX on QEMU
Note: This section is outdated, please check this article for updates...
- ["Apache NuttX RTOS on Arm Cortex-A53: How it might run on PinePhone"](https://lupyuen.github.io/articles/arm)
[Apache NuttX RTOS](https://nuttx.apache.org/docs/latest/) now runs on Arm Cortex-A53 with Multi-Core SMP...
- [nuttx/boards/arm64/qemu/qemu-a53](https://github.com/apache/incubator-nuttx/tree/master/boards/arm64/qemu/qemu-a53)
PinePhone is based on [Allwinner A64 SoC](https://linux-sunxi.org/A64) with 4 Cores of Arm Cortex-A53...
- [PinePhone Wiki](https://wiki.pine64.org/index.php/PinePhone)
We start with NuttX Mainline, run it on QEMU, then mod it for PinePhone.
## Download NuttX
Download the Source Code for NuttX Mainline, which supports Arm Cortex-A53...
```bash
## Create NuttX Directory
mkdir nuttx
cd nuttx
## Download NuttX OS
git clone \
--recursive \
--branch arm64 \
https://github.com/lupyuen/incubator-nuttx \
nuttx
## Download NuttX Apps
git clone \
--recursive \
--branch arm64 \
https://github.com/lupyuen/incubator-nuttx-apps \
apps
## We'll build NuttX inside nuttx/nuttx
cd nuttx
```
Install the Build Prerequisites, skip the RISC-V Toolchain...
- ["Install Prerequisites"](https://lupyuen.github.io/articles/nuttx#install-prerequisites)
## Download Toolchain
Download the Arm Toolchain for AArch64 ELF Bare-Metal Target (`aarch64-none-elf`)...
- [Arm GNU Toolchain Downloads](https://developer.arm.com/downloads/-/arm-gnu-toolchain-downloads)
For Linux x64 and WSL:
- [gcc-arm-11.2-2022.02-x86_64-aarch64-none-elf.tar.xz](https://developer.arm.com/-/media/Files/downloads/gnu/11.2-2022.02/binrel/gcc-arm-11.2-2022.02-x86_64-aarch64-none-elf.tar.xz)
For macOS:
- [arm-gnu-toolchain-11.3.rel1-darwin-x86_64-aarch64-none-elf.pkg](https://developer.arm.com/-/media/Files/downloads/gnu/11.3.rel1/binrel/arm-gnu-toolchain-11.3.rel1-darwin-x86_64-aarch64-none-elf.pkg)
(I don't recommend building NuttX on Plain Old Windows CMD, please use WSL instead)
Add it to the `PATH`...
```bash
## For Linux x64 and WSL:
export PATH="$PATH:$HOME/gcc-arm-11.2-2022.02-x86_64-aarch64-none-elf/bin"
## For macOS:
export PATH="$PATH:/Applications/ArmGNUToolchain/11.3.rel1/aarch64-none-elf/bin"
```
Check the toolchain...
```bash
aarch64-none-elf-gcc -v
```
[(Based on the instructions here)](https://github.com/apache/incubator-nuttx/tree/master/boards/arm64/qemu/qemu-a53)
## Download QEMU
Download and install QEMU...
- [Download QEMU](https://www.qemu.org/download/)
For macOS we may use `brew`...
```bash
brew install qemu
```
## Build NuttX: Single Core
First we build NuttX for a Single Core of Arm Cortex-A53...
```bash
## Configure NuttX for Single Core
./tools/configure.sh -l qemu-a53:nsh
## Build NuttX
make
## Dump the disassembly to nuttx.S
aarch64-none-elf-objdump \
-t -S --demangle --line-numbers --wide \
nuttx \
>nuttx.S \
2>&1
```
The NuttX Output Files may be found here...
- [NuttX for Arm Cortex-A53 Single Core](https://github.com/lupyuen/pinephone-nuttx/releases/tag/v1.0.1)
## Test NuttX with QEMU: Single Core
This is how we test NuttX on QEMU with a Single Core of Arm Cortex-A53...
```bash
## Start QEMU (Single Core) with NuttX
qemu-system-aarch64 \
-cpu cortex-a53 \
-nographic \
-machine virt,virtualization=on,gic-version=3 \
-net none \
-chardev stdio,id=con,mux=on \
-serial chardev:con \
-mon chardev=con,mode=readline \
-kernel ./nuttx
```
Here's NuttX with a Single Core running on QEMU...
```text
- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
nx_start: Entry
up_allocate_heap: heap_start=0x0x402c4000, heap_size=0x7d3c000
gic_validate_dist_version: GICv3 version detect
gic_validate_dist_version: GICD_TYPER = 0x37a0007
gic_validate_dist_version: 224 SPIs implemented
gic_validate_dist_version: 0 Extended SPIs implemented
gic_validate_dist_version: Distributor has no Range Selector support
gic_validate_redist_version: GICD_TYPER = 0x1000011
gic_validate_redist_version: 16 PPIs implemented
gic_validate_redist_version: no VLPI support, no direct LPI support
up_timer_initialize: up_timer_initialize: cp15 timer(s) running at 62.50MHz, cycle 62500
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
work_start_highpri: Starting high-priority kernel worker thread(s)
nx_start_application: Starting init thread
lib_cxx_initialize: _sinit: 0x402a7000 _einit: 0x402a7000 _stext: 0x40280000 _etext: 0x402a8000
nsh: sysinit: fopen failed: 2
nsh: mkfatfs: command not found
NuttShell (NSH) NuttX-10.3.0-RC2
nsh> nx_start: CPU0: Beginning Idle Loop
nsh> help
help usage: help [-v] []
. cd dmesg help mount rmdir true xd
[ cp echo hexdump mv set truncate
? cmp exec kill printf sleep uname
basename dirname exit ls ps source umount
break dd false mkdir pwd test unset
cat df free mkrd rm time usleep
Builtin Apps:
getprime hello nsh ostest sh
nsh> uname -a
NuttX 10.3.0-RC2 1e8f2a8 Aug 23 2022 07:04:54 arm64 qemu-a53
nsh> hello
task_spawn: name=hello entry=0x4029b594 file_actions=0x402c9580 attr=0x402c9588 argv=0x402c96d0
spawn_execattrs: Setting policy=2 priority=100 for pid=3
Hello, World!!
nsh> ls /
/:
dev/
etc/
proc/
nsh> ls /dev
/dev:
console
null
ram0
ram2
ttyS0
zero
nsh> ls /proc
/proc:
0/
1/
2/
meminfo
memdump
fs/
self/
uptime
version
nsh> ls /etc
/etc:
init.d/
nsh> ls /etc/init.d
/etc/init.d:
rcS
nsh> cat /etc/init.d/rcS
# Create a RAMDISK and mount it at /tmp
mkrd -m 2 -s 512 1024
mkfatfs /dev/ram2
mount -t vfat /dev/ram2 /tmp
```
NuttX is [POSIX Compliant](https://nuttx.apache.org/docs/latest/introduction/inviolables.html), so the developer experience feels very much like Linux. (But much smaller)
And NuttX runs everything in RAM, no File System needed. (For now)
## Build NuttX: Multi Core
From Single Core to Multi Core! Now we build NuttX for 4 Cores of Arm Cortex-A53...
```bash
## Erase the NuttX Configuration
make distclean
## Configure NuttX for 4 Cores
./tools/configure.sh -l qemu-a53:nsh_smp
## Build NuttX
make
## Dump the disassembly to nuttx.S
aarch64-none-elf-objdump \
-t -S --demangle --line-numbers --wide \
nuttx \
>nuttx.S \
2>&1
```
The NuttX Output Files may be found here...
- [NuttX for Arm Cortex-A53 Multi-Core](https://github.com/lupyuen/pinephone-nuttx/releases/tag/v1.0.0)
## Test NuttX with QEMU: Multi Core
And this is how we test NuttX on QEMU with 4 Cores of Arm Cortex-A53...
```bash
## Start QEMU (4 Cores) with NuttX
qemu-system-aarch64 \
-smp 4 \
-cpu cortex-a53 \
-nographic \
-machine virt,virtualization=on,gic-version=3 \
-net none \
-chardev stdio,id=con,mux=on \
-serial chardev:con \
-mon chardev=con,mode=readline \
-kernel ./nuttx
```
Note that `smp` is set to 4. [(Symmetric Multi-Processing)](https://developer.arm.com/documentation/den0024/a/Multi-core-processors/Multi-processing-systems/Symmetric-multi-processing?lang=en)
Here's NuttX with 4 Cores running on QEMU...
```text
- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
[CPU0] psci_detect: Detected PSCI v1.1
[CPU0] nx_start: Entry
[CPU0] up_allocate_heap: heap_start=0x0x402db000, heap_size=0x7d25000
[CPU0] gic_validate_dist_version: GICv3 version detect
[CPU0] gic_validate_dist_version: GICD_TYPER = 0x37a0007
[CPU0] gic_validate_dist_version: 224 SPIs implemented
[CPU0] gic_validate_dist_version: 0 Extended SPIs implemented
[CPU0] gic_validate_dist_version: Distributor has no Range Selector support
[CPU0] gic_validate_redist_version: GICD_TYPER = 0x1000001
[CPU0] gic_validate_redist_version: 16 PPIs implemented
[CPU0] gic_validate_redist_version: no VLPI support, no direct LPI support
[CPU0] up_timer_initialize: up_timer_initialize: cp15 timer(s) running at 62.50MHz, cycle 62500
[CPU0] uart_register: Registering /dev/console
[CPU0] uart_register: Registering /dev/ttyS0
- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
[CPU1] gic_validate_redist_version: GICD_TYPER = 0x101000101
[CPU1] gic_validate_redist_version: 16 PPIs implemented
[CPU1] gic_validate_redist_version: no VLPI support, no direct LPI support
[CPU1] nx_idle_trampoline: CPU1: Beginning Idle Loop
[CPU0] arm64_start_cpu: Secondary CPU core 1 (MPID:0x1) is up
- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
[CPU2] gic_validate_redist_version: GICD_TYPER = 0x201000201
[CPU2] gic_validate_redist_version: 16 PPIs implemented
[CPU2] gic_validate_redist_version: no VLPI support, no direct LPI support
[CPU2] nx_idle_trampoline: CPU2: Beginning Idle Loop
[CPU0] arm64_start_cpu: Secondary CPU core 2 (MPID:0x2) is up
- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
[CPU3] gic_validate_redist_version: GICD_TYPER = 0x301000311
[CPU3] gic_validate_redist_version: 16 PPIs implemented
[CPU3] gic_validate_redist_version: no VLPI support, no direct LPI support
[CPU0] arm64_start_cpu: Secondary CPU core 3 (MPID:0x3) is up
[CPU0] work_start_highpri: Starting high-priority kernel worker thread(s)
[CPU0] nx_start_application: Starting init thread
[CPU3] nx_idle_trampoline: CPU3: Beginning Idle Loop
[CPU0] nx_start: CPU0: Beginning Idle Loop
nsh: sysinit: fopen failed: 2
nsh: mkfatfs: command not found
NuttShell (NSH) NuttX-10.3.0-RC2
nsh> help
help usage: help [-v] []
. cd dmesg help mount rmdir true xd
[ cp echo hexdump mv set truncate
? cmp exec kill printf sleep uname
basename dirname exit ls ps source umount
break dd false mkdir pwd test unset
cat df free mkrd rm time usleep
Builtin Apps:
getprime hello nsh ostest sh smp taskset
nsh> uname -a
NuttX 10.3.0-RC2 1e8f2a8 Aug 21 2022 15:57:35 arm64 qemu-a53
nsh> hello
[CPU0] task_spawn: name=hello entry=0x4029cee4 file_actions=0x402e52b0 attr=0x402e52b8 argv=0x402e5400
[CPU0] spawn_execattrs: Setting policy=2 priority=100 for pid=6
Hello, World!
```
We see each of the 4 Cores starting NuttX (CPU0 to CPU3). That's so cool!
(Can we use QEMU to partially emulate PinePhone? That would be extremely helpful!)
## Inside NuttX for Cortex-A53
Now we browse the Source Files for the implementation of Cortex-A53 on NuttX.
NuttX treats QEMU as a Target Board (as though it was a dev board). Here are the Source Files and Build Configuration for the QEMU Board...
- [nuttx/boards/arm64/qemu/qemu-a53](https://github.com/apache/incubator-nuttx/tree/master/boards/arm64/qemu/qemu-a53)
(We'll clone this to create a Target Board for PinePhone)
The Board-Specific Drivers for QEMU are started in [qemu-a53/src/qemu_bringup.c](https://github.com/apache/incubator-nuttx/blob/master/boards/arm64/qemu/qemu-a53/src/qemu_bringup.c)
(We'll start the PinePhone Drivers here)
The QEMU Board calls the QEMU Architecture-Specific Drivers at...
- [nuttx/arch/arm64/src/qemu](https://github.com/apache/incubator-nuttx/tree/master/arch/arm64/src/qemu)
The UART Driver is located at [qemu/qemu_serial.c](https://github.com/apache/incubator-nuttx/blob/master/arch/arm64/src/qemu/qemu_serial.c) and [qemu/qemu_lowputc.S](https://github.com/apache/incubator-nuttx/blob/master/arch/arm64/src/qemu/qemu_lowputc.S)
(For PinePhone we'll create a UART Driver for Allwinner A64 SoC. I2C, SPI and other Low-Level A64 Drivers will be located here too)
The QEMU Functions (Board and Architecture) call the Arm64 Architecture Functions at...
- [nuttx/arch/arm64/src/common](https://github.com/apache/incubator-nuttx/tree/master/arch/arm64/src/common)
Which implements all kinds of Arm64 Features: [FPU](https://github.com/apache/incubator-nuttx/blob/master/arch/arm64/src/common/arm64_fpu.c), [Interrupts](https://github.com/apache/incubator-nuttx/blob/master/arch/arm64/src/common/arm64_gicv3.c), [MMU](https://github.com/apache/incubator-nuttx/blob/master/arch/arm64/src/common/arm64_mmu.c), [Tasks](https://github.com/apache/incubator-nuttx/blob/master/arch/arm64/src/common/arm64_task_sched.c), [Timers](https://github.com/apache/incubator-nuttx/blob/master/arch/arm64/src/common/arm64_arch_timer.c)...
(We'll reuse them for PinePhone)
## NuttX Image
Next we analyse the NuttX Image with [Ghidra](https://ghidra-sre.org/), to understand the NuttX Image Header and Startup Code.
Here's the [NuttX ELF Image `nuttx`](https://github.com/lupyuen/pinephone-nuttx/releases/download/v1.0.0/nuttx) analysed by Ghidra...
Note that the NuttX Image jumps to `real_start` (to skip the Image Header)...
```text
40280000 4d 5a 00 91 add x13,x18,#0x16
40280004 0f 00 00 14 b real_start
```
`real_start` is defined at 0x4028 0040 with the Startup Code...
We see something interesting: The Magic Number `ARM\x64` appears at address 0x4028 0038.
Searching the net for this Magic Number reveals that it's actually an Arm64 Linux Kernel Header!
When we refer to the [NuttX Arm64 Disassembly `nuttx.S`](https://github.com/lupyuen/pinephone-nuttx/releases/download/v1.0.0/nuttx.S), we find happiness: [arch/arm64/src/common/arm64_head.S](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_head.S#L79-L117)
```text
/* Kernel startup entry point.
* ---------------------------
*
* The requirements are:
* MMU = off, D-cache = off, I-cache = on or off,
* x0 = physical address to the FDT blob.
* it will be used when NuttX support device tree in the future
*
* This must be the very first address in the loaded image.
* It should be loaded at any 4K-aligned address.
*/
.globl __start;
__start:
/* DO NOT MODIFY. Image header expected by Linux boot-loaders.
*
* This add instruction has no meaningful effect except that
* its opcode forms the magic "MZ" signature of a PE/COFF file
* that is required for UEFI applications.
*
* Some bootloader (such imx8 uboot) checking the magic "MZ" to see
* if the image is a valid Linux image. but modifying the bootLoader is
* unnecessary unless we need to do a customize secure boot.
* so just put the ''MZ" in the header to make bootloader happiness
*/
add x13, x18, #0x16 /* the magic "MZ" signature */
b real_start /* branch to kernel start */
.quad 0x480000 /* Image load offset from start of RAM */
.quad _e_initstack - __start /* Effective size of kernel image, little-endian */
.quad __HEAD_FLAGS /* Informative flags, little-endian */
.quad 0 /* reserved */
.quad 0 /* reserved */
.quad 0 /* reserved */
.ascii "ARM\x64" /* Magic number, "ARM\x64" */
.long 0 /* reserved */
real_start:
/* Disable all exceptions and interrupts */
```
NuttX Image actually follows the Arm64 Linux Kernel Image Format! As defined here...
- ["Booting AArch64 Linux"](https://www.kernel.org/doc/html/latest/arm64/booting.html)
Arm64 Linux Kernel Image contains a 64-byte header...
```text
u32 code0; /* Executable code */
u32 code1; /* Executable code */
u64 text_offset; /* Image load offset, little endian */
u64 image_size; /* Effective Image size, little endian */
u64 flags; /* kernel flags, little endian */
u64 res2 = 0; /* reserved */
u64 res3 = 0; /* reserved */
u64 res4 = 0; /* reserved */
u32 magic = 0x644d5241; /* Magic number, little endian, "ARM\x64" */
u32 res5; /* reserved (used for PE COFF offset) */
```
Start of RAM is 0x4000 0000. The Image Load Offset in our NuttX Image Header is 0x48 0000 according to [arch/arm64/src/common/arm64_head.S](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_head.S#L107)
```text
.quad 0x480000 /* Image load offset from start of RAM */
```
This means that our NuttX Image will be loaded at 0x4048 0000.
I wonder if this Image Load Offset should have been 0x28 0000? (Instead of 0x48 0000)
Remember that Ghidra (and the Arm Disassembly) says that our NuttX Image is actually loaded at 0x4028 0000. (Instead of 0x4048 0000)
RAM Size and RAM Start are defined in the NuttX Configuration: [boards/arm64/qemu/qemu-a53/configs/nsh_smp/defconfig](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/boards/arm64/qemu/qemu-a53/configs/nsh_smp/defconfig#L47-L48)
```text
CONFIG_RAM_SIZE=134217728
CONFIG_RAM_START=0x40000000
```
That's 128 MB RAM. Which should fit inside PinePhone's 2 GB RAM.
The NuttX Image was built with this Linker Command, based on `make --trace`...
```bash
aarch64-none-elf-ld \
--entry=__start \
-nostdlib \
--cref \
-Map=nuttx/nuttx/nuttx.map \
-Tnuttx/nuttx/boards/arm64/qemu/qemu-a53/scripts/dramboot.ld \
-L nuttx/nuttx/staging \
-L nuttx/nuttx/arch/arm64/src/board \
-o nuttx/nuttx/nuttx arm64_head.o \
--start-group \
-lsched \
-ldrivers \
-lboards \
-lc \
-lmm \
-larch \
-lapps \
-lfs \
-lbinfmt \
-lboard /Applications/ArmGNUToolchain/11.3.rel1/aarch64-none-elf/bin/../lib/gcc/aarch64-none-elf/11.3.1/libgcc.a /Applications/ArmGNUToolchain/11.3.rel1/aarch64-none-elf/bin/../lib/gcc/aarch64-none-elf/11.3.1/../../../../aarch64-none-elf/lib/libm.a \
--end-group
```
NuttX Image begins at `__start`, which is defined as 0x4028 0000 in the NuttX Linker Script: [boards/arm64/qemu/qemu-a53/scripts/dramboot.ld](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/boards/arm64/qemu/qemu-a53/scripts/dramboot.ld#L30-L33)
```text
SECTIONS
{
. = 0x40280000; /* uboot load address */
_start = .;
```
We'll change this to 0x4008 0000 for PinePhone, since Kernel Start Address is 0x4008 0000 and Image Load Offset is 0. (See below)
We've seen the NuttX Image (which looks like a Linux Kernel Image), let's compare with a PinePhone Linux Kernel Image and see how NuttX needs to be tweaked...
## PinePhone Image
Will NuttX run on PinePhone? Let's analyse a PinePhone Linux Kernel Image with Ghidra, to look at the Linux Kernel Header and Startup Code.
We'll use the PinePhone Jumpdrive Image, since it's small...
https://github.com/dreemurrs-embedded/Jumpdrive
Download https://github.com/dreemurrs-embedded/Jumpdrive/releases/download/0.8/pine64-pinephone.img.xz
Expand `pine64-pinephone.img.xz`
Expand the files inside...
```bash
gunzip Image.gz
gunzip initramfs.gz
tar xvf initramfs
```
Import the uncompressed `Image` (Linux Kernel) into Ghidra.
For "Language" select AARCH64:LE:v8A:default...
- Processor: AARCH64
- Variant: v8A
- Size: 64
- Endian: little
- Compiler: default
Here's the Jumpdrive `Image` (Linux Kernel) in Ghidra...
According to the Linux Kernel Header...
- ["Booting AArch64 Linux"](https://www.kernel.org/doc/html/latest/arm64/booting.html)
We see Linux Kernel Magic Number `ARM\x64` at offset 0x38.
Image Load Offset is 0, according to the header.
Kernel Start Address on PinePhone is 0x4008 0000.
So we shift `Image` in Ghidra to start at 0x4008 0000...
- Click Window > Memory Map
- Click "ram"
- Click the 4-Arrows icon ("Move a block to another address")
- Change "New Start Address" to 40080000
## Will NuttX Boot On PinePhone?
_So will NuttX boot on PinePhone?_
It's highly plausible! We discovered (with happiness) that NuttX already generates an Arm64 Linux Kernel Header.
So NuttX could be a drop-in replacement for the PinePhone Linux Kernel! We just need to...
- Write PinePhone Jumpdrive to a microSD Card (with Etcher, in FAT format)
- Overwrite `Image.gz` by the (gzipped) NuttX Binary Image `nuttx.bin.gz`
- Insert the microSD Card into PinePhone
- Power on PinePhone
And NuttX should (theoretically) boot on PinePhone!
As mentioned earlier, we should rebuild NuttX so that `__start` is changed to 0x4008 0000 (from 0x4028 0000), as defined in the NuttX Linker Script: [boards/arm64/qemu/qemu-a53/scripts/dramboot.ld](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/boards/arm64/qemu/qemu-a53/scripts/dramboot.ld#L30-L33)
```text
SECTIONS
{
SECTIONS
{
. = 0x40080000; /* PinePhone uboot load address (kernel_addr_r) */
/* Previously: . = 0x40280000; */ /* uboot load address */
_start = .;
```
Also the Image Load Offset in our NuttX Image Header should be changed to 0x0 (from 0x48 0000): [arch/arm64/src/common/arm64_head.S](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_head.S#L107)
```text
.quad 0x0000 /* PinePhone Image load offset from start of RAM */
# Previously: .quad 0x480000 /* Image load offset from start of RAM */
```
Later we'll increase the RAM Size to 2 GB (from 128 MB): [boards/arm64/qemu/qemu-a53/configs/nsh_smp/defconfig](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/boards/arm64/qemu/qemu-a53/configs/nsh_smp/defconfig#L47-L48)
```text
/* TODO: Increase to 2 GB for PinePhone */
CONFIG_RAM_SIZE=134217728
CONFIG_RAM_START=0x40000000
```
But not right now, because it might clash with the Device Tree and RAM File System.
_But will we see anything when NuttX boots on PinePhone?_
Not yet. We'll need to implement the UART Driver for NuttX...
# UART Driver for NuttX
Read the article...
- ["NuttX RTOS for PinePhone: UART Driver"](https://lupyuen.github.io/articles/serial)
We won't see any output from NuttX until we implement the UART Driver for NuttX.
These are the Source Files for the QEMU UART Driver (PL011)...
- [arch/arm64/src/qemu/qemu_serial.c](https://github.com/apache/incubator-nuttx/blob/master/arch/arm64/src/qemu/qemu_serial.c)
- [arch/arm64/src/qemu/qemu_lowputc.S](https://github.com/apache/incubator-nuttx/blob/master/arch/arm64/src/qemu/qemu_lowputc.S)
[(More about PL011 UART)](https://krinkinmu.github.io/2020/11/29/PL011.html)
We'll replace the code above with the UART Driver for Allwinner A64 SoC...
- [UART0 Memory Map](https://linux-sunxi.org/A64/Memory_map)
- [Allwinner A64 UART](https://linux-sunxi.org/UART)
- [Allwinner A64 User Manual](https://linux-sunxi.org/File:Allwinner_A64_User_Manual_V1.1.pdf)
- [Allwinner A64 Info](https://linux-sunxi.org/A64)
To access the UART Port on PinePhone, we'll use this USB Serial Debug Cable...
- [PinePhone Serial Debug Cable](https://wiki.pine64.org/index.php/PinePhone#Serial_console)
Which connects to the Headphone Port. Genius!
[(Remember to flip the Headphone Switch to OFF)](https://wiki.pine64.org/index.php/PinePhone#Privacy_switch_configuration)
[_PinePhone UART Port in disguise_](https://wiki.pine64.org/index.php/PinePhone#Serial_console)
## Garbled Console Output
The log appears garbled when `printf` is called by our NuttX Test Apps, due to concurrent printing by multiple tasks...
```text
nx_start_application: Starting init thread
lib_cxx_initialize: _sinit: 0x400e9000 _einit: 0x400e9000
nsh: sysinit: fopen failed: 2
nshn:x _msktfaarttf:s :C PcUo0m:m aBnedg innonti nfgo uInddleLoNouptt
Shell (NSH) NuttX-11.0.0-RC2
```
[(Source)](https://gist.github.com/lupyuen/e49a22a9e39b7c024b984bea40377712)
It's supposed to show...
```text
nsh: sysinit: fopen failed: 2
nsh: mkfatfs: command not found
NuttShell (NSH) NuttX-11.0.0-RC2
nsh> nx_start: CPU0: Beginning Idle Loop
```
[(Source)](https://gist.github.com/lupyuen/7537da777d728a22ab379b1ef234a2d1)
_Why is the output garbled?_
The output is garbled because there 2 ways of writing to the Console Output...
- `up_putc`: Called by the NuttX Kernel to log messages (like `_info`)
- `a64_uart_send`: Allwinner A64 UART Driver is called by NuttX Apps to print messages (like `printf`)
To fix the output garbled at startup: Disable "Scheduler Informational Output" in...
"Build Setup > Debug Options > Enable Debug Features > Scheduler Debug Features"
This prevents `sinfo` from garbling the `printf` output...
- `sinfo` writes directly to UART Port character by character...
```text
nx_start: CPU0: Beginning Idle Loop
```
- Whereas `printf` is buffered and writes the buffer to the UART Driver...
```text
nsh: mkfatfs: command not found
NuttShell (NSH) NuttX-11.0.0-RC2
```
Also we have a temporary workaround until this gets fixed: [a64_serial.c](https://github.com/lupyuen2/wip-pinephone-nuttx/blob/91516c7cef28a79516d6196f070320a16649cf90/arch/arm64/src/a64/a64_serial.c#L424-L452)
```c
static void a64_uart_send(struct uart_dev_s *dev, int ch)
{
const struct a64_uart_port_s *port = (struct a64_uart_port_s *)dev->priv;
const struct a64_uart_config *config = &port->config;
#define BUFFER_UART
#ifdef BUFFER_UART
//// TODO: Fix the garbled output. Buffer the chars until we see CR or LF.
static char buf[256];
static int pos = 0;
if (ch != '>' && ch != '\r' && ch != '\n' && pos < sizeof(buf))
{
buf[pos] = ch;
pos += 1;
return;
}
for (int i = 0; i < pos; i++)
{
up_putc(buf[i]);
}
pos = 0;
up_putc(ch);
UNUSED(config);
#else
/* Write char to Transmit Holding Register (UART_THR) */
putreg8(ch, UART_THR(config->uart));
#endif // BUFFER_UART
}
```
This forces `a64_uart_send` to call `up_putc` to print an entire line of text. (Instead of character by character)
This also means that the `nsh` prompt will no longer echo text character by character. We need to press Enter to see the `nsh` output.
FYI: `printf` Console Output Stream is locked and unlocked with a Mutex. Let's log the locking and unlocking of the Mutex...
[nuttx/libs/libc/stdio/lib_libfilelock.c](https://github.com/apache/nuttx/blob/master/libs/libc/stdio/lib_libfilelock.c#L39-L64)
```c
void flockfile(FAR struct file_struct *stream)
{
up_putc('{'); // Log the Mutex Locking
nxrmutex_lock(&stream->fs_lock);
}
...
void funlockfile(FAR struct file_struct *stream)
{
up_putc('}'); // Log the Mutex Unlocking
nxrmutex_unlock(&stream->fs_lock);
}
```
Output log shows that `{` and `}` (Mutex Locking and Unlocking) are nested...
```text
nx_start_application: Starting init thread
lib_cxx_initialize: _sinit: 0x400e9000 _einit: 0x400e9000
{{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{n}s}h{:} {s}y{s}i{n}i{t}:{ }f{o}p{e}n{ }f{a}i{l}e{d}:{ }2{
}
{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{n}s}h{:} {m{k}f}a{t}f{s{:} }c{o{m}m}a{n{d} }n{o{t} }f{o{u}n}d{
```
How can be it locked twice without unlocking?
[`nxrmutex_lock`](https://github.com/apache/nuttx/blob/master/include/nuttx/mutex.h#L335-L377) calls...
- [`nxmutex_lock`](https://github.com/apache/nuttx/blob/master/include/nuttx/mutex.h#L135-L179), which calls...
- [`nxsem_wait`](https://github.com/apache/nuttx/blob/master/sched/semaphore/sem_wait.c#L42-L210), which calls...
- [`up_switch_context`](https://github.com/apache/nuttx/blob/master/arch/arm64/src/common/arm64_switchcontext.c#L41-L103)
Let's print the Thread ID and Mutex Count...
```text
void flockfile(FAR struct file_struct *stream)
{
nxrmutex_lock(&stream->fs_lock);
_info("%p, thread=%d, mutex.count=%d\n", stream, gettid(), stream->fs_lock.count); // Log the Thread ID and Mutex Count
}
void funlockfile(FAR struct file_struct *stream)
{
_info("%p, thread=%d, mutex.count=%d\n", stream, gettid(), stream->fs_lock.count); // Log the Thread ID and Mutex Count
nxrmutex_unlock(&stream->fs_lock);
}
```
Thread ID is always the same. Mutex Count goes from 1 to 3 and drops to 2...
```text
lib_cxx_initialize: _sinit: 0x400e9000 _einit: 0x400e9000
flockfile: 0x40a5cc78, thread=2, mutex.count=1
flockfile: 0x40a5cc78, thread=2, mutex.count=2
flockfile: 0x40a5cc78, thread=2, mutex.count=3
funlockfile: 0x40a5cc78, thread=2, mutex.count=3
funlockfile: 0x40a5cc78, thread=2, mutex.count=2
```
Why? That's because [`nxrmutex_lock`](https://github.com/apache/nuttx/blob/master/include/nuttx/mutex.h#L335-L377) allows the Mutex to be locked multiple times __within the same thread.__
FYI: Here's how we verify whether our code is called by multiple CPU Cores...
```c
#include "../arch/arm64/src/common/arm64_arch.h"
_info("up_cpu_index=%d\n", MPIDR_TO_CORE(GET_MPIDR()));
// Shows: `up_cpu_index=0`
_info("mpidr_el1=%p\n", read_sysreg(mpidr_el1));
// Shows `mpidr_el1=0x80000000`
```
FYI: How `printf` works...
[`printf`](https://github.com/apache/nuttx/blob/master/libs/libc/stdio/lib_printf.c#L32-L51) calls...
- [`vfprintf`](https://github.com/apache/nuttx/blob/master/libs/libc/stdio/lib_vfprintf.c#L34-L56), which calls...
- [`libvsprintf`](https://github.com/apache/nuttx/blob/master/libs/libc/stdio/lib_libvsprintf.c#L1336-L1381), which calls...
- [`vsprintf_internal`](https://github.com/apache/nuttx/blob/master/libs/libc/stdio/lib_libvsprintf.c#L171-L1332), which calls...
- [`stream_putc`](https://github.com/apache/nuttx/blob/master/libs/libc/stdio/lib_libvsprintf.c#L70), which calls...
- ???, which calls...
- [`fputc`](https://github.com/apache/nuttx/blob/master/libs/libc/stdio/lib_fputc.c#L31-L61), which calls...
- [`libfwrite`](https://github.com/apache/nuttx/blob/master/libs/libc/stdio/lib_libfwrite.c#L41-L180)
[`fputc`](https://github.com/apache/nuttx/blob/master/libs/libc/stdio/lib_fputc.c#L31-L61) also calls...
- [`lib_libfflush`](https://github.com/apache/nuttx/blob/master/libs/libc/stdio/lib_libfflush.c#L40-L171)
## Configure UART Port
To support multiple UART Ports, we copied the following functions from the [Allwinner A1X UART Driver](https://github.com/apache/nuttx/blob/master/arch/arm/src/a1x/a1x_serial.c#L695-L987)...
- [`arm64_earlyserialinit`](https://github.com/lupyuen2/wip-pinephone-nuttx/blob/uart3/arch/arm64/src/a64/a64_serial.c#L1323-L1398)
(Init the UART Ports)
- [`a64_uart_init`](https://github.com/lupyuen2/wip-pinephone-nuttx/blob/uart3/arch/arm64/src/a64/a64_serial.c#L944-L996)
(Init the UART Pins. Called by [`arm64_earlyserialinit`](https://github.com/lupyuen2/wip-pinephone-nuttx/blob/uart3/arch/arm64/src/a64/a64_serial.c#L1323-L1398))
- [`a64_uart_setup`](https://github.com/lupyuen2/wip-pinephone-nuttx/blob/uart3/arch/arm64/src/a64/a64_serial.c#L453-L582)
(Configure the UART Port. Called when the UART Port is opened)
- [`a64_uart_irq_handler`](https://github.com/lupyuen2/wip-pinephone-nuttx/blob/uart3/arch/arm64/src/a64/a64_serial.c#L301-L415)
(UART Interrupt Handler)
- [`a64_uart_receive`](https://github.com/lupyuen2/wip-pinephone-nuttx/blob/uart3/arch/arm64/src/a64/a64_serial.c#L727-L753)
(Receive UART Data)
[(Here's the log)](https://github.com/lupyuen2/wip-pinephone-nuttx-apps/blob/e5aa2baba64c8d904d6c16b7c5dbc68cd5c8f1e1/examples/hello/hello_main.c#L72-L270)
Inside the UART Driver, be careful when logging to the UART Port!
If the UART Port is busy doing logging, it won't allow us to the Baud Rate...
> "This register may only be accessed when the DLAB bit (UART_LCR[7]) is set and the UART is not busy (UART_USR[0] is zero)"
[(Allwinner A64 User Manual, Page 564)](https://lupyuen.github.io/images/Allwinner_A64_User_Manual_V1.1.pdf)
That's why we always wait for UART Not Busy before setting the Baud Rate. [(Like this)](https://github.com/lupyuen2/wip-pinephone-nuttx/blob/uart3/arch/arm64/src/a64/a64_serial.c#L417-L451)
_What happens if UART Is Busy when we set the Baud Rate?_
```text
up_setup: Enter DLAB=1
up_serialout: addr=0x1c2800c, before=0x3, after=0x83
up_setup: Set the BAUD divisor
up_serialout: addr=0x1c28004, before=0x0, after=0x0
up_serialout: addr=0x1c28000, before=0x0, after=0xd
up_setup: Clear DLAB
up_serialout: addr=0x1c2800c, before=0x3, after=0x3
```
[(Source)](https://github.com/lupyuen2/wip-pinephone-nuttx/blob/83566af2e58f92ae42fd434ce97f919404245986/arch/arm64/src/a64/a64_serial.c#L1306-L1313)
From above, we see that the Baud Rate is not read correctly...
```text
up_serialout: addr=0x1c28000, before=0x0, after=0xd
// For UART0: Before should be 0xd
```
And DLAB doesn't get cleared correctly...
```text
up_setup: Enter DLAB=1
up_serialout: addr=0x1c2800c, before=0x3, after=0x83
...
up_setup: Clear DLAB
up_serialout: addr=0x1c2800c, before=0x3, after=0x3
// Before should be 0x83, not 0x3
```
We fix this by disabling the logging and waiting for UART Not Busy before setting the Baud Rate. [(Like this)](https://github.com/lupyuen2/wip-pinephone-nuttx/blob/uart3/arch/arm64/src/a64/a64_serial.c#L530-L549)
The changes have been upstreamed to NuttX Mainline...
- [__Pull Request: Support multiple UART Ports__](https://github.com/apache/nuttx/pull/9243)
_How do we enable a UART Port? Like UART3?_
Head over to the NuttX Build Configuration...
```bash
make menuconfig
```
Then select...
- System Type > Allwinner A64 Peripheral Selection > UART3
To configure the Baud Rate, Parity and Stop Bits...
- Device Drivers > Serial Driver Support > UART3 Configuration
(The default Baud Rate / Bits / Parity / Stop Bits work OK with the PinePhone LTE Modem on UART3: 115.2 kbps / 8 Bits / No Parity / 1 Stop Bit)
Let's test the PinePhone LTE Modem on UART3...
## Test UART3 Port
In the previous section we have configured UART3 for PinePhone's [4G LTE Modem](https://lupyuen.github.io/articles/lte).
This is how we read and write the UART3 port via `/dev/ttyS1`: [hello_main.c](https://github.com/lupyuen2/wip-pinephone-nuttx-apps/blob/e5aa2baba64c8d904d6c16b7c5dbc68cd5c8f1e1/examples/hello/hello_main.c#L38-L70)
```c
// Open /dev/ttyS1 (UART3)
int fd = open("/dev/ttyS1", O_RDWR);
printf("Open /dev/ttyS1: fd=%d\n", fd);
assert(fd > 0);
// Repeat 5 times: Write command and read response
for (int i = 0; i < 5; i++) {
// Write command
const char cmd[] = "AT\r";
ssize_t nbytes = write(fd, cmd, sizeof(cmd));
printf("Write command: nbytes=%ld\n", nbytes);
assert(nbytes == sizeof(cmd));
// Read response
static char buf[1024];
nbytes = read(fd, buf, sizeof(buf) - 1);
if (nbytes >= 0) { buf[nbytes] = 0; }
printf("Response: nbytes=%ld\n%s\n", nbytes, buf);
// Wait a while
sleep(2);
}
// Close the device
close(fd);
```
LTE Modem works OK with UART3 on NuttX yay! See the test log here...
- ["Test UART with NuttX"](https://lupyuen.github.io/articles/lte#test-uart-with-nuttx)
__Note:__ Modem UART flow control is broken
> "Not resolved in v1.2 -- assumption is that USB will be used for high-bandwidth modem I/O.
> BB-TX and BB-RX are connected to UART3 (PD0/PD1). BB-RTS and BB-CTS are connected to UART4 (PD4/PD5). To use hardware flow control, TX/RX would need to be connected to UART4, swapping PD0/PD1 with the motor control and rear camera reset GPIOs at PD2/PD3. This would need a device tree change.
> Hardware flow control can be disabled with the AT+IFC command, and USB can also be used for commands instead of the UART. So the impact of this problem is unclear."
[(Source)](https://wiki.pine64.org/wiki/PinePhone_v1.1_-_Braveheart#Modem_UART_flow_control_is_broken)
# Boot NuttX on PinePhone
Read the article...
- ["PinePhone boots Apache NuttX RTOS"](https://lupyuen.github.io/articles/uboot)
## PinePhone U-Boot Log
Before starting the Linux Kernel, PinePhone boots by running the U-Boot Bootloader...
- [A64 Boot ROM](https://linux-sunxi.org/BROM#A64)
- [A64 U-Boot](https://linux-sunxi.org/U-Boot)
- [A64 U-Boot SPL](https://linux-sunxi.org/BROM#U-Boot_SPL_limitations)
- [SD Card Layout](https://linux-sunxi.org/Bootable_SD_card#SD_Card_Layout)
Here's the PinePhone U-Boot Log captured with the USB Serial Debug Cable...
(Press Enter repeatedly when PinePhone powers on to enter the U-Boot Prompt)
```text
$ screen /dev/ttyUSB0 115200
DRAM: 2048 MiB
Trying to boot from MMC1
NOTICE: BL31: v2.2(release):v2.2-904-gf9ea3a629
NOTICE: BL31: Built : 15:32:12, Apr 9 2020
NOTICE: BL31: Detected Allwinner A64/H64/R18 SoC (1689)
NOTICE: BL31: Found U-Boot DTB at 0x4064410, model: PinePhone
NOTICE: PSCI: System suspend is unavailable
U-Boot 2020.07 (Nov 08 2020 - 00:15:12 +0100)
DRAM: 2 GiB
MMC: Device 'mmc@1c11000': seq 1 is in use by 'mmc@1c10000'
mmc@1c0f000: 0, mmc@1c10000: 2, mmc@1c11000: 1
Loading Environment from FAT... *** Warning - bad CRC, using default environment
starting USB...
No working controllers found
Hit any key to stop autoboot:
=> help
? - alias for 'help'
base - print or set address offset
bdinfo - print Board Info structure
blkcache - block cache diagnostics and control
boot - boot default, i.e., run 'bootcmd'
bootd - boot default, i.e., run 'bootcmd'
bootelf - Boot from an ELF image in memory
booti - boot Linux kernel 'Image' format from memory
bootm - boot application image from memory
bootvx - Boot vxWorks from an ELF image
cmp - memory compare
coninfo - print console devices and information
cp - memory copy
crc32 - checksum calculation
dm - Driver model low level access
echo - echo args to console
editenv - edit environment variable
env - environment handling commands
exit - exit script
ext2load - load binary file from a Ext2 filesystem
ext2ls - list files in a directory (default /)
ext4load - load binary file from a Ext4 filesystem
ext4ls - list files in a directory (default /)
ext4size - determine a file's size
false - do nothing, unsuccessfully
fatinfo - print information about filesystem
fatload - load binary file from a dos filesystem
fatls - list files in a directory (default /)
fatmkdir - create a directory
fatrm - delete a file
fatsize - determine a file's size
fatwrite - write file into a dos filesystem
fdt - flattened device tree utility commands
fstype - Look up a filesystem type
go - start application at address 'addr'
gpio - query and control gpio pins
gpt - GUID Partition Table
gzwrite - unzip and write memory to block device
help - print command description/usage
iminfo - print header information for application image
imxtract - extract a part of a multi-image
itest - return true/false on integer compare
ln - Create a symbolic link
load - load binary file from a filesystem
loadb - load binary file over serial line (kermit mode)
loads - load S-Record file over serial line
loadx - load binary file over serial line (xmodem mode)
loady - load binary file over serial line (ymodem mode)
loop - infinite loop on address range
ls - list files in a directory (default /)
lzmadec - lzma uncompress a memory region
md - memory display
mm - memory modify (auto-incrementing address)
mmc - MMC sub system
mmcinfo - display MMC info
mw - memory write (fill)
nm - memory modify (constant address)
part - disk partition related commands
poweroff - Perform POWEROFF of the device
printenv - print environment variables
random - fill memory with random pattern
reset - Perform RESET of the CPU
run - run commands in an environment variable
save - save file to a filesystem
saveenv - save environment variables to persistent storage
setenv - set environment variables
setexpr - set environment variable as the result of eval expression
sf - SPI flash sub-system
showvar - print local hushshell variables
size - determine a file's size
sleep - delay execution for some time
source - run script from memory
sysboot - command to get and boot from syslinux files
test - minimal test like /bin/sh
true - do nothing, successfully
unlz4 - lz4 uncompress a memory region
unzip - unzip a memory region
usb - USB sub-system
usbboot - boot from USB device
version - print monitor, compiler and linker version
=> printenv
arch=arm
baudrate=115200
board=sunxi
board_name=sunxi
boot_a_script=load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} ${prefix}${script}; source ${scriptaddr}
boot_extlinux=sysboot ${devtype} ${devnum}:${distro_bootpart} any ${scriptaddr} ${prefix}${boot_syslinux_conf}
boot_net_usb_start=usb start
boot_prefixes=/ /boot/
boot_script_dhcp=boot.scr.uimg
boot_scripts=boot.scr.uimg boot.scr
boot_syslinux_conf=extlinux/extlinux.conf
boot_targets=fel mmc_auto usb0
bootcmd=run distro_bootcmd
bootcmd_fel=if test -n ${fel_booted} && test -n ${fel_scriptaddr}; then echo '(FEL boot)'; source ${fel_scriptaddr}; fi
bootcmd_mmc0=devnum=0; run mmc_boot
bootcmd_mmc1=devnum=1; run mmc_boot
bootcmd_mmc_auto=if test ${mmc_bootdev} -eq 1; then run bootcmd_mmc1; run bootcmd_mmc0; elif test ${mmc_bootdev} -eq 0; then run bootcmd_mmc0; run bootcmd_mmc1; fi
bootcmd_usb0=devnum=0; run usb_boot
bootdelay=0
bootm_size=0xa000000
console=ttyS0,115200
cpu=armv8
dfu_alt_info_ram=kernel ram 0x40080000 0x1000000;fdt ram 0x4FA00000 0x100000;ramdisk ram 0x4FE00000 0x4000000
distro_bootcmd=for target in ${boot_targets}; do run bootcmd_${target}; done
ethaddr=02:ba:8c:73:bf:ca
fdt_addr_r=0x4FA00000
fdtcontroladdr=bbf4dd40
fdtfile=allwinner/sun50i-a64-pinephone.dtb
kernel_addr_r=0x40080000
mmc_boot=if mmc dev ${devnum}; then devtype=mmc; run scan_dev_for_boot_part; fi
mmc_bootdev=0
partitions=name=loader1,start=8k,size=32k,uuid=${uuid_gpt_loader1};name=loader2,size=984k,uuid=${uuid_gpt_loader2};name=esp,size=128M,bootable,uuid=${uuid_gpt_esp};name=system,size=-,uuid=${uuid_gpt_system};
preboot=usb start
pxefile_addr_r=0x4FD00000
ramdisk_addr_r=0x4FE00000
scan_dev_for_boot=echo Scanning ${devtype} ${devnum}:${distro_bootpart}...; for prefix in ${boot_prefixes}; do run scan_dev_for_extlinux; run scan_dev_for_scripts; done;
scan_dev_for_boot_part=part list ${devtype} ${devnum} -bootable devplist; env exists devplist || setenv devplist 1; for distro_bootpart in ${devplist}; do if fstype ${devtype} ${devnum}:${distro_bootpart} bootfstype; then run scan_dev_for_boot; fi; done; setenv devplist
scan_dev_for_extlinux=if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${boot_syslinux_conf}; then echo Found ${prefix}${boot_syslinux_conf}; run boot_extlinux; echo SCRIPT FAILED: continuing...; fi
scan_dev_for_scripts=for script in ${boot_scripts}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${script}; then echo Found U-Boot script ${prefix}${script}; run boot_a_script; echo SCRIPT FAILED: continuing...; fi; done
scriptaddr=0x4FC00000
serial#=92c07dba8c73bfca
soc=sunxi
stderr=serial@1c28000
stdin=serial@1c28000
stdout=serial@1c28000
usb_boot=usb start; if usb dev ${devnum}; then devtype=usb; run scan_dev_for_boot_part; fi
uuid_gpt_esp=c12a7328-f81f-11d2-ba4b-00a0c93ec93b
uuid_gpt_system=b921b045-1df0-41c3-af44-4c6f280d3fae
Environment size: 2861/131068 bytes
=> boot
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
653 bytes read in 3 ms (211.9 KiB/s)
## Executing script at 4fc00000
gpio: pin 114 (gpio 114) value is 1
4275261 bytes read in 192 ms (21.2 MiB/s)
Uncompressed size: 10170376 = 0x9B3008
36162 bytes read in 4 ms (8.6 MiB/s)
1078500 bytes read in 51 ms (20.2 MiB/s)
## Flattened Device Tree blob at 4fa00000
Booting using the fdt blob at 0x4fa00000
Loading Ramdisk to 49ef8000, end 49fff4e4 ... OK
Loading Device Tree to 0000000049eec000, end 0000000049ef7d41 ... OK
Starting kernel ...
/ #
```
According to the U-Boot Log, the Start of RAM `kernel_addr_r` is 0x4008 0000.
We need to set this in the NuttX Linker Script and the NuttX Header...
## NuttX Boots On PinePhone
In the previous section, U-Boot says that the Start of RAM `kernel_addr_r` is 0x4008 0000.
Let's set this in the NuttX Linker Script and the NuttX Header...
- Change Image Load Offset in NuttX Header to 0x0 (from 0x48000)
[(See the changes)](https://github.com/lupyuen/incubator-nuttx/commit/9916b52f9dba17944a35aafd4c21fb9eabb17c0e#diff-a830678a9f1b0773c404196c86ad45d1ef7d7e51a52b935cd08df35f5949aaf8)
- Change NuttX Linker Script to set the Start Address `_start` to 0x4008 0000 (from 0x4028 0000)
[(See the changes)](https://github.com/lupyuen/incubator-nuttx/commit/9916b52f9dba17944a35aafd4c21fb9eabb17c0e#diff-d8d987cb5ba644b5f79987e42663217799e03e384552b2e8dbb041f145fa8ad1)
For PinePhone Allwinner A64 UART: We reused the previous code for transmitting output to UART...
```text
/* PL011 UART transmit character
* xb: register which contains the UART base address
* wt: register which contains the character to transmit
*/
.macro early_uart_transmit xb, wt
strb \wt, [\xb] /* -> UARTDR (Data Register) */
.endm
```
[(Source)](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/qemu/qemu_lowputc.S#L87-L94)
But we updated the UART Register Address for Allwinner A64 UART...
```text
/* 32-bit register definition for qemu pl011 uart */
/* PinePhone Allwinner A64 UART0 Base Address: */
#define UART1_BASE_ADDRESS 0x01C28000
/* Previously: #define UART1_BASE_ADDRESS 0x9000000 */
#define EARLY_UART_PL011_BAUD_RATE 115200
```
[(Source)](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/qemu/qemu_lowputc.S#L40-L45)
Right now we don't check if UART is ready to transmit, so our UART output will have missing characters. This needs to be fixed...
```text
/* PL011 UART wait UART to be ready to transmit
* xb: register which contains the UART base address
* c: scratch register number
*/
.macro early_uart_ready xb, wt
1:
# TODO: Wait for PinePhone Allwinner A64 UART
# ldrh \wt, [\xb, #0x18] /* <- UARTFR (Flag register) */
# tst \wt, #0x8 /* Check BUSY bit */
# b.ne 1b /* Wait for the UART to be ready */
.endm
```
[(Source)](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/qemu/qemu_lowputc.S#L74-L85)
We don't init the UART Port because U-Boot has kindly done it for us. This needs to be fixed...
```text
/* PL011 UART initialization
* xb: register which contains the UART base address
* c: scratch register number
*/
GTEXT(up_earlyserialinit)
SECTION_FUNC(text, up_earlyserialinit)
# TODO: Set PinePhone Allwinner A64 Baud Rate Divisor: UART_LCR (DLAB), UART_DLL, UART_DLH
# ldr x15, =UART1_BASE_ADDRESS
# mov x0, #(7372800 / EARLY_UART_PL011_BAUD_RATE % 16)
# strh w0, [x15, #0x28] /* -> UARTFBRD (Baud divisor fraction) */
# mov x0, #(7372800 / EARLY_UART_PL011_BAUD_RATE / 16)
# strh w0, [x15, #0x24] /* -> UARTIBRD (Baud divisor integer) */
# mov x0, #0x60 /* 8n1 */
# str w0, [x15, #0x2C] /* -> UARTLCR_H (Line control) */
# ldr x0, =0x00000301 /* RXE | TXE | UARTEN */
# str w0, [x15, #0x30] /* -> UARTCR (Control Register) */
ret
```
[(Source)](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/qemu/qemu_lowputc.S#L55-L72)
With the above changes, NuttX boots on PinePhone yay!
[__Watch the Demo on YouTube__](https://youtube.com/shorts/WmRzfCiWV6o?feature=share)
## NuttX Boot Log
This is how we build NuttX for PinePhone...
```bash
## TODO: Install Build Prerequisites
## https://lupyuen.github.io/articles/uboot#install-prerequisites
## Create NuttX Directory
mkdir nuttx
cd nuttx
## Download NuttX OS for PinePhone
git clone \
--recursive \
--branch pinephone \
https://github.com/lupyuen/incubator-nuttx \
nuttx
## Download NuttX Apps for PinePhone
git clone \
--recursive \
--branch pinephone \
https://github.com/lupyuen/incubator-nuttx-apps \
apps
## We'll build NuttX inside nuttx/nuttx
cd nuttx
## Configure NuttX for Single Core
./tools/configure.sh -l qemu-a53:nsh
## Build NuttX
make
## Dump the disassembly to nuttx.S
aarch64-none-elf-objdump \
-t -S --demangle --line-numbers --wide \
nuttx \
>nuttx.S \
2>&1
## Compress the NuttX Binary Image
cp nuttx.bin Image
rm -f Image.gz
gzip Image
## Copy compressed NuttX Binary Image to Jumpdrive microSD.
## How to create Jumpdrive microSD: https://lupyuen.github.io/articles/uboot#pinephone-jumpdrive
## TODO: Change the microSD Path
cp Image.gz "/Volumes/NO NAME"
```
Insert the Jumpdrive microSD into PinePhone and power up.
Here's the UART Log of NuttX booting on PinePhone...
```text
DRAM: 2048 MiB
Trying to boot from MMC1
NOTICE: BL31: v2.2(release):v2.2-904-gf9ea3a629
NOTICE: BL31: Built : 15:32:12, Apr 9 2020
NOTICE: BL31: Detected Allwinner A64/H64/R18 SoC (1689)
NOTICE: BL31: Found U-Boot DTB at 0x4064410, model: PinePhone
NOTICE: PSCI: System suspend is unavailable
U-Boot 2020.07 (Nov 08 2020 - 00:15:12 +0100)
DRAM: 2 GiB
MMC: Device 'mmc@1c11000': seq 1 is in use by 'mmc@1c10000'
mmc@1c0f000: 0, mmc@1c10000: 2, mmc@1c11000: 1
Loading Environment from FAT... *** Warning - bad CRC, using default environment
starting USB...
No working controllers found
Hit any key to stop autoboot: 0
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
653 bytes read in 3 ms (211.9 KiB/s)
## Executing script at 4fc00000
gpio: pin 114 (gpio 114) value is 1
99784 bytes read in 8 ms (11.9 MiB/s)
Uncompressed size: 278528 = 0x44000
36162 bytes read in 4 ms (8.6 MiB/s)
1078500 bytes read in 51 ms (20.2 MiB/s)
## Flattened Device Tree blob at 4fa00000
Booting using the fdt blob at 0x4fa00000
Loading Ramdisk to 49ef8000, end 49fff4e4 ... OK
Loading Device Tree to 0000000049eec000, end 0000000049ef7d41 ... OK
Starting kernel ...
HELLO NUTTX ON PINEPHONE!
- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
nx_start: Entry
up_allocate_heap: heap_start=0x0x400c4000, heap_size=0x7f3c000
arm64_gic_initialize: TODO: Init GIC for PinePhone
arm64_gic_initialize: CONFIG_GICD_BASE=0x1c81000
arm64_gic_initialize: CONFIG_GICR_BASE=0x1c82000
arm64_gic_initialize: GIC Version is 2
up_timer_initialize: up_timer_initialize: cp15 timer(s) running at 24.00MHz, cycle 24000
up_timer_initialize: _vector_table=0x400a7000
up_timer_initialize: Before writing: vbar_el1=0x40227000
up_timer_initialize: After writing: vbar_el1=0x400a7000
uart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
work_start_highpri: Starting high-priority kernel worker thread(s)
nx_start_application: Starting init thread
lib_cxx_initialize: _sinit: 0x400a7000 _einit: 0x400a7000 _stext: 0x40080000 _etext: 0x400a8000
nsh: sysinit: fopen failed: 2
nshn:x _msktfaarttf:s :C PcUo0m:m aBnedg innonti nfgo uInddle L oNouptt
Shell (NSH) NuttX-10.3.0-RC2
nsh> uname -a
NuttX 10.3.0-RC2 fc909c6-dirty Sep 1 2022 17:05:44 arm64 qemu-a53
nsh> help
help usage: help [-v] []
. cd dmesg help mount rmdir true xd
[ cp echo hexdump mv set truncate
? cmp exec kill printf sleep uname
basename dirname exit ls ps source umount
break dd false mkdir pwd test unset
cat df free mkrd rm time usleep
Builtin Apps:
getprime hello nsh ostest sh
nsh> hello
task_spawn: name=hello entry=0x4009b1a0 file_actions=0x400c9580 attr=0x400c9588 argv=0x400c96d0
spawn_execattrs: Setting policy=2 priority=100 for pid=3
Hello, World!!
nsh> ls /dev
/dev:
console
null
ram0
ram2
ttyS0
zero
```
[__Watch the Demo on YouTube__](https://youtube.com/shorts/WmRzfCiWV6o?feature=share)
The output is slightly garbled, the UART Driver needs fixing.
# Interrupt Controller
Read the article...
- ["NuttX RTOS for PinePhone: Fixing the Interrupts"](https://lupyuen.github.io/articles/interrupt)
Let's talk about the __Arm Generic Interrupt Controller (GIC)__ for PinePhone...
```text
arm64_gic_initialize: TODO: Init GIC for PinePhone
arm64_gic_initialize: CONFIG_GICD_BASE=0x1c81000
arm64_gic_initialize: CONFIG_GICR_BASE=0x1c82000
arm64_gic_initialize: GIC Version is 2
```
This is the current implementation of [Arm GIC Version 3](https://developer.arm.com/documentation/ihi0069/latest) in NuttX Arm64...
- [arch/arm64/src/common/arm64_gicv3.c](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_gicv3.c)
- [arch/arm64/src/common/arm64_gic.h](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_gic.h)
This implementation won't work on PinePhone, so we have commented out the existing code and inserted our own implementation.
_Why won't Arm GIC Version 3 work on PinePhone?_
According to the Allwinner A64 SoC User Manual (page 210, "GIC"), PinePhone's Interrupt Controller runs on...
- [Arm GIC PL400](https://developer.arm.com/documentation/ddi0471/b/introduction/about-the-gic-400), which is based on...
- [Arm GIC Version 2](https://developer.arm.com/documentation/ihi0048/latest/)
We'll have to downgrade [arm64_gicv3.c](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_gicv3.c) to support Arm GIC Version 2 for PinePhone.
_Does NuttX implement Arm GIC Version 2?_
NuttX has an implementation of Arm GIC Version 2, but it's based on Arm32. We'll port it from Arm32 to Arm64...
- [arch/arm/src/armv7-a/arm_gicv2.c](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm/src/armv7-a/arm_gicv2.c)
- [arch/arm/src/armv7-a/arm_gicv2_dump.c](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm/src/armv7-a/arm_gicv2_dump.c)
- [arch/arm/src/armv7-a/gic.h](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm/src/armv7-a/gic.h)
- [arch/arm/src/armv7-a/mpcore.h](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm/src/armv7-a/mpcore.h)
- [arch/arm/src/imx6/chip.h](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm/src/imx6/chip.h)
- [arch/arm/src/imx6/hardware/imx_memorymap.h](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm/src/imx6/hardware/imx_memorymap.h)
By reusing the code above, we have implemented Arm GIC Version 2 for PinePhone...
- [arch/arm64/src/common/arm64_gicv3.c](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_gicv3.c#L765-L823)
We made minor tweaks to NuttX's implementation of GIC Version 2...
- [Changes for arch/arm/src/armv7-a/arm_gicv2.c](https://github.com/lupyuen/incubator-nuttx/commit/6fa0e7e5d2beddad07890c83d2ee428a3f2b8a62#diff-6e1132aef124dabaf94c200ab06d65c7bc2b9967bf76a46aba71a7f43b5fb219)
- [Changes for arch/arm/src/armv7-a/arm_gicv2_dump.c](https://github.com/lupyuen/incubator-nuttx/commit/4fc2669fef62d12ba1dd428f2daf03d3bc362501#diff-eb05c977988d59202a9472f6fa7f9dc290724662ad6d15a4ba99b8f1fc1dc8f8)
- [Changes for arch/arm/src/armv7-a/gic.h](https://github.com/lupyuen/incubator-nuttx/commit/6fa0e7e5d2beddad07890c83d2ee428a3f2b8a62#diff-b4fcb67b71de954c942ead9bb0868e720a5802c90743f0a1883f84b7565e1a0f)
_Where in memory is the GIC located?_
According to the Allwinner A64 SoC User Manual (page 74, "Memory Mapping"), the GIC is located at this address...
| Module | Address (It is for Cluster CPU) | Remarks
| :----- | :------ | :------
|SCU space | 0x01C80000| (What's this?)
| | GIC_DIST: 0x01C80000 + 0x1000| GIC Distributor (GICD)
|CPUS can’t access | GIC_CPUIF:0x01C80000 + 0x2000| GIC CPU Interface (GICC)
(Why "CPUS can’t access"?)
The __Interrupt Sources__ are defined in the Allwinner A64 SoC User Manual (page 210, "GIC")...
- 16 x Software-Generated Interrupts (SGI)
"This is an interrupt generated by software writing to a GICD_SGIR register in the GIC. The system uses SGIs for interprocessor communication."
- 16 x Private Peripheral Interrupts (PPI)
"This is a peripheral interrupt that is specific to a single processor"
- 125 x Shared Peripheral Interrupts (SPI)
"This is a peripheral interrupt that the Distributor can route to any of a specified combination of processors"
To verify the GIC Version, read the __Peripheral ID2 Register (ICPIDR2)__ at Offset 0xFE8 of GIC Distributor.
Bits 4 to 7 of ICPIDR2 are...
- 0x1 for GIC Version 1
- 0x2 for GIC Version 2
This is how we implement the GIC Version verification: [arch/arm64/src/common/arm64_gicv3.c](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_gicv3.c#L710-L734)
```c
// Init GIC v2 for PinePhone. See https://github.com/lupyuen/pinephone-nuttx#interrupt-controller
int arm64_gic_initialize(void)
{
sinfo("TODO: Init GIC for PinePhone\n");
// To verify the GIC Version, read the Peripheral ID2 Register (ICPIDR2) at Offset 0xFE8 of GIC Distributor.
// Bits 4 to 7 of ICPIDR2 are...
// - 0x1 for GIC Version 1
// - 0x2 for GIC Version 2
// GIC Distributor is at 0x01C80000 + 0x1000.
// See https://github.com/lupyuen/pinephone-nuttx#interrupt-controller
const uint8_t *ICPIDR2 = (const uint8_t *) (CONFIG_GICD_BASE + 0xFE8);
uint8_t version = (*ICPIDR2 >> 4) & 0b1111;
sinfo("GIC Version is %d\n", version);
DEBUGASSERT(version == 2);
// arm_gic0_initialize must be called on CPU0
arm_gic0_initialize();
// arm_gic_initialize must be called for all CPUs
// TODO: Move to arm64_gic_secondary_init
arm_gic_initialize();
return 0;
}
```
See below for the [GIC Register Dump](https://github.com/lupyuen/pinephone-nuttx#gic-register-dump).
## Multi Core SMP
Right now NuttX is configured to run on a Single Core for PinePhone.
To support all 4 Cores on PinePhone with SMP (Symmetric Multi-Processing), we need to fix these in the Generic Interrupt Controller Version 2...
https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_gicv3.c#L717-L743
```c
// Init GIC v2 for PinePhone. See https://github.com/lupyuen/pinephone-nuttx#interrupt-controller
int arm64_gic_initialize(void)
{
...
// arm_gic0_initialize must be called on CPU0
arm_gic0_initialize();
// arm_gic_initialize must be called for all CPUs
// TODO: Move to arm64_gic_secondary_init
arm_gic_initialize();
```
https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_gicv3.c#L746-L761
```c
#ifdef CONFIG_SMP
...
// TODO: Init GIC for PinePhone
void arm64_gic_secondary_init(void)
{
sinfo("TODO: Init GIC Secondary for PinePhone\n");
// TODO: arm_gic_initialize must be called for all CPUs.
arm_gic_initialize();
}
#endif // NOTUSED
```
And we rebuild NuttX for Multi Core...
```bash
## Erase the NuttX Configuration
make distclean
## Configure NuttX for 4 Cores
./tools/configure.sh -l qemu-a53:nsh_smp
## Build NuttX
make
## Dump the disassembly to nuttx.S
aarch64-none-elf-objdump \
-t -S --demangle --line-numbers --wide \
nuttx \
>nuttx.S \
2>&1
```
## System Timer
NuttX starts the System Timer when it boots. Here's how the System Timer is started: [arch/arm64/src/common/arm64_arch_timer.c](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_arch_timer.c#L212-L233)
```c
void up_timer_initialize(void)
{
uint64_t curr_cycle;
arch_timer_rate = arm64_arch_timer_get_cntfrq();
cycle_per_tick = ((uint64_t)arch_timer_rate / (uint64_t)TICK_PER_SEC);
sinfo("%s: cp15 timer(s) running at %lu.%02luMHz, cycle %ld\n", __func__,
(unsigned long)arch_timer_rate / 1000000,
(unsigned long)(arch_timer_rate / 10000) % 100, cycle_per_tick);
irq_attach(ARM_ARCH_TIMER_IRQ, arm64_arch_timer_compare_isr, 0);
arm64_gic_irq_set_priority(ARM_ARCH_TIMER_IRQ, ARM_ARCH_TIMER_PRIO,
ARM_ARCH_TIMER_FLAGS);
curr_cycle = arm64_arch_timer_count();
arm64_arch_timer_set_compare(curr_cycle + cycle_per_tick);
arm64_arch_timer_enable(true);
up_enable_irq(ARM_ARCH_TIMER_IRQ);
arm64_arch_timer_set_irq_mask(false);
}
```
At every tick, the System Timer triggers an interrupt that calls [`arm64_arch_timer_compare_isr`](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_arch_timer.c#L109-L169)
(`CONFIG_SCHED_TICKLESS` is undefined)
__Timer IRQ `ARM_ARCH_TIMER_IRQ`__ is defined in [arch/arm64/src/common/arm64_arch_timer.h](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_arch_timer.h#L38-L45)
```c
#define CONFIG_ARM_TIMER_SECURE_IRQ (GIC_PPI_INT_BASE + 13)
#define CONFIG_ARM_TIMER_NON_SECURE_IRQ (GIC_PPI_INT_BASE + 14)
#define CONFIG_ARM_TIMER_VIRTUAL_IRQ (GIC_PPI_INT_BASE + 11)
#define CONFIG_ARM_TIMER_HYP_IRQ (GIC_PPI_INT_BASE + 10)
#define ARM_ARCH_TIMER_IRQ CONFIG_ARM_TIMER_VIRTUAL_IRQ
#define ARM_ARCH_TIMER_PRIO IRQ_DEFAULT_PRIORITY
#define ARM_ARCH_TIMER_FLAGS IRQ_TYPE_LEVEL
```
`GIC_PPI_INT_BASE` is defined in [arch/arm64/src/common/arm64_gic.h](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_gic.h#L120-L128)
```c
#define GIC_SGI_INT_BASE 0
#define GIC_PPI_INT_BASE 16
#define GIC_IS_SGI(intid) (((intid) >= GIC_SGI_INT_BASE) && \
((intid) < GIC_PPI_INT_BASE))
#define GIC_SPI_INT_BASE 32
#define GIC_NUM_INTR_PER_REG 32
#define GIC_NUM_CFG_PER_REG 16
#define GIC_NUM_PRI_PER_REG 4
```
## Timer Interrupt Isn't Handled
Previously NuttX hangs midsentence while booting on PinePhone, let's find out how we fixed it...
```text
arm64_gic_initialize: TODO: Init GIC for PinePhone
arm64_gic_initialize: CONFIG_GICD_BASE=0x1c81000
arm64_gic_initialize: CONFIG_GICR_BASE=0x1c82000
arm64_gic_initialize: GIC Version is 2
up_timer_initialize: up_timer_initialize: cp15 timer(s) running at 24.00MHz, cycle 24000
uart_regi
```
Based on our experiments, it seems the [System Timer](https://github.com/lupyuen/pinephone-nuttx#system-timer) triggered a Timer Interrupt, and NuttX hangs while attempting to handle the Timer Interrupt.
The Timer Interrupt Handler [`arm64_arch_timer_compare_isr`](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_arch_timer.c#L109-L169) is never called. (We checked using [`up_putc`](https://github.com/lupyuen/pinephone-nuttx#boot-debugging))
_Is it caused by PinePhone's GIC?_
This problem doesn't seem to be caused by [PinePhone's Generic Interrupt Controller (GIC)](https://github.com/lupyuen/pinephone-nuttx#interrupt-controller) that we have implemented. We [successfully tested PinePhone's GIC](https://github.com/lupyuen/pinephone-nuttx#test-pinephone-gic-with-qemu) with QEMU.
Let's troubleshoot the Timer Interrupt...
- We called [`up_putc`](https://github.com/lupyuen/pinephone-nuttx#boot-debugging) to understand [how Interrupts are handled on NuttX](https://github.com/lupyuen/pinephone-nuttx#handling-interrupts).
We also added Debug Code to the [Arm64 Interrupt Handler](https://github.com/lupyuen/pinephone-nuttx#interrupt-debugging).
[(Maybe we should have used GDB with QEMU)](https://github.com/apache/incubator-nuttx/tree/master/boards/arm64/qemu/qemu-a53)
- We [dumped the Interrupt Vector Table](https://github.com/lupyuen/pinephone-nuttx#dump-interrupt-vector-table).
We verified that the Timer Interrupt Handler Address in the table is correct.
- We confirmed that [Interrupt Dispatcher `irq_dispatch`](https://github.com/lupyuen/pinephone-nuttx#handling-interrupts) isn't called.
And [Unexpected Interrupt Handler `irq_unexpected_isr`](https://github.com/lupyuen/pinephone-nuttx#handling-interrupts) isn't called either.
- Let's backtrack, maybe there's a problem in the Arm64 Interrupt Handler?
But [`arm64_enter_exception`](https://github.com/lupyuen/pinephone-nuttx#handling-interrupts) and [`arm64_irq_handler`](https://github.com/lupyuen/pinephone-nuttx#handling-interrupts) aren't called either.
- Maybe the __Arm64 Vector Table `_vector_table`__ isn't correctly configured?
[arch/arm64/src/common/arm64_vector_table.S](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_vector_table.S#L93-L232)
And we're right! The Arm64 Vector Table is indeed incorrectly configured! Here why...
## Arm64 Vector Table Is Wrong
Earlier we saw that the Interrupt Handler wasn't called for System Timer Interrupt. And it might be due to problems in the __Arm64 Vector Table `_vector_table`__: [arch/arm64/src/common/arm64_vector_table.S](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_vector_table.S#L93-L232)
Let's check whether the Arm64 Vector Table `_vector_table` is correctly configured in the Arm CPU: [arch/arm64/src/common/arm64_arch_timer.c](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_arch_timer.c#L212-L235)
```c
void up_timer_initialize(void)
{
...
// Attach System Timer Interrupt Handler
irq_attach(ARM_ARCH_TIMER_IRQ, arm64_arch_timer_compare_isr, 0);
// For PinePhone: Read Vector Base Address Register EL1
extern void *_vector_table[];
sinfo("_vector_table=%p\n", _vector_table);
sinfo("Before writing: vbar_el1=%p\n", read_sysreg(vbar_el1));
```
After attaching the Interrupt Handler for System Timer, we read the Arm64 [Vector Base Address Register EL1](https://github.com/lupyuen/pinephone-nuttx#handling-interrupts). Here's the output...
```text
up_timer_initialize: up_timer_initialize: cp15 timer(s) running at 24.00MHz, cycle 24000
up_timer_initialize: _vector_table=0x400a7000
up_timer_initialize: Before writing: vbar_el1=0x40227000
```
Aha! `_vector_table` is at 0x400a7000... But Vector Base Address Register EL1 says 0x40227000!
Our Arm64 CPU is pointing to the wrong Arm64 Vector Table... Hence our Interrupt Handler is never called!
Let's fix the Vector Base Address Register EL1: [arch/arm64/src/common/arm64_arch_timer.c](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_arch_timer.c#L212-L235)
```c
// For PinePhone: Write Vector Base Address Register EL1
write_sysreg((uint64_t)_vector_table, vbar_el1);
ARM64_ISB();
// For PinePhone: Read Vector Base Address Register EL1
sinfo("After writing: vbar_el1=%p\n", read_sysreg(vbar_el1));
```
This writes the correct value of `_vector_table` back into Vector Base Address Register EL1. Here's the output...
```text
up_timer_initialize: up_timer_initialize: cp15 timer(s) running at 24.00MHz, cycle 24000
up_timer_initialize: _vector_table=0x400a7000
up_timer_initialize: Before writing: vbar_el1=0x40227000
up_timer_initialize: After writing: vbar_el1=0x400a7000
```
Yep Vector Base Address Register EL1 is now correct.
And our Interrupt Handlers are now working fine yay!
## Test PinePhone GIC with QEMU
This is how we build NuttX for QEMU with [Generic Interrupt Controller (GIC) Version 2](https://github.com/lupyuen/pinephone-nuttx#interrupt-controller)...
```bash
## TODO: Install Build Prerequisites
## https://lupyuen.github.io/articles/uboot#install-prerequisites
## Create NuttX Directory
mkdir nuttx
cd nuttx
## Download NuttX OS for QEMU with GIC Version 2
git clone \
--recursive \
--branch gicv2 \
https://github.com/lupyuen/incubator-nuttx \
nuttx
## Download NuttX Apps for QEMU
git clone \
--recursive \
--branch arm64 \
https://github.com/lupyuen/incubator-nuttx-apps \
apps
## We'll build NuttX inside nuttx/nuttx
cd nuttx
## Configure NuttX for Single Core
./tools/configure.sh -l qemu-a53:nsh
## Build NuttX
make
## Dump the disassembly to nuttx.S
aarch64-none-elf-objdump \
-t -S --demangle --line-numbers --wide \
nuttx \
>nuttx.S \
2>&1
```
And this is how we tested PinePhone's GIC Version 2 with QEMU...
```bash
## Run GIC v2 with QEMU
qemu-system-aarch64 \
-smp 4 \
-cpu cortex-a53 \
-nographic \
-machine virt,virtualization=on,gic-version=2 \
-net none \
-chardev stdio,id=con,mux=on \
-serial chardev:con \
-mon chardev=con,mode=readline \
-kernel ./nuttx
```
Note that `gic-version=2`, instead of the usual GIC Version 3 for NuttX Arm64.
Also we simulated 4 Cores of Arm Cortex-A53 (similar to PinePhone): `-smp 4`
QEMU boots OK with PinePhone's GIC Version 2...
```text
- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
nx_start: Entry
up_allocate_heap: heap_start=0x0x402c4000, heap_size=0x7d3c000
arm64_gic_initialize: TODO: Init GIC for PinePhone
arm64_gic_initialize: CONFIG_GICD_BASE=0x8000000
arm64_gic_initialize: CONFIG_GICR_BASE=0x8010000
arm64_gic_initialize: GIC Version is 2
EFGHup_timer_initialize: up_timer_initialize: cp15 timer(s) running at 62.50MHz, cycle 62500
AKLMNOPBIJuart_register: Registering /dev/console
uart_register: Registering /dev/ttyS0
AKLMNOPBIJwork_start_highpri: Starting high-priority kernel worker thread(s)
nx_start_application: Starting init thread
lib_cxx_initialize: _sinit: 0x402a7000 _einit: 0x402a7000 _stext: 0x40280000 _etext: 0x402a8000
nsh: sysinit: fopen failed: 2
nsh: mkfatfs: command not found
NuttShell (NSH) NuttX-10.3.0-RC2
nsh> nx_start: CPU0: Beginning Idle Loop
```
So our implementation of GIC Version 2 for PinePhone is probably OK.
_Is the Timer Interrupt triggered correctly with PinePhone GIC?_
Yes, we verified that the Timer Interrupt Handler [`arm64_arch_timer_compare_isr`](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_arch_timer.c#L109-L169) is called periodically. (We checked using [`up_putc`](https://github.com/lupyuen/pinephone-nuttx#boot-debugging))
_How did we get the GIC Base Addresses?_
```text
arm64_gic_initialize: CONFIG_GICD_BASE=0x8000000
arm64_gic_initialize: CONFIG_GICR_BASE=0x8010000
```
We got the GIC v2 Base Addresses for GIC Distributor (`CONFIG_GICD_BASE`) and GIC CPU Interface (`CONFIG_GICR_BASE`) by dumping the Device Tree from QEMU...
```bash
## GIC v2 Dump Device Tree
qemu-system-aarch64 \
-smp 4 \
-cpu cortex-a53 \
-nographic \
-machine virt,virtualization=on,gic-version=2,dumpdtb=gicv2.dtb \
-net none \
-chardev stdio,id=con,mux=on \
-serial chardev:con \
-mon chardev=con,mode=readline \
-kernel ./nuttx
## Convert Device Tree to text format
dtc -o gicv2.dts -O dts -I dtb gicv2.dtb
```
The Base Addresses are revealed in the GIC v2 Device Tree: [gicv2.dts](https://github.com/lupyuen/incubator-nuttx/blob/gicv2/gicv2.dts#L324)...
```text
intc@8000000 {
reg = <
0x00 0x8000000 0x00 0x10000 // GIC Distributor: 0x8000000
0x00 0x8010000 0x00 0x10000 // GIC CPU Interface: 0x8010000
0x00 0x8030000 0x00 0x10000 // VGIC Virtual Interface Control: 0x8030000
0x00 0x8040000 0x00 0x10000 // VGIC Virtual CPU Interface: 0x8040000
>;
compatible = "arm,cortex-a15-gic";
```
[(More about this)](https://www.kernel.org/doc/Documentation/devicetree/bindings/interrupt-controller/arm%2Cgic.txt)
We defined the Base Addresses in [arch/arm64/include/qemu/chip.h](https://github.com/lupyuen/incubator-nuttx/blob/gicv2/arch/arm64/include/qemu/chip.h#L38-L40)
Compare the above Base Addresses with the GIC v3 Device Tree: [gicv3.dts](https://github.com/lupyuen/incubator-nuttx/blob/gicv2/gicv3.dts#L324)
```text
intc@8000000 {
reg = <
0x00 0x8000000 0x00 0x10000 // GIC Distributor: 0x8000000
0x00 0x80a0000 0x00 0xf60000 // GIC CPU Interface: 0x80a0000
>;
#redistributor-regions = <0x01>;
compatible = "arm,gic-v3";
```
This is how we copied the PinePhone GIC v2 Source Files into NuttX QEMU Arm64 for testing...
```bash
cp ~/PinePhone/nuttx/nuttx/arch/arm64/src/common/arm64_gicv3.c ~/gicv2/nuttx/nuttx/arch/arm64/src/common/arm64_gicv3.c
cp ~/PinePhone/nuttx/nuttx/arch/arm/src/armv7-a/arm_gicv2.c ~/gicv2/nuttx/nuttx/arch/arm/src/armv7-a/arm_gicv2.c
cp ~/PinePhone/nuttx/nuttx/arch/arm/src/armv7-a/gic.h ~/gicv2/nuttx/nuttx/arch/arm/src/armv7-a/gic.h
cp ~/PinePhone/nuttx/nuttx/arch/arm/src/armv7-a/arm_gicv2_dump.c ~/gicv2/nuttx/nuttx/arch/arm/src/armv7-a/arm_gicv2_dump.c
cp ~/PinePhone/nuttx/nuttx/arch/arm64/src/common/arm64_arch_timer.c ~/gicv2/nuttx/nuttx/arch/arm64/src/common/arm64_arch_timer.c
```
## Handling Interrupts
Let's talk about NuttX and how it handles interrupts.
The __Interrupt Vector Table__ is defined in [sched/irq/irq_initialize.c](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/sched/irq/irq_initialize.c#L47-L53)
```c
/* This is the interrupt vector table */
struct irq_info_s g_irqvector[NR_IRQS];
```
(Next section talks about dumping the Interrupt Vector Table)
At startup, the Interrupt Vector Table is initialised to the __Unexpected Interrupt Handler `irq_unexpected_isr`__: [sched/irq/irq_initialize.c](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/sched/irq/irq_initialize.c#L59-L85)
```c
/****************************************************************************
* Name: irq_initialize
* Description:
* Configure the IRQ subsystem
****************************************************************************/
void irq_initialize(void)
{
/* Point all interrupt vectors to the unexpected interrupt */
for (i = 0; i < NR_IRQS; i++)
{
g_irqvector[i].handler = irq_unexpected_isr;
}
up_irqinitialize();
}
```
__Unexpected Interrupt Handler `irq_unexpected_isr`__ is called when an Interrupt is triggered and there's no Interrupt Handler attached to the Interrupt: [sched/irq/irq_unexpectedisr.c](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/sched/irq/irq_unexpectedisr.c#L38-L59)
```c
/****************************************************************************
* Name: irq_unexpected_isr
* Description:
* An interrupt has been received for an IRQ that was never registered
* with the system.
****************************************************************************/
int irq_unexpected_isr(int irq, FAR void *context, FAR void *arg)
{
up_irq_save();
_err("ERROR irq: %d\n", irq);
PANIC();
```
To __attach an Interrupt Handler__, we set the Handler and the Argument in the Interrupt Vector Table: [sched/irq/irq_attach.c](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/sched/irq/irq_attach.c#L37-L136)
```c
/****************************************************************************
* Name: irq_attach
* Description:
* Configure the IRQ subsystem so that IRQ number 'irq' is dispatched to
* 'isr'
****************************************************************************/
int irq_attach(int irq, xcpt_t isr, FAR void *arg)
{
...
/* Save the new ISR and its argument in the table. */
g_irqvector[irq].handler = isr;
g_irqvector[irq].arg = arg;
```
When an __Interrupt is triggered__...
1. Arm CPU looks up the __Arm64 Vector Table `_vector_table`__: [arch/arm64/src/common/arm64_vector_table.S](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_vector_table.S#L93-L232)
```text
/* Four types of exceptions:
* - synchronous: aborts from MMU, SP/CP alignment checking, unallocated
* instructions, SVCs/SMCs/HVCs, ...)
* - IRQ: group 1 (normal) interrupts
* - FIQ: group 0 or secure interrupts
* - SError: fatal system errors
*
* Four different contexts:
* - from same exception level, when using the SP_EL0 stack pointer
* - from same exception level, when using the SP_ELx stack pointer
* - from lower exception level, when this is AArch64
* - from lower exception level, when this is AArch32
*
* +------------------+------------------+-------------------------+
* | Address | Exception type | Description |
* +------------------+------------------+-------------------------+
* | VBAR_ELn + 0x000 | Synchronous | Current EL with SP0 |
* | + 0x080 | IRQ / vIRQ | |
* | + 0x100 | FIQ / vFIQ | |
* | + 0x180 | SError / vSError | |
* +------------------+------------------+-------------------------+
* | + 0x200 | Synchronous | Current EL with SPx |
* | + 0x280 | IRQ / vIRQ | |
* | + 0x300 | FIQ / vFIQ | |
* | + 0x380 | SError / vSError | |
* +------------------+------------------+-------------------------+
* | + 0x400 | Synchronous | Lower EL using AArch64 |
* | + 0x480 | IRQ / vIRQ | |
* | + 0x500 | FIQ / vFIQ | |
* | + 0x580 | SError / vSError | |
* +------------------+------------------+-------------------------+
* | + 0x600 | Synchronous | Lower EL using AArch64 |
* | + 0x680 | IRQ / vIRQ | |
* | + 0x700 | FIQ / vFIQ | |
* | + 0x780 | SError / vSError | |
* +------------------+------------------+-------------------------+
*/
GTEXT(_vector_table)
SECTION_SUBSEC_FUNC(exc_vector_table,_vector_table_section,_vector_table)
...
/* Current EL with SP0 / IRQ */
.align 7
arm64_enter_exception x0, x1
b arm64_irq_handler
...
/* Current EL with SPx / IRQ */
.align 7
arm64_enter_exception x0, x1
b arm64_irq_handler
```
[(`arm64_enter_exception` is defined here)](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_vector_table.S#L41-L87)
1. Based on the Arm64 Vector Table `_vector_table`, Arm CPU jumps to `arm64_irq_handler`: [arch/arm64/src/common/arm64_vectors.S](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_vectors.S#L326-L413)
```text
/****************************************************************************
* Name: arm64_irq_handler
* Description:
* Interrupt exception handler
****************************************************************************/
GTEXT(arm64_irq_handler)
SECTION_FUNC(text, arm64_irq_handler)
...
/* Call arm64_decodeirq() on the interrupt stack
* with interrupts disabled
*/
bl arm64_decodeirq
```
1. `arm64_irq_handler` calls `arm64_decodeirq` to decode the Interrupt: [arch/arm64/src/common/arm64_gicv3.c](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_gicv3.c#L800-L829)
```c
/***************************************************************************
* Name: arm64_decodeirq
* Description:
* This function is called from the IRQ vector handler in arm64_vectors.S.
* At this point, the interrupt has been taken and the registers have
* been saved on the stack. This function simply needs to determine the
* the irq number of the interrupt and then to call arm_doirq to dispatch
* the interrupt.
* Input Parameters:
* regs - A pointer to the register save area on the stack.
***************************************************************************/
// Decode IRQ for PinePhone, based on arm_decodeirq in arm_gicv2.c
uint64_t * arm64_decodeirq(uint64_t * regs)
{
...
if (irq < NR_IRQS)
{
/* Dispatch the interrupt */
regs = arm64_doirq(irq, regs);
```
1. `arm64_decodeirq` calls `arm64_doirq` to dispatch the Interrupt: [arch/arm64/src/common/arm64_doirq.c](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_doirq.c#L64-L119)
```c
/****************************************************************************
* Name: arm64_doirq
* Description:
* Receives the decoded GIC interrupt information and dispatches control
* to the attached interrupt handler.
*
****************************************************************************/
uint64_t *arm64_doirq(int irq, uint64_t * regs)
{
...
/* Deliver the IRQ */
irq_dispatch(irq, regs);
```
1. `irq_dispatch` calls the Interrupt Handler fetched from the Interrupt Vector Table: [sched/irq/irq_dispatch.c](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/sched/irq/irq_dispatch.c#L115-L173)
```c
/****************************************************************************
* Name: irq_dispatch
* Description:
* This function must be called from the architecture-specific logic in
* order to dispatch an interrupt to the appropriate, registered handling
* logic.
****************************************************************************/
void irq_dispatch(int irq, FAR void *context)
{
if ((unsigned)irq < NR_IRQS)
{
if (g_irqvector[ndx].handler)
{
vector = g_irqvector[ndx].handler;
arg = g_irqvector[ndx].arg;
}
}
/* Then dispatch to the interrupt handler */
CALL_VECTOR(ndx, vector, irq, context, arg);
```
_How is the [Arm64 Vector Table `_vector_table`](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_vector_table.S#L93-L232) configured in the Arm CPU?_
The [Arm64 Vector Table `_vector_table`](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_vector_table.S#L93-L232) is configured in the Arm CPU during EL1 Init by `arm64_boot_el1_init`: [arch/arm64/src/common/arm64_boot.c](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_boot.c#L132-L162)
```c
void arm64_boot_el1_init(void)
{
/* Setup vector table */
write_sysreg((uint64_t)_vector_table, vbar_el1);
ARM64_ISB();
```
`vbar_el1` refers to __Vector Base Address Register EL1__.
[(See Arm Cortex-A53 Technical Reference Manual, page 4-121, "Vector Base Address Register, EL1")](https://documentation-service.arm.com/static/5e9075f9c8052b1608761519?token=)
[(Arm64 Vector Table is also configured during EL3 Init by `arm64_boot_el3_init`)](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_boot.c#L39-L75)
EL1 Init `arm64_boot_el1_init` is called by our Startup Code: [arch/arm64/src/common/arm64_head.S](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_head.S#L216-L230)
```text
PRINT(switch_el1, "- Boot from EL1\r\n")
/* EL1 init */
bl arm64_boot_el1_init
/* set SP_ELx and Enable SError interrupts */
msr SPSel, #1
msr DAIFClr, #(DAIFCLR_ABT_BIT)
isb
jump_to_c_entry:
PRINT(jump_to_c_entry, "- Boot to C runtime for OS Initialize\r\n")
ret x25
```
_What are EL1 and EL3?_
According to [Arm Cortex-A53 Technical Reference Manual](https://documentation-service.arm.com/static/5e9075f9c8052b1608761519?token=) page 3-5 ("Exception Level")...
> The ARMv8 exception model defines exception levels EL0-EL3, where:
> - EL0 has the lowest software execution privilege, and execution at EL0 is called unprivileged execution.
> - Increased exception levels, from 1 to 3, indicate increased software execution privilege.
> - EL2 provides support for processor virtualization.
> - EL3 provides support for a secure state, see Security state on page 3-6.
PinePhone only uses EL1 and EL2 (but not EL3)...
```text
HELLO NUTTX ON PINEPHONE!
- Ready to Boot CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
```
From this we see that NuttX runs mostly in EL1.
(EL1 is less privileged than EL2, which supports Processor Virtualization)
## Dump Interrupt Vector Table
This is how we dump the Interrupt Vector Table to troubleshoot Interrupts...
Based on [arch/arm64/src/common/arm64_arch_timer.c](https://github.com/lupyuen/incubator-nuttx/blob/pinephone/arch/arm64/src/common/arm64_arch_timer.c#L210-L240)
```c
#include "irq/irq.h" // For dumping Interrupt Vector Table
void up_timer_initialize(void)
{
...
// Attach System Timer Interrupt Handler
irq_attach(ARM_ARCH_TIMER_IRQ, arm64_arch_timer_compare_isr, 0);
// Begin dumping Interrupt Vector Table
sinfo("ARM_ARCH_TIMER_IRQ=%d\n", ARM_ARCH_TIMER_IRQ);
sinfo("arm64_arch_timer_compare_isr=%p\n", arm64_arch_timer_compare_isr);
sinfo("irq_unexpected_isr=%p\n", irq_unexpected_isr);
for (int i = 0; i < NR_IRQS; i++)
{
sinfo("g_irqvector[%d].handler=%p\n", i, g_irqvector[i].handler);
}
// End dumping Interrupt Vector Table