Testing embedded systems is difficult. Unit tests will catch bugs in one module, but can't catch bugs that span multiple modules. Integration tests validate that code works across multiple modules, but these tests run on real hardware, making them slow. Additionally, integration testing can't test error paths well. How do you, on real hardware, mock out your temperature sensor to ensure that the temperature sensor failing to report a value triggers the correct response?
With these issues in mind, emulation is an attractive way to solve these problems. Emulation allows you to have an infinite amount of virtual hardware, allowing your integration tests to run in parallel. Emulation also lets you control the response of your hardware, so you can indeed have your emulated temperature sensor return an error value
Given the potential upsides of emulation, I wanted to do some experimentation to see the pros and cons of testing this way. Because I wanted a simple system to test on - and because I already had one lying around - the system I decided to emulate was a Raspberry Pi 4. The Pi would run a server that exposed a REST API to store and retrieve data from an EEPROM [1]. The build process for the app would then spin up a virtual Pi using QEMU, and would run the integration tests against that. This would ensure that changes I made to the code didn't break anything, which mimics how this type of testing is done in a real CI pipeline
At the time, I assumed that getting the emulated Pi setup would be the easy part. I imagined most of my time would be spent writing the app, the integration tests, and the accompanying blog post
I was wrong. Very wrong. So wrong that this entire article exists just to document the pain I went through trying to get my virtual Pi to talk to a virtual EEPROM
The goal of writing this is for you, dear reader, to come away with a better understanding of how QEMU works, a good understanding of how to emulate a Pi using QEMU, and, hopefully, more humility for future projects than I had for this one
The problems began as soon as I started trying to boot QEMU. In a previous post, I had gotten QEMU to work with a custom Pi 4 kernel and device tree, so I figured that I would just copy that launch script and start there. I grabbed a Pi kernel and device tree from an existing SD card loaded with Raspbian, and tried to boot the Pi using this launch script
#!/bin/bash
qemu-system-aarch64 \
-nographic \
-machine raspi4b \
-cpu cortex-a72 \
-m 2G -smp 4 \
-kernel $1 \
-dtb $2 \
--initrd $3 \
-serial null \
-chardev stdio,id=uart1 \
-serial chardev:uart1 \
--monitor noneBut when I tried to start it, I got no output. It just sat here forever
./old_launch.sh qemu_boot_files/kernel8.img qemu_boot_files/custom_device_tree.dtb qemu_boot_files/init.cpio
qemu-system-aarch64: warning: bcm2711 dtc: brcm,bcm2711-pcie has been disabled!
qemu-system-aarch64: warning: bcm2711 dtc: brcm,bcm2711-rng200 has been disabled!
qemu-system-aarch64: warning: bcm2711 dtc: brcm,bcm2711-thermal has been disabled!
qemu-system-aarch64: warning: bcm2711 dtc: brcm,bcm2711-genet-v5 has been disabled!So it seems that I couldn't even print anything to the screen. Most interesting. Last time I used QEMU I actually had the same issue, but had managed to fumble around the internet until the SEO gods saw fit to bestow upon me the following magic flags
-serial null \
-chardev stdio,id=uart1 \
-serial chardev:uart1 \Which had enabled my screen to be blessed with the glorious output of the Linux CLI
But this time the flags weren't working
Given that I had been here before, and yet I still didn't know what was
actually wrong, I figured that it was going to be easier for me to just
read the QEMU source code and understand what
-serial actually does. The alternative was to sit here and
hit the Google slot machine for a few more hours, waiting for it to spit
out another set of magic flags. And I wasn't super hopeful about that
approach working again
[2]
-serial worksLuckily for me, Silicon Valley VCs have decided that they enjoy giving away free money (in the form of free LLM tokens). And these LLMs are very good at reading code bases I'm unfamiliar with and telling me what files to read
So, with chatbot in hand (or rather in CLI), I was able to unlock the
mysteries of the QEMU -serial flag
When you pass -serial to QEMU, QEMU allocates a serial device
for you to use. When your emulated UART writes to that device, QEMU routes
the bytes you wrote to whatever back end you configured. That back end
could be a socket, a file, or, in the normal case, the
stdio of your terminal!
Looking at the Pi 4's source code, you can see the emulated UARTs setup here [3]
/* UART0 */
qdev_prop_set_chr(DEVICE(&s->uart0), "chardev", serial_hd(0));
if (!sysbus_realize(SYS_BUS_DEVICE(&s->uart0), errp)) {
return;
}
memory_region_add_subregion(&s->peri_mr, UART0_OFFSET,
sysbus_mmio_get_region(SYS_BUS_DEVICE(&s->uart0), 0));
sysbus_connect_irq(SYS_BUS_DEVICE(&s->uart0), 0,
qdev_get_gpio_in_named(DEVICE(&s->ic), BCM2835_IC_GPU_IRQ,
INTERRUPT_UART0));
/* AUX / UART1 */
qdev_prop_set_chr(DEVICE(&s->aux), "chardev", serial_hd(1));
if (!sysbus_realize(SYS_BUS_DEVICE(&s->aux), errp)) {
return;
}
In my case there are only two UARTs, so I only need to pass it at most two
serial flags. The first serial flag configures where the output of
uart0 goes, and the second serial flag configures where the
output of the aux UART goes
So in my case, given that I was configuring my serial ports using this
-serial null \
-chardev stdio,id=uart1 \
-serial chardev:uart1 \
This meant that uart0 was going to NULL, and the
aux UART was going to stdio. Given that I don't see any
output, I guess Linux isn't sending its output to the
aux UART?
So how do I configure what UART Linux uses to print?
On the Linux side things were much easier. Google pretty quickly revealed that Linux has a device tree property that tells it where to write its output to. That property is the stdout-path, which was set to this
chosen {
stdout-path = "serial0:115200n8";
/* Rest ignored for clarity*/
};
Here the device tree is telling Linux to find the
serial0 alias and use that as the stdout path. Looking at the
serial0 node I see this
serial0 = "/soc/serial@7e201000";
Which means that Linux will use that device tree node as the stdout path
for printing. Looking at that node, I see that it’s the
uart0 for the Pi
serial@7e201000 {
compatible = "arm,pl011", "arm,primecell";
reg = <0x7e201000 0x200>;
interrupts = <0x00 0x79 0x04>;
clocks = <0x08 0x13 0x08 0x14>;
clock-names = "uartclk", "apb_pclk";
arm,primecell-periphid = <0x341011>;
status = "okay";
cts-event-workaround;
pinctrl-names = "default";
pinctrl-0 = <0x09>;
uart-has-rtscts;
skip-init;
phandle = <0x43>;
bluetooth {
compatible = "brcm,bcm43438-bt";
max-speed = <0x2dc6c0>;
shutdown-gpios = <0x0b 0x00 0x00>;
local-bd-address = [00 00 00 00 00 00];
fallback-bd-address;
status = "disabled";
phandle = <0x41>;
};
};
Ah, well that's a problem. This is the device tree node for
uart0, which is NOT the aux UART
So with the Linux and the serial side done, things are now clear. My
device tree defines stdout as the uart0, but
when I invoke QEMU, I'm setting the first serial device, which is
uart0, to NULL. So everything that my terminal prints is
being discarded by QEMU
I went back and checked, and in my previous post, the device tree I was
using was set up differently. It had its stdout directed to
the aux UART instead, which is why the old flags worked
Luckily this is very easy to fix. I just changed my serial flag setup to this [4]
-serial mon:stdio And now I'm getting output!
qemu-system-aarch64: warning: bcm2711 dtb: brcm,bcm2711-pcie has been disabled!
qemu-system-aarch64: warning: bcm2711 dtb: brcm,bcm2711-rng200 has been disabled!
qemu-system-aarch64: warning: bcm2711 dtb: brcm,bcm2711-thermal has been disabled!
qemu-system-aarch64: warning: bcm2711 dtb: brcm,bcm2711-genet-v5 has been disabled!
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[ 0.000000] KASLR disabled due to lack of seed
[ 0.000000] Machine model: Raspberry Pi 4 Model B
[ 0.000000] efi: UEFI not found.
[ 0.000000] Reserved memory: created CMA memory pool at 0x000000002c000000, size 64 MiBMy joy was short-lived, however, since ~30 seconds after QEMU started I saw this
[ 9.376777] systemd[1]: Detected architecture arm64.
Welcome to Debian GNU/Linux 13 (trixie)!
[ 9.495035] systemd[1]: Hostname set to <NickPI>;.
[ 10.336334] systemd[1]: bpf-restrict-fs: BPF LSM hook not enabled in the kernel, BPF LSM not supported.
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]I'm pretty sure the boot timestamps aren't supposed to reset back to 0. That probably indicates that something has gone wrong
I let it run for a little while longer, but all it would do is boot loop. Once I realized this was going to keep happening, I killed the process and started thinking
After about 10 minutes of sitting there with no idea what was wrong, I just dumped the entire boot message into a chat bot and asked it why this was happening. And, to my genuine surprise, it identified the issue! The watchdog was rebooting the Pi! [5]
Given that I was already running the Pi on my x86 desktop, the emulated Pi was already very slow, due to the ARM to x86 translation that QEMU does. So anything timing related, like the watchdog, was never going to work, even if it didn't cause crashes. With all that in mind, I decided to just try turning off the watchdog and hope for the best
watchdog@7e100000 {
compatible = "brcm,bcm2711-pm", "brcm,bcm2835-pm-wdt";
#power-domain-cells = <0x01>;
#reset-cells = <0x01>;
reg = <0x7e100000 0x114 0x7e00a000 0x24 0x7ec11000 0x20>;
reg-names = "pm", "asb", "rpivid_asb";
clocks = <0x08 0x15 0x08 0x1d 0x08 0x17 0x08 0x16>;
clock-names = "v3d", "peri_image", "h264", "isp";
system-power-controller;
phandle = <0x49>;
/* This turns off the watchdog, which hopefully fixes the reboot issue */
status = "disabled";
};And that fixed it! I restarted QEMU, and, after much waiting, was finally greeted with a login screen!
[ OK ] Started cups.path - CUPS Scheduler.
[ OK ] Closed cups.socket - CUPS Scheduler.
Stopping cups.socket - CUPS Scheduler...
[ OK ] Listening on cups.socket - CUPS Scheduler.
Starting cups.service - CUPS Scheduler...
[ OK ] Started cups.service - CUPS Scheduler.
[ OK ] Started cups-browsed.service - Mak…te CUPS printers available locally.
[ OK ] Finished logrotate.service - Rotate log files.
[ TIME ] Timed out waiting for device dev-dri-card0.device - /dev/dri/card0.
[ TIME ] Timed out waiting for device dev-d…rD128.device - /dev/dri/renderD128.
Starting lightdm.service - Light Display Manager...
[ 110.959580] sh[1023]: Completed socket interaction for boot stage final
Debian GNU/Linux 13 NickPI ttyAMA0
My IP address is 10.0.2.15 fec0::5054:ff:fe12:3457I was in. Surely this would be the last issue I had...
foreshadowing is a literary device that writers utilize as a means to indicate or hint to readers something that is to follow or appear later in a story
So Linux was now printing characters to the screen and not crashing. Good progress. It was time for the final part of the process - getting the EEPROM to work
To that end I setup an EEPROM on I2C bus 1 at address 0x50 (adding this to the launch script) [6]
-drive if=none,id=i2c_storage,format=raw,file=eeprom.bin \
-device at24c-eeprom,bus=i2c-bus.1,address=0x50,drive=i2c_storage,rom-size=4096 \
The first line of that command tells QEMU to create a drive called
i2c_storage that uses eeprom.bin as the backing
file
The second line tells QEMU to create an EEPROM device at address
0x50 on I2C bus 1. The EEPROM uses the drive that I just
created (i2c_storage) to store all of its data
I then added an EEPROM to the appropriate I2C bus in my device tree
i2c@7e804000 {
compatible = "brcm,bcm2711-i2c", "brcm,bcm2835-i2c";
reg = <0x7e804000 0x1000>;
interrupts = <0x00 0x75 0x04>;
clocks = <0x08 0x14>;
#address-cells = <0x01>;
#size-cells = <0x00>;
status = "okay";
pinctrl-names = "default";
pinctrl-0 = <0x15>;
clock-frequency = <0x186a0>;
phandle = <0x48>;
/** We should see an EEPROM now */
eeprom@50 {
compatible = "atmel,24c32";
status = "okay";
reg = <0x50>;
pagesize = <0x100>;
};
};and booted up Linux
After waiting for Linux to boot, I looked for the EEPROM on I2C bus 1 and found... nothing
ls -l /sys/bus/i2c/devices/1-0050/
total 0
-r--r--r-- 1 root root 4096 May 17 15:34 modalias
-r--r--r-- 1 root root 4096 May 17 15:26 name
lrwxrwxrwx 1 root root 0 May 17 15:34 of_node -> ../../../../../../firmware/devicetree/base/soc/i2c@7e804000/eeprom@50
drwxr-xr-x 2 root root 0 May 17 15:34 power
lrwxrwxrwx 1 root root 0 May 17 15:26 subsystem -> ../../../../../../bus/i2c
-rw-r--r-- 1 root root 4096 May 17 15:26 uevent
-r--r--r-- 1 root root 4096 May 17 15:34 waiting_for_supplierHmmm. There should be a file called eeprom here
Why is there no EEPROM?
I wasn't sure if this was a driver issue, a device tree issue, or a QEMU issue. So I tried out a few different tools to get a better idea of what was going on
Would i2cdetect find anything?
i2cdetect -y 1
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: -- -- -- -- -- -- -- --Nope
Could i2cget read any data?
i2cget -y 1 0x50
Error: Read failedNo
Did dmesg have a helpful error message saying this is the issue?
dmesg | grep -i at24
[ 43.375424] at24 1-0050: supply vcc not found, using dummy regulatorShocker, but there was in fact nothing useful there either! [7]
To double check that I had indeed setup the EEPROM on I2C bus 1, I used QEMU's monitor to see what devices were attached to I2C bus 1
dev: bcm2835-i2c, id ""
gpio-out "sysbus-irq" 1
mmio ffffffffffffffff/0000000000000024
bus: i2c-bus.1
type i2c-bus
dev: at24c-eeprom, id ""
rom-size = 4096 (0x1000)
address-size = 2 (0x2)
writable = true
drive = "i2c_storage"
address = 80 (0x50)And yes, my EEPROM was there all right. I just couldn't talk to it
This was getting ridiculous!
I had assumed that getting the Pi to work would be the easy part, but as evidenced by this post's existence, it was most certainly not easy. I had come too far at this point to give up. This thing was going to talk to me, whether or not it liked it. But where to start the hunt for the bug?
Given that I didn't know at what layer of the stack this issue was happening, I decided to start small. I would get a one byte read from the EEPROM to work and go from there
To that end, there were four things I needed to know
Looking at the EEPROM datasheet, a current address read only requires two things [8]
So reading from the actual EEPROM is very straightforward. What about reading from QEMU's virtual EEPROM?
On the QEMU side, the emulated EEPROM is also easy to read from. There's only a few data members that we care about
struct EEPROMState {
/* Most members removed for clarity */
/* The current address */
uint16_t cur;
/* total size in bytes */
uint32_t rsize;
/* Stores the EEPROM data */
uint8_t *mem;
/* The backing device on the host system */
BlockBackend *blk;
};
When I do a read from QEMU's EEPROM, QEMU calls the
recv() callback, which just reads data from the current
address and then increments that address by one
static
uint8_t at24c_eeprom_recv(I2CSlave *s)
{
EEPROMState *ee = AT24C_EE(s);
uint8_t ret;
/* omitted for clarity */
/* Read data from the current address (starts off at 0) */
ret = ee->mem[ee->cur];
/* Increments the address by 1, so subsequent reads will read from the next address */
ee->cur = (ee->cur + 1u) % ee->rsize;
return ret;
}
So the emulated EEPROM behaves exactly the same as the physical EEPROM,
which is good. Invoking the recv() function should indeed
return 0x00, since the entire EEPROM is all zeros. No surprises there
Onto the Pi's I2C controller
Chapter 3 of the manual lists the registers that are available on the I2C controller
| Offset | Name | Description |
|---|---|---|
| 0x00 | C | Control |
| 0x04 | S | Status |
| 0x08 | DLEN | Data length |
| 0x0C | A | Address |
| 0x10 | FIFO | Data FIFO |
| 0x14 | DIV | Clock divider |
| 0x18 | DEL | Data Delay |
| 0x1C | CLKT | Clock Stretch Timeout |
Luckily only four of these matter: the A, DLEN, C, and S registers
The address (A) register controls the address of the I2C device the Pi is talking to, which is 0x50 in my case
The data length (DLEN) register controls how many bytes of data will be transferred during the transaction. In my case that's one
The status (S) register tells us various things about the I2C controller, such as how much data is in the FIFO [9]
The control (C) register controls a few different things. Importantly, this register controls
So in my case the register should be setup to start a read and then trigger an interrupt when there's data in the FIFO to be read
With all of that context, sending a transaction should be pretty straightforward
And indeed, if I look at the Pi's I2C driver in Linux, this is exactly what Linux does
static void bcm2835_i2c_start_transfer(struct bcm2835_i2c_dev *i2c_dev)
{
u32 c = BCM2835_I2C_C_ST | BCM2835_I2C_C_I2CEN;
struct i2c_msg *msg = i2c_dev->curr_msg;
bool last_msg = (i2c_dev->num_msgs == 1);
if (!i2c_dev->num_msgs)
return;
i2c_dev->num_msgs--;
i2c_dev->msg_buf = msg->buf;
i2c_dev->msg_buf_remaining = msg->len;
if (msg->flags & I2C_M_RD)
c |= BCM2835_I2C_C_READ | BCM2835_I2C_C_INTR;
else
c |= BCM2835_I2C_C_INTT;
if (last_msg)
c |= BCM2835_I2C_C_INTD;
bcm2835_i2c_writel(i2c_dev, BCM2835_I2C_A, msg->addr);
bcm2835_i2c_writel(i2c_dev, BCM2835_I2C_DLEN, msg->len);
bcm2835_i2c_writel(i2c_dev, BCM2835_I2C_C, c);
}First the C register is setup to enable the I2C controller and start the transfer
u32 c = BCM2835_I2C_C_ST | BCM2835_I2C_C_I2CEN;Then Linux specifies that it wants a read, and that it wants to trigger an interrupt on the FIFO having data to read
if (msg->flags & I2C_M_RD)
c |= BCM2835_I2C_C_READ | BCM2835_I2C_C_INTR;Then it writes to the A register
bcm2835_i2c_writel(i2c_dev, BCM2835_I2C_A, msg->addr);Then it writes the DLEN register
bcm2835_i2c_writel(i2c_dev, BCM2835_I2C_DLEN, msg->len);Finally, Linux writes to the control register to actually kick off the read
bcm2835_i2c_writel(i2c_dev, BCM2835_I2C_C, c);Then, whenever an interrupt triggers, this ISR is called (which just reads from the FIFO if there's data available)
static irqreturn_t bcm2835_i2c_isr(int this_irq, void *data)
{
struct bcm2835_i2c_dev *i2c_dev = data;
u32 val, err;
/** Read from the status register, since Linux needs to know what condition triggered the ISR */
val = bcm2835_i2c_readl(i2c_dev, BCM2835_I2C_S);
/** Details omitted for simplicity */
/** Check if the ISR was triggered by the FIFO condition */
if (val & BCM2835_I2C_S_RXR) {
if (!i2c_dev->msg_buf_remaining) {
i2c_dev->msg_err = val | BCM2835_I2C_S_LEN;
goto complete;
}
/** Linux reads from the FIFO */
bcm2835_drain_rxfifo(i2c_dev);
return IRQ_HANDLED;
}
/** Details omitted for clarity */
}Ok, so now that I know how Linux talks to the I2C controller, the last piece of information that I need is how QEMU emulates the I2C controller
Looking at the QEMU source code, we see the same registers that were defined in the manual
struct BCM2835I2CState {
/* <private>; */
SysBusDevice parent_obj;
/* <public>; */
MemoryRegion iomem;
I2CBus *bus;
qemu_irq irq;
uint32_t c;
uint32_t s;
uint32_t dlen;
uint32_t a;
uint32_t div;
uint32_t del;
uint32_t clkt;
uint32_t last_dlen;
};This is the handler that runs when QEMU traps on a write to one of the I2C controller registers
static void bcm2835_i2c_write(void *opaque, hwaddr addr,
uint64_t value, unsigned int size)
{
BCM2835I2CState *s = opaque;
uint32_t writeval = value;
switch (addr) {
case BCM2835_I2C_C:
/* ST is a one-shot operation; it must read back as 0 */
s->c = writeval & ~BCM2835_I2C_C_ST;
/* Start transfer */
if (writeval & (BCM2835_I2C_C_ST | BCM2835_I2C_C_I2CEN)) {
bcm2835_i2c_begin_transfer(s);
/*
* Handle special case where transfer starts with zero data length.
* Required for zero length i2c quick messages to work.
*/
if (s->dlen == 0) {
bcm2835_i2c_finish_transfer(s);
}
}
bcm2835_i2c_update_interrupt(s);
break;
case BCM2835_I2C_DLEN:
s->dlen = writeval;
s->last_dlen = writeval;
break;
case BCM2835_I2C_A:
s->a = writeval;
break;
/* omitted for clarity */
}
}So when doing a write, Linux first writes to the A and DLEN registers. Both of those writes set the corresponding register values in the emulated i2c device state
case BCM2835_I2C_A:
s->a = writeval;
break;case BCM2835_I2C_DLEN:
s->dlen = writeval;
s->last_dlen = writeval;
break;Linux then writes to the control register, which is more involved
Here QEMU first checks if we're starting a transaction (via the
BCM2835_I2C_C_ST bit in the C register). If a transaction is
being started, then QEMU begins the i2c transfer via
bcm2835_i2c_begin_transfer, which does some internal
bookkeeping and other setup code
if (writeval & (BCM2835_I2C_C_ST | BCM2835_I2C_C_I2CEN)) {
bcm2835_i2c_begin_transfer(s);
QEMU then calls
bcm2835_i2c_update_interrupt();
which triggers any interrupts that need to occur. In my case, since I'm
starting a read, the first if statement evaluates to true and QEMU will
trigger the FIFO interrupt
static void bcm2835_i2c_update_interrupt(BCM2835I2CState *s)
{
int do_interrupt = 0;
/* Interrupt on RXR (Needs reading) */
if (s->c & BCM2835_I2C_C_INTR && s->s & BCM2835_I2C_S_RXR) {
do_interrupt = 1;
}
/* Interrupt on TXW (Needs writing) */
if (s->c & BCM2835_I2C_C_INTT && s->s & BCM2835_I2C_S_TXW) {
do_interrupt = 1;
}
/* Interrupt on DONE (Transfer complete) */
if (s->c & BCM2835_I2C_C_INTD && s->s & BCM2835_I2C_S_DONE) {
do_interrupt = 1;
}
qemu_set_irq(s->irq, do_interrupt);
}The FIFO interrupt, in turn, will then result in the ISR from earlier being called, at which point Linux will drain the FIFO. When Linux tries to read from the FIFO address, QEMU will trap that access and call this code
static uint64_t bcm2835_i2c_read(void *opaque, hwaddr addr, unsigned size)
{
BCM2835I2CState *s = opaque;
uint32_t readval = 0;
switch (addr) {
case BCM2835_I2C_FIFO:
/* We receive I2C messages directly instead of using FIFOs */
if (s->s & BCM2835_I2C_S_TA) {
/** Here we call the EEPROM's recv() function and get 0x00 (ideally) */
readval = i2c_recv(s->bus);
s->dlen -= 1;
if (s->dlen == 0) {
bcm2835_i2c_finish_transfer(s);
}
}
bcm2835_i2c_update_interrupt(s);
break;
/* Details omitted for clarity */
}
}
And it's at this point that we actually trigger the read from the virtual
EEPROM by calling its recv() callback
[10]
So, now that I knew how everything worked, I expected the following to
happen when I called i2cget
i2cget makes a request to read one byte from address 0x50
recv() function
recv() (which should be 0 in
this case)
i2cget prints out 0x00
Now armed with some idea of what was supposed to happen, I added a debug
printf to all QEMU register accesses and re-ran my
i2cget command. And I got this
i2cget -y 1 0x50
BCM: write addr=0x0c val=0x00000050
BCM: A <- 0x50
BCM: write addr=0x08 val=0x00000001
BCM: DLEN <- 1
BCM: write addr=0x00 val=0x00008581
BCM: C <- 0x00008581 (ST=1 I2CEN=1 READ=1)
BCM: triggering begin_transfer (ST=1 I2CEN=1)
BCM: begin_transfer addr=0x50 dir=READ dlen=1
at24c-eeprom : clear
BCM: direction=READ, setting RXR|RXD
BCM: update_interrupt
BCM: RXR interrupt
BCM: do_interrupt=1
BCM: write addr=0x00 val=0x00000010
BCM: C <- 0x00000010 (ST=0 I2CEN=0 READ=0)
BCM: update_interrupt
BCM: do_interrupt=0
Error: Read failedAlright, let's go through this. The first thing to note is that the read is being setup correctly
Earlier I said that I expect these three steps
And that's exactly what's happening here [11]
i2cget -y 1 0x50
BCM: write addr=0x0c val=0x00000050
BCM: A <- 0x50
BCM: write addr=0x08 val=0x00000001
BCM: DLEN <- 1
BCM: write addr=0x00 val=0x00008581
BCM: C <- 0x00008581 (ST=1 I2CEN=1 READ=1)
BCM: triggering begin_transfer (ST=1 I2CEN=1)
BCM: begin_transfer addr=0x50 dir=READ dlen=1So Linux is doing its part. The next steps I expected to see were
I don't expect to see a message for Linux going to sleep (since that's not going to touch the I2C controller registers), but I do see QEMU trying to trigger the interrupt
BCM: update_interrupt
BCM: RXR interrupt
BCM: do_interrupt=1However, I never see Linux trying to read from the FIFO in response to the FIFO interrupt that QEMU triggered. All that happens after QEMU triggers the FIFO interrupt is Linux eventually doing this write to the C register [12]
BCM: update_interrupt
BCM: RXR interrupt
BCM: do_interrupt=1
BCM: write addr=0x00 val=0x00000010
BCM: C <- 0x00000010 (ST=0 I2CEN=0 READ=0)
BCM: update_interruptAnd then the read fails
BCM: do_interrupt=0
Error: Read failedSo Linux is triggering the read. QEMU sees that read and tries to interrupt Linux saying that there's data to be read. But Linux never tries to read the data. Why?
It seems there are two options here
To test out which of these was true, I added a call to
trace_printk() to the bcm2835_isr(), recompiled
the kernel, and did the same i2cget
Checked the trace logs
Nothing
So the interrupt wasn't firing at all... [13]
Now I had an issue, because not only do I not know how interrupts work in QEMU, I don't even know how they work on the Pi. So back to the manual it was, this time to read about interrupts!
The Pi 4 has two interrupt controllers (see chapter 6 of the manual for details). The first is the legacy interrupt controller, which is carried over from previous versions of the Pi. The second one, which is new for the Pi 4, is the GIC (generic interrupt controller) 400 [14]
According to the documentation, the Pi should be using the GIC by default. So in theory I2C interrupts should be routed from the I2C controller to the GIC, and then from the GIC to Linux. But something in that chain isn't working
So, let’s start with the most basic thing first; did I actually configure Linux to use the GIC? Looking at the device tree I see this
/dts-v1/;
/memreserve/ 0x0000000000000000 0x0000000000001000;
/ {
compatible = "raspberrypi,4-model-b", "brcm,bcm2711";
model = "Raspberry Pi 4 Model B";
#address-cells = <0x02>;
#size-cells = <0x01>;
interrupt-parent = <0x01>;
Which tells me that the interrupt controller is whatever has
phandle = <0x1>. Searching for that, I do indeed see the GIC
interrupt-controller@40041000 {
interrupt-controller;
#interrupt-cells = <0x03>;
compatible = "arm,gic-400";
reg = <0x40041000 0x1000 0x40042000 0x2000 0x40044000 0x2000 0x40046000 0x2000>;
interrupts = <0x01 0x09 0xf04>;
phandle = <0x01>;
};So Linux is using the correct interrupt controller. Onto the next question; is the I2C interrupt actually being routed to the correct spot?
After some digging I found this in bmc2835_peripherals.c
for (n = 0; n < ORGATED_I2C_IRQ_COUNT; n++) {
sysbus_connect_irq(SYS_BUS_DEVICE(&s->i2c[n]), 0,
qdev_get_gpio_in(DEVICE(&s->orgated_i2c_irq), n));
}
qdev_connect_gpio_out(DEVICE(&s->orgated_i2c_irq), 0,
qdev_get_gpio_in_named(DEVICE(&s->ic),
BCM2835_IC_GPU_IRQ,
INTERRUPT_I2C));So the I2C interrupt is indeed being routed to the legacy interrupt controller, not to the GIC. So when an interrupt triggers on the I2C bus, that interrupt goes into the legacy interrupt controller, which Linux isn't listening to
Mystery solved! Or so I thought
Because then I saw this in the same file. The UART interrupt is also wired into the legacy interrupt controller!
sysbus_connect_irq(SYS_BUS_DEVICE(&s->uart0), 0,
qdev_get_gpio_in_named(DEVICE(&s->ic), BCM2835_IC_GPU_IRQ,
INTERRUPT_UART0));But the UART was definitely working. It's how I was seeing output on the screen and interacting with the terminal. So if the UART and I2C were both routed to the legacy interrupt controller, why was only I2C not working?
There must be something else going on.... but what??
It seemed like the best thing to do was to follow one of the interrupts that I knew was working, and see where it's used. Given that I know the UARTs are working, I started there
Grepping for uart0 shows that we're only using it in two
spots (the max78000 and the zynq files can be
ignored)
[15]
rgc "\->uart0"
qemu/hw/misc/zynq_slcr.c
285: clock_set(s->uart0_ref_clk,
315: clock_propagate(s->uart0_ref_clk);
qemu/hw/misc/max78000_gcr.c
160: device_cold_reset(s->uart0);
qemu/hw/arm/bcm2835_peripherals.c
100: object_initialize_child(obj, "uart0", &s->uart0, TYPE_PL011);
290: qdev_connect_clock_in(DEVICE(&s->uart0), "clk",
317: qdev_prop_set_chr(DEVICE(&s->uart0), "chardev", serial_hd(0));
318: if (!sysbus_realize(SYS_BUS_DEVICE(&s->uart0), errp)) {
323: sysbus_mmio_get_region(SYS_BUS_DEVICE(&s->uart0), 0));
324: sysbus_connect_irq(SYS_BUS_DEVICE(&s->uart0), 0,
qemu/hw/arm/bcm2838.c
180: sysbus_connect_irq(SYS_BUS_DEVICE(&ps_base->uart0), 0,
Ahhh, now this is promising! The BCM2838 is the SOC used by
the Pi 4. So the fact that I see an IRQ for uart0
in that file is very promising. Opening up the file I then see this
/* Connect UART0 to the interrupt controller */
sysbus_connect_irq(SYS_BUS_DEVICE(&ps_base->uart0), 0,
qdev_get_gpio_in(gicdev, GIC_SPI_INTERRUPT_UART0));
So it does indeed appear that the GIC is being wired up separately from
the initial uart0 interrupt line. And if I go search for the
same behavior using the I2C interrupt line, I don't see the I2C interrupt
in
bcm2835_peripherals.c
rgc "orgated_i2c_irq"
hw/arm/bcm2835_peripherals.c
34:#define ORGATED_I2C_IRQ_COUNT 3
179: &s->orgated_i2c_irq, TYPE_OR_IRQ);
180: object_property_set_int(OBJECT(&s->orgated_i2c_irq), "num-lines",
181: ORGATED_I2C_IRQ_COUNT, &error_abort);
500: if (!qdev_realize(DEVICE(&s->orgated_i2c_irq), NULL, errp)) {
503: for (n = 0; n < ORGATED_I2C_IRQ_COUNT; n++) {
505: qdev_get_gpio_in(DEVICE(&s->orgated_i2c_irq), n));
507: qdev_connect_gpio_out(DEVICE(&s->orgated_i2c_irq), 0,So the I2C interrupt is not being hooked up anywhere else! That's it. That's the bug. No one ever added the I2C interrupt to the GIC, so QEMU's interrupts are going to the legacy interrupt controller, which Linux isn't checking
With that in mind I made this patch to the QEMU source to redirect the interrupt from the legacy interrupt controller to the GIC
qdev_connect_gpio_out(DEVICE(&ps_base->orgated_i2c_irq), 0,
qdev_get_gpio_in(gicdev, GIC_SPI_INTERRUPT_I2C));Along with the appropriate header definition
#define GIC_SPI_INTERRUPT_I2C 117
Then I recompiled QEMU and tested out my i2cget again. And
this time it worked!
i2cget -y 1 0x50
0x00But reading one byte from the EEPROM was small potatoes. It was time for the real test. Would enabling the EEPROM in the device tree work this time?
I turned the EEPROM back on, booted the system, and waited
Time for the moment of truth!
ls -l /sys/bus/i2c/devices/i2c-1/1-0050/ | grep -i eeprom
-rw------- 1 root root 4096 May 17 16:57 eepromAnd success! The EEPROM was there!
And I could read from it and write to it as well!
echo "hello there" | sudo tee /sys/bus/i2c/devices/i2c-1/1-0050/eeprom
hello theresudo cat /sys/bus/i2c/devices/1-0050/eeprom
hello thereIt may have taken much, much longer than I expected, but I now have a virtual Raspberry Pi with an attached EEPROM!
So, after all of that, I finally have a virtual Pi that I can use for my initial use case. There are actually a few other things I didn't get into in this post, such as trying to get the networking setup. But I can save that for a future post, since that's when I'll actually need the networking to function
But, to borrow a phrase from standup, I am now unblocked! I can finally get onto the thing I actually wanted to do and see just how useful QEMU is for testing my Pi!