scudo's junkie site - The NetBSD OpenSSL PowerPC oopsie

The NetBSD OpenSSL PowerPC oopsie

Jun 7th, 2025 - scudo

Last year, the NetBSD team has started to maintain an official port of the operating system to the Nintendo Wii, powered by a custom PowerPC processor by IBM.

Out of curiosity, I tried to run it on my console.

Everything went smooth, until when I tried running pkg_add pkgin; the program would just hang immediately and wouldn't respond.

I then whipped out gdb and ran the program through the debugger; I discovered that pkg_add crashed during OpenSSL initialization. More specifically, it was crashing with an illegal instruction exception in OPENSSL_ppc64_probe() from /usr/bin/libcrypto.so.15.

The first crash

gdb showing the program crashed with a SIGILL on an illegal instruction

Wait, what?

PPC64? But the Wii only has a 32 bit CPU, no wonder it's crashing!

I spent some more time disassembling parts of libcrypto to figure out why an illegal instruction exception was thrown in the first place. I ended up taking a look at the same function that crashed, OPENSSL_ppc64_probe. This is what I found:

The culprit

A decompilation showing a fcfid instruction present in the code

That fcfid instruction is, indeed, present only on 64-bit PowerPC CPUs, as you can see from IBM's website:

The culprit

IBM web page for the fcfid instruction; says 'Converts a 64-bit signed integer stored in a double into a double-precision floating-point value. This intrinsic is valid on any 64-bit PowerPC architecture.'

Testing on real hardware confirms this (again!): the GNU assembler (as) indeed does not recognize the instruction.

The culprit

GNU assembler saying 'unrecognized opcode: fcfid'

Digging out NetBSD's source code, I can indeed confirm that libcrypto contains this function. Its scope appears to check if the CPU the OS is running on supports 64-bit integer and floating point operations.

The problem is that this code is part of the powerpc source tree, which is the 32-bit flavour of the architecture, so running this function will always raise an illegal instruction exception:

The culprit

In the end, I gave up trying to figure this out - but this bug still persists in what was the latest PowerPC release when all of this happened (April 23, 2025).