Status Update: The official Orange Pi 5 image isn't updated often, so I tried their official 3rd party image by Joshua Riek. Wow. Updated kernel, no crashes, and build-in graphics acceleration from the get-go.
https://github.com/Joshua-Riek/ubuntu-rockchip
Handles my NVMe drives without problems, the UI (using Ubuntu Desktop) is accelerated so it's smooth and usable. It's not smooth like an i9 with a high end nVidia card, but for this board, it's smooth and totally usable.
So this is my current setup and will be for a while, since everything works exactly as expected. Quite impressed.