I am using journalctl -o short-precise -k -b -1 for the previous boot per the DMESG post above and the boot is from yesterday so it appears that when it is in the “red light” state it does in fact never try to boot, at least far enough to log anything.
And to fundamentally restate what is happening. If I unplug and replug, (simulating a freeze/crash with a improper shutdown), within 2 minutes, it just does nothing and has solid red LED on. If I wait 3 minutes it usually boots and has the green LED blinking with the solid red LED on. It is like there is some sort of capacitor energy left over somehow.
UPDATE:
This was wrong. This was on my host Arch system. There is no persistent storage for DMESG to save boot logs it appears:
patch@patchbox:~ $ journalctl -o short-precise -k -b -1
Specifying boot ID or boot offset has no effect, no persistent journal was found.
patch@patchbox:~ $ journalctl --list-boot
0 c261c521f71c4dce8c478d4db9589073 Thu 2020-12-17 17:17:01 GMT—Thu 2020-12-17 18:34:03 GMT
UPDATE2: From reading more Pi LED indicator states here What do system LEDs signify? - Raspberry Pi Stack Exchange, solid RED is good, means power is correct, and the flashing GREEN at boot indicates SD card activity, which never happens in this 3 minute window. I don’t think it is FSCK because if I simply wait for 3 minutes it boots right up with GREEN activity. There seems to be something in that 3 minutes that happens.
UPDATE3: Finally figured out how to enable persistent journalctl storage. I’ll try to crash it now with the MIDI passthrough to the synth.
sudo vim /etc/systemd/journald.conf (change line: Storage=auto TO Storage=persistent)
mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
UPDATE4: K, confirmed that when it is in this RED only state with no SD card activity with GREEN that there is in fact no boot logs per journalctl --output short-precise --dmesg --boot -1 as that shows the previous successful boot logs, not the failed boot with RED state. The 3 minute wait seems to be working 100% of the time now. And after the 3 minutes it boots within 30 seconds every time (SSH works then).
UPDATE5: When I do a proper sudo reboot, I am always getting “Dirty bit is set”:
patch@patchbox:~ $ sudo fsck.vfat -a /dev/mmcblk0p1
fsck.fat 4.1 (2017-01-24)
0x41: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
Automatically removing dirty bit.
Performing changes.
/dev/mmcblk0p1: 232 files, 80116/516190 clusters
Others are seeing same issue here > Dirty bit is always found set at system boot · Issue #688 · cyoung/stratux · GitHub
UPDATE6: Shortly after I tested the above fsck, the Pi crashed again with SOLID RED and GREEN. I am now putting the card through some tests and full formatting on my host Arch system.|
UPDATE7: Running a sudo f3probe /dev/sda on this on Arch now per this recommendation > bad blocks - How can I test the full capacity of an SD card in Linux? - Super User.
UPDATE8: Looks good with f3probe, gonna fully reformat and flash with balenaEtcher again.
❯ sudo f3probe /dev/sda
F3 probe 8.0
Copyright (C) 2010 Digirati Internet LTDA.
This is free software; see the source for copying conditions.
WARNING: Probing normally takes from a few seconds to 15 minutes, but
it can take longer. Please be patient.
Probe finished, recovering blocks... Done
Good news: The device '/dev/sda' is the real thing
Device geometry:
*Usable* size: 29.81 GB (62521344 blocks)
Announced size: 29.81 GB (62521344 blocks)
Module: 32.00 GB (2^35 Bytes)
Approximate cache size: 0.00 Byte (0 blocks), need-reset=no
Physical block size: 512.00 Byte (2^9 Bytes)
Probe time: 1'38"
UPDATE9: Did a full fat32 reformat (erase data, took 20 minutes) with Disk Utility and then flashed again with balenaEtcher. no crash so far after an hour or so. Still has the dirty bit issue with a normal reboot, fwiw. I also did a memtester 500 1 to test 500MB of ram, which passed. There is a total of 1024MB/1GB so I cannot test all of it.
UPDATE10: K, I revisited the comments here and tested this, and confirmed that it only happens with the Pisound attached. If I take the Pisound off and turn it off by pulling the power plug, it boots up right away with GREEN SD activity instantly after plugging it back in, just the Pi 3 by itself.
UPDATE11: Okay, so I reattached the pisound, but left the case off, and was able to get it booting almost every single time with a manual unplug with the exception of 2 times, which I then unplugged the ethernet cable and got it going instantly again. So it does seem like maybe something to do with the case or pressure somewhere, maybe the way I was holding it. I have tried at least 10 times now and it seems like it is working. I could see it maybe being temperature, but also could see it being some type of pressure. I am going to leave the case off for a while and see if I can just use it for a few days without it crashing.
UPDATE12: Okay, seems to be overheating that causes it. Done a bunch more tests, and this seems likely for the crashes too. I was able to get it to not boot right away after a bit of keeping it on and then put a bag of peas underneath the plastic bottom of the case and it was able to boot within 5-10 seconds of it being unplugged instead of the usual 3 minutes. I have been monitoring temp here cat /sys/class/thermal/thermal_zone0/temp and went from ~46000 (46 Celsius/115 Fahrenheit, divide by 1000) to currently 55 C/131 F in just 10 minutes or so with the Pisound on and the case is still off.
I also found this thread of overheating Pisound (Raspberry Pi Audio/MIDI interface) is here! - #78 by cata - Equipment - lines where another user has reported, but said they never posted here, and just don’t use it because of that:
No solutions here - but just wanted to say I’ve also experienced similar seemingly heat related issues with my Pisound 
Very often has trouble booting after being on for ~15 minutes prior, waiting another 5 minutes seems to fix it.
Multiple Pisound owners/users have reported in the thread above. This all makes sense now.
UPDATE13: I am going to see if I can find my Pi heatsink and/or put the case back on and keep an icepack underneath it to see if I can get some stability this way. I suspect either the Pisound or the Pi itself has a hardware issue since otherwise this would be widely reported, but it was hard to figure out for myself, so maybe not.
UPDATE14: Hope, it appears that it maybe the SD card itself being temperature dependent! Temp Sensitive Boot Problem - Raspberry Pi Forums (
Temp Sensitive Boot Problem)
The the Pi is first turned on it boots correctly and it can be powered down within the first 30 seconds or so and it will behave normally, booting up reliably.
Once it has been running for longer than 30 seconds and it is disconnected and then reconnected. the red power LED comes on and nothing else happens. If unplug it and plug it back in the same thing happens. The only way to make it boot again is to physically disconnect it for around 10 seconds and connected it back again at which point it boots as expected.
i found an old San disk 2gig card and it seems to work perfectly! this is mad, it seems the SD cards are temperature dependant
I think too the SD card temperature is the problem. I stated before that I had a kingston card that rebooted everytime.
UPDATE15: I definitely think it is the card now. Now I need to know what make and model card to get that people have success with! The make/model I have that has problems is Samsung EVO Plus 32GB.
The maximum operating temperature of the Raspberry Pi is 85°C so 40-50°C is probably OK.
source: pi 3 - How Hot Is Too Hot? - Raspberry Pi Stack Exchange