It’s probably not the plugins but the fluidsynth library or host not handling it correctly.
this was linked from another forum which has the same problem with fluidsynth plugins. He solved it thru host
Fluida Issue#18
Though this problem doesn’t exist for windows and ubuntu x86_64. I did compile some fluidsynth plugins and it works fine except for RPI arm64.
My only option for now is either not use fluidsynth or keep at least one plugin attached and avoid switching pedalboards.