You are almost certainly overthinking this. In my benchmark tests (Benchmarking audio performance) I get over 40 simple, but realistic synth voices at very low latency on a Raspberry Pi 3b. With tuning the system a little, you can get double that. This is in single-core SuperCollider. Your performance with Pd should be similar.
I’d say just build the 4 voice polyphonic synth directly in Pd, the normal way… Wait until you hit limitations before you introduce the complicating factor of pd~.