Try running alsamixer and reduce the volume levels, sometimes too high values distort the audio.
@Giedrius I have already tried that, sadly no luck at all.
@Henk I thought about buying one for the future, but I am currently doing a build with an 8" screen that sits on a tabletop with a 3d printed enclosure.
Update: I managed to borrow a Rpi4b and everything worked perfectly using the USB audio. Now I am even more focused on trying to figure out why it crackles so badly when using a Pi3b.