Program Change Clusters with values 1 or 124 and duplicate all note off messages with value 0 or 123

okay - I have an issue that I can’t figure out. While it’s entirely possible it’s the midihub pipeline I’ve programmed, I think it could also be a midihub bug. And the repro case is hard to describe as I believe it might be activated by multiple messages coming into the midi hub at once.

Anyways, I have this intermittent behavior where the midihub will spew program change messages when it’s not supposed to. It will always look like three PC messages being sent at a time and they will either have the value of 1 or 124 and they will go to the channels assigned to the given pipeline (I set my pipelines up to only operate on a single channel). The behavior of the issue is not consistent so, sometimes I’ll get values of 1, and the next three times the bug happens i’ll get values of 1, and then suddenly i’ll get values of 124. But I never get PC values other than 1 or 124 in these erroneous PC messages.

The pipeline itself (file attached) converts CC values to to Play and Start messages on specific channels so I can use CC’s to start/stop other sequencers from my main sequencer. PC values are supposed to just ‘passthrough’ and when I send a PC on it’s own, it most certainly does - I send a PC into my midi hub pipeline I get a single PC out. Any time I send just a PC through these pipelines, the PC seems to arrive just fine.

However, when my sequencer loads a song it sends a few PCs and CC messages and sometimes this causes the error to occur - sometimes it doesn’t (I’m saying that sometimes it works totally fine). Because the behavior has this “random” characteristic to it, that’s why I’m not entirely sure its my midi pipelines I’ve programmed and potentially a bug in the midihub.

midhub_repro.mhp (598 Bytes)

I basically repro this by loading songs in my sequencer. When a song loads in the sequencer, it sends All Note Off messages to each channel (so one all note off to all 16 channels), it also sends PCs against channels I have activated. I use the midi hub for two channels specifically (these are synths that have sequencers in them)

In the above picture, this is how things are supposed to look. I have a mirror of Virt A to output C and a mirror of Virt B to output D for monitoring purposes. X-touch is “virt a” and USB Midi is “virt b” in the monitoring software. I load one of my songs and you see it sends the respective PC through virt A and those All ntoes off as I mentioned, I press another button to send PC value 9 to channel 10 - so the PC pass through is working just fine. Then I load a second song, this song actually doesn’t send any PCs when it loads so you just see the All Notes off stuff that my sequencer is sending - though my sequencer is only sending the Note Off with value 0, I believe the Note off 123 messages are erroneous created by the midi hub.

Now I’d say 1 in every three times I do this exact same procedure without changing anything else, I get output that looks like this. I color coded this so you could connect the two images. The Blue line is the same as the blue line in the first image I posted - There’s pretty much the same all notes off messages and PC messages sent. But the pink line, this is the error i’m talking about, in this case both of the pipelines sent erroneous PC messages - you can see that three PCs are sent to channel six with a value of 1 - then the all notes off, then you see three PCs sent to channel 10. And I guess in this case, one more PC sent to channel 6 with a value of 1 for good measure.

Again, sometimes the value of these PCs will be 1, and sometimes it will be 124. The value is always the same for the cluster of PCs (it will be a cluster of 3+ pcs value 1, or it will be a cluster of 3+ pcs value 124)

oi, i hope you folks don’t think i’m crazy. I went through a long and arduous process with my sequencer manufacturer and have been studying the PC outputs from my sequencer for a few weeks now - these aren’t PCs that my sequencer is sending, I’ve verified this like 50 times at this point - I’m certain that these are being created in the midihub either by my poor programming or due to a bug. And I speculate that this behavior might have something to do with the “All Notes Off” messages that are being sent to all 16 channels at once.

There’s a second symptom which may or may not be related in that, it looks like it’s duplicating the notes off messages to each channel with values of 123 or 0, but this is intermittent too. When I monitor my sequencer directly (no midihub), I get Note off values 0 every time, no matter how many times I do it - so there’s something going on in the midihub generating this too I believe.

I realize this is a really hard behavior to communicate and that there are many variables, but I really think it could very well be a bug. Intermittent bugs (heisenbugs as I like to call them) are the worst. That said, if it’s my midi pipelines I’m down to take full responsibility. I swear i’m not crazy, but this is really weird - any help I could get would be appreciated.

I have the 1.11.10 firmware installed and I originally found this behavior on 1.11.7

1 Like

Hey, thank you for reporting this. This kind of looks like some MIDI serialization issue, but could be something else.

Could you try and produce MIDI clips with the entire data that’s sent to Midihub inputs? This way we could play back the same clips into it and observe this behavior ourselves. You could do that by forwarding unmodified MIDI X to USB X ports in Midihub and recording all of that data on the host computer.

Another thought is, as this is all DIN-5 MIDI, there might be some physical issue, like bad contact of the ports, or a cable issue. I once hit an issue with a broken MIDI cable which was producing some very weird stuff, changing out the cable fixed it. :slight_smile:

Yay - thanks for responding. Regarding cables - I’ve definitely heard from folks what a bad din5 can do but stastically it’s less likely - I say this because I’m sending the same input to two ports and reproducing the issue on two identical (i hope) pipelines. So, it could be that both cables are bad (aren’t those situations THE WORST), but I don’t think so - fwiw of course.

Also, I got a really good piece of data today while trying to get you a repro – I can only produce this issue when I have the midi clock enabled. When I disable the clock I can’t reproduce the issue. This suggest that the issue might be relating to how much data is coming in at once - like some memory is overflowing or something of that nature. I promise not to try to overdiagnose this from my end.

A second data point that I got was - because I have the same data coming into two hardware ports I decided to unplug the second port and see if I could reproduce the issue with data coming into a single port operating a single pipeline, which I can.

Okay - so, I have a midi file that reproduces the issue in an extreme fashion - but I have to admit that this file isn’t exact to what my sequencer, I’ll explain why. First, the file:

repro_extreme.mid (1.5 KB)

The fundamental problem I faced was that I couldn’t find a program for OS X that could record either the clock or record midi CC 123. Midi CC 123 is ‘all note off’ and this CC apparently is special in every DAW on the planet. The program I normally use is logic pro - and while logic will record note data, Program change messages, and MIDI CC’s 0-122 and 124-127 it won’t record CC123 - I can’t for the life of me figure out how to record CC123 and I think i’ve been digging around the net for like 2 hours trying to figure this out.

Now you might recall that in my post that my sequencer sends a CC123 value 0 to all 16 midi channels every time you load a track. Just one big blast of them. For me to get a good midi file for you, I feel that this value is important - so I worked really hard at capturing specifically what my sequencer was outputting - but I just can’t. If you have some advise on how to do this on a Mac I am all ears - who knew CC123 would be my achilles heel.

Anyways - I refuse to give up so here’s what I did after I figured I was never going to be able to record this - I just recorded everything else via logic. As you suggested, I re-routed my sequencer unmodified midi stream to the USB, I disabled all other midi inputs in logic, I pressed record and then I just loaded songs over and over every few seconds. This reproduced the bug a few times - here’s the modified output (run through my pipelie on port A) when I was recording:

So we see those bursts of PC 1 as well as the CC123 (all note off) messages with value “123” – I learned something fun about CC123 which is it doesn’t actually care what the value is - but my sequencer only sends the value zero - so the value 123 here is an error.

Okay, then what I did was I wanted to see if I could replay my midi file from logic through the midihub to see if I could reproduce the bursts without my sequencer - no dice. I couldn’t make the error occur. This made me think that perhaps the burst of CC123 messages to all 16 channels was important in tracking this bug.

And this is where I things get weird - since I couldn’t record CC123 I used the event editor in logic and I programmed in a burst of CC123 to all 16 channels whenever it looked like I was doing a load. Since I would load a song every few seconds, I would just look for gaps between the PC messages being sent and then I would put down 16 CC123 val 0 events.

So every so often before the PC (and bank selection) messages, there’s a burst that looks like this:

This creates what looks like an extreme case for this bug. Instead of getting little bursts of 3 PC messages, I get bursts of 16 PC messages whenever these things send. Check this out:

Note the first ‘All Notes Off’ message, then there’s a burst of PC messages. Then you’ll notice there’s some kinda quiet notes off messages - so the bug didn’t quite trigger and them BOOOOOM - the midihub goes crazy and bursts a ton of these messages.

Now I think this behavior is different than my sequencer because my computer is sending the CC123 messages in a faster burst than my sequencer does. And, my hypothesis is, when all these messages come in they overwhelm the midihub somehow and then some of the clock pulses or CC 123 messages somehow become PC messages. It’s just a hypothesis based on my observation - so take it with a grain of salt.

Anyways - so to repro this, first you’ve gotta have the midi file, but you also need to make sure that whatever is playing the midi file is exporting a clock (for me this was set to 131bpm). Clearly my logic reproduction of this is different than I can do on my sequencer, but the midihub response was similar enough I felt that this could very well be tickling the same conditions i’m seeing on my sequencer.

Again - if there’s any way I could record CC123 I will totally do that. As an alternative, I can try to hand edit the events in logic to slow down the delivery of the CCs to mimic the sequencer but I sort of have no idea how they’re supposed to be timed - logic will force me to time them musically relative to bars/beats whereas I think my sequencer just sends them as fast as its little microcontroller can.

Holy crap this is hard to sort out. Anyways - I really appreciate you responding to me and I will do whatever I can to help figure this out - if there’s any way I can capture the CC123 more accurately I will jump on it.

Phew, lol, I think I need a brain break after that - this one is rough. Thanks again.

1 Like

I thought this might be helpful - so I did a quick capture of the load output from my sequencer - the main thing I was curious about was what the timing looked like for the burst CC123 messages

So these probably don’t send as fast as my computer - but they send a lot faster than the clock messages. The timing of these correlates more with the error PC output bursts than that of the clock (the clock pulses are further apart than these CC123 messages are). It seems that there’s correlation to these rapid fire CC123 messages and the issue.

okay, i promise i’ll stop now.

1 Like

Hmm, I’ve tried reproducing the issue using your preset and mid file, but I couldn’t get it to produce those bursts of messages, regardless if sent over MIDI cables or USB, with clock or without…

I think it’s a good plan to come up with a .mid clip that’d demonstrate the issue, even if it’s not 100% equivalent of what the sequencer sends.

Could you see how far can you simplify the preset on Midihub and keep getting the issue to occur? Like try bypassing sections of modifier pipes, to see how far the preset can be reduced.

Also, does the issue occur if messages are sent over USB only?

well - hrm. I found that when I just fired up the midi file again after not looking at it, I couldn’t get it to repro the issue. Of course, when I made it the issue seemed obvious. Now, when I hook up to my sequencer and it does what it does - sure enough, I can repro my issue. I need to test a litle more to verify - but it’s entirely possible something enters some kind of state and whatever it is that enters that state I’ve failed to capture in the midi file.

So that sucks - but maybe i’ll come up with a way to give you a repro case without having my hardware (which is the squarp pyramid).

I did do the exercise where you suggestet disabling elements of the pipeline to see if the issue reproduced. I disabled almost everything and I get the error with a fairly minimal set - maybe the main filter object is doing something weird?

Every time I see this error via my pyramid i always get chunks of 3 pcs sent with values 1 or 124. I have three filters, three bad messages? I know, i’m grasping at straws here and maybe trying to think about how I can help you get a good case. Well, anyway - i’ll attach the file of me reducing everything (maybe i’ll get lucky lol):

preset_bypass.mhp (612 Bytes)

The only other thing I can share is that I made the observation that when I added more filters it goes worse. I went through an exercise where I added another couple layers to the pipeline - the goal was to see if I could specifically filter out the two error messages. Since I always get values of 1 or 124 in a set of 3 pc messages, I created a filter for those values specifically and use some filters in those pipelines - the issue got worse for me. And naturally I don’t have that file to show you - but that’s probably not useful anyways. Maybe you know someone with a pyramid? :smiley:

I still need to give the usb a shot - i’m doing all this on din cables. Anyways - if nothing else, thanks for listening, i’ll keep trying to figure out a way around this.

1 Like

In the latest preset, what was connected to Midihub’s IN A and IN B ports? Which Midihub’s MIDI output sends the unexpected messages?

My sequencer is connected to a splitter (a banana split) which has two connection running to midihub - one running to Midihub port a, and one running to midihub port b (so the same signal, though I realize now I could probably just use one input port :-D). I’ve observed this behavior on all four din output ports (haven’t looked on usb output).

The banana split simply acts as parallel THRU ports?

I think we should try to reproduce the issue with a direct MIDI cable to a single port, just to keep reducing the variables and zero in on where the issue is hiding. :slight_smile: