🎨 Help designing version 2!

[rdoursenaud]

Hey @mido community 😄

I've started preliminary design for version 2 :
✏️ https://github.com/mido/mido/wiki/Version-2-Design

📝 Let me know what you think and feel free to suggest architectural or programming interface changes no matter how small or large!

🚀 I'd like to have a stable target early October to dedicate the last quarter of 2023 to implementation and optimistically release in quarter 1 of 2024.

👷 Of course, mido being an Open Source project, all volunteers are welcome to chime in.

Thanks,
with ❤️
Raphaël

P.S.: While you're here, don't forget to vote for your favorite backend!

[rdoursenaud]

Relevant comments:

• Defer computing merged_track #565 (comment)
• Faster merge tracks #474 (comment)
• Enable a bunch more Ruff categories #566 (comment)

[akx]

Some random thoughts:

• That's some thorough documentation! 👍
• I think it might be worth it to codegen the various MIDI event/message classes, and their (de)serialization and base their design on e.g. frozen dataclasses (though they don't necessarily need to be actual dataclasses if they're codegen'd).
  • For backward compatibility (if desired), mido.Message could remain a factory function.
• As discussed in Faster merge tracks #474, per-object validation should probably be very opt-in.
• Similarly, it should be possible to leniently parse streams when possible; there's a lot of broken MIDI out there which is still worth being able to read.
• The validation APIs should probably allow for validating an entire MidiFile with an error generator pattern (i.e. .iter_errors(), and a top-level .validate() that could e.g. raise an ExceptionGroup (when available, Python 3.11+)).

[rdoursenaud]

1. Thanks! It was sorely needed to allow myself and hopefully others to wrap my (their) head(s) around the unexpected complexity.
2. Since there's a definite number of messages properly defined in the specifications, I don't really see the benefit of codegen versus dedicated classes and a catch all for undefined messages. The serdes of MIDI is not straightforward (lots of corner cases and special treatments: Running status, Realtime priority, SysEx, aborts…) and requires fine tuning that is better served by dedicated processes IMHO.
3. Dully noted!
4. Agreed. I intent to raise warnings when appropriate to keep things explicit and only error out when unable to proceed further.
5. Great suggestion. I'm not familiar with this pattern. Some research needed on my part.

[akx]

Regarding codegen – fair enough, but care should anyway be taken to ensure the objects will be small and immutable (probably __slots__ worthy!)

By the way, I could gather up a collection of various suitably (CC or similar) licensed MIDI files that could be used as a performance/smoke test suite – would that be useful?

[rdoursenaud]

Hey, sorry for the late reply and thanks for the precious insights.

We're currently using the Lakh MIDI Dataset informally. I'll probably use it more formally for testing and validating version 2's implementation.

[akx]

Ah, good to know someone's done that data work already (and of course it'd be Colin...).

It would probably be worth it to mirror e.g. the clean_midi dataset on GitHub (e.g. as a GitHub release in a new mido/datasets project) instead of downloading it from hog.ee.columbia.edu all the time (especially for CI purposes). At that point, it's probably also best to re-compress it with Zstandard; one can squeeze about 40 megabytes from the package size, which would probably add up over time! 😅

($ zstd -vv -T0 --ultra -20 clean_midi.tar)

-rw-r--r--@ 1 akx  staff  829941760 Aug  9  2016 clean_midi.tar
-rw-r--r--@ 1 akx  staff  234283029 Aug  9  2016 clean_midi.tar.gz
-rw-r--r--@ 1 akx  staff  195512425 Aug  9  2016 clean_midi.tar.zst