Re: [Lilypond-auto] Issue 4042 in lilypond: Patch: Support for controlling MIDI expression

[lilypond]

Comment #1 on issue 4042 by address@hidden: Patch: Support for  
controlling MIDI expression
http://code.google.com/p/lilypond/issues/detail?id=4042

This patch adds support for controlling the "expression level" of MIDI
channels using the "Staff.midiExpression" context property.  The property
accepts a number value between 0.0 and 1.0.

The patch itself is quite straightforward: it simply relies on the
infrastructure introduced in issue 3581 to translate changes to the new
context property into MIDI Control Change events in the outputted MIDI.

The main purpose of this patch is to add to LilyPond (the technical)
possibility of adjusting the perceived "volume" of even already sounding
notes on a MIDI channel, to allow creating effects (for example, crescendos
or decrescendos on long tied notes) which cannot be realized with the
current implementation, which assigns notes "velocities" that apply only at
the start of a note.

----

Proof-of-concept example:

\version "2.19.11"

adjustExpression=
#(define-music-function (parser location initialvalue finalvalue music)
                        (number? number? ly:music?)
  ;; Helper function to emit a sequence of changes in the midiExpression
  ;; context property from "initialvalue" to "finalvalue" during "music".
  (letrec ((adj (lambda (unit numsteps silence i)
                  (if (>= i numsteps)
                      '()
                      (cons
                        #{
                          \set midiExpression = #(+ initialvalue (* i unit))
                          #silence
                        #}
                        (adj unit numsteps silence (+ i 1)))))))
    (let* ((duration (ly:music-length music))
           (step (ly:make-moment 1 64))
           (silence (ly:music-compress #{ s1 #} step))
           (numsteps (floor (ly:moment-main (ly:moment-div duration step))))
           (unit (/ (- finalvalue initialvalue) numsteps)))
      (make-simultaneous-music
        (list music (make-sequential-music (adj unit numsteps silence  
0)))))))

mus = <<
  {
    a'1~\p -\tag #'layout \<
    a'1~
    a'1~ -\tag #'layout \>
    a'1

    a'2 -\tag #'layout \sf
    r2
  }
  {
    \adjustExpression #0.3 #0.8 { s1*2 }
    \adjustExpression #0.8 #0.3 { s1*2 }

    \adjustExpression #0.3 #1.0 { s8 }
    \adjustExpression #1.0 #0.3 { s4 }
    s8
  }


\score {
  \new Staff \keepWithTag #'layout \mus
  \layout { }
}

\score {
  \new Staff \with {
    midiExpression = #0.3
    midiInstrument = #"clarinet"
  } \keepWithTag #'midi \mus
  \midi { }
}

----

Warning: The possibilities opened by this patch (and, due to the
lack of more abstract interfaces for using the new functionality, the
resulting difficulty of understanding its implications on the
handling of "MIDI volume" in LilyPond without technical knowledge about
the implementation) may raise serious questions about whether it makes
sense to provide the low-level interface exposed by this patch for all users
in its current form.

While it could be convenient for advanced users to expose more MIDI
controls in LilyPond (so that a single LilyPond source file could be used
to generate both layout and MIDI, with even all the MIDI effects
"implemented" in the source file using LilyPond syntax), there's likely to
be a limit to how far this should go - after all, most people likely use
LilyPond as a music typesetting engine, and there exist plenty of other
tools for MIDI editing.

As is already quite apparent from the example (due to the use of a helper
function, a separate \score block to generate the MIDI, and tags to
prevent the standard dynamic performer from touching the velocities of
notes), the main "flaw" of exposing the Expression control this way is
the lack of integration with the dynamic performer.  Furthermore, without
careful manual effort to make sure that the combined effect of note
velocities and MIDI expression on the perceived MIDI volume remains
consistent (see the example below), careless adjustment of the MIDI
expression alone can lead to unexpected results with the perceived MIDI
volume.

    From what I've learned by searching information about MIDI, my
    current understanding is that the actual volume of a note is formed
    first as a combination of the following MIDI channel controls:

    1. The "MIDI volume" (CC#7) assigned to the note's MIDI channel.  In
       LilyPond, this control is not adjustable by the user - nevertheless,
       for reasons that I haven't tried to track down, LilyPond is quite
       aggressive in emitting CC#7 messages on dynamic changes, repeatedly
       setting the volume of a MIDI channel to 100 (in the range 0-127
       allowed for CC#7) on dynamic changes in the input file.

    2. The "MIDI expression" (CC#11) assigned to the note's MIDI
       channel (0-127).

    The General MIDI System Developer guidelines
    (http://www.midi.org/techspecs/gmguide2.pdf, pp. 9-10) suggest that
    the values of these controls should be multiplied (by a synthesizer
    interpreting a MIDI file) together so that the actual volume of a
    MIDI channel is given by the equation
        (CC#7 * CC#11) / (127 * 127)
    (as a fraction of the channel's maximum available volume).

    The question about perceived MIDI volume is made more complicated by
    the fact that also the "velocity" assigned to an individual note may
    be used to adjust the note's perceived volume, although, to my
    current understanding, the MIDI standard leaves open how the
    velocity, or, in other words, the "level of force" in which a note is
    to be played, ought to be interpreted.  Even though the "level of
    force" in which a note is played on a real instrument likely has
    some correlation to the note's "volume" (making the direct
    "velocity-as-volume" interpretation a possible choice for a MIDI
    synthesizer), there are also other possibilities that could be
    imagined, such as using separate sound samples for notes played with
    different velocities (or velocity ranges).

    As LilyPond's dynamic performer translates dynamics directly into
    note velocities, LilyPond seems to assume this direct
    "velocity-as-volume" interpretation.  However, since the velocity of
    a note that is still playing cannot be changed after its initial
    "note-on" MIDI event (at least I've failed in my searches to find a
    way how it could be done in MIDI), the only way to alter the volume of
    already sounding notes without generating new "note-on" events is to
    use the channel controls for this purpose.  (The General MIDI System
    Developer guidelines document even recommends using the MIDI
    expression control for creating crescendo/descrendo effects.)

    (Of course, the reason why "volume" in LilyPond is implemented using
    note velocity and not the MIDI channel controls is probably the fact
    that, unlike the MIDI controls, which always apply to all notes
    playing at the same time on a MIDI channel, the velocity of a note
    can be controlled independently of all other notes even within a
    channel, which is certainly more flexible.)

    However, even under the direct "velocity-as-volume" interpretation, it
    is likely again up to a synthesizer implementation to decide how the
    velocity of a note is combined with the values of CC#7 and CC#11 to
    obtain the note's actual volume.  A straightforward method would be
    to use the velocity just as an additional factor in the calculation of
    the volume level, and compute the volume level of each note using the
    equation (CC#7 * CC#11 * velocity) / (127 * 127 * 127).

Assuming that CC#7, CC#11 and note velocity all have equal significance
(so the last equation above holds), simulating a crescendo effect from \pp
to \ff on a tied note using the expression control could then be realized
as

----

\version "2.19.11"

adjustExpression = ... % from the previous example

mus = {
  \set midiExpression = #1.0

  a'1\pp  % note volume: (100/127 [MIDI channel volume]
          %               * 1.0   [expression]
          %               * 0.49  [velocity for \pp, see scm/midi.scm])
          %              of maximum volume

  \adjustExpression #0.49    % velocity for \pp, scm/midi.scm
                    #0.80    % velocity for \ff, scm/midi.scm
                    {
                      a'1~\sf % set the note velocity to the maximum value  
1.0;
                              % since expression starts to increase from  
0.49,
                              % the actual initial volume of this note  
should
                              % be the same as the previous note's volume
                      a'1
                    }

  \set midiExpression = #1.0
  a'1\ff  % note volume: (100/127 [MIDI channel volume]
          %               * 1.0   [expression]
          %               * 0.80  [velocity for \ff])
          %              of maximum volume
}

\score {
  \new Staff \with {
    midiMinimumVolume = #0.0   % use the full available volume range
    midiMaximumVolume = #1.0   % to make velocity calculation simpler
    midiInstrument = #"clarinet"
  } \mus
  \midi { }
}

----

Needless to say, applying custom equalization to MIDI instruments will add
another scaling factor which should be taken into account when calculating
the absolute velocity values to which \pp, \ff etc. correspond, to make the
transitions in volume "smoother" at the \adjustExpression endpoints (which
still assumes a lot about how a MIDI synthesizer combines note velocity with
the MIDI controls into a volume level).



In summary, this patch (and, in fact, already the performer for trapping
certain SetProperty events added in issue 3581) represent my experiments
at trying to extend LilyPond with the functionality to adjust the volume
of already sounding notes (to support implementing certain new effects such
as crescendos or decrescendos on tied notes directly in the LilyPond
input).  However, from the examples it is clear that, due to the
"interference" of the new control with the way LilyPond traditionally
handles volume, the minimal interface added by this patch (which only
exposes the MIDI control) leaves probably too much to be desired in order
for the interface to be accessible (or easily documentable for) the average
user in its current form.  Then again, such a user may not have any need
for this functionality; also, the new MIDI context property is a strict
extension of the current feature set and should have no implications on the
handling of any source files which do not use it.  Maybe the usability
could be improved with better high-level Scheme helper functions; this
patch only makes a first step in adding low-level support for controlling
the expression level.


Regards,
Heikki Tauriainen


--
You received this message because this project is configured to send all  
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings