Visualizing the First Two “Razumovsky” Quartets

Having spent this past Friday writing about how Quatuor Ébène approached the performing (and recording) of the first two of Ludwig van Beethoven’s Opus 59 (“Razumovsky”) quartets, I decided to revisit Stephen Malinowski’s YouTube playlist of his visualizations of the Beethoven quartets to take a look at how he approached both of these quartets. Since this involved material with which I was familiar through both recordings and concert performances, I was curious as to how his videos would have an impact after one set of recorded performances was still fresh in memory. Once again, the experience had assets and liabilities.

Perhaps highest among the assets is the extent to which these visualizations provide a representation of one way in which the marks on the score pages may be parsed. It is important to bear in mind that there is no reason why any composition by Beethoven (or any other composer) should be parsed in a single way. Indeed, from a grammatical point of view, performance itself is a means of expressing a particular approach to parsing. In that sense Malinowski’s video interpretations take a syntactic structure arising from a recorded performance and add a second layer of parsing in the visual domain to the one that performance has established in the auditory domain. If nothing else, this is a highly inventive approach to how, with the proper tools, one might be able to document a listening experience by working in the same time-based domain that provides the foundation for listening itself.

This suggests that each of these videos reflects a highly personal interpretation. This is not meant as a criticism. Rather, it suggests that it identifies strategies that others might consider when trying to account for their own respective listening experiences, not necessarily by creating their own visualizations but, rather, by coming to appreciate what those visualizations express and then finding other means to express them.

Nevertheless, there are certain aspects that concern me because they might distract when they should be indicating. From a personal point of view, I find the frequent use of blurred images to be an almost painful distraction, even though I realize that, from Malinowski’s point of view, it is simply one of several means to express the variety of different approaches to performance. On the other hand I appreciate that Malinowski’s approach to “scrolling through time” does not always provide all “future information.” Every now and then events simply “spring out of the void;” and, from a rhetorical point of view, they tend to have the same effect as that of certain abrupt introductions of thematic material in the score itself.

Still, from an information theoretic point of view, there is an almost overwhelming abundance of content in both of the quartets being visualized, meaning that every movement, even when taken on its own, serves up considerable “bandwidth,” which then requires equally considerable “processing effort” on the part of the viewer. I would postulate that such effort is far greater than what is engaged when one watches the members of a string quartet perform these pieces. My explanation would be that the visual experience of performance reveals cues about how the performers are taking their “parsing of the score” and realizing it as a listening experience. They are not providing a blueprint. They are providing their impressions of that blueprint, and it is through those impressions that those of us on the audience side come away with an absorbing listening experience.

Thus, to return to the issue of a “fatigue factor” that continues to occupy my appreciation of Malinowski’s visualizations, I would say that the bandwidth of a concert experience is more manageable that that of watching those visualizations, particularly if one wishes to experience a single quartet from start to finish (as it would be performed). Mind you, the same issue arises when one is trying to follow the score. There are just too many bits on the printed page for mind to keep up with the steady flow coming in through auditory sensation. Thus, when we follow a score during a performance, we tend to select where we look because we want to look for certain features for the sake of understanding how they are interpreted. One cannot be that selective when viewing one of Malinowski’s visualizations; and, to be fair, I do not think he expects his viewers to approach his work that way.

Still, I have to grant that any one of those visualizations almost always calls my attention to a variety of features that have always registered with listening but may not have been appreciated for the full worth of their contributions to the overall design. There is no doubt that, for any of the Beethoven quartets, there is always likely to be more than “meets the ear,” whether the ear is in the concert hall or listening to a recording. Malinowski consistently provides insights into what that “more” may be. Sometimes, however, those insights tend to overwhelm the viewer, tempting him/her to close his/her eyes and just get “back to the Beethoven.”

Further Thoughts about Fatigue and Multimedia

Having rambled my way around the question of whether or not visualization can compensate for the fatigue that tends to arise when listening to a long and challenging piece of music, such as the original version of Ludwig van Beethoven’s Opus 130 string quartet in B-flat major with the “Große Fuge” as its final movement, I realized that the recital by Calidore String Quartet that prompted these thoughts had a “punch line” that I had overlooked. Having felt as if I had been pushed to some limit of endurance by the “Große Fuge,” I discovered that my attention span was not going to get a break during the encore. The selection served up more Beethoven in the form of the second (Adagio ma non troppo) movement from the Opus 74 (“Harp”) quartet in E-flat major. My reaction at the time was to describe this selection as pushing too-much-of-a-good-thing beyond the limits of “reasonable endurance!”

To be fair, I have a rather unconventional history with this particular movement, which dates back to when I had been using a music data-entry system on an early Macintosh laptop. Unless I am mistaken, I was using an early generation of Lime™ software, whose manual demonstrated a notational “acid test” taken from that particular Opus 74 movement:

An example of challenging data entry from the second movement of Beethoven’s Opus 74 string quartet (from the Dover Publications reprint of the 1863 Breitkopf und Härtel edition, from IMSLP, public domain)

It is relatively easy to see that there is a lot going on in this excerpt, all of which could be handled in Lime™ by a patient user.

On the other hand, this is the sort of composition that poses as many challenges to the listener as to the performers. Even the most avid score-follower runs the risk of letting his/her eyes glaze over during this passage. As a result, I was very curious as to what Stephen Malinowski would present to the attentive listener in the visualization he created for this particular movement.

The good news is that he was spot-on in accounting for every note and aligning it properly with the accompanying recording of the Alexander String Quartet playing this movement. On the other hand my own subjective impression was that watching this single movement tended to impose significantly more fatigue than I had encountered in yesterday’s exercise with the “long version” of Opus 130. Was this a matter of my personal state of well-being; or was it actually a “content-related” issue?

I suspect the answer has much to do with what the attentive listener knows about this music and what (s)he expects of the experience of listening to it. Let’s not kid ourselves. Even the most attentive among us will probably confess to incidents of “tuning out” for some reason or another, perhaps for something as trivial as checking to make sure that the phone was turned off. We are all all-too-human; and mind can wander off on its own without our necessarily paying much heed to it.

However, I would conjecture that, when an auditory experience is reinforced by an animated visual one, attention is less inclined to wander, simply because so much of the “space” is now being “covered.” After all, even the slightest movement tends to draw attention; and, in accounting for the many subtleties in this one Adagio ma non troppo movement, attention is being not only prompted but also drawn to many different loci on the screen. In other words keeping track of what is happening on the screen is more demanding than watching the activities of a gathering of four musicians.

Mind you, every now and then we come across evidence that suggests that such a high level of attentiveness is not always necessary. There is an old joke that Igor Stravinsky claimed that the music of Franz Schubert would consistently put him to sleep. However, he then continued by observing that, when he awoke, he discovered he was in Heaven! The reader is free to decide whether this is nothing but a joke or a suggestion that attentive listening may involve some kind of subconscious substrate in the background that complements the wealth of detail one encounters in the foreground.

Another old joke is the observation, “Sometimes I sit and think, and other times I just sit.” Perhaps we all need to come to terms with the relationship between sitting and thinking. I have no idea how to begin to approach that relationship, but I am pretty confident that it is not a static one!

Attentive Listening and the Fatigue Factor

Every now and then I am reminded that I have been falling behind in keeping up with Stephen Malinowski and his innovative approach to creating animated visualizations of music that augment a recorded performance with a “video track.” On this site I wrote about his work most recently in July of 2016, when he had completed visualizations of the preludes and fugues in the first book of Johann Sebastian Bach’s The Well-Tempered Clavier. Malinowski has never shied away from major challenges, and my first encounter with his work took place back when I was writing for Examiner.com and I viewed the video he had created in honor of the 100th anniversary of the first performance of the ballet “The Rite of Spring,” for which Igor Stravinsky had provided the music.

The final measures of Beethoven’s “Große Fuge” (Artaria first edition, from IMSLP, public domain)

At the end of last month I learned of his latest “anniversary” project. In honor of next year’s celebration of Ludwig van Beethoven’s 250th birthday, Malinowski has created visualizations of all of that composer’s music for string quartet, working with recordings made by the Alexander String Quartet. The entire collection has its own playlist on YouTube.

Last month San Francisco Performances got the celebration off to an early start with a recital by the Calidore String Quartet in which Beethoven’s Opus 130 quartet in B-flat major was played in its original version, concluding with the “Große Fuge,” which would later be published separately as Opus 133 and replaced by a shorter concluding movement for Opus 130. As a result, Beethoven’s quartets are already on my mind; and I was curious as to how visualization might relate to the impressions of the Calidore performance I had documented.

Those who read my account of that performance know that I felt it necessary to address the problem of fatigue. From a mathematical point of view, the “information content” of the original version of Opus 130 is impressively high, quite possibly noticeably higher than any previously composed music. Whether or not one appreciates such a mathematical argument, the subjective consequence is that, when it comes to processing the auditory stimuli, mind has a lot on its plate. Indeed, there may well be more than most minds can handle, meaning that, before the performance has concluded, some (many?) of those minds have begun to “tune out” due to that fatigue factor.

At that particular recital, my own mind had already sustained quite a workout from the performance of of Joseph Haydn’s Hoboken III/32 quartet in C major, whose final movement is a fugue with four subjects (or, as H. C. Robbins Landon preferred to call it, a double fugue with two countersubjects). Haydn clearly believed in challenging both performers and listeners; and, since Beethoven’s experiences with Haydn as a pupil did not go particularly well, it is understandable that the pupil would have an ongoing need to one-up the teacher! Beethoven was still scratching that itch late in life; and the original version of Opus 130 remains one of the most imposing challenges in the repertoire for both performers and listeners.

I was thus curious about whether my own personal battle with fatigue would arise while viewing Malinowski’s visual interpretation of this music. My guess is that he may have wondered the same. In the playlist he created, Opus 133 follows the Cavatina (fifth) movement of Opus 130, as Beethoven had originally planned; and the “replacement” Finale is the next item. In other words listening to the published version of Opus 130 requires “manual intervention” between the fifth and sixth movements.

I should probably note, by way of context, that I have been following recordings with scores ever since I was in high school. Just about every music course I took as an undergraduate or a graduate student involve score-following in one context or another. Indeed, much later in life, when I attended a pre-concert talk that Steve Reich gave at the University of California, Los Angeles about his “New York Counterpoint,” he concluded by playing a recording while setting out the score for those of us in the audience to follow. It was only after I began to write about music on this site that I realized that paying too much attention to marks on score pages might distract from the experience of listening itself, and these days I only consult scores after a performance has taken place.

To some extent watching a Malinowski visualization is an alternative to score following. He clearly understands the information content of the score; but, on the basis of my past experiences with his videos, I would say that his visualizations have more to do with auditory impressions than with the marks on paper beyond the performance itself. On the other hand, the score is not ignored entirely. Thus, when the instrumental voices cross over each other in Beethoven’s score, one sees that crossing in the unfolding images. Similarly, he introduces a “splash” image to represent notated pizzicato playing.

Nevertheless, the overall images themselves and the ways in which those images unfold tend to have more to do with auditory impressions, rather than with the marks on paper behind the performance itself. Thus, the visualization reflects rhetorical interpretations that escalate the performance above merely getting all the notes right. It is through that attention to rhetoric that one appreciates a view of “music being made,” rather than “notes being interpreted.” One might think that this would lead to greater “information overload” than what comes with listening in a concert setting. However, as one settles into the “image vocabulary,” that risk of mind having to process too much seems to have been diminished.

This may be due, in part, to the variation of the “image vocabulary” as one advances through the movements. Thus, in my own situation, I no longer faced fatigue during the Cavatina. Indeed, I was more immersed in its uniquely sensuous rhetoric than I usually am when listening to this music in recital or on recording. Mind you, I do not think that Malinowski was deliberately going after sensuous images; but the visualization brought to the foreground the devices through which the subjective nature of that music flourished. (On the other hand I have to confess to some disagreement over foreground-background decisions; but those involve picking nits that would distract from the significance of the visualization technique itself.)

My conclusion (at least where this repertoire is concerned) is that, when there is more coming through the “auditory channel” than mind can handle, adding another medium to the mix may be beneficial!

Tim Thompson and his Space Palette Pro

Last night at the Community Music Center, Outsound Presents launched the concert portion of its seventeenth annual New Music Summit, a series of five evening events that will continue throughout this week. The title of the program was Sonic Foundry III: a night of unique invention, because it was the third in an ongoing series providing a showcase for invented instruments. The first set was taken by Tim Thompson, who has brought earlier incarnations of his equipment to the annual Touch the Gear expo that Outsound Presents holds on the Sunday before the Summit concerts begin.

For his concert performance he brought his latest creation, which he calls the Space Palette Pro. Like a palette, it is a flat horizontal surface, which Thompson controls with both hands. There is also a moderately small chromatic keyboard at the front and a display that seems primarily to show how the components of the instrument are configured.

The primary components are four Sensei Morphs arranged roughly as the corners of a square. A Sensei Morph is a rectangular touch-sensitive pad. It can sense multiple points of contact, as well as the pressure applied at each of those points. The Space Palette Pro basically translates input from the Sensei Morphs (and, occasionally, the keyboard) into both audio and video signals, the latter being projected on a large audience-friendly screen.

Ever since my first encounter with Joel Davel’s Marimba Lumina (which he plays in his performances with the Paul Dresher Ensemble), I have been sensitive to the distinction between an “instrument” and a “controller.” Clearly, all musical instruments “control” the creation of sound by virtue of how the performer interacts with an “interface.” However, a controller is basically a source of signals that is independent of the device that takes those signals as input.

Thus, in my personal lexicon I would prefer to call the Space Palette Pro a controller, rather than an instrument. However, this is far from an attempt to demean Thompson’s work. Indeed, to the contrary, it is the only controller I have encountered in which frequently, if not always, the same signals are passed to both audio and video devices. As a result performance consistently results in a smooth coordination between what you hear and what you see, very much in the same vein as the polished visualizations that Stephen Malinowski creates for recordings of classical music.

This is not to suggest that Thompson’s images follow the same sort of note-by-note account that one finds in Malinowski’s videos. Indeed, the very concept of “note” feels somewhat outmoded when one attends to the experience of both listening to and viewing a Thompson performance. I was particularly struck by how Thomson could convert his signals into images that unfold as cascades of a prodigious variety of symmetries. (The mathematical concept of symmetry involves far more than “mirror images.” Indeed, during the Q&A after the performance, I could not resist asking if Thompson had drawn on the systematic classification of symmetry types based on group theory and discussed in Hermann Weyl’s Symmetry monograph. They were not, which led me to admire all the more the extent of Thompson’s visual imagination!)

The performance itself clearly went through a series of episodes, each of which involved different interpretations of the signals from the Sensei Morphs. Thompson’s intuitions about the durations of those episodes could not have been more acute. No episode ever overstayed its welcome. Indeed, my only disappointment was that the performance, as a whole, never really managed to register a “sense of an ending.” At a certain point the flow of control signals ceased, and that was that. Mind you, there are any number of compositions that have this same trait. However, because there was always a clear sense of tonality on the audio side of his signal processing, it was hard to resist that Schenkerian urge to listen for an overall tonic-dominant-tonic cadence.

Nevertheless, Thompson is clearly playing by his own rules, rather than Schenker’s. Those rules are not just about listening but about the integration of listening and viewing. It would not surprise me if those rules go through further refinement as Thompson continues to explore approaches to generating signals that are interpreted for both listening and viewing. Hopefully, there will be future opportunities to observe how his efforts progress.

Reproducing Music Without a Microphone

I suppose that my background in technology has a lot to do with my fascination with player pianos. Back when I was teaching at the University of Pennsylvania in the Seventies, I knew someone who was passionate about restoring old player pianos; and it was from him that I first learned that, in the most advanced models, there could be up to five additional columns for holes that would control the amplitude. The makers of player pianos understood the digital-to-analog converter long before the electrical circuit for such a device had been built!

Nevertheless, there is a downside to that story. It involves the promotion of the Welte-Mignon, a reproducing piano that was first introduced in 1904. To promote the instrument, members of the Welte family sought out as many of the best pianists of that period as they could muster, inviting them all to allow their performances to be captured by their technology. The reason I can remember that there were five additional columns comes from a story I heard about what happened when Artur Schnabel was approached. Whichever Welte was making the pitch to him boasted that their system could capture 32 levels of loudness. Schnabel replied curtly, “I have 33;” and refused to make any recordings for the instrument!

It this respect it is interesting that MIDI representations are based on eight-bit “words,” meaning that, unless extended by different interpretation software, the representation of amplitude has 256 levels (including silence). I was actually reluctant to doing anything with MIDI when it first appeared. This was not because of Schnabel but because I was interested in microtonal tuning at the time and had not yet encountered a representation of pitch that would satisfy the representation I had developed for my own EUTERPE language, which divided the octave into 72 equal parts.

However, when I moved to Singapore in 1991, I knew that I was not going to bring along my Baldwin grand. So I visited a Yamaha showroom and got myself a Clavinova with sampled sounds and support (sort-of) for three functioning pedals. (“Sort-of” meant that the “virtual dampers” controlled how long a note would be sustained but not how it would cause other open strings to reverberate.) Because this instrument supported MIDI control, it was not long before I bought a Macintosh, perched it on the instrument, and started playing around with MIDI software.

In Singapore Yamaha ran a music school in conjunction will selling instruments. I got to know one of the piano teachers, and she started asking me questions about MIDI. I happened to have one piece of software that would provide numerical representation of MIDI content in terms of time (both start and end), piano key, and degree of loudness. I showed her a printout of my own performance of Wolfgang Amadeus Mozart’s K. 2 minuet (which has only 24 measures). Once I explained what the numbers meant, she immediately started to critique my performance; and she could explain what she was saying in terms of the “raw data” of the numbers themselves.

Thus began one of the most interesting projects from my time in Singapore. Two colleagues and I developed a music-based visualization of MIDI recordings. This meant that the display was structured around score notation constructs. Amplitude was represented by coloring the note heads on a color scale that would “dissolve” from high-intensity red (loudest) to high-intensity blue (softest). We also developed a “virtual metronome” that would insert tick marks corresponding to the position of the beat with respect to the notation. Finally, there was a representation of “articulation,” basically how much of a pause there was from one note to the next.

Such visualization provided a new “channel” through which teacher could communicate with student. The object may have been to get the student to be a better listener, but establishing what the student should pay attention to while listening could be tricky. Visual cues, even relatively primitive ones, could go a long way towards guiding auditory attention.

Having established a relationship with Yamaha, I could now devote some of my attention to some of their own efforts to promote reproducing pianos, such as the Disklavier, which was basically a grand piano with the ability to translate MIDI data into physical hammer and pedal actions. Yamaha was trying to market a library of floppy discs recording performances by famous classical and jazz pianists. (As I recall there was even a floppy with MIDI transcriptions of player piano rolls recorded by George Gershwin.)

Needless to say, these sounded pretty feeble on my Clavinova. However, I discovered that the effect was not much better when I listened to the same recorded performance on a Disklavier in the Yamaha showroom. Was this just another example of Schnabel’s complaint that there were “not enough bits?”

My current thinking is that this was not the case. I thought about some of the vinyl recordings I had of Welte-Mignon performances; and I recalled that the audio recordings were based on using the instrument on which the piano roll had initially be created. This led to the hypothesis that reproduction was as much a matter of what the pianist heard as of what the pianist did physically. Our research in visualization had offered a representation of audible features; but there was a listening-playing loop that would be unique to any performance, regardless of how it was visualized.

This hypothesis has two serious corollaries. One involves “scientific” approaches to researching the nature of piano performance. One can take recordings of pianists and abstract them to a MIDI-like representation. That representation will provide a representation of what the pianist was doing that would be at least adequate. However, when isolated from the sounds of the recording, it abstracts out any information about how the pianist is responding to the sounds (s)he is creating. This seems to be a critical gap in the data that are available. It also suggests that, when a pianist is warming up on an unfamiliar instrument (s)he is aware of not only the physical responsiveness of the keys and pedals but also of how the whole instrument reverberates and how some frequencies may be stronger than others.

The second corollary goes back to Yamaha’s interest in marketing those floppy discs. Because every piano has its own unique sonorities, playing one of those floppies on a piano that is different from the one on which it is recorded may well have unsatisfying results. In other words the abstraction of the data on the floppy, which is probably grounded in MIDI, cannot capture the impact of listening on the nature of the performance.

Can the capture technology be improved? This would require a richer understanding of the nature of that listening-playing loop. At the present time we can do little more than come up with some highly general hypotheses. However, this is one of those cases in which the devil is in the details; and we still have quite a ways to go before we even know which of those details are the relevant ones!

Your Brain on Music is a Highly Distributed Processor

BBC Science Reporter Victoria Gill in currently in San Diego covering the annual meeting of the American Association for the Advancement of Science. After yesterday's proceedings she filed a story that almost immediately grabbed my attention. Here is the opening summary:

Teaching stroke patients to sing "rewires" their brains, helping them recover their speech, say scientists.

By singing, patients use a different area of the brain from the area involved in speech.

If a person's "speech centre" is damaged by a stroke, they can learn to use their "singing centre" instead.

The paper was delivered by Gottfried Schlaug, Professor of Neurology at Harvard Medical School, practicing at Beth Israel Deaconess Medical Center. The research has been based on a therapeutic technique for stroke patients that has been used for some time. It is based on the premise that the brain has separate areas for processing speech and music. The conclusion of the above excerpt reasons that, if the patient cannot use language through speaking, linguistic performance may still be exercised through singing.

Not only is this principle familiar, it has led to any number of jokes. Had Figaro decided to ask Susanna about the price of the ribbons on her new little hat, one might have ended up with an elaborate duet (of a sort that only Wolfgang Amadeus Mozart could have composed) debating the equally elaborate fine points of the cost of living in seventeenth-century Seville! More seriously, Schlaug's contribution has escalated the study of the stroke-afflicted brain from a successful therapeutic technique to new results in brain imaging. His claim is that he can show "what is actually going on in the brain" (his words) as patients learn to sing their words rather than speak them.

More interesting for me, however, was a reaction Gill reported from another research community:

Dr Aniruddh Patel from the Neurosciences Institute in San Diego, said the study was an example of the "explosion in research into music and the brain" over the last decade.

"People sometimes ask where in the brain music is processed and the answer is everywhere above the neck," said Dr Patel.

"Music engages huge swathes of the brain - it's not just lighting up a spot in the auditory cortex."

Again, there has been talk about the brain as a distributed processor for some time. This inspired the whole connectionist approach to artificial intelligence that was so popular 25 years ago and probably still has its allotment of die-hard supporters. Connectionism never progressed very far beyond serving as a new approach to applying optimization to pattern classification, which is only a small part of what probably occupies "your brain on music" (to appropriate the turn of phrase by Daniel J. Levitin). The practice of music is a far more complex behavior, and researchers like Patel may be leading us into intricacies of brain function that could be only vaguely hinted in Levitin's exposition.

Needless to say, we are still quite some distance from any understanding of music behavior that accounts for the objective, subjective, and social worlds in any remotely comprehensive way. The good news from Professor Schlaug is the imaging technology keeps getting better; and, each time it improves, researchers come up with new ways to leverage those improvements. As we proceed down this path, there will be less talk about the "musical brain" at cocktail parties and more talk at scientific conferences!

Visualizing Statistics

Seattle photographer Chris Jordan has an interesting project. He calls it Running the Numbers, and it is a unique and striking exercise in visualizing some of the more extreme statistics that define American life. Here is his own description from his Web site:

This new series looks at contemporary American culture through the austere lens of statistics. Each image portrays a specific quantity of something: fifteen million sheets of office paper (five minutes of paper use); 106,000 aluminum cans (thirty seconds of can consumption) and so on. My hope is that images representing these quantities might have a different effect than the raw numbers alone, such as we find daily in articles and books. Statistics can feel abstract and anesthetizing, making it difficult to connect with and make meaning of 3.6 million SUV sales in one year, for example, or 2.3 million Americans in prison, or 426,000 cell phones retired every day. This project visually examines these vast and bizarre measures of our society, in large intricately detailed prints assembled from thousands of smaller photographs.

Given the scale of the results, the process of assembly must require an inordinate amount of time (and therefore patience); but the results certainly justify the dedication and effort. There are quite a few of those results on the Web page, which means that it takes some time for the page to load. Also, Jordan emphasizes that these images have their greatest impact when viewed at their original scale. However, since this currently entails a trip to Seattle, I am glad that he has provided an opportunity to get an initial feel for what he has done. My favorite is what he did with those 106,000 aluminum cans: He basically adapted the pointillist style of Seurat's "Grande Jatte," reproducing Seurat's canvas by using the colors of the cans as his "points." Since I never really appreciated the original until I was standing in front of it at the Art Institute of Chicago, this is the image that I most want to see at scale; and, alas, no trip to Seattle is currently in the cards!