Harmony Space

Harmony Space is an interactive digital music system designed to give beginners and experts insight into a wide range of musical tasks ranging from performance and analysis to composition. There are various version of Harmony Space.

  • A Desktop version that runs on Macintosh, Windows and Unix desktops & laptops.
  • A Whole body version that works work with camera tracking and floor projection.
  • The Song Walker version – with multiple dance mats, wii remotes, and a large projection screen.

Because Harmony Space links fundamental concepts in tonal harmony to simple spatial concepts, it enables beginners to carry out a wide range of relatively complex musical activities relatively quickly, playing, composing and analysing harmony, while at the same time affording new insights to expert musicians.

What is Harmony Space?

Based on the Longuet-Higgins (1962) and Balzano (1980) theories of harmonic perception, Harmony Space (Holland 1994) is designed to teach novices principles of tonal harmony and harmonic progression, by exploiting intuitive prior knowledge of, for example, straight-line movement, proximity judgments and visual grouping. This reduces the need for prior specialized domain knowledge of musical concepts when playing, composing and analysing harmony.

Publications

  • Holland,S., Wilkie,K., Bouwer, A., Dalgleish.M and Mulholland,P. (2011) Whole Body Interaction in Abstract Domains, In England, D. (Ed.) Whole Body Interaction. Human–Computer Interaction Series, Springer Verlag, London. ISBN 978-0-85729-432-6. Pre-Print WholeBodyInteractionInAbstractDomains.pdf.
  • Holland, S. (2010) Asymmetrical Multi-User Co-operative Whole Body Interaction in Abstract Domains. Paper presented at Workshop on Whole Body Interaction, CHI 2010, Atlanta. Holland Asymmetrical WholeBody CHI2010.pdf
  • Holland, S., Marshall, P., Bird, J., Dalton, S. N., Morris, R., Pantidi, N., Rogers, Y. and Clark, A. (2009) Running up Blueberry Hill: Prototyping Whole Body Interaction in Harmony Space. TEI 2009. Proceedings of the Third Conference on Tangible and Embodied Interaction. ISBN 978-1-60558-493-5 pp 92-98. ACM New York. Oro ID 17966. Pre-Print BlueberryHillWholeBodyInteraction.pdf
  • Wilkie, K., Holland, S. and Mulholland, P. (2009) Evaluating Musical Software Using Conceptual Metaphors. In Proceedings of BCS Human Computer Interaction, Ed. Blackwell, A. 2009 pp 232-237. EWIC, ISSN 1477-9358. Oro ID 17967. BCS HCI 2009 Wilkie Evaluating musical software.pdf
  • Holland, S., Marshall, P. and Bird, J. (2009) Prototyping Whole Body Navigation of Harmony Space. Paper presented at Workshop on Whole Body Interaction, CHI 2009, Boston. Holland CHI2009.pdf
  • Bird, J., Holland, S., Marshall, P., Rogers, Y. and Clark, A. (2008) Feel the Force: Using Tactile Technologies to Investigate the Extended Mind. In Proceedings of Devices that Alter Perception Workshop (DAP 08), pp. 1-4 as part of Ubicomp 2008, Seoul, Korea (winner of best paper award). pdf here.
  • Holland, S. (2000) Artificial Intelligence in Music Education: a critical review. In Miranda, E. (ed.) Readings in Music and Artificial Intelligence, Contemporary Music Studies Vol. 20. Pages 239-274, ISBN 90-5755-094-6, ISSN 0891-5415. Harwood Academic Publishers, Amsterdam. Artificial Intelligence MusicEd.pdf.
  • Howard, P., Holland, S. and Whitelock, D. (1994) Keyboard Harmony: some applications of computers in music education. In Musical Times, ISSN 0027 46666 Volume CXXXV No. 1817 pages 467- 471. Computers in Music Education.pdf
  • Holland, S. (1994) Learning about harmony with Harmony Space: an overview. In Smith, M. and Wiggins, G. (Eds.) Music Education: An Artificial Intelligence Approach. Springer Verlag, London.
  • Holland, S. (1992) Interface Design for Empowerment: A case study from music in Holland, S., and Edwards, A. (1992) (Eds) Multimedia Interface Design in Education. Springer Verlag, Hiedelberg.
  • Holland, S. (1987) Direct Manipulation tools for novices based on new cognitive theories of harmony. pp 182 -189 Proceedings of 1987 International Computer Music Conference.
  • Holland, S. (1986) Design considerations for a human computer interface using 12-tone three-dimensional Harmony space to aid novices to learn aspects of harmony and composition. OU IET CITE report no. 7, August 1986.

Reviews

  • Camilleri, Professor Lelio, (1987). Letture Critiche. Three page critical review of Harmony Space In BeQuadro, April.
  • Durham, Tony, 1987. “Musician and Machine in perfect Harmony?”. 2 page review, in ‘Computing’, February 26 1987.

Analysing Songs

Harmony Space links fundamental concepts in tonal harmony to simple spatial concepts. Consequently it can be used to visually explore analyses that faithfully represent harmonic concepts. At the same time,

When combined with whole-body interaction in the Song Walker version of Harmony Space, beginners are able to enact complex harmonic phenomena in physically simple ways. This approach affords rapid, concrete means of reasoning about and manipulating abstract musical entities and relationships by exploiting intuitions associated with bodily movement and navigation.

Analyses can sound abstract on paper, but are quickly grasped when played actively using Harmony Space, given only examples of a very small number of cultural conventions. Beginners typically get the hang of playing simple songs within seconds, and many start to gain deeper insights within minutes.

Example Analyses

Isn’t She Lovely – Stevie Wonder
Pachelbel’s Canon
Pachelbel’s Cannon – more analytical view
Heart of Gold Neil Young
Sultans of Swing Dire Straits
Suffragette City David Bowie
Honey Pie The Beatles
Scarborough Fair Traditional
All the things you are – Jerome Kern
Giant Steps John Coltrane
Ticket to Ride The Beatles
Hey Jude
Miles Davis Decoy – quartal progressions
Mr Blue Sky – an ingenious harmonic dorian to major modulation
Spirited Away- Quartal Textures
Charlie Parker Swedish Blues

Song Walker

Harmony Space is an interactive digital music representation system designed to give beginners and experts insight into a wide range of musical tasks ranging from performance and analysis to composition.

When combined with whole-body interaction to create the Song Walker version (Holland et al., 2009), Harmony Space allows users to enact complex harmonic phenomena physically via mappings between a) bodily movement and b) conceptual metaphors and blends for musical abstractions. This approach affords rapid, concrete means of reasoning about and manipulating abstract musical entities and relationships by exploiting intuitions associated with bodily movement and navigation.

Song Walker employs four dance mats, four wireless controllers (wii remotes and nunchucks), various foot pedals, a large projection screen and a synthesizer. When operated by a solo player, one dance mat is used to navigate around in the projected space. Squeezing the trigger on the wii remote visibly and audibly plays the chord associated with the current location and situation. When operated by multiple players, additional dance mats may be used simultaneously in a variety of collaborative ways, for example, to navigate key changes or to harmonically invert or otherwise alter the chords. Single players , or groups, are typically able to play, and start to gain insight into, relatively complex music within a matter of minutes.

  • Holland,S., Wilkie,K., Bouwer, A., Dalgleish.M and Mulholland,P. (2011) Whole Body Interaction in Abstract Domains, In England, D. (Ed.) Whole Body Interaction. Human–Computer Interaction Series, Springer Verlag, London. ISBN 978-0-85729-432-6. Pre-Print WholeBodyInteractionInAbstractDomains.pdf
  • Holland, S. (2010) Asymmetrical Multi-User Co-operative Whole Body Interaction in Abstract Domains. Paper presented at Workshop on Whole Body Interaction, CHI 2010, Atlanta. Holland Asymmetrical WholeBody CHI2010.pdf

Whole Body

Whole body Harmony Space works work with camera tracking and floor projection. A future version will work with the Kinect.

  • Holland, S., Marshall, P., Bird, J., Dalton, S. N., Morris, R., Pantidi, N., Rogers, Y. and Clark, A. (2009) Running up Blueberry Hill: Prototyping Whole Body Interaction in Harmony Space. TEI 2009. Proceedings of the Third Conference on Tangible and Embodied Interaction. ISBN 978-1-60558-493-5 pp 92-98. ACM New York. Oro ID 17966. Pre-Print BlueberryHillWholeBodyInteraction.pdf

Neurophony

Neurophony is a first, exploratory design on a brain computer music interface using musical harmony. Neurophony uses EEG (electroencephalography) sensors and analysis software to control a system called Harmony Space to generate harmonic sequences reflecting the user’s mood.

Neurophony maps positions in the two dimensional Valence / Arousal space into harmonic processes. For this initial design, these processes took the form of simple chord sequences. To determine the nature of the chords, we were guided by insights into harmonic processes derived from the musical tool Harmony Space.

Mapping from EEG to Music – design goals

There are at least two simple, diametrically opposed, principles for guiding mappings from EEG signals to musical output: homeostasis and mirroring. For a system that implements the principle of homestasis, the aim is to maintain neutral levels of valence and arousal (or indeed to try to maintain any desired set point). So for example such a system might try to counter a high level of arousal by generating calming, slowing music, or might try to counter a negative valence (corresponding to negative emotion) by playing euphonious, pleasant music. By contrast, for a system designed to implement the principle of mirroring, the aim is simply to reflect the levels of valence and arousal detected. In this case, for example, a high level of arousal might lead to busy, excited music, and an extreme level of negative valence might lead to ‘ugly’ music.

Neurophony is designed explore the mirroring principle rather than the principle of homestasis. There are an unlimited number of ways in which EEG signals might be mapped into music to reflect this scheme. Various broad design goals were considered to guide our choice of mapping, as follows:

  1. To use a simple mapping in the first instance;
  2. To use a linked graphical and musical output in tandem;
  3. To preserve a principled connection between the mappings used for musical and graphical output;
  4. To minimise, within reasonable bounds, the extent to which the musical mappin relies on learned associations from a particular genre;
  5. To choose a mapping capable of principled elaboration as further dimensions of EEG signal become available.

This work was carried out by Fabien Leon with guidance from Eduardo Miranda and Simon Holland, in collaboration with, and with kind support of, the Interdisciplinary Centre for Computer Music Research at the University of Plymouth.

Project Report report_Fabien_leon.pdf

Embodied Cognition and Harmony

One of the reason why Harmony Space works so well seems to be that it allows people to repurpose human spatial and navigations skills for musical purposes. Inspection of numerous harmonic analyses carried out using Harmony Space suggests that the success of this repurposing is no accident – we hypothesise that human harmonic skills draw upon repurposed spatial abilities. We are working to test and refine this hypothesis this empirically. This work relates strongly to research we have already carried out applying Embodied Cognition theory to Interaction Design for Music.

CMJ Special Issue

Computer Music Journal Special Issue on HCI

Computer Music Journal Winter 2010, Vol. 34, No. 4: 1–2

The Editor of Computer Music Journal writes in this issue:

“Back cover. This diagram from the article by Katie Wilkie et al. shows the user interface of Harmony Space, one of two software programs that the article evaluates with respect to support for musicians’ conceptual metaphors.”

“Also applying established theories from HCI to interactions with music software, in “What Can the Language of Musicians Tell Us about Music Interaction Design?” Katie Wilkie, Simon Holland, and Paul Mulholland examine the ways in which people map generic image schemas, such as containment, spatial orientation, and balance, onto musical concepts to create the conceptual metaphors we use to understand music. The authors analyze two software programs in terms of the image schemas they suggest or support, and discuss the relationships between diverse image schemas or conceptual metaphors and the specific purposes or goals of the software. This article primarily focuses on how and what we know when experiencing music, forming a nice contrast to that of Duignan, Noble, and Biddle, which looks at how and what we do when creating music.”

This work was also highlighted in the editor’s podcast accompanying the issue (approximately 12 minutes 30 seconds into the podcast).

Wilkie, K., Holland,S. and Mulholland, P. (2010) What Can the Language of Musicians Tell Us about Music Interaction Design? Computer Music Journal, 34(4).

Events

Learn About Fair

A participative demo of Song Walker Harmony Space took place at the at the Learn About Fair 10 Feb 2010 in the Ambient Lab of The Jenny Lee Building from 11.30 am to 2.30 pm at the Open University.

Performance Event in Utrecht

Harmony Space the game, 1990. Two hour embodied participative videoed research game with active participation of some twenty musicians, music computing researchers and artificial intelligence researchers in collaboration with Peter Desain, Henkjan Honing and the Faculty of Art, at the Utrecht Hogeschool voor de Kunsten (Utrecht School of Art), 7th March 1990.

Broadcast Media

BBC Radio 4 Program on Maths and Music
BBC Maths Music.mp3

BBC Radio Scotland News item
BBC Radio Scotland HSP.mp3

Presentations

Running up Blueberry Hill.

Presentation at TEI 2009 about Whole Body Harmony Space using Floor Projection and position tracking. TEI2009 Whole Body PRESENTATION.pdf

HCI poster

A poster outlining some basic properties of Harmony Space
HCI poster.pdf

Related Work

Lego Harmony
Labyrinth etc
Mathieu

Partial re-implementations by other researchers of Harmony Space

Since the original presentation of Harmony Space at the Interactional Computer Music Conference in 1989, others have implemented various simplified versions of Harmony Space. These include:

Related Work

  • Tornil and Baptiste-Jessel, 2005. Used a haptic version of the harmony space representation with blind users for music analysis.
  • Bergstrom, Karahalios, and Hart, 2007, used isochords, related to Harmony Space by a simple affine transformation, for the purpose of visualising live music.

Use of early versions by Music Educators

Italian Music Educator Professor Lelio Camileri, UK Music Educator Dr Patricia Howard, and Composer Nigel Morgan have reported on a wide variety of musical uses of prototype versions as an aid to composition, a pointer to harmonic theory, and as an analytical tool, using wide-ranging repertoire from Rameau and Correlli, through Bach and Mozart, to Duke Ellington, John Coltrane, Stevie Wonder, and others.

  • Nigel Morgan, Composer and Music Educator, used an early version of Harmony Space at Denbigh School, Milton Keynes in 1996 to teach Jazz harmony to three key stage 4 classes.
  • Dr Patricia Howard, Music Department, Open University, used an early version of Harmony Space at OU Music Summer School 1994, for teaching beginners to harmonize Bach chorales “on the fly”.
  • Conrad Cork, Jazz improvisation Teacher at Leicester Polytechnic (Now deMonfort University) Incorporated elements of Harmony Space diagrams into his ‘Harmony with Lego Bricks’ System after visits to the OU and discussion about Harmony Space in 1990.

Guide to Practical Use

Playing Tasks

Some example songs played easily in Harmony Space

Pachelbel’s Canon

Pachelbel’s Canon can easily be performed solo, but makes a good team piece.

  • Player one traces the path of the chords. Black notes are switched off.
  • The path starts simply with a diagonal pattern of six moves alternating upward and downward.
  • The pattern is completed with a path of three moves in a straight line to the home location.
  • Then the pattern repeats.
  • Player 3 simply alternates the inversion between second and first inversion for each chord.
  • Player 3 waits for the event before reaching home and alters the chord to a suspended 4th.

A team of three complete beginners can typically can typically learn to play this chord sequence note perfect in 5 – 10 minutes.

Giant Steps John Coltrane

The chord sequence of John Coltrane’s Giant Steps. The upper figure shows the path navigated by one player to control the root path. The lower figure shows the path navigated by another player controlling the key changes. The trace of the performance corresponds to a detailed abstract harmonic analysis of the piece.

For more examples see Analysing Songs

Analytical Tasks

A root trajectory showing David Bowie’s Suffragette City, used in an analytical task. The white shaded ‘preferred territory’ is physically moved under the red nodes and blue arrows that trace out the movement of the song, until the white territory fully contains the red nodes.

Settings and options