Visual Perception of Music Notation: On-Line and Off-Line Recognition Susan E. George University of South Australia, Australia IDEA GROUP PUBLISHING Visual Perception ofVisual Perception of Visual Perception ofVisual Perception of Visual Perception of Music Notation:Music Notation: Music Notation:Music Notation: Music Notation: On-Line and Off-LineOn-Line and Off-Line On-Line and Off-LineOn-Line and Off-Line On-Line and Off-Line RecognitionRecognition RecognitionRecognition Recognition Susan E. George University of South Australia, Australia IRM Press Publisher of innovative scholarly and professional information technology titles in the cyberage Hershey • London • Melbourne • Singapore Acquisitions Editor: Mehdi Khosrow-Pour Senior Managing Editor: Jan Travers Managing Editor: Amanda Appicello Development Editor: Michele Rossi Copy Editor: Michelle Wilgenburg Typesetter: Amanda Appicello Cover Design: Lisa Tosheff Printed at: Integrated Book Technology Published in the United States of America by IRM Press (an imprint of Idea Group Inc.) 701 E. Chocolate Avenue, Suite 200 Hershey PA 17033-1240 Tel: 717-533-8845 Fax: 717-533-8661 E-mail: cust@idea-group.com Web site: http://www.irm-press.com and in the United Kingdom by IRM Press (an imprint of Idea Group Inc.) 3 Henrietta Street Covent Garden London WC2E 8LU Tel: 44 20 7240 0856 Fax: 44 20 7379 3313 Web site: http://www.eurospan.co.uk Copyright © 2005 by IRM Press. All rights reserved. No part of this book may be reproduced in any form or by any means, electronic or mechanical, including photocopying, without written permission from the publisher. Library of Congress Cataloging-in-Publication Data George, Susan Ella. Visual perception of music notation : on-line and off-line recognition / Susan Ella George. p. cm. Includes bibliographical references and index. ISBN 1-931777-94-2 (pbk.) ISBN 1-931777-95-0 (ebook) 1. Musical notation Data processing. 2. Artificial intelligence Musical applications. I. Title. ML73.G46 2005 780'.1'48028564 dc21 2003008875 ISBN 1-59140-298-0 (h/c) British Cataloguing in Publication Data A Cataloguing in Publication record for this book is available from the British Library. All work contributed to this book is new, previously-unpublished material. The views expressed in this book are those of the authors, but not necessarily of the publisher. New Releases from IRM Press Excellent additions to your institution’s library! Recommend these titles to your Librarian! To receive a copy of the IRM Press catalog, please contact 1/717-533-8845, fax 1/717-533-8661, or visit the IRM Press Online Bookstore at: [http://www.irm-press.com]! Note: All IRM Press books are also available as ebooks on netlibrary.com as well as other ebook sources. Contact Ms. Carrie Skovrinskie at [cskovrinskie@idea-group.com] to receive a complete list of sources where you can obtain ebook information or IRM Press titles. • Visual Perception of Music Notation: On-Line and Off-Line Recognition Susan Ella George ISBN: 1-931777-94-2; eISBN: 1-931777-95-0 / © 2004 • 3D Modeling and Animation: Synthesis and Analysis Techniques for the Human Body Nikos Sarris & Michael G. Strintzis ISBN: 1-931777-98-5; eISBN: 1-931777-99-3 / © 2004 • Innovations of Knowledge Management Bonnie Montano ISBN: 1-59140-229-8; eISBN: 1-59140-230-1 / © 2004 • e-Collaborations and Virtual Organizations Michelle W. L. Fong ISBN: 1-59140-231-X; eISBN: 1-59140-232-8 / © 2004 • Information Security and Ethics: Social and Organizational Issues Marian Quigley ISBN: 1-59140-233-6; eISBN: 1-59140-234-4 / © 2004 • Issues of Human Computer Interaction Anabela Sarmento ISBN: 1-59140-235-2; eISBN: 1-59140-236-0 / © 2004 • Instructional Technologies: Cognitive Aspects of Online Programs Paul Darbyshire ISBN: 1-59140-237-9; eISBN: 1-59140-238-7 / © 2004 • E-Commerce and M-Commerce Technologies P. Candace Deans ISBN: 1-59140-239-5; eISBN: 1-59140-240-9 / © 2004 Visual Perception ofVisual Perception of Visual Perception ofVisual Perception of Visual Perception of Music Notation:Music Notation: Music Notation:Music Notation: Music Notation: On-Line and Off-Line RecognitionOn-Line and Off-Line Recognition On-Line and Off-Line RecognitionOn-Line and Off-Line Recognition On-Line and Off-Line Recognition Table of ContentsTable of Contents Table of ContentsTable of Contents Table of Contents Preface vi Susan E. George, University of South Australia, Australia Section 1: Off-Line Music ProcessingSection 1: Off-Line Music Processing Section 1: Off-Line Music ProcessingSection 1: Off-Line Music Processing Section 1: Off-Line Music Processing Chapter 1 Staff Detection and Removal 1 Ichiro Fujinaga, McGill University, Canada Chapter 2 An Off-Line Optical Music Sheet Recognition 40 Pierfrancesco Bellini, University of Florence, Italy Ivan Bruno, University of Florence, Italy Paolo Nesi, University of Florence, Italy Chapter 3 Wavelets for Dealing with Super-Imposed Objects in Recognition of Music Notation 78 Susan E. George, University of South Australia, Australia Section 2: Handwritten Music RecognitionSection 2: Handwritten Music Recognition Section 2: Handwritten Music RecognitionSection 2: Handwritten Music Recognition Section 2: Handwritten Music Recognition Chapter 4 Optical Music Analysis for Printed Music Score and Handwritten Music Manuscript 108 Kia Ng, University of Leeds, United Kingdom Chapter 5 Pen-Based Input for On-Line Handwritten Music Notation 128 Susan E. George, University of South Australia, Australia Section 3: Lyric RecognitionSection 3: Lyric Recognition Section 3: Lyric RecognitionSection 3: Lyric Recognition Section 3: Lyric Recognition Chapter 6 Multilingual Lyric Modeling and Management 162 Pierfrancesco Bellini, University of Florence, Italy Ivan Bruno, University of Florence, Italy Paolo Nesi, University of Florence, Italy Chapter 7 Lyric Recognition and Christian Music 198 Susan E. George, University of South Australia, Australia Section 4: Music Description and its ApplicationsSection 4: Music Description and its Applications Section 4: Music Description and its ApplicationsSection 4: Music Description and its Applications Section 4: Music Description and its Applications Chapter 8 Towards Constructing Emotional Landscapes with Music 227 Dave Billinge, University of Portsmouth, United Kingdom Tom Addis, University of Portsmouth, United Kingdom and University of Bath, United Kingdom Chapter 9 Modeling Music Notation in the Internet Multimedia Age 272 Pierfrancesco Bellini, University of Florence, Italy Paolo Nesi, University of Florence, Italy Section 5: EvaluationSection 5: Evaluation Section 5: EvaluationSection 5: Evaluation Section 5: Evaluation Chapter 10 Evaluation in the Visual Perception of Music 304 Susan E. George, University of South Australia, Australia About the Editor 350 About the Authors 351 Index 354 PrefacePreface PrefacePreface Preface vi Overview of Subject Matter and Topic Context The computer recognition of music notation, its interpretation and use within various applications, raises many challenges and questions with regards to the appropriate algorithms, techniques and methods with which to automatically understand music notation. Modern day music notation is one of the most widely recognised international languages of all time. It has developed over many years as requirements of consistency and precision led to the develop- ment of both music theory and representation. Graphic forms of notation are first known from the 7th Century, with the modern system for notes developed in Europe during the 14th Century. This volume consolidates the successes, challenges and questions raised by the computer perception of this music notation language. The computer perception of music notation began with the field of Optical Music Recognition (OMR) as researchers tackled the problem of recognising and interpreting the symbols of printed music notation from a scanned image. More recently, interest in automatic perception has extended to all components of song lyric, melody and other symbols, even broadening to multi-lingual handwritten components. With the advent of pen-based input systems, automatic recognition of notation has also extended into the on-line context — moving away from processing static scanned images, to recognising dynamically constructed pen strokes. New applications, including concert- planning systems sensitive to the emotional content of music, have placed new demands upon description, representation and recognition. Summary of Sections and Chapters This special volume consists of both invited chapters and open-solicited chapters written by leading researchers in the field. All papers were peer reviewed by at least two recognised reviewers. This book contains 10 chapters divided into five sections: vii Section 1 is concerned with the processing of music images, or Optical Music Recognition (OMR). A focus is made upon recognising printed typeset music from a scanned image of music score. Section 2 extends the recognition of music notation to handwritten rather than printed typeset music, and also moves into the on-line context with a consideration of dynamic pen-based input. Section 3 focuses upon lyric recognition and the identification and representation of conventional lyric text combined with the symbols of music notation. Section 4 considers the importance of music description languages with emerging applications, including the context of Web-based multi-media and concert planning systems sensitive to the emotional content of music. Finally, Section 5 considers the difficulty of evaluating automatic perceptive systems, discussing the issues and providing some benchmark test data. Section 1: Off-Line Music Processing • Chapter 1: Staff Detection and Removal, Ichiro Fujinaga • Chapter 2: An Off-line Optical Music Sheet Recognition, Pierfrancesco Bellini, Ivan Bruno, Paolo Nesi • Chapter 3: Wavelets for Dealing with Super-Imposed Objects in Recognition of Music Notation, Susan E. George Section 2: Handwritten Music Recognition • Chapter 4: Optical Music Analysis for Printed Music Score and Handwritten Music Manuscript, Kia Ng • Chapter 5: Pen-Based Input for On-Line Handwritten Music Nota- tion, Susan E. George Section 3: Lyric Recognition • Chapter 6: Multilingual Lyric Modeling and Management, Pierfrancesco Bellini, Ivan Bruno, Paolo Nesi • Chapter 7: Lyric Recognition and Christian Music, Susan E. George Section 4: Music Description and its Applications • Chapter 8: Towards Constructing Emotional Landscapes with Music, Dave Billinge, Tom Addis • Chapter 9: Modeling Music Notation in the Internet Multimedia Age, Pierfrancesco Bellini, Paolo Nesi viii Section 5: Evaluation • Chapter 10: Evaluation in the Visual Perception of Music, Susan E. George Description of Each Chapter In Chapter 1, Dr. Ichiro Fujinaga describes the issues involved in the detection and removal of stafflines of musical scores. This removal process is an important step for many optical music recognition systems and facilitates the segmenta- tion and recognition of musical symbols. The process is complicated by the fact that most music symbols are placed on top of stafflines and these lines are often neither straight nor parallel to each other. The challenge here is to remove as much of the stafflines as possible while preserving the shapes of the musical symbols, which are superimposed on stafflines. Various problematic examples are illustrated and a detailed explanation of an algorithm is presented. Image processing techniques used in the algorithm include: run- length coding, connected-component analysis, and projections. In Chapter 2, Professor Pierfrancesco Bellini, Mr. Ivan Bruno and Professor Paolo Nesi compare OMR with OCR and discuss the O 3 MR system. An overview of the main issues and a survey of the main related works are discussed. The O 3 MR system (Object Oriented Optical Music Recognition) system is also described. The used approach in such system is based on the adoption of projections for the extraction of basic symbols that constitute graphic elements of the music notation. Algorithms and a set of examples are also included to better focus concepts and adopted solutions. In Chapter 3, Dr. Susan E. George investigates a problem that arises in OMR when notes and other music notation symbols are super-imposed upon stavelines in the music image. A general purpose knowledge-free method of image filtering using two-dimensional wavelets is investigated to separate the super-imposed objects. The filtering provides a unified theory of staveline removal/symbol segmentation, and practically is a useful pre-processing method for OMR. In Chapter 4, Dr. Kia Ng examines a method of recognising printed music — both handwritten and typeset. The chapter presents a brief background of the field, discusses the main obstacles, and presents the processes involved for printed music scores processing; using a divide-and-conquer approach to sub- segment compound musical symbols (e.g., chords) and inter-connected groups (e.g., beamed quavers) into lower-level graphical primitives such as lines and ellipses before recognition and reconstruction. This is followed by discussions on the developments of a handwritten manuscripts prototype with a segmen- tation approach to separate handwritten musical primitives. Issues and ix approaches for recognition, reconstruction and revalidation using basic music syntax and high-level domain knowledge, and data representation are also presented. In Chapter 5, Dr. Susan E. George concentrates upon the recognition of handwritten music entered in a dynamic editing context with use of pen-based input. The chapter makes a survey of the current scope of on-line (or dynamic) handwritten input of music notation, presenting the outstanding problems in recognition. A solution using the multi-layer perception artificial neural network is presented, explaining experiments in music symbol recognition from a study involving notation writing from some 25 people using a pressure- sensitive digitiser for input. Results suggest that a voting system among networks trained to recognize individual symbols produces the best recogni- tion rate. In Chapter 6, Professor Pierfrancesco Bellini, Mr. Ivan Bruno and Professor Paolo Nesi present an object-oriented language capable of modelling music notation and lyrics. This new model makes it possible to “plug” on the symbolic score different lyrics depending on the language. This is done by keeping separate the music notation model and the lyrics model. An object-oriented model of music notation and lyrics are presented with many examples. These models have been implemented in the music editor produced within the WEDELMUSIC IST project. A specific language has been developed to associate the lyrics with the score. The most important music notation formats are reviewed focusing on their representation of multilingual lyrics. In Chapter 7, Dr. Susan E. George presents a consideration of lyric recognition in OMR in the context of Christian music. Lyrics are obviously found in other music contexts, but they are of primary importance in Christian music — where the words are as integral as the notation. This chapter (i) identifies the inseparability of notation and word in Christian music, (ii) isolates the challenges of lyric recognition in OMR providing some examples of lyric recognition achieved by current OMR software and (iii) considers some solutions outlining page segmentation and character/word recognition ap- proaches, particularly focusing upon the target of recognition, as a high level representation language, that integrates the music with lyrics. In Chapter 8, Dr. Dave Billinge and Professor Tom Addis investigate language to describe the emotional and perceptual content of music in linguistic terms. They aim for a new paradigm in human-computer interaction that they call tropic mediation and describe the origins of the research in a wish to provide a concert planner with an expert system. Some consideration is given to how music might have arisen within human culture and in particular why it presents unique problems of verbal description. An initial investigation into a discrete, stable lexicon of musical effect is summarised and the authors explain how and [...]... integrated with the model of music notation, such as: automatic formatting, music notation navigation, synchronization of music notation with real audio, etc In this chapter, the WEDELMUSIC XML format for multimedia music applications of music notation is presented It includes a music notation format in XML and a format for modelling multimedia elements, their relationships and synchronization with... comparison of this new model with the most important and emerging models is reported The taxonomy used can be useful for assessing and comparing suitability of music notation models and formats for their adoption in new emerging applications and for their usage in classical music editors In Chapter 10, Dr Susan E George considers the problem of evaluating the recognition music notation in both the on-line and. .. describes the issues involved in the detection and removal of stavelines of musical scores This removal process is an important step for many Optical Music Recognition systems and facilitates the segmentation and recognition of musical symbols The process is complicated by the fact that most music symbols are placed on top of stavelines and these lines are often neither straight nor parallel to each other... handwritten music) and hence a target representation provided for recognition processes Initial consideration of the range of test data that is needed (MusicBase I and II) is also made Conclusion This book will be useful to researchers and students in the field of pattern recognition, document analysis and pen-based computing, as well as potential users and vendors in the specific field of music recognition. .. and off-line (traditional OMR) contexts The chapter presents a summary of reviews that have been performed for commercial OMR systems and addresses some of the issues in evaluation that must be taken into account to enable adequate comparison of recognition performance A representation language (HEART) is suggested, such that the semantics of music is captured (including the dynamics of handwritten music) ... before, during and after the birth of our beautiful twins, Joanna and Abigail; received with much joy during the course of this project! Susan E George Editor S ECTION 1: OFF-LINE MUSIC PROCESSING Copyright © 2004, Idea Group Inc Copying or distributing in print or electronic forms without written permission of Idea Group Inc is prohibited 1: STAFF DETECTION AND REMOVAL 1 1 STAFF DETECTION AND REMOVAL... used from the eleventh to the 13th century and the five-line staff did not become standard until mid-17th century, (some keyboard music of the 16th and 17th centuries employed staves of as many as 15 lines)” (Read, 1979, p 28) Today, percussion parts may have one to several numbers of lines The placement and the size of staves may vary on a given page because of an auxiliary staff, which is an alternate... permission of Idea Group Inc is prohibited 1: STAFF DETECTION AND REMOVAL 3 Overview of OMR Research The OMR research began with two MIT doctoral dissertations (Prusslin, 1966, 1970) With the availability of inexpensive optical scanners, much research began in the 1980s Excellent historical reviews of OMR systems are given in Blostein and Baird (1992) and in Bainbridge and Carter (1997) After Prusslin and. .. another run Y, if Y is on the next column (n + 1) and X and Y are connected Y is called a child of X In a depth-first search, all children of a given father are searched first recursively, before finding other relatives such as grandfathers Note that a father can have any number of sons and each son may have any number of fathers Also, by definition of run-length coding, no two runs in the same column... (1997), and Ng (1995) Many commercial OMR software exists today, such as capella-scan, OMeR, PhotoScore, SharpEye, and SmartScore Background The following procedure for detecting and removing staves may seem overly complex, but it was found necessary in order to deal with the variety of staff configurations and distortions such as skewing The detection of staves is complicated by the variety of staves . of Visual Perception ofVisual Perception of Visual Perception of Music Notation :Music Notation: Music Notation :Music Notation: Music Notation: On-Line and Off-LineOn-Line and Off-Line On-Line and Off-LineOn-Line. of Music Notation :Music Notation: Music Notation :Music Notation: Music Notation: On-Line and Off-Line RecognitionOn-Line and Off-Line Recognition On-Line and Off-Line RecognitionOn-Line and Off-Line. Visual Perception of Music Notation: On-Line and Off-Line Recognition Susan E. George University of South Australia, Australia IDEA GROUP PUBLISHING Visual Perception ofVisual Perception of Visual