Call for Papers: TISMIR Special Collection on Multi-Modal Music Information Retrieval

TISMIR Special Collection on Multi-Modal Music Information Retrieval
(see also the PDF version on the TISMIR web page)

Deadline for Submissions

Scope of the Special Collection
Data related to and associated with music can be retrieved from a variety of sources or modalities: audio tracks; digital scores; lyrics; video clips and concert recordings; artist photos and album covers; expert annotations and reviews; listener social tags from the Internet; and so on. Essentially, the ways humans deal with music are very diverse: we listen to it, read reviews, ask friends for recommendations, enjoy visual performances during concerts, dance and perform rituals, play musical instruments, or rearrange scores.

As such, it is hardly surprising that we have discovered multi-modal data to be so effective in a range of technical tasks that model human experience and expertise. Former studies have already confirmed that music classification scenarios may significantly benefit when several modalities are taken into account. Other works focused on cross-modal analysis, e.g., generating a missing modality from existing ones or aligning the information between different modalities.

The current upswing of disruptive artificial intelligence technologies, deep learning, and big data analytics is quickly changing the world we are living in, and inevitably impacts MIR research as well. Facilitating the ability to learn from very diverse data sources by means of these powerful approaches may not only bring the solutions to related applications to new levels of quality, robustness, and efficiency, but will also help to demonstrate and enhance the breadth and interconnected nature of music science research and the understanding of relationships between different kinds of musical

In this special collection, we invite papers on multi-modal systems in all their diversity. We particularly encourage under-explored repertoire, new connections between fields, and novel research areas. Contributions consisting of pure algorithmic improvements, empirical studies, theoretical discussions, surveys, guidelines for future research, and introductions of new data sets are all welcome, as the special collection will not only address multi-modal MIR, but also cover multi-perspective ideas, developments, and opinions from diverse scientific communities.

Sample Possible Topics
● State-of-the-art music classification or regression systems which are based on several
● Deeper analysis of correlation between distinct modalities and features derived from them
● Presentation of new multi-modal data sets, including the possibility of formal analysis and theoretical discussion of practices for constructing better data sets in future
● Cross-modal analysis, e.g., with the goal of predicting a modality from another one
● Creative and generative AI systems which produce multiple modalities
● Explicit analysis of individual drawbacks and advantages of modalities for specific MIR tasks
● Approaches for training set selection and augmentation techniques for multi-modal classifier systems
● Applying transfer learning, large language models, and neural architecture search to
multi-modal contexts
● Multi-modal perception, cognition, or neuroscience research
● Multi-objective evaluation of multi-modal MIR systems, e.g., not only focusing on the quality, but also on robustness, interpretability, or reduction of the environmental impact during the training of deep neural networks

Guest Editors
● Igor Vatolkin (lead) – Akademischer Rat (Assistant Professor) at the Department of Computer Science, RWTH Aachen University, Germany
● Mark Gotham – Assistant professor at the Department of Computer Science, Durham
University, UK
● Xiao Hu – Associated professor at the University of Hong Kong
● Cory McKay – Professor of music and humanities at Marianopolis College, Canada
● Rui Pedro Paiva – Professor at the Department of Informatics Engineering of the University of Coimbra, Portugal

Submission Guidelines
Please, submit through, and note in your cover letter that your paper is intended to be part of this Special Collection on Multi-Modal MIR.
Submissions should adhere to formatting guidelines of the TISMIR journal: Specifically, articles must not be longer than 8,000 words in length, including referencing, citation and notes.

Please also note that if the paper extends or combines the authors‘ previously published research, it is expected that there is a significant novel contribution in the submission (as a rule of thumb, we would expect at least 50% of the underlying work – the ideas, concepts, methods, results, analysis and discussion – to be new).

In case you are considering submitting to this special issue, it would greatly help our planning if you let us know by replying to igor.vatolkin [AT]

Veröffentlicht unter Conferences & Calls | Hinterlasse einen Kommentar


The program of the following 56th SIGMA meeting on 15.02.2024, 14:00-16:20, which takes place at the Chair for AI Methodology, RWTH Aachen, Theaterstr. 35-39, room 325, and online (please send an email to igor.vatolkin [at] if you wish to get the Zoom link):

14:00-14:05 Welcome greetings

14:05-14:30 Conference study
Fabian Ostermann: Adaptive video game music as a multi-objective benchmark for conditional autoregressive models

14:30-15:20 Research study
Claus Weihs: Optimized decision trees – how to improve model quality in music data analysis

15:20-16:10 Research discussion
Martin Ebeling: Is that what you hear? How ambiguities in hearing disturb the modelling of auditory perception

16:10-16:20 Conferences and calls, teaching activities, miscellaneous, next meeting

Veröffentlicht unter SIGMA | Hinterlasse einen Kommentar

Two papers accepted for EvoMUSART 2024

(1) J. Dettmer, I. Vatolkin, and T. Glasmachers: Weighted Initialisation of Evolutionary Instrument and Pitch Detection in Polyphonic Music. Accepted for Proceedings of the 13th International Conference on Artificial Intelligence in Music, Sound, Art and Design (EvoMUSART)

Abstract: Current state-of-the-art methods for instrument and pitch detection in polyphonic music often require large datasets and long training times; resources which are sparse in the field of music information retrieval, presenting a need for unsupervised alternative methods that do not require such prerequisites. We present a modification to an evolutionary algorithm for polyphonic music approximation through synthesis that uses spectral information to initialise populations with probable pitches. This algorithm can perform joint instrument and pitch detection on polyphonic music pieces without any of the aforementioned constraints. Sets of tuples of (instrument, style, pitch) are graded with a COSH distance fitness function and finally determine the algorithm’s instrument and pitch labels for a given part of a music piece. Further investigation into this fitness function indicates that it tends to create false positives which may conceal the true potential of our modified approach. Regardless of that, our modification still shows significantly faster convergence speed and slightly improved pitch and instrument detection errors over the baseline algorithm on both single onset and full piece experiments.

(2) L. Fricke, M. Gotham, F. Ostermann, and I. Vatolkin: Adaptation and Optimization of AugmentedNet for Roman Numeral Analysis Applied to Audio Signals. Accepted for Proceedings of the 13th International Conference on Artificial Intelligence in Music, Sound, Art and Design (EvoMUSART)

Abstract: Automatic music harmony analysis has recently been significantly improved by AugmentedNet, a convolutional recurrent neural network for predicting Roman numeral labels. The original network receives perfect note annotations from the digital score as inputs and predicts various tonal descriptors: key, chord root, bass note, harmonic rhythm, etc. However, for many music tracks the score is not available at hand. For this study, we have first adjusted AugmentedNet for a direct application to audio signals represented either by chromagrams or semitone spectra. Second, we have implemented and compared further modifications to the network architecture: a preprocessing block designed to learn pitch spellings, increase of the network size, and addition of dropout layers. The statistical analysis helped to identify the best among all proposed configurations and has shown that some of the optimization steps significantly increased the classification performance. Besides, AugmentedNet can reach similar accuracies with audio features as inputs, compared to the perfect annotations that it was originally designed for.

Veröffentlicht unter General | Hinterlasse einen Kommentar

Proseminar on AI in music data analysis

During the summer term 2024, a proseminar „Artificial Intelligence in Music Data Analysis“ will take place the Chair for Artificial Intelligence Methodology, RWTH Aachen.

Veröffentlicht unter Teaching Activities | Hinterlasse einen Kommentar

Job offer as student assistant at the Chair for AI Methodology, RWTH

A position as student assistant for the support in teaching activities and empirical studies for optimisation of music data analysis using Artificial Intelligence is open and should be filled as soon as possible. For details, please see the PDF.

Veröffentlicht unter Jobs | Hinterlasse einen Kommentar

Call for Papers: EvoMUSART 2024

The 13th International Conference on Artificial Intelligence in Music, Sound, Art and Design (EvoMUSART) will take place on 3-5 April 2024, as part of the evo* event.

EvoMUSART webpage:

Submission deadline: 1 November 2023

Conference: 3-5 April 2024

EvoMUSART is a multidisciplinary conference that brings together researchers who are working on the application of Artificial Neural Networks, Evolutionary Computation, Swarm Intelligence, Cellular Automata, Alife, and other Artificial Intelligence techniques in creative and artistic fields such as Visual Art, Music, Architecture, Video, Digital Games, Poetry, or Design. This conference gives researchers in the field the opportunity to promote, present and discuss ongoing work in the area.

Submissions must be at most 14 pages long, excluding references, in Springer LNCS format. Each submission must be anonymized for a double-blind review process.

Accepted papers will be presented orally or as posters at the event and included in the EvoMUSART proceedings published by Springer Nature in a dedicated volume of the Lecture Notes in Computer Science series.

More information on the submission process and the topics of EvoMUSART:

Flyer of EvoMUSART 2024:

Veröffentlicht unter Conferences & Calls | Hinterlasse einen Kommentar


The program of the following 55th SIGMA meeting on 16.06.2023, 14:00-16:00, which takes place at the Chair of Algorithm Engineering, Deparment of Computer Science, TU Dortmund, Otto-Hahn-Str. 14, room 202, and online (please send an email to igor.vatolkin [at] if you wish to get the Zoom link):

14:00-14:05 Welcome greetings

14:05-14:35 Bachelor’s thesis (results)
Alhuseen Ali: Comparison between Artificial Neural Networks and Traditional Classifiers for the Suppression of Background Noise

14:35-15:05 Research study
Leonard FrickeApplication of AugmentedNet for Different Sound Bodies

15:05-15:35 Conference study
Igor Vatolkin: Musical Genre Recognition based on Deep Neural Network Predictions of Harmony, Instrumentation, and Segments

15:35-15:55 Conferences and calls, miscellaneous, next meeting

Veröffentlicht unter SIGMA | Hinterlasse einen Kommentar

Paper describing Artificial Audio Multitracks (AAM) dataset published in EURASIP Journal on Audio, Speech, and Music Processing

F. Ostermann, I. Vatolkin, and M. Ebeling: AAM: a Dataset of Artificial Audio Multitracks for Diverse Music Information Retrieval Tasks. EURASIP Journal on Audio, Speech, and Music Processing, 13, 2023.

Zenodo link:

Abstract: We present a new dataset of 3000 artificial music tracks with rich annotations based on real instrument samples and generated by algorithmic composition with respect to music theory. Our collection provides ground truth onset information and has several advantages compared to many available datasets. It can be used to compare and optimize algorithms for various music information retrieval tasks like music segmentation, instrument recognition, source separation, onset detection, key and chord recognition, or tempo estimation. As the audio is perfectly aligned to original MIDIs, all annotations (onsets, pitches, instruments, keys, tempos, chords, beats, and segment boundaries) are absolutely precise. Because of that, specific scenarios can be addressed, for instance, detection of segment boundaries with instrument and key change only, or onset detection only in tracks with drums and slow tempo. This allows for the exhaustive evaluation and identification of individual weak points of algorithms. In contrast to datasets with commercial music, all audio tracks are freely available, allowing for extraction of own audio features. All music pieces are stored as single instrument audio tracks and a mix track, so that different augmentations and DSP effects can be applied to extend training sets and create individual mixes, e.g., for deep neural networks. In three case studies, we show how different algorithms and neural network models can be analyzed and compared for music segmentation, instrument recognition, and onset detection. In future, the dataset can be easily extended under consideration of specific demands to the composition process.

Veröffentlicht unter MIR Research, Publications | Hinterlasse einen Kommentar

Slides from the MDA lecture by Meinard Müller

On 03.02.2023, the last unit of  the interdisciplinary lecture on Music Data Analysis was organized by Prof. Meinard Müller, Audiolabs Erlangen. The slides on FMP Notebooks are available here (handout version).

Veröffentlicht unter Teaching Activities | Hinterlasse einen Kommentar

Two papers accepted for EvoMUSART

(1) I. Vatolkin, M. Gotham, N. Nápoles López, and F. Ostermann: Musical Genre Recognition based on Deep Descriptors of Harmony, Instrumentation, and Segments. Accepted for Proceedings of the 12th International Conference on Artificial Intelligence in Music, Sound, Art and Design (EvoMUSART)

Abstract: Deep learning has recently established itself as a cluster of methods of choice for almost all classification tasks in music information retrieval. However, despite very good classification performance, it sometimes brings disadvantages including long training times and higher energy costs, lower interpretability of classification models, or an increased risk of overfitting when applied to small training sets due to a very large number of trainable parameters. In this paper, we investigate the combination of both deep and shallow algorithms for recognition of musical genres using a transfer learning approach. We train deep classification models once to predict harmonic, instrumental, and segment properties from datasets with respective annotations. Their predictions for another dataset with annotated genres are used as features for shallow classification methods. They can be trained over and again for different categories, and are particularly useful when the training sets are small, in a real world scenario when listeners define various musical categories selecting only a few prototype tracks. The experiments show the potential of the proposed approach for genre recognition. In particular, when combined with evolutionary feature selection which identifies the most relevant deep feature dimensions, the classification errors became significantly lower in almost all cases, compared to a baseline based on MFCCs or results reported in the previous work.

(2) L. Fricke, I. Vatolkin, and F. Ostermann: Application of Neural Architecture Search to Instrument Recognition in Polyphonic Audio. Accepted for Proceedings of the 12th International Conference on Artificial Intelligence in Music, Sound, Art and Design (EvoMUSART)

Abstract: Instrument recognition in polyphonic audio signals is a very challenging classification task. It helps to improve related application scenarios, like music transcription and recommendation, organization of large music collections, or analysis of historical trends and properties of musical styles. Recently, the classification performance could be improved by the integration of deep convolutional neural networks. However, in to date published studies, the network architectures and parameter settings were usually adopted from image recognition tasks and manually adjusted, without a systematic optimization. In this paper, we show how two different neural architecture search strategies can be successfully applied for improvement of the prediction of nine instrument classes, significantly outperforming the classification performance of three fixed baseline architectures from previous works. Although high computing efforts for model optimization are required, the training of the final architecture is done only once for later prediction of instruments in a possibly unlimited number of musical tracks.

Veröffentlicht unter MIR Research, Publications | Hinterlasse einen Kommentar