demo_chroma.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7e27922b",
   "metadata": {},
   "source": [
    "# Sonification of Chromagrams\n",
    "In this notebook, we illustrate the sonification of chromagrams using the ```libsoni.core.chroma``` module. A chromagram can be seen as a representation of the distribution of musical pitch classes, a feature often used in MIR Applications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d42a3d8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import numpy as np\n",
    "import librosa, librosa.display\n",
    "import pandas as pd\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "import IPython.display as ipd\n",
    "\n",
    "from libsoni.util.utils import mix_sonification_and_original\n",
    "from libsoni.core.chroma import sonify_chromagram\n",
    "from libsoni.core.methods import generate_shepard_tone\n",
    "\n",
    "sys.path.append('..')\n",
    "import libfmp.b\n",
    "import libfmp.c8\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "Fs = 22050\n",
    "H = 1024"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ca99ac0",
   "metadata": {},
   "source": [
    "## Simple Scenario: C Major Cadence\n",
    "To start with a simple example, let's look at the chords of a C major cadence: **C:maj**, **F:maj** and **G:maj**.\n",
    "<img src=\"figures/demo_chroma/C_Major_Cadence.png\" alt=\"C-Major-Triad\" width=\"500\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1d74b79",
   "metadata": {},
   "outputs": [],
   "source": [
    "C_Maj_Cadence_Chords_pitchclasses = [(0,4,7), (5,9,0), (2,7,11), (0,4,7)]\n",
    "\n",
    "starts = [0, 1, 2, 3]\n",
    "\n",
    "durations = [0.75, 0.75, 0.75, 0.75]\n",
    "\n",
    "Fs_frame = Fs / H\n",
    "\n",
    "\n",
    "\n",
    "Chromagram_C_Maj_Cadence = np.zeros((12,int((starts[-1]+durations[-1])*Fs/H)))\n",
    "\n",
    "\n",
    "for i in range(len(starts)):\n",
    "    Chromagram_C_Maj_Cadence[C_Maj_Cadence_Chords_pitchclasses[i],int(starts[i]*Fs/H):int((starts[i]+durations[i])*Fs/H)]=1\n",
    "\n",
    "title = 'Chromagram for C Major Cadence'\n",
    "libfmp.b.plot_chromagram(Chromagram_C_Maj_Cadence, Fs=Fs_frame, title=title, figsize=(8, 2.5))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8826df7b",
   "metadata": {},
   "source": [
    "### Sonification using Shepard Tones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "835dcdd5",
   "metadata": {},
   "outputs": [],
   "source": [
    "shepard_tone = generate_shepard_tone()\n",
    "\n",
    "ipd.display(ipd.Audio(shepard_tone, rate=Fs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "316f1bec",
   "metadata": {},
   "outputs": [],
   "source": [
    "one_octave_shepard = np.asarray([])\n",
    "\n",
    "for pitch_class in np.arange(0,13):\n",
    "    \n",
    "    one_octave_shepard=np.append(one_octave_shepard, generate_shepard_tone(pitch_class%12))\n",
    "\n",
    "print('Shepard tone for one octave:')\n",
    "ipd.display(ipd.Audio(one_octave_shepard,rate=Fs))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "288671eb",
   "metadata": {},
   "source": [
    "### Sonified C Major Cadence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "975be605",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sonification using libsoni\n",
    "x_chroma_ann = sonify_chromagram(Chromagram_C_Maj_Cadence,H=512,fs=Fs)\n",
    "\n",
    "print('Sonification of chroma-based respresentation of the C Major Cadence:')\n",
    "ipd.display(ipd.Audio(x_chroma_ann, rate=Fs))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72c7bffa",
   "metadata": {},
   "source": [
    "### Customizing the Sonification\n",
    "\n",
    "Some sound changing parameters can be passed to ```sonify_chromagram``` in order to change the shepard tones. \n",
    "\n",
    "- ```pitch_range```: Tuple defining which pitch range to encounter in shepard tone\n",
    "- ```filter```: bool, deciding, if shepard tones are filtered or not\n",
    "- ```f_center``` : float, center_frequency in Hertz for bell-shaped filter\n",
    "- ```octave_cutoff```: int, determines, at which multiple of f_center, the harmonics get attenuated by 2.\n",
    "Let's show some examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94d790eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Shepard tone, default\n",
    "shepard_tone = generate_shepard_tone()\n",
    "print('Shepard tone, default:')\n",
    "ipd.display(ipd.Audio(shepard_tone, rate=Fs))\n",
    "\n",
    "# Shepard tone with small, high pitch range\n",
    "shepard_tone_small_pitch_range_high = generate_shepard_tone(pitch_range=(60,84))\n",
    "print('Shepard tone with small, high pitch range:')\n",
    "ipd.display(ipd.Audio(shepard_tone_small_pitch_range_high, rate=Fs))\n",
    "\n",
    "# Shepard tone with small, low pitch range\n",
    "shepard_tone_small_pitch_range_low = generate_shepard_tone(pitch_range=(36,60))\n",
    "print('Shepard tone with small, low pitch range:')\n",
    "ipd.display(ipd.Audio(shepard_tone_small_pitch_range_low, rate=Fs))\n",
    "\n",
    "# Narrow filtered Shepard tone with low center frequency:\n",
    "shepard_tone_filtered_low = generate_shepard_tone(filter=True, f_center=100, octave_cutoff=1)\n",
    "print('Narrow filtered Shepard tone with low center frequency:')\n",
    "ipd.display(ipd.Audio(shepard_tone_filtered_low, rate=Fs))\n",
    "\n",
    "# Wide filtered Shepard tone with high center frequency:\n",
    "shepard_tone_filtered_low = generate_shepard_tone(pitch_range=(50,127), filter=True, f_center=1000, octave_cutoff=3)\n",
    "print('Wide filtered Shepard tone with high center frequency:')\n",
    "ipd.display(ipd.Audio(shepard_tone_filtered_low, rate=Fs))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "911ec57b",
   "metadata": {},
   "source": [
    "## Scenario: *Frühlingstraum (Dream of Spring)* by *Franz Schubert*\n",
    "<img src=\"figures/demo_chroma/FMP_B_Sonify_Pitch_Schubert_D911-11_A.png\" alt=\"Frühlingstraum\" width=\"600\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "736aa226",
   "metadata": {},
   "outputs": [],
   "source": [
    "schubert_audio, Fs = librosa.load(os.path.join('data_audio', 'demo_chroma', 'FMP_B_Sonify_Pitch_Schubert_D911-11_SC06.wav'), sr=Fs)\n",
    "\n",
    "print('Frühlingstraum by Franz Schubert:')\n",
    "ipd.display(ipd.Audio(schubert_audio,rate=Fs))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c86a05ce",
   "metadata": {},
   "source": [
    "### Preparing the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b3bb4ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "schubert_df = pd.read_csv(os.path.join('data_csv','demo_chroma','FMP_B_Sonify_Pitch_Schubert_D911-11_SC06.csv'), sep=';')\n",
    "\n",
    "note_list = [[start, duration, pitch, vel, ins] for i, \n",
    "             (start, duration, pitch, vel, ins) in schubert_df.iterrows()]\n",
    "\n",
    "num_frames = int(len(schubert_audio) / H)\n",
    "Fs_frame = Fs / H\n",
    "C_annotated_schubert = libfmp.b.list_to_chromagram(note_list, num_frames, Fs_frame)\n",
    "ipd.display(schubert_df[0:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a893b04",
   "metadata": {},
   "outputs": [],
   "source": [
    "libfmp.b.plot_chromagram(C_annotated_schubert, Fs=Fs_frame, title='Frühlingstraum, Franz Schubert', figsize=(8, 2.5))\n",
    "plt.show()\n",
    "\n",
    "# Sonified Chromagram from annotation\n",
    "c_annotated_schubert_sonified = sonify_chromagram(C_annotated_schubert,H=H,fs=Fs)\n",
    "print('Sonification of annotated Chromagram:')\n",
    "ipd.display(ipd.Audio(c_annotated_schubert_sonified, rate=Fs))\n",
    "\n",
    "schubert_annotated_sonified_w_original = mix_sonification_and_original(sonification=c_annotated_schubert_sonified,\n",
    "                                                         original_audio=schubert_audio,\n",
    "                                                         gain_lin_sonification=1,\n",
    "                                                         gain_lin_original_audio=1)\n",
    "\n",
    "print('Sonification of original Audio and annotated Chromagram:')\n",
    "ipd.display(ipd.Audio(schubert_annotated_sonified_w_original, rate=Fs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3dad48a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Computation of Chromagram from audio using librosa\n",
    "N = 2048\n",
    "H = 1024\n",
    "C_schubert = librosa.feature.chroma_stft(y=schubert_audio, sr=Fs, tuning=0, norm=2, hop_length=H, n_fft=N)\n",
    "\n",
    "libfmp.b.plot_chromagram(C_schubert, Fs=Fs_frame, title='Frühlingstraum, Franz Schubert', figsize=(8, 2.5))\n",
    "plt.show()\n",
    "\n",
    "\n",
    "c_schubert_sonified = sonify_chromagram(C_schubert,H=H,fs=Fs)\n",
    "\n",
    "\n",
    "print('Sonification of computed Chromagram:')\n",
    "ipd.display(ipd.Audio(c_schubert_sonified, rate=Fs))\n",
    "\n",
    "\n",
    "schubert_sonified_w_original = mix_sonification_and_original(sonification=c_schubert_sonified,\n",
    "                                                             original_audio=schubert_audio,\n",
    "                                                             gain_lin_sonification=1,\n",
    "                                                             gain_lin_original_audio=1)\n",
    "\n",
    "print('Sonification of computed Chromagram and annotated Chromagram:')\n",
    "ipd.display(ipd.Audio(schubert_sonified_w_original, rate=Fs))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}