Basic filters for convolutional neural networks applied to music: Training or design?

Abstract

When convolutional neural networks are used to tackle learning problems based on music or other time series, raw one-dimensional data are commonly preprocessed to obtain spectrogram or mel-spectrogram coefficients, which are then used as input to the actual neural network. In this contribution, we investigate, both theoretically and experimentally, the influence of this pre-processing step on the network’s performance and pose the question whether replacing it by applying adaptive or learned filters directly to the raw data can improve learning success. The theoretical results show that approximately reproducing mel-spectrogram coefficients by applying adaptive filters and subsequent time-averaging on the squared amplitudes is in principle possible. We also conducted extensive experimental work on the task of singing voice detection in music. The results of these experiments show that for classification based on convolutional neural networks the features obtained from adaptive filter banks followed by time-averaging the squared modulus of the filters’ output perform better than the canonical Fourier transform-based mel-spectrogram coefficients. Alternative adaptive approaches with center frequencies or time-averaging lengths learned from training data perform equally well.

Appendix A: Proof of Theorem 1

In order to include the situation described in Theorem 1, we assume the situation in which the original spectrogram is sub-sampled, in other words, we start the computations concerning a signal f from

$$\begin{aligned} S_0 ( \alpha l, \beta k)= |{\mathcal {V}} _g f (\alpha l, \beta k) |^2 = |{\mathcal {F}} (f\cdot T_{\alpha l} g)(\beta k)|^2. \end{aligned}$$

The proof is based on the observation that the mel-spectrogram can be written via the operation of so-called STFT- or Gabor multipliers, cf. [17], on any given function in the sense of a bilinear form. Before deriving the involved correspondence, we thus introduce this important class of operators.

Given a window function g, time- and frequency-sub-sampling parameters $\alpha , \beta$, respectively, and a function $\mathbf{{m}}: {\mathbb {Z}} \times {\mathbb {Z}} \mapsto {\mathbb {C}}$, the corresponding Gabor multiplier $G^{\alpha ,\beta }_{g, \mathbf{{m}}}$ is defined as

$$\begin{aligned} G^{\alpha ,\beta }_{g, \mathbf{{m}}} f = \sum _k \sum _l \mathbf{m} (k,l) \langle f, M_{\beta k} T_{\alpha l} g\rangle M_{\beta k} T_{\alpha l} g . \end{aligned}$$

We next derive the expression of a mel-spectrogram by an appropriately chosen Gabor multiplier. Using sub-sampling factors $\alpha$ in time and $\beta$ in frequency as before, we start from (4) and reformulate as follows:

$$\begin{aligned} {{\text {MS}}}_{g}(f) (b,\nu )=&\sum _k |{\mathcal {F}} (f\cdot T_b g)(\beta k)|^2 \cdot \varLambda _\nu (\beta k)\\ =&\sum _k \langle f, M_{\beta k} T_b g\rangle \overline{\langle f, M_{\beta k} T_b g\rangle } \varLambda _\nu (\beta k)\\ =&\left\langle \sum _k \varLambda _\nu ( \beta k) \langle f, M_{\beta k} T_b g\rangle M_{\beta k} T_b g , f\right\rangle \\ =&\left\langle \sum _k \sum _l \mathbf{m} (k,l) \langle f, M_{\beta k} T_{\alpha l} g\rangle M_{\beta k} T_{\alpha l} g , f\right\rangle \end{aligned}$$

with $\mathbf{m} (k,l) = \delta (\alpha l-b)\varLambda _\nu (\beta k)$. We see that the mel-coefficients can thus be interpreted via a Gabor multiplier: ${{\text {MS}}}_{g}(f) (b,\nu ) = \langle G^{\alpha ,\beta }_{g, \mathbf{{m}}}f, f \rangle$.

The next step is to switch to an alternative operator representation. Indeed, as shown in [16], every operator H can equally be written by means of its spreading function$\eta _H$ as

$$\begin{aligned} Hf (t) = \int _x \int _\xi \eta _H (x,\xi ) f (t-x) e^{2\pi i t \xi }{\mathrm{d}}\xi {\mathrm{d}}x. \end{aligned}$$

(12)

We note that two operators $H_1$, $H_2$ are equal if and only if their spreading functions coincide, see [15, 16] for details.

As shown in [15], a Gabor multiplier’s spreading function $\eta ^{\alpha ,\beta }_{{g, \mathbf{m}} }$ is given by

$$\begin{aligned} \eta ^{\alpha ,\beta }_{{g, \mathbf{m}} } (x,\xi ) = {\mathcal {M}} (x,\xi ) {\mathcal {V}} _g g(x,\xi ), \end{aligned}$$

(13)

where ${\mathcal {M}} (x,\xi )$ denotes the $(\beta ^{-1}, \alpha ^{-1})$-periodic symplectic Fourier transform of $\mathbf{m}$, i.e.,

$$\begin{aligned} {\mathcal {M}} (x,\xi ) = \mathcal {F}_s ( \mathbf{m} )(x,\xi ) = \sum _k\sum _l \mathbf{m} (k,l) e^{-2\pi i (\alpha l \xi - \beta kx )}. \end{aligned}$$

(14)

We now equally rewrite the time-averaging operation applied to a filtered signal, as defined in (6), as a Gabor multiplier. As before, we set $\check{h}_\nu (t) = \overline{h_\nu (-t)}$ and have

$$\begin{aligned} {{\text {FB}}}_{h_\nu }(f) (b,\nu )&=\sum _l |(f*h_\nu )(\alpha l)|^2 \cdot \varpi _\nu (\alpha l-b) = \sum _l |\sum _n f(n) \check{h}_\nu (n-\alpha l)|^2 \cdot \varpi _\nu (\alpha l-b)\\&=\sum _k \sum _l |\langle f, M_{\beta k} T_{\alpha l} \check{h}_\nu \rangle |^2 \cdot \varpi _\nu (\alpha l-b)\delta (\beta k)= \langle G^{\alpha ,\beta }_{\check{h}_{\nu }, \mathbf{m}_F} f, f\rangle . \end{aligned}$$

with $\mathbf{m}_F (k,l) = T_b \varpi _\nu (l) \delta (\beta k)$. To obtain the error estimate in Corollary 1, first note that by straightforward computation using the operators’ representation by their spreading functions as in (12)

$$\begin{aligned}&|{{\text {MS}}}_{g}(f) (b,\nu )-{{\text {FB}}}_{h_\nu }(f)(b,\nu )| = \left| \left\langle \left( G^{\alpha ,\beta }_{g, \mathbf{m}} - G^{\alpha ,\beta }_{\check{h}_{\nu }, \mathbf{m}_F}\right) f, f\right\rangle \right| \nonumber \\&\quad = \left| \left\langle \left( \eta _{g_\alpha ^\beta , \mathbf{m}} - \eta _{\check{h}_{\alpha \nu }^\beta , \mathbf{m}_F}\right) , {\mathcal {V}}_f f\right\rangle \right| \le \left\| \eta ^{\alpha , \beta }_{g, \mathbf{m}} - \eta ^{\alpha , \beta }_{\check{h}_{ \nu } ,\mathbf{m}_F}\right\| \cdot \Vert f\Vert _2^2 \end{aligned}$$

and we can estimate the error by the difference of the spreading functions. We write the sampled version of $\varLambda _\nu$ by using the Dirac comb Ш$_\beta$: $\varLambda _\nu (\beta k) = ($Ш$_\beta \varLambda _\nu ) (t) = \sum _k \varLambda _\nu (t) \delta (t-\beta k)$ and analogously for $\varpi _\nu$ using Ш$_\alpha$ to obtain $\mathbf{m} =T_b \delta (\alpha l) \cdot$Ш$_\beta \varLambda _\nu$ and $\mathbf{m}_F =$Ш$_\alpha T_b \varpi _\nu \cdot \delta (\beta k)$. Applying the symplectic Fourier transform (14) to $\mathbf{m}$ then gives:

Now it is a well-known fact that the Fourier transform turns sampling with sampling interval $\beta$ into periodization by $1/\beta$, in other words, into a convolution with Ш$_{\frac{1}{\beta }}$:

hence

$$\begin{aligned} {\mathcal {M}}^\nu (x,\xi ) = \sum _l T_{\frac{l}{\beta } }{\mathcal {F}}^{-1} (\varLambda _\nu ) (x) \cdot e^{-2\pi i b \xi }. \end{aligned}$$

Completely analogous considerations for $\varpi _\nu$ and Ш$_\alpha$ lead to the periodization of $\mathcal {F}(\varpi _\nu )$ and thus the following expression for the symplectic Fourier transform of $\mathbf{m}_F$:

$$\begin{aligned} {\mathcal {M}}^\nu _F(x,\xi ) = \sum _l T_{\frac{l}{\alpha } }{\mathcal {F}}(\varpi _\nu ) (\xi ) \cdot e^{-2\pi i b \xi }. \end{aligned}$$

Plugging these expressions into (13) gives the bound (8).

Remark 5

It is interesting to interpret the action of an operator in terms of its spreading function. In view of (12), we see that the spreading function determines the amount of shift in time and frequency, which the action of the operator imposes on a function. For Gabor multipliers, if well-concentrated window functions are used, it is immediately obvious that the amount of shifting is moderate as well as determined by the window’s eccentricity. At the same time, the aliasing effects introduced by coarse sub-sampling are reflected in the periodic nature of ${\mathcal {M}}$. Since, for $\mathcal {F}^{-1} (\varLambda _\nu )$ the sub-sampling density in frequency, determined by $\beta$, and for $\mathcal {F}(\varpi _\nu )$ the sub-sampling density in time, determined by $\alpha$, determine the amount of aliasing, the overall approximation quality deteriorates with increasing sub-sampling factors.