Deep CockTail Networks

Abstract

Transferable deep representations for visual domain adaptation (DA) provides a route to learn from labeled source images to recognize target images without the aid of target-domain supervision. Relevant researches increasingly arouse a great amount of interest due to its potential industrial prospect for non-laborious annotation and remarkable generalization. However, DA presumes source images are identically sampled from a single source while Multi-Source DA (MSDA) is ubiquitous in the real-world. In MSDA, the domain shifts exist not only between source and target domains but also among the sources; especially, the multi-source and target domains may disagree on their semantics (e.g., category shifts). This issue challenges the existing solutions for MSDAs. In this paper, we propose Deep CockTail Network (DCTN), a universal and flexibly-deployed framework to address the problems. DCTN uses a multi-way adversarial learning pipeline to minimize the domain discrepancy between the target and each of the multiple in order to learn domain-invariant features. The derived source-specific perplexity scores measure how similar each target feature appears as a feature from one of source domains. The multi-source category classifiers are integrated with the perplexity scores to categorize target images. We accordingly derive a theoretical analysis towards DCTN, including the interpretation why DCTN can be successful without precisely crafting the source-specific hyper-parameters, and target expected loss upper bounds in terms of domain and category shifts. In our experiments, DCTNs have been evaluated on four benchmarks, whose empirical studies involve vanilla and three challenging category-shift transfer problems in MSDA, i.e., source-shift, target-shift and source-target-shift scenarios. The results thoroughly reveal that DCTN significantly boosts classification accuracies in MSDA and performs extraordinarily to resist negative transfers across different MSDA scenarios.

Appendices

Appendix A: Proofs

Proof

(Proof (Lemma 1)) Suppose the optimal feature extractor is $F^*$, then M source-target adversarial learning pairs can be separately considered and correspond to the optimization objectives w.r.t. $\{D_j\}^M_{j=1}$, respectively. Substitute $F^*({\varvec{x}})$ by x, and there is

$$\begin{aligned} D^{*}_{j}(x) = \frac{{\mathscr {P}}_{j}(x)}{{\mathscr {P}}_{j}(x)+{\mathscr {P}}_{t}(x)} \end{aligned}$$

(22)

is exactly derived from Theorem.1 in Goodfellow et al. (2014).

$$\begin{aligned} \begin{aligned} h_t(x)&=\sum ^{M}_{j=1}\frac{-\log \left( \frac{{\mathscr {P}}_{t}(x)}{{\mathscr {P}}_{j}(x)+{\mathscr {P}}_{t}(x)}\right) h_j(x)}{\sum ^{M}_{k=1}-\log \left( \frac{{\mathscr {P}}_{t}(x)}{{\mathscr {P}}_{j}(x)+{\mathscr {P}}_{t}(x)}\right) }\\&=\sum ^{M}_{j=1}\frac{\log \left( 1+ \frac{{\mathscr {P}}_{j}(x)}{{\mathscr {P}}_{t}(x)}\right) h_j(x)}{\sum ^{M}_{k=1}\log \left( 1+ \frac{{\mathscr {P}}_{k}(x)}{{\mathscr {P}}_{t}(x)}\right) }. \end{aligned} \end{aligned}$$

(23)

$\square$

Proof

(Proof (Proposition 2)) Given Lemma 1, it holds

$$\begin{aligned} \begin{aligned} h_t(x)&=\sum ^{M}_{j=1}\frac{\log \left( 1+ \frac{{\mathscr {P}}_{j}(x)}{{\mathscr {P}}_{t}(x)}\right) }{\sum ^{M}_{k=1}\log \left( 1+ \frac{{\mathscr {P}}_{k}(x)}{{\mathscr {P}}_{t}(x)}\right) }h_j(x)\\&\le \sum ^{M}_{j=1}\frac{\log \left( 1+ \frac{{\mathscr {P}}_{j}(x)}{{\mathscr {P}}_{t}(x)}\right) \ h_j(x)}{\sum ^{M}_{k=1}\log (1+ \alpha _k)}\\&\le \sum ^{M}_{j=1}\frac{{\mathscr {P}}_{k}(x) \ h_j(x)}{{\mathscr {P}}_{t}(x)\sum ^{M}_{k=1}\log (1+ \alpha _k)}, \end{aligned} \end{aligned}$$

(24)

To this end, given a target feature x,

$$\begin{aligned}&{\mathscr {L}}({\mathscr {P}}_t,h_t,f)(x)\nonumber \\&\quad =L\left( \sum ^{M}_{j=1}\frac{\log (1+ \frac{{\mathscr {P}}_{j}(x)}{{\mathscr {P}}_{t}(x)})}{\sum ^{M}_{k=1}\log \left( 1+ \frac{{\mathscr {P}}_{k}(x)}{{\mathscr {P}}_{t}(x)}\right) }h_j(x),f(x)\right) {\mathscr {P}}_t(x)\nonumber \\&\quad \le \sum ^{M}_{j=1}\frac{\log \left( 1+ \frac{{\mathscr {P}}_{j}(x)}{{\mathscr {P}}_{t}(x)}\right) }{\sum ^{M}_{k=1}\log (1+ \frac{{\mathscr {P}}_{k}(x)}{{\mathscr {P}}_{t}(x)})} {\mathscr {P}}_t(x)L\left( h_j(x),f(x)\right) \nonumber \\&\quad \le \sum ^{M}_{j=1}\frac{{\mathscr {P}}_{j}(x)}{{\mathscr {P}}_{t}(x)\sum ^{M}_{k=1}\log (1+ \alpha _k)} {\mathscr {P}}_t(x)L\left( h_j(x),f(x)\right) \nonumber \\&\quad = \sum ^{M}_{j=1}\frac{{\mathscr {P}}_{j}(x)}{\sum ^{M}_{k=1}\log (1+ \alpha _k)} L\left( h_j(x),f(x)\right) \end{aligned}$$

(25)

in which the first inequality is derived from the convexity of the loss function $L(\cdot ,\cdot )$. $\square$

Proof

(Proposition 3) In terms of Proposition 1, we provide the upper bound of ${\mathscr {L}}({\mathscr {P}}_t,h_t,f)$. Specifically,

$$\begin{aligned} \begin{aligned} {\mathscr {L}}({\mathscr {P}}_t,h_t,f)&=\int _{x\in {\mathscr {X}}}{\mathscr {L}}({\mathscr {P}}_t,h_t,f)(x)dx\\&\le \int _{x\in {\mathscr {X}}}\sum ^{M}_{j=1}\frac{{\mathscr {P}}_{j}(x)}{\sum ^{M}_{k=1}\log (1+ \alpha _k)} L\big (h_j(x),f(x)\big )dx\\&=\frac{1}{\sum ^{M}_{k=1}\log (1+ \alpha _k)}\sum ^{M}_{j=1}\\&\quad \int _{x\in {\mathscr {X}}}{\mathscr {P}}_{j}(x) L\big (h_j(x),f(x)\big )dx \end{aligned} \end{aligned}$$

(26)

Since $\rho$ indicates the proportion of wrongly-labeled target data by the auto-annotating strategy in the discriminative adaptation phase; $f'(x)$ represents the wrong target function w.r.t. x, namely, $\forall x\in {\mathscr {X}}$, it holds $f'(x)\ne f(x)$ and $L\big (h_j(x),f(x)\big )\le L\big (h_j(x),f'(x)\big )$. Therefore,

$$\begin{aligned} \begin{aligned} {\mathscr {L}}({\mathscr {P}}_t,h_t,f)&\le \frac{1}{\sum ^{M}_{k=1}\log (1+ \alpha _k)}\\&\quad \sum ^{M}_{j=1}\int _{x\in {\mathscr {X}}} L\big (h_j(x),f(x)\big )dx\\&= \frac{1}{\sum ^{M}_{k=1}\log (1+ \alpha _k)}\\&\quad \sum ^{M}_{j=1}\Bigg (\int _{x\in {\mathscr {X}}}{\mathscr {P}}_{j}(x)\bigg ((1-\rho ) L\big (h_j(x),f(x)\big )\\&\quad +\rho L\big (h_j(x),f(x)\big )\bigg )dx\Bigg ),\\&\le \frac{1}{\sum ^{M}_{k=1}\log (1+ \alpha _k)}\\&\quad \sum ^{M}_{j=1}\Bigg (\int _{x\in {\mathscr {X}}}{\mathscr {P}}_{j}(x)\bigg ((1-\rho ) L\big (h_j(x),f(x)\big )\\&\quad +\rho L\big (h_j(x),f'(x)\big )\bigg )dx\Bigg ),\\ \end{aligned} \end{aligned}$$

(27)

Due to the assumption of the 0-1 loss function on L, it follows the analysis in Saito et al. (2017) and holds

$$\begin{aligned} \begin{aligned}&\int _{x\in {\mathscr {X}}} {\mathscr {P}}_j(x)L\big (h_j(x),f(x)\big )dx\\&\quad \le \int _{x\in {\mathscr {X}}} {\mathscr {P}}_j(x)L\big (h_j(x),f'(x)\big )dx\\&\quad \le \int _{x\in {\mathscr {X}}} {\mathscr {P}}_j(x)dx = 1. \end{aligned} \end{aligned}$$

(28)

so that

$$\begin{aligned} \begin{aligned} {\mathscr {L}}({\mathscr {P}}_t,h_t,f)&\le \frac{1}{\sum ^{M}_{k=1}\log (1+ \alpha _k)}\big ((1-\rho )\\&\quad {\sum _{j\in [M]}\epsilon _j}+M\rho \big ) \end{aligned} \end{aligned}$$

(29)

Conclude the proof. $\square$

Proof

(Proposition 4) According to the results of Proposition.21, we have

$$\begin{aligned} \begin{aligned} {\mathscr {L}}({\mathscr {P}}_t,h_t,f)(x)&\le \frac{1}{\sum ^{M}_{k=1}\log (1+ \alpha _k)}\\&\quad \sum ^{M}_{j=1}{\mathscr {P}}_{j}(x) L\big (h_j(x),f(x)\big ). \end{aligned} \end{aligned}$$

Therefore

$$\begin{aligned} \begin{aligned} {\mathscr {L}}({\mathscr {P}}_t,h_t,f)&=\frac{1}{\sum ^{M}_{k=1}\log (1+ \alpha _k)}\\&\quad \int _{x\in {\mathscr {X}}}\sum ^{M}_{j=1}{\mathscr {P}}_{j}(x) L\big (h_j(x),f(x)\big )(x)dx\\&\le \frac{1}{\sum ^{M}_{k=1} \log (1+ \alpha _k)}\\&\quad \sum _{j\in [M]}\Bigg ((1-\rho ')\underset{x\in {\mathscr {X}}}{\int } {\mathscr {P}}_j(x)L\big (h_j(x),f(x)\big )dx\\&\quad \underset{x\in {\mathscr {X}}}{\int } {\mathscr {P}}_j(x)L\big (h_j(x),f'(x)\big )dx\Bigg )\\&\le \frac{1}{\sum ^{M}_{k=1} \log (1+ \alpha _k)}\\&\quad \sum _{j\in [M]} \Bigg ( (1-\rho ')\bigg ( (1-\rho )\underset{x\in {\mathscr {X}}}{\int } {\mathscr {P}}_t(x)L\big (h_j(x),f(x)\big )dx\\&\quad +\rho \underset{x\in {\mathscr {X}}}{\int } {\mathscr {P}}_t(x)L\big (h_j(x),f'(x)\big )dx \bigg )\\&\quad +\rho '\underset{x\in {\mathscr {X}}}{\int } {\mathscr {P}}_t(x)L\big (h_j(x),f'(x)\big )dx\Bigg ), \end{aligned} \end{aligned}$$

(30)

where the first, third and fifth inequalities are derived from the proof of Proposition 2; the second inequality is developed from the entropy-based unknown category discovery strategy, which is specified in target-category-shift and source-target-category-shift scenarios (Notice that, since the unknown class discovery is executed ahead of the pseudo-labeling strategy, it makes the the inequality w.r.t. $\rho$ nested in the inequality w.r.t. $\rho '$); the fourth inequality is derived from the 0–1 loss function upper bound discussed in Saito et al. (2017) .

Due to the assumption of the 0–1 loss function on L, it follows (Saito et al. 2017) and holds

$$\begin{aligned} \begin{aligned} {\mathscr {L}}({\mathscr {P}}_t,h_t,f)&\le \sum _{j\in [M]}\frac{1}{\sum ^{M}_{k=1} \log (1+ \alpha _k)}\Big ((1-\rho ')(1-\rho )\\&\quad \underset{x\in {\mathscr {X}}}{\int } {\mathscr {P}}_t(x)L\big (h_j(x),f(x)\big )dx+\rho (1-\rho ')+\rho '\Big )\\&\le \frac{1}{\sum ^{M}_{k=1}\log (1+ \alpha _k)}\Big ((1\,\, -\,\, \rho ')(1\,\, -\,\, \rho ){\sum _{j\in [M]}\epsilon _j}\,\,\\&\quad +\big (\,\, (1\,\, -\,\, \rho ')\rho +\rho '\big )M\Big ) \end{aligned} \end{aligned}$$

(31)

$\square$

Appendix B: Implementation details

The setup of $\gamma$ and $\zeta$ The the pseudo labeling strategies of DCTN rely on hyper-parameters $\gamma$ and $\zeta$. The threshold $\gamma$ is leveraged to select a part of target candidates, which are annotated as “high-confident” and augmented with multi-source examples to train the multi-source classifiers. We set the value over 90% to ensure the quality of selecte target samples. Instead of choosing a specific threshold $\gamma$, we rank the target examples according to their entropy values on the source-specific classifiers by a monotonically decreasing order, then choose the top 15% as the “unknown” candidates: 300/120/140 as the “unknown” candidates in A/D/W domains, respectively. This manner promises adequate “unknown” examples to train a reliable classifier for each source domain. The schemes is adopted the same in the experiment of DomainNet (Details see Table 7).

Network Implementation details For the recognitions in Office-31 and ImageCLEF-DA, existing deep DA approaches (Long et al. 2015, 2016) routinely employ Alexnet (Krizhevsky et al. 2012) as their backbones. For a fair comparison, we choose a DCTN architecture deriving from the Alexnet pipeline. As Fig. 11 illustrated, the representation module F is designed as a five-layer fully-convolutional network with three max-pooling operators, and the (multi-source) category classifier C is a three-layer fully-connected multi-task network. They are stacked into an exactly Alexnet-like pipeline to categorize examples. We adopt a CNN with a two-head classifier as domain discriminator D.

For the sake of legibility, we apply the sigmoid cross entropy loss to denote the multi-way adversarial learning inducing the perplexity score in our paper. Under M adversarial adaptation context, this loss function leads to the gradient vanishing and behaves extremely unstable during training. To overcome this issue, we replace it with the least square measure (Mao et al. 2017) in practice to ensure robust adversarial learning:

$$\begin{aligned} \begin{aligned} {\mathscr {L}}^{(ls)}_{adv}(F, D)&= \frac{1}{M}\sum _{j}^{M}{\mathbb {E}}_{{\varvec{x}}\sim {\mathbf {X}}_{j}}[(D_{j}(F({\varvec{x}})))^2] \\&\quad + {\mathbb {E}}_{{\varvec{x}}^{(t)}\sim {\mathbf {X}}_{t}}[(1- D_{j}(F({\varvec{x}}^{(t)})))^2].\\ \end{aligned} \end{aligned}$$

(32)

Accordingly, the confusion loss has been revised as

$$\begin{aligned} \ \begin{aligned} {\mathscr {L}}^{(ls)}_{cf}({\varvec{x}};F,D_{j}) = \left( D_{j}(F({\varvec{x}}))-\frac{1}{2}\right) ^2. \end{aligned} \end{aligned}$$

(33)

Then given a target instance ${\varvec{x}}^{(t)}$, a least square perplexity score is

$$\begin{aligned} \begin{aligned} s({\varvec{x}}^{(t)};F,D_{j}) = (D_{j}(F({\varvec{x}}^{(t)})))^2. \end{aligned} \end{aligned}$$

(34)

Fig. 11

The representation module, domain discriminator and category classifier we used in the experiments about object recognition. (Best viewed in color)

Fig. 12

The representation module, domain discriminator and category classifier we used in the experiments about digit recognition. (Best viewed in color)

The implementation keeps consistent with all our analysis in the paper. No matter in training or test, we need a perplexity score weighting scheme to predict the class of the target instance. While in the adversarial learning process, the domain discriminator D must be gradually trained to accommodate the learning of feature extractor F. It means that in those previous epoches, the perplexity scores are not capable of providing reliable probablistic relations between target and each source. This hurts the pseudo-labeling scheme and further spoils the adversary at the next alternative step. Empirically, this negative effect mostly attributes to the unstable predictions to target instances. Hence we utilize the moving average to calculate the perpelxity score for each target instance.

$$\begin{aligned} \begin{aligned} s({\varvec{x}}_{N_T}^{(t)};F,D_{j}) = \frac{1}{N_{T}}\sum _{i}^{N_T}(D_{j}(F({\varvec{x}}_i^{(t)})))^2, \end{aligned} \end{aligned}$$

(35)

where $N_T$ denotes the number of times that the target samples have been visited to train our model (one target mini-batch as the measurement unit); $x_{N_T}^t$ denotes the current target instance being considered.

Hyper-parameter setting of training In visual object recognition experiments (Office-31 and ImageCLEF), we initiate our DCTN by following the same way of DAN (Long et al. 2015). In terms of digit recognition, DCTN learns from scratch. In order to execute online hard domain mining, we construct our mini-batch by sampling an equal number of images per domain. For instance, consider a case of two-source domain adaptation with a domain batch size of 32. Then we have mini-batches with the sizes as $96 = 32 \times (2+1)$ (2 and 1 denote two source domains and one target domain). In this situation, the length of one epoch is decided by the size of the domain containing most instances. Finally, we adopt Adam (Kingma and Ba 2015) solver with $momentum = (0.9, 0.99)$ in all experiments to update our networks (Fig. 12).

Table 7 The hyper-parameters setting in our experiment

More hyper-parameter details are shown in Table 7.