Time series anomaly detection has attracted extensive research attention owing to its real-world applications. Existing deep learning based anomaly detectors usually require a separate training phase for each dataset. However, the long training time restricts their practicality in the industry use. To address this limitation, we propose a novel deep learning based discord search method named DeepDiscord, which is a multi-scale anomaly detector capable of directly examining unseen datasets after pre-training. To the best of our knowledge, our study is the first to introduce contrastive learning in the discord search, in order to provide a flexible and effective similarity measure for various kinds of data. We innovatively divide the data into two categories according to their roles in discord search, and combine dual learning with contrastive learning, which improves the efficiency and efficacy of discord search. Furthermore, a novel pretext task is proposed based on our dual contrastive learning setting. We evaluate DeepDiscord comprehensively on five anomaly detection benchmarks. Experimental results show that DeepDiscord achieves the state-of-the-art results on the four out of five benchmarks.