Publications
2024
-
Privacy versus Emotion Preservation Trade-offs in Emotion-Preserving Speaker AnonymizationZexin Cai, Henry Li Xinyuan, Ashi Grag, and 5 more authorsIn IEEE Spoken Language Technology Workshop, 2024Advances in speech technology now allow unprecedented access to personally identifiable information through speech. To protect such information, the differential privacy field has explored ways to anonymize speech while preserving its utility, including linguistic and paralinguistic aspects. However, anonymizing speech while maintaining emotional state remains challenging. We explore this problem in the context of the VoicePrivacy 2024 challenge. Specifically, we developed various speaker anonymization pipelines and find that approaches either excel at anonymization or preserving emotion state, but not both simultaneously. Achieving both would require an in-domain emotion recognizer. Additionally, we found that it is feasible to train a semi-effective speaker verification system using only emotion representations, demonstrating the challenge of separating these two modalities.
@inproceedings{CAI2024privacy, author = {Cai, Zexin and Li Xinyuan, Henry and Grag, Ashi and Garc\'ia-Perera, Leibny Paola and Duh, Kevin and Khudanpur, Sanjeev and Andrews, Nicholas and Wiesner, Matthew}, booktitle = {IEEE Spoken Language Technology Workshop}, title = {Privacy versus Emotion Preservation Trade-offs in Emotion-Preserving Speaker Anonymization}, year = {2024}, volume = {}, number = {}, pages = {}, } -
Integrating Frame-Level Boundary Detection and Deepfake Detection for Locating Manipulated Regions in Partially Spoofed Audio Forgery AttacksComputer Speech & Language, 2024Partially fake audio, a variant of deep fake that involves manipulating audio utterances through the incorporation of fake or externally-sourced bona fide audio clips, constitutes a growing threat as an audio forgery attack impacting both human and artificial intelligence applications. Researchers have recently developed valuable databases to aid in the development of effective countermeasures against such attacks. While existing countermeasures mainly focus on identifying partially fake audio at the level of entire utterances or segments, this paper introduces a paradigm shift by proposing frame-level systems. These systems are designed to detect manipulated utterances and pinpoint the specific regions within partially fake audio where the manipulation occurs. Our approach leverages acoustic features extracted from large-scale self-supervised pre-training models, delivering promising results evaluated on diverse, publicly accessible databases. Additionally, we study the integration of boundary and deepfake detection systems, exploring their potential synergies and shortcomings. Importantly, our techniques have yielded impressive results. We have achieved state-of-the-art performance on the test dataset of the Track 2 of ADD 2022 challenge with an equal error rate of 4.4%. Furthermore, our methods exhibit remarkable performance in locating manipulated regions in Track 2 of the ADD 2023 challenge, resulting in a final ADD score of 0.6713 and securing the top position.
@article{CAI2024integrating, title = {Integrating Frame-Level Boundary Detection and Deepfake Detection for Locating Manipulated Regions in Partially Spoofed Audio Forgery Attacks}, journal = {Computer Speech & Language}, volume = {85}, pages = {101597}, year = {2024}, issn = {0885-2308}, doi = {https://doi.org/10.1016/j.csl.2023.101597}, author = {Cai, Zexin and Li, Ming}, } -
INVERTIBLE VOICE CONVERSION WITH PARALLEL DATAIn IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024This paper introduces an innovative deep learning framework for parallel voice conversion to mitigate inherent risks associated with such systems. Our approach focuses on developing an invertible model capable of countering potential spoofing threats. Specifically, we present a conversion model that allows for the retrieval of source voices, thereby facilitating the identification of the source speaker. This framework is constructed using a series of invertible modules composed of affine coupling layers to ensure the reversibility of the conversion process. We conduct comprehensive training and evaluation of the proposed framework using parallel training data. Our experimental results reveal that this approach achieves comparable performance to non-invertible systems in voice conversion tasks. Notably, the converted outputs can be seamlessly reverted to the original source inputs using the same parameters employed during the forwarding process. This advancement holds considerable promise for elevating the security and reliability of voice conversion.
@inproceedings{CAI2024invertible, author = {Cai, Zexin and Li, Ming}, booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, title = {INVERTIBLE VOICE CONVERSION WITH PARALLEL DATA}, year = {2024}, volume = {}, number = {}, pages = {}, }
2023
-
Waveform Boundary Detection for Partially Spoofed AudioIn IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023The present paper proposes a waveform boundary detection system for audio spoofing attacks containing partially manipulated segments. Partially spoofed/fake audio, where part of the utterance is replaced, either with synthetic or natural audio clips, has recently been reported as one scenario of audio deepfakes. As deepfakes can be a threat to social security, the detection of such spoofing audio is essential. Accordingly, we propose to address the problem with a deep learning-based frame-level detection system that can detect partially spoofed audio and locate the manipulated pieces. Our proposed method is trained and evaluated on data provided by the ADD2022 Challenge. We evaluate our detection model concerning various acoustic features and network configurations. As a result, our detection system achieves an equal error rate (EER) of 6.58% on the ADD2022 challenge test set, which is the best performance in partially spoofed audio detection systems that can locate manipulated clips.
@inproceedings{CAI2023waveform, author = {Cai, Zexin and Wang, Weiqing and Li, Ming}, booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, title = {Waveform Boundary Detection for Partially Spoofed Audio}, year = {2023}, volume = {}, number = {}, pages = {1-5}, doi = {10.1109/ICASSP49357.2023.10094774}, } -
Cross-Lingual Multi-Speaker Speech Synthesis with Limited Bilingual Training DataComputer Speech & Language, 2023Modeling voices for multiple speakers and multiple languages with one speech synthesis system has been a challenge for a long time, especially in low-resource cases. This paper presents two approaches to achieve cross-lingual multi-speaker text-to-speech (TTS) and code-switching synthesis under two training scenarios: (1) cross-lingual synthesis with sufficient data, (2) cross-lingual synthesis with limited data per speaker. Accordingly, a novel TTS synthesis model and a non-autoregressive multi-speaker voice conversion model are proposed. The TTS model designed for sufficient-data cases has a Tacotron-based structure that uses shared phonemic representations associated with numeric language ID codes. As for the data-limited scenario, we adopt a framework cascading several speech modules to achieve our goal. In particular, we proposed a non-autoregressive many-to-many voice conversion module to address multi-speaker synthesis for data-insufficient cases. Experimental results on speaker similarity show that our proposed voice conversion module can maintain the voice characteristics well in data-limited cases. Both approaches use limited bilingual data and demonstrate impressive performance in cross-lingual synthesis, which can deliver fluent foreign speech and even code-switching speech for monolingual speakers.
@article{CAI2023cross, title = {Cross-Lingual Multi-Speaker Speech Synthesis with Limited Bilingual Training Data}, journal = {Computer Speech & Language}, volume = {77}, pages = {101427}, year = {2023}, issn = {0885-2308}, doi = {https://doi.org/10.1016/j.csl.2022.101427}, author = {Cai, Zexin and Yang, Yaogen and Li, Ming}, } -
Electrolaryngeal Speech Enhancement Based on A Two Stage Framework with Bottleneck Feature Refinement and Voice ConversionYaogen Yang, Haozhe Zhang, Zexin Cai, and 6 more authorsBiomedical Signal Processing and Control, 2023An electrolarynx (EL) is a medical device that generates speech for people who lost their biological larynx. However, EL speech signals are unnatural and unintelligible due to the monotonous pitch and the mechanical excitation of the EL device. This paper proposes an end-to-end voice conversion method to enhance EL speech. We adopt a speaker-independent automatic speech recognition model to extract bottleneck features as the intermediate phonetic features for enhancement. Our system includes two stages: the bottleneck feature vectors of the EL speech are mapped by a parallel non-autoregressive model to the corresponding feature vectors of the normal speech in stage one. Then another voice conversion model maps normal speech’s bottleneck feature vectors directly to normal speech’s Mel-spectrogram in stage two, followed by a MelGAN-based vocoder to convert the Mel-spectrogram into waveform. In addition, we incorporate data augmentation and transfer learning to improve conversion performance. Experimental results show that the proposed method outperforms our baseline methods and performs well in terms of naturalness and intelligibility. The audio samples are available online.
@article{YANG2023Electrolaryngeal, title = {Electrolaryngeal Speech Enhancement Based on A Two Stage Framework with Bottleneck Feature Refinement and Voice Conversion}, journal = {Biomedical Signal Processing and Control}, volume = {80}, pages = {104279}, year = {2023}, issn = {1746-8094}, doi = {https://doi.org/10.1016/j.bspc.2022.104279}, author = {Yang, Yaogen and Zhang, Haozhe and Cai, Zexin and Shi, Yao and Li, Ming and Zhang, Dong and Ding, Xiaojun and Deng, Jianhua and Wang, Jie} } -
Identifying Source Speakers for Voice Conversion Based Spoofing Attacks on Speaker Verification SystemsIn IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023An automatic speaker verification system aims to verify the speaker identity of a speech signal. However, a voice conversion system could manipulate a person’s speech signal to make it sound like another speaker’s voice and deceive the speaker verification system. Most countermeasures for voice conversion-based spoofing attacks are designed to discriminate bona fide speech from spoofed speech for speaker verification systems. In this paper, we investigate the problem of source speaker identification – inferring the identity of the source speaker given the voice converted speech. To perform source speaker identification, we simply add voice-converted speech data with the label of source speaker identity to the genuine speech dataset during speaker embedding network training. Experimental results show the feasibility of source speaker identification when training and testing with converted speeches from the same voice conversion model(s). In addition, our results demonstrate that having more converted utterances from various voice conversion model for training helps improve the source speaker identification performance on converted utterances from unseen voice conversion models.
@inproceedings{cai2023identifying, author = {Cai, Danwei and Cai, Zexin and Li, Ming}, booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, title = {Identifying Source Speakers for Voice Conversion Based Spoofing Attacks on Speaker Verification Systems}, year = {2023}, volume = {}, number = {}, pages = {}, doi = {10.1109/ICASSP49357.2023.10096733}, }
2022
-
SIG-VC: A Speaker Information Guided Zero-Shot Voice Conversion System for Both Human Beings and MachinesHaozhe Zhang, Zexin Cai, Xiaoyi Qin, and 1 more authorIn IEEE International Conference on Acoustics, Speech and Signal Processing, 2022Nowadays, as more and more systems achieve good performance in traditional voice conversion (VC) tasks, people’s attention gradually turns to VC tasks under extreme conditions. In this paper, we propose a novel method for zero-shot voice conversion. We aim to obtain intermediate representations for speaker-content disentanglement of speech to better remove speaker information and get pure content information. Accordingly, our proposed framework contains a module that removes the speaker information from the acoustic feature of the source speaker. Moreover, speaker information control is added to our system to maintain the voice cloning performance. The proposed system is evaluated by subjective and objective metrics. Results show that our proposed system significantly reduces the trade-off problem in zero-shot voice conversion, while it also manages to have high spoofing power to the speaker verification system.
@inproceedings{zhang2022sigvc, author = {Zhang, Haozhe and Cai, Zexin and Qin, Xiaoyi and Li, Ming}, booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing}, title = {SIG-VC: A Speaker Information Guided Zero-Shot Voice Conversion System for Both Human Beings and Machines}, year = {2022}, volume = {}, number = {}, pages = {6567-6571}, doi = {10.1109/ICASSP43922.2022.9746048} }
2020
-
From Speaker Verification to Multispeaker Speech Synthesis, Deep Transfer with Feedback ConstraintIn Conference of the International Speech Communication Association (INTERSPEECH), 2020High-fidelity speech can be synthesized by end-to-end text-to-speech models in recent years. However, accessing and controlling speech attributes such as speaker identity, prosody, and emotion in a text-to-speech system remains a challenge. This paper presents a system involving feedback constraints for multispeaker speech synthesis. We manage to enhance the knowledge transfer from the speaker verification to the speech synthesis by engaging the speaker verification network. The constraint is taken by an added loss related to the speaker identity, which is centralized to improve the speaker similarity between the synthesized speech and its natural reference audio. The model is trained and evaluated on publicly available datasets. Experimental results, including visualization on speaker embedding space, show significant improvement in terms of speaker identity cloning in the spectrogram level. In addition, synthesized samples are available online for listening.
@inproceedings{cai2020from, author = {Cai, Zexin and Zhang, Chuxiong and Li, Ming}, title = {{From Speaker Verification to Multispeaker Speech Synthesis, Deep Transfer with Feedback Constraint}}, year = {2020}, booktitle = {Conference of the International Speech Communication Association (INTERSPEECH)}, pages = {3974--3978}, doi = {10.21437/Interspeech.2020-1032}, }
2019
-
Polyphone Disambiguation for Mandarin Chinese Using Conditional Neural Network with Multi-level Embedding FeaturesIn Conference of the International Speech Communication Association (INTERSPEECH), 2019This paper describes a conditional neural network architecture for Mandarin Chinese polyphone disambiguation. The system is composed of a bidirectional recurrent neural network component acting as a sentence encoder to accumulate the context correlations, followed by a prediction network that maps the polyphonic character embeddings along with the conditions to corresponding pronunciations. We obtain the word-level condition from a pre-trained word-to-vector lookup table. One goal of polyphone disambiguation is to address the homograph problem existing in the front-end processing of Mandarin Chinese text-to-speech system. Our system achieves an accuracy of 94.69% on a publicly available polyphonic character dataset. To further validate our choices on the conditional feature, we investigate polyphone disambiguation systems with multi-level conditions respectively. The experimental results show that both the sentence-level and the word-level conditional embedding features are able to attain good performance for Mandarin Chinese polyphone disambiguation.
@inproceedings{cai2019polyphone, author = {Cai, Zexin and Zhang, Chuxiong and Li, Ming}, title = {{Polyphone Disambiguation for Mandarin Chinese Using Conditional Neural Network with Multi-level Embedding Features}}, year = {2019}, booktitle = {Conference of the International Speech Communication Association (INTERSPEECH)}, pages = {2110--2114}, doi = {10.21437/Interspeech.2019-1235}, } -
F0 Contour Estimation Using Phonetic Feature in Electrolaryngeal Speech EnhancementIn IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019Pitch plays a significant role in understanding a tone based language like Mandarin. In this paper, we present a new method that estimates F0 contour for electrolaryngeal (EL) speech enhancement in Mandarin. Our system explores the usage of phonetic feature to improve the quality of EL speech. First, we train an acoustic model for EL speech and generate the phoneme posterior probabilities feature sequence for each input EL speech utterance. Then we employ the phonetic feature for F0 contour generation rather than the acoustic feature. The experimental results indicate that the EL speech is significantly enhanced under the adoption of the phonetic feature. Experimental results demonstrate that the proposed method achieves notable improvement regarding the intelligibility and the similarity with normal speech.
@inproceedings{cai2019f0contour, author = {Cai, Zexin and Xu, Zhicheng and Li, Ming}, booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, title = {F0 Contour Estimation Using Phonetic Feature in Electrolaryngeal Speech Enhancement}, year = {2019}, volume = {}, number = {}, pages = {6490-6494}, doi = {10.1109/ICASSP.2019.8683435}, }
2018
-
The DKU-JNU-EMA Electromagnetic Articulography Database on Mandarin and Chinese Dialects with Tandem Feature Based Acoustic-to-Articulatory InversionZexin Cai, Xiaoyi Qin, Danwei Cai, and 3 more authorsIn International Symposium on Chinese Spoken Language Processing (ISCSLP), 2018This paper presents the acquisition of the Duke Kunshan University Jinan University Electromagnetic Articulography (DKU-JNU-EMA) database in terms of aligned acoustics and articulatory data on Mandarin and Chinese dialects. This database currently includes data from multiple individuals in Mandarin and three Chinese dialects, namely Cantonese, Hakka, Teochew. There are 2–7 native speakers for each language or dialect. Acoustic data is obtained by one headmounted close talk microphone while articulatory data is obtained by the NDI electromagnetic articulography wave research system. The DKU-JNU-EMA database is now in preparation for public release to help advance research in areas of acoustic-to-articulatory inversion, speech production, dialect recognition, and experimental phonetics. Along with the database, we propose an acoustic-to-articulatory inversion baseline using deep neural networks. Moreover, we show that by concatenating the dimension reduced phoneme posterior probability feature with MFCC features at the feature level as tandem feature, the inversion system performance is enhanced.
@inproceedings{cai2018ema, author = {Cai, Zexin and Qin, Xiaoyi and Cai, Danwei and Li, Ming and Liu, Xinzhong and Zhong, Haibin}, booktitle = {International Symposium on Chinese Spoken Language Processing (ISCSLP)}, title = {The DKU-JNU-EMA Electromagnetic Articulography Database on Mandarin and Chinese Dialects with Tandem Feature Based Acoustic-to-Articulatory Inversion}, year = {2018}, volume = {}, number = {}, pages = {235-239}, doi = {10.1109/ISCSLP.2018.8706629}, } -
Insights in-to-End Learning Scheme for Language IdentificationWeicheng Cai, Zexin Cai, Wenbo Liu, and 2 more authorsIn IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018A novel interpretable end-to-end learning scheme for language identification is proposed. It is in line with the classical GMM i-vector methods both theoretically and practically. In the end-to-end pipeline, a general encoding layer is employed on top of the frontend CNN, so that it can encode the variable-length input sequence into an utterance level vector automatically. After comparing with the state-of-the-art GMM i-vector methods, we give insights into CNN, and reveal its role and effect in the whole pipeline. We further introduce a general encoding layer, illustrating the reason why they might be appropriate for language identification. We elaborate on several typical encoding layers, including a temporal average pooling layer, a recurrent encoding layer and a novel learnable dictionary encoding layer. We conducted experiment on NIST LRE07 closed-set task, and the results show that our proposed end-to-end systems achieve state-of-the-art performance.
@inproceedings{cai2018insights, author = {Cai, Weicheng and Cai, Zexin and Liu, Wenbo and Wang, Xiaoqi and Li, Ming}, booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, title = {Insights in-to-End Learning Scheme for Language Identification}, year = {2018}, volume = {}, number = {}, pages = {5209-5213}, doi = {10.1109/ICASSP.2018.8462026}, } -
A Novel Learnable Dictionary Encoding Layer for End-to-End Language IdentificationWeicheng Cai, Zexin Cai, Xiang Zhang, and 2 more authorsIn IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018A novel learnable dictionary encoding layer is proposed in this paper for end-to-end language identification. It is inline with the conventional GMM i-vector approach both theoretically and practically. We imitate the mechanism of traditional GMM training and Supervector encoding procedure on the top of CNN. The proposed layer can accumulate high-order statistics from variable-length input sequence and generate an utterance level fixed-dimensional vector representation. Unlike the conventional methods, our new approach provides an end-to-end learning framework, where the inherent dictionary are learned directly from the loss function. The dictionaries and the encoding representation for the classifier are learned jointly. The representation is orderless and therefore appropriate for language identification. We conducted a preliminary experiment on NIST LRE07 closed-set task, and the results reveal that our proposed dictionary encoding layer achieves significant error reduction comparing with the simple average pooling.
@inproceedings{cai2018novel, author = {Cai, Weicheng and Cai, Zexin and Zhang, Xiang and Wang, Xiaoqi and Li, Ming}, booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, title = {A Novel Learnable Dictionary Encoding Layer for End-to-End Language Identification}, year = {2018}, volume = {}, number = {}, pages = {5189-5193}, doi = {10.1109/ICASSP.2018.8462025}, } -
Unsupervised Query by Example Spoken Term Detection Using Features Concatenated with Self-Organizing Map DistancesIn International Symposium on Chinese Spoken Language Processing (ISCSLP), 2018In the task of the unsupervised query by example spoken term detection (QbE-STD), we concatenate the features extracted by a Self-Organizing Map (SOM) and features learned by an unsupervised GMM based model at the feature level to enhance the performance. More specifically, The SOM features are represented by the distances between the current feature vector and the weight vectors of SOM neurons learned in an unsupervised manner. After fetching these features, we apply sub-sequence Dynamic Time Warping (S-DTW) to detect the occurrences of keywords in the test data. We evaluate the performance of these features on the TIMIT English database. After concatenating the SOM features and the GMM based features together, we achieve an improvement of 7.77% and 7.74% on Mean Average Precision (MAP) and P@10 on average.
@inproceedings{wu2018unsupervised, author = {Wu, Haiwei and Li, Ming and Cai, Zexin and Zhong, Haibin}, booktitle = {International Symposium on Chinese Spoken Language Processing (ISCSLP)}, title = {Unsupervised Query by Example Spoken Term Detection Using Features Concatenated with Self-Organizing Map Distances}, year = {2018}, volume = {}, number = {}, pages = {1-5}, doi = {10.1109/ISCSLP.2018.8706580}, } -
End-to-end Language Identification Using NetFV and NetVLADJinkun Chen, Weicheng Cai, Danwei Cai, and 3 more authorsIn International Symposium on Chinese Spoken Language Processing (ISCSLP), 2018In this paper, we apply the NetFV and NetVLAD layers for the end-to-end language identification task. NetFV and NetVLAD layers are the differentiable implementations of the standard Fisher Vector and Vector of Locally Aggregated Descriptors (VLAD) methods, respectively. Both of them can encode a sequence of feature vectors into a fixed dimensional vector which is very important to process those variable-length utterances. We first present the relevances and differences between the classical i-vector and the aforementioned encoding schemes. Then, we construct a flexible end-to-end framework including a convolutional neural network (CNN) architecture and an encoding layer (NetFV or NetVLAD) for the language identification task. Experimental results on the NIST LRE 2007 close-set task show that the proposed system achieves significant EER reductions against the conventional i-vector baseline and the CNN temporal average pooling system, respectively.
@inproceedings{chen2018lid, author = {Chen, Jinkun and Cai, Weicheng and Cai, Danwei and Cai, Zexin and Zhong, Haibin and Li, Ming}, booktitle = {International Symposium on Chinese Spoken Language Processing (ISCSLP)}, title = {End-to-end Language Identification Using NetFV and NetVLAD}, year = {2018}, volume = {}, number = {}, pages = {319-323}, doi = {10.1109/ISCSLP.2018.8706687}, } -
Deep Speaker Embeddings with Convolutional Neural Network on Supervector for Text-Independent Speaker RecognitionIn Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), 2018Lexical content variability in different utterances is the key challenge for text-independent speaker verification. In this paper, we investigate using supervector which has ability to reduce the impact of lexical content mismatch among different utterances for supervised speaker embedding learning. A DNN acoustic model is used to align a feature sequence to a set of senones and generate centered and normalized first order statistics supervector. Statistics vectors from similar senones are placed together and reshaped to an image to maintain the local continuity and correlation. The supervector image is then fed into residual convolutional neural network. The deep speaker embedding features are the outputs of the last hidden layer of the network and we employ a PLDA back-end for the subsequent modeling. Experimental results show that the proposed method outperforms the conventional GMM-UBM i-vector system and is complementary to the DNN-UBM i-vector system. The score level fusion system achieves 1.26% ERR and 0.260 DCF10 cost on the NIST SRE 10 extended core condition 5 task.
@inproceedings{cai2018deepspk, author = {Cai, Danwei and Cai, Zexin and Li, Ming}, booktitle = {Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)}, title = {Deep Speaker Embeddings with Convolutional Neural Network on Supervector for Text-Independent Speaker Recognition}, year = {2018}, volume = {}, number = {}, pages = {1478-1482}, doi = {10.23919/APSIPA.2018.8659595}, }
