Privacy versus Emotion Preservation Trade-offs in
Emotion-Preserving Speaker Anonymization

Zexin Cai, Henry Li Xinyuan, Ashi Garg, Leibny Paola Garc'ia-Perera, Kevin Duh, Sanjeev Khudanpur, Nicholas Andrews, Matthew Wiesner

Advances in speech technology now allow unprecedented access to personally identifiable information through speech. To protect such information, the differential privacy field has explored ways to anonymize speech while preserving its utility, including linguistic and paralinguistic aspects. However, anonymizing speech while maintaining emotional state remains challenging. We explore this problem in the context of the VoicePrivacy 2024 challenge. Specifically, we developed various speaker anonymization pipelines and find that approaches either excel at anonymization or preserving emotion state, but not both simultaneously. Achieving both would require an in-domain emotion recognizer. Additionally, we found that it is feasible to train a semi-effective speaker verification system using only emotion representations, demonstrating the challenge of separating these two modalities.

Paper is accepted by 2024 IEEE Spoken Language Technology Workshop

Anonymization Samples

Origin The original, unanonymized speech.
ConVec2Mel-VC The speaker embedding-conditioned VC system we developed. During anonymization, the target embedding is extracted from a randomly selected utterance from LibriTTS.
kNN-VC The kNN-based VC method. For each utterance, the WavLM feature pool is obtained from a randomly chosen target speaker in LibriTTS, with the target pool comprising at least 5 minutes of audio.
ConVec2Mel-VC-XTTS This system utilizes the cascaded ASR-TTS method. For each source utterance, the prompt speech is the corresponding anonymized speech from ConVec2Mel-VC.
kNN-VC-XTTS This system follows the cascaded ASR-TTS approach. For each source utterance, the prompt speech is the corresponding anonymized speech from the kNN-VC system.
XTTS The cascaded ASR-TTS anonymization method, where the prompt utterance during inference is randomly chosen from the LibriTTS dataset.
EmoMSP-XTTS The system that employs the emotion-proxy anonymization strategy. This system is based on the out-of-domain emotion recognizer.
EmoIEMOCAP-XTTS The system that employs the emotion-proxy anonymization strategy. This system is based on the in-domain emotion recognizer.