Parallel Data Augmentation for Formality Style Transfer

ACL, pp. 3221-3228, 2020.

Cited by: 1|Bibtex|Views48|Links
EI
Keywords:
simultaneous trainingstyle transfermulti-task transferback translationparallel datumMore(18+)
Weibo:
We propose novel data augmentation methods for formality style transfer

Abstract:

The main barrier to progress in the task of Formality Style Transfer is the inadequacy of training data. In this paper, we study how to augment parallel data and propose novel and simple data augmentation methods for this task to obtain useful sentence pairs with easily accessible models and systems. Experiments demonstrate that our aug...More

Code:

Data:

0
Introduction
  • Formality style transfer (FST) is defined as the task of automatically transforming a piece of text in one particular formality style into another (Rao and Tetreault, 2018).
  • The performance of the neural network approaches is still limited by the inadequacy of training data: the public parallel corpus for FST training – GYAFC (Rao and Tetreault, 2018) – contains only approximately 100K sentence pairs, which can hardly satiate the neural models with millions of parameters.
  • To tackle the data sparsity problem for FST, the authors propose to augment parallel data with three specific data augmentation methods to help improve the model’s generalization ability and reduce the overfitting risk.
Highlights
  • Formality style transfer (FST) is defined as the task of automatically transforming a piece of text in one particular formality style into another (Rao and Tetreault, 2018)
  • The performance of the neural network approaches is still limited by the inadequacy of training data: the public parallel corpus for Formality style transfer training – GYAFC (Rao and Tetreault, 2018) – contains only approximately 100K sentence pairs, which can hardly satiate the neural models with millions of parameters
  • We study three data augmentation methods for formality style transfer: back translation, formality discrimination, and multi-task transfer
  • We focus on informal→formal style transfer since it is more practical in real application scenarios
  • According to the observation that an informal sentence tends to become a formal sentence after a round-trip translation by Machine Translation models that are mainly trained with formal text like news, we propose a novel method called formality discrimination to generate formal rewrites of informal source sentences by means of cross-lingual Machine Translation models
  • We propose novel data augmentation methods for formality style transfer
Methods
  • The authors present the experimental settings and related experimental results.
  • The authors use GYAFC benchmark dataset (Rao and Tetreault, 2018) for training and evaluation.
  • GYAFC’s training split contains a total of 110K annotated informal-formal parallel sentences, which are annotated via crowd-sourcing of two domains: Entertainment & Music (E&M) and Family & Relationships (F&R).
  • There are 1,146 and 1,332 informal sentences in E&M and F&R domain respectively and each informal sentence has 4 referential formal rewrites.
  • The authors use all the three data augmentation methods the authors introduced and obtain a total of 4.9M augmented pairs.
  • 1.6M are generated by back-translating (BT) formal sentences identified by the formality discriminator in E&M and F&R domain on Yahoo
Results
  • 3.2.1 Effect of Proposed Approach

    Table 1 compares the results of the models trained with simultaneous training (ST) and pre-training & fine-tuning (PT&FT).
  • 3.2.1 Effect of Proposed Approach.
  • Table 1 compares the results of the models trained with simultaneous training (ST) and pre-training & fine-tuning (PT&FT).
  • ST with the augmented and original data leads to a performance decline, because the noisy augmented data cannot achieve desirable performance by itself and may distract the model from exploiting the original data in simultaneous training.
  • PT&FT only uses.
  • Pre-training & Fine-tuning + BT + F-Dis + M-Task.
  • + BT + M-Task + F-Dis 72.63 77.01 System
Conclusion
  • The authors propose novel data augmentation methods for formality style transfer.
  • The authors' proposed data augmentation methods can effectively generate diverse augmented data with various formality style transfer knowledge.
  • The augmented data can significantly help improve the performance when it is used for pre-training the model and leads to the state-of-the-art results in the formality style transfer benchmark dataset
Summary
  • Introduction:

    Formality style transfer (FST) is defined as the task of automatically transforming a piece of text in one particular formality style into another (Rao and Tetreault, 2018).
  • The performance of the neural network approaches is still limited by the inadequacy of training data: the public parallel corpus for FST training – GYAFC (Rao and Tetreault, 2018) – contains only approximately 100K sentence pairs, which can hardly satiate the neural models with millions of parameters.
  • To tackle the data sparsity problem for FST, the authors propose to augment parallel data with three specific data augmentation methods to help improve the model’s generalization ability and reduce the overfitting risk.
  • Methods:

    The authors present the experimental settings and related experimental results.
  • The authors use GYAFC benchmark dataset (Rao and Tetreault, 2018) for training and evaluation.
  • GYAFC’s training split contains a total of 110K annotated informal-formal parallel sentences, which are annotated via crowd-sourcing of two domains: Entertainment & Music (E&M) and Family & Relationships (F&R).
  • There are 1,146 and 1,332 informal sentences in E&M and F&R domain respectively and each informal sentence has 4 referential formal rewrites.
  • The authors use all the three data augmentation methods the authors introduced and obtain a total of 4.9M augmented pairs.
  • 1.6M are generated by back-translating (BT) formal sentences identified by the formality discriminator in E&M and F&R domain on Yahoo
  • Results:

    3.2.1 Effect of Proposed Approach

    Table 1 compares the results of the models trained with simultaneous training (ST) and pre-training & fine-tuning (PT&FT).
  • 3.2.1 Effect of Proposed Approach.
  • Table 1 compares the results of the models trained with simultaneous training (ST) and pre-training & fine-tuning (PT&FT).
  • ST with the augmented and original data leads to a performance decline, because the noisy augmented data cannot achieve desirable performance by itself and may distract the model from exploiting the original data in simultaneous training.
  • PT&FT only uses.
  • Pre-training & Fine-tuning + BT + F-Dis + M-Task.
  • + BT + M-Task + F-Dis 72.63 77.01 System
  • Conclusion:

    The authors propose novel data augmentation methods for formality style transfer.
  • The authors' proposed data augmentation methods can effectively generate diverse augmented data with various formality style transfer knowledge.
  • The augmented data can significantly help improve the performance when it is used for pre-training the model and leads to the state-of-the-art results in the formality style transfer benchmark dataset
Tables
  • Table1: The comparison of simultaneous training (ST) and Pre-train & Fine-tuning (PT&FT). Down-sampling and up-sampling are for balancing the size of the augmented data and the original data. Specifically, downsampling samples augmented data, while up-sampling increases the frequency of the original data
  • Table2: The comparison of different data augmentation methods for FST
  • Table3: The comparison of our approach to the stateof-the-art results. * denotes the ensemble results
  • Table4: Results of human evaluation of FST. Scores marked with */† are significantly different from the scores of Original data / NMT-MTL (p < 0.05 in significance test)
  • Table5: The sizes of augmented datasets generated by F-Dis based on different pivot languages
  • Table6: Performances of formality discrimination based on different pivot languages: French (Fr), German (De) and Chinese (Zh)
Download tables as Excel
Related work
Funding
  • This work is partly supported by Beijing Academy of Artificial Intelligence
Reference
  • Albert C Baugh and Thomas Cable. 1993. A history of the English language. Routledge.
    Google ScholarFindings
  • Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. CoRR, abs/1406.1078.
    Findings
  • Daniel Dahlmeier, Hwee Tou Ng, and Siew Mei Wu. 201Building a large annotated corpus of learner english: The nus corpus of learner english. In Proceedings of the eighth workshop on innovative use of NLP for building educational applications, pages 22–31.
    Google ScholarLocate open access versionFindings
  • Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. arXiv preprint arXiv:1808.09381.
    Findings
  • Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 2: Short Papers, pages 567–573.
    Google ScholarLocate open access versionFindings
  • Tao Ge, Furu Wei, and Ming Zhou. 2018a. Fluency boost learning and inference for neural grammatical error correction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1055–1065.
    Google ScholarLocate open access versionFindings
  • Tao Ge, Furu Wei, and Ming Zhou. 2018b. Reaching human-level performance in automatic grammatical error correction: An empirical study. arXiv preprint arXiv:1807.01270.
    Findings
  • Tao Ge, Xingxing Zhang, Furu Wei, and Ming Zhou. 2019. Automatic grammatical error correction for sequence-to-sequence text generation: An empirical study. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6059–6064.
    Google ScholarLocate open access versionFindings
  • Shun Kiyono, Jun Suzuki, Masato Mita, Tomoya Mizumoto, and Kentaro Inui. 201An empirical study of incorporating pseudo data into grammatical error correction. arXiv preprint arXiv:1909.00502.
    Findings
  • Rumeng Li, Xun Wang, and Hong Yu. 2019.
    Google ScholarFindings
  • Fuli Luo, Peng Li, Jie Zhou, Pengcheng Yang, Baobao Chang, Xu Sun, and Zhifang Sui. 2019. A dual reinforcement learning framework for unsupervised text style transfer. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 5116–5122. ijcai.org.
    Google ScholarLocate open access versionFindings
  • Tomoya Mizumoto, Mamoru Komachi, Masaaki Nagata, and Yuji Matsumoto. 2011. Mining revision log of language learning sns for automated japanese error correction of second language learners. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 147–155.
    Google ScholarLocate open access versionFindings
  • Xing Niu, Sudha Rao, and Marine Carpuat. 2018. Multi-task neural models for translating between styles within and across languages. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018, pages 1008– 1021.
    Google ScholarLocate open access versionFindings
  • Alberto Poncelas, Dimitar Shterionov, Andy Way, Gideon Maillette de Buy Wenniger, and Peyman Passban. 2018. Investigating backtranslation in neural machine translation. CoRR, abs/1804.06189.
    Findings
  • Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9.
    Google ScholarLocate open access versionFindings
  • Roman Grundkiewicz, Marcin Junczys-Dowmunt, and Kenneth Heafield. 2019. Neural grammatical error correction systems with unsupervised pre-training on synthetic data. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 252–263.
    Google ScholarLocate open access versionFindings
  • Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016. Dual learning for machine translation. In Advances in Neural Information Processing Systems, pages 820–828.
    Google ScholarLocate open access versionFindings
  • Harsh Jhamtani, Varun Gangal, Eduard H. Hovy, and Eric Nyberg. 2017. Shakespearizing modern language using copy-enriched sequence-to-sequence models. CoRR, abs/1707.01161.
    Findings
  • Alina Karakanta, Jon Dehdari, and Josef van Genabith. 2018. Neural machine translation for low-resource languages without parallel corpora. Machine Translation, 32(1-2):167–189.
    Google ScholarLocate open access versionFindings
  • Sudha Rao and Joel R. Tetreault. 2018. Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 129–140.
    Google ScholarLocate open access versionFindings
  • Marek Rei, Mariano Felice, Zheng Yuan, and Ted Briscoe. 2017. Artificial error generation with machine translation and syntactic patterns. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, BEA@EMNLP 2017, Copenhagen, Denmark, September 8, 2017, pages 287–292.
    Google ScholarLocate open access versionFindings
  • Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proceedings of the
    Google ScholarLocate open access versionFindings
  • 54th Annual Meeting of the Association for Compu- Ruochen Xu, Tao Ge, and Furu Wei. 2019. Formality tational Linguistics, ACL 2016, August 7-12, 2016, style transfer with hybrid textual annotations. CoRR, Berlin, Germany, Volume 1: Long Papers.
    Google ScholarLocate open access versionFindings
  • Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers.
    Google ScholarLocate open access versionFindings
  • Wei Xu, Alan Ritter, Bill Dolan, Ralph Grishman, and Colin Cherry. 2012. Paraphrasing for style. In COLING 2012, 24th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, 8-15 December 2012, Mumbai, India, pages 2899–2914.
    Google ScholarLocate open access versionFindings
  • Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3104–3112.
    Google ScholarLocate open access versionFindings
  • Toshikazu Tajiri, Mamoru Komachi, and Yuji Matsumoto. 2012. Tense and aspect error correction for esl learners using global context. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2, pages 198–202. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008.
    Google ScholarLocate open access versionFindings
  • Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig. 2018. Switchout: an efficient data augmentation algorithm for neural machine translation. CoRR, abs/1808.07512.
    Findings
  • Yunli Wang, Yu Wu, Lili Mou, Zhoujun Li, and Wenhan Chao. 2019. Harnessing pre-trained neural networks with rules for formality style transfer. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3571– 3576, Hong Kong, China. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Chen Wu, Xuancheng Ren, Fuli Luo, and Xu Sun. 2019. A hierarchical reinforced sequence operation method for unsupervised text style transfer. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4873–4883. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Ziang Xie, Guillaume Genthial, Stanley Xie, Andrew Ng, and Dan Jurafsky. 2018. Noising and denoising natural language: Diverse backtranslation for grammar correction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 619–628.
    Google ScholarLocate open access versionFindings
  • Zheng Yuan, Ted Briscoe, and Mariano Felice. 2016. Candidate re-ranking for smt-based grammatical error correction. In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, BEA@NAACL-HLT 2016, June 16, 2016, San Diego, California, USA, pages 256– 266.
    Google ScholarLocate open access versionFindings
  • Yi Zhang, Jingjing Xu, Pengcheng Yang, and Xu Sun. 2018a. Learning sentiment memories for sentiment modification without parallel data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 1103–1108. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Zhirui Zhang, Shujie Liu, Mu Li, Ming Zhou, and Enhong Chen. 2018b. Joint training for neural machine translation models with monolingual data. In ThirtySecond AAAI Conference on Artificial Intelligence.
    Google ScholarLocate open access versionFindings
  • Wei Zhao, Liang Wang, Kewei Shen, Ruoyu Jia, and Jingming Liu. 2019. Improving grammatical error correction via pre-training a copy-augmented architecture with unlabeled data. arXiv preprint arXiv:1903.00138.
    Findings
  • Wangchunshu Zhou, Tao Ge, Chang Mu, Ke Xu, Furu Wei, and Ming Zhou. 2019. Improving grammatical error correction with machine translation pairs. arXiv preprint arXiv:1911.02825.
    Findings
Your rating :
0

 

Tags
Comments