Expertise Style Transfer: A New Task Towards Better Communication between Experts and Laymen

ACL, pp. 1061-1071, 2020.

Cited by: 0|Bibtex|Views200|Links
EI
Keywords:
expertise style transferexpertise levelstyle transferUnified Medical Language Systemautomatic evaluationMore(11+)
Weibo:
We proposed a practical task of expertise style transfer and constructed a high-quality dataset, MSD

Abstract:

The curse of knowledge can impede communication between experts and laymen. We propose a new task of expertise style transfer and contribute a manually annotated dataset with the goal of alleviating such cognitive biases. Solving this task not only simplifies the professional language, but also improves the accuracy and expertise level ...More

Code:

Data:

0
Introduction
  • The curse of knowledge (Camerer et al, 1989) is a pervasive cognitive bias exhibited across all domains, leading to discrepancies between an expert’s advice and a layman’s understanding of it (Tan and Goonawardene, 2017).
  • Take medical consultations as an example: patients often find it difficult to understand their doctors’ language.
  • The authors propose a new task of text style transfer between expert language and layman language, namely Expertise Style Transfer, and contribute a manually annotated dataset in the medical
Highlights
  • The curse of knowledge (Camerer et al, 1989) is a pervasive cognitive bias exhibited across all domains, leading to discrepancies between an expert’s advice and a layman’s understanding of it (Tan and Goonawardene, 2017)
  • We propose a new task of text style transfer between expert language and layman language, namely Expertise Style Transfer, and contribute a manually annotated dataset in the medical
  • Content Similarity measures how much content is preserved during style transfer
  • We proposed a practical task of expertise style transfer and constructed a high-quality dataset, MSD
  • The results shown a significant gap between machine and human performance
  • Our further discussion analyzed the challenges of existing metrics
Methods
  • The authors reimplement five SOTA models from prior TS and ST studies on both MSD and SimpWiki datasets.
  • The models for ST task selected are: (1) Disentanglement method ControlledGen (Hu et al., 2017) that utilizes VAEs to learn content representations following a Gaussian prior, and reconstructs a style vector via a discriminator; (2) Manipulation method DeleteAndRetrieve (Li et al, 2018) that first identifies style words with a statistical method, replaces them with target style words derived from given corpus; and (3) Translation method StyleTransformer (Dai et al, 2019) that uses cyclic reconstruction to learn content and style vectors without parallel data.
  • The authors adopt early stopping and dropout rate is set to 0.5 for both encoder and decoder
Results
  • Following Dai et al (2019), the authors make an automatic evaluation on three aspects: Style Accuracy aims to measure how accurate the model controls sentence style.
  • The authors train two classifiers on the training set of each dataset using fasttext (Joulin et al, 2017).
  • Fluency is usually measured by the perplexity of the transferred sentence.
  • The authors fine-tune the state-of-the-art pretrained language model, Bert (Devlin et al, 2019), on the training set of each dataset for each style.
  • Content Similarity measures how much content is preserved during style transfer.
  • The authors calculate 4-gram BLEU (Papineni et al, 2002) between model outputs and inputs, and between outputs and gold human references
Conclusion
  • Both tasks lack parallel data for training and evaluation. This prevents researchers from exploring more advanced models concerning the knowledge gap as well as linguistic modification of lexicons and structures.
  • The model does not succeed in the style transfer task, and learns to add the word doctors into layman sentences while almost keeping the other words unchanged; and adding the word eg into the expertise sentences
  • It achieves good performance on all of the three ST measures, but makes little useful modifications.The authors proposed a practical task of expertise style transfer and constructed a high-quality dataset, MSD.
  • The authors are interested in injecting knowledge into text representation learning (Cao et al, 2017, 2018b) for deeply understanding expert language, and will help to generate knowledgeenhanced questions (Pan et al, 2019) for laymen
Summary
  • Introduction:

    The curse of knowledge (Camerer et al, 1989) is a pervasive cognitive bias exhibited across all domains, leading to discrepancies between an expert’s advice and a layman’s understanding of it (Tan and Goonawardene, 2017).
  • Take medical consultations as an example: patients often find it difficult to understand their doctors’ language.
  • The authors propose a new task of text style transfer between expert language and layman language, namely Expertise Style Transfer, and contribute a manually annotated dataset in the medical
  • Methods:

    The authors reimplement five SOTA models from prior TS and ST studies on both MSD and SimpWiki datasets.
  • The models for ST task selected are: (1) Disentanglement method ControlledGen (Hu et al., 2017) that utilizes VAEs to learn content representations following a Gaussian prior, and reconstructs a style vector via a discriminator; (2) Manipulation method DeleteAndRetrieve (Li et al, 2018) that first identifies style words with a statistical method, replaces them with target style words derived from given corpus; and (3) Translation method StyleTransformer (Dai et al, 2019) that uses cyclic reconstruction to learn content and style vectors without parallel data.
  • The authors adopt early stopping and dropout rate is set to 0.5 for both encoder and decoder
  • Results:

    Following Dai et al (2019), the authors make an automatic evaluation on three aspects: Style Accuracy aims to measure how accurate the model controls sentence style.
  • The authors train two classifiers on the training set of each dataset using fasttext (Joulin et al, 2017).
  • Fluency is usually measured by the perplexity of the transferred sentence.
  • The authors fine-tune the state-of-the-art pretrained language model, Bert (Devlin et al, 2019), on the training set of each dataset for each style.
  • Content Similarity measures how much content is preserved during style transfer.
  • The authors calculate 4-gram BLEU (Papineni et al, 2002) between model outputs and inputs, and between outputs and gold human references
  • Conclusion:

    Both tasks lack parallel data for training and evaluation. This prevents researchers from exploring more advanced models concerning the knowledge gap as well as linguistic modification of lexicons and structures.
  • The model does not succeed in the style transfer task, and learns to add the word doctors into layman sentences while almost keeping the other words unchanged; and adding the word eg into the expertise sentences
  • It achieves good performance on all of the three ST measures, but makes little useful modifications.The authors proposed a practical task of expertise style transfer and constructed a high-quality dataset, MSD.
  • The authors are interested in injecting knowledge into text representation learning (Cao et al, 2017, 2018b) for deeply understanding expert language, and will help to generate knowledgeenhanced questions (Pan et al, 2019) for laymen
Tables
  • Table1: Examples of parallel annotation in MSD, where the red fonts in brackets denote UMLS concepts
  • Table2: Statistics of MSD and SimpWiki. One annotation may contain multiple sentences, and MSD Train has no parallel annotations due to expensive expert cost. The ratio of layman to expert according to each metric denotes the gap between the two styles, and a higher value implies smaller differences except that for #Sentence
  • Table3: BLEU (4-gram) and edit distance (ED ) scores between parallel sentences. Concept words are masked for ED computation (<a class="ref-link" id="cFu_et+al_2019_a" href="#rFu_et+al_2019_a">Fu et al, 2019</a>). Higher BLEUs imply two more similar sentences, while higher edit distances imply more heterogeneous structures
  • Table4: Overall performance based on style transfer evaluation metrics from expertise to laymen language (marked as E2L) and in the opposite direction (L2E). Gold denotes human references
  • Table5: Performance using SARI
  • Table6: Examples of model outputs. Red/blue words with underlines highlight model/expected modifications
Download tables as Excel
Related work
  • 2.1 Text Style Transfer

    Existing ST work has achieved promising results on the styles of sentiment (Hu et al, 2017; Shen et al, 2017), formality (Rao and Tetreault, 2018), offensiveness (dos Santos et al, 2018), politeness (Sennrich et al, 2016), authorship (Xu et al, 2012), gender and ages (Prabhumoye et al, 2018; Lample et al, 2019), etc. Nevertheless, only a few of them focus on supervised methods due to the limited availability of parallel corpora. Jhamtani et al (2017) extract modern language based Shakespeare’s play from the educational site, while Rao and Tetreault (2018) and Li et al (2018) utilize crowdsourcing techniques to rewrite sentences from Yahoo Answers, Yelp and Amazon reviews, which are then utilized for training neural machine translation (NMT) models and evaluation.

    More practically, there is an enthusiasm for unsupervised methods without parallel data. There are three groups. The first group is Disentanglement methods that learn disentangled representations of style and content, and then directly manipulating these latent representations to control style-specific text generation. Shen et al (2017) propose a cross-aligned autoencoder that learns a shared latent content space between true samples and generated samples through an adversarial classifier. Hu et al (2017) utilize neural generative model, Variational Autoencoders (VAEs) (Kingma and Welling, 2013), to represent the content as continuous variables with standard Gaussian prior, and reconstruct style vector from the generated samples via an attribute discriminator. To improve the ability of style-specific generation, Fu et al (2018) utilize multiple generators, which are then extended by a Wasserstein distance regularizer (Zhao et al, 2018). SHAPED (Zhang et al, 2018a) learns a shared and several private encoder–decoder frameworks to capture both common and distinguishing features. Some variants further investigate the auxiliary tasks to better preserve contents (John et al, 2019), or domain adaptation (Li et al, 2019).
Funding
  • This research is supported by the National Research Foundation, Singapore under its International Research Centres in Singapore Funding Initiative
Reference
  • Laurens Van den Bercken, Robert-Jan Sips, and Christoph Lofi. 2019. Evaluating neural text simplification in the medical domain. In WWW.
    Google ScholarLocate open access versionFindings
  • Olivier Bodenreider. 2004. The unified medical language system (umls): integrating biomedical terminology. Nucleic acids research.
    Google ScholarFindings
  • Colin Camerer, George Loewenstein, and Martin Weber. 1989. The curse of knowledge in economic settings: An experimental analysis. Journal of political Economy.
    Google ScholarLocate open access versionFindings
  • Yixin Cao, Lei Hou, Juanzi Li, and Zhiyuan Liu. 2018a. Neural collective entity linking. In COLING.
    Google ScholarLocate open access versionFindings
  • Yixin Cao, Lei Hou, Juanzi Li, Zhiyuan Liu, Chengjiang Li, Xu Chen, and Tiansi Dong. 2018b. Joint representation learning of cross-lingual words and entities via attentive distant supervision. In EMNLP.
    Google ScholarFindings
  • Yixin Cao, Zikun Hu, Tat-seng Chua, Zhiyuan Liu, and Heng Ji. 2019. Low-resource name tagging learned with weakly labeled data. In EMNLP.
    Google ScholarFindings
  • Yixin Cao, Lifu Huang, Heng Ji, Xu Chen, and Juanzi Li. 201Bridge text and knowledge by learning multi-prototype entity mention embedding. In ACL.
    Google ScholarFindings
  • John Carroll, Guido Minnen, Darren Pearce, Yvonne Canning, Siobhan Devlin, and John Tait. 1999. Simplifying text for language-impaired readers. In EACL.
    Google ScholarFindings
  • Raman Chandrasekar and Bangalore Srinivas. 1997. Automatic induction of rules for text simplification. KBS.
    Google ScholarFindings
  • Meri Coleman and Ta Lin Liau. 1975. A computer readability formula designed for machine scoring. Journal of Applied Psychology.
    Google ScholarLocate open access versionFindings
  • William Coster and David Kauchak. 20Learning to simplify sentences using wikipedia. In Proceedings of the workshop on monolingual text-to-text generation. ACL.
    Google ScholarLocate open access versionFindings
  • Ning Dai, Jianze Liang, Xipeng Qiu, and Xuanjing Huang. 2019. Style transformer: Unpaired text style transfer without disentangled latent representation. In ACL.
    Google ScholarFindings
  • Louise Deleger and Pierre Zweigenbaum. 2008. Paraphrase acquisition from comparable medical corpora of specialized and lay texts. In AMIA Annual Symposium Proceedings.
    Google ScholarLocate open access versionFindings
  • Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL.
    Google ScholarFindings
  • Yao Fu, Hao Zhou, Jiaze Chen, and Lei Li. 2019. Rethinking text attribute transfer: A lexical analysis. ArXiv.
    Google ScholarFindings
  • Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2018. Style transfer in text: Exploration and evaluation. In AAAI.
    Google ScholarFindings
  • Goran Glavas and Sanja Stajner. 2015. Simplifying lexical simplification: Do we need simplified corpora? In ACL.
    Google ScholarLocate open access versionFindings
  • Robert Gunning. 1968. Technique of clear writing.
    Google ScholarFindings
  • Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Toward controlled generation of text. In ICML.
    Google ScholarFindings
  • Harsh Jhamtani, Varun Gangal, Eduard Hovy, and Eric Nyberg. 2017. Shakespearizing modern language using copy-enriched sequence-to-sequence models. EMNLP.
    Google ScholarLocate open access versionFindings
  • Zhijing Jin, Di Jin, Jonas Mueller, Nicholas Matthews, and Enrico Santus. 2019. Unsupervised text attribute transfer via iterative matching and translation. In EMNLP.
    Google ScholarFindings
  • Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2019. Disentangled representation learning for non-parallel text style transfer. AAAI.
    Google ScholarFindings
  • Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In EACL.
    Google ScholarFindings
  • J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel.
    Google ScholarFindings
  • Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.
    Google ScholarFindings
  • Diederik P. Kingma and Max Welling. 2013. Autoencoding variational bayes. ICLR.
    Google ScholarLocate open access versionFindings
  • Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. In ACL.
    Google ScholarFindings
  • Nicholas Kloehn, Gondy Leroy, David Kauchak, Yang Gu, Sonia Colina, Nicole P Yuan, and Debra Revere. 2018. Improving consumer understanding of medical text: Development and validation of a new subsimplify algorithm to automatically generate term explanations in english and spanish. JMIR.
    Google ScholarLocate open access versionFindings
  • Guillaume Lample, Sandeep Subramanian, Eric Michael Smith, Ludovic Denoyer, Marc’Aurelio Ranzato, and Y-Lan Boureau. 2019. Multiple-attribute text rewriting. In ICLR.
    Google ScholarFindings
  • Dianqi Li, Yizhe Zhang, Zhe Gan, Yu Cheng, Chris Brockett, Ming-Ting Sun, and William B. Dolan. 2019. Domain adaptive text style transfer. EMNLP.
    Google ScholarFindings
  • Juncen Li, Robin Jia, He He, and Percy Liang. 2018.
    Google ScholarFindings
  • Kevin Miller. 2009. Wordnet-about.
    Google ScholarFindings
  • Gustavo Paetzold and Lucia Specia. 2016. Unsupervised lexical simplification for non-native speakers. In AAAI.
    Google ScholarFindings
  • Liangming Pan, Wenqiang Lei, Tat-Seng Chua, and Min-Yen Kan. 2019. Recent advances in neural question generation. arXiv preprint arXiv:1905.08949.
    Findings
  • Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL.
    Google ScholarFindings
  • Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP.
    Google ScholarFindings
  • Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W. Black. 2018. Style transfer through back-translation. In ACL.
    Google ScholarFindings
  • Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may i introduce the gyafc dataset: Corpus, benchmarks and metrics for formality style transfer. In NAACL.
    Google ScholarFindings
  • Cicero Nogueira dos Santos, Igor Melnyk, and Inkit Padhi. 2018. Fighting offensive language on social media with unsupervised text style transfer. In ACL.
    Google ScholarFindings
  • Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Controlling politeness in neural machine translation via side constraints. In NAACL.
    Google ScholarFindings
  • Matthew Shardlow and Raheel Nawaz. 2019. Neural text simplification of clinical letters with a domain specific phrase table. In ACL.
    Google ScholarFindings
  • Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In NIPS.
    Google ScholarFindings
  • Luca Soldaini and Nazli Goharian. 2016. Quickumls: a fast, unsupervised approach for medical concept extraction. In MedIR workshop, sigir.
    Google ScholarLocate open access versionFindings
  • Elior Sulem, Omri Abend, and Ari Rappoport. 2018a. Bleu is not suitable for the evaluation of text simplification. In EMNLP.
    Google ScholarFindings
  • Elior Sulem, Omri Abend, and Ari Rappoport. 2018b. Simple and effective text simplification using semantic and neural methods. In ACL.
    Google ScholarFindings
  • Sai Surya, Abhijit Mishra, Anirban Laha, Parag Jain, and Karthik Sankaranarayanan. 2019. Unsupervised neural text simplification. In ACL.
    Google ScholarFindings
  • Sharon Swee-Lin Tan and Nadee Goonawardene. 2017. Internet health information seeking and the patientphysician relationship: a systematic review. JMIR.
    Google ScholarLocate open access versionFindings
  • David Vickrey and Daphne Koller. 2008. Sentence simplification for semantic role labeling. In ACL.
    Google ScholarFindings
  • Kristian Woodsend and Mirella Lapata. 2011. Learning to simplify sentences with quasi-synchronous grammar and integer programming. In EMNLP.
    Google ScholarFindings
  • Sander Wubben, Antal Van Den Bosch, and Emiel Krahmer. 2012. Sentence simplification by monolingual machine translation. In ACL.
    Google ScholarFindings
  • Jingjing Xu, Xu Sun, Qi Zeng, Xuancheng Ren, Xiaodong Zhang, Houfeng Wang, and Wenjie Li. 2018. Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach. In ACL.
    Google ScholarFindings
  • Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in current text simplification research: New data can help. TACL.
    Google ScholarLocate open access versionFindings
  • Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. TACL.
    Google ScholarFindings
  • Wei Xu, Alan Ritter, Bill Dolan, Ralph Grishman, and Colin Cherry. 2012. Paraphrasing for style. In COLING.
    Google ScholarFindings
  • Xingxing Zhang and Mirella Lapata. 2017. Sentence simplification with deep reinforcement learning. In EMNLP.
    Google ScholarFindings
  • Ye Zhang, Nan Ding, and Radu Soricut. 2018a. Shaped: Shared-private encoder-decoder for text style adaptation. In NAACL.
    Google ScholarFindings
  • Yi Zhang, Jingjing Xu, Pengcheng Yang, and Xu Sun. 2018b. Learning sentiment memories for sentiment modification without parallel data. In EMNLP.
    Google ScholarFindings
  • Zhirui Zhang, Shuo Ren, Shujie Liu, Jianyong Wang, Peng Chen, Mu Li, Ming Zhou, and Enhong Chen. 2019. Style transfer as unsupervised machine translation. AAAI.
    Google ScholarLocate open access versionFindings
  • Jake Zhao, Yoon Kim, Kelly Zhang, Alexander M Rush, and Yann LeCun. 2018. Adversarially regularized autoencoders. In ICML.
    Google ScholarFindings
  • Zhemin Zhu, Delphine Bernhard, and Iryna Gurevych. 2010. A monolingual tree-based translation model for sentence simplification. In COLING.
    Google ScholarFindings
Your rating :
0

 

Tags
Comments