Salience and Market-aware Skill Extraction for Job Targeting

KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining Virtual Event CA USA July, 2020, pp. 2871-2879, 2020.

Cited by: 0|Bibtex|Views24|Links
EI
Keywords:
Job Targeting skillJobs You May Be Interested InQuality Applicant skilllinkedinjob postingMore(10+)
Weibo:
We show that the job targeting skills generated by Job2Skills better captures such market diversity by modeling salience and market signals jointly

Abstract:

At LinkedIn, we want to create economic opportunity for everyone in the global workforce. To make this happen, LinkedIn offers a reactive Job Search system, and a proactive Jobs You May Be Interested In (JYMBII) system to match the best candidates with their dream jobs. One of the most challenging tasks for developing these systems is to ...More

Code:

Data:

0
Introduction
  • LinkedIn is the world’s largest professional network whose vision is to “create economic opportunity for every member of the global workforce”
  • To achieve this vision, it is crucial for LinkedIn to match Jaewon Yang.
  • It is crucial for LinkedIn to match Jaewon Yang
  • Qi He job postings to quality applicants who are both qualified and willing to apply for the job.
  • In other words, when a member comes to the job recommendation page, the authors recommend job postings whose targeting skills match the member’s skills
Highlights
  • LinkedIn is the world’s largest professional network whose vision is to “create economic opportunity for every member of the global workforce”
  • We propose a skill extraction framework to target job postings by skill salience and market-awareness, which is different from traditional entity recognition based method
  • We find that the top-ranked quality applicant skills largely overlap with the job poster selections
  • After we describe the procedure we use to collect the ground truth for salience and market-aware job targeting skill extraction, here we discuss how we build the proposed Job2Skills using multi-resolution skill salience features and market-aware signals
  • We show that the job targeting skills generated by Job2Skills better captures such market diversity by modeling salience and market signals jointly
  • In Tab. 7, we presented the top-5 job targeting skills of US government and technology industries generated by the baseline and the proposed Job2Skills
Methods
  • The authors conduct an extensive set of experiments with both offline and online A/B tests to demonstrate the effectiveness of the proposed Job2Skills model compared to the market-agnostic production model.
  • Note that the job-level salience sub-model the authors used in production Job2Skills model is a FastText-based model instead of the BERT model the authors tried offline.
  • This is because the authors observed significant latency reduction with only 3% salience accuracy drop.
  • The market-agnostic production model the authors compared against is a logistic regression model trained with skill appearance features, e.g. job-level features such as is the skill mentioned in the text?, where the skill is mentioned?, and global-level features such as mention frequency
Results
  • The authors add signals representing the job market supplies and the salience of skills in the model and significantly improve the performance in the offline evaluation and online A/B test.
  • The authors observe that when only using one feature group, model trained with deep learning-based salience feature is 0.95% better than market-feature only model.
  • By combining both group of features, the authors further improve the AUROC by 1.56% comparing to the salience only model.
  • As shown in Tab. 4, the skills recommended by Job2Skills are notably better than the existing production model because the recruiters are 31.44% less likely to manually add a job targeting skill and 33.71% less likely to reject recommendation
Conclusion
  • The authors proposed salience and market-aware skill extraction task, discussed two data collection strategies, and presented Job2Skills, which models skill salience using deep learning methods and market supply signals using engineered features.
  • The authors conducted extensive experiments and showed that Job2Skills significantly improves the quality of multiple LinkedIn products including job targeting skill suggestions and job recommendation.
  • The authors performed large-scale case studies to explore interesting insights the authors obtained by analyzing Job2Skills results.
  • The authors plan to add temporal information into the model and explore advanced methods to learn skill embeddings
Summary
  • Introduction:

    LinkedIn is the world’s largest professional network whose vision is to “create economic opportunity for every member of the global workforce”
  • To achieve this vision, it is crucial for LinkedIn to match Jaewon Yang.
  • It is crucial for LinkedIn to match Jaewon Yang
  • Qi He job postings to quality applicants who are both qualified and willing to apply for the job.
  • In other words, when a member comes to the job recommendation page, the authors recommend job postings whose targeting skills match the member’s skills
  • Methods:

    The authors conduct an extensive set of experiments with both offline and online A/B tests to demonstrate the effectiveness of the proposed Job2Skills model compared to the market-agnostic production model.
  • Note that the job-level salience sub-model the authors used in production Job2Skills model is a FastText-based model instead of the BERT model the authors tried offline.
  • This is because the authors observed significant latency reduction with only 3% salience accuracy drop.
  • The market-agnostic production model the authors compared against is a logistic regression model trained with skill appearance features, e.g. job-level features such as is the skill mentioned in the text?, where the skill is mentioned?, and global-level features such as mention frequency
  • Results:

    The authors add signals representing the job market supplies and the salience of skills in the model and significantly improve the performance in the offline evaluation and online A/B test.
  • The authors observe that when only using one feature group, model trained with deep learning-based salience feature is 0.95% better than market-feature only model.
  • By combining both group of features, the authors further improve the AUROC by 1.56% comparing to the salience only model.
  • As shown in Tab. 4, the skills recommended by Job2Skills are notably better than the existing production model because the recruiters are 31.44% less likely to manually add a job targeting skill and 33.71% less likely to reject recommendation
  • Conclusion:

    The authors proposed salience and market-aware skill extraction task, discussed two data collection strategies, and presented Job2Skills, which models skill salience using deep learning methods and market supply signals using engineered features.
  • The authors conducted extensive experiments and showed that Job2Skills significantly improves the quality of multiple LinkedIn products including job targeting skill suggestions and job recommendation.
  • The authors performed large-scale case studies to explore interesting insights the authors obtained by analyzing Job2Skills results.
  • The authors plan to add temporal information into the model and explore advanced methods to learn skill embeddings
Tables
  • Table1: Relative skill extraction AUROC improvement on JT and QA datsets
  • Table2: Feature ablation test on job targeting skill inference
  • Table3: Online A/B test result on the LinkedIn Job Recommendation (JYMBII [<a class="ref-link" id="c18" href="#r18">18</a>]) page
  • Table4: A/B test result of job targeting skill suggestions
  • Table5: Sample of top-10 job targeting skills extracted by the salience and market-agnostic baseline and Job2Skills
  • Table6: Sample of top-5 job targeting skills per industry and country extracted by the salience and market-agnostic baseline and Job2Skills. Red squared skills are sub-optimal
  • Table7: Top-5 job targeting skills of US government and technology generated the salience and market-agnostic baseline and Job2Skills. Red squared skills are sub-optimal
Download tables as Excel
Related work
  • Job Recommendation. Previous work usually treat the job targeting problem as job recommendation [14, 37], and optimizes the model using direct user interaction signals such as click, dismiss, bookmark, etc. [2, 3, 16, 40]. Borisyuk et al proposed LiJar [6] to redistribute job targeting audiences and improve marketplace efficiency. Dave et al designed a representation-learning method to perform job and skill recommendations [11] . Li et al used career history to predict next position [20]. None of the previous works address the most pressing job targeting issue, which is how to properly represent jobs with relevant, important attribute entities to improve the number of quality applicants a job can reach. Skill Analysis. Traditionally, skill analysis are often conducted by experts manually to either gain insights [27] or curate structured taxonomy [10]. Recently, SPTM [41] used topic modeling to measure the popularity of 1, 729 IT skills from 892, 454 jobs. TATF [39] is a trend-aware tensor factorization method that models time-aware skill popularity. DuerQuiz [29] is proposed to create in-depth skill assessment questions for applicant evaluation. These methods were applied to small-sacle IT jobs only and are not designed to extract skills for job targeting purpose. Recently Xiao proposed a social signal-based method for members’ skill validation [42]. However it is not applicable to jobs due to the lack of such signals. Job Market Analysis. Modeling job targeting and recommendation using skills is mostly inspired by economic research which analyzes the labor market using skills as the most direct and vital signal [4, 32]. However these works are either conducted on a very small scale or using only a handful of hand-crafted general skill categories. Woon et al [38] performed a case study to learn occupational skill changes, but the skills are limited to 35 skills provided by O*NET [26]. Radermacher et al [30] studied the skill gap between fresh graduates and industry expectations based on the feedback of 23 managers and hiring personnel using 16 hand picked skills. Recently, [17, 36] analyzed labor demand and skill fungibility using a skill taxonomy with 1, 351 skills in the IT industry. APJFNN [28] and other resume-based method [45] are developed to predict person-job fit by comparing the job description and resume. HIPO [43] identifies high potential talent by conducting neural network-based social profiling. OSCN [35] and HCPNN [23], use recurrent neural networks and attention mechanism to predict organization and individual level job mobility. However, none of these works addresses the market-aware job targeting task, and they all use a limited skill taxonomy that contains at most a thousand skills.
Reference
  • Fabian Abel. 2015. We know where you should work next summer: job recommendations. In RecSys.
    Google ScholarFindings
  • Fabian Abel, Yashar Deldjoo, Mehdi Elahi, and Daniel Kohlsdorf. 2017. Recsys challenge 2017: Offline and online evaluation. In RecSys.
    Google ScholarFindings
  • Deepak Agarwal, Bee-Chung Chen, and Pradheep Elango. 2009. Spatio-temporal models for estimating click-through rate. In WWW.
    Google ScholarFindings
  • David H Autor, Frank Levy, and Richard J Murnane. 2003. The skill content of recent technological change: An empirical exploration. The Quarterly journal of economics 118, 4 (2003).
    Google ScholarLocate open access versionFindings
  • Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. TACL 5 (2017).
    Google ScholarLocate open access versionFindings
  • Fedor Borisyuk, Liang Zhang, and Krishnaram Kenthapadi. 2017. LiJAR: A system for job application redistribution towards efficient career marketplace. In KDD.
    Google ScholarFindings
  • Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175 (2018).
    Findings
  • Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In KDD.
    Google ScholarFindings
  • Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016.
    Google ScholarLocate open access versionFindings
  • National Research Council et al. 20A database for a changing economy: Review of the Occupational Information Network (O* NET).
    Google ScholarFindings
  • Vachik S Dave, Baichuan Zhang, Mohammad Al Hasan, Khalifeh AlJadda, and Mohammed Korayem. 2018. A combined representation learning approach for better job and skill recommendation. In CIKM.
    Google ScholarFindings
  • Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
    Findings
  • Yupeng Gu, Bo Zhao, David Hardtke, and Yizhou Sun. 2016. Learning global term weights for content-based recommender systems. In WWW.
    Google ScholarFindings
  • Cheng Guo, Hongyu Lu, Shaoyun Shi, Bin Hao, Bin Liu, Min Zhang, Yiqun Liu, and Shaoping Ma. 2017. How Integration helps on Cold-Start Recommendations. In RecSys Challenge 2017.
    Google ScholarLocate open access versionFindings
  • Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. In WWW.
    Google ScholarLocate open access versionFindings
  • Chao Huang, Xian Wu, Xuchao Zhang, Chuxu Zhang, Jiashu Zhao, Dawei Yin, and Nitesh V Chawla. 2019. Online Purchase Prediction via Multi-Scale Modeling of Behavior Dynamics. In KDD.
    Google ScholarFindings
  • Brian Johnston, Benjamin Zweig, Michael Peran, Charlie Wang, and Rachel Rosenfeld. 20Estimating skill fungibility and forecasting services labor demand. In Big Data.
    Google ScholarFindings
  • Krishnaram Kenthapadi, Benjamin Le, and Ganesh Venkataraman. 2017. Personalized job recommendation system at linkedin: Practical challenges and lessons learned. In RecSys.
    Google ScholarFindings
  • Jia Li, Dhruv Arya, Viet Ha-Thuc, and Shakti Sinha. 2016. How to get them a dream job?: Entity-aware features for personalized job search ranking. In KDD.
    Google ScholarFindings
  • Liangyue Li, How Jing, Hanghang Tong, Jaewon Yang, Qi He, and Bee-Chung Chen. 2017. Nemo: Next career move prediction with contextual embedding. In WWW.
    Google ScholarFindings
  • Shan Li, Baoxu Shi, Jaewon Yang, Ji Yan, Shuai Wang, Fei Chen, and Qi He. 2020. Deep Job Understanding at LinkedIn. In SIGIR.
    Google ScholarFindings
  • Yin Lou and Mikhail Obukhov. 2017. BDT: Gradient Boosted Decision Tables for High Accuracy and Scoring Efficiency. In KDD.
    Google ScholarFindings
  • Qingxin Meng, Hengshu Zhu, Keli Xiao, Le Zhang, and Hui Xiong. 2019. A Hierarchical Career-Path-Aware Neural Network for Job Mobility Prediction. In KDD. ACM.
    Google ScholarLocate open access versionFindings
  • David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. Lingvisticae Investigationes 30, 1 (2007).
    Google ScholarLocate open access versionFindings
  • Ioannis Paparrizos, B Barla Cambazoglu, and Aristides Gionis. 2011. Machine learned job recommendation. In RecSys.
    Google ScholarFindings
  • Norman G Peterson, Michael D Mumford, Walter C Borman, P Richard Jeanneret, Edwin A Fleishman, Kerry Y Levin, Michael A Campion, Melinda S Mayfield, Frederick P Morgeson, Kenneth Pearlman, et al. 2001. Understanding work using the Occupational Information Network (O* NET): Implications for practice and research. Personnel Psychology 54, 2 (2001).
    Google ScholarLocate open access versionFindings
  • Bipin Prabhakar, Charles R Litecky, and Kirk Arnett. 2005. IT skills in a tough job market. Commun. ACM 48, 10 (2005).
    Google ScholarLocate open access versionFindings
  • Chuan Qin, Hengshu Zhu, Tong Xu, Chen Zhu, Liang Jiang, Enhong Chen, and Hui Xiong. 2018. Enhancing person-job fit for talent recruitment: An abilityaware neural network approach. In SIGIR.
    Google ScholarFindings
  • Chuan Qin, Hengshu Zhu, Chen Zhu, Tong Xu, Fuzhen Zhuang, Chao Ma, Jingshuai Zhang, and Hui Xiong. 2019. DuerQuiz: A Personalized Question Recommender System for Intelligent Job Interview. In KDD.
    Google ScholarFindings
  • Alex Radermacher, Gursimran Walia, and Dean Knudson. 2014. Investigating the skill gap between graduating students and industry expectations. In ICSE.
    Google ScholarFindings
  • Rohan Ramanath, Hakan Inan, Gungor Polatkan, Bo Hu, Qi Guo, Cagri Ozcaglar, Xianren Wu, Krishnaram Kenthapadi, and Sahin Cem Geyik. 2018. Towards Deep and Representation Learning for Talent Search at LinkedIn. In CIKM. 2253–2261.
    Google ScholarFindings
  • Ellu Saar and Mari Liis Räis. 2017. Participation in job-related training in European countries: the impact of skill supply and demand characteristics. Journal of Education and Work 30, 5 (2017).
    Google ScholarLocate open access versionFindings
  • Baoxu Shi, Shan Li, Jaewon Yang, Mustafa Emre Kazdagli, and Qi He. 2020. Learning to Ask Screening Questions for Job Postings. In SIGIR.
    Google ScholarFindings
  • Baoxu Shi, Jaewon Yang, Tim Weninger, Jing How, and Qi He. 2019. Representation Learning in Heterogeneous Professional Social Networks with Ambiguous Social Connections. In IEEE BigData.
    Google ScholarLocate open access versionFindings
  • Ying Sun, Fuzhen Zhuang, Hengshu Zhu, Xin Song, Qing He, and Hui Xiong. 2019. The Impact of Person-Organization Fit on Talent Management: A StructureAware Convolutional Neural Network Approach. In KDD.
    Google ScholarLocate open access versionFindings
  • Shrihari Vasudevan, Moninder Singh, Joydeep Mondal, Michael Peran, Benjamin Zweig, Brian Johnston, and Rachel Rosenfeld. 2018. Estimating Fungibility Between Skills by Combining Skill Similarities Obtained from Multiple Data Sources. Data Science and Engineering 3, 3 (2018).
    Google ScholarLocate open access versionFindings
  • Maksims Volkovs, Guang Wei Yu, and Tomi Poutanen. 2017. Content-based Neighbor Models for Cold Start in Recommender Systems. In RecSys.
    Google ScholarFindings
  • Wei Lee Woon, Zeyar Aung, Wala AlKhader, Davor Svetinovic, and Mohammad Atif Omar. 2015. Changes in occupational skills-a case study using nonnegative matrix factorization. In ICONIP.
    Google ScholarFindings
  • Xunxian Wu, Tong Xu, Hengshu Zhu, Le Zhang, Enhong Chen, and Hui Xiong. 2019. Trend-aware tensor factorization for job skill demand analysis. In IJCAI.
    Google ScholarFindings
  • Lianghao Xia, Chao Huang, Yong Xu, Peng Dai, Bo Zhang, and Liefeng Bo. 2020. Multiplex Behavioral Relation Learning for Recommendation via Memory Augmented Transformer Network. In SIGIR.
    Google ScholarFindings
  • Tong Xu, Hengshu Zhu, Chen Zhu, Pan Li, and Hui Xiong. 2018. Measuring the popularity of job skills in recruitment market: A multi-criteria approach. In AAAI.
    Google ScholarFindings
  • Xiao Yan, Jaewon Yang, Mikhail Obukhov, Lin Zhu, Joey Bai, Shiqi Wu, and Qi He. 2019. Social Skill Validation at LinkedIn. In KDD.
    Google ScholarFindings
  • Yuyang Ye, Hengshu Zhu, Tong Xu, Fuzhen Zhuang, Runlong Yu, and Hui Xiong. 2019. Identifying High Potential Talent: A Neural Network Based Dynamic Social Profiling Approach. In ICDM.
    Google ScholarFindings
  • XianXing Zhang, Yitong Zhou, Yiming Ma, Bee-Chung Chen, Liang Zhang, and Deepak Agarwal. 2016. Glmix: Generalized linear mixed models for large-scale response prediction. In KDD.
    Google ScholarFindings
  • Chen Zhu, Hengshu Zhu, Hui Xiong, Chao Ma, Fang Xie, Pengliang Ding, and Pan Li. 2018. Person-Job Fit: Adapting the Right Talent for the Right Job with Joint Representation Learning. TMIS 9, 3 (2018).
    Google ScholarLocate open access versionFindings
Your rating :
0

 

Tags
Comments