AI helps you reading Science

AI generates interpretation videos

AI extracts and analyses the key points of the paper to generate videos automatically


pub
Go Generating

AI Traceability

AI parses the academic lineage of this thesis


Master Reading Tree
Generate MRT

AI Insight

AI extracts a summary of this paper


Weibo:
We believe this technique can provide a basis for providing automated feedback to hundreds of thousands of students learning from online introductory programming courses that are being taught by MITx, Coursera, and Udacity

Automated feedback generation for introductory programming assignments

Special Interest Group on Programming Languages, no. 6 (2013): 15-26

Cited by: 362|Views162
EI

Abstract

We present a new method for automatically providing feedback for introductory programming problems. In order to use this method, we need a reference implementation of the assignment, and an error model consisting of potential corrections to errors that students might make. Using this information, the system automatically derives minimal c...More

Code:

Data:

Introduction
  • There has been a lot of interest recently in making quality education more accessible to students worldwide using information technology
  • Several education initiatives such as EdX, Coursera, and Udacity are racing to provide online courses on various collegelevel subjects ranging from computer science to psychology.
  • These courses, called massive open online courses (MOOC), are typically taken by thousands of students worldwide, and present many interesting scalability challenges.
  • In test-case based feedback, the student program is run on a set of test cases and the failing test cases
Highlights
  • There has been a lot of interest recently in making quality education more accessible to students worldwide using information technology
  • This paper addresses the challenge of providing personalized feedback for programming assignments in introductory programming courses
  • We show that the problem of providing automated feedback for introductory programming assignments can be framed as a synthesis problem
  • We presented a new technique of automatically providing feedback for introductory programming assignments that can complement manual and test-cases based techniques
  • We have evaluated our technique on a large set of benchmarks and it can correct 64% of incorrect solutions in our benchmark set
  • We believe this technique can provide a basis for providing automated feedback to hundreds of thousands of students learning from online introductory programming courses that are being taught by MITx, Coursera, and Udacity
Methods
  • The authors present various experiments the authors performed to evaluate the tool on the benchmark problems.

    Performance Table 1 shows the number of student attempts corrected for each benchmark problem as well as the time taken by the tool to provide the feedback.
  • The authors present various experiments the authors performed to evaluate the tool on the benchmark problems.
  • Performance Table 1 shows the number of student attempts corrected for each benchmark problem as well as the time taken by the tool to provide the feedback.
  • The tool was able to provide appropriate corrections as feedback for 64% of all incorrect student attempts in around 10 seconds on average.
  • The remaining 36% of incorrect student attempts on which the tool could not provide feedback fall in one of the following categories:
Results
  • Timeout: In the experiments, the authors found less than 5% of the student attempts timed out.
Conclusion
  • The authors presented a new technique of automatically providing feedback for introductory programming assignments that can complement manual and test-cases based techniques.
  • The authors have evaluated the technique on a large set of benchmarks and it can correct 64% of incorrect solutions in the benchmark set.
  • The authors believe this technique can provide a basis for providing automated feedback to hundreds of thousands of students learning from online introductory programming courses that are being taught by MITx, Coursera, and Udacity
Tables
  • Table1: The percentage of student attempts corrected and the time taken for correction for the benchmark problems
Download tables as Excel
Related work
  • In this section, we describe several related work to our technique from the areas of automated programming tutors, automated program repair, fault localization, automated debugging, automated grading, and program synthesis.

    7.1 AI based programming tutors

    There has been a lot of work done in the AI community for building automated tutors for helping novice programmers learn programming by providing feedback about semantic errors. These tutoring systems can be categorized into the following two major classes: Code-based matching approaches: LAURA [1] converts teacher’s and student’s program into a graph based representation and compares them heuristically by applying program transformations while reporting mismatches as potential bugs. TALUS [31] matches a student’s attempt with a collection of teacher’s algorithms. It first tries to recognize the algorithm used and then tentatively replaces the top-level expressions in the student’s attempt with the recognized algorithm for generating correction feedback. The problem with these approach is that the enumeration of all possible algorithms (with its variants) for covering all corrections is very large and tedious on part of the teacher.

    Intention-based matching approaches: LISP tutor [13] creates a model of the student goals and updates it dynamically as the student makes edits. The drawback of this approach is that it forces students to write code in a certain pre-defined structure and limits their freedom. MENO-II [39] parses student programs into a deep syntax tree whose nodes are annotated with plan tags. This annotated tree is then matched with the plans obtained from teacher’s solution. PROUST [24], on the other hand, uses a knowledge base of goals and their corresponding plans for implementing them for each programming problem. It first tries to find correspondence of these plans in the student’s code and then performs matching to find discrepancies. CHIRON [32] is its improved version in which the goals and plans in the knowledge base are organized in a hierarchical manner based on their generality and uses machine learning techniques for plan identification in the student code. These approaches require teacher to provide all possible plans a student can use to solve the goals of a given problem and do not perform well if the student’s attempt uses a plan not present in the knowledge base.
Funding
  • This work is supported in part by National Science Foundation grants Expeditions in Computer Augmented Program Engineering (ExCAPE NSF-1139056) and NSF-1139056, MIT EECS Super UROP program, Microsoft Research and a Microsoft Research PhD fellowship. We thank Ned Batchelder, Sarina Canelake, Piotr Mitros, and Victor Shnayder from the EdX team for their support and for providing us the 6.00x data
Reference
  • A. Adam and J.-P. H. Laurent. LAURA, A System to Debug Student Programs. Artif. Intell., 15(1-2):75–122, 1980.
    Google ScholarLocate open access versionFindings
  • U. Ahmed, S. Gulwani, and A. Karkare. Automatically generating problems and solutions for natural deduction. In IJCAI, 2013.
    Google ScholarLocate open access versionFindings
  • R. Alur, L. D’Antoni, S. Gulwani, D. Kini, and M. Viswanathan. Automated grading of dfa constructions. In IJCAI, 2013.
    Google ScholarFindings
  • E. Andersen, S. Gulwani, and Z. Popovic. A trace-based framework for analyzing and synthesizing educational progressions. In CHI, 2013.
    Google ScholarFindings
  • A. Arcuri. On the automation of fixing software bugs. In ICSE Companion, 2008.
    Google ScholarLocate open access versionFindings
  • T. Ball, M. Naik, and S. K. Rajamani. From symptom to cause: localizing errors in counterexample traces. In POPL, 2003.
    Google ScholarFindings
  • M. S. Bernstein, G. Little, R. C. Miller, B. Hartmann, M. S. Ackerman, D. R. Karger, D. Crowell, and K. Panovich. Soylent: a word processor with a crowd inside. In UIST, 2010.
    Google ScholarLocate open access versionFindings
  • R. Bodík, S. Chandra, J. Galenson, D. Kimelman, N. Tung, S. Barman, and C. Rodarmor. Programming with angelic nondeterminism. In POPL, 2010.
    Google ScholarLocate open access versionFindings
  • C. Daly. Roboprof and an introductory computer programming course. ITiCSE, 1999.
    Google ScholarFindings
  • V. Debroy and W. Wong. Using mutation to automatically suggest fixes for faulty programs. In ICST, 2010.
    Google ScholarLocate open access versionFindings
  • C. Douce, D. Livingstone, and J. Orwell. Automatic test-based assessment of programming: A review. J. Educ. Resour. Comput., 5(3), Sept. 2005.
    Google ScholarLocate open access versionFindings
  • P. Ertmer, J. Richardson, B. Belland, D. Camin, P. Connolly, G. Coulthard, K. Lei, and C. Mong. Using peer feedback to enhance the quality of student online postings: An exploratory study. Journal of Computer-Mediated Communication, 12(2):412–433, 2007.
    Google ScholarLocate open access versionFindings
  • R. G. Farrell, J. R. Anderson, and B. J. Reiser. An interactive computer-based tutor for lisp. In AAAI, 1984.
    Google ScholarLocate open access versionFindings
  • S. Forrest, T. Nguyen, W. Weimer, and C. L. Goues. A genetic programming approach to automated software repair. In GECCO, 2009.
    Google ScholarLocate open access versionFindings
  • A. Groce, S. Chaki, D. Kroening, and O. Strichman. Error explanation with distance metrics. STTT, 8(3):229–247, 2006.
    Google ScholarLocate open access versionFindings
  • S. Gulwani, S. Srivastava, and R. Venkatesan. Program analysis as constraint solving. In PLDI, 2008.
    Google ScholarLocate open access versionFindings
  • S. Gulwani, V. A. Korthikanti, and A. Tiwari. Synthesizing geometry constructions. In PLDI, 2011.
    Google ScholarLocate open access versionFindings
  • S. Gulwani, W. R. Harris, and R. Singh. Spreadsheet data manipulation using examples. In CACM, 2012.
    Google ScholarLocate open access versionFindings
  • P. Hawkins, A. Aiken, K. Fisher, M. C. Rinard, and M. Sagiv. Data representation synthesis. In PLDI, 2011.
    Google ScholarLocate open access versionFindings
  • P. Hawkins, A. Aiken, K. Fisher, M. C. Rinard, and M. Sagiv. Concurrent data representation synthesis. In PLDI, 2012.
    Google ScholarLocate open access versionFindings
  • J. B. Hext and J. W. Winings. An automatic grading scheme for simple programming exercises. Commun. ACM, 12(5), May 1969.
    Google ScholarLocate open access versionFindings
  • D. Jackson and M. Usher. Grading student programs using assyst. SIGCSE, 1997.
    Google ScholarLocate open access versionFindings
  • B. Jobstmann, A. Griesmayer, and R. Bloem. Program repair as a game. In CAV, pages 226–238, 2005.
    Google ScholarLocate open access versionFindings
  • W. L. Johnson and E. Soloway. Proust: Knowledge-based program understanding. IEEE Trans. Software Eng., 11(3):267–275, 1985.
    Google ScholarLocate open access versionFindings
  • M. Jose and R. Majumdar. Cause clue clauses: error localization using maximum satisfiability. In PLDI, 2011.
    Google ScholarLocate open access versionFindings
  • U. Junker. QUICKXPLAIN: preferred explanations and relaxations for over-constrained problems. In AAAI, 2004.
    Google ScholarLocate open access versionFindings
  • R. Könighofer and R. P. Bloem. Automated error localization and correction for imperative programs. In FMCAD, 2011.
    Google ScholarLocate open access versionFindings
  • C. Kulkarni and S. R. Klemmer. Learning design wisdom by augmenting physical studio critique with online self-assessment. Technical report, Stanford University, 2012.
    Google ScholarFindings
  • V. Kuncak, M. Mayer, R. Piskac, and P. Suter. Complete functional synthesis. PLDI, 2010.
    Google ScholarLocate open access versionFindings
  • G. Little, L. B. Chilton, M. Goldman, and R. C. Miller. Turkit: human computation algorithms on mechanical turk. In UIST, 2010.
    Google ScholarLocate open access versionFindings
  • W. R. Murray. Automatic program debugging for intelligent tutoring systems. Computational Intelligence, 3:1–16, 1987.
    Google ScholarLocate open access versionFindings
  • W. Sack, E. Soloway, and P. Weingrad. From PROUST to CHIRON: Its design as iterative engineering: Intermediate results are important! In In J.H. Larkin and R.W. Chabay (Eds.), Computer-Assisted Instruction and Intelligent Tutoring Systems: Shared Goals and Complementary Approaches., pages 239–274, 1992.
    Google ScholarLocate open access versionFindings
  • R. Singh and S. Gulwani. Learning semantic string transformations from examples. PVLDB, 5, 2012.
    Google ScholarLocate open access versionFindings
  • R. Singh and A. Solar-Lezama. Synthesizing data structure manipulations from storyboards. In SIGSOFT FSE, 2011.
    Google ScholarLocate open access versionFindings
  • R. Singh, S. Gulwani, and S. K. Rajamani. Automatically generating algebra problems. In AAAI, 2012.
    Google ScholarLocate open access versionFindings
  • R. Singh, S. Gulwani, and A. Solar-Lezama. Automated semantic grading of programs. CoRR, abs/1204.1751, 2012.
    Findings
  • A. Solar-Lezama. Program Synthesis By Sketching. PhD thesis, EECS Dept., UC Berkeley, 2008.
    Google ScholarFindings
  • A. Solar-Lezama, R. Rabbah, R. Bodik, and K. Ebcioglu. Programming by sketching for bit-streaming programs. In PLDI, 2005.
    Google ScholarLocate open access versionFindings
  • E. Soloway, B. P. Woolf, E. Rubin, and P. Barth. Meno-II: An Intelligent Tutoring System for Novice Programmers. In IJCAI, 1981.
    Google ScholarLocate open access versionFindings
  • S. Srivastava, S. Gulwani, and J. Foster. From program verification to program synthesis. POPL, 2010.
    Google ScholarLocate open access versionFindings
  • S. S. Staber, B. Jobstmann, and R. P. Bloem. Finding and fixing faults. In Correct Hardware Design and Verification Methods, Lecture notes in computer science, pages 35 – 49, 2005.
    Google ScholarLocate open access versionFindings
  • M. Vechev, E. Yahav, and G. Yorsh. Abstraction-guided synthesis of synchronization. In POPL, 2010.
    Google ScholarLocate open access versionFindings
  • D. S. Weld, E. Adar, L. Chilton, R. Hoffmann, and E. Horvitz. Personalized online education - a crowdsourcing challenge. In Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence, 2012.
    Google ScholarLocate open access versionFindings
  • A. Zeller and R. Hildebrandt. Simplifying and isolating failureinducing input. IEEE Transactions on Software Engineering, 28:183– 200, 2002.
    Google ScholarLocate open access versionFindings
Your rating :
0

 

Tags
Comments
数据免责声明
页面数据均来自互联网公开来源、合作出版商和通过AI技术自动分析结果,我们不对页面数据的有效性、准确性、正确性、可靠性、完整性和及时性做出任何承诺和保证。若有疑问,可以通过电子邮件方式联系我们:report@aminer.cn
小科