Analysis Of Container Based Vs. Jailed Sandbox Autograding Systems

Traditionally, automated testing and grading of student programming assignments has been done in either a jailed sandbox environment or within a virtual machine (VM). For a VM, each submission is given its own instantiation of a guest operating system (OS) running atop the host OS, with no ability for a given submission to affect anything outside the VM. However, using a VM is expensive in terms of system resource usages, especially for RAM and memory, making it less than ideal for solutions without unlimited resources. Jailed sandboxes on the other hand allow student submissions to run directly on the server. Sufficient security measures must be implemented to ensure that students cannot access each other's submissions or the server at large, and must prevent runaway programs, over-utilization of system resources. Jailed sandboxes have a larger attack vector than VMs. Within the past several years, container systems have been gaining popularity and usage within the computer science industry, primarily through solutions such as Docker. These containers give similar security protections as a VM, but with better performance due to being able to utilize of resources installed within the host OS and other containers. However, containers do not have the full isolation of a VM, and thus implementing Docker for autograding ends up facing its own set of security concerns, as well as with the increased system resource usage. In this poster, we will analyze how well containers work, measuring system resources and throughput of submissions of containers against the traditional jailed environment.