Task-fidelity Assessment for Programming Tasks Using Semantic Code Analysis

In computer science and related technical fields, researchers, educators, and practitioners are continuously automating recurring tasks for high efficiency in a wide variety of fields. In higher education, such tasks that educators face are the recurring review and assessment process of students' programming coursework. Thus, various attempts exist to automate the assessment and feedback generation for course homework and practicals in higher education. Those approaches for automated programming task assessment often comprise running automated tests to check for limited functional correctness and potentially style checking for various violations (LINTing). Educators familiar with large-scale automated task assessment are likely used to seeing hard-coded solutions specifically or accidentally designed to just pass the required tests, ignoring or misinterpreting the actual task requirements. Detecting such issues in arbitrary code is non-trivial and an ongoing research topic in software engineering. Software engineering research has yielded various semantic analysis frameworks, such as GitHub's CodeQL, which can be adapted for programming task assessment. We present a work-in-progress programming task analysis framework which employs CodeQL's analysis technology to identify the actual use of task-description-mandated syntactic and semantic elements such as loop structures or the use of mandated data blocks in branching conditions. This allows extending existing course work analysis frameworks to include a semantic check of an uploaded program which exceeds the relatively simple set of input-output test cases provided by unit tests. We use a running example of entry level programming tasks and several solution attempts to introduce and explain our proposed control flow and data flow -based analysis method. We discuss the benefits of including semantic analysis as an additional method in the automated programming task assessment toolbox. Our main contribution is the adaptation of an semantic analysis code framework to analyse syntactic and semantic components in students' programming coursework.