AutoStyle: Automatic Hints for Improving Student Code Style

[bibshow file=fox.bib]

Students: HeZheng Yin (PhD, on leave), Joseph Moghadam (MS 2015), Rohan Roy Choudhury (MS 2015), Antares Chen, Eliane Wiese (postdoc 2018, now faculty at University of Utah)

Advised by Armando Fox

Our focus is on the (relatively under-researched) area of helping “post-CS1” students build the skills expected of experienced programmers. The upper-division Intro to Software Engineering course and the accompanying project course are representative of those skills.

Among the most important are working effectively in a software team (a group skill we address in our Teamscope work) and writing maintainable code (an individual skill). In Robert C. “Uncle Bob” Martin’s book Clean Code: A Handbook of Agile Software Craftsmanship, Ward Christensen is quoted as saying that beautiful code “makes it look like the language was made for the problem.” Clean code is more maintainable because, to an expert, it more readily reveals its design intent and is well factored.

Inspired in part by my experience teaching Berkeley’s first MOOC on software engineering, the idea behind AutoStyle was that when many students submit functionally-correct solutions to the same programming problem, some students’ code will be closer to what an expert would produce than that of other students. Could we automatically identify which aspects of the “better” solutions made them better, and synthesize hints to give to students who came up with weaker solutions?

We built a tool called AutoStyle that did just this, by using existing software engineering static-analysis tools to identify “better styled” code (empirically, we found that a low “ABC score”, which counts for assignments, branches, and conditions, was a good proxy for conciseness), analyzing the abstract syntax trees of correct submissions, and arranging them into a tree whose root represents (one of the) “best” solutions and in which an edge connects a child to a parent if the child is within a threshold Zhang-Shasha edit distance of the parent. We then find a “chain” of transformations from any given student’s solution to the root, and propose the required transformations as hints to the student to gradually refine their code.

We did a randomized controlled trial [bibcite key=autostyle-its2016] in which students were asked to continue to improve and resubmit their code until the style “score” exceeded some threshold. Students who received automated restructuring hints from AutoStyle produced statistically significantly better-styled code than a control group who were given the same style guidelines but no hints.

But this promising result was dampened by other findings. In our study, the best results were achieved by students whose overall “game plan” for writing the code was sound (i.e. close to that of an experienced programmer); in that case, AST-based style hints would get them the rest of the way there. But students who started with a poor game plan–usually because they failed to recognize that the coding problem lent itself to the use of particular patterns or idioms–were unable to use the hints productively; the best advice to them would basically be “start over with a different game plan.” This suggested that before asking students to write code and receive hints, we should ensure that they had identified the correct “game plan” (programming pattern) for solving the problem.

Second, we subsequently found [bibcite key=autostyle-las2017] that while working with the AutoStyle “tutor” helps students write better code, they are not able to transfer that knowledge once the tutor’s scaffolding is removed. Part of the reason for this lack of transfer is that novice students cannot always recognize beautiful code, so they don’t share an expert’s opinion that that code is better [bibcite key=autostyle-readability-comprehension-icse2019], but we noted that exposure to AutoStyle did improve their ability to recognize beautiful code.

We concluded that we needed to (a) repeatedly expose students to “expert” programming patterns so that they would be sure to recognize them when appropriate for a programming problem, and (b) find a way to elicit from the student some evidence that they knew which pattern was appropriate to use for a given problem, before starting to write code for the problem and getting style feedback. Those observations ultimately led to our work on Scaffolding the Learning of Programming Patterns and Subgoal Decomposition.