Property-Based Testing for the People

Loading...
Thumbnail Image
Degree type
Doctor of Philosophy (PhD)
Graduate group
Computer and Information Science
Discipline
Computer Sciences
Subject
human factors
programming languages
property-based testing
random testing
software testing
Funder
Grant number
License
Copyright date
01/01/2024
Distributor
Related resources
Author
Goldstein, Harrison, Joseph
Contributor
Abstract

Software errors are expensive and dangerous. Best practices around testinghelp to improve software quality, but not all testing tools are created equal. In recent years, automated testing, which helps to save time and avoid developers' blind-spots, has begun to improve this state of affairs. In particular, property-based testing is a powerful automated testingtechnique that gives developers some of the power of formal methods without the high cost. PBT is incredibly effective at finding important bugs in real systems, and it has been established as a go-to technique for testing in some localized developer communities. To bring the power of PBT to a larger demographic, I shift focus to PBTusers. My work is motivated by conversations with real PBT users, accentuating the benefits that they get from PBT and reducing the drawbacks. My work begins with Property-Based Testing in Practice. This in-depth user study establishes a rich set of observations about PBT's use in practice, along with a wide array of ideas for future research that are motivated by the needs of real PBT users. The rest of the work in the dissertation is informed by these results. One critical observation is that PBT users struggle with the random data generation that is a key aspect of PBT's operation. To address this problem, I establish a new foundation for random data generators in Parsing Randomness and extend that foundation to be more flexible and powerful in Reflecting on Randomness. These projects contribute new algorithms that increase developers' leverage during testing while decreasing developer effort. I also observed that PBT users are not always good at evaluating whether their testing was effective. In Understanding Randomness, I establish a new PBT paradigm that gives developers critical insights into their tests' performance and enables new ways of interacting with a PBT system. The Tyche interface developed in that project is now an open source tool with real-world users. By blending tools from programming languages, human-computer interaction, andsoftware engineering, my work increases the reach and impact of PBT and builds a foundation for a future with better software assurance.

Advisor
Pierce, Benjamin, C.
Weirich, Stephanie, C.
Date of degree
2024
Date Range for Data Collection (Start Date)
Date Range for Data Collection (End Date)
Digital Object Identifier
Series name and number
Volume number
Issue number
Publisher
Publisher DOI
Journal Issue
Comments
Recommended citation