Browsing by Author "Dewey, Marco"

Now showing 1 - 1 of 1
  • Thumbnail Image
    Item
    Large language models and software testing
    (Stellenbosch : Stellenbosch University, 2024-03) Dewey, Marco; Inggs, Cornelia P. ; Visser, Willem; Stellenbosch University. Faculty of Science. Dept. of Computer Science.
    ENGLISH ABSTRACT: This thesis examines the viability of leveraging transformer-based large language models, exemplified by Codex, f or the a utomated g eneration of test suites in production software. By leveraging the abilities large language models exhibit for understanding and generating natural and coding languages, these models can analyze code and comments to generate contextually relevant test cases. Using these models in the domain of automatic software testing presents a potential solution to the oracle problem. The research involves a comparative analysis between Codex and a promi- nent automatic testing tool, EvoSuite, using the Commons-Lang library from the Defects4J benchmark. This comparison draws insights regarding Codex’s efficacy in ge nerating co verage te sts an d id entifying fa ulty be havior within production code. The findings o f t his thesis a rgue t hat C odex w hile demon- strating promise, exhibits limitations as an automatic testing tool in achieving high test coverage and uncovering software bugs. Moreover, the study high- lights potential challenges associated with utilizing open-source repositories for training and testing code generation by large language models, including the risk of incorporating inconsistent coding conventions and suboptimal software testing practices into these models.