FinalUnit: AI driven Unit Test Generation for Golang
Unit testing in software development is fundamental when creating successful and long term software products. I like to look at unit tests as a mechanism to anchor the current behaviour of a piece of functionality. Therefore, by writing sufficient unit tests, we can ensure that the behaviour of pieces of software does not change over time. Having this behavioural anchoring, developers working on a software project can create new features while knowing existing functionality does not break. This results in an overall productivity improvement in the long term. Golang offers all necessary tools for writing proper unit tests out of the box and a lot of open source options are available as well such as Goblin, Ginko and go-testdeep. Personally, I am a big fan of the testify library.
Now that we understand the importance of unit tests and how they can help us, let us discuss some of the challenges. Writing unit tests may take up a significant amount of time. Especially when working towards an MVP, we do not always allow ourselves the time to write proper unit tests. Furthermore, managers or product owners might be more interested in creating new features and do not always understand the importance of software quality and the impact on projects. Therefore, the big question is: how can we achieve the best of both worlds? Well, what about automating the creation of unit tests? That way, we make sure to still anchor the behaviour of our code, but save up a considerable amount of time. That is exactly what FinalUnit does. FinalUnit is a command line tool to generate unit tests for a given folder containing Golang source code.
How does it work
FinalUnit consists of three steps. First inputs are created for functions. Afterwards, using Evolutionary Based Machine learning, the initial values are evolved in order to try and increase the overall test coverage as much as possible. When the evolution is complete, FinalUnit runs the source code with the generated inputs and records the output of the functions. The final result are written to a test file.
Input generation
First of all, functions present in the source code are analysed. For every function parameter and function receiver type, random input values are generated. Per default, FinalUnit creates 10 test cases for every function. If imports are used, FinalUnit will also try to create values for imported types. Private types and exported interfaces with private functions in imports are skipped, as those values can not be set from outside a package in Go.
Evolution
FinalUnit tries to maximise the overall test coverage for the generated test set. This is done using Evolutionary Machine Learning. The idea behind this type of machine learning is very straight forward. We start with a population of organisms, a group of beings with some form of DNA. This population then starts evolving. Every organisms has a fitness, this fitness score depicts how well that organism is functioning in the world. Organisms with a high fitness have a bigger chance to reproduce. The theory states that this will increase the overall fitness of the population over time, after all the strongest genes have a higher chance to end up in the next generation, thus increasing the overall fitness.
Every organism in FinalUnit consists a set of test cases for all functions in a given directory. The fitness of such organism, is defined by the test coverage of the test set. By evolving this population, FinalUnit tries to increase the overall coverage over time and find a test set with a maximum coverage. The generator halts when either the target coverage is hit, or a given amount of generations did not show any improvements.
Behaviour recording
When the evolution is complete, the organism with the best fitness is taken. This test set is executed and the outputs of all functions which are tested are recorded and written to file as the resulting test cases. In case function outputs are non deterministic, the generator will add a FIXME telling the user to add assert statements manually.
Example
Let us take a look at an example. Below a snippet is shown containing a dummy function called Bar. This function contains an if else chain with 5 so called code branches. Each of these branches can be taken based on the input value x.
We start the FinalUnit cmd tool in verbose mode using: finalunit -v. The output of the generator is shown in the snippet below. The generator starts by creating the first generation of organisms. Afterwards, it is going to start evolving the population. For every generation the best fit is determined and the average fit of the generation is logged. Furthermore, the generator logs which generation we are looking at and how many generations we have encountered without encountering a new best fit. This example consisted of three generations:
- Generation zero: One organism with a test coverage of 88.9% was found and the population has an average fit of 72.24%
- Generation one: The best fit remains the same, but the average fitness increased to 81.13%
- Generation two: This generation contains an organism with a fitness of 100%. This means the generator can halt and create the test results
The generated test suite is shown in the snippet below. Note that for every branch an assert statement is created. Another interesting thing to note are the random variable names. This makes the code somewhat less readable, but ensures that the code always compiles.
Limitations
Nobody is perfect. FinalUnit is incapable of handling functions that terminate a program e.g. calling os.Exit. This might also be called by some log functions such as log.Fatal. Another limitation is that FinalUnit is not always able to generate meaningful values for types. Take for example the time.Time struct from the standard library. This struct only contains private fields, which makes it impossible for FinalUnit to create a meaningful time.Time struct. To cope with these limitations, FinalUnit provides so-called decorators. These decorators allow users to tell the generator to either, exclude functions and files from generation, or to add custom values for input parameters or function receiver types. Details about using the decorators are available at the documentation page.
What is next?
We want to use the coming period to gather as much information as possible. First of all, we want to know what people like about our tool and more importantly, what could be improved. The generator also does not create input values for channels yet, so that will be one of the first new features.
Conclusion
Nice, automatically generated Unit Tests, does this mean that I never have to write unit tests ever again? No. It does not. However, we think FinalUnit is a great tool to quickly generate a set of tests for your source code. By looking at the generated unit tests we can determine if the written functionality behaves as expected. It can also be a warm welcome when you want to refactor a project containing legacy code. Being able to simply create unit tests for legacy code makes refactoring much easier. Furthermore, the tool could be used to increase the overall test coverage on packages by generating additional test cases next to existing ones.
Friendly warning: the tool does execute your source code, so always make sure no unwanted behaviour can be triggered by executing your code. Interested? You can find download instructions here. We sincerely hope you find it useful. If you like the product or have any suggestions for improvements please let us know!
Happy holidays everyone!
