The development team of the open source mailer 'Thunderbird' explains 'development techniques to reduce bugs'



In the email software '

Thunderbird ,' the development team runs automated tests on a daily basis to improve the quality of the software. The Thunderbird development team explains the reasons and methods behind this.

Automated Testing: How We Catch Thunderbird Bugs Before You Do
https://blog.thunderbird.net/2024/04/automated-testing-how-we-catch-thunderbird-bugs-before-you-do/



◆ Purpose and benefits of automated testing
The Thunderbird development project places emphasis on 'automated testing' to minimize the occurrence of bugs due to code changes. According to the development team, every time a change is made to Thunderbird's code or features, automated tests are run on Windows, macOS, and Linux test machines to detect mistakes and unintended consequences. It is also said that testing time for one change takes 60 to 80 hours.

The development team further stated, 'By implementing automated testing, we were able to significantly reduce the time spent testing updated code before it was released with each monthly update. This has significantly reduced the burden and pressure on the team.'



On the other hand, the development team explains the reason for performing automated testing, saying, 'We don't just run automated tests to make ourselves feel good.'

・Prevent mistakes
By testing whether a piece of code works as expected, development teams can find and fix bugs without having to wait for users to report them.

- Check for Firefox bugs
Thunderbird shares a large portion of its code with Firefox, and changes to Firefox result in changes to Thunderbird's code, so running automated tests helps catch bugs in the Firefox code early.

・Reducing the burden on human testers
The introduction of automated testing frees humans from tedious, repetitive tasks that are difficult to set up, allowing for the proper use of limited human resources. Jeff Lankow of the development team said, 'I recently added tests to the automated tests that check for 22 ways to trigger email retrieval and 10 situations in which email retrieval may not work. It would be difficult for a human tester to check all of these, but by having a computer perform the tests, it is possible to test multiple times a day.'



・Discovering existing bugs
Some of Thunderbird's source code contains very old code. By testing these codes with a computer, it is possible to identify mistakes made when the code was written and where defects have developed over time. In addition, running automated tests is also useful for understanding the contents of old code.

In addition, the development team revealed that since the release of version 115, they have found ways to perform automated tests for 'communication from mail servers using encrypted channels,' 'two-factor authentication on mail servers,' and 'communication with web servers using specific addresses that should not use unencrypted channels.' These methods were not possible when human testers were conducting the tests.

◆Code coverage management
The development team records and

publishes progress on ' code coverage, ' which indicates the percentage of code that has been executed during testing.

Below is a graph showing the progress of code coverage. The vertical axis shows the code coverage value, and the horizontal axis shows the date when the code coverage was recorded. Looking at the graph, we can see that the code coverage value is slightly decreasing. This indicates that 'the amount of third-party code that is not involved in the development team has increased.'



Code coverage records also help make unmaintained code visible. Development teams are already deleting unused code found in code coverage records.

The development team cites the benefits of recording code coverage as making it easier to find incorrect code and memory leaks, and making it easier to remove unused code, resulting in reduced maintenance time.

◆ Test suite migration
Thunderbird has announced that it will discontinue its test suite, Mozmill , at the end of 2023 and move to another test suite called Mochitest . The development team said, 'Mozmill was clever but used strange tricks, so we moved to Mochitest, which uses more orthodox methods.'

◆Increasing test code
The percentage of test code in Thunderbird's total code has been increasing year by year, from about 19% in January 2019 to about 27% in January 2024. The development team said, 'We plan to ask contributors to add tests to their patches more frequently in the future.'



◆ Forum is currently open
A forum related to this article has been set up on the official GIGAZINE Discord server . Anyone can post freely, so please feel free to comment! If you do not have a Discord account, please refer to the account creation procedure explanation article to create an account!

• Discord | 'Tell us what you do to reduce bugs in software development!' | GIGAZINE
https://discord.com/channels/1037961069903216680/1227909826965930065

Related Posts:

in Software, Posted by log1r_ut