The verification optimizer feature of PinDown has several functions that reduces the initial test time, i.e. before the actual debugging starts. This saves a lot of time, computer resources and provides a better understanding for what is happening while the customers test scripts are running.

Real Time Status

The key to how PinDown views the test phase is that it does not have to wait until the end of the regression runs in order to see what is happening. It regularly polls the status (the frequency is user defined) which allows it to take action as soon as it sees failures appearing. It also gives the user a view of how far into the regression run we are currently.

realtimestatus 650x151

 Fig 1. Real Time Status

Stopping the tests at an optimal time

PinDown can be setup to react immediately to incoming test failures. For regression runs which normally are clean, i.e. that most of the time come back with all tests passing, it is possible to save a lot of time to tell PinDown to stop all test jobs and start debugging immediately when there is a test failure appearing. The only reason to stop the test jobs is to avoid the debug jobs being queued on the farm after the test jobs, because in the case no time is saved. However, this may not always be necessary. After killing the test jobs on the farm, PinDown debugs the issue and sends out a bug report as fast as possible. It is then possible for PinDown to go back to the test phase and kick off a new test run immediately but the second time it will not stop if it encounters the same issue again. This can allow the customer to get bug reports hours earlier and provide fast feedback to the engineers, but it only works for test suites that are passing most of the time. Otherwise you will miss many of the failures that would have been reported later in the test run.

VO2

Fig 2. Stopping based on historical statistics

There is a more fine-tuned approach to this where the user does not specify PinDown to stop and debug on the first new failure, but when a number of failures have occurred. The user can tell PinDown to stop the tests when there is a 90% chance that all bugs have been found, based on historical statistics. PinDown looks at the historical runs for this specific test suites and determines at which point it is possible to stop the test suite. This opens up the possibility to stop early most of the time and run the complete run only some times. If 90% of the bugs are debugged earlier thanks to this, and the remaining 10% of the bugs slower, there is an overall gain on both debug time and computer resources.

Generating wave forms and verbose logs

Sometimes one of the first thing engineers do sometimes is to re-run a failing test in order to generate wave forms or verbose logs. It is not possible to do this for all tests because that would consume too much disk space, which is why this manual re-run is often necessary. The downside is that time is lost doing this. PinDown can be setup to generate wave forms or verbose logs for just one test per bugs saving the engineers this hassle and making the testing more efficient by providing all the needed information from start.