Reevaluating the small-scope testing hypothesis of answer set programs

As we increasingly rely on artificial intelligence systems, we must ensure that those systems are reliable and need to know how much we can rely on them. In software quality assurance, testing is a useful method to highlight and fix issues during development to avoid unexpected behavior after the system has been deployed. Artificial intelligence engineers are increasingly becoming aware of quality assurance as a requirement. Previous results in the area of answer set programming suggest that a high proportion of errors can be found when testing a program against a small scope, i.e. by inputs from a small domain. However, these results are based on assumptions that may be impractical for testing. To find out whether small scopes remain sufficient in practice, we evaluate several benchmarks against actual test oracles. Our findings suggest that small scopes can indeed find a high proportion of errors, but results depend on the observed benchmark and appropriate test oracles are required to achieve reliable scores.