[XviD-devel] automated xvid system test?

[Mat Hostetter]

I've been thinking about writing an automated xvid "system test" as a
way of learning the encoding/decoding API.

I'm envisioning a test that encodes and then decodes a video sequence,
with each resulting frame compared against the original frame.  For
each frame, compute an error metric and have the test verify that the
average per-frame error rate and the max error rate of any frame fall
within some bounds.  It could also verify that the total encoded bits
generated have some reasonable relation to the requested bitrate.

Such a test could be run with various combinations of encoding flags
and bitrates to test much of the xvid code.  I suppose a code coverage
tool could even say what cases were not hit and the test could be improved.

As for the test input video, it could be a real video snippet, but
that would bloat the download, especially if uncompressed.
Alternatively I could write a program that generates numerous frames
of geometric shapes bouncing about the screen, with some tweaks to
make the test more interesting like slowly mutating sizes and colors,
rotations, gradient fill patterns, etc.  A few small multi-frame
animations of some real-world video bouncing around the screen might
add a touch of less-cartoony realism to the test too.  Generating
these images in a larger frame and then scaling it down (antialiasing)
to the actual test frames would result in a more realistic test
(objects would effectively be subpixel-positioned).

I'm not sure what the right error metric would be.  An ideal error
metric would take into account properties of the human visual system,
but that's not realistic.  A simple error metric would be the
sum-of-absolute-differences of the pixel values before and after the
round-trip compression/decompression.  That has the drawback that a
vertical line shifted over one pixel is a larger error than having the
vertical line disappear altogether, which could be bad.  Perhaps
taking the "min" of absolute differences between a "new" pixel and
each of the 9 nearest "original" pixels would be a more reasonable
error metric, since the cases where this is pathologically inaccurate
are less likely to occur.

-Mat