[XviD-devel] mpeg intra quantization

[Radek Czyz]

Hi everyone, long time no see :)

Alex_W on irc pointed out that xvid's intra quantization is weird, 
mostly when compared to libav's. I went on to debug it, and it seems 
that it's rounding coefficients *up*, creating huge frames at low psnr.

For example, if mpeg matrix entry is 32 at Q=1, a level of 1 is 
quantized to 1:

level = ((level << 4) + (intra_matrix[i] >> 1)) / intra_matrix[i];
level = ((level + quantd) * mult) >> SCALEBITS;

which is
level = (1*16+16)/32 = 1;
level = (1+1)*65537/(2*65536) = 1;

This is later dequantized to 4.

Bad comparison happens at different quants too, so this is not limited 
to Q1 or anything like that.

In this simple example, changing quantd=0 gives:
18:30:11 : <Alex_W> xvid 1.1: 17772 KB 41.8 dB
18:30:11 : <Alex_W> xvid quantd = 0: 10813 KB 46.42 dB

Can someone (Skal?:) ) explain what was *meant* to be happening here?

:)
Radek