[XviD-devel] big trellis bug

[Radek Czyz]

Hi everyone,

I spent this day trying to catch an ugly bug. In the end, I got the
following debug output from mbtransquant, showing the processing of
one (chroma) block:

[912] AFTER QUANT
[912] -181, 0, 1, 0, 0, 0, 0, 0
[912] -3, 2, 0, 0, 0, 0, 0, 0
[912] 1, 0, 0, 0, 0, 0, 0, 0
[912] -1, 0, 0, 0, 0, 0, 0, 0
[912] 1, 0, 0, 0, 0, 0, 0, 0
[912] 0, 0, 0, 0, 0, 0, 0, 0
[912] 0, 0, 0, 0, 0, 0, 0, 0
[912] 0, 0, 0, 0, 0, 0, 0, 0

[912] AFER TRELLIS QUANT
[912] 0, 1, 1, 0, 0, 0, 0, 0
[912] -3, 2, 0, 0, 0, 0, 0, 0
[912] 1, -1, 0, 0, 0, 0, 0, 0
[912] -1, 0, 0, 0, 0, 0, 0, 0
[912] 1, 0, 0, 0, 0, 0, 0, 0
[912] 0, 0, 0, 0, 0, 0, 0, 0
[912] 0, 0, 0, 0, 0, 0, 0, 0
[912] 0, 0, 0, 0, 0, 0, 0, 0

See what happened to DC value? Gone...
I was doing my best to understand what happened to it. Here is the
reconstruction of the block from all trellis nodes:

[912] node 9, value -1 run 1
[912] node 8, value -1 run 3
[912] node 5, value 1 run 1
[912] node 4, value 2 run 1
[912] node 3, value 1 run 1
[912] node 2, value -3 run 1
[912] node 1, value 1 run 2

... which is consistent with the resulting block.

Unfortunately I'm completely stucked. The idea that trellis
*increases* the level of coefficients from 0 to +/-1, thus
*increasing* the filesize and *increasing* the psnr at fixed quant
(look at the first AC coeff here) is completely against my
understanding of trellis quantization, and quantization in general.

There must be something very smart happening here but unfortunatly I
was unable to figure out what.

I hope you can help me.... or at least you can fix the bug. Thanks.

Radek