[XviD-devel] big trellis bug

[Christoph Lampert]

On Sat, 22 Nov 2003, Radek Czyz wrote:
> Unfortunately I'm completely stucked. The idea that trellis
> *increases* the level of coefficients from 0 to +/-1, thus
> *increasing* the filesize and *increasing* the psnr at fixed quant
> (look at the first AC coeff here) is completely against my
> understanding of trellis quantization, and quantization in general.

The thing is that the interval around 0 is larger than half step size.  
(dead zone) See Skal's quantization page: 
http://skal.planet-d.net/coding/quantize.html

When quantization step size is 6, all values from -6 to 6 (more?)  are
quantized to 0, although for +-5,+-6 of course +-1 which is dequantized to
+-9 would be less distorsion. This is a very basic form of Rate-Distorsion
opt., because 0s are likely to cost much fewer bits than +-1s. But not
always, so trellis quant checks if maybe the 0 was a "false" 0, and +-1
would be better not only in PSNR but also in R-D sense.  That's why
Trellis has a chance of increasing PSNR at all!

gruel