[XviD-devel] big trellis bug

[Radek Czyz]

> gruel wrote:
> Lambda = 0xfe00/(q*q) = 65024/4 = 16256
> 727*727*Lambda = 8591767424 which has 33 bits!

I found that too ;) I tried 1<<8 times smaller multiplier for lambda
and bits and everything went well. Nice. I sure hope we don't need
such a huge precision.


> The thing is that the interval around 0 is larger than half step size.
> (dead zone) See Skal's quantization page:
> http://skal.planet-d.net/coding/quantize.html

Great page :)

Now, I have a question: do we do that for mpeg quant? I'd say no, but
this division-by-multiply doesn't make the picture clear.
If we don't do that, I can turn the "specialized loop for -1, 0, 1"
into "specialized loop for -1, 1" correct? Or in fact, I don't need
any specialized loop.

If I understand well, I just need to use mpeg dequantization formula
to calculate dQ1 and dQ2 and I should have trellis for mpeg ready :]