[XviD-devel] big trellis bug

[skal]

	Hi all,

On Sun, 2003-11-23 at 11:51, Radek Czyz wrote:
> > gruel wrote:
> > Lambda = 0xfe00/(q*q) = 65024/4 = 16256
> > 727*727*Lambda = 8591767424 which has 33 bits!

	OMG, Gruel is right. I overlooked big levels...

> 
> I found that too ;) I tried 1<<8 times smaller multiplier for lambda
> and bits and everything went well. Nice. I sure hope we don't need
> such a huge precision.

	Sure, for big levels, chances are high they are already
	coded Esc2 or Esc3 (32bits!). We shouldn't even try to
	RD-optimize them. I propose a threshold on levels being
	examined...


> 
> Now, I have a question: do we do that for mpeg quant? I'd say no, but
> this division-by-multiply doesn't make the picture clear.

	i've already sent GomGom a patch for mpeg-quant. It's just
	a matter of having the variable 'Mult' and 'Bias' within the
	loop. Not a big deal...

> If we don't do that, I can turn the "specialized loop for -1, 0, 1"
> into "specialized loop for -1, 1" correct? Or in fact, I don't need
> any specialized loop.
> 

	There's no need for a special loop. In fact 'level=0' is
	never examined in itself. Only 'Level' and 'Level-1' are.
	The case Level=0 is handled separately, disregarding
	the value of Level (silly, for big levels!:), during the
	Run_Costs[] update.

> If I understand well, I just need to use mpeg dequantization formula
> to calculate dQ1 and dQ2 and I should have trellis for mpeg ready :]

	later!
Skal