[XviD-devel] big trellis bug

[Christoph Lampert]

On Sun, 23 Nov 2003, Radek Czyz wrote:
> > gruel wrote:
> > Lambda = 0xfe00/(q*q) = 65024/4 = 16256
> > 727*727*Lambda = 8591767424 which has 33 bits!
> 
> I found that too ;) I tried 1<<8 times smaller multiplier for lambda
> and bits and everything went well. Nice. I sure hope we don't need
> such a huge precision.


I hope you changed the other shifts as well? ;-)

> this division-by-multiply doesn't make the picture clear.
> If we don't do that, I can turn the "specialized loop for -1, 0, 1"
> into "specialized loop for -1, 1" correct? Or in fact, I don't need
> any specialized loop.

Hm, it might still happen that strange Run-Level combinations 
have less bits when a 0 is turned a 1, right?

gruel