[XviD-devel] big trellis bug

[Christoph Lampert]

Hi guys,

I can't believe it's for R-D reasons that a value of 
-727 is quantized to 0 by Trellis. That's distorsion of 
528529 instead of 1, no lambda can compensation for that. 
Not even for quant 31, but definitely not for quant 2. 
I believe it's a bug. Maybe overflow.

Lambda = 0xfe00/(q*q) = 65024/4 = 16256
727*727*Lambda = 8591767424 which has 33 bits!

How about an upper limit for coeffs to be zeroed, 
or a lower fixpoint accuracy for lambda?

chl



On Sun, 23 Nov 2003, Radek Czyz wrote:
> Hi skal,
> 
> Gee, I even had the original values pasted to the mail, but then my
> bug-producing part of the brain told me "who needs it, it's a flood".
> 
> One more time:
> 
> [684] after dct
> [684] -727, 4, 7, -2, 1, 3, 1, 0
> [684] -13, 9, -1, 2, 2, 0, 0, 0
> [684] 5, -4, 1, -1, -1, 1, 0, 0
> [684] -5, 4, -1, 0, 1, 1, 0, 0
> [684] 5, 0, -1, 0, 0, 0, 0, 0
> [684] 0, -2, 0, 0, 0, 0, 0, 0
> [684] 0, 2, 1, 0, 1, 0, 0, 0
> [684] 0, 0, 1, 0, -1, 1, 0, 0
> 
> [684] NORMAL QUANT
> [684] -181, 0, 1, 0, 0, 0, 0, 0
> [684] -3, 2, 0, 0, 0, 0, 0, 0
> [684] 1, 0, 0, 0, 0, 0, 0, 0
> [684] -1, 0, 0, 0, 0, 0, 0, 0
> [684] 1, 0, 0, 0, 0, 0, 0, 0
> [684] 0, 0, 0, 0, 0, 0, 0, 0
> [684] 0, 0, 0, 0, 0, 0, 0, 0
> [684] 0, 0, 0, 0, 0, 0, 0, 0
> 
> [684] TRELLIS QUANT
> [684] 0, 1, 1, 0, 0, 0, 0, 0
> [684] -3, 2, 0, 0, 0, 0, 0, 0
> [684] 1, -1, 0, 0, 0, 0, 0, 0
> [684] -1, 0, 0, 0, 0, 0, 0, 0
> [684] 1, 0, 0, 0, 0, 0, 0, 0
> [684] 0, 0, 0, 0, 0, 0, 0, 0
> [684] 0, 0, 0, 0, 0, 0, 0, 0
> [684] 0, 0, 0, 0, 0, 0, 0, 0
> 
> As you have already guessed, quantizer is 2 :)
> 
> Thanks,
> Radek
> 
> _______________________________________________
> XviD-devel mailing list
> XviD-devel at xvid.org
> http://list.xvid.org/mailman/listinfo/xvid-devel
>