[XviD-devel] big trellis bug

[Christoph Lampert]

On Sun, 23 Nov 2003, Edouard Gomez wrote:

> Edouard Gomez (ed.gomez at free.fr) wrote:
> > I'm currently  testing the changes.  I also  introduced TL_SHIFT=10 to
> > avoid overflows.  It doesn't fix the real problem but avoids its nasty
> > effects.
> 
> Ok tests done:
>  - h263 works hopefully as before (output matches exactly if i set
>    TL_SHIFT 16)
>  - MPEG seems to benefit from trellis as well ;-)
> 
> Nota bene, i suffered the chroma bug in trellis for the first time while
> running a mpeg+trellis  test (a red block appeared in  a white zone).  I
> guess we should really leave DC unoptimized for high values.
> 
> May a simple thresholding be enough ?

Yes, of course. Current 16bit shifts can only overflow for coefficients
with more than 8 bits. Also, I doubt that anything above +-128 will ever
be quantized to 0, so checking for abs(coeff)<256 would be the safe side.

gruel