[XviD-devel] Possible bug in bitstream/mbcoding.c

[Martin Thierer]

Hi,

using transcode 0.6.1 (linux) I found that the Xvid snapshot 20020822 
crashed while the snapshot 20020728 worked fine.

After I tried to debug the problem, I think it's a bug in Xvid (that's why 
I post here and not to the transcode-ML).

The following code is from the file bitstream/mbcoding.c, init_vlc_table() 
around line 131:

  run -= max_run_ptr[abs(level)] + 1;	// and change run

  if ((abs(level) <= max_level_ptr[run]) && 
      (run <= (uint32_t) max_run_ptr[abs(level)])) {

	vlc[intra]->code = 0x0e;
	vlc[intra]->len = 9;
	goto loop_end;
  }
  
  run += max_run_ptr[abs(level)] + 1;

The crash seems to happen when max_level_ptr[run] is accessed. It _might_ 
be because "run" becomes negative. "Unfortunatly" the code doesn't crash 
anymore when I insert debug-output, so I can't tell if this is really the 
problem. Anyway, the index becoming negative seems odd to me, but then 
again, I don't understand at all what this function is supposed to do :-) 
So it might be perfectly normal. At least I'm pretty sure the the crash 
happens here.

Please note that this code hasn't changed since snapshot 20020728, but as 
little changes in the code also make the problem go away for me, that 
hasn't much to say. It's probably just a matter of a few bytes if an 
illegal page is hit or not.

Bye,

Martin