[XviD-devel] Bug+patch: mem corruption w/ unaligned bitstream writes, take 2

[Alex Volkov]

Hello,

I am reposting this as I never got any response the first time.

There is a bug in 1.1-HEAD Bitstream init/writing functions that causes
memory corruption when a bitstream passed to enc_encode() is not aligned on
32bit boundary. This can wreak havoc on encoder output buffer. If you do not
write out the data immediately after enc_encode() returns it, and instead,
simply move the bitstream ptr, on the next enc_encode() operation, the
previous couple bytes (the unaligned ones) get clobbered.
For example, you may choose to forego flushing the dx50-compat empty marker,
which is 7 bytes long, and simply advance the output ptr by 7 bytes. On the
next enc_encode() operation, the last 3 bytes (which contain the
time_increment) get clobbered. This of course causes severe block artifacts
during decoding.

The patch to fix unaligned bitstream inits is attached.

-Alex.

-------------- next part --------------
Index: bitstream.h
===================================================================
RCS file: /xvid/xvidcore/src/bitstream/bitstream.h,v
retrieving revision 1.22
diff -u -u -p -r1.22 bitstream.h
--- bitstream.h	23 May 2005 09:29:43 -0000	1.22
+++ bitstream.h	9 Dec 2005 15:17:22 -0000
@@ -193,8 +193,12 @@ BitstreamInit(Bitstream * const bs,
 #endif
 	bs->bufb = tmp;
 
-	bs->buf = 0;
 	bs->pos = bs->initpos = bitpos*8;
+	/* preserve the intervening bytes */
+	if (bs->initpos > 0)
+		bs->buf = bs->bufa & ~((1 << (32 - bs->initpos)) - 1);
+	else
+		bs->buf = 0;
 	bs->length = length;
 }
 
@@ -220,7 +224,11 @@ BitstreamReset(Bitstream * const bs)
 #endif
 	bs->bufb = tmp;
 
-	bs->buf = 0;
+	/* preserve the intervening bytes */
+	if (bs->initpos > 0)
+		bs->buf = bs->bufa & ~((1 << (32 - bs->initpos)) - 1);
+	else
+		bs->buf = 0;
 	bs->pos = bs->initpos;
 }