[XviD-devel] Bug+patch: mem corruption w/ unaligned bitstream writes

[Alex Volkov]

Hello, everyone.
I am new to the XviD development and this list, so please forgive me if this
is not the most appropriate place for this (I could not find a functioning
bugtracker, or a FAQ on how to post patches).

There is a bug in 1.1-CVS Bitstream init/writing functions that causes
memory corruption when a bitstream passed to enc_encode() is not aligned on
32bit boundary. This wreaks havoc on my encoder output buffer. I do not
write out the data immediately after enc_encode() returns it if the returned
data is too small (for example, the dx50-compat empty marker, which is 7
bytes long). Instead, I simply move the bitstream ptr by 7 bytes. On the
next enc_encode() op the last 3 bytes (which contain the time_increment) get
clobbered. This of course causes severe block artifacts during decoding.

The patch to fix unaligned bitstream inits:

Index: src/bitstream/bitstream.h
===================================================================
RCS file: /xvid/xvidcore/src/bitstream/bitstream.h,v
retrieving revision 1.22
diff -u -r1.22 bitstream.h
--- src/bitstream/bitstream.h	23 May 2005 09:29:43 -0000	1.22
+++ src/bitstream/bitstream.h	16 Aug 2005 21:40:03 -0000
@@ -193,8 +193,12 @@
 #endif
 	bs->bufb = tmp;
 
-	bs->buf = 0;
 	bs->pos = bs->initpos = bitpos*8;
+	/* preserve the intervening bytes */
+	if (bs->initpos > 0)
+		bs->buf = bs->bufa & ~((1 << (32 - bs->initpos)) - 1);
+	else
+		bs->buf = 0;
 	bs->length = length;
 }
 
@@ -220,7 +224,11 @@
 #endif
 	bs->bufb = tmp;
 
-	bs->buf = 0;
+	/* preserve the intervening bytes */
+	if (bs->initpos > 0)
+		bs->buf = bs->bufa & ~((1 << (32 - bs->initpos)) - 1);
+	else
+		bs->buf = 0;
 	bs->pos = bs->initpos;
 }