[XviD-devel] Crash in BGR_TO_YV12/bgr_to_yv12_mmx

Matthew Allen fret at memecode.com
Mon Mar 30 05:17:40 CEST 2009


I'm new here... so go easy. Just wanted to report a crash and some analysis of the problem. I'm not sure on the best way to fix it, but if you know the what and the where... the "how" should be easy right?

Ok, so when I feed 640x480 frames into xvidcore 1.2.1 via AVIStreamWrite I get a crash in one of the instances of the BGR_TO_YV12 macro, specifically the movd mm1, [x_ptr+%1] instruction. In my case I'm passing in BGR 24bit pixels, so the macro is bgr_to_yv12_mmx. So anyway, it's processing pixels in 2x2 groups right, and it does this fine until it reaches the end of the first scanline. My bitmap are Y fliped, so the end fo the first scanline is also the end of the block of memory containing the input bitmap. Can you guess where this is going? lol. So it gets up to the last 2x2 block of pixels, 6 bytes on the last scanline, and 6 bytes on the 2nd last scanline... and does 4 x movd to read them in... and how many bytes does movd read? 4... this line (colorspace_rgb_mmx.asm:136):

  movd mm1, [x_ptr+%1]            ; src[%1...]

Expands to 

  movd mm1, [edi+3]
And when that is pointed at the last 3 bytes of the memory buffer (i.e. where edi/x_ptr has 6 bytes left).... bang, access violation as it does a read 1 past the end of the bitmap buffer.

I guess if xvidcore has control over the buffer allocation, you could just alloc 1 more byte (it gets thrown away). Or alternatively do some special processing for the last pixel block to avoid reading off the end of the memory block.

In any case I've worked around it by using the C implementation instead. But it'd be great to see a fix rolled into the next release.

Matthew Allen

More information about the Xvid-devel mailing list