[XviD-devel] Crash in BGR_TO_YV12/bgr_to_yv12_mmx

Michael Militzer michael at xvid.org
Mon Mar 30 17:06:54 CEST 2009


Hi,

I think you are right. RGB24 conversion will read one byte beyond the
input buffer for image widths that are multiple of 16.

It is known that the MMX code cannot handle arbitrary image sizes.
Therefore, there's the safe_packed_conv() function in src/image/image.c
which is supposed to split up the image into an area that can be safely
converted using the MMX functions and a remaining part that's handled
using the C implementation.

Obviously, safe_packed_conv() did not cover all potentially problematic
cases for RGB24 conversion properly. I've commited a patch to the HEAD
branch that should fix your issue. You may want to give it a try and
report back if it solves your problem.

Thanks,
Michael


Quoting Matthew Allen <fret at memecode.com>:

> Hi,
>
> I'm new here... so go easy. Just wanted to report a crash and some   
> analysis of the problem. I'm not sure on the best way to fix it, but  
>  if you know the what and the where... the "how" should be easy right?
>
> Ok, so when I feed 640x480 frames into xvidcore 1.2.1 via   
> AVIStreamWrite I get a crash in one of the instances of the   
> BGR_TO_YV12 macro, specifically the movd mm1, [x_ptr+%1]   
> instruction. In my case I'm passing in BGR 24bit pixels, so the   
> macro is bgr_to_yv12_mmx. So anyway, it's processing pixels in 2x2   
> groups right, and it does this fine until it reaches the end of the   
> first scanline. My bitmap are Y fliped, so the end fo the first   
> scanline is also the end of the block of memory containing the input  
>  bitmap. Can you guess where this is going? lol. So it gets up to  
> the  last 2x2 block of pixels, 6 bytes on the last scanline, and 6  
> bytes  on the 2nd last scanline... and does 4 x movd to read them  
> in... and  how many bytes does movd read? 4... this line   
> (colorspace_rgb_mmx.asm:136):
>
>   movd mm1, [x_ptr+%1]            ; src[%1...]
>
> Expands to
>
>   movd mm1, [edi+3]
>
> And when that is pointed at the last 3 bytes of the memory buffer   
> (i.e. where edi/x_ptr has 6 bytes left).... bang, access violation   
> as it does a read 1 past the end of the bitmap buffer.
>
> I guess if xvidcore has control over the buffer allocation, you   
> could just alloc 1 more byte (it gets thrown away). Or alternatively  
>  do some special processing for the last pixel block to avoid  
> reading  off the end of the memory block.
>
> In any case I've worked around it by using the C implementation   
> instead. But it'd be great to see a fix rolled into the next release.
>
> Regards
> --
> Matthew Allen
>
> _______________________________________________
> Xvid-devel mailing list
> Xvid-devel at xvid.org
> http://list.xvid.org/mailman/listinfo/xvid-devel
>
>





More information about the Xvid-devel mailing list