[XviD-devel] malloc(0) and potential underflow error?

[Sriram Sankaranarayanan]

Dear Developers,

I am a program verification researcher at NEC Labs. I am trying to use our
tool F-Soft to find bugs
in open source projects (xvidcore being one of them).

In the process of examining our tool reports, I came across a potential
malloc(0) and an underflow related to it. I am not familiar
with the details of your code (even though I am working to familiarize
myself). I attach some details. Please email me
if you have questions.

Project: xvidcore version 1.2.1
            Ditto for xvidcore version 1.2.2 (latest version).
             http://downloads.xvid.org/downloads/xvidcore-1.2.2.tar.gz

File: xvidcore/src/plugins/plugin2_pass2.c

Function: scale_curve_for_vbv_compliancy (plugin2_pass2.c: line 1520).
            The tool claims that
                 1. Value of  num_scenes at  lines 1565-1569 can be zero
(malloc(0) is likely)
                 2.  potential underflow at line 1591
(scenelength[num_scenes -1] is the expression accessed, num_scenes=0).

A small pedantic note about malloc(0) :-)
        http://blogs.ethz.ch/syslab/2007/04/02/malloc0-and-freenull/

Our own examination suggests that this is likely if  the condition
                  if ((rc->stats[i].type == XVID_TYPE_IVOP) &&
(i-last_scene>min_scenelength)) {
at line 1559 is not enabled.

If at all possible, feedback on the possibility of this bug will be
appreciated.

Thanks,
Sriram Sankaranarayanan