[XviD-devel] qpel - problems I found

[Radek 'sysKin' Czyz]

Hi everyone

I did my best to understand how qpel (especially qpel ME) works and I
have to say that I succeed ;-)

As a result, I found some problems and minor mistakes that have been
done so far (so far == not in a long time ;> it's a new code after
all).

First: decoding of qpel frames fails on other decoders. We all know
that, and I've found a reason.
This is because when you do motion compensation for chroma, there is:

         dx = (dx >> 1) | (dx & 1);
         dy = (dy >> 1) | (dy & 1);

Well, if I remember correctly doing dx >>1 on a potenetialy negative
value doesn't work as it should - am I right? (not a rethorical
question, I'm not sure).
Anyway, changing it to
        dx = dx/2;
        dy = dy/2;
fixed the problem on all decoders, and also decreased filesize. Same
thing exist in decoder.

Second:
Right before qpel refine, SADs consist of 'real' SAD and d_mv_bits,
where d_mv_bits is the vector weight *for halfpel conditons*. When you
do qpel refinement, you compare it against 'real' sad + d_mv_bits(qpel
conditions). As d_mv_bits[qpel] is statistically twice as big, there
is a big possibility that comparsion is not fair and some vectors
(qpel refinement vectors) have worse chances.

I've made a fix - or rather a hack. I substract d_mv_bits[hpel] and
add d_mv_bits[qpel] before qpel refinement.

It's a bit more complicated with 8x8 search - I have to substract
mv_bits[qpel] and add mv_bits[hpel] at the beginning. Then, I do
halfpel search and move to qpel again and do qpel refinement....

I'm sure there is a better way for all of this (like using
d_mv_bits[qpel] from the beginning) but I won't fix it until the code
settles more.

There is a reasonable filesize gain after this fix.

Third:
While doing "normal" search maximum vector ranges are different. This
has been taken into account by decreasing fcode by 1 which is quite
smart.
However, when you move to qpel refinement there is no other way but
re-calculate "real" maximum range, and this has been forgotten.
The filesize/psnr gain from this fix is also.. well not big but
measurable.

Now, a question: what happened to hinted ME? second pass crashes,
even for normal halfpel.

Now, a suggestion: checking of qpel vectors consists of: making average
of two pictures and computing SAD between this average and reference.
This is ok, but tell me: how is it different then doing sad8bi from
the beginning? Of course sad8bi doesn't have rounding value but it
shouldn't be a problem.
Doing it this way seems to be the fastest.

Best regards,
Radek