[XviD-devel] better trellis?

[Radek Czyz]

Hi everyone.

I'm starting to think about a backend for HVS modules. One of the 
possible ways of using HVS data in encoder is to modulate lambda during 
trellis quantization - more important parts of the picture get higher 
lambda, less important get lower.

It's a good method, more precise and less costly than adaptive quantization.

There is a problem with current trellis though. It is only able to 
decrease bitrate with low lambda, I can't count on it when I try to 
increase quality.

This is because of the way it works: Firstly, normal quantization is 
performed. Secondly, trellis optimizes the coefficients, but mostly by 
decreasing levels. It will try to increase some zeroes to ones but only 
some.
The most extreme case is when block is zero after "normal" quantization. 
Trellis will not be able to touch it at all, even if HVS model suggests 
high lambda.

What I have done: instead of doing "normal" quantization before trellis, 
I'm using a "minimum error" quantization. Such quantization has no 
deadzone and no bias, and if the coefficients were kept this way, it 
would be horribly inefficient. But this is where trellis kicks in: it 
lowers (ONLY lowers) the levels to make it efficient. Since it only 
lowers, it does less work (in many cases) than before, as there is only 
one "direction" to check.

Depending on lambda, it is now perfectly possible to lower the 
coefficients less or more, to achive a per-block quality that we need.

My current test results are a bit surprising. Bitrate at Q4 goes up by 
whole 6%, PSNR goes up accordingly. At 2-pass, average PSNR is the same 
as before (+/- 0.02dB) but the minimum PSNR (worst frame) goes up by as 
much as 0.5dB. This seems to be the only benefit until we have HVS 
models to modulate the quality - but this benefit is quite high, as it's 
exactly the worst-case that is seen.

I don't have a proper code to show you yet, but I'll create something 
interesting soon. In particular, I'll try moving trellis to separate 
file - it's so messy currently :/ There is also a couple of ther fixes I 
want to try (cbp cost in not 2 bits per coded block, especially not in 
bframes).

Comments, suggestions?

Radek

PS. perhaps such trellis will like another lambda by default - quality 
might increase