[XviD-devel] multithreading

[Radek Czyz]

Hi Lars,

 > fewer threads give me much lower cpu usage

...but faster encoding :)
"CPU usage" in this case rises only because cost and slowness of 
interactions between threads goes higher.

 > You suggest as many threads as cores available?

I can't answer that, I coded that for an early athlon X2 and haven't run 
this code since.

 > I don't know anything about Win stuff. I'm on a Linux box.
 > As mentioned before the VfW is non of my business.

It's not a matter of what would work for you, it's a matter of what 
constraints we put on xvid's API. And it so happens that compatibility 
with VfW is too important for it. Relatively few people use XviD now and 
those that do won't change their toolchain (which is usually virtualdub 
producing standalone-boxes-compatible avi files).

 > - initialize two buffers. one for future I-frames one for future P-frames

You're assuming the two GOPs are independent and they only are in 
constant-quantizer mode. In any other ratecontrol, overflow/underflow 
from one gop has to be incorporated in compression strength of following 
GOP.

And just to illustrate that knowing frame types in advance had nothing 
to do with it: as you accept frames, you could run a frame type decision 
and still use the same algorithm in exactly the same way. It's a valid 
way to parallelize, and if you didn't know frame types in advance it 
would work just as well.

Radek

Lars Täuber wrote:
> Hallo Radek,
> 
> Radek Czyz <radoslaw at syskin.cjb.net> schrieb:
>> Hi,
>>
>>  > In avidemux I set the xvid threads manually to 12
>>
>> Ugh, why did you do that, is it any faster than 2 or 3? If my tests are 
>> any indication it should be significantly slower.
> 
> fewer threads give me much lower cpu usage. But I'll try again. You suggest as many threads as cores available?
> 
>>  > only get a cpu usage of around 250% even on the second run of 2-pass 
>> encoding. The first run is at about 175% cpu usage max.
>>
>> Yeah that's a limitation of both current threading model as well as 
>> current implementation.
>> If someone wants to rewrite implementation with proper mutexes etc (or 
>> in any other way drop Sleep(); win98 compatibility is no longer the 
>> issue it used to be) please go ahead.
> 
> I don't know anything about Win stuff. I'm on a Linux box.
> 
>>  > Could you elaborate the threading model for xvid in simple words? Are 
>> all threads encoding a single frame until it's finished?
>>
>> Yes all threads are motion-estimating (not even fully encoding) a single 
>> frame. For the most part this is required if xvid is supposed to stay 
>> VfW-compatible, because VfW requires that a frame is encoded and written 
>> before next frame can be delivered.
> 
> As mentioned before the VfW is non of my business.
> 
>>  > I took some thinking about how I would do it and therefore liked to 
>> know if the first run on a 2-pass encoding fixes the series of frame types?
>>
>> Yes, frame types are written to stats file by first pass and reused. In 
>> case you wonder this doesn't do anything useful for any parallization.
> 
> I don't agree. Because if you know which frame will be encoded independently (I-Frame) or independently from future frames (P-Frame) you could encode like this:
> 
> - initialize two buffers. one for future I-frames one for future P-frames
> - a dispatcher thread cares that at min. 2(?) encoded I-frames are in the I-buffer
> - same with P-buffer
> - next free thread gets to encode a B-frame if a future P or I-frame are available in the buffers
> --- otherwise get this thread to encode an P or I-frame
> - each thread encodes one frame completely
> 
> When I'm right then there is an encoding thread never waiting for something.
> This might not be possible with VfW as of your comment above. But I don't care.
> 
> How much faster/slower are the B-frames encoded in contrast to P or I-frames? Is there a rule of thumb?
> 
> What do you think about this?
> 
> Lars