[XviD-devel] XviD 1.1.3 and daily snapshot segfaults on Mac OS X 10.5 Intel (with mencoder) and assembly enabled

[Thibaut VARENE]

Hi,

I've been trying to get XviD to work on my Core2 Duo Mac Mini running
10.5 without luck so far.

In order to get 1.1.3 to build I had to follow the receipe here:
http://rob.opendot.cl/index.php/useful-stuff/xvid-with-asm-on-os-x/

but then mencoder would segfault when using the xvid codec.

I tried using the daily snapshot from yesterday (20080919) with the
same result (also tried mencoder snapshot). Disabling assembly in the
configure option yielded a working XviD codec, but obviously with
perfs that made it totally useless.

I've narrowed down the problem to fdct_sse2_skal (so far):

gdb mencoder
GNU gdb 6.3.50-20050815 (Apple version gdb-960) (Sun May 18 18:38:33 UTC 2008)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-apple-darwin"...Reading symbols for
shared libraries .................... done

(gdb) run dvd://4 -dvd-device /Volumes/Toto -nosound -ovc xvid
-xvidencopts pass=1 -o /dev/null
Starting program: /Users/varenet/MPlayer-1.0rc2/mencoder dvd://4
-dvd-device /Volumes/Toto -nosound -ovc xvid -xvidencopts pass=1 -o
/dev/null
Reading symbols for shared libraries
+++++++++++++++++++.......................................................................
done
MEncoder 1.0rc2-4.0.1 (C) 2000-2007 MPlayer Team
CPU: Intel(R) Core(TM)2 CPU         T5600  @ 1.83GHz (Family: 6,
Model: 15, Stepping: 2)
CPUflags: Type: 6 MMX: 1 MMX2: 1 3DNow: 0 3DNow2: 0 SSE: 1 SSE2: 1
Compiled for x86 CPU with extensions: MMX MMX2 SSE SSE2

There are 6 titles on this DVD.
There are 4 chapters in this DVD title.
There are 1 angles in this DVD title.
audio stream: 0 format: ac3 (stereo) language: fr aid: 128.
audio stream: 1 format: ac3 (stereo) language: en aid: 129.
number of audio channels on disk: 2.
subtitle ( sid ): 0 language: fr
subtitle ( sid ): 1 language: en
number of subtitles on disk: 2
success: format: 2  data: 0x0 - 0x718f3800
MPEG-PS file format detected.
VIDEO:  MPEG2  720x576  (aspect 2)  25.000 fps  9200.0 kbps (1150.0 kbyte/s)
[V] filefmt:2  fourcc:0x10000002  size:720x576  fps:25.00  ftime:=0.0400
xvid: using library version 1.1.3 (build xvid-1.1.3)
Opening video filter: [expand osd=1]
Expand: -1 x -1, -1 ; -1, osd: 1, aspect: 0.000000, round: 1
==========================================================================
Opening video decoder: [mpegpes] MPEG 1/2 Video passthrough
VDec: vo config request - 720 x 576 (preferred colorspace: Mpeg PES)
Could not find matching colorspace - retrying with -vf scale...
Opening video filter: [scale]
The selected video_out device is incompatible with this codec.
Try appending the scale filter to your filter list,
e.g. -vf spp,scale instead of -vf spp.
VDecoder init failed :(
Opening video decoder: [libmpeg2] MPEG 1/2 Video decoder libmpeg2-v0.4.0b
Selected video codec: [mpeg12] vfm: libmpeg2 (MPEG-1 or 2 (libmpeg2))
==========================================================================
VDec: vo config request - 720 x 576 (preferred colorspace: Planar YV12)
VDec: using Planar YV12 as output csp (no 0)
Movie-Aspect is 1.33:1 - prescaling to correct movie aspect.
videocodec: XviD (720x576 fourcc=44495658 [XVID])
xvid: par=0/0 (vga11), displayed=768x576, sampled=720x576
xvid: 2Pass Rate Control -- 1st pass
Pos:   0.0s      1f ( 0%)  0.00fps Trem:   0min   0mb  A-V:0.000 [0:0]
1 duplicate frame(s)!
Writing header...
ODML: vprp aspect is 4:3.
Writing header...
ODML: vprp aspect is 4:3.
Pos:   0.0s      2f ( 0%)  0.00fps Trem:   0min   0mb  A-V:0.000 [0:0]
Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: 13 at address: 0x00000000
0x005b8a09 in fdct_sse2_skal ()
(gdb) bt
#0  0x005b8a09 in fdct_sse2_skal ()
#1  0x005b5aaf in MBfDCT [inlined] () at ../../src/utils/mbtransquant.c:93
Previous frame inner to this frame (gdb could not unwind past this frame)
(gdb) x/i 0x005b8a09
0x5b8a09 <fdct_sse2_skal+185>:  movdqa 0x7320ea,%xmm4
(gdb) disassemble 0x005b89f0 0x005b8a39
Dump of assembler code from 0x5b89f0 to 0x5b8a39:
0x005b89f0 <fdct_sse2_skal+160>:        paddsw %xmm5,%xmm5
0x005b89f4 <fdct_sse2_skal+164>:        paddsw %xmm2,%xmm2
0x005b89f8 <fdct_sse2_skal+168>:        paddsw %xmm4,%xmm5
0x005b89fc <fdct_sse2_skal+172>:        movdqa %xmm4,0x40(%ecx)
0x005b8a01 <fdct_sse2_skal+177>:        paddsw %xmm1,%xmm2
0x005b8a05 <fdct_sse2_skal+181>:        movdqa %xmm5,(%ecx)
0x005b8a09 <fdct_sse2_skal+185>:        movdqa 0x7320ea,%xmm4
0x005b8a11 <fdct_sse2_skal+193>:        pmulhw %xmm7,%xmm4
0x005b8a15 <fdct_sse2_skal+197>:        movdqa 0x7320ea,%xmm5
0x005b8a1d <fdct_sse2_skal+205>:        psubsw %xmm6,%xmm4
0x005b8a21 <fdct_sse2_skal+209>:        pmulhw %xmm6,%xmm5
0x005b8a25 <fdct_sse2_skal+213>:        paddsw %xmm7,%xmm5
0x005b8a29 <fdct_sse2_skal+217>:        movdqa 0x73210a,%xmm6
0x005b8a31 <fdct_sse2_skal+225>:        movdqa 0x7323fa,%xmm7
End of assembler dump.
(gdb) p $xmm4
$3 = {
 v4_float = {0, 0, 0, 0},
 v2_double = {0, 0},
 v16_int8 = '\0' <repeats 15 times>,
 v8_int16 = {0, 0, 0, 0, 0, 0, 0, 0},
 v4_int32 = {0, 0, 0, 0},
 v2_int64 = {0, 0},
 uint128 = 0
}


I'm unfortunately absolutely not fluent in x86 asm so I can't be of
much help there as to why it doesn't work. I'd be willing to help
figuring out what's wrong if given proper indications tho.

If I read this http://www.ews.uiuc.edu/~cjiang/reference/vc183.htm
right tho, the left operand should be 16-byte aligned and unless I'm
mistaken, 0x7320ea is not. nasm goofed?

HTH

T-Bone

PS: please CC me on replies, I'm not subscribed.

--
Thibaut VARENE
http://www.parisc-linux.org/~varenet/