[Xvid-devel] 1.3.5 segfault in check_cpu_features (xvidcore/src/utils/x86_asm/cpuid.asm:97)

[Dominik 'Rathann' Mierzejewski]

Dear developers,
on Fedora 29, we started getting segmentation faults in
check_cpu_features. Here's a downstream bug report:
https://bugzilla.rpmfusion.org/show_bug.cgi?id=5141

Same nasm version was used to build both on Fedora 28 and 29.
There are differences in gcc version (8.0.1 vs. 8.2.1), binutils
(2.29.1 vs. 2.31.1). Links to each build:
http://koji.rpmfusion.org/koji/buildinfo?buildID=6763 (F28)
http://koji.rpmfusion.org/koji/buildinfo?buildID=8508 (F29)

The issue can be reproduced with ffmpeg. Here's the backtrace:
$ coredumpctl gdb ffmpeg
...
Core was generated by `ffmpeg -i arrival-tlr2_h1080p.mov -c:a copy -c:v libxvid out.mp4'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f5e62980920 in check_cpu_features () from /lib64/libxvidcore.so.4
[Current thread is 1 (Thread 0x7f5e61523e80 (LWP 23309))]
(gdb) where
#0  0x00007f5e62980920 in check_cpu_features () from /lib64/libxvidcore.so.4
#1  0x00007f5e628fa12b in detect_cpu_flags () at ../../src/xvid.c:156
#2  0x00007f5e628fb235 in xvid_gbl_init (init=0x7fff3250c044, init=0x7fff3250c044) at ../../src/xvid.c:792
#3  xvid_global (handle=handle at entry=0x0, opt=opt at entry=0, param1=param1 at entry=0x7fff3250c044, param2=param2 at entry=0x0)
    at ../../src/xvid.c:816
#4  0x00007f5e653e9157 in xvid_encode_init (avctx=0x5566b6afd3c0) at libavcodec/libxvid.c:458
#5  0x00007f5e6588414e in avcodec_open2 (avctx=0x5566b6afd3c0, codec=0x7f5e65e82660 <ff_libxvid_encoder>,
    options=0x5566b6acc2d8) at libavcodec/utils.c:923
#6  0x00005566b607cfaa in init_output_stream (ost=<optimized out>, error=0x7fff3250c790 "", error_len=1024)
    at fftools/ffmpeg.c:3473
#7  0x00005566b607ec31 in reap_filters (flush=0) at fftools/ffmpeg.c:1432
#8  0x00005566b6061eb0 in transcode_step () at fftools/ffmpeg.c:4597
#9  transcode () at fftools/ffmpeg.c:4641
#10 main (argc=<optimized out>, argv=<optimized out>) at fftools/ffmpeg.c:4841
(gdb)

My CPU is:
$ lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  2
Core(s) per socket:  2
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               69
Model name:          Intel(R) Core(TM) i7-4500U CPU @ 1.80GHz
Stepping:            1
CPU MHz:             838.433
CPU max MHz:         3000.0000
CPU min MHz:         800.0000
BogoMIPS:            4788.77
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            4096K
NUMA node0 CPU(s):   0-3
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts flush_l1d

Any ideas how to debug this?

Regards,
Dominik
-- 
Fedora   https://getfedora.org  |  RPM Fusion  http://rpmfusion.org
There should be a science of discontent. People need hard times and
oppression to develop psychic muscles.
        -- from "Collected Sayings of Muad'Dib" by the Princess Irulan