[XviD-devel] [Fwd: quant4_inter_xmm large quant bug - hopefully fixed]

Dirk Knop xvid-devel@xvid.org
Sat, 28 Dec 2002 18:39:07 +0100


This is a multi-part message in MIME format.
--------------080503010809000104060206
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit



-------- Original Message --------
Subject: quant4_inter_xmm large quant bug - hopefully fixed
Date: Sat, 28 Dec 2002 18:44:54 +0200
From: Jaan Kalda <kalda@ioc.ee>
To: <koepi@roeder.goe.net>



Hi!

Unfortunately there was a bug (as mentioned in the mailing list).
The sum (return value) was not calculated properly for q>19.

The fix is as follows: in file 
quantize4_xmm.asm, starting from line 579 should be

  paddw mm2,[ebx+8]   ;sum  
  paddw mm5,mm0 ;final result
  paddw mm7,mm3
  movq mm0,[edi]
  movq mm3,[edi]
  pmulhuw mm5, mm6  ; mm0 = (mm0 / 2Q) >> 16
  pmulhuw mm7, mm6  ;  (level ) / quant (0<quant<32)
  add esi,byte 2
  psrlw   mm5, 1   ; (level ) / (2*quant)
  paddw  mm2,mm5 ;sum += x1
  psrlw   mm7, 1  
  movq  [ebx],mm7 ;store x2  

(was: 
paddw mm5,mm0
.........
  psrlw   mm7, 1 )

Best wishes,
Jaan




--------------080503010809000104060206
Content-Type: application/x-zip-compressed;
 name="quantize4_xmm.zip"
Content-Transfer-Encoding: base64
Content-Disposition: inline;
 filename="quantize4_xmm.zip"

UEsDBBQAAAAIAKCDnC045lXqthMAADxcAAARAAAAcXVhbnRpemU0X3htbS5hc23tHGtT20jy
s1OV/9BfUmBjG0s2mI1Dqtgs2eNu81gg+yiKomRpjOdWlpyRDObqfvx192g0ErITc7FJ9m61
GyNNP6dnprun9RjsNtZ6PH0ygAas8TAcz8cygZEMBeDfqadSiEfw2438oQkejJQQ8Ob98Y+t
HtzIQMQgIj8OhNoNBP8tctycjqQOqedFICfTUExElHqpjCPS1cu1jiMBsYJJrHKlf2Gl0zgO
kxJHL4FkKnw5kiIAGcHJ2bvdk+NX4PR63+23XEhSLwo8FbTJQHEiECcVUSCja2QGM2xI0W5F
jkk8Sm89JVB8MAuJAMbIgJtiZcFThXA/TYAuvOBGJiJAZl4KMi3pyFIm3h1yGikULEDMZZLi
GXYYlUkT4uvH0zslr8dp0gRUGf/dQTLzx0RdseNtPAsDGKLclJHYcndbyOg2AiWTP7i7qC9y
lJEXQiBuRBhPhSLzmh5/otesAqLtjgWpNpmiOqwXihsm4uMM1QYRyDRWNJrUcSFViaOmkgL7
cyvDEI14IyCKIZTeUIYyvYNRrKh3RiWrQ6wWzR5SDcfZ5wnDNguEkjd4eSMSUkCJeNSGZcfj
znBecKZDA7iLZ+B7ESgR4NArOZylAmRKptuNle7ZHTVUOc4i7CZ1D1KhJom2loAf336AH0Uk
FI7u+9kwlD78JH0RJYLWxJRakrHASXJX4Ujkr0m/M2Pw1zFKYbsOQEgyJdwIleA1uJlAw72J
hq9w3PZS6qKCeEpM6jx7Qy+1fNoA38jI2AEgh8FdG+PKMEtXT9ahoJk5moVNQMwlHH89Of/b
uw/ncPT2d/j16PT06O357wOkT8cx0uCKY+7s6cg7oaWVF6V3ZNAlHN8cn776G7I5+v7kp5Pz
38nUr0/O3x6fncHrd6dwBO+PTs9PXn346egU3n84ff/u7BgNeyYECVrE8VPTZGRcbCBST4ZJ
+2uNzO+4OJIxuzR2Ekr4Atd0AB57xc/P+ApHL4yjax4J9izkq6+VNxmAHEEUp+iRlEzRaHF1
LSzR0a6QJpxEfrsJe9/BuaAgBu9DzxdNOJsRz26304Tv4yQlzDdHAB3XcZyW0+304cPZ0Ubs
CMxyrccusnz6ZFPJz8cZLoQeLq3rRA/5UIgIRnIugufoTOoQo9sYhfEtoeiJ6qVKzkHonCGB
WnFGCuQX0mg6gKguyIR4sfPjeJXKdKbj7WQWjm95Yujz2S1sy7Oz43qJIUe0WUIUiJRKWr8K
OiM8xiCjJBVeQLPS6XQ6zhi2qwrCITiDIkcZaQfje+RCHU6CgoBUFEgtjJw7lNlkTAJLmm/Y
M4eCqbjX6ZGKJ5TNKO9Ky0aB28M6KJqppLpQKlZsvNBT1wJ5iBGGT0n24x5qMsvRGLdkv/Q2
bmGHp+BNcRV5/vg5wlSS6uu5nOj8bZvFiqCsY4D5SJ3ZEZQC0yzBoRrwslMiwU4T1BiZJTKs
bM17IzORCcL9MZHOOOuK0a6hP6OIw+R+rJTwSbF2jQ5WamOLBK66QSRgNItYJGeEOv1J5TAU
errRLONgQlCOkxP5LxGY7BX1D3VGRAz/0YepnIpQRiLRMlprPUhG57u247bdTscFOErH6DGN
VlmC5ceRCZQ0LH/3vAj+4YWBBxvQh/4bAHpYbzcVczRSKK8jGvta6QBA4c8CMULLwNFPJz++
hQPbcHZ0/uH06PyYmA0pC++6dPpMjhCBIuibo/OrV+9ev3765Oz41fnJu7fQJpEsF/FEmIhF
IK3NIUmivcOImU48X8XgX4fxkJYo6lXLBL0/PX598hte1zLg1TMG14yeeIlNRMASLSI3Ghn0
l4UUxaFthIqWiNPAz4iziEvFcXfJsJPJvwRKDQLoNDuoBjbMr2h3lkryR71acFsDh8fucH1H
tqoCSTtVmnruz5B6tJKwed2iTG95MukOouAECsdAvnTRPl6SEKIEBy+UmEKXzJdZAoJbgO1t
B168AGevDruItwNOnQbCUsodRxsayZeJ/qRk+Jzo/dVEb2jIOBxlznszg5WtA5zwv9E4YNeL
Hd9+5tSp74XZDERllk0xXF5hiAuXgxZDnMXNmzOnMJnFRqxZ6ItQpouLm6utxlLLISHJsO75
lzfOwXvolht+hp5tiOJpD374HjoHP4ybnX4Pf9x9/ClhdCk1ApFMm8O7VEAZ6MIkvmEg/itB
+hAMkXFAPH36oTP+r8xhX7MfTpvBbazQ863b7Dy0nAEHm3RrT59oGSt6rm0enYak9cPnP++6
dVpTfF5f6ks2NPWzlLDk/jdkJvK7KO6KzbWSteRjWyPx0pnSiXYoJzJNNjNldDhy9o1Vum5/
v381kdEsuXI7vX4Nj4IdthneIkjdEhxYgoNFBActgmgC5goFDLrWEMsEihwMMeUlRQBnJ+h/
KIJiWOBlWx1Jd2+PoiKCTdjoc9gAyWHjk8Oa6A0FtCk/3dA438Qy0K7hisNL72o+mWzjqbN/
lUJDb+GaNPg1OvwY96JgwPqqwRlrs4wyQ5yuizjMexkw8BPchniqPtjg7MqSHZM360qA7i11
Fr1WpYUyVvTqNeHNm3CBfh1wsA4uazW9YcjBfgHsuJcIZl45PCjAe0zO9iT4dJaMayKR9jwo
nA/ndI6xIWMFCG3q3Ni0IG2z5fRqNaaaU0Fg0kGcTuG6i9dduvYnU9aW45dDLf8UtUh4Ctof
nTCOp1Wk7xjrGjRWaJAoXmkDwT0DtxHF6PuxNpk41HdvTpZriAQnt+OSCUhNOIQLJ0kvWVWs
l9xCTSvvwKCFvwUuvTIXNnSHLJ3T1nQ/e0za4+aJN09uAXKe/06U/+8yhEmyDrmal/IQAqS4
swcDXpRIV4b1CJbZPAlDbGQpuDGhroV004NWea+M0M0QMkIM9znhRSnZKxkLysjdpchkEWMy
YNy95kUl86zytvj95fiWvalfEXvUvdzUp6kGmq3Nh6wURHHLQkr6lBD3K4hVRcJb5oi6gG1i
WtTFTo5sDri2JRv7/Wzse0ayVlBnMCgRFwI04ABacGCVY9XMhq2MZEfLGMgtNvS5oWoca8BO
dQiqI2bRu0vQrZ3K2nTua9O1vSLZIpDFfnZ1S61WkFojTmQDu4q36c8u5kt1ePkS43gZu59j
w7ZeGjtZGkoxkM9gu/OCT1503brxXEaQYwQ1DoGWI0nLkQz/HiLlg1uipKpkMqVSuYUXiYyT
wZ6WTU0YewvBQLZleJ/gelsgtcd02WFG16AdK7tDQLcYxJFAM7L39/xQ1ww5jlx0LmEXk4Es
HNbzcKjTRJ4rFCN0EGHXRz/75P4M7tMnAISYzEHHI9pCsMvE4WRIrUbBA2HZNXBcKnNk3GSs
NAwc9mYYheAwFwS7LiN5SjPsol/1g49UEr0RfhorLD6OCYPGhtkgFhgu5MJ/gRb9aTnMZzZE
LJ+x8JpMaa5J0aKmeldk9OVQqo3DtrnIcrCdHg0QQSROQGbF4ZU05iunz1mIRNyaP9c99Fm3
QD6ns11rVBytfLEOtbEuC6GYG3Z6tinJmrSvKO0WHbPseSLh3EHZJhGgCXAI3lwHVCW49ljJ
XDBEP0Zo3flmAysen42uDwyttQHZSiZUyR+KrPIXPCDgEgtgqMN1a09GfA+DKurMju5B017S
eWhkHnD13txU4TsaGwrWxrLYUL7HAvEo78PGIzoNLcVTttzHF4fO/oIYn8+CVKCWN144E/Di
MH/cZJUUwPIwt3O+MCew8bXMvmRJHkzNYc35AAzsTacvzA0KupM1zY2ysu4Pzx4W9S9R2ard
ozCzJDdwsZEu+K5mvUzXZ7r/jVShsgX752Sq2zhrsJHggLZX/99BYO27q4URYDWX/ifcmH1l
J7/ApX+JK1+bC69s6+7hQCU2Vfz+Y/n7Nbr7zbv5x9wxVkLKItptt8EX/8PRBGqFWh1Uw8mG
Ssh5NdfWUYVaWEb+wiry16kS676QNYttf9I6sdnQ6mVKm3C7UXXZZ4bCI4zSdjYKuImxWgcw
MDnJCoVn9hQXREksLg0a3Rh9PW5Cn356Pfxxu/iDvDUB4qJwwl579fqv4vV60yuhHlC8Fmrl
HEmoh+VIQn1xjvT5LEioFbMgoTZevLbZjVleKxeqq3mHUA+qOwu1SrLhNrOFDIBTcjb5NmvS
izOLapS1ytbM4FGnYOcQ5rbIU7vI/FwfoWmsBMzdx08vSv51L/evHeNga9bDukZP7MvSzKTl
7H8uOTEo/fWWwcuzyc5xG6Tc3oLSLLVWq7PcWi3QYrOeJCQpYFHAqwdV1hN6Fk39P8Lgo92A
XIsU6HFxVC2Nqfn5WF6Pjb6BQbTU45y6w9Qm+05j4pPpGnAqQXi12l8l4c2UhIX64pKwUEtL
wgx9cEmYqb64JLyGELimkvAD4uSfoCS8JNJWY+hKtdr1xdNabcVK7n8VZz9Trl2pTLvOKPq1
tunLgmWtts5w+Vcp+CvuVRZ6/tVc+Z9wm/OVnfujloKrrvsLyr4r+/vHc/drcvNfZzO1tHBb
LdgujxjVoPPNRZFvaTe2pDa8sRd0+AHjQJSescU1Z2vDpuwLC46FhWIujS6nqNaNl+E+7sPI
AKTE2zgVz7FHECv+yEJsnrQXsCW8RIZ3W024FTBVopWM5Sjlt0u5G3IkhWI+FHh6bfreRtRk
OG09W7fIkhz5tn606LKhXcyFvMwWEnhKmNdXMz4zGQbgGR349WOPvuVxwm/JRnHU4sfe4yi8
g1sUB15k35SOfX+mRNDWvH4VqMosAY9HfOr5fyC73VmUnaHMWw8VDOLZMBSkrCF8EytBPHW/
kXesJtyrCW2Ptyu94MDVLLwV3QTRvm43mZvpvSx0v06mwu7g/6FI+N38SKA+tidyBJnR2EZR
nGpm/lj4f4gA7RFNZykoL7oWCaE81/g0kBctx+038R+ut6JrZ5jTbuMbAJdNT/OjSjr3wgC7
zmUbGMY/jTUc+RtrPxz//AG/7NA7eXt+evTmDb+9BpA5f6dpqvL+nB4AhAY8cyjQGMfvA/4Y
W1pCThKLNIB+i2i6hL5laXYcpirGaidvsGkhgM4ULYptsTicFmYJo73WSSK/bUuKgNWSQ6dn
QieuJV5KIjN+Aa9v8eyrtgCF9ASgfFPbdvySjVWenSXi/lJitpolZ2s1yrqZAG8bOAPqGglj
bTLSkfrILgDibB43slVjImiRpsthYTUaawbSh2louaxAY3KD1Wj8yfQ61R3Sj66WW7uFVsoE
ZkmuVMe25UK72RQxqpspY9TaY6NSdDdcHNIyST2VQiSutSMcot/K8UxAX47HE1sr4bBi5bZe
rpjNx6hLMCAHW2jvltrNgrGKDgAzPpmMywoAWFSj61JUxi3emAsKM/PSTimuzcpLi11CBJrD
PJcsMq96+yZs6Q0vc2v1fjLQDSj7WND4nMhZNOT3V3c4uS4tY8jvruo1pWdZcY0jtPTGHWpO
HBs61deGy+csd512lm4bzgR/VYE+CtDOpQVGWg+lmfu8fDuWdhe1DC+Zc4Gawhz7SxaT3yyl
HwAwbWZVdk0bM0sknYfC4+6TyJar1S3dtu261MT3aL3CPdppjJbkHe1HVoLHgb/+4PkC4giS
lCauHVvP3J21LVlO2qc2ieazNXdnH4vpXRwLPOyz6GUXbCjtLKDNeiUqocS8Ss/v2uG1vqGN
658lImQRoUMoSBkWkHCmgz5DG9ntBi4b5NxfxMWtCCKe1/Y6uyM+zXeQ08tFfHhtA5TuN3R5
V8yNdqnhmhnOTTOJMbg7vUWMFxttb1Hj/qJG6jSU319GUQe8tpTY6GuFxTUtFK9pm/bD6nl/
I7v+ZOL/iA+M8KTPMnn4lVNq2q36sAMJbh/9OrT4pOXX6+AVPjajDG0SU647xkUbCk51u/yZ
ngTTyhfoHfzDQ/xFKPgvO7xZiAR9ryptGwbEfs7fubHftaEIu2WCZqc530LtkHoE85fEruXw
+YtOzuQk3ULl6LM75sNVAsQc3a9vPpBILd4wicNZKnR5pv15ny5U1adzY9mnAxR8KaAz5cO6
VIvl51gHBsu4eYtWjhEWjVV4UDSwEaflcPZpSyr0AyZAOAX3vc/JDrabYhTv38uu3PgG83yO
pXZM5lt+egaA/SD/6ZXtzndVS9mEzsoPGrqA2284lHrZhN7m8ja1srmKxnLg0ExgPLYpbRbq
RgT1oldfJCjLswlc3ABUsjhGYLyuEbVVr4hieyd5euvo0YadjKKM4pq4aVGIqR02G24Zh4o8
roHmDNwMuqWh97c7SWkHtJc32YyrV5DR8ouQPeZvIFuaud2/UDbFQLqCQ+M3ikh7JaQ9g4S9
NHKssXpZL6FlsCxSOZGnzVrLsrKzz1odbAbpj3HrSxmkflV9BJEQgQju59HdlaisD70VW4o9
m0opJVHiWiYpVQLGQok2vI+R9dSTCnm02+3ywJjFSHwqWwpeOXY3wuhQLUnbOVweYCKwmybT
1ClvmpZvfuxg2M1PFX/13qzeDbrkrniBzQf1lLZL0U5hu3nKlbabJd2yt2hm2G2QElSuNGNc
2QcxvxVM665uWncF09r1Z3W207Ks8+dtYxdOZWNZmf8rmqbL7Kyeln65mrZ2U/VJXVPO0e02
LuV+qRib+DtrcVjc2FUmUsttHFAEYQZ2h1fl7v633M3+0b2/fyRKKllDHuWyxzYXi9FWHM9G
erhJrY6j/xtCCegw0Mmg94CuBuoDKStWLAVsIriPYepreuVRa2VnxihIUmop7xrIwOV8vWeI
aCNnzOk29rtoPv3uKJhU/j9QSwECFAAUAAAACACgg5wtOOZV6rYTAAA8XAAAEQAAAAAAAAAA
ACAAAAAAAAAAcXVhbnRpemU0X3htbS5hc21QSwUGAAAAAAEAAQA/AAAA5RMAAAAA
--------------080503010809000104060206--