Integer Overflow in tjexample.c in libjpeg-turbo/libjpeg-turbo
May 27th 2023
tjexample.c example program uses
tjAlloc() function to allocate the output buffer of the JPEG buffer.
malloc() which takes a
size_t number of bytes (an unsigned integer).
tjAlloc() itself takes the number of bytes as a signed integer: https://github.com/libjpeg-turbo/libjpeg-turbo/blob/188.8.131.52/turbojpeg.c#L487
tjexample.c the calculation of this parameter is
width * height * tjPixelSize[pixelFormat]: https://github.com/libjpeg-turbo/libjpeg-turbo/blob/184.108.40.206/tjexample.c#L334
All the three factors are defined as a signed integer.
When a very big number below the maximum limit (
JPEG_MAX_DIMENSION=0xffdc) of a JPEG picture is used for the width or height, the resulting calculation exceeds the capacity of the data type, causing it to overflow and become a negative integer. Subsequently, when this negative integer is converted to a size_t number, it becomes an exceedingly large value.
Thus, when using the upper limit for a width/height of a JPEG picture (0xffdc), the calculation's product will overflow and become a negative integer. Then, when converted to a
size_t number it becomes a very big number.
Proof of Concept
> ./tjexample poc.jpg out.jpg Input Image: 65500 x 65500 pixels, 4:4:4 subsampling, YCbCr colorspace /home/user/libjpeg-turbo-220.127.116.11/tjexample.c:334:50: runtime error: signed integer overflow: 65500 * 65500 cannot be represented in type 'int' ERROR in line 336 while allocating uncompressed image buffer: Cannot allocate memory
> ./tjexample poc.jpg out.jpg Input Image: 65500 x 65500 pixels, 4:4:4 subsampling, YCbCr colorspace  2906 IOT instruction ./tjexample poc.jpg out.jpg [305561.738682] potentially unexpected fatal signal 6. [305561.745706] CPU: 31 PID: 4386 Comm: tjexample Not tainted 18.104.22.168-microsoft-standard-WSL2 #1 [305561.746141] RIP: 0033:0x7fc1bd389a7c [305561.746958] Code: 41 89 c5 41 f7 dd eb 80 66 0f 1f 44 00 00 b8 ba 00 00 00 0f 05 89 c5 e8 42 56 05 00 44 89 e2 89 ee 89 c7 b8 ea 00 00 00 0f 05 <41> 89 c5 41 f7 dd 3d 00 f0 ff ff b8 00 00 00 00 44 0f 46 e8 e9 6d [305561.747398] RSP: 002b:00007ffc7f4f5860 EFLAGS: 00000246 ORIG_RAX: 00000000000000ea [305561.747772] RAX: 0000000000000000 RBX: 00007fc1bd2f0800 RCX: 00007fc1bd389a7c [305561.747945] RDX: 0000000000000006 RSI: 0000000000000b5a RDI: 0000000000000b5a [305561.748120] RBP: 0000000000000b5a R08: 00007ffc7f4f5930 R09: 000000007fffffff [305561.748290] R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000006 [305561.748462] R13: 0000000000000016 R14: 0000560f59072c90 R15: 00007fc1bd625040 [305561.748633] FS: 00007fc1bd2f0800 GS: 0000000000000000
It may result in a denial of service for applications that utilize
libjpeg-turbo and rely on integer overflow sanitization mechanisms like trapv