Integer Overflow in tjexample.c in libjpeg-turbo/libjpeg-turbo


Reported on

May 27th 2023


The tjexample.c example program uses tjAlloc() function to allocate the output buffer of the JPEG buffer.
tjAlloc() uses malloc() which takes a size_t number of bytes (an unsigned integer).
However, tjAlloc() itself takes the number of bytes as a signed integer:

In tjexample.c the calculation of this parameter is width * height * tjPixelSize[pixelFormat]:
All the three factors are defined as a signed integer.

When a very big number below the maximum limit (JPEG_MAX_DIMENSION=0xffdc) of a JPEG picture is used for the width or height, the resulting calculation exceeds the capacity of the data type, causing it to overflow and become a negative integer. Subsequently, when this negative integer is converted to a size_t number, it becomes an exceedingly large value.
Thus, when using the upper limit for a width/height of a JPEG picture (0xffdc), the calculation's product will overflow and become a negative integer. Then, when converted to a size_t number it becomes a very big number.

Proof of Concept



> ./tjexample poc.jpg out.jpg
Input Image:  65500 x 65500 pixels, 4:4:4 subsampling, YCbCr colorspace
/home/user/libjpeg-turbo- runtime error: signed integer overflow: 65500 * 65500 cannot be represented in type 'int'
ERROR in line 336 while allocating uncompressed image buffer:
Cannot allocate memory


> ./tjexample poc.jpg out.jpg
Input Image:  65500 x 65500 pixels, 4:4:4 subsampling, YCbCr colorspace
[1]    2906 IOT instruction  ./tjexample poc.jpg out.jpg

[305561.738682] potentially unexpected fatal signal 6.
[305561.745706] CPU: 31 PID: 4386 Comm: tjexample Not tainted #1
[305561.746141] RIP: 0033:0x7fc1bd389a7c
[305561.746958] Code: 41 89 c5 41 f7 dd eb 80 66 0f 1f 44 00 00 b8 ba 00 00 00 0f 05 89 c5 e8 42 56 05 00 44 89 e2 89 ee 89 c7 b8 ea 00 00 00 0f 05 <41> 89 c5 41 f7 dd 3d 00 f0 ff ff b8 00 00 00 00 44 0f 46 e8 e9 6d
[305561.747398] RSP: 002b:00007ffc7f4f5860 EFLAGS: 00000246 ORIG_RAX: 00000000000000ea
[305561.747772] RAX: 0000000000000000 RBX: 00007fc1bd2f0800 RCX: 00007fc1bd389a7c
[305561.747945] RDX: 0000000000000006 RSI: 0000000000000b5a RDI: 0000000000000b5a
[305561.748120] RBP: 0000000000000b5a R08: 00007ffc7f4f5930 R09: 000000007fffffff
[305561.748290] R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000006
[305561.748462] R13: 0000000000000016 R14: 0000560f59072c90 R15: 00007fc1bd625040
[305561.748633] FS:  00007fc1bd2f0800 GS:  0000000000000000


It may result in a denial of service for applications that utilize libjpeg-turbo and rely on integer overflow sanitization mechanisms like trapv

We are processing your report and will contact the libjpeg-turbo team within 24 hours. 4 months ago
We created a GitHub Issue asking the maintainers to create a 4 months ago
Yair Mizrahi
4 months ago


@admin Has the report been submitted to dcommander yet?

Pavlos modified the Severity from Medium (5.5) to None (0) 3 months ago
The researcher has received a minor penalty to their credibility for miscalculating the severity: -1
Pavlos validated this vulnerability 3 months ago
Yair Mizrahi has been awarded the disclosure bounty
The fix bounty is now up for grabs
The researcher's credibility has increased: +7
Pavlos marked this as fixed in 3.0beta with commit 6b506e 3 months ago
The fix bounty has been dropped
This vulnerability will not receive a CVE
Pavlos published this vulnerability 3 months ago
3 months ago


From the maintainer:

It just wasn't a security issue, since the application in question is not distributed. I fixed the issue in order to prevent downstream developers from interpreting tjexample.c literally and introducing similar issues into their own code.

to join this conversation