Description

The tjexample.c example program uses tjAlloc() function to allocate the output buffer of the JPEG buffer.
tjAlloc() uses malloc() which takes a size_t number of bytes (an unsigned integer).
However, tjAlloc() itself takes the number of bytes as a signed integer: https://github.com/libjpeg-turbo/libjpeg-turbo/blob/2.1.5.1/turbojpeg.c#L487

In tjexample.c the calculation of this parameter is width * height * tjPixelSize[pixelFormat]: https://github.com/libjpeg-turbo/libjpeg-turbo/blob/2.1.5.1/tjexample.c#L334
All the three factors are defined as a signed integer.

When a very big number below the maximum limit (JPEG_MAX_DIMENSION=0xffdc) of a JPEG picture is used for the width or height, the resulting calculation exceeds the capacity of the data type, causing it to overflow and become a negative integer. Subsequently, when this negative integer is converted to a size_t number, it becomes an exceedingly large value.
Thus, when using the upper limit for a width/height of a JPEG picture (0xffdc), the calculation's product will overflow and become a negative integer. Then, when converted to a size_t number it becomes a very big number.

Proof of Concept

poc.jpg

UBSAN

> ./tjexample poc.jpg out.jpg
Input Image:  65500 x 65500 pixels, 4:4:4 subsampling, YCbCr colorspace
/home/user/libjpeg-turbo-2.1.5.1/tjexample.c:334:50: runtime error: signed integer overflow: 65500 * 65500 cannot be represented in type 'int'
ERROR in line 336 while allocating uncompressed image buffer:
Cannot allocate memory

TRAPV

> ./tjexample poc.jpg out.jpg
Input Image:  65500 x 65500 pixels, 4:4:4 subsampling, YCbCr colorspace
[1]    2906 IOT instruction  ./tjexample poc.jpg out.jpg


[305561.738682] potentially unexpected fatal signal 6.
[305561.745706] CPU: 31 PID: 4386 Comm: tjexample Not tainted 5.15.90.1-microsoft-standard-WSL2 #1
[305561.746141] RIP: 0033:0x7fc1bd389a7c
[305561.746958] Code: 41 89 c5 41 f7 dd eb 80 66 0f 1f 44 00 00 b8 ba 00 00 00 0f 05 89 c5 e8 42 56 05 00 44 89 e2 89 ee 89 c7 b8 ea 00 00 00 0f 05 <41> 89 c5 41 f7 dd 3d 00 f0 ff ff b8 00 00 00 00 44 0f 46 e8 e9 6d
[305561.747398] RSP: 002b:00007ffc7f4f5860 EFLAGS: 00000246 ORIG_RAX: 00000000000000ea
[305561.747772] RAX: 0000000000000000 RBX: 00007fc1bd2f0800 RCX: 00007fc1bd389a7c
[305561.747945] RDX: 0000000000000006 RSI: 0000000000000b5a RDI: 0000000000000b5a
[305561.748120] RBP: 0000000000000b5a R08: 00007ffc7f4f5930 R09: 000000007fffffff
[305561.748290] R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000006
[305561.748462] R13: 0000000000000016 R14: 0000560f59072c90 R15: 00007fc1bd625040
[305561.748633] FS:  00007fc1bd2f0800 GS:  0000000000000000

Impact

It may result in a denial of service for applications that utilize libjpeg-turbo and rely on integer overflow sanitization mechanisms like trapv