Why Isn't JPEG Dead and Is HEIF the Solution?

It feels like a bad dream, that just got worse. Just when you thought the nightmares were over, you awake in a cold sweat plagued by the same recurring terrors. Yes, the JPEG really is that bad so why do we keep clinging to it? Isn't it time we killed it off and moved on to something better?

The need for JPEG format was a very real one — transfer images across extremely low bandwidth networks for display. Low bandwidth applied to both the storage medium, as well as the transfer channel. That could be a hard disk drive or floppy disk, over dial-up networks, or perhaps the more favored sneaker-net! Up until that point, images had been stored as raw bitmaps (BMP) which contained each individual RGB or grayscale pixel values. This was vastly inefficient for data storage and transfer but was the cornerstone for simple algorithms. Methods to compress images were rapidly developed, such as Run Length Encoding (RLE), which identified contiguous pixels that were the same value and just stored this information. Even within this context, there was a competing trade-off between resolution, file size, and loading speed. I distinctly remember waiting tens of seconds for individual grayscale photos to load on my PC running the MS-DOS operating system. Of course, these compression systems didn't work particularly well for continuous-tone images, such as photos. An alternative was needed that worked with photos over low bandwidths. Enter the JPEG.

JPEG is an acronym for the Joint Photographic Experts Group, which was formed as a sub-group of the ISO/IEC Joint Technical Committee 1, Subcommittee 29, Working Group 1. For simplicity, it was known as ISO/IEC JTC 1/SC 29/WG 1! Given the importance of the problem the two main standards organizations (ISO and IEC) came together and released the JPEG format for digital still images in 1992. Importantly, it was royalty free.

Philosophically, the JPEG committee didn't mind if they butchered the original image, as long as it looked good. That was the key to unlocking the easy storage and transfer of digital images. More specifically, it had to look appealing to the human eye. Regardless of the input data, the RGB layers (or the single grayscale) are reduced to 8-bit data before smoothing to the point that it looks the same — or almost the same — as the original image. The primary setting is "quality" which aggressively increases the amount of compression applied at the expense of efficacy. However, the benefit is smaller files, with compression ratios of 10:1 common, along with faster write speeds.

The algorithm underpinning the file format uses a discrete cosine transformation (DCT) over 8x8 windows of pixels (or kernels). The DCT is "fitted" to the underlying image data — the accuracy of the fit determines the degree of compression, with a closer fit requiring more storage. Whilst effective, the use of a DCT has led to some significant problems in the quality of the final images. Firstly, it is fairly common to see "blocky" artifacts, particularly in heavily compressed images, which result from the underlying 8x8 kernels. Secondly, the DCT kernels are also the reason for posterization which results in pixillated transitions across smooth tonal changes (as the image below shows). Thirdly, every time you re-save a JPEG the DCT is reapplied which causes progressive degradation. One of the most common causes of this is image rotation followed by further edited. As rotation simply changes the pixel identified as the origin of the image, there is no need for re-saving and so this was the reason for the introduction of lossless rotation.

The Future?

The JPEG in its original incarnation was a 1980s solution to a 1970s problem. It met the need head-on and was a perfect fit for the subsequent explosion in digital cameras. However, it rapidly pushed up against the inherent problems noted above and so an alternative was required. The JPEG2000 standard entered service in 2000 and used the discrete wavelet transformation (DWT) to achieve much greater flexibility, but only with mild increases in compression. Ultimately it was too complex for its time, was not backward compatible, and had significantly greater processing requirements. It's a salutary lesson on how standards can shape technological development — the JPEG was never envisaged to be the de facto image format for 28 years and counting, however, that is how it has ended up.

We desperately need a new format for one reason: JPEG is appallingly bad for any kind of long term archival.

If you want a butchered image with no latitude whatsoever to be the lasting record for future generations then go ahead and continue using JPEG, however it really has no place in the photographic arsenal. Of the three primary benefits, arguably none of them exist any longer. Backward compatibility has been important for wide-ranging support in capture and display systems, however, there are good alternatives that need wide adoption. The royalty free status is important which is where we need standards organizations to again step in. The file size is far less important than it ever was; ~24 megapixel sensors strike the sweet spot for resolution and you can happily shoot all day on current memory cards. Higher resolution sensors — such as found in the Fuji GFX100 — still cause a data headache but you find these in the hands of those familiar with the data processing issues. There is perhaps some rationale in JPEG use for photographers that require fast shooting speeds, however with Canon's 1DX Mk III able to shoot at 20 fps for up to 1,000 raw files, this difference is rapidly diminishing as the technology trickles down.

Camera manufacturers are partly culpable for the situation we now find ourselves in and certainly have a part to play. It's notable that even with Adobe's push to popularize the DNF raw format, most still stick to their proprietary types although at least these are widely supported. However, JPEG remains a stalwart when there should be a far wider take-up of other formats. Part of the culprit here — besides a lack of industry focus — has been the failure of manufacturers to continue updating firmware. For example, my aged Synology DS112 NAS runs the same version of the firmware that sits on the latest Diskstation 220j: why can't camera manufacturers be as proactive? The longevity of the JPEG is such a critical issue that manufacturers should feel an implicit requirement to update older cameras.

Of course, there is an heir apparently waiting in the wings in the form of the HEIF format which was developed by the Moving Picture Experts Group (MPEG) and is also royalty free, sports better compression, and has a greater bit depth, all of which are important for mobile shooting. It's the reason Apple incorporated it into iOS in 2017 and, because of this, HEIF is now widely implemented across other operating systems and applications. Canon is the first camera manufacturer to incorporate support (into the 1DX Mk III), however, it is disappointing that other manufacturers haven't yet followed suit.

That said, it remains to be seen if HEIF is the format that stills photographers have been waiting for or if it is being exploited to the best of its abilities. Apple and Canon both appear to be focusing on file size, whilst maintaining the same or better quality. Given the use cases involved, this is in order to minimize upload times. However, what we need is far wider adoption for straight-out-of-camera (SOOC) shooters that implement lower levels of compression. The other format on the horizon, and approaching ratification, is JPEG XL which should provide this functionality, better balancing compression, quality, and encoding speed. What this highlights is how poor camera manufacturers are in pushing their firmware platforms forward: there needs to be a sea-change in how they achieve this and the current paradigm is not working for photographers. Perhaps the Zeiss ZX1 is the way forward that will allow a plug and play architecture for the deployment of a range of new functionalities. Whatever the future holds, current manufacturers are letting their users down.

Apple has never been shy to adopt new standards in the pursuit of performance gains and we desperately need camera manufacturers to likewise follow suit. I for one don't want to see an option for JPEG shooting in any future cameras.

Lead image courtesy Michael Gäbler vis Wikipedia.