the website of dither8

The past, present and future of image formats

Published on 2021-08-18

Anyone who has dabbled in website creation, has no doubt had to wrangle with an image codec. You have to do this because of bandwidth, since images too large will waste time for those on slow connections, and cost you dearly if you're popular.

Since the late-1990s, two dominate image formats came to be used all over the web (and off the web too). JPEG (est. 1992) and PNG (est. 1996). Both these image formats had different use cases. JPEG became infamous for it's lossy compression, widely used with photographs where the loss was less visible. While PNG was better suited to computer graphics, since it supported transparency, and it's lossless compression performs well on uncomplex images.

Before moving on I should also mention that GIF was big at this time. It can be considered a proto-PNG format, because of it's historical use in the same domain. GIF is from the late-80's, so it is limited to 256-colours. However GIF animations are widely supported, so this format lived long past it's expiry date.

JPEG (named after the committee who created it), tried as early as the year 2000 to replace the original standard, with "JPEG 2000". They were ultimately unsuccessful dethroning JPEG 1, despite large improvements with it's compression. It caused less information loss, and looked better then a JPEG image at equivalent file sizes. There was also a large range of expansions, such as the support for motion and lossless compression. Ultimately is was probably patents and the associated license costs that prevented JP2 from going mainstream, a lesson which would later be learnt.

As I'm sure you understand, getting a better picture at lower file sizes is big business for the web. So for the better part of the 2010's we had three formats thrown at the wall, with various degrees of adoption. Google brought us WebP in 2010, MPEG released HEIC in 2015, and AOM with AVIF in 2019. One thing all these standards have in common, is that they are actually based of video codecs. This is due to the innovations made in that domain. Needless to say, all three support motion, but none of them support progressive decoding due to the fact they are video codecs.

WebP puts out marginally better compression in some cases. I don't think this format is worth the hassle. As Johannes Siipola pointed out in his blog post last year, a finely tuned JPEG encoder will do just as well. WebP has become widely supported, though far too late. WebP has better lossless compression than PNG, but does worse than JPEG when it comes to hi-fi images (such as those from cameras). I don't see WebP used long-term since it's only purpose, the web, will simply move on to better formats once ready.

HEIC and AVIF are far more interesting. They're based on far more complex codecs, so they both leave everything else behind when it comes to lo-fi images. AVIF shows improvement with hi-fi images, but HEIC is no better than JPEG. HEIC, based on H.265 video, means it's encumbered with patents (moreso then anything before it). So it has no support outside of Apple. AVIF coming from AV1, is patent-free. So it'll likely gain traction similar to WebP. Using video codecs for images has the advantage of being able to leverage, hardware accelerated decoding for better performance and battery life. But as we're about to find out, this might be all for nothing.

Since 2018, the JPEG committee began working on JPEG XL, a royalty-free image format that would take into account innovations made in the last 30 years. When it comes to compression, JXL does a little worse than AVIF at lo-fi images, otherwise JXL is better than everything else.

Some features added to JPEG XL that were missing from the original JPEG include transparency, depth mapping, and HDR support (WebP misses the last two). Due to the fact that JPEG XL is designed for images, it doesn't have as strict limitations compared to the video-based counterparts, with resolution being a notable example.

Progressive decoding is a useful feature for image loading perception. What is does is structure the file in a way so that first couple of KB's contains a full (very un-detailed) image, it will then load more detail onto of the base, rather than loading a full-quality image top-to-bottom. This feature is only useful to images codecs, so it's no surprise that both JPEG and JXL have this feature. Again this is only perception, it doesn't actually make it load faster. Still I believe this will play a role in JPEG XL becoming preferred on the web.

So then, what can be done with all the existing images encoded in JPEG? Well, JPEG XL has a feature that lets you losslessly convert JPEG's into JXL's. Which gives you ~20% file size savings per .jpg, without adding another layer of compression on top of what JPEG did. This is not a common feature with lossy codecs. But since most cameras don't do RAW, it was needed.

To do these tests I took this photo taken by Alessio Soggetti. I then down-scaled and cropped it. I started at 0 quality for all encoders and worked my way up in 10 step increments. I tried to aim for a little over 100KB. I did bump JPEG XL a few steps (50 to 53) to keep it competitive with AVIF (JXL came in at 100KB at q50 while AVIF came in at 110KB at q40).

AVIF and JPEG XL are visually indistinguishable. But here we get to see the visual signatures that each leave:

Loss of quality really doesn't matter at these microscopic image sizes. Obviously you'd use a higher quality for photographs, and websites will try and downsize the image further. JPEG XL beats AVIF in efficiency when creating hi-fi images.

But honestly who knows when, or if any of this happens? It took WebP nearly 10 years to get adopted. But, the cat is out of the bag, both JPEG and WebP are seriously lacking as formats for photos online. so it's only inevitable that JPEG XL or AVIF takes over.

In my personal opinion JPEG XL is the most likely to become unanimous, as it is far more versatile as an image format.

Updated 2022-06-07: A little bit more terse