AVIF is now being hosted here.

First there was AV1. Now there’s AVIF

A while back during my video encoding sessions, I toyed with the concept of serving AV1 videos on my site. The results were not great out of respect that we could not find a great way of detecting the client’s capabilities of video decoding without writing some shitty JavaScript system to snoop around and break privacy rights all the way. However, there’s a branch of the Av1 system known as “AVIF” which is designed to use for still images. It boasts about getting better compression ratio’s to that of even WebP images that we discussed a while back.

Why did we implement AVIF here? Read on if you want to know more.

Cel Phone Towers.

Getting from point A to B.

One of the biggest reasons for all of these new and weird codecs appearing out of the woodwork is that when you graphically design something such as a website. That you want to be able to make it look good on high res monitors. But you also want to make it load as fast as a GeoCities website from the ’90s. Everyone is getting faster and faster so the concept of compression becomes irrelevant for major networks that want to hit you wish 1.8Mb of text and 22.7Mb of Javascript like CNN. That’s effectively an application every time you load their webpage! Now, you have a lot of big-boy companies out there that are serving millions of images per second. If you could deliver near-perfect quality content to your audience and save 30-50-70 percent of your bandwidth in doing so. Then it’s a win for everyone.

In our article about maybe not thinking with Google, you have people who believe that time is a temporal paradox that you can have 12 times the amount of people leaving your site than entering if your site exceeds 5 seconds.

That example is extreme; However, as a web designer, you want to at LEAST deliver the content of your site as fast as possible as there could be people on crappy cellphone towers or in countries where the internet is oftentimes not fantastic. Like Colorado. Yes, we’re calling you out Colorado.

Tor Networks blog title.

Also noted that about %10 of our readership comes from the Tor/Onion network environment. Because of the nature of Tor where of obfuscates your IP by bouncing your connection through a series of proxies connection time can vary from barely tolerable DSL connection speed to 56k modem mode depending on the locale and also depending on what supernode you are going through on their network.

Tor Results of S-Config.comPressing “12” will pull up the “network” tab for you to see your traffic.

Because the Tor/Onion client is built upon firefox builds it’s currently taking advantage of the WebP conversion that is happening within this site. The moment Tor adopts the later builds of Firefox those will be able to pull in AVIF files which will result in faster load times for our readers on the Onion/Darknet.

Deployment.

Now, we’re not going to use a PHP WordPress plugin in order to accomplish such feats. We -used- to use EWWW image optimizer back in the days where we were on shared hosting. but the settings it uses for WebP conversion were rather poor. To top it all off it will probably take years if it all for them to implement AV1F image formats.

Instead, we’re going to use some cron.script magic and also a little bit of Nginx fuckery.

Disclaimer.

Here at S-Config.com before you perform any script modifications of additions to your operating system you should always backup your data to preserve anything that happens from here. S-Config.com is not responsible for any damages that occur including, mental, physical, technological. Always act responsibly and double-check your data as not even WE are experts in these matters.

Nginx MIME action.

You’ve probably noticed in the network logs of our Tor/Onion browser that we are requesting a PNG or JPG but the “Type” category is a WebP. This is because the Nginx web-server is intercepting the request from my blog and converting the file path on the flow.

Let’s dive into Nginx to get this whole process going here.

First we need to tell Nginx what the *.avif file really is. So we need to open our mime.types within the root of nginx and add the following lines:

image/png png         apng;
image/webp            webp;
image/avif            avif;
image/avif-sequence   avifs;

and save.

Future versions of Nginx may already have these format-types set. But as of 1.19.x we certainly know avif is not apart of Nginx.

Moving onto the real action.

Nginx /sites-available/ redirection magic.

This is probably the point where you want to go to your /nginx/sites-available folder and find the site profile that you wish to modify. This is all done after your locations are defined within your profile.

    # for PNG and JPG - This will test if a browser can run AVIF and fall back to WEBP

    set $webp_suffix "";
    if ($http_accept ~* "webp") {
        set $webp_suffix ".webp";
    }
    if ($http_accept ~* "avif") {
        set $webp_suffix ".avif";
    }

    location ~ \.(jpg|png)$ {
       expires 1y;
       add_header Cache-Control "public, no-transform";
       add_header Vary "Accept-Encoding";
       try_files $uri$webp_suffix $uri $uri/ =404;
    }

    # for GIF - This will test for WEBP and then fallback to APNG files.


     set $gifv_suffix "";
    if ($http_accept ~* "apng") {
        set $gifv_suffix ".apng";
    }
    if ($http_accept ~* "webp") {
        set $gifv_suffix ".webp";
    }

    if ($http_accept ~* "avif") {
        set $gifv_suffix ".avif";
    }

    location ~ \.(gif)$ {
       expires 1y;
       add_header Cache-Control "public, no-transform";
       add_header Vary "Accept-Encoding";
       try_files $uri$gifv_suffix $uri $uri/ =404;
    }

This is essentially a modified version of the Nginx script to force WebP if it’s available. It firsts asks the client if they can accept a certain format such as WEBP first, then AVIF . This way, If a Tor/Onion user enters my site they will at least get the advantage of WebP. If a Google Chrome user enters my site then it will bypass WebP imaging and go right for AVIF.

Finally, the script TRIES the file to see if an appropriate WEBP or AVIF is out there. for example, if our blog has a link to “./logo.png” it will search for “./logo.png.webp” or “./logo.png.avif” whichever the browser admits to accepting. If the browser does not respond to Nginx or the user is operating on a Potato for a PC then Nginx will naturally serve them the default .jpg or .png files.

Apache time.

We haven’t really tested this as we’ve moved away from apache long ago. But as this too is based on the webp format you could just force everyone to avif all the same through the .htaccess file of your shared provider:

AddType image/avif .avif

<IfModule mod_rewrite.c>
  RewriteEngine On

  RewriteCond %{HTTP_ACCEPT} image/avif
  RewriteCond %{REQUEST_FILENAME} -f
  RewriteCond %{REQUEST_FILENAME}.avif -f
  RewriteRule ^/?(.+?)\.(jpe?g|png|gif)$ /$1.$2.avif [NC,T=image/avif,E=EXISTING:1,E=ADDVARY:1,L]

  <IfModule mod_headers.c>
     <FilesMatch "(?i)\.(jpe?g|png|gif)$">
     Header append "Vary" "Accept"
  </FilesMatch>
 </IfModule>
</IfModule>

CRON time!

Alright, so we addressed if the files are there or not. if we had a small website we could probably use a service like “squoosh” for example. No need to install files onto the server. Just convert the files you need to upload with the proper extensions and be done with it. However, as the image format wars wage onward and we like trying new things we needed a methodology mass conversion.

We’re about to show our CRON job. We are not fantastic at scripting so to those Linux users that are slapping their foreheads with the way we wrote it. Let us know. We would very much like to improve as well.

#!/bin/bash

Username=WebUser

cd /home/of/your/www/
shopt -s globstar
for i in **/*.jpg;
do
  if [ ! -f ${i}.flag ]; then
    echo "optimizing jpg ${i} "
    /usr/bin/nice -n 20 sudo -u $Username /usr/bin/jpegoptim --strip-all ${i}
    /usr/bin/nice -n 20 sudo -u $Username /usr/bin/convert -strip -sampling-factor 4:2:0 -interlace Plane -quality 75% -colorspace RGB -define jpeg:dct-method=float ${i} ${i}

    if [ ! -f ${i}.avif ]; then
      echo "optimizing avif ${i} "
      /usr/bin/nice -n 20 sudo -u $Username /usr/bin/avifenc --min 0 --max 25 --minalpha 0 --maxalpha 25 --speed 0 ${i} ${i}.avif
    fi

    if [ ! -f ${i}.webp ]; then
      echo "optimizing webp ${i} "
      /usr/bin/nice -n 20 sudo -u $Username /usr/bin/cwebp ${i} -q 90 -m 6 -pass 10 -o ${i}.webp
    fi
    echo "adding flag ${i}"
    /usr/bin/nice -n 20 sudo -u $Username echo "Optimization Complete Flag." >> ${i}.flag
    /bin/chown $Username:$Username ${i}.flag
  fi
done

for i in **/*.png;
do
  if [ ! -f ${i}.flag ]; then
    echo "optimizing png ${i} "
    /usr/bin/nice -n 20 sudo -u $Username /usr/bin/optipng -o7 -zm1-9 -strip all ${i}

    if [ ! -f ${i}.avif ]; then
      echo "processing avif ${i}"
      /usr/bin/nice -n 20 sudo -u $Username /usr/bin/avifenc --min 0 --max 25 --minalpha 0 --maxalpha 25 --speed 0 ${i} ${i}.avif
    fi

    if [ ! -f ${i}.webp ]; then
      echo "processing webp ${i}"
      /usr/bin/nice -n 20 sudo -u $Username /usr/bin/cwebp ${i} -q 90 -m 6 -pass 10 -alpha_q 90 -o ${i}.webp
    fi
    echo "adding flag ${i}"
    /usr/bin/nice -n 20 sudo -u $Username echo "Optimization Complete Flag." >> ${i}.flag
    /bin/chown $Username:$Username ${i}.flag
  fi
done

for i in **/*.gif;
do
  if [ ! -f ${i}.flag ]; then

    if [ ! -f ${i}.webp ]; then
      echo "processing webp. ${i}"
      /usr/bin/nice -n 20 sudo -u $Username /usr/bin/gif2webp -q 90 -m 6 ${i} -o ${i}.webp
    fi

    if [ ! -f ${i}.avif ]; then
      /usr/bin/nice -n 20 sudo -u $Username /usr/bin/ffmpeg -i ${i} -hide_banner -loglevel panic -pix_fmt yuv420p ${i}.y4m
      echo "processing to animated avif ${i}"
      if [ -f ${i}.y4m ]; then
        echo "processing to animated avif ${i}"
        /usr/bin/nice -n 20 sudo -u $Username /usr/bin/avifenc --min 0 --max 25 --minalpha 0 --maxalpha 25 --speed 0 ${i}.y4m ${i}.avif
        if [ ! -f ${i}.apng ]; then
          echo "appending animated apng ${i}"
          /usr/bin/nice -n 20 sudo -u $Username /usr/bin/gif2apng -z0 ${i} ${i}.apng
        fi
      fi
    fi

    if [ ! -f ${i}.y4m ]; then
      if [ ! -f ${i}.apng ]; then
        echo "processing non-animated apng ${i}"
        /usr/bin/nice -n 20 sudo -u $Username /usr/bin/gif2apng -z0 ${i} ${i}.apng
      fi
    fi
    rm -f ${i}.y4m
    echo "adding flag ${i}"
    /usr/bin/nice -n 20 sudo -u $Username echo "Optimization Complete Flag." >> ${i}.flag
    /bin/chown $Username:$Username ${i}.flag
  fi
done

for i in **/*.webp;
do if [ ! -f ${i:0:-5} ]; then
echo "removing ${i} as ${i:0:-5} is no longer existing."
rm ${i}
fi
done

for i in **/*.avif;
do if [ ! -f ${i:0:-5} ]; then
echo "removing ${i} as ${i:0:-5} is no longer existing."
rm ${i}
fi
done

for i in **/*.flag;
do if [ ! -f ${i:0:-5} ]; then
echo "removing ${i} as ${i:0:-5} is no longer existing."
rm ${i}
fi
done

for i in **/*.apng;
do if [ ! -f ${i:0:-5} ]; then
echo "removing ${i} as ${i:0:-5} is no longer existing."
rm ${i}
fi
done

Wow, that’s a lot of scripting action. Let us try to break it down a little for you.

In the beggining!
#!/bin/bash 

Username=WebUser 

cd /home/of/your/www/

The first part is probably the area you want to point onto your VPS if you were to use this script. We are using user control because scripts tend to run as root of the OS which would be rather problematic in terms of access rights depending on your web browser.

Conversion of .JPG files.

The script begins with checking for a .jpg.flag file. This is for sanity reasons. We don’t want to have the script constantly compressing the same jpg files over and over again. Then, following from what we learned about Jpeg compression from a previous blog we begin compression. If the flag isn’t present we begin compressing our jpeg images within our site more.

/usr/bin/convert -strip -sampling-factor 4:2:0 -interlace Plane -quality 75% -colorspace RGB -define jpeg:dct-method=float ${i} ${i}

Deviating a little bit from our earlier blogs about non-destructively compressing jpeg files. You may find yourself running a social network and other types of public services which involve forcing you to compress jpeg files. Especially if you are accepting images from people with mobile phones where they kind of suck at compression. We decided to use ImageMagicks ‘convert’ feature to re-compress jpeg images into something that search engines would consider ‘optimized.’

Keep in mind, although this command is lossy and destructive. It only happens once before the .flag file is dropped and thus is never accessed again. If you don’t want to do this simply delete that line out of the script to maintain lossless optimization.

JPG <-> AVIF

Now, we’re using colorists avifenc on my site as we can control the threads and memory usage which we simply could not do with Kagami’s Go-Avif .

The command we keep on passing to AVIF is the following.

avifenc --min 0 --max 25 --minalpha 0 --maxalpha 25 --speed 0 ${i} ${i}.avif

We setup a minimum to maximum compression to 25 which is the default. If you’re a photographer and you want to get the most compression out of AVIF without any perceptual quality loss I would recommend setting the –max and –maxalpha to 25. There’s a bit of gaussian blurring that happens in finely detailed images but you -really- have to look for it to notice.  Of course, sitting the compression lower will result in bigger files. So it’s a bit of a balancing act as to what you are trying to accomplish.

–speed 0 even though just like our PNG command is overkill. But it’s a background task on my VPS. I don’t care.

If you have no fucks to give about your still images like YouTube you’d probably be more inclined to up the compression to 30-32.

JPG <-> WEBP
/usr/bin/cwebp ${i} -q 90 -m 6 -pass 10 -o ${i}.webp

A lot of this was covered already in this blog article. Yes, we are passing a -q 90 to have some lossless destruction of our files. But we find this setting to be almost indistinguishable from the original file.

Dropping the flag.
 echo "adding flag ${i}" 
/usr/bin/nice -n 20 sudo -u $Username echo "Optimization Complete Flag." >> ${i}.flag 
/bin/chown $Username:$Username ${i}.flag

This is important because once the flag is dropped we don’t have to waste CPU time on our VPS to ever have to process this file again.

Conversion of .PNG files.

Now as for my commands passed on PNG files. We discussed all of those options in a previous blog. We are using a more aggressive command to optimize our PNG files however we find this does not affect the VPS in any way as we set ourselves a low priority. If we were processing several hundred png files at a time we may ease off so that daily.cron can keep up. But for now, this is totally fine.

Nothing really special changed here. The only warning we will say about this script is if your site has mixed animated .png files instead of the more traditional .apng format then the optimizer may destroy the animation within your .PNG files.

Conversion of .GIF files.

This is where the script gets a little crazy.

GIF <-> WEBP

This part is identical to how we convert everything else. Thankfully the WebP conversion utility can sense frames without issues.

GIF <-> AVIF

Because as of version 0.90 of avifenc can how to process animated movies as .avif. HOWEVER, we can’t jump immediately from gif to avif.

/usr/bin/ffmpeg -i ${i} -hide_banner -loglevel panic -pix_fmt yuv420p ${i}.y4m

To accomplish this, we turn to ffmpeg to take a GIF and convert it into the .y4m format which is YUV4Mpeg to which a lot of people on the net tend to use this format as a ‘master’ for digital media as the format is totally uncompressed. And the only time I want to hear from ffmpeg in my cron.daily is if something happens during conversion.

/usr/bin/avifenc --min 0 --max 25 --minalpha 0 --maxalpha 25 --speed 0 ${i}.y4m ${i}.avif

Afterward, we can convert the .y4m to avif and get amazing compression results versus GIF while preserving all of the frames in the animation. After the conversion is complete we delete that y4m as quickly as possible as those files can get huge rather quickly depending on the length of the animated GIF you are pushing through.

GIF <-> APNG

Using the gif2apng utility for Linux allows us to make appropriate animated png files or .apng that can be served to those who cannot understand webp or avif.

Script cleanup.

Each time we are converting an image on this site we echo a .flag file. This is important because we initially run this script outside of CRON to it takes the time to process all of the .png and .jpg files with ridiculous settings for compression. On my site, it took a solid week before it was able to place a flag file on all of them. So it’s a good idea to run the script first! We insert the script into our cron.daily so that it skips past all of the “.flag”ed files and only processes new files that come into this blog. Which is not as painful as doing the entire folder!

Finally, it does a reverse lookup on the original file. If we delete something in WordPress (or practically any other CMS script really) then the script will remove the files it has made as well keeping my directories clean.

Security considerations with WordPress.

You’ll probably be as ambitious as we were by instead of simply pointing this to the /wp-content/images folder that you’ll just point it to the root folder of WordPress. This way you’ll not only optimize images but any images that your plugins will be using. This is OKAY but you have to mind that if you are running any security plugins like WordFence of Sucuri Security you will more than likely piss those programs off. WordPress security software looks for any changes to the core of WordPress. and by adding *.webp, *.avif, *.flag files all over the place. Guess what? change detected.

So before you run a script like this (and if you are running a CMS similar to WordPress) make sure that the appropriate extensions are white-listed so you don’t come back to thousands of false-positives because of what you have done.

Shared hosting with AVIF/WEBP.

Chances are, your web-host administrator will probably not let you install a whole bunch of third-party binaries like ffmpeg, gif2png, cwebp, avifenc, and jpegoptim. If you have the option of SSH so you can transfer files to your account you could try to rsync your files back and forth using an offsite process like a Raspberry Pi to handle all of the image conversions for you. Sure, it would eat away at your bandwidth costs a little but it keeps your hard processing offline and just you serving a website online.

We’re using a Virtual-Private-Server (VPS) so we don’t need to do the back and forth r-sync services for something like that.

Final thoughts.

Serving AVIF through Chrome Browser.Serving AVIF files through the Chrome Browser.

 

Serving all of these different formats media-types does eat up the hard drive space. However, within the server world; It is no longer a question of how much space your website takes up but how fast your webserver delivers the media. Media Storage is cheap. Bandwidth is not.

Until next time.

Server protect you.

+++END OF LINE

Leave a Comment to the Void