How do I quickly generate low quality animated GIFs with ffmpeg?

We generate a lot of thumbnail GIFs whose quality does not matter nearly as much as the time it takes to generate them. Generating high quality GIFs with ffmpeg is very well covered, but I'm not having much luck figuring out how to generate low quality ones as quickly as possible.

The palette computation takes up most of the execution time with the following command (taken from the multi-chain filtergraph answer here: How to efficiently create a best-palette gif from a video portion straight from the web):

ffmpeg -y -threads 8 -r 24 -f image2 -start_number 1 -i "frames.%04d.jpg" -filter_complex "fps=24,scale=150:-1:flags=fast_bilinear,split=2 [a][b]; [a] palettegen [pal] fifo [b]; [b] [pal] paletteuse" output.gif

The execution time of that command with 1000 frames is about 72 seconds. About 67 seconds of that is the palette pass, and then it blazes through the actual GIF generation in about 5 seconds. I would like to get the whole execution time down as much as possible, and willing to sacrifice a lot of image quality for speed.

Rjak

Posted 2018-08-08T05:25:21.073

Reputation: 221

1Your command is syntactically incorrect; you're missing a [b] label. Either way, using the palette is about 200⨉ slower judging from a quick test. Not using it at all is not an option for you? – slhck – 2018-08-08T09:00:26.213

@slhck I am finding the ffmpeg docs around this stuff pretty tough to grok, but I came up with this for eliminating the palette: ffmpeg -y -r 24 -f image2 -start_number 1 -i "frames.%04d.jpg" -filter:v "scale=150:-1:flags=fast_bilinear" output.gif The two problems with this: 1) It only saved 13 seconds (59 seconds instead of 72), and 2) I'm not 100% sure this command eliminates palette computation altogether (only that the palette computation I was specifying before is not included). – Rjak – 2018-08-08T21:58:44.377

1Yes, that's basically what I meant. Sorry for not being so explicit, I thought you knew what the filter command you were using was doing. In my tests, leaving out the palette altogether actually sped up the command significantly. What led you to the original conclusion that the actual encoding only took 5 seconds? – slhck – 2018-08-09T07:08:15.880

Observation of the output, which is probably not a great indicator. When I run the command without palette generation, the "current frame" indicator spins up immediately and we start processing frames. When I run it with palette generation, the current frame sits at 0 until about 5 seconds before the end of processing, then it spins all the way through the frames very quickly. – Rjak – 2018-08-09T22:24:20.343

1Yes, that's also what I'm observing, but it's much faster then. But I guess it depends on the length of the video. To be honest, I don't know if there's any faster way to generate the GIF than just doing ffmpeg -i <input> <scale> <output.gif>. – slhck – 2018-08-10T06:33:38.323

Yeah it sure seems like I'm maxing it out. I am now working on another method where we skip frames if the clip is long enough (e.g. if the clip is 15,000 frames we only use every 100th frame so that we end up with an animated GIF with 150 frames in it). The hurdle there is that I am in fact ending up with a 150 frame GIF, but it appears to be doing computation on all the frames I want to skip as well so it is taking the same amount of time to complete! Getting this with -filter_complex "select='not(mod(n\,100))',scale=150:-1:flags=fast_bilinear" – Rjak – 2018-08-13T04:08:18.940

Answers

I was tasked with reducing the amount of time it took to generate an animated GIF as close as possible to 30 frames in length at 150 pixels width. Most of the sequences we generate are under 1000 frames. We had a 15,000 frame sequence and our render nodes were taking 17 minutes to produce this ~30-frame GIF, which is unacceptably slow.

We were using ffmpeg as a demuxer and piping to imagemagick. Through several hours of experimentation, I arrived at the following conclusions:

The number of input frames you ask ffmpeg to process is BY FAR the most impactful input in terms of execution speed. If using the concat demuxer to skip input frames is an option, this will make the biggest performance difference. By taking every 5th frame, I was able to reduce total computation time to 1 minute 45 seconds with high-quality lanczos rescaling and per-frame palette computation. Generating our 30-frame preview thumbnail now takes under 1 second.
The rescaling algorithm was the next largest performance impactor (but a far distant second). Using fast_bilinear instead of lanczos saved 150 seconds of compute time over all 15,000 frames.
The least impactful variable was palette computation, and this varied with rescale algorithm. Over 15,000 frames using lanczos, we saved around 17 seconds of execution time if we eliminated palette computation. Using fast_bilinear, we saved around 75 seconds of execution time.

Because the rescaling algorithm and palette computation were negligible, we ended up keeping them at the highest quality. We have reduced our computation time from 17 minutes to under 1 second mostly by telling ffmpeg to skip reading input files.

KEY TAKEAWAY: SKIPPING INPUT FRAMES vs SKIPPING OUTPUT FRAMES

The reason our process was taking so long is that frame dropping does not help execution time when using the image2 demuxer. If you muck with the -r flag and the fps filter, you will affect the number of frames which appear in the final GIF, but ffmpeg appears to still do something with all 15,000 input frames.

The only way I could find to have ffmpeg skip input frames is by using the concat demuxer.

Here is how I now generate high-quality animated GIF thumbnails on my dev machine in under 1 second by skipping input frames:

# create text file which describes the ~30 input frames we want ffmpeg to process
seq -f "file 'left_frames.%04g.jpg'" 10000 500 25000 > tmp.txt

# generate the animated gif using ffmpeg only
ffmpeg -f concat -i tmp.txt -filter_complex "scale=150:-1:flags=lanczos,split=2 [a][b]; [a] palettegen [pal]; [b] fifo [b]; [b] [pal] paletteuse" output.gif

Rjak

Posted 2018-08-08T05:25:21.073

Reputation: 221

Your use of the palettegen/paletteuse filters is making the command run slower. The simple way to achieve a lower-quality GIF would be:

ffmpeg -f image2 -i "frames.%04d.jpg" output.gif

With additional scaling:

ffmpeg -f image2 -i "frames.%04d.jpg" -vf scale=150:-1 output.gif

You can also drop frames in the output GIF, i.e. sample the frames, so that not all of them are processed. E.g. to have only 1 FPS output, by using an fps filter:

ffmpeg -i "frames.%04d.jpg" -vf "fps=fps=1,scale=150:-1" output.gif

slhck

Posted 2018-08-08T05:25:21.073

Reputation: 182 472

I think there's a typo in that last example there ... should be `-vf "fps=1,scale=150:-1". The execution time of the second example is 68.48s. Specifying fps=1 does result in a 1 FPS GIF, but the source is 120fps so the result plays back too slowly and the execution time is not improved enough (59.44s). I am experimenting with using the concat loader and having ffmpeg read every Nth frame. Doing that, if I sample every 5th frame using Lanczos rescaling and per-frame palette generation, the temporal quality looks great, the color quality looks great, and the command executes in under 8 seconds. – Rjak – 2018-08-15T03:34:44.123

Will share the whole thing in a comment here once I have fully tested and have execution timings. – Rjak – 2018-08-15T03:35:30.960

@Rjak The first option of the fps filter is named fps, so these are equivalent. If your source is 120fps, you need to specify -framerate 120 -i "frames…", because the default for input is 24. – slhck – 2018-08-15T11:05:28.563