Enhancing videos with Video2x

For those who are interested in enhancing upscaled videos, there is a tool known as Video2x that uses machine learning to "fill in the gaps" when upscaling. There are releases for Windows but getting it to run on Linux was actually much easier than I thought.

First, there is always the docker option. You can find instructions here from the repo's wiki page. I shall not touch more on this because it is actually quite straightforward.

I followed the instructions here to get it running on Linux Mint 20.

The first step is to clone the repo.

git clone https://github.com/k4yt3x/video2x.git
cd video2x/src

Video2x requires Python3 and FFMPEG to run. Python should already be on your system, but FFMPEG can usually be installed by

sudo apt install ffmpeg

on Debian-based systems.

Then, install the require Python libraries.

pip install -r requirements.txt

The tricky step is finding the drivers to install. Of the various drivers that Video2x supports, I found precompiled binaries for waifu2x-ncnn-vulkan and srmd-ncnn-vulkan. Download the precompiled Linux binaries from their respective release pages, then uncompress the folders. I renamed the folders to waifu2x-ncnn-vulkan and srmd-ncnn-vulkan respectively and place them in a dependencies folder inside video2x/src.

This is actually all you need to get started. Then, just run

python3 video2x_gui.py

to start up the GUI. You will need to go to the Drivers tab to select the path to your drivers, and the FFMPEG tab to tell it where to find ffmpeg. This needs to be done each time... so the better way is to edit video2x/src/video2x.yaml with the respective paths, like below.

 

waifu2x_ncnn_vulkan:
  path: 'dependencies/waifu2x-ncnn-vulkan/waifu2x-ncnn-vulkan'

srmd_ncnn_vulkan:
  path: 'dependencies/srmd-ncnn-vulkan/srmd-ncnn-vulkan'

ffmpeg:
  ffmpeg_path: '/usr/bin'

(Without leading slash, the path is taken as a relative one; the leading slash makes the path an absolute one.)

Then, Video2x should be able to find the drivers and ffmpeg executable each time without problem.

Video2x is not... super great on generic videos, but the results are much better than using the various upscaling/denoising tools in FFMPEG. It works best on anime since that was what the waifu2x driver was originally developed for. Video2x can be very slow, though. A 1080p video gets processed at about 2 seconds per frame... so any full-length movie will take days to process. Upscaling a 55min video from 720p to 1080p took me around 11 hours...

By the way, to use Nvidia GPUs, you will need to install CUDA and cuDNN for your system. I can't really give much advice on this... it was a struggle for me to get them installed on my current build (Linux Mint 20 on Ryzen 3600 with GTX 1660 Super). Those running Ubuntu probably have it easier, so if you are going to build a PC just for such work, I highly recommend installing Ubuntu instead.

Update on 7 Sep 2020: Dandere2x looks promising too, but I have not been able to get it to work. Keep getting broken pipe errors. I think it has to do with PIL and Pillow (I think I somehow managed to get them both installed, and they could be interfering with each other). Anyway, if I find the time and motivation, I will see if I can get it to work.