{"id":2722,"date":"2020-05-06T20:34:51","date_gmt":"2020-05-06T20:34:51","guid":{"rendered":"https:\/\/aviancetechnologies.com\/blog\/?p=2722"},"modified":"2020-05-06T20:34:51","modified_gmt":"2020-05-06T20:34:51","slug":"how-to-upscale-video-content-to-4k-8k-and-beyond","status":"publish","type":"post","link":"https:\/\/aviancetechnologies.com\/blog\/how-to-upscale-video-content-to-4k-8k-and-beyond\/","title":{"rendered":"How to Upscale Video Content to 4K, 8K, and Beyond"},"content":{"rendered":"

\u00a0  <\/p>\n

[responsivevoice_button rate=”1\u2033 pitch=”1.2\u2033 volume=”0.8\u2033 voice=”US English Female” buttontext=”Story in Audio”]<\/p>\n

How to Upscale Video Content to 4K, 8K, and Beyond<\/h2>\n

<\/p>\n

\n
\n
\n \t \t \"https:\/\/www.extremetech.com\/\"<\/div>\n<\/p><\/div>\n

For the past three months, I\u2019ve been working on what I\u2019ve named the Deep Space Nine Upscale Project (DS9UP). The goal of DS9UP is to create a new, much-improved version of the show by applying modern processing techniques to the original DVD source before using AI-based software to create a higher-resolution version of the show. It\u2019s had me thinking about upscaling and upscalers in general. Upscaling isn\u2019t a feature we talk about much, but how your TV handles it (or, alternately, how you use the capability on a PC) can have a significant impact on how you experience content.<\/p>\n

The word \u201cupscale\u201d generically means \u201cto improve the value or quality of something.\u201d In the video and PC space, it\u2019s almost always a reference to increasing the effective resolution of a piece of content. There is typically an understood difference between upscaling and native resolution. If you upscale a 1080p video into 4K, it means you are taking a 1080p input and using a combination of hardware and software to create a larger image. This upscaled image will not be identical to a native 4K signal, but it should offer a better picture than what was previously available on your 720p or 1080p television.<\/p>\n

Keyword: \u201cshould.\u201d Video scalar quality in TVs can vary widely between different product families. In some cases, you might be better off using a GPU to drive a picture than relying on the TV\u2019s native rescaling capability, while other TVs have excellent upscalers. Manufacturers rarely disclose their upscaling hardware choices, but higher-end TVs should have improved upscaling capabilities. If you have a UHD Blu-ray player paired with an older or lower-quality 1080p or 4K TV, you might even get better results by running all video signals through the Blu-ray player rather than the television. Generally speaking, a good TV upscaler is considered to be as good or better than a GPU.<\/p>\n

How the Scalar Sausage Gets Made<\/h3>\n

The most basic function of a video scaler is to take whatever image it receives \u2014 480i, 720p, 1080p \u2014 and stretch it across the entire screen. Without this functionality, a 1080p signal would take up just a fraction of a 4K television\u2019s display. This simple resizing is typically done by taking each individual 1080p pixel and creating four pixels out of it (remember, 4K is four times the pixels of 1080p).<\/p>\n

But many TVs and Blu-ray players do more than just perform a simple 1:4 mapping. They also use video processing techniques to extrapolate what details ought<\/em> to be present in the scene. How well this works depends on the type of content being<\/p>\n

\"https:\/\/www.extremetech.com\/\"<\/p>\n

Image by DisplayNinja<\/p>\n<\/div>\n

In the image above, you can see how the upscaled 4K is much more nuanced than the simple 1:4 mapping in the second grid from the left. If you\u2019re having trouble seeing the difference between the 1080p upscale and the native 4K, look at the left-side blocks in the very first row and the right-side blocks in the very last row. The native 4K image resolves into distinctly different colors than the 1080p image in R1C2 (That\u2019s 1st row, 2nd column), R1C3, and R8C8. As the amount of available horsepower in televisions has improved, the quality and sophistication of their integrated upscalers have grown as well. Some modern TVs have sophisticated sharpening algorithms to reverse the blur created by upscaling and interpolation algorithms good enough to almost match the precision of a native 4K signal.<\/p>\n

How does all this look with real content? A 2017 Rtings article can help answer that question. The first image below is from a 4K set displaying 1080p in upscaled 4K, while the second is native 1080p.<\/p>\n

\"https:\/\/www.extremetech.com\/\"<\/p>\n

Image by Rtings<\/p>\n<\/div>\n

If you have trouble seeing a difference between the two images, open both of them in a new tab and focus your eyes near the center of the image. See the house with a brown roof near the center of the image, with three windows facing approximately south-southwest and a fourth pointed east? (All directional cues based on north being \u201cup\u201d, not the direction of the sunlight). Look at that specific spot in both images, and the roofs in the buildings immediately adjacent. The difference should jump out at you. In this case, even using TVs that date back to 2015, the 1080p upscale to 4K is better than the 4K image.<\/p>\n

\"https:\/\/www.extremetech.com\/\"<\/p>\n

Image by Rtings<\/p>\n<\/div>\n

If you have an older TV or a budget 4K model, there\u2019s one obvious method of improving your TV\u2019s upscaling: Buy a better television. Unfortunately, it\u2019s impossible to predict how well<\/em> this will work without knowing exactly what you own now and what you plan to purchase to replace it. The older your current TV, the better the chances that a new set will deliver upgrades in all respects, but many of those improvements may have nothing to do with the way your upscaler handles <4K content.<\/p>\n

If you aren\u2019t happy with your current TV, can\u2019t replace it at the moment, and happen to own a high-end Blu-ray or UHD player, you can also try running content through its upscaler rather than relying on the television to handle it. In some cases, a top-end UHD Blu-ray player may deliver a better experience than an entry-level 4K TV from a few years back. If you\u2019re still using a DVD player to feed a picture to a mediocre 1080p or 4K panel when\/if you play DVDs, and you can swap over to a high-end Blu-ray \/ UHD Blu-ray player instead, I\u2019d try it. It may or may not help, but it definitely won\u2019t hurt. What you\u2019re trying to do here is route the signal through the upscaler that\u2019ll give it the best quality kick.<\/p>\n

Still need a higher-quality picture? You\u2019re in luck.<\/p>\n

Real-Time AI Processing<\/h3>\n

I haven\u2019t tested the most recent Nvidia Shield myself, but there\u2019s a demo you can actually play with on Nvidia.com to apply the effect the TV offers. Here\u2019s a screenshot of the effect. I\u2019ve positioned the slider over the lizard\u2019s eye because it\u2019s the easiest place to see the upscaler\u2019s impact:<\/p>\n

\"Upscale-Comparison\"<\/p>\n

Left: Regular upscale
Right: Nvidia Shield<\/p>\n<\/div>\n

Still not clear? Here\u2019s an enlarged version of the same screenshot.<\/p>\n

\"https:\/\/www.extremetech.com\/\"<\/p>\n

The image on the left is a traditional upscaler, the image on the right is Nvidia\u2019s Shield when asked to scale up 720p or 1080p content to 4K (content below 720p is not supported for upscaling, at least not yet). The AI component of the upscaler obviously improves the overall image quality. In my experience with applications like TVEAI, this is a fair representation of the improvements that can be achieved.<\/p>\n

Third-party reviews of the Shield agree. Slashgear writes that when it works, the effect is \u201cfairly astonishing.\u201d So far as I\u2019m aware, the Shield is currently the only set-top or 2D box offering this kind of functionality.<\/p>\n

Video Upscaling via Third-Party Software<\/h3>\n

Finally, there\u2019s the option to use a third-party upscaler, like Topaz Video Enhance AI. I\u2019ve made extensive use of TVEAI as part of the Deep Space Nine Upscale Project, and can confirm that the application is capable of yielding stunning results. One major limitation of TVEAI, however, is that it currently only supports Intel and Nvidia platforms for GPU-accelerated processing. CPU processing is available, but likely too slow to be all that useful.<\/p>\n

Thanks to AI-based upscaling, historic sci-fi canards like the \u201cEnhance\u201d function are now a reality. It\u2019s a truism in video encoding that no application on Earth can put data back where it never existed, and that\u2019s still true today. The reason we can now \u201cenhance\u201d images to improve their clarity is that AI-based applications are capable of analyzing a video field and estimating what detail would<\/em> exist if the image were in higher quality already.<\/p>\n

One of the typical ways to train a video-enhancing application like TVEAI is to provide the neural net with the same image or video in both high and low quality. The neural net is then tasked with finding a way to make the low-quality source look as much like the high-quality source as possible. Instead of trying to teach a computer what lines and curves look like by providing painstaking examples, we\u2019ve developed the ability to make the computer do the work of teaching\u00a0itself.<\/em> That\u2019s why \u201cEnhance\u201d has gone from a complete joke to a plausible reality in a matter of a few years.<\/p>\n

\"https:\/\/www.extremetech.com\/\"<\/p>\n

You may need to zoom to see subtle differences, but this isn\u2019t an upscale comparison. It\u2019s a comparison of performing my upscale work in separate applications and steps (left-side) versus ingesting the entire video workload into Topaz Video Enhance AI using the AVFS virtual file system and performing the entire operation at once. If you\u2019re thinking \u201cWow, that\u2019s a tiny difference, who would even care?\u201d well, you may be right \u2014 but I wasn\u2019t kidding when I said I was going to use the highest-quality source possible.<\/p>\n<\/div>\n

Applications like Topaz Video Enhance AI can cost several hundred dollars and they don\u2019t run in real-time \u2014 the RTX 2080 appears capable of 90-110 frames per minute when upscaling 640\u00d7480 video to 2560\u00d71920. The result of using these applications, however, is a vastly better picture than you\u2019ll see from any other source.<\/p>\n

\"https:\/\/www.extremetech.com\/\"<\/p>\n

DVD source on the left, upscale on the right.<\/p>\n<\/div>\n

I suspect we\u2019ll see AI video processing and upscaling become more important over time as Intel, Nvidia, and AMD introduce their next-generation graphics processors. There\u2019s an awful lot of old content orphaned on bad-quality source, and building specialized AI processors to fix it will likely consume some silicon for all of the concerned parties over the next few years.<\/p>\n

Finally, I\u2019ve embedded the current version of DS9UP\u2019s opening credits for the show below. This is new footage that hasn\u2019t appeared in a previous DS9UP article. There is no audio in this clip and you\u2019ll need to raise the resolution to 4K, but I\u2019ve created this file based on the DVD source with modifications in AviSynth, DaVinci Resolve, and Topaz Video Enhance AI.<\/p>\n