Essays, Posts & Presentations

Computational Video Premieres in Wolfram Language 12.1

Shadi Ashnai

Author

Shadi Ashnai

Title

Computational Video Premieres in Wolfram Language 12.1

Description

Powerful Video object released with Wolfram Language 12.1. How to use the Video object, access parts of a video, analyze and compute with video, image and audio processing, generate video.

Computational Video Premieres in Wolfram Language 12.1

May 19, 2020

Shadi Ashnai, Manager of Sound & Vision, Algorithms R&D

Version 12.1 of the Wolfram Language introduces the long-awaited

Video

object. The Video object is completely (and only) out-of-core; it can link to an extensive list of video containers with almost any codec. Most importantly, it is bundled with complete stacks for image and audio processing, machine learning and neural nets, statistics and visualization and many more capabilities. This already makes the Wolfram Language a powerful video computation platform, but there are still more features to explore.

The Video Object A video file typically has a video and an audio track. Here is a Video object linked to a video file:

In[]:=

Video["ExampleData/Caminandes.mp4"]

Out[]=

In Version 12.1, by default, the Video object is displayed as a small thumbnail and can be played in an external player. There are other appearances to enable in-notebook players, like the Video object with a basic player:

In[]:=

Video["ExampleData/Caminandes.mp4",Appearance"Basic"]

Out[]=

Now you can inspect the Video object:

In[]:=

Duration[

]

Out[]=

min

30.036

In[]:=

Information[

]

Out[]=

Most video containers support multiple video, audio and subtitle tracks. Having multiple audio or subtitle tracks in a single file is more common than having more than one video track.

This is an example of a Video object linking to a file with multiple audio and subtitle tracks:

In[]:=

Information[Video["ExampleData/bullfinch.mkv"]]

Out[]=

Accessing Parts of a Video There are several parts of a video you may be interested in extracting. Use

VideoFrameList

and

VideoExtractFrames

to extract specific video frames. You can also use VideoFrameList to sample the video uniformly or randomly with

frames:

In[]:=

VideoFrameList[

,3]

Out[]=

Use this function to create a thumbnail grid (a group of smaller images that summarizes the whole video):

In[]:=

VideoFrameList[

,12]//ImageCollage

Out[]=

You can also trim a segment of a video:

In[]:=

VideoTrim[

,{30,60}]

Out[]=

Or extract only the audio track from a video to analyze it:

In[]:=

Audio[

]

Out[]=

In[]:=

Spectrogram[%]

Out[]=

Performing Analysis In Version 12.1, we have introduced

VideoTimeSeries

, which works on frames of a video file to perform any computation—either one frame at a time or a list of frames all at once. This is a powerful tool capable of analysis like in the examples below.

Compute the mean color of each frame over time:

In[]:=

VideoTimeSeries[Mean,

]//ListLinePlot#,

Rule[

]

&

Out[]=

Count the number of objects (cars, for example) detected in each frame of a video:

In[]:=

v=Video["http://exampledata.wolfram.com/cars.avi"];

In[]:=

ts=VideoTimeSeries[Point[ImagePosition[#,Entity["Word","car"]]]&,v]

Out[]=

TimeSeries

Time: 0.02 to 9.98
Data points: 250



Plot the number of objects (again, using cars as an example) detected in each frame:

In[]:=

TimeSeriesMap[Length@@#&,ts]//ListLinePlot

Out[]=

Highlight the position of all detected objects (cars) on a sample frame:

In[]:=

HighlightImage[VideoExtractFrames[v,1],{AbsolutePointSize[3],Flatten@Values[ts]}]

Out[]=

We can also use the multiframe version of the function to perform any analysis that requires multiple frames.

By looking at consecutive frames from a pixabay video and computing the difference between four views, we can find the transition times from one view to another and then use those times to extract one frame per scene:

In[]:=

v=Video["Musician.mp4"]

Out[]=

In[]:=

diffs=VideoTimeSeriesImageDistance@@#&,v,

frames

frame



Out[]=

TimeSeries

Time: 0.0334 to 25.3
Data points: 758



In[]:=

ListLinePlot[diffs,PlotRangeAll]

Out[]=

In[]:=

times=FindPeaks[diffs,Automatic,Automatic,150]["Times"]

Out[]=

{8.008,15.1151,20.9876}

In[]:=

VideoExtractFrames[v,Prepend[times,0]]

Out[]=





Process a Video The Wolfram Language already included a variety of image and audio processing functions.

VideoFrameMap

is a function that takes one frame or a list of video frames, filters them and writes them to a new video file. Let’s use the bullfinch video:

In[]:=

v=Video["ExampleData/bullfinch.mkv"];VideoFrameList[v,3]

Out[]=

We can start with a color negation as a simple “Hello, World!” example:

In[]:=

VideoFrameMap[ColorNegate,v]//VideoFrameList[#,3]&

Out[]=

Or posterize frames to create a cartoonish effect:

In[]:=

f=With[{tmp=ColorQuantize[#,16,DitheringFalse]},tmp-EdgeDetect[tmp]]&;

In[]:=

VideoFrameMap[f,v]//VideoFrameList[#,3]&

Out[]=

Use a neural net to perform semantic segmentation on the previously used video of cars:

In[]:=

v=Video["http://exampledata.wolfram.com/cars.avi"];

In[]:=

segment[img_]:=Block[{net,encData,dec,mean,var,prob},net=NetModel["Dilated ResNet-38 Trained on Cityscapes Data"];encData=Normal@NetExtract[net,"input_0"];dec=NetExtract[net,"Output"];{mean,var}=Lookup[encData,{"MeanImage","VarianceImage"}];Colorize@NetReplacePart[net,{"input_0"NetEncoder[{"Image",ImageDimensions@img,"MeanImage"mean,"VarianceImage"var}],"Output"dec}][img]]

In[]:=

VideoFrameList[VideoFrameMap[segment,v],3]

Out[]=





Next is a video stabilization example, which is a vastly simplified version of this Version 12.0 product example. The input video is another pick from pixabay:

In[]:=

v=Video["soap_bubble.mp4"]

Out[]=

Here is the mask over the ground to make sure the shaking soap bubble movement does not affect our stabilization algorithm:

In[]:=

mask=

;

Next is a routine to find correspondence and geometric transformation between every two consecutive frames, iteratively composed with the previous transformation to get a stabilization all the way to the initial frame:

In[]:=

f=Identity;VideoFrameMapModule{tmp},tmp=Last@FindGeometricTransform##,

Rule[

]

&@@ImageCorrespondingPointsSequence@@#,

;f=Composition[tmp,f];ImagePerspectiveTransformation#[[2]],f,

&,v,

frames

frame

;

From Manipulate to Video Let’s switch the topic to generation of video.

Manipulate

has been a core way of creating animations in the Wolfram Language for over a decade. In Version 12.1, Manipulate expressions can easily be converted to video.

This is a Manipulate from the Wolfram Demonstrations Project:

In[]:=

m=ResourceData["Demonstrations Project: Day and Night World Clock"]

Out[]=

And a video generated from it:

In[]:=

Video[m]

Out[]=

A video can also be generated from a Manipulate and a

Sound

Audio

object:

In[]:=

Export["file.mp4",{"Animation"m,"Audio"ExampleData[{"Audio","PianoScale"}]},"Rules"]//Video

Out[]=

A Short Note about Supported Codecs The Wolfram Language by default uses the operating system as well as a limited version of FFmpeg for decoding and encoding a large number of multimedia containers and codecs.

$VideoEncoders

$VideoDecoders

$AudioEncoders

, etc. list supported encoders and decoders.

Codec support can be expanded even further by installing FFmpeg (Version 4.0.0 or higher). This is the number of decoders and the list of MP4 video decoders on macOS with FFmpeg installed:

In[]:=

Length/@$VideoDecoders

Out[]=

AVI141,Matroska237,MP421,Ogg1,QuickTime63,VideoFormat259

In[]:=

$VideoDecoders["MP4"][[All,1]]

Out[]=

{DIRAC,H264,H264-AVF,H264-VIDEOTOOLBOX,HEVC,HEVC-AVF,HEVC-VIDEOTOOLBOX,JPEG2000,JPEG-AVF,LIBAOM-AV1,LIBOPENJPEG,LIBVPX-VP9,MJPEG,MPEG1VIDEO,MPEG2VIDEO,MPEG4,MPEGVIDEO,PNG,TSCC2,VC1,VP9}

More to Come Video computation in the Wolfram Language is only at its beginning stages. The new capabilities featured here are only part of an already powerful collection of video basics, and we are actively designing and developing updates to existing functions and additional capabilities for future versions, with machine learning and neural net integration at the top of the list. Let us know what you think in the comments—bugs, suggestions and feature requests are always welcome.

Cite this as: Shadi Ashnai, "Computational Video Premieres in Wolfram Language 12.1" from the Notebook Archive (2020), https://notebookarchive.org/2020-05-8rvs1dk