Analytics in 360 Video


I remember the early days of Oculus DK1 when most of developers were simply adding a VR mode to existing games. Fortunately these days are gone and more developers make decisions to create a VR-only games or add some features that will justify adding a VR mode.

There are several ways how we can learn about users’ experience and improve our VR games. After your game is out on a market it’s good to have some sort of Analytics system to help us understand player’s behaviour. Unity provides a cool Analytics system that allows you to monitor DAU or trigger custom events. On top of the Analytics system, there is a pretty neat Heatmap editor. It’s basically a plugin that sits on top of Analytics custom events. It allows you to get a raw data (Pro feature) and import it right into your editor and render it in a human-understandable way.

In this article I will focus on one part of VR experience – 360 video. We will try to use Analytics data to display traditional heatmaps over the video and try to figure out where our users look at. I also want to help you to understand how the 360 video works in Unity, what are the different types of projection, how to visualise data as a heatmap and how to update gaze positions as the video is being played.

 

User’s gaze position

One way to analyse where player focuses on when watching a 360 video could be gathering information about gaze position. Current generation of VR headsets doesn’t have eye-tracking technology that’s why we have to do some simplifications. Let’s assume that main camera is inside the sphere with our 30 video at the centre of the scene at position (0, 0, 0). Our gaze position in this situation will be a forward direction of the camera.

For my tests I decided to use a Star Wars 360 experience from YouTube.

I gave it to my colleagues at the office to watch. After the video was played for about 30 times I gathered all the analytics data and put it into my Unity project to display it as a heatmap. It was interesting to see how people reacted to the video. I must say that the gaze positions were quite consistent. The Star Wars experience was made with a lot of thought into it. At some part of the video you can notice that people’s sight is directed by fast moving objects (like the friend X-Wing).

The current version of the project on GitHub uses data saved and read from binary file. MovieAnalytics.cs class handles the updates of gaze positions that are later sent to Heatmap.cs.

MovieAnalytics has two modes: Read and Write.

In the “Write” mode, once per second TryToGetAnalyticsPoint() checks if the camera’s angle difference is greater than minAngleDifference, if so, it will create a new Vector4. XYZ components represent forward direction of the camera and W component specifies the time when the vector was created. Later, vector is added to a list inside a ViewData. After the video is finished ViewData is added to the AnalyticsData and then saved as a binary file.

In the “Read” mode, UpdateGazePositions() reads the data that was recorded, interpolates the positions from ViewDatas and adds them to a list of positions that is later sent to Heatmap.cs to display it on a sphere with a video.

 

360 Video in Unity

Unity 5.6 brought many interesting changes and new features. The one we’re interested here in particular is 4K video player. 360 videos by definition provide video wherever you look. What that means is if you want to achieve a similar quality to let’s say 720p or 1080p the source video has to be rendered in much higher resolution. To add a 360 video to your scene, simply drag your video asset onto any game object and set render mode to “render texture” or “override material”. With this mode video will be rendered into your material. Since we are going to watch the video from the inside we need to flip normals in our material:

Material that you are rendering to needs to be placed on a surface which is usually cube or sphere. In this example we are going to render it on a sphere since it’s the most popular method, but I want to make you aware that there are different types of projections:

1.Equirectangular

Equirectangular projection is the most widely spread technique. It uses the same principle as the world map being stretched on a 2D surface. You can notice that the further you go from the equator the more stretched the image becomes. This means that pixels close to the poles are overrepresented. As a result we have a lot of redundant information.

Here is an example from a real 360 video:

2. Cubemap

Cubemaps are everywhere in computer graphics and games they are used for skyboxes or reflections. This technique is relatively simple. Pixels from the video are mapped onto the 6 faces of the cube:

Big advantage of this approach is that we have much less distortion within the faces and pixels have better distribution. Facebook developers claim that by using heatmaps they are able to reduce video size by 25%: https://code.facebook.com/posts/1126354007399553/next-generation-video-encoding-techniques-for-360-video-and-vr/

3. Equi-Angular Cubemap (EAC)

Cubemaps are not without flaws. The main problem we have here is that the centers of the cube faces are close to the sphere, while the corners are further away:

This can be corrected by changing sampling of the video. On a picture above you can see that distance between the pixels on each face have equal distance between each other.
Here is an image provided by google engineers with saturation of an image when using different techniques:

Green colour represents an optimal pixel density; red, orange and yellow represent insufficient pixel density;  blue indicated over-sampled area with wasted resources. 

 

Visualisation

Heatmap is one of the standard techniques to visualise spatial distribution of data. It’s widely used in fields like meteorology and other data analysis. Some web companies use them to visualise user’s focus. We will try to apply similar technique to our 360 video.
Let’s start with a simple example. Imagine that you have user looking at the certain point on the sphere and we want to visualise how far every pixel is from the centre of this point. It’s much easier to operate on values from 0 to 1 that’s why we will try to find a function that match our needs.

Falloff curve

Let’s start with a following function:

Let’s try to plot it using WolframAlpha for DistMax of 30:

You can see that it is a linear function that is dropping together with the distance (horizontal axis).

The only way to represents values in shaders is by using colours. Let’s generate a simple texture with hue from blue to red:

After passing it to our shader here is what we have:

Pure blue colour represents distances bigger than our max distance, and the closer we get to the distance = 0 then more red colour we have.

Here is our shader code:

 

We can try to smoothen colour’s transition as the distance approaches MaxDistance by taking our value to power of 2:

Here is the modified GetFalloff function in our shader:

To smoothen transition close to Distance = 0 we need to take a power of 2 of our distance divided by max distance:

Here is the modified GetFalloff function:

Finally we can combine those two:

Here is the modified Falloff function:

Congratulations! You’ve just found your own falloff curve! This is actually exactly the same function as the one described in this article Wikipedia about metaballs. We can read there:

When seeking a more efficient falloff function, several qualities are desired:

  • Finite support. A function with finite support goes to zero at a maximum radius. When evaluating the metaball field, any points beyond their maximum radius from the sample point can be ignored. A hierarchical culling system can thus ensure only the closest metaballs will need to be evaluated regardless of the total number in the field.
  • Smoothness. Because the isosurface is the result of adding the fields together, its smoothness is dependent on the smoothness of the falloff curves.

The simplest falloff curve that satisfies these criteria is f(r)=(1-r^2)2 where r is the distance to the point. This formulation avoids expensive square root calls.”

We have to bear in mind that our method uses for loop in the fragment shader to calculate these values. That’s why we should try to keep our falloff function as simple as possible.

Isosurfaces

Now, when we have discovered a suitable falloff curve we can proceed to generating isosurfaces in our shader.

 

At this point it’s as simple as adding contributions from each point. We need to introduce a for loop into our shader. For each point in the array that we passed we have to calculate the falloff and add it together:

Here is a result with 3 arbitrary points:

Since we have to pass a list of points to our shader, we have to use a MaterialPropertyBlock and SetVectorArray .

Here is our modified shader:

We have a small problem here. For a large amount of points our view will be totally obscured with large amount of red colour and it will be much harder to reason from data.

One thing that we can do is to divide the sum by number of points:

This is a mean value of all the contributions. We can think of our result colour of how close our pixel is to all the points:

Let’s take a look at the modified GetFalloff function:

 

We can try to model appearance of our data by modifying our look-up texture. If you want to represent the falloff values with different colour values.  I found that this one fits nice in my needs:

If we bring points closer together we get back our red colour:

After I put the video playing part and analytics part together the distance di in GetFalloff function was replaced with the angle between the fragment position and gaze forward direction. I made that decision because the video is displayed on a sphere so it was easier for me to operate on angles instead of positions:

 

Final result

 

 

You can find source code, shaders and textures used in this article on my GitHub.

 


 

I hope that in the future I can come back to this project and gather data that includes eye tracking information.

Share:

+ There are no comments

Add yours