I’m now working for more than two years on Mixed Reality concepts and projects with the HoloLens device. While the device is for me one of the best Mixed Reality devices and can do incredible things when it comes to 3D visualization, connecting to services and much more, it sometimes let you think everything is possible. And… that is not the case. Writing applications for HoloLens expect much more from you than understanding the HoloLens SDK and writing C# code.

There is so much you need to understand to have great performance with your HoloLens application. This post is not intended as a deep and complicated post regarding some of the things you can do to allow your app to perform, but tries to explain on a somewhat understandable level the tips (and if you want to call it tricks) to take into account regarding performance and optimization when writing HoloLens applications.

This post contains examples of things i have crossed myself, my team during builds and stuff i found online. The purpose of the post is having some tools to get your app running as it should be.

So let’s start!

FPS

The first and most important thing: Know how your FPS is. FPS? FPS stands for Frames Per Second and can be displayed using the “FPSDisplay” prefab in the Mixed Reality Tool Kit. It continuously shows the FPS of your application when running your HoloLens app. FPS is important! Not going to much in detail, but try to keep the FPS of your application running somewhere between 40 en 60 FPS. Rather 60 FPS than lower. Be aware.. using the preview camera in the Windows Device Portal will drop the FPS by 30. So that can give you some strange and incorrect numbers. As soon as the FPS will drop, the stability of your application will drop and can result in many different things. Stability in spatial mapping, positioning of Holograms and other nasty effects.

Toolkits & Unity Store assets

Try to limit the use of the Mixed Reality Tool Kit. Yes. I just told you that there is a great prefab in the MRTK to monitor your FPS, but still… try to minimize the use. Why? Because it is a toolkit. A toolkit means a lot of stuff you are not using which is compiled together with your application. Yes..It makes your live easier when building applications, but keep in mind that as soon as you start using something from a toolkit, you get a lot of other components and stuff which is part of or related to it. That can influence the performance and stability of your application. An to be honest…i have seen it happen multiple times when using the MRTK in one of our projects. I’m now nagging about the MRTK. But this remark is of course for any toolkit you want to use. Unity has a great store of stuff you can use in your applications. But that is the same as all the toolkits. You are importing a lot of functionality in your application and in most cases more than you want. The solution? Use toolkits or imported stuff from the Unity store partially. Remove everything which you can remove. Don’t try to use those create prefabs (like the HoloLens camera) when you can easily built one for yourself. Create for those things your own lean and mean library with prefabs and functionality which you can use in your applications.

CPU depending code

Don’t execute code which is heavily depending on the CPU of HoloLens. Think of heavy calculations, image splitting functions and handling video feeds. The power of HoloLens lies in the fact that it has a simple but good CPU and makes use of a powerful GPU and the incredible HPU. But keep in mind that the device is not intended to do heavy CPU activities. If you have the need for heavy processes like calculations and other stuff use, depending on your app and its environment, a local system/ PC or service in the Cloud. An example could be doing some calculations which give you specific numbers for running your app through a service which is called by an employee before it sets a data bundle ready for the HoloLens. The HoloLens itself offload the calculated data to its memory and does the 3D rendering and all other actions.

Unity related stuff

Another thing which can help you out is the configuration settings of your Unity project. There are a lot of configuration settings which can help to improve performance and stability. For this i have created a configuration list to make it a little bit easier.

• Quality settings of the project – A lot of user manuals tells you to set your quality level of a HoloLens application to “very low”. But that is ridiculous. Yes.. the lower the less quality, the less GPU and HPU needed to draw holograms. But it is better to check per application what is needed and how high you can go with the quality level. It is a balance between quality and performance which you have to decide what works the best for you. So keep it as high as possible while keeping the performance as high as possible and quality suits the best for your type of application.
• Light / Directional Light – Unity allows you to add light to your scene. It is additionally added to your scene as Directional Light. Light or directional light is in most cases with HoloLens applications not needed. Light causes to have a heavier scene to display in your HoloLens. Most shaders take care of the lightning in the scene and have impact on the calculations made to display graphical information like holograms.
• HoloLens specific shaders – Shaders are used in the process of drawing materials on GameObjects in Unity.  It does some actions on a material which is drawn on the 3D model. The way how it is drawn is determined by the shader. More information here. See it like you use light in your scene. The default shader takes into account all light sources and calculates the lighting on your 3D models. You can imagine that in some cases this is far too much you need in your app and cost a lot of memory and time to redraw the 3D model. The Mixed Reality Toolkit has a number of shaders written specifically for HoloLens applications. Or you could write your own. Keep in mind that writing shaders is not that easy. Check out the following article at Unity documentation regarding performance of built-in shaders.
• Vuforia – It allows you to use 2D or 3D recognition of objects and provide you the exact 3D coordinates in space. This allows you to project cool holograms in top of markers, images and even 3D scanned objects in real-time. The trouble with Vuforia is, that the engine is heavy. So you have to determine if you really need to use Vuforia in your application. Secondly you need to think about when to use it. Vuforia is enabled through a checkbox in the player settings which enables it for all the scenes in your project. So you need to think when you need it and turn the system on/off depending on where you are in the application.
• Code in your Update routine – This is actually a no-brainer for Unity developers. But not all HoloLens developers are Unity developers. The update method is called every frame (around 30 times per second) per enable game object in your scene. Code running in your Update needs to be fast and simple. Think about the concept of your application and what kind of objects it is going to use. Than think about what you need to do during real-time execution and keep that to a bare minimum.
• Non-movable game objects – If you are working with game objects which do not move in the scene, it would be wise to define them statically. The option is at the top of the inspector window for each game object in the scene.
• Textures – If you are using textures it will have influence on the performance of your application. Each texture exists of pixels which are drawn all separately. If your textures are not that detailed try to use the setting to 1/8 Texture resolution.
• Colliders – The default game objects like cubes, spheres and others have colliders by default added to the game object. Often imported game objects have the same. Think about if you really need to use the collider. Do you need to communicate with that game object or not? If possible remove the colliders which will have in its turn less impact on the process/flow of the applications performance
• Optimization settings – When you go to player settings of your project there is a tab called “Other settings”. Part of it is called optimization and has a lot of settings which can optimize your application.  Think of shaders which keeps loaded during your app execution. Or the ability to optimize your mesh data. Try to play with these settings if it fits your application.
• Clipping planes – One of the settings of the camera in Unity is the “Clipping planes” setting. This setting defines what of the scene you are seeing or not. You have the ability to adjust the “Near” and the “Far” distance in meters. Your application will not show/render any game objects which are outside these parameters. Keep in mind that objects which are not drawn are still active. Meaning that code written around those objects is still active. You could also decide to write your own code which takes care of activating/deactivating game objects in your scene based on the viewing distance.
• Level of detail – When a GameObject in the scene is a long way from the camera, the amount of detail that can be seen on it is greatly reduced. However, the same number of triangles will be used to render the object, even though the detail will not be noticed. An optimisation technique called Level Of Detail (LOD) rendering allows you to reduce the number of triangles rendered for an object as its distance from the camera increases. More about this topic.

Application build

The standard process for building HoloLens applications is generating a Visual Studio solution from the Unity build dialog. When you start the project from the “app” folder in Visual Studio it opens 3 projects as part of your solution. One of them contains your code, the second is the firstpass of the solution taking for example plugins into account. The third is the actual UWP solution which is deployed to your HoloLens device. When you build a package or deploys your solution directly to a device you have actually 3 options. Most people only uses debug or release. But there is a third option called master. The difference is explained below:

• Debug – Speaks for itself and creates a debug version on your device and allows you to debug your code by attaching it or use remote debugging.
• Release – This build will optimize your code for better performance. Yes! it seriously does. We have seen a difference in FPS of 20 between debug and release.
• Master – Master is normally only used when you want to submit your app to the store. It does all the things what release does including the profiler support is stripped out. Making your app even quicker.

So even if you do not want to deploy your app to the store, it could be interesting to see how it performance when building a master deployment. Keep in mind that it will take some time to build. And.. be aware… there are options and configurations which are allowed to use in your app, but are not allowed when building the app for the store. So far as I know in those cases the app will not build when Master is selected.

Memory usage

Building a HoloLens application will give you a maximum of 900 MB memory which can be used. Be careful what you are using. The more memory you use will influence the performance and stability of your application. When building application which use a lot of memory think about what you really need to have in memory. Offload (storage on the device) your content. Is there a internet connection? Think about getting your content in pieces from an online service. Maybe zip your content to storage and only load the content you need at the moment. be careful with coding which uses (even for a short time) a lot of memory for storing temporarily content. Don’t under estimate memory usage. Keep it as low as possible.

Other references

There are a lot of good posts about performance and stability regarding the use of Unity and in some cases in combination with a HoloLens device. A lot of the things you can do are not even mentioned by me. So i created a list which is interesting to read.

• Optimizing graphics rendering in Unity – Good performance is critical to the success of many games. Below are some simple guidelines for maximizing the speed of your game’s rendering
• How To Plan Optimizations with Unity – Unity provides a number of tools and settings to help make games perform smoothly. For this project, we chose ones we thought could prove to be troublesome and analyzed how they affected game performance on Intel® graphics processors.
• Performance recommendations for Unity – For the most part, the general performance recommendations for HoloLens apps apply to Unity as well, however there are a few Unity specific things you can do as well.
• Maximizing Your Unity Game’s Performance – A game’s performance can make or break the experience. Playing a game that is properly optimized feels smooth and responsive, and it will have a larger audience since it can run on lower-end platforms as well.
• HoloLens–The Path to 60fps – Just a great blog post about performance for HoloLens applications. A must read!!
• Hologram stability – To achieve stable holograms, HoloLens has a built-in image stabilization pipeline. The stabilization pipeline works automatically in the background, so there are no extra steps required to enable it. However, developers should exercise techniques that improve hologram stability and avoid scenarios that reduce stability.
• Performance recommendations for HoloLens apps – HoloLens is capable of presenting an immersive holographic experience to the user without being tethered to a PC, phone or external cameras. To accomplish this, HoloLens has more compute power than the average laptop with passive cooling. The passive cooling means that there is no fan and no noise that disturbs the user’s experience.

Conclusion

These are just a few things you need to consider when building a Mixed Reality/ HoloLens application. They will help you in some way to optimize your application to let it run as smooth at possible.

A thank you to one of my colleague Martijn Takken for mentioning some of the performance gains and issues.

I presume that i will be updating this post frequently and if you have some additional tips around performance, please let me know and i will add them to the list.