Challenges in Creating Mobile AR Challenges in Creating Mobile AR

By Penjami Rantakangas


The tools for creating a Mobile AR application are more advanced than ever before, but developing for AR still involves many challenges. This article is an overview of the major challenges I faced while creating a modern mobile AR application.

Developing for AR is slightly different from other software development, because there is a bunch of different things, not strictly related to programming, to take into account. Creating AR is as much about the surrounding environment as it is about the code in the software.

1. Maintaining Immersion

Immersion is a huge part of creating a successful and visually pleasing Augmented Reality experience. Everything from the lighting conditions to the detected object or image can have an effect on how immersive the application is. Most Augmented Reality SDKs use a combination of the frames from the device camera and the gyro and accelerometer sensors to determine the movement and location of the device in relation to the real world. Dynamic environments are usually bad for AR, because they have a lot of moving parts, which is hard to track. Most of the time the best environment for AR is inside, because there you can affect every detail to maximize immersion. You can set up the lighting and the wall textures so that everything works, even on devices that don’t have the latest sensors.

2. Optimizing for Mobile AR

Optimizing for mobile devices is a huge challenge on its own. Tracking the environment takes up a large chunk of the processing power needed to render the Augmented Reality scene. For an immersive AR experience you need the framerate of your application to be as close to 60 as possible. You rarely get one big break when optimizing software, usually optimizing consists of changing a bunch of small things to help performance. it’s important to first check the usual suspects: complex materials, high polygon count 3d-models, huge textures, demanding post-processing effects like anti-aliasing and bloom.

Particles are a very easy way to add complex effects to a project, but they come with a performance cost. Usually under a thousand simple particles doesn’t affect the performance that much, but more than that can significantly affect performance. Sometimes it’s better to substitute particle effects with different ways of showing a similiar effect. One such way is to use models and textures with shaders instead.

AR Demo Image The image above is from an in house made AR application, that showcases Intopalo Digital’s expertise in different fields of technology.

3. Dealing with Scaling

One unit in an Augmented Reality coordinate system equals one meter in the real world. So bigger scenes need to be scaled down, but scaling objects too much leads to some of the components not working or behaving in strange ways. In Unity there are a lot of components that don’t work with scaling, for example: Particlesystems, Physics components and navigation components. This can be avoided by scaling the camera that is used to observe the content and then moving the camera so that it looks like the content is being scaled down even though in reality only the virtual camera rendering the AR content is moving/scaling.

4. Creating Convincing Interactability

Interactability is something that enhances an AR experience by more than you would think. Adding some interactability to your AR project gives the user a new dimension to work with. The challenge is how to add meaningful and non-disruptive types of interactability. The obvious way of communicating with a Mobile device is the touch screen. Touch screen can be used to trigger events or move objects inside the AR application. But the majority of mobile devices have a bunch of sensors that you can also use to create interactability. You can use the position in relation to the content in the real world to trigger events. You can also use the camera to procedurally modify the AR content based on the colors currently on screen. AR in mobile devices is still fresh from the oven, and all the interactability you can achieve hasn’t been figured out.

Final Thoughts

In the future you might not need to worry about performance or immersion because they are overcome by technological advancements, but today it’s best to make sure that the mobile AR application you create is the best it can be, by not overlooking the challenges mentioned in this article.