Augmented reality as a technology is no longer a novelty. It has been around for quite a while and is rapidly developing. Its main application area is the development of applications and games for mobile devices and AR-glasses. The technology of augmented reality became more accessible with the appearance of tools that make it easier to work with, such as Google’s ARCore and Apple’s ARKit. However, despite the massive interest in it and the support of major companies, there are still problems in the development of augmented reality applications, both physical and technological. The most serious of them are:
1. Determining the camera angle
2. Recognizing physical objects
3. Identifying plane boundaries and occlusion
4. Lack of computing power
This article is going to be useful for both beginners interested in this technology and customers who want to use this technology in their projects.
If you are unfamiliar with the concept of augmented reality, it is, in short, the art of bringing virtual objects into the physical world. For example, you can play billiards in a nearby park, or kill a whole bunch of ghosts in the neighborhood, or measure the floor space of your room, using only your phone’s camera. What difficulties can we expect in the development of such applications? Let’s see.
Problem 1: determining the camera angle.
The picture shows the scene the camera is pointing at, and on top of it, there is a virtual object. Seems easy, right? However, it’s easy only at first glance. What is actually going on here?
The application defines the plane on which the virtual object should appear and correctly positions it in this plane, taking into account the angle from which you are looking at it. And here we encounter our first problem: the camera does not provide information about its location, that is, the application does not understand from which side you are looking at a virtual object, by how many degrees the camera is turned, and what part of the object has to be shown at the moment.
There are some algorithms that deal with this problem. For example, you can place an object on a mark, like in image 1. In this case, it is possible to find special points of the mark to understand how to change the camera position. A point can be called special if it has a descriptor that distinguishes it from many other points. Image 2 shows an example of such points.
You can also use surfaces instead of marks. Algorithms that recognize surfaces through specific points are currently available. These algorithms help determine the position of the camera against the plane using additional information from the phone’s gyroscope and accelerometer sensors. However, these algorithms are not designed to scan all types of surfaces. The texture of the surface must not be monotonous, transparent, and must not shine. There is another significant limitation of ARCore and ARKit technologies: all physical objects must be static, otherwise they are ignored.
Problem 2: Recognition of physical objects.
Let’s look at another example. You want to scan an object, let’s say, an armchair. Once the scanning is complete, the app determines that the item is an armchair, and it also determines your location against the seat. After that, the application can place a virtual object on it, for example, an animated character. What difficulties do we have here? The main problem is that there are many armchairs in the world, and it is impossible to choose one standard armchair which would match all other armchairs. How can we identify things, the appearance of which is very different from each other? In this case, modern artificial intelligence algorithms come in handy. For example, you can scan many armchairs of various types, and over time, the algorithm begins to distinguish them from other things with fairly good accuracy.
Problem 3: Identifying plane boundaries and occlusion.
Your camera shows a 2D image, and the algorithms can only recognize some key points in that image. However, to define the boundaries of objects and the location of physical and virtual objects relative to each other is an extremely complicated task. For example, if you put a virtual object behind a physical object, a part of it should not be displayed. It must be hidden behind a physical object, not created on top of it. This is called occlusion, a situation where a part of a physical object overlaps a virtual object. Different startups are trying to find solutions to this problem. Here is the most successful of them.
Problem 4: Lack of computing power.
This problem exists because augmented reality is most commonly used in mobile applications. Smartphones have to be compact, and the smaller the device, the less power it has. Calculations to solve the tasks mentioned above require massive resources, which imposes limitations on the development process. Here is a list of some devices that support ARKit technology:
● iPhone SE
● iPhone 6s
● iPhone 6s Plus
● iPhone 7
● iPhone 7 Plus and above
● iPad Pro (9.7, 10.5 and 12.9 inches)
● iPad (2017)
You can find the devices with ARCore support on the official website of this technology.
The issues discussed in this article are only a small part of the problems of augmented reality. Nevertheless, technological development does not stand still. It is likely that soon the issues discussed above will be solved, and this technology will become part of our daily lives.
