AR and AI: How to understand furnishing styles from images and present meaningful AR visualizations

Christian Nietner, Founder and CTO of the company RoomAR, gave his presentation “Understanding furnishing styles from images” at the Data Festival 2018. He provided insights into the use of AR technology in an e-commerce context and explained how AI, especially neural networks, are used to enable image recognition and meaningful recommendations based on images.

How things got started or how to use AR to enhance furnishing

When Christian Nietner and his Co-Founder and CEO Naimah Schütter were furnishing, they realized how difficult it is to actually imagine how furniture looks like in one’s own room. Coming from a tech background, Christian Nietner thought about how this issue might be solved using techniques such as machine learning and computer vision.

The emergence of AR technology

Christian Nietner’s idea coincided with the emergence of AR (augmented reality) technology, that had reached a new level of profoundness. All the big players in the digital world, such as Google, Apple, and others, are developing their own AR technology and devices.

AR technology is about to reach everyday life. By 2020, 4.3 billion smartphones will be running AR and the market size for this technology is expected to multiply (almost) 6-fold in the short, five-year time window between 2018 and 2023. This rapid development is not only driven by the availability of devices, but also by the e-commerce industry that adopted this trend early on.

In order to capitalize on this development in AR, RoomAR was founded to become one of the leading tech companies to empower AR commerce. The company’s purpose is to provide a solution for the furniture industry to accelerate sales and increase efficiency.

However, in order to enrich the real-world environment with virtual images, you also need to understand that environment. That is where AI enters the game: image processing is crucial to understand the environment that is intended to be augmented.

The development of RoomAR

The company was founded in fall 2017, supported by funding. In the first steps, a business model needed to be developed and a demo app was built. Soon after, the company acquired its first investor, which enabled the company to hire four employees in the fields of data science, business, and app development. By 2018, the company had launched its first product – a smart AR product for interior products. During the same time period, another investor joined the business.

How to benefit from AR technology in the furnishing industry (and other e-commerce companies)

Christian Nietner is convinced that AR is going to change the way in which products are purchased, especially in the furniture business. The use of AR (in the furniture industry) provides a number of benefits:

  • acceleration of sales, both online and offline
  • an innovative touch
  • brand experience is scaled by technology
  • new target groups can be addressed

The furniture industry can harness all these benefits by integrating AR into their product to provide recommendations.

Two major steps are to be taken to enrich user experience with AR and AI:

  • Visualization with AR enriches the real world with digital content. However, at the same time, it also represents a new data source – information can be obtained from interactions, the product choices people make, or the customer’s configuration of furniture. This data can be used in various ways.
  • Personalization with AI by using vast amounts of data in order to enhance the experience for customers on one hand and to gain insights for companies themselves on the other hand.

How the AR experience can be personalized using AI techniques

By using AI methods in an AR context, RoomAR’s main objective is to apply machine learning to understand furnishing styles from images (à image recognition) and thereby make appropriate and meaningful product recommendations.

The main challenge within that goal is the question of how to match pictures from the video screen with products from the database.

Pictures as a basis for recommendations

A large number of approaches can be used to realize the idea of using pictures as a basis for recommendations. They range from methods such as using deep learning-based large-scale visual recommendations, deep depth completion of a single RGB-D image, ScanNet, and many more.

However, Christian Nietner preferred to begin with the simplest idea possible and then expand on it by iteration.

RoomAR’s approach to using pictures for recommendations

RoomAR’s approach can be illustrated in four major steps:

  • Build up a labeled database of interior images.

The company’s (room) pictures need to be matched with suitable products. The data sources for this step are pictures and products – therefore, a furniture catalogue can be used as initial inspiration for how to combine these images with products. The starting point is to build a database with images that are related to products or styles.

In order to achieve this, more detailed information is necessary to classify products and images by room type, style, or colors, for example. On that basis, products can be matched by color, for instance. Besides these classifiers and products, the database contains objects such as sideboards or armchairs.

  • Train neural networks for feature extraction.

Deep neural networks learn hierarchical feature representations. The deep neural network consists of an input layer, several hidden layers, and an output layer. When a picture is transferred through such a network, the first step is feature extraction where different representations are produced by several hidden layers.

In the next step, a feature vector contains different aspects that describe the picture’s characteristics. These characteristics are then classified in order to identify the room type or style, for example. This step enables meaningful and specific classification. However, a lot of data is necessary in this step, therefore RoomAR uses transfer learning. Transfer learning describes a machine learning method in which the results of an already trained neural network are applied to a new task.

Therefore, RoomAR processes images through a pre-trained neural network that has very general feature extractions in order to produce a feature vector.

  • Define distance metrics for similar images.

In this step, images are compared to visualizations of the actual environment.. Image clustering is one method that can be used in this step: similar images can be identified using distance metrics in the feature space.

In the recommendation process, the input of the user’s room image is identified, the database is checked for similar images, and suitable products are then recommended.

  • Improve via image triplets and hinge gloss

Christian Nietner’s key learnings and takeaways

Christian Nietner identifies several key learnings for the use of AR technology:

  • Since AR technology has matured, it is now enterprise-ready and can be used to address the mobile mass market
  • AR offers huge potential for brand experience and the acceleration of sales
  • AR allows to enrich the real world with digital content
  • AR also provides access to new and insightful visual data sources
  • Decades of computer vision research results can be exploited to leverage the potential of AR across industries

 

Want to learn more about AR and its use in the furniture industry? Check out our full-length video of Christian Nietner’s presentation here.