CS491 - Virtual and Augmented Reality

Student's Choice Presentation

Sketch AR | Art with Mixed Reality

SketchAR is an application that applied Augmented Reality to create art works in real world. When the user is using the device, he will be able to draw images, just like tracing. The image outline is projected on the selected space, which will be used as the canvas. This app is a very simple but effective usage of augmented reality. While this can be used as a fun application, it can also help people to create high quality copies of existing art and also reproduce them easily on a massive scale. This application is also available for other platforms such as iOS, Android, etc. But among all these, Hololens seems to be the most suitable place to use this app.

How it Works?

  • The application scans the surface on which the image is to be drawn.
  • Once the surface is scanned, the image that is to be drawn is selected.
  • The image is then resized according to the user.
  • User can then start drawing.

Supported Platforms

Mobile - iOS and Android

SketchAR is available for both iOS and Android mobile platforms. In both these platform, the user will be able to scan a proper A4 or A5 sheet of paper as a drawing surface on which the image is superimposed in the mobile screen. With this the user will be able to sketch the image on the paper.

The current version of the app for iOS does not support projecting images on large surfaces such as walls etc. But this limitation will be resolved in the next update since it will be based on the new AR Kit.

Android with Project Tango

Link to Project Tango

In Android, the application is also available for devices that support Google's Project Tango. Tango is an augmented reality platform that was developed by Google. Tango employs computer vision to detect the device's position relative to the world around it. With Tango, developers were able to create indoor navigation, 3D mapping applications etc. But this has already been retired and Google's ARCore has taken its place. Project Tango was implemented only in 2 devices - Lonovo Phab 2 Pro and Asus Zenfone AR. SketchAR for Tango devices helps the user to project the image on any large surface, including walls etc. This can be seen in the video below.

Hololens

SketchAR is an ideal application for use with Hololens. With Hololens, the image to be drawn can be projected on any surface and to any scale. It also allows the creator of the art to move freely without having to change the position of the image that is projected. With SketchAR for Hololens, walls can be used as canvas and huge graffiti street arts can also be created easily. It also allows replication for complex artworks easily.

Spatial Awareness

Spatial Awareness is the basis of SketchAR. Both Android's Project Tango and Microsoft's Holographic Computing with Hololens stand out in their implementation of Spatial Awareness. In face, Spatial Awareness is the USP of both these technologies. Spatial Awareness includes motion tracking, Depth sensing, area learning, surface reconstructing, plane and void detection, etc. Tango and MS Holographic computing were platforms that were built completely to implement spatial awareness while apple devices started to include support for spatial awareness from their iPhone 7 onwards.

SketchAR and Spatial Awareness

SketchAR, though a simple application still utilizes spatial awareness features. Most of the work goes into identifying the environment with its spatial data. Then when the user selects the image to be displayed, it remembers the location in the environment chosen by the user to render the image. Different platforms have implemented this feature in a different way.

Real Usecases

As already reiterated many times, the main application of SketchAR app is creating real work artworks of different scales. Depending on the device in which SketchAR is used, the scale of the artwork can change. With iOS and normal Android devices, recreational artwork can be created. The SketchAR app allows you to select any image you want from the ones available in the system and returns it as a sketch which will be projected on the required canvas space. So with the help of iOS and Android Devices, it will be easier to create notbook sized artwork.

With the help of Hololens or any other HMD, the real use case of this app would be to create wall art. This wall art can be both street art and interior designs. It can be utilized commercially by design companies to effectively create drawings, where the company can reduce the cost invested in high skilled labor. When people want to create wall art in their living rooms, they don't have to hire a professional anymore because it will become much simpler to trace your favorite art on the wall by yourselves.

Hololens vs Mobile Devices

With a HMD such as Hololens, there is no distraction for the user. The user can also use both the hands effectively. In case of mobile devices, in most cases, the user has to hold them in one hand or has to place them in a fixed location, which he has to look at from time to time while he is drawing. This ability makes hololens the natural medium of use for the SketchAR app.

Limitations

With Mobile Applications for iOS and Android, there arises a need for extra support for the device to be stable in order to use it for serious art. This negates the purpose of mobile technology as it gets fixed to a location.

Also, if it is handheld, the device becomes shaky and getting the image to be fixed in a place becomes difficult. When tracing artwork in this setup, the output is often bad.

Hololens has the problem of the holograms having very poor brightness. This is due to light of the room or the environment interfering with the rendering. It is still to be tested how this works in an actual Hololens, which can be tested anytime with the Hololens at the Lab.

One other limitation will be the real world problem with paints and materials which the application has no control over.

Similar Applications

Sources: