'Feels Like Paper!': An intuitive AI interface that uses paper



Unlike computers, paper is said to be good for reading books and leaving simple notes and doodles. Designer

Lucas Morrow has developed ' Feels like Paper! ', which allows you to write directly on paper and the AI will perform various conversions.

'Feels Like Paper!' · Interfacing Artificial Intelligence through Paper
https://lukasmoro.com/paper

'Feels like Paper!' is a series consisting of three prototypes: 'Maths & Questions,' 'Mark & Comment,' and 'Draw & Dream.'

◆Maths & Questions
Maths & Questions is an application that allows users to write mathematical expressions on paper while wearing a mixed reality (MR) headset, and the application displays the results. The application uses a wrapper using Apple's Vision Framework API to automatically recognize equal signs and question marks, and then transmits the read mathematical expressions to a large-scale language model to calculate the results.




All of the pseudo-characters were handwritten by Morrow on a computer in advance, and Morrow said, 'Each number and letter is animated as if it were being written by a friendly ghost.'



◆Mark & Comment
'Mark & Comment' allows users to highlight text on paper and have the same text highlighted on the computer in real time.




In addition, if you hold the paper with your pinky finger and speak, your voice will be automatically recorded and entered into the transcription AI Whisper. The transcribed comment will automatically be displayed above the highlight.



In addition, the Mark & Comment process converts the text into a string of character codes using an OpenCV optical character recognition pipeline, and then forwards the prompt to GPT-4o: 'If highlighted text appears, please answer with text only. Otherwise, return an empty answer.' As a result, it becomes possible to recognize only the highlighted text and digitally highlight it.



◆Draw & Dream
'Draw & Dream' is an application that converts the act of drawing on paper into another video in real time. The way it works is that the user draws on paper and automatically inputs it into the image generation AI

StreamDiffusion , which generates an image according to the prompt. The video feed of this generated image is rendered in real time through the Unity application.




According to Morrow, he implemented an image processing pipeline on Python to feed the drawn image to StreamDiffusion. He also introduced OpenCV to detect the four sides of the paper and input the image to StreamDiffusion. In addition, by using a ' Brute-Force matcher ' that matches features between consecutive frames and homography that allows you to always see the image from a bird's-eye view, it is possible to properly input even if the paper is partially obscured by a hand, pen, or brush.



Speaking about these prototypes, Morrow said: 'These three approaches frame the relationship between humans and AI in different ways. Maths & Questions proposes integrating or 'merging' with AI to complement our own intelligence. Mark & Comment harnesses the capabilities of AI to use it as an assistant to understand the world. Draw & Dream uses generative AI not just as a tool, but to help you create your own work and provide interactions that further inspire your creativity.'

Related Posts:

in Software, Posted by log1r_ut