Startup uses artificial intelligence with mechanism to generate required code when reading screenshot of GUI

ByGwydion M Williams

It is Denmark's IT system startupUIzard TechnologiesDeveloped an application that uses Artificial Intelligence (AI) to read screenshots of the graphical user interface (GUI), and outputs the necessary code (storyboard file and HTML file) for that GUI. The demonstration video on YouTube is released.

Pix2code - YouTube


"Pix2code" generates code from the GUI screenshot.


First we prepared GUI for mobile terminal. It is just a screen shot of a PNG image.


This is loaded into pix2code. Pix2code seems to be able to choose the target platform from "iOS", "Android", "Web", I chose iOS here.


Since the movie is recorded in real time, it takes time to wait for processing after loading ......


Within 15 seconds the files "mobile-gui-demo.gui" and "mobile-gui-demo.storyboard" were generated. The storyboard file is used for application development for iOS.


When I opened the file, the same GUI that I saw in the screenshot was completed.


The position of the slider and the contents of the character string allocated to the menu can also be edited here.


For web-based services the way is almost the same. Screenshot to load this.


Pix2code Platform for "Web" as option at run time.


This took about 30 seconds. Now the file "web-gui-demo.gui" "web-gui-demo.html" has been completed.


When opening the HTML file in the browser, the screen shot and the contents which were not different from the previous one were completed.


According to UIzard Technologies, the accuracy of making a GUI from screenshots is currently around 77%. It is said that accuracy will improve by further learning. The dataset used is published on GitHub, and it is planned to release all code of pix2code within 2017.

GitHub - tonybeltramelli / pix2code: pix2code: Generating Code from a Graphical User Interface Screenshot
https://github.com/tonybeltramelli/pix2code