Microsoft Interns Built An App That Turns Napkin Sketches Into UI

Entertainment By Elena Boaghi |

Even in an era when digital proficiency is higher than ever, there’s still a gap between concept and product. And Microsoft dreams of eliminating that gap entirely with an experimental Windows app called Ink to Code. It lets you literally sketch out your app design–complete with buttons, text fields, and headers–then export a machine-readable layout file to Microsoft’s Visual Studio, which allows you to add the code for either a quick and dirty prototype or fully fledged Windows and Android apps.

The app itself was born as an intern project at Microsoft Garage, where the company rapidly prototypes ideas that may or may not be brought to market. Even at the Garage, ideas are typically born the old-fashioned way. “We have all these app ideas sketched out on paper,” says George Matthews, who managed Ink to Code. “Every time one of our teams will build out a feature, the first thing we do is we draw it by hand, before we even touch a computer. For me it’s natural, it feels more natural. [But] we thought we could do something better.”

Watching Matthews demo the app himself, it’s clear that Microsoft is on its way to that amorphous “better.” Using a Surface Studio, Matthews quickly draws up a tipping app, hastily sketching UI elements that transform into crisp boxes and buttons. What I don’t notice at first, though, is that while the software is using AI to recognize his intent, he’s also drawing in a sort of language. A box with many lines in it is for text. A box with a circle is a button. A box with a circle is an image.

“We got inspiration from looking at app designer notebooks–we asked them to show us their notebooks,” Matthews says with a laugh. As it turned out, many designers already sketch their apps with these visual shorthands, so it made sense for Microsoft to build them right into the app.

Especially for professionals that already use Microsoft Visual Studio in their work, Ink to Code might save a step or two in the early development process, getting them to a working prototype faster. Yet it’s neither turnkey nor foolproof. While it helps you generate an annotated layout for your app, you still have to connect all those placeholder buttons and images with real code inside another editor. So for now, Ink to Code isn’t the dream app that the first type of people in this world–the non-coding, non-designing people–need to realize their visions. Though keep in mind, it’s not being billed as that. It’s a free experiment you can try. Microsoft doesn’t even call it a “product” at this time.

Even still, talking to the team behind Ink to Code, it’s clear that Microsoft would like to one day build a full end-to-end system for app creation that could be as simple as sketching. “I’ll admit to being a little bit obsessed with this idea right now. I care very deeply about it,” says Matthews. “Much to the chagrin of my family at home, I say, ‘I have a great idea of what we can add to it!’”

Leave a Reply

Your email address will not be published. Required fields are marked *

fifteen − 6 =