I’m in a strangely lit subterranean room in Kyoto, Japan, and for the sake of the experiment in which I am participating, I’m pretending to be lost. A large “Mall Map” is mounted on a wall in front of me. I move toward it at a leisurely pace, in the manner of a man trying hard not to draw attention to himself. When I reach the map, I stop. A whirring sound of gears moving in a motor rises up behind me. I hear the thrush of wheels passing quickly over carpet. Something mechanical this way comes.



“Konichiwa,” says a cheery voice that sounds like it’s emerged from a Casio keyboard. I recognize the greeting: “Good afternoon.” I turn and see two enormous black eyes staring up at me. “My name is Robovie,” says the robot in Japanese. “I am here to help. Are you lost?”

“Yes,” I answer.

Robovie swivels on his omni-directional wheelbase, extends his motorized arm to the left corner of the room, and points to a large sheet of paper displaying the word “Shoes.”

“May I recommend the shoe store?” Robovie asks. “It’s right over there.”

“Dōmo arigatō,” I tell the robot. (“Thank you very much.”) It bows and wheels slowly backward. The experiment concludes successfully.

Welcome to Japan, which has been one of the world’s most important centers for robotics research since Ichiro Kato unveiled his WAP-1, a bipedal walking robot, at Waseda University in 1969. Fifteen years later, Kato presented a robot that could read sheet music and play an electric organ.

Robovie may seem like a step back compared with an assemblage of metal and wire that can sit down and coerce Kitaro’s “Silk Road” from a keyboard without missing a note. But Robovie, in fact, is far more human than his most impressive predecessors. In ways that are subtle but nonetheless significant, he represents an important turning point in the field. He’s also a moving, talking poster boy for all that is wonderful about Japanese robotic research. The future of human–machine interaction can be found in Robovie’s dark, watchful eyes.

Japan: Robot Central

MIT-trained robotics engineer Dylan Glas is one of Robovie’s very many chaperones. He’s lived in Japan for eight years now, and this has given him a uniquely international perspective on robotics culture. He also represents a reverse brain drain. He holds multiple degrees from the most prestigious technical school in the United States, but he left his country of birth to pursue better research opportunities abroad. Glas says that the allure of Japan wasn’t financial. He had plenty of offers to design robots in the United States. The problem, as he explains it, was that he didn’t want to build war machines.

A participant in MIT’s Middle East Education Through Technology program, Glas worked teaching JAVA programming to Israeli and Palestinian high-school students in Jerusalem in 2004. The experience was instructive.

“I saw how people who are parts of larger warring groups can form friendships,” says Glas. “So I came straight from trying to make peace to looking at building things that killed people.” He explored other research opportunities online and found a picture of Robovie (an earlier iteration) hugging a little girl. He knew at that moment he was moving to Japan.

The United States and Japan lead the world in robotics research. But the two countries are dramatically apart on what they’re building these bots to do. The United States, which spends more on its military than do the next 45 spenders combined, has devoted most of its robotics research funding, on a national scale, to putting machines in dangerous battlefield situations, deep behind enemy lines, over the mountains of Afghanistan and Pakistan in the place of humans. The goal is not so much to replace the human soldier but to automate the deadliest parts of the job so the soldier becomes more technician, less target. The iRobot Corporation, the most successful private robot manufacturer in the United States, didn’t get its start building Roomba vacuum cleaners but designing military machines like the PackBot.



Japan is looking to fill a very different need. Demographically, it’s the oldest country in the world; nearly 20% of the population is older than 65. In the rural countryside, the proportion is closer to 25%. Japan is also shrinking. The number of children under age 15 reached a record low of 16.9 million in 2010. Many of Japan’s best-known robotics research projects, such as Asimo, indirectly address the rising population of seniors and growing dearth of able-bodied workers.

Meeting Social Challenges

Many Japanese argue that the country could address its demographic challenges through policy, such as allowing more willing immigrants into the country. There’s evidence to suggest that a more relaxed immigration policy would benefit Japan economically. But immigrants here face social and even linguistic barriers to real integration. Japanese is a tough language to learn; rules and usage can vary tremendously from prefecture to prefecture, between superiors and subordinates, between waiters and restaurant goers, and even between men and women. Linguistic and social customs can be very important to older Japanese, even if hip and media-savvy kids in Tokyo don’t think much of these cultural norms.

Formality and routine are particularly important in work settings, as anyone who has lived in Japan can testify. The degree of professionalism, focus, and seriousness that people bring to even menial jobs is impressive. This is not a country where you encounter baristas texting while they’re making lattes. That emphasis on completing tasks in a very specific “right” way contributes to greater acceptance of automation, says Glas.

“At work, there is no deviation from the established best practice,” he notes. “When I go to the supermarket, they always say exactly the same thing and deliver customer service exactly the same way. So I think the idea of robots doing that sort of work is very natural.”

All of these factors—aging and decreasing population, lack of immigrant labor, electronics expertise, available research funding, and cultural openness to automation—make Japan the key destination for humanoid robotic research, the study of how humans and robots interact in casual, civilian settings.

The Intelligent Robotics and Communication Laboratories, where Glas works, puts people and robots together in interesting settings. Their field tests offer a snapshot of a future where humans and machines work and play side by side. One of Glas’s favorite experiments involved a teleoperated robot who served as a tour guide in the town of Nara, Japan’s imperial capital some 900 years ago and home to some of the most important Buddhist temples in the world.

Touring Nara is more fun with the right guide, someone who has spent awhile learning the history and who knows a secret or two about the place (such as where to find the sacred herd of deer that eat directly from your hand). But the average age of a tour guide here is 74. Therein lies the problem. The walk from the train station to the best sites, like the famous giant bronze Buddha, can be challenging for young bodies, let alone someone in her 70s. Glas and his colleagues saw an opportunity to put a remote-controlled robot (as opposed to a fully autonomous one) in a unique setting to serve as the voice, eyes, and ears of a real person.

“Having this robot there helps [the guides] be there from home, so they can still talk and share their enthusiasm for Nara and the history of Nara,” Glas notes. “When I tell people this, a lot of Americans say, with a blasé shrug, ‘interesting.’ Japanese people light up and say, ‘Oh, we really need that!’ The perception of necessity is very different. That’s a cultural difference that guides the way people perceive how robots should be in society.”

My conversation with Robovie reenacts another field test that took place in an Osaka shopping mall in 2008 and 2009. The goal in that situation was not so much to empower people through telerobotics as to instruct robots how to interact with humans. The setting was a strip of stores by the entrance to Universal Studios Japan. Robovie had a 20-meter turf, sandwiched between a collection of clothing and accessory boutiques and a balcony. The first challenge was learning to distinguish between people who were passing through the area in a hurry from those who were just window-shopping or who were lost. The second group might be open to a little conversation; the first group represented a hazard.

The mall test is a classic example of the sort of pattern-recognition task that humans are great at, but robots just don’t do; there are too many open questions. How do you explain human states like “in a hurry,” “window-shopping,” and “lost” in binary code, a language that the robot can understand?



The researchers outfitted the area with six sensors (SICK LMS-200 laser range finders) and, over the course of one week, collected 11,063 samples of people walking, running, and window-shopping. They analyzed the data in terms of each mall goer’s velocity and direction, and isolated four distinct behaviors: fast walkers, idle walkers, wanderers, and people who were stopped or stuck looking at a map. These classifications helped Robovie learn how to recognize different types of people based on their behavior.

Next, Robovie had to say something to the folks he chose to converse with, and the back-and-forth had to seem fluid and natural to the human participant. You would assume that teaching a robot to make chitchat would be a snap after all the time humans have spent over the last decade talking to computerized agents over the phone. But in a real-world setting, the interaction is a lot harder for the machine to handle gracefully. “People think [computerized] speech recognition is so great,” says Glas. “It is, if you have a mic next to your mouth. But if you have a mic that’s far away or on a noisy robot, or there’s music in the background and a group of three people is trying to talk to the robot at once, it’s not feasible.”

Robots have the same hearing problem that afflicts people with King–Kopetzky syndrome, also called auditory processing disorder. Picking up sound isn’t the issue. It’s distinguishing words and meaning from all the other background noises. The problem lies in the brain, which is where most of what we call hearing actually occurs.

To compensate for these deficiencies, the researchers made sure Robovie could call for backup. A human operator would monitor the exchanges from an isolated location, and if Robovie heard a word he didn’t recognize (or got himself lost in some corner of the mall), the operator could chime in and help the robot with the recognition problem. This cut down on the time it took the robot to respond to questions. Human–robotic interaction will likely proceed along these lines—partially human, partially robot—for the foreseeable future, says Glas.

“Even in big automated factories, you need a human. You always will,” Glas avers. “My goal is to increase the automation level, decrease the role of the operator, and work towards more automation. So instead of one person fully operating a telerobot, you have one person monitoring 400 robots; when one runs into a novel operation scenario, he calls the operator.”

The lab’s field tests have yielded a plethora of interesting and counterintuitive findings. For one thing, people trust robots that look like they just came out of the garage, with bolts and hinges exposed, more than they do bots concealed in slick plastic frames. Also, kids and adults interact with robots in very different ways. Adults kept asking Robovie for directions and treated him like a servant, while kids asked the robot’s age and favorite color.

These experiences are why Glas loves his job. They also reveal how the study of humanistic robots involves much more than sensors and hardware. It draws from psychology, anthropology, and a host of other so-called soft sciences. It makes use of intuition and observation in a way that formal robotics research under a military grant doesn’t. This, in part, explains why one of the most important figures in human–robotic interaction research is himself an artist.
The Oil Painter

In the myth of Pygmalion, a sculptor creates a female statue so convincing that the gods make her real. Japanese roboticist Hiroshi Ishiguro has never heard of Pygmalion, but shares the tragic hero’s obsession: creating a work of art so lifelike that—in the imaginations of those who behold her—she becomes real. Ishiguro is known internationally for his very particular robotic creations modeled after real people, including himself, his daughter, and various beautiful Japanese women. He’s also one of the senior fellows at the Intelligent Robotics and Communication Labs.

On a warm November day, I get to meet him at his office at Osaka University. My friend and I are shown into a large space decorated in modern furniture of plastic and glass. At the far end of the room, a man draped entirely in black and leather, and sporting an Elvis-like pompadour that extends from his forehead, is watching two different television monitors and smoking with a feverish intensity. He looks not so much like one of the most important figures in modern robotics as a Los Angeles record producer circa 1985.

Ishiguro began his university studies with a dream of becoming a visual artist. Computer science was a backup. Eventually, he was forced to give up on oil painting as a career. “I couldn’t get the right shade of green,” he says. Ishiguro has put his arts training to good use. It’s his artistic sensibility that informs his unique approach to robotic design. “Oil painting is the same thing [as building robots]; the meaning of oil painting is to re-create the self on canvas.”

Ishiguro believes that some understanding of humanity (and some formal humanities training) is essential to success as an engineer. “We need to start with the question, What is human?” he says, “and then design and build from there. Our brain function is for recognizing humans, but in engineering robots, we don’t study the importance of the human appearance.… We need to establish a new type of engineering. Appearance is key, then movement, then perception and conversation.”

He takes us across the hall to show us his lab. A mannequin-like figure is sitting erect on a metal stool. I ask if I can investigate, and he nods. I step hesitantly forward and poke the android in the cheek. Its eyes open wide, and it turns to stare in my direction. The mouth hangs slightly open in an expression of surprise.

The demonstration is simultaneously amazing and unnerving. Ishiguro admits that his creations have secured a reputation for creepiness. When his daughter met her android doppelgänger, she burst into tears. Like a true artist, Ishiguro says he’s thankful for every honest response he gets.



“People are surprisingly sensitive of the humanlike appearance. The older people accept this type of robot. I exhibited one at the World Expo. Elderly people came up and asked ‘Where is the robot?’ Young people knew right away.”

Ishiguro has recently turned his attention to the theater. Two years ago, he and Japanese playwright Oriza Hirata began the Robot-Human Theater project, an ongoing collaboration placing flesh-and-blood actors next to tele-operated androids and other robots. Last year, on a trip to Tokyo, I caught a performance of Hataraku Watashi (I, Worker), a staid, 30-minute piece exploring themes of mortality, autonomy, and what it means to be human. Hataraku Watashi also serves as a live experiment in human–robotic interaction. Takeo and Momoko, the play’s robotic stars, faced the same challenges as their human co-stars: line delivery, timing, blocking, and conveying emotion and meaning.

Ishiguro and Hirata’s most recent piece, Sayonara, starred actress Bryerly Long and Geminoid F, a female tele-operated android. Following the play’s debut last November, Long told Reuters that she felt “a bit of a distance” between herself and her co-star. “It’s not some kind of human presence,” she said.

Ishiguro expressed confidence that future performances will get better. “We think we need humans for theater, but this is not so,” he told me. “The android can change the theater itself.”

This assertion begs a question that is either philosophical or semantic depending on the answer: Can robots act?

The late Russian literary critic M. M. Bahtkin might say Yes, insisting that the success of any piece of art, including theatrical performance, rests entirely on the reaction it creates in the audience. By this view, Takeo, Momoko, and Geminoid F are already accomplished actors.

The late Lee Strasberg, founder of method acting, would argue otherwise. The robot has no internal memories, no painful or elating life episodes with which to breathe credibility into the performance. “The human being who acts is the human being who lives,” he said. The presence of life, ergo, is a necessary precondition to “acting.” The robot is a prop, or, in the case of a remotely controlled android, just a puppet. It’s an interesting gimmick but not a thespian. Real-life experience is a precursor to genuine acting, and a robot will never be able to experience life in the way that humans do.

Or will it?

The Pattern-Recognition Machine

Turn your attention back to Robovie for a moment. Picture him standing alone in his stretch of mall. A potential conversational partner enters his area of operation. Robovie has to make a decision: Is it safe to approach or isn’t it? The window for that decision is closing.

“You don’t want [the robot] to go super fast in crowded commercial spaces,” says Glas. “But people walk quickly. If you’re walking through that door, and Robovie wants to talk to you, he has to start early. We have to predict people’s behavior in the future, predict not only where they are, but what they’re going to be doing and where they’re going. The system gets a couple of seconds of data.”

Herein is the reason Robovie represents a great leap forward in artificial intelligence. He’ll never play chess as well as IBM’s Deep Blue played against Garry Kasparov. He won’t win on Jeopardy and can’t vacuum better than Roomba. What Robovie does is learn about the people in his environment. He takes in information about his setting and the live actors in that setting and responds on the basis of a perceived pattern, moving toward reward and away from threat. This is incredibly human.

In his 2004 book On Intelligence: How a New Understanding of the Brain Will Lead to the Creation of Truly Intelligent Machines, neuroscientist and Palm Computing founder Jeff Hawkins argues that the neocortex evolved expressly for the purpose of turning sensory data, in the form of lived experiences, into predictions.

“The brain uses vast amounts of memory to create a model of the world,” Hawkins writes. “Everything you know and have learned is stored in this model. The brain uses this memory-based model to make continuous predictions of future events. It is the ability to make predictions about the future that is the crux of intelligence.”

This ability to anticipate is a function of the neocortex, which can be traced back to humankind’s reptilian ancestors of the Carboniferous Period. Over the course of millions of years, the neocortex increased in complexity and size and emerged as the central component in human cognitive intelligence—not because of what it is, which has changed materially, but because of what it does.

The process of prediction forms the very basis of what makes us human. We see, we gather data from our senses, we predict, therefore we are. By that metric, Robovie’s every interaction, his every encounter, every question he asks, and every response he picks up brings him a little bit closer to humanity.