NEW YORK TIMES
July 6, 2004
By Margaret Werhteim

In a dusty valley in southern Lebanon, "Sgt. John Smith" of the Special Forces scans the scene in front of him. Ahead is a village known as Talle. His immediate mission: to find out who the local headman is and make his way to that house.

All discussions with the villagers will have to be conducted in Arabic, and Sergeant Smith must comport himself with the utmost awareness of local customs so as not to arouse hostility. If successful, he will be paving the way for the rest of his unit to begin reconstruction work in the village.

Sergeant Smith is not a real soldier, but the leading character in a video game being developed at the University of Southern California's School of Engineering as a tool for teaching soldiers to speak Arabic. Both the game's environment and the characters who populate it have a high degree of realism, in an effort to simulate the kinds of situations troops will face in the Middle East. Talle is modeled on an actual Lebanese village, while the game's characters are driven by artificial-intelligence software that enables them to behave autonomously and react realistically to Sergeant Smith.

The Tactical Language Project, as it is called, is being developed at U.S.C.'s Center for Research in Technology for Education, in cooperation with the Special Operations Command. From July 12 to 16, real Special Forces soldiers at Fort Bragg in Northern California will test the game and put Sergeant Smith through his paces.

The user plays Sergeant Smith, while the other characters are virtual constructs. Using a laptop, the user speaks for the sergeant, in Arabic, through a microphone headset and controls the character's actions by typing keyboard instructions.

The project is part of a major initiative, financed by the Defense Advanced Research Projects Agency, or Darpa, to explore new ways of training troops by making use of the large installed base of existing technology, especially laptops.

"I'd like to be able to send something like this to every soldier stationed in a foreign country," said Dr. Ralph Chatham, the Darpa project manager.

The philosophy is to deliver what Dr. Chatham calls "tactical language," linguistic skills sufficient to the task at hand.

Dr. Lewis Johnson, the director of the Center for Research in Technology for Education, or Carte, said, "The basic assumption is that there's certain situations you need to face -- such as establishing a rapport with the people you meet and finding out where the headman lives -- and how do you cope effectively with those situations."

No one is going to be able to read Omar Khayyam after this training, but the agency hopes it will enable soldiers to navigate more easily and safely through the Arab world. In its current version, the game teaches Lebanese Arabic. The U.S.C. team is also working on an Iraqi Arabic version. Darpa hopes to have at least some preliminary version to the military by the fall, Dr. Chatham said.

Dr. Johnson, a linguist and an artificial intelligence expert, noted that for English speakers, Arabic is a relatively difficult language, containing sounds that they find hard to distinguish. Moreover, Arabic dialects differ considerably by region.

"People who are taught literary Arabic typically have a lot of difficulty on the street," he said.

But it is on the street that soldiers need to be most effective. One of the tools the Carte team has developed is a virtual tutor that uses artificial intelligence software to coach individual students through the minefield of pronunciation. To do this, the researchers have had to design speech recognition software tailored specifically for language learners.

At the United States Military Academy, senior-level Arabic students who tested an early version last October were "very enthusiastic" about it, said Sherri Bellinger, director of West Point's Center for Technology Enhanced Language Learning.

"We've had a vision of learner language speech recognition for a long time, but until recently we didn't have the computer power to make this possible," Ms. Bellinger said.

Communicating is not just about uttering the right words, Dr. Johnson said. It also involves a huge amount of nonverbal interaction. The Tactical Language Project was born of Darpa's realization that in addition to basic vocabulary soldiers in foreign countries also need to understand basic cultural and gestural cues. Dr. Chatham tells the story of a soldier in Afghanistan, soon after the start of war there: soldiers in his unit came to a village and realized that not only did they not understand a word being spoken, they could not interpret people's nonverbal cues.

In tense situations like those induced by war, nonverbal messages may be just as important as words themselves. The Tactical Language Project game is intended to teach such skills. Users learn, for example, that when Sergeant Smith starts or finishes a conversation with an important person, he can cross his right hand over his heart and bow slightly, a common gesture of respect in the Arab world.

As with speech, nonverbal communication is a two-way process, and here, said Dr. Hannes Vilhjalmsson, a Carte scientist, realism becomes a critical quality. A great deal of the team's research has been directed at getting the game's virtual characters, or "agents," to behave in realistic ways.

Dr. Vilhjalmsson pointed out a simple example: "When you are talking to someone, you want them to be facing you."

Humans take this for granted, but agents have to be taught it.

Such intense levels of realism may sound like a luxury, but Dr. Chatham notes that research on memory formation suggests that people retain more information when they are in a heightened state of mental engagement with their surroundings. In order to make the game's characters as realistic as possible, each one is programmed with what the researchers call a belief system. Each character has its own individual set of beliefs about the world and about Sergeant Smith that will change in response to his actions, said Mei Si, a doctoral student in charge of coding this element.

One of their most critical beliefs is their trust level, Ms. Si said. If Sergeant Smith behaves appropriately, he will gain the characters' trust and they will help him; if not, he is likely to cause suspicion.

"You don't have to be obnoxious," Dr. Vilhjalmsson said. "Mainly, you just have to be impolite, or not seem to care about what you are saying."

Dr. Johnson noted that one of the first things many users have to learn is simply to say thank you.

"Most video gamers are not used to saying thank you in the context of a game," he said.

Developing so-called intelligent agents is currently a hot research topic and U.S.C.'s Information Sciences Institute, where Carte is based, is home to world leaders in this field.

Two institute scientists, Dr. David Pynadath and Dr. Stacy Marsella, have developed a program called PsychSim to model individual and group behavior among agents. PsychSim is the software platform guiding the behavior of the Tactical Language characters.

Another characteristic the agents possess is what Dr. Vilhjalmsson calls their "arousal level." One way of understanding this, he said, is as a form of virtual anxiety: "If their anxiety level gets too high, it will trigger them to act."

In a scene in a cafe, Sergeant Smith must try to find out who the village headman is. If he doesn't act properly, one of the cafe patrons will jump up and demand to know who he really is. If tensions escalate, the patron will eventually accuse the sergeant of being a C.I.A. agent.

Intelligent agents have been used before in research environments, but this is the first time such sophisticated behavioral modeling has been put in a video game, Dr. Johnson said.

In the second phase of the project, beginning late this year, two further languages will be included. One will probably be Dari, a major language in Afghanistan. Another under consideration is one of the Indonesian languages. Once the basic platform is designed, Dr. Chatham said, the team hopes to use it with many different languages and cultural contexts. "We're spending a lot on developing this," Dr. Chatham said, noting that the cost is about $7.2 million. But the hope, he said, is that eventually such intelligent games could be used not just for teaching languages but ''for other kinds of memory-intensive training tasks."

CORRECTION - July 10, 2004: An article in Science Times on Tuesday about a project at the University of Southern California that uses artificial intelligence to help soldiers learn Arabic gave an incorrect literary example. Omar Khayyam's poetry is in Persian, not Arabic. Because of an editing error, the article also misstated the location of Fort Bragg, where Special Forces soldiers are to test a video game this month as part of the project. It is in North Carolina, not Northern California.

Copyright 2004 The New York Times Company

» More press articles