Networks of compromised computers controlled by a central server, better known as botnets, are a Swiss Army knife of tools for online criminals. Hackers can use these co-opted systems to churn out spam, host malicious code, hide their tracks on the Internet, or flood a corporate network to cut off its access to the Web.

Credit: Technology Review

Whenever a new botnet appears, researchers race to reverse engineer the software it installs on a victim's machine, and to decode the way each bot communicates with the controlling server. Because these communications are often encrypted, such analyses can take weeks or months. Now researchers from the University of California at Berkeley and Carnegie Mellon University have created a way to automatically reverse engineer the communications between compromised computers and their controlling servers.

In a paper to be presented this week at the Association for Computing Machinery's Conference on Computer and Communications Security, the researchers show how automatic reverse engineering can decipher the structure and purpose of the communications between a command-and-control server and its bots.

"The communications protocol of the botnet is the core of the botnet," says Juan Caballero, a PhD student affiliated with both the University of California at Berkeley and Carnegie Mellon University, and lead author of the paper. "That is how the attacker sends commands to the botnet."

When researchers have previously tried to automatically analyze botnet communicationprotocols, they focused on deciphering the commands received by the client. Yet Caballero, together with UC Berkeley assistant professor Dawn Song and two other colleagues, has developed a technique that translates both the commands received by a client and the responses it sends.

The researchers then ran the botnet code on a virtual machine and analyzed the movement of information to and from a computer's registers--memory components within a machine's processor--before it was encrypted. Watching for changes in the memory registers--the researchers call this "buffer deconstruction"-- allowed them to derive the structure of the botnet communications and infer the function of the various components of each command.