Publications

Can a Game Teach Practical Cyber Security?

Lincoln Laboratory’s Capture the Flag competition challenges college students
to defend cyberspace.

Thousands of teams around the world bearing names like the Plaid Parliament of Pwning, gettohackers, or Shellphish compete each year in contests to infiltrate opponents' computer services while defending their own systems from cyber attacks. In these competitions, teams playing on networks of virtual machines earn points by breaching other teams' services to capture information that the contest administrators hide within the programming. Called a flag, the information is typically a lengthy string of random, hard-to-guess code. The first of these Capture the Flag (CTF) events was held at the 1996 DEF CON,¹ now one of the world's largest hacker conventions. Since then, CTF competitions have sprouted up in dozens of countries, often organized by university departments and technology companies seeking to improve students' and employees' skills in devising techniques and tools to ensure network security.

Success at Capture the Flag depends a great deal on the teamwork exhibited by the competitors as they plan attack and defend strategies.

To investigate what educational benefits CTF competitions provide to participants and whether CTF play leads to the development of innovative strategies applicable to real-world cyber defense, researchers from Lincoln Laboratory developed a CTF event for college students.

Early on, the core team of technical staff members from the Cyber Security and Information Sciences Division—Joseph Werther, Michael Zhivich, Timothy Leek, and Andrew Davis—decided that their CTF competition would be structured as an attack-defend format. Some CTFs focus on either offensive or defensive actions. In contests in which only attacks earn points, competitors focus on techniques to breach security and forgo protecting their systems. In defense-only matches, players employ functions to keep their services running despite assaults the CTF administrators have embedded in their virtual systems; these players do not face the pressure of devising defenses while also crafting attacks against others and foiling a steady barrage of onslaughts from other teams.

The dual format has significant advantages. Requiring success at both attack (capturing flags) and defense (securing services) to score points compels players to interact continually with opponents and their own systems. The attack-defend approach creates a dynamic, realistic environment in which defensive techniques must be developed under pressure and time constraints. To further simulate the demanding pace of real attack mitigations, the Lincoln Laboratory CTF also allowed teams to score points only if their services were operational. "Requiring that team services be up in order to score points either offensively or defensively provides a very strong incentive for every team to risk running services as soon as and as much as possible," says Davis. Finally, the attack-defend format is more challenging and, the Laboratory's organizers believe, more fun than a single-focus one.

Lincoln Laboratory's first CTF competition was held at MIT on 2 and 3 April 2011 and was open to Boston-area college students. Forty-five registered players from six schools showed up to spend 18 hours of their weekend attacking and defending a web application server. The virtual system was modeled as a Linux operating system running an Apache server and employing a MySQL database and Hypertext Preprocessor scripting language. So that students could experiment on an application whose code would be accessible, the open-source WordPress content management software for creating websites and blogs was chosen as the target. In addition, because the flexible architecture of WordPress allows plugins, the organizers could periodically add new vulnerabilities for the students to mitigate.

Many universities and businesses that run CTF competitions do so online, with registered teams downloading the necessary software and instructions so that they can tackle the challenges made available on contest days. Lincoln Laboratory’s CTF organizers chose to hold an onsite event. "The competition is so much more exciting live. The energy in the room is invigorating," says Leek. "There is a lot more interaction between team members."

The Laboratory's CTF development team, which in 2011 also included Nickolai Zeldovich from MIT's Computer Science and Artificial Intelligence Laboratory, found that the algorithm used for scoring the play is vital to the dynamic nature of the competition. Designing a scheme that rewards players for achieving the defensive goals of maintaining data confidentiality, availability, and integrity and that also awards points for offensive successes is a balancing act. After the first day of the 2011 event, the CTF organizers noticed that equally weighting offensive and defensive results encouraged teams to shut down their servers when they were planning their offense, thereby denying attackers access to the servers and increasing their own scores for maintaining data confidentiality and integrity. A revised weighting method to reward teams whose services were accessible created the motivation for them to focus efforts toward more defensive actions.

Patrick Hulin, a member of MIT's winning team in the 2012 CTF competition and now on staff in the Cyber System Assessments Group, credits his team's success to their emphasis on defense. "We narrowly focused on the essential tasks we had to complete in order to succeed under the scoring algorithm. It was more important to keep your services operating than to attack teams other than the leaders, so we wasted very little time working on the fun but not necessarily relevant offensive moves that took a lot of work for very little actual gain in the standings."

According to the surveys that participants filled out online after the competition, the 2011 CTF event was a success. The students appreciated the challenges presented by the game; most of them thought they had improved their skills; and many reported an increased interest in a career in cyber security (though these students did note a previous interest in such a career).

Era Vuksani, now a researcher in the Laboratory's Cyber Systems and Operations Group and formerly a member of Wellesley College's 2011 CTF team, says, "I learned a lot from being in that environment where you had to be very proactive in defending yourself from adversaries as well as be ready to wipe your machine and start over as need be. You had to be adaptable at a moment's notice."

The organizers applied lessons learned from the 2011 event to full-scale competitions held in 2012 and 2013 at MIT and a few practice sessions, or mini-CTFs, offered in 2014 at the MIT Lincoln Laboratory Beaver Works center near the MIT campus in Cambridge, Massachusetts.

In 2012, the students were tasked with sustaining the security of an enterprise web server, and the 2013 competition charged participants with supporting several apps for an Android platform and the corresponding back-end services on Linux virtual machines. As word of the Laboratory's CTF spread, participation grew: in 2012, 62 students from six schools participated, and 165 players from 10 regional universities took on the 2013 Android challenge. The events also grew longer; in 2013, the students were in competition for 48 hours straight, eating while working and taking turns catching naps.

Lincoln Laboratory’s third Capture the Flag (CTF) competition drew 165 college students to MIT for a 48-hour marathon of attacking and defending Android services. Students came from MIT, Boston University, UMass-Boston, Northeastern, Brandeis, Wellesley, Worcester Polytechnic Institute, Rensselaer Polytechnic Institute, New York University Polytechnic School of Engineering, and Dartmouth College. Many of the participants and event organizers posed for a post-event photo-graph. The official MIT Lincoln Laboratory CTF flag is held by a participant in the back, while in front a member of one of the top three teams is holding a replica of a check for the team’s prize money.

Lincoln Laboratory’s third Capture the Flag (CTF) competition drew 165 college students to MIT for a 48-hour marathon of attacking and defending Android services. Students came from MIT, Boston University, UMass-Boston, Northeastern, Brandeis, Wellesley, Worcester Polytechnic Institute, Rensselaer Polytechnic Institute, New York University Polytechnic School of Engineering, and Dartmouth College. Many of the participants and event organizers posed for a post-event photo-graph. The official MIT Lincoln Laboratory CTF flag is held by a participant in the back, while in front a member of one of the top three teams is holding a replica of a check for the team’s prize money.

The 2013 CTF event also introduced a new element—evaluating an outside organization's technology. Employees from Raytheon BBN Technologies tested out their Advanced Adaptive Application (A3) Environment prototype by trying to defend the CTF's App Store against attacks from the competition teams. After a flaw identified in the A3 Environment software on the first day was remedied, the prototype was able to secure the App Store. The Laboratory's CTF organizers concluded that "CTFs can provide a sandbox in which prototype technologies, both defensive and offensive, can be tested and evaluated," but with the caveat that the technology developers need to be on hand to fix problems. From their participation in CTF, the A3 Environment researchers gained improvements to their code, validation of their defensive policies, and a corpus of attacks they could use in building later iterations of their technology.

Creating a challenging, well-functioning CTF competition requires a significant investment in software development. The Laboratory researchers devoted a great deal of effort to conceiving the scenarios, cyber vulnerabilities, and scoring strategies, and then to building the software and interfaces that enabled these.

Lincoln Laboratory's CTF experience has been successful on a number of fronts. First, the researchers who worked on developing the competitions acquired some answers to their initial questions.

• Do CTF events help educate students in cyber security?
  The answer is a qualified yes. Students who are already inclined to engage in cyber defense, who have perhaps tried online CTF games, will strengthen their competencies in computer security. Zhivich likens the learning to that of athletes sharpening skills through practice: "Good ballplayers get better as they play more." It is harder to say how much CTF participation teaches students who do not have prior experience in computer security; in the Laboratory's CTF games, the less experienced students did not amass high scores but felt they took away new awareness of the cyber field. Although precompetition tutorials on cyber defense tactics, common cyber tools, and web applications were appreciated by the students who attended these sessions, the researchers cannot say definitively that the tutorials resulted in helping to "level the playing field" for the inexperienced CTF teams. "We believe CTF works as a kind of group self-guided, project-based instruction,"” says the CTF research team's 2014 paper chronicling their findings from hosting the events.²
• Can CTF events generate new ideas for real-world cyber defense tactics and tools?
  Again, the answer is not definitive. The researchers monitoring the competitions directed their attention to keeping the game running smoothly. They state in their 2014 paper that they would like to understand better what teams do to win CTF competitions and that "it may be possible to discover new advanced techniques for attack and defense by providing college students a safe [i.e., not incurring legal repercussions for hacking real networks] place to play." However, the experience with the A3 Environment shows that a CTF event can be used as a test bed for new technology.

On another front, the collegiate CTF competition introduced the Laboratory and many talented young people to each other. Indeed, five staff members hired into the Cyber Security and Information Sciences Division had participated in one of the Laboratory's CTF events. As the world becomes more dependent on computer networks to conduct all its activities, nations and private businesses are eager to find the best-qualified people to secure their network services. Hosting a CTF event could be one avenue for organizations to meet those people.

Finally, the CTF events resulted in personal successes for participants and organizers. In the post-game surveys, students cited not only increased understanding of cyber security but also improved teaming skills as takeaways from the competitions. Resolving the technical demands of crafting the scenarios and developing the automated scoring application were interesting projects for the Laboratory staff members. Furthermore, both students and the CTF staff had fun.

The researchers who conducted the CTF events under funding from the National Security Agency have completed their investigation into CTF's role in enhancing education and development in cyber security techniques. Their published experiences can serve as a road map for future Lincoln Laboratory CTF events and for other organizations considering the establishment of CTF competitions.^2,3 In addition, the research team is looking to make their infrastructure available as an open-source codebase.

¹ DEF CON is an annual event that attracts not only computer hackers but also researchers from academia, industry, and government agencies.

² A. Davis, T. Leek, M. Zhivich, K. Gwinnup, and W. Leonard, "The Fun and Future of CTF," 2014 USENIX Summit on Gaming, Games, and Gamification in Security Education, available at https://www.usenix.org/node/184963.

³ J. Werther, M. Zhivich, T. Leek, and N. Zeldovich, "Experiences in Cyber Security Education: The MIT Lincoln Laboratory Capture-the-Flag Exercise," Proceedings of the 4th Conference on Cyber Security Experimentation and Test, 2011, available at http://www. ll.mit.edu/mission/cybersec/publications/publication-files/full_papers/2011_08_08_ Werther_CSET_FP.pdf.