Coordinated Response
Services and tools for incident response management

Case Study

Penn State and a Coordinated Response

Penn State was hacked, but returned “a coordinated and deliberate response.”

NOTE: Our firm, Coordinated Response was NOT involved, but we can all learn from Penn State’s response. The response includes a number of best practices highlighted in this article.

On May 15, 2015 Penn State announced the discovery of two sophisticated cyber attacks. “In a coordinated and deliberate response by Penn State, the College of Engineering’s computer network has been disconnected from the Internet and a large-scale operation to securely recover all systems is underway.”

For more details read the announcement on the Penn State web site, but here are some best practices.

An Effective Communications Plan

Penn State announced the breach on May 15. On the same day, they released the web site to keep students and faculty informed of events. The URL suggests they were prepared for any eventuality. Penn State also notified research partners and individuals who may have had sensitive data exposed to the intruder.

Detect and Analyze

On November 21, 2014, the FBI alerted Penn State of a cyber attack with unknown origins and scope. The University’s security experts along with outside experts began an extensive investigation. Mandiant, a FireEye company, was one of the experts. The investigation determined that the earliest confirmed date of intrusion was September 2012 — over 2 years earlier.

Contain: Protect and Preserve

Steps were immediately taken to further protect and preserve critical information and sensitive data on the College of Engineering Systems.

Cone of Silence

No announcements were made to avoid alerting the intruder and to avoid unwanted damage or destruction.

Eradicate and Recover

On May 15, once the scope and nature of the intrusion were known, the engineering systems were taken offline to remove malware, secure the systems, and restore them to operational status. This is expected to take several days. All passwords are being reset to address potentially compromised credentials.


Once the investigation was completed the communication plan was launched.

Lessons Learned

Two factor authentication for remote access to the Engineering systems is being implemented now. This will be extended to the rest of the University later this year. Additional measures were taken to improve the security posture.

Coordinated Response

One might question why it took over 2 years and an outside organization to discover the intrusion. But, experience suggests state actors operate “slow and low”. First, the intruders observe without leaving “foot prints” for an extended period. Then, instead of leaving 10 foot prints a day, they leave 1 a day for 10 days. They work hard to make it hard to detect.

But, once alerted, Penn State responded with a well planned and well coordinated response. Let us help your organization with an incident response plan review. Together, we can apply best practices to improve the plan and the outcome of a cyber incident.

Department of Energy Hack Raises Issues

The Washington Free Beacon reported on 2/4/2013, “Computer networks at the Energy Department were attacked by sophisticated hackers in a major cyber incident two weeks ago and personal information on several hundred employees was compromised by the intruders”. A total of 14 computer servers and 20 workstations at the headquarters were penetrated during the attack.

This article and other articles in the recent past all raise the same issue: inadequate security measures stemming from (pick one or more): improperly trained administrators, inexperienced security staff, budgetary constraints, and/or “institutional hubris”.  Government has a responsibility to protect the information entrusted to it by its citizens.  However, the government – all branches – has failed in this endeavor and will likely continue to fail until they wake up to reality and get smarter than those attempting to compromise their systems.

Mandatory security testing and training must be implemented at all levels of IT and operations throughout the government.  If sensitive information is involved, training must be held.  I am not talking about awareness training; I am talking about training the administrators, IT managers and security staff on what to look for, how to properly program and configure and, most importantly, how to test systems and how to properly conduct attack and penetration tests.

Do not rely on hiring people with long strings of certifications behind their names.  In many cases, they are merely cert collectors who have no clue as to what the certs really mean – other than the more certs you have the better chances of getting a job.  Establish real training programs.  Work with groups such as the GIAC (Global Information Assurance Credentials) which has programs that REQUIRE a practical exercise before a cert can be awarded. NSA relies on GIAC certified individuals, why shouldn’t the rest of government?

Finally, forget sending trained staff away to conferences.  Not only will the conferences be a waste of time – it seems only controversial, contrarian views are desired for talk topics these days – but you will leave your networks and systems in the hands of those not as qualified to deal with crises should the inevitable happen.  Everyone likes to go to conferences (if for no other reason than to collect suitcases full of vendor-supplied swag) but the best bet on training spending is on real training as supplied by organizations such as the SANS Institute.

CSO magazine analyzing this story provided a number of sources that support the same conclusion.

New York Times Hacked Again

The New York Times incident response and incident handling provides examples of best practices, as well as, lessons learned.

According to the New York Times, Hackers in China Attacked the Times for Four Months. The Times Incident Response Plan provides examples of best practices as well as lessons learned.

  • Include your ISP in your response plan.
  • Identify appropriate specialists.
  • Malware detection is not enough.
  • Investigate, then eradicate.
  • Involve law enforcement when appropriate.
  • Get specialists involved early.
  • Enhanced monitoring is important.
  • Review the incident postmortem.

And the most important best practice is to make security improvements based on the postmortem and lessons learned

In the case of the New York Times, the hackers were able to crack the password for every Times employee. Then the hackers used the passwords to gain access to the personal computers of 53 employees. The attack was discovered on October 25. The investigation placed the initial compromise on or around September 13 or 6 weeks earlier.

First, the Times engaged their Internet Service Provider (ISP), AT&T, to watch for unusual activity as part of AT&T’s intrusion detection protocol. The Times received an alert from AT&T that coincided with the publication of a story on the Chinese Prime Minister’s family wealth.

The New York Times briefed the Federal Bureau of Investigation (FBI). The FBI has jurisdiction over cybercrimes in the United States.

At the start of week 9, when the response team, even with AT&T assistance, was unable to eliminate the malicious code, the Times engaged Mandiant, a firm specializing in cybersecurity breaches. This is both a lesson learned and a best practice. Expect to need additional help when a significant incident occurs, but it’s best to engage the specialists when the response plan is first developed.

The response team identified 45 pieces of malware that provided the intruders with an extensive tool set to extract information. Virus protection only identified 1 piece. This is an important lesson. Advanced or enhanced malware may be difficult to detect. Defense in depth requires added security controls. In this case, monitoring provided the alert.

At this point, the response team “allowed hackers to spin a digital web for four months to identify every digital back door the hackers used”. This allowed the Times to (1) fully eradicate the intruders, (2) determine the extent of the intrusion and data extraction, and (3) build improved defenses for the future. Again, this is a best practice – investigation precedes eradication.

I found the work habits of the hackers particularly interesting. They worked a standard day starting at 8 A.M. Beijing time. Occasionally they worked as late as mid-night including November 6, election night.