What happened during and after a Florida city paid a $460,000 ransom to the Russian eCrime group, Grim Spider: An interview with Brian Hawkins, ex-Head of IT for Lake City, Florida.
Ransomware is the scourge of businesses, both large and small. The malware that encrypts your files and brings business to a halt is arguably one of the most malicious programs ever to be developed. Security analyst firm, Cybersecurity Ventures, has predicted that firms will experience a ransomware attack every 14 seconds by the end of 2019. They describe the technique of financial extortion by holding files for ransom as the “go-to method” for cybercriminals.
UPDATE: WatchPoint has hired Brian as our Technical Dispatch Manager.
UPDATE2: The New York Times picked up this story and has written a full article:When Ransomware Cripples a City, Who’s to Blame? This I.T. Chief Is Fighting Back
This brings us to the discussion point of this article. The ex-Head of IT for the City of Lake City, Florida, Brian Hawkins, was fired after a recent ransomware attack on the city. The attack resulted in the city paying a ransom of $460,000 in Bitcoin. Although it was reported by several media outlets that the attack caused the city to be shut-down for three weeks, their ERP and primary management systems were up and running in 24 hours. The city’s local servers and shared files were inaccessible due to the on-premise backups being deleted, leading to the decision to pay the ransom.
WatchPoint Data talked to Brian about his experience.
A First-Hand Account of a Ransomware Attack
WatchPoint: What variant of ransomware hit your network at Lake City?
Brain Hawkins: Ryuk -- I had not previously heard of this particular version of ransomware. This variant is known as a “triple threat.” It exploits a weakness using social engineering, e.g., using phishing emails to trick a user into clicking on a malicious link. This begins the infection process, starting with a trojan that crawls through the network. The trojan goes undetected. In the meantime, the trojan steals credentials, and the result is the upload of the ransomware payload.
WatchPoint additional notes: Ryuk is the ransomware of choice of the Russian eCrime group, Grim Spider. It specifically targets enterprises. It first appeared in August 2018 and is believed to have made the gang operating the ransomware $3,701,893.
WatchPoint: What were the effects on the network? Were specific file types impacted?
Brain Hawkins: Most of our servers had at least some encrypted files on them. The ransomware focused on finding key 
points in our server infrastructure, like file servers.
WatchPoint: How did you discover you were infected?
Brain Hawkins: We noticed that email was running slowly over the weekend before the Monday morning that we discovered the infection. On logging in on Monday, we found the encrypted files on our exchange server. On investigation, we also found other infected servers. At this point, we shut down some servers and disconnected all infected servers from the network and the internet.
WatchPoint: Did your Data Recovery (DR) plan work as expected? Considering that the Ryuk ransomware also impacts back-ups?
Brain Hawkins: Once we understood what we were dealing with, I instructed my staff to recover from back-up. At that point, we realized the back-ups had been deleted by the incident; this part of the plan failed. The second part of the plan was our ERP system (city management application). Back in 2017, I had ensured our ERP system was migrated to the cloud. Fortunately, this meant this system was unaffected. This decision allowed the city to continue to conduct business. Our ERP was essential to allow all aspects of city government to function, even after the infection by RYUK.
We then explored all options in recovering the encrypted files; this included sending the hard drives from the backup servers to recovery specialists.
WatchPoint: What made the city decide to pay the ransom?
Brain Hawkins: Timing was one aspect – how long do we wait to see if we can recover the data? Should we try and negotiate to recover our data? The city used its insurance firm to hire a third-party company to negotiate on the city’s behalf. The negotiators used an email address associated with the ransom to contact the cybercriminals behind the attack. There was no actual initial ransom or deadline until the hackers were contacted by the company.
Ultimately, the city paid 42 bitcoins, which is worth around half a million dollars.
WatchPoint: How much did the deleted back-up files factor into paying the ransom?
Brain Hawkins: A great deal. The recovery process on the files was taking a long time. This was because it was a Linux based system and because of the structure of the backups. The lengthy process was a major factor in paying the ransom.
I was involved in discussing the payment but did not make the final decision.
WatchPoint: What measures did you previously propose that could have helped to mitigate the damage of the attack?
Brain Hawkins: The issue came down to the fact our backup solution was on-premise. I suggested options of having a cloud backup solution, along with the option of the ERP migration to the cloud. However, the lowest cost option, which was to migrate our ERP, was chosen. A cloud back-up solution would have helped expedite recovery.
WatchPoint: What was it like to be in the midst of a ransomware attack?
Brain Hawkins: It was horrifying. Unbelievable. How could this happen to us? We are a small town in Florida, of around 12,000 residents, why would we be a target for ransomware?
WatchPoint: How long was the city’s network shut down?
Brain Hawkins: Our network was never actually down, only our server infrastructure. Fortunately, the cloud-based ERP system was still functional. We also held a set of ‘clean’ computers which were set aside for such a disaster, which we used after disconnecting other computers. We connected these computers to the ERP system, and because of this solution, we were able to open the next day for business.
WatchPoint: How did you feel when the city terminated your employment?
Brain Hawkins: We discovered the attack on June 10th; on June 21st I was terminated. I had been with the city just over five years and in charge of IT for 2.5 years. I went through pretty much every negative emotion that day; disbelief, hurt, anger, sadness. I wasn’t finished with my job and felt I still had projects to complete; I still wanted to give to the city and always worked with the best interest of the city in mind; this was my work ethic. I felt that myself and the team had done a lot of good for the city. I was in disbelief it had come to this.
WatchPoint thoughts: As security specialists, we believe that Brian, as Director of IT, was the most important person to keep around through this emergency and recovery period. He was the best person to execute recovery in the short and long term.
WatchPoint: What was it like to find out your name was being splashed across the Internet with headlines like “Florida Man Fired After Lake City Suffers Massive Ransomware Attack.”
Brain Hawkins: I was shocked, angry; I felt betrayed by the city I had given so much to. I felt it was completely unnecessary for my name to be splashed around the news media. They could have told the story without ruining my reputation.
WatchPoint: Why do you feel you were terminated from Lake City?
Brain Hawkins: I was a scapegoat. It was a political move. Someone’s head had to roll, and it wasn’t going to be the City Manager’s.
WatchPoint: What security upgrades could have been implemented to prevent the attack?
Brain Hawkins: In this case, a cloud back-up and disaster recovery solution would have expedited the recovery process.
WatchPoint: What percentage of similar city councils, do you believe, are vulnerable to this kind of attack?
Brain Hawkins: I would say 100%.
WatchPoint: Did the City install anti-virus and firewalls and were they up to date?
Brain Hawkins: We had every form of security you can imagine, and it was all up to date. This may seem enough if you aren’t familiar with cybersecurity, but these tools alone are not sufficient.
WatchPoint: What can other IT professionals do to protect their jobs?
Brain Hawkins: Secure the data, regardless of the cost. I managed an IT department operating on a budget. I have never sat through a budget meeting where the topic was anything other than cut the budget. The City of Lake City discovered the cost of valuing the pennies saved over the security of its data.
WatchPoint: For your next job, will you run toward or away from IT Security?
Brain Hawkins: Definitely toward. Cybercrime is not going away. Cybersecurity is the most important part of the data network. It is imperative to use the proper security and use solutions that work. Not only from the prevention standpoint but from the standpoint of what to do when the attack happens – because it will happen. It is vital to think about what happens after the attack.
Closing thoughts from WatchPoint Founder, CEO, and ransomware expert – Greg Edwards:
WatchPoint Data would like to thank Brian Hawkins for his candid and highly informative interview. Brian’s experience is not one any of us want to go through. Using his wisdom, we can look to implement a more comprehensive way of dealing with cyber-threats.
Steps to prevent and mitigate ransomware:
- Use an encrypted offsite backup solution and stop relying on local backup. Brian suggested this and was denied by the same people that fired him.
Recommendation: WatchPoint Bare-Metal Backup
- PATCH – Every station and every application needs to be up-to-date to close security vulnerabilities.
Recommendation: ConnectWise Automate
- Educate – Train users to recognize phishing emails and social engineering.
Recommendation: KnowBe4
- Firewall and Antivirus – Make sure they are up-to-date.
Recommendation: Firewall: SonicWall, Fortinet or PaloAlto NextGen AV: Carbon Black, Webroot, Sophos
- A dedicated ransomware detection tool - deception technology
Recommendation: WatchPoint CryptoStopper – 1 through 4 are necessary but still won’t stop every ransomware attack. CryptoStopper is your last line of defense and will stop actively running attacks that get past your other defenses. We’ve made it free for home users and small offices.
As Brian mentions, cybercrime and ransomware are not going away anytime soon. Companies, local governments, and individuals all must take additional steps to mitigate and prevent the damage attackers are doing. Stopping the effects of ransomware is the primary reason WatchPoint was founded. Businesses and governments have to stop paying these ransoms for the attacks to stop happening, but when faced with losing critical data or paying the ransom, sometimes paying is the only choice. Make the decision now to follow the steps above so that paying isn’t your only choice.
