Vanadium incident (Part 1): you have up-to-date backups but you're in for a nightmare

Streamscan specializes in the detection and elimination of intrusions in computer networks. We are constantly called upon to intervene directly in companies to limit the damage during a computer attack. Our interventions are based on the most advanced technologies in artificial intelligence and also on the expertise of our cybersecurity experts.

During a deployment of our incident response team, we documented the process of our response. This is the Vanadium case.

Overview of the Vanadium incident

It is common to hear that to effectively protect yourself against ransomware you just need to have up-to-date backups. It seems obvious and everyone repeats it. But is this really the case?

In one incident we managed in the Montreal area, the organization had up-to-date backups and yet the management of the incident was very challenging.

The article is separated in 2 parts:

1 - Containment of the incident (ransomware)

2- Never give up

Part 1 - Containment of the incident

Saturday 11:03 am - initial contact: our phone rings. On the other end of the line, a Montreal-based retail company has been hacked. 18 of its servers have been infected by ransomware and the hacker is demanding $200,000 US to be paid in bitcoin. The incident took place on Friday around 2AM and no one was able to work during the day. The internal IT team tried to fix the problem and after several hours of hard work without rest, they are not sure if they can fix the problem. In addition, the team suspects that the hacker is still in the network. So the IT team decided to seek outside help to ensure that the incident was handled properly.

Saturday at 11:30 am - debriefing: we gather our incident response team members and debrief the situation. 2 members of our team are dispatched to the premises of the company victim of the ransomware. In parallel, we assist the company remotely to implement immediate containment measures.

Saturday at 12:43 - arrival at the company's premises and information gathering meeting: our incident response team arrives at the hacked company's premises. We hold an additional information gathering meeting and ask to know all the measures taken since the beginning. It turns out that all 18 infected servers are Windows-based. A ransomware file was dropped in each directory encrypted by the ransomware. The organization does not have an incident response plan and a member of senior management has requested to contact the hacker to find out his intentions. The hacker is demanding payment of US$200,000, payable in bitcoins.

The situation is tense, the stress level is at its highest, and senior management is putting pressure on the IT team to know when the problem will be fixed. What a nightmare for the IT team.

Saturday at 1:05 p.m. - the company appears to have backups: It appears that the organization has remote backups at a vendor located in Toronto. This vendor has been contacted to confirm if the backups were not affected by the ransomware. The IT team is awaiting a response from the vendor.

Saturday 1:08pm - we take charge and establish the action plan to manage the incident and get back to production as soon as possible. The first action is to clearly present the situation to the top management in a transparent way: it's a big problem, it won't be easy but we'll get there. It is useless to put pressure on the IT team because it will not change anything. We are rolling out our action plan:

1- check that the incident is totally contained and cannot spread any further

2- make sure that the hacker is no longer in the network

3- Etc.

We present our action plan to the top management and we have their approval to move forward. Let's go! The serious business can begin.

Saturday 1:34 PM: complete network isolation: while doing our initial investigations, we quickly discover that a copy of the ransomware has been deposited on other servers of the network, but have not been executed (fortunately).

This discovery shocks the IT team and increases their stress level. They are no longer able to confirm the exact number of servers the organization has. They are also unable to confirm whether the organization has servers in the cloud or not.

In such a case, the best thing to do is to shut down the entire Internet network. If the hacker is still in the network, he will automatically be ejected. Shutting down the Internet takes the pressure off and the IT manager will say 'thank goodness you're here! For a moment I was afraid that the hacker would discover that we had seen him and that all our other servers would start to be encrypted'. Phew of relief for the IT team.

Saturday, 1:57 p.m. - the white smoke is billowing - the backups are fine: the IT team of the victim organization receives the answer from their backup solution provider. Everyone holds their breath.

Finally the backups are good and can be restored. Phew! What a relief. Senior management is informed and we ask them to stop responding to the hacker. The IT team is busy starting to download the backups.

Saturday, 2:48 p.m. - identification of all servers on which a copy of the ransomware has been deposited. We plug our CDS cyber threat detection technology into the network. In addition to its ability to detect and block attacks that target a network, this technology can scan the organization's entire IT estate to identify computers on which a copy of ransomware has been dropped.

We find another 13 servers and 4 workstations with a copy of the ransomware. At this stage, the victim organization knows exactly how many machines in its computer pool the hacker had access to.

We have all the affected machines isolated and senior management is informed.

Saturday 3:14pm - start of in-depth investigations: we start in-depth investigations to find out how the hacker got into the network. We collect logs on some relevant servers that were infected.

In parallel with the analysis, a copy of the ransomware was sent to our malware reverse-engineering team for remote analysis in our lab.

Saturday 3:43pm - start rebuilding ransomware-infected servers: when a machine is infected with ransomware, it must be rebuilt from scratch. The internal IT team starts reinstalling the servers. Once the servers are reinstalled we will harden them to strengthen their security. Then we can install applications and restore data.

Saturday 3:47 pm - first shock: the IT team has started downloading the backups but there is bad news. The backups are large (several tens of Terabytes) and the bandwidth of the victim organization's headquarters network is not very high. It will take about 60 hours to download the backups! Total shock for everyone, the IT Director first! It's 2.5 days! Unthinkable!

Senior management was informed. There are no other viable solutions at this time, so the data download is restarted. The download time is used to build servers and continue the investigations.

The IT team takes the opportunity to take a break: the team members have not rested since Friday.

Saturday at 5pm - we found out how the hacker got into the network. Our investigations have identified that the hacker entered the network via an attack on the RDP remote access server. It has been over 2 months since the hacker took control of the network. The attack caused quite a stir, but the victim organization did not have any intrusion detection technology (IDS/IPS/NDR) and did not monitor the security of its network. A boon for the hackers.

Saturday at 6pm: the download of the backups continues. Time to order the Pizza as it could be a long night. We try to cheer up the internal IT team, it's the reality of managing cybersecurity incidents, but we're here to help.

We also make sure that there is enough coffee because the night might be long... Very long. As you can imagine, coffee is a working tool for incident response teams!

Saturday at 8:37 p.m. - the nightmare begins: the data download stops abruptly again. Everything has to start again. Another shattered hope. The in-house IT team collapses, but this is no time to give up.

We continue to motivate the troop: this is something that happens often and we have to find a solution. We'll get there!

Read more of the story: Part 2 - Never give up!