Noteable is a collaborative data notebook that enables data-driven teams to use and visualize data, together. Doing so, we offer a free plan on our multi-tenant with unlimited compute. There is no gotcha and no limit in terms of product usage. We became attractive to malicious actors who wanted to run scripts on Noteable for different reasons, such as Cryptojacking. We are sharing how cryptojacking threats can be significantly reduced by implementing various security measures depending on the organization’s needs.
Cryptojacking or malicious crypto mining is a security threat in which attackers abuse compute resources to mine cryptocurrency without the organization’s consent. This can impact performance, increase business costs, and generate profits for malicious users through the mined cryptocurrency.
Due to the nature of our product and the kernels allowing arbitrary code execution, Noteable quickly became a target for cryptojacking attacks. In the hope of helping other companies to deal with these, we are sharing our journey that walks through how we:
- detect malicious activity,
- identify the malicious patterns and;
- mitigate existing and future cryptojacking activity
We leveraged our own product to manage the complete incident response ranging from digging into the logs to acting and blocking accounts using Auth0 APIs.
Detecting Cryptojacking
One of the immediate signals for us was the increase in CPU-intensive tasks and a spike in cloud computing costs. In general, both are significant indicators to monitor continuously for early detection of a potential attack.
We built a Noteable notebook (another example of a public notebook) to analyze data collected from cloud billing reports. We could not share the actual notebook due to the confidentiality of the data, but we are sharing some visualizations. It helped us to detect cost patterns and show recent anomalies quickly.
Leveraging our interactive data explorer, within the first hour of the incident, we had a clear picture of the scope of the issue. In addition, our collaboration functionality could bring the entire team on the same page.
How bad can it get?
We started with some occasional occurrences and just a couple of detected cryptominers; it quickly evolved to a more complex and hard-to-manage scenario with multiple engineers involved across DevOps and Backend and an effort required to track down and kill crypto mining processes.
Noteable secrets allowed a smooth integration with Auth0 API to retrieve a list of cryptominers and their locations. The below tile map visualization shows the cryptominers distribution around the world.
How Noteable implemented security measures for cryptojacking
- We followed the best practices for securing Kubernetes platform. Firewalls and network policies are configured to prevent unauthorized communications.
- We put in place resource usage monitors to detect potential crypto mining activity. Crypto mining requires a lot of processing power and can consume a significant amount of resources, such as CPU, memory, and network bandwidth. If there is an unusual spike in resource usage, it could be a sign of crypto mining activity.
- We scheduled our noteable notebooks to run daily monitoring compute costs, usage and detect spikes as potential indicators of cryptojacking.
- We integrated Sysdig Secure for detection. Sysdig Secure deploys different types of policies for threat detection, including access to crypto mining networks, suspicious crypto mining rules, and machine learning policies to detect cryptominers. It is possible to control what should happen to affected containers if the policy rules are breached in an automated way.
- We implemented Seon fraud prevention tools. Seon evaluates risks and helps to detect malicious accounts during the sign-up process. It is a powerful tool to identify fraudulent behavior patterns with scoring engine features.
Below we show the rules’ effectiveness over time. Customized parameters and increased fraud scores rejected a higher number of crypto miners.
We can also look at the applied rule statistics and recurrence:
Finally, we set up notification channels whenever thresholds and violations occurred for broader visibility. We have slack notifications from our notebooks to our slack channels. We leverage Secrets to store Slack credentials, and a Python code cell sends the notification. It is super easy, and we can have conditional notifications whenever specific criteria are met.
The workflow above gives us enough feedback loop and data to update our rules to improve detection continuously. In security, observation and fine-tuning configurations are key. Attackers live to come up with ways to abuse while we live to find that new way as soon as possible!
Our most recent data shows an outstanding decrease in attacks:
Long-term mitigations require more than putting a process in place. It’s an ongoing effort to stay up to date with the latest security events in our infrastructure and the implementation of effective policies. Using Noteable no-code visualizations capability simplifies the fight against cryptojacking by reducing time to insights to take quick action and customize measures based on crypto mining patterns. Aggregating data from different sources, running Python or SQL code, and then visualizing the results took less than a day of work. Most importantly, having the entire team be able to collaborate made the incident response process streamlined, as it was a single document that the entire team was using to orchestrate multiple prongs of the response.
We would love to connect if you want to learn more about our ongoing effort to tackle and mitigate malicious attacks.
