In the dynamic world of Web3, comprehensive security measures are essential. While it’s impossible to entirely eliminate the risk of security breaches, a well-structured incident response plan can significantly reduce potential harm to your Web3 project and its users. In this blog, we’ll explore the intricacies of crafting an effective incident response plan, tailored specifically for Web3 security. Frontal, your trusted partner in Web3 security, is here to guide you through the process step by step.
Defining Web3 Security Incidents
Understanding what constitutes a “security incident” in the realm of Web3 is the first critical step in creating an effective response plan. This ensures that your team can quickly recognize potential threats and respond appropriately. Here are specific scenarios that should trigger your protocol’s emergency response efforts:
- Discovery of Smart Contract Vulnerabilities: This includes identifying a bug or vulnerability in your protocol’s smart contracts that puts users’ funds at risk.
- Underlying Protocol Vulnerabilities: The discovery of a bug or vulnerability in an underlying protocol utilized by your decentralized application (dApp).
- Third-Party Infrastructure Issues: This covers bugs or vulnerabilities discovered in third-party infrastructure and tools integral to your project, such as oracles, wallet address generators, bridges, multisig wallets, programming languages, and more.
- Ethical Hacker or Researcher Findings: When a bug or vulnerability is uncovered by an ethical hacker, bug bounty hunter, or security researcher.
- Active Exploits or Hacks: The detection and reporting of an active exploit or hack of your protocol, whether reported by a user, community member, or an unidentified party.
- Key Internal Infrastructure Compromise: Instances where key internal infrastructure, such as founders’ private keys, is compromised.
Each of these situations necessitates a swift and coordinated response to minimize potential damage, as exemplified by the Wintermute incident, which resulted in a loss of $160 million due to an address generator vulnerability.
Creating an Incident Response Plan for Web3 Projects
- Identifying Critical Roles:The foundation of an effective incident response plan lies in defining the individuals responsible for specific tasks throughout the process. This proactive approach ensures that you have the right people with the necessary skills and qualities, such as crisis management and the ability to work under pressure. Consider roles like Operations, Strategy, and Communications, each with distinct responsibilities.
- Operations: This role oversees both on and off-chain monitoring of threats, triages and prioritizes security issues, and notifies relevant personnel about security incidents. They also coordinate the deployment of patches to fix smart contract bugs and document incident details for post-mortem reports.
- Strategy: The Strategy role analyzes threats, formulates remediation and eradication plans, and facilitates discussions and collaborations within the team. They should be experienced in handling high-stakes security incidents.
- Communications: While Operations can handle communications, having a dedicated Communications role can enhance efficiency. This role manages information flow within and outside the war room, communicates with third parties like auditors and protocol developers, coordinates internal team communication, vets incoming information, and engages with the protocol’s community and stakeholders.
- Setting Up a War Room: A “war room” serves as a central hub where team members and experts collaborate to address major emergencies. Besides the assigned roles, invite relevant experts, including protocol developers, UI developers, auditors, and white hat hackers, as needed. Your response plan should outline criteria for inviting external parties and establish communication channels within the war room.
- War Room Structure: While there are no strict standards for structuring a war room, ensure it offers useful communication and collaboration features. It can range from a private chat room on platforms like Signal, Telegram, or Discord to video conferencing tools like Zoom or Google Hangouts.
- War Room Structure: While there are no strict standards for structuring a war room, ensure it offers useful communication and collaboration features. It can range from a private chat room on platforms like Signal, Telegram, or Discord to video conferencing tools like Zoom or Google Hangouts.
- Evaluating Security Threats: After assembling the team and setting up the war room, it’s crucial to assess the security threat.
Key questions to consider include:
- Concrete Evidence: Is there concrete evidence confirming the issue, such as recent transactions or social media reports from affected users?
- Scope of Vulnerability: Does the vulnerability affect multiple components in the protocol, potentially causing domino effects?
- Risk to User Funds: Are users’ funds at risk, and if so, to what extent? The amount at stake may inform your team’s response plans.
- Immediate Actions: If user funds are at risk, what immediate actions can be taken to mitigate losses.
Collaborative efforts and tools like Tenderly Debugger, Foundry Debugger, Brownie, and EthTX can facilitate the process of understanding and addressing vulnerabilities.
- Executing Defensive Security Measures: In some cases, security incidents may necessitate immediate actions to minimize fund losses, especially if an exploit is ongoing or a vulnerability remains active. Consider the following defensive actions:
- Pausing Smart Contracts: Temporarily pause smart contracts to prevent further harm.
- UI Updates: Modify the user interface to reflect information about security incidents or disable specific user operations like deposits and withdrawals.
- Whitehat Rescues: Engage ethical hackers to execute fund rescues from vulnerable smart contracts.
Assign specific tasks in advance to expedite execution, such as listing EOA or multisig wallet accounts required for transactions and preparing scripts for deploying mitigative measures.
- Security Crisis Communication With Users: Transparent, reliable, and consistent communication with users is a critical aspect of effective incident response. Follow these best practices:
- Information Review: Ensure someone reviews all outgoing information for accuracy and to prevent disclosing information that could further jeopardize security efforts.
- Avoid Premature Commitments: Refrain from making commitments regarding remediation until all facts concerning the situation are understood.
- Appropriate Channels: Utilize the best communication channels available, starting with project-specific platforms. Communication may begin with a post on the project’s Discord server or Telegram channel before publishing an announcement on official social media.
Communication tasks are typically handled by the Communications role or someone acting in that capacity, such as a community manager or public relations officer.
- Developing and Deploying Bug Fixes: After implementing temporary mitigative measures, focus on developing a permanent solution. This phase can be divided into several sub-tasks:
- Solution Exploration: Different participants in the war room, coordinated by the Strategy Lead, explore various solutions. Each solution is evaluated based on criteria such as complexity, implementation timeline, and minimization of losses for users. Teams may need to resolve disagreements and address concerns before settling on an acceptable solution.
- Testing and Review: After reaching a consensus, the team conducts tests to confirm that the patch addresses the vulnerability. Tools like Ganache and Hardhat can be used to fork blockchain state and test contracts locally. The patch should also undergo review by security auditors and other developers to ensure it doesn’t introduce new vulnerabilities. Consider independent reviews through platforms like Code4rena.
- Implementation: The responsible party, assigned in advance, implements the chosen solution. The incident response plan document should include contract deployment/upgrade scripts, relevant multisig wallet addresses, and other necessary details.
- Conducting a Security Incident Post-Mortem: The post-mortem phase is the final step of the security incident response process. During this phase, team members and other relevant parties gather to reevaluate the incident, identify any additional root causes not previously mentioned, and provide feedback on the overall incident response process. This step yields valuable insights that can be used to prevent future exploits, improve internal security measures, and enhance incident response processes.
Additionally, the post-mortem serves as a source of information for the vulnerability disclosure statement. This document outlines the security incident, actions taken by the team to address the vulnerability, and is made available to users, developers, and stakeholders. It demonstrates your team’s commitment to actively managing security issues and protecting users’ funds.
In Conclusion
Creating a comprehensive incident response plan for your Web3 project positions you to respond effectively and efficiently to discovered vulnerabilities. However, it’s not enough to merely draft a plan—you should conduct periodic drills to ensure your team is well-prepared to handle security incidents. This proactive approach enhances the effectiveness of your incident response and distinguishes your Web3 project as a leader in security preparedness.
For expert assistance in developing a strategic security incident response plan tailored to your Web3 project, don’t hesitate to contact Frontal—we identify as frontliners of the web3 ecosystem.
