ChatGPT can’t beat human smart contract auditors yet: OpenZeppelin’s Ethernaut challenges

While generative artificial intelligence (AI) is capable of performing a wide variety of tasks, OpenAI’s ChatGPT-4 is currently unable to audit smart contracts as effectively as human auditors, according to recent testing.

In an effort to determine whether AI tools can replace human auditors, blockchain security firm OpenZeppelin’s Mariko Wakabayashi and Felix Wegener pitted ChatGPT-4 against the firm’s Ethernaut security. challenge,

Although the AI ​​model passed most levels, it struggled with new ones introduced after the September 2021 training data cutoff date, as a plugin that enables web connectivity was not included in the test.

Ethernaut is a wargame played within the Ethereum Virtual Machine in which 28 smart contracts – or levels – are hacked. In other words, levels are completed once the correct exploit is found.

According to testing by OpenZeppelin’s AI team, ChatGPT-4 was able to detect the exploit and pass 20 out of 28 levels, but some additional hints were needed to help solve some levels after the initial hint : “Does one of the following smart contracts have a vulnerability?”

In response to questions from Cointelegraph, Wegener said that OpenZeppelin expects its auditors to be able to meet all Ethernaut levels, as all competent authors should be able to do.

While Wakabayashi and Wegener concluded that ChatGPT-4 is currently unable to replace human auditors, they highlighted that it should still be used as a tool to increase the efficiency of smart contract auditors and detect security vulnerabilities. Can be used as, note:

“To the community of web3 builders, we have one word of comfort – your job is safe! If you know what you’re doing, AI can be leveraged to improve your efficiency.”

When asked whether tools that increase the efficiency of human auditors will mean that firms like OpenZeppelin won’t need as many, Wegener told Cointelegraph that the overall demand for audits exceeds the capacity to provide high-quality audits, And they expect the number of people employed. To keep growing as auditors in Web3.

Connected: Satoshi Nak-AI-moto: Bitcoin’s creator turns AI chatbot

May 31 in Twitter ThreadWakabayashi said that large language models (LLMs) such as ChatGPT are not yet ready for smart contract security auditing, as this is a task that requires a high degree of accuracy, and LLMs need to be able to generate text and perform human-like interactions. is adapted to.

Because LLMs try to predict the most likely outcome each time, the output is not consistent.

This is obviously a big problem for tasks requiring a high degree of certainty and accuracy in results.

— marico (@mwkby) May 31, 2023

However, Wakabayashi suggested that AI models trained using tailored data and output goals could provide more reliable solutions than chatbots currently available to the public trained on large amounts of data.

What does this mean for AI in Web3 security?

If we train AI models with more targeted vulnerability data and specific output goals, we can create more accurate and reliable solutions than powerful LLMs trained on large amounts of data.

— marico (@mwkby) May 31, 2023

AI Eye: 25K traders bet on ChatGPT’s stock picks, AI sucks at dice throws and more