Artificial intelligence (AI) has emerged as a potentially powerful tool for the blockchain industry. One of the use cases explored right now is its utility in the area of security audits. With that, Coinbase recently conducted an experiment using OpenAI’s ChatGPT to assess whether or not it could carry out a token security review with accuracy. While the results were promising, some of the crucial mistakes it made along the way led to the consensus that AI cannot replace human auditors anytime soon.
The Nature of the Experiment
During the experiment carried out by Coinbase, they reviewed 20 smart contracts with the help of ChatGPT. While the AI tool matched the results of the manual review in 12 instances, it missed the mark on the remaining eight. Among those, five high-risk assets were wrongly labeled as low-risk.
The experiment revealed that ChatGPT has critical limitations. It generated inconsistent results in some cases where the outputs varied from one iteration to the next.
The Potential for Complementary Quality Assurance Checks
While ChatGPT has its limits, the Coinbase team sees potential for the tool to improve its precision through further engineering efforts. In its current state, it could be more useful as a tool for complementary quality assurance checking.
Other experts in the blockchain industry also acknowledge the potential for AI tools to help auditors detect vulnerabilities that humans may overlook. Nevertheless, they echo the same sentiments of Coinbase, which says that AI tools cannot replace human auditors at present.
Smart contract audits should concentrate on assessing complicated business logic, which is the primary attack surface and loophole that AI tools cannot evaluate with a high rate of precision yet.
The Future of AI Tools in Blockchain Security
The potential implementation of AI tools in the blockchain security sector is gaining acceptance, with experts recognizing their usefulness for specific tasks. However, the general consensus is that AI tools cannot replace human auditors.
According to Connie Lam, a solutions architecture head at Certik, while AI tools like ChatGPT can help create new things, it’s hard for them to replace human auditors at this point. Presently, manual and specialized security audit tools are still preferred over ChatGPT. However, the development of OpenAI’s technology, such as its transition from the 3.5 to 4.0 version, suggests that further advancements will improve its capabilities.
Importance of Accurate Token Security Review in the Blockchain
Having a high level of accuracy in token security review is crucial in the blockchain industry as it ensures that listed tokens on exchanges meet certain standards of security and reliability. Token security review involves analyzing the smart contracts underlying the token to identify vulnerabilities and potential attack surfaces.
If a token has a security vulnerability, it could be exploited by bad actors, leading to financial losses for investors and damage to the reputation of the exchange listing the token. Therefore, ensuring that listed tokens meet security standards is critical for maintaining the trust of investors and ensuring the overall stability of the blockchain ecosystem.
Inaccurate security reviews, either from human auditors or AI tools, could result in vulnerabilities being overlooked or false positives being flagged, leading to unnecessary delays and potential loss of business. Therefore, having a high level of accuracy in token security review is essential for maintaining the integrity of the blockchain ecosystem and ensuring the continued growth of the industry.
Final Thoughts
Given the facts provided by Coinbase, AI tools like ChatGPT have the potential to be powerful complementary tools for quality assurance checks in the future. However, they cannot replace human auditors yet based on the inconsistency of their results. For now, traditional security audit tools are still superior to AI tools in locating vulnerabilities.
Nevertheless, the blockchain industry must embrace and integrate these tools to encourage further progress in the field. As the industry continues to explore the potential of AI, we can expect to see further innovations and improvements that could enhance the security of the blockchain ecosystem going forward.
Giancarlo is an economist and researcher by profession. Prior to his addition to Blockzeit’s dynamic team, he was handling several crypto projects for both the government and private sectors as a Project Manager of a consultancy firm.
