OpenAI’s ChatGPT was good at solving coding problems before its training cutoff date of 2021, but the chatbot struggled to solve problems after that year, according to a study published by IEEE Transactions on Software Engineering.
ADVERTISEMENT
ChatGPT’s ability to solve coding problems dropped by a wide margin if the problem set was outside its training data as language model was unable to even understand the meaning of basic problems, the study noted.
The AI chatbot’s success rates of 89% (easy), 71% (medium) and 40% (hard) for coding problems dipped after 2021 to 52% (easy). For hard problems, the rate fell to 0.66%.
ADVERTISEMENT
The study covered 728 coding problems from Leetcode testing website in five programming languages -- C, C++, Java, JavaScript and Python. The chatbot’s prowess for tested on functionality, complexity and security.
(For top technology news of the day, subscribe to our tech newsletter Today’s Cache)
The research also highlighted that within the Leetcode dataset, ChatGPT was able to code better than humans in some cases, but carried security concerns. ChatGPT was also good at fixing compiling errors in a code but not very good at correcting its own mistakes. One of the researchers involved in the study, Yutian Tang from the University of Glasgow stated, “ChatGPT may generate incorrect code because it does not understand the meaning of algorithm problems, thus, this simple error feedback information is not enough.”
Tang said it was important to understand exactly where these AI models were limited so coders using them could find ways to work around it. “For example, when encountering more complex programming problems, developers can provide relevant knowledge as much as possible, and tell ChatGPT in the prompt which potential vulnerabilities to be aware of,” he added.