Automated testing for software engineering job candidates is widely used today, with many companies relying on such techniques to identify the most talented programmers. But these tests are not without their faults, and the burgeoning field of ChatGPT may add further caution to their use.
Although it is early, there are grounds for concern. Automated coding tests that screen developers for skills could be subject to AI-driven manipulation. That’s on top of manipulation already evidenced.
Generic coding tests tend to be highly inefficient as most are automated and can be manipulated. According to a survey released by skills-based hiring platform Filtered, coding tests are vulnerable to fraud, with more than half the respondents reporting knowing someone who has cheated on a coding test as part of an interview process.
Matters only become worse with the widespread availability of AI-powered tools, such as ChatGPT, which have made cheating on these tests easier than ever before. Examples are arising.
When Jason Wodicka, staff engineer at interviewing cloud platform Karat, administered an interview with ChatGPT, he found the tool was able to generate a valid solution. But the way in which it reached its answer was what caught his eye. Wodicka described this in a series of blog posts on ChatGPT in technical interviews.
“It behaved more like someone who had memorized the answer to this particular problem in advance but did not have the needed skills to solve it independently, which is consistent with how it works,” he wrote.
ChatGPT does not have a model of the problem, only questions, and plausible responses, he said. Moreover, he witnessed “wild changes to its algorithm” and the way that its explanations were misaligned with its actions.
He concluded ChatGPT’s results were very unlike a human solving a problem, and it did not handle probing questions in a way that created confidence in its understanding.
Whether they get the job or not, if candidates were to use the ChatGPT tool for a prescreening assessment and clear that key hurdle, they are hampering the hiring process in many ways. That is per Ravinder Goyal, cofounder and managing director of Erekrut.
First, it defeats the purpose of the test itself, which is to evaluate the candidate’s knowledge and understanding of the subject, he said. Secondly, it undermines the credibility of the test, leading to doubt and mistrust among employers. Thirdly, it could lead to false positives and inaccurate results, ultimately leading to the wrong candidate being chosen for the job position.
Until better solutions are created, it is perhaps safe to say that traditional coding tests will be considered unreliable as a sole indicator of a candidate’s abilities.
So what’s next? For his part, Wodicka sees an AI-driven future with fewer automated coding tests that need a candidate to reach a known solution, and more interviews with a person that test how a candidate approaches, explains and solves problems that have many possible answers.
“I see AI making software development — and technical interviews — more accessible and more human. This is a positive development,” he said. “AI tools don’t remove the need for programmers, they just relieve the cognitive burden of translating ideas into code and shift the level of intent up to a more human level,” he said.
Future technical interviewing will assess the fundamentally human portion of the task. That is “Problem-solving and thought processes required to make machines do new and exciting things,” Wodicka blogged.
In his experience, the future of technical interviewing will hinge on subtle shades of meaning — and ultimately be more predictive of on-the-job performance — than an automated coding test that produces a binary “pass/fail” result. Over time, ChatGPT may be just another tool in a typical developer’s tool box.
“It’s also that nuance that renders a candidate’s use of ChatGPT somewhat meaningless — in fact, we allow candidates to use resources like Stack Overflow or Google during their interview, just like they have access to those resources on the job. I don’t see ChatGPT being any different in this regard,” Wodicka added.
Meanwhile, in the face of concerns over ChatGPT manipulation, automated coding assessments are likely to continue to find greater use. They help talent teams and hiring managers alike.
The automated coding assessment is still the first step in many technical recruitment processes, as it helps evaluate the engineer’s understanding of fundamental programming concepts, such as data structures and algorithms, and their ability to write code that is efficient, accurate and easy to debug. Automated tests can also save time, as they allow a large group of candidates to be tested at once.
“Currently, hiring for mission-critical roles is paramount, and automated coding assessments can speed up the process and assess a large number of candidates simultaneously –- reducing the amount of time and effort required to manually assess,” said Sujit Karpe, CTO and cofounder of skills assessment software, iMocha. With a large candidate pool, this would be a “must have” for the recruitment teams, Karpe continued.
Pratik Vaidya, managing director and chief vision officer at Karma Global, a tech-enabled HR and compliance organization, seconds this opinion.
Evaluating candidates’ coding skills with tech-friendly, hands-on programming tests is the surest way of getting the right tech personnel on board, Vaidya said.
“Coding tests are used by many corporations to determine the caliber of right candidates, especially for technical positions,” he said, citing a high degree of fabrications possible in an alternative source for evaluation — that is programmers’ resumes.
Automated tools, including coding tests, can be useful in the screening process, but they should not be relied on as the sole means of evaluation, said Peeush Bajpai, CEO and founder of SpringPeople. And others concurred.
“Companies should adopt a holistic and human-centric approach to hiring that takes into account not only a candidate’s technical skills and experience, but also their cultural fit and potential for growth within the company,” he said.
Code reviews, technical interviews, behavioral interviews and work sample assessments represent various important means to gain a comprehensive understanding of a candidate’s abilities and fit for a role and a company,” Bajpai said.
Chinese
Japanese
Korean
