An expert of Indian Institute of Technology Delhi (IIT-D) has punched holes in the IIT-Madras report submitted to the Supreme Court at the behest of the National Testing Agency (NTA) on the National Eligibility Cum Entrance Test (NEET-UG) 2024 paper leak issue, which claims that there were “no abnormalities” in the way marks and ranks were distributed for 23 lakh students who appeared for the exam.
The IIT-Madras report had analysed data of number of students who appeared versus their score at a national level for two years — 2023 and 2024 — and stated that it showed a bell curve. It also stated that the distribution of marks is ‘quite normal,’ on comparing two years of data.
The bell curve plotted by the IIT-Madras team was analysed by Vishal Vaibhav, Assistant Professor, Optics and Photonics Centre, IIT-D for the petitioners in the NEET 2024 case and is going to be submitted to the Supreme Court for the July 18 hearing. The lead petitioners who have taken on the NTA for alleged irregularities in NEET 2024 are Vanshika Yadav and Shivangi Mishra.
The report submitted by Dr. Vaibhav, a copy of which has been accessed by The Hindu, states that this test proposed by the IIT-M team does not qualify as a valid statistical test. “The recipe proposed by the IIT-M team does not rise up to the level of a statistical test because it lacks in the requisite specificity to aid decision making,” Dr. Vaibhav’s report states.
Logarithmic scale
From the bell curve graph submitted in court, Dr. Vaibhav could extract raw data accounting for approximately eight lakh students. “On plotting this data on a linear curve for 2023 and 2024, there were no major abnormalities seen as submitted by IIT-M, but when we plot it on logarithmic scale, which is like using a magnifying glass to see the data comparison, large gaps are picked up at extreme ends of the curve. On the extreme right end, we find more students scoring higher scores in 2024 as compared to 2023. It is evident that a logarithmic plot is better suited to visualise rank inflation, the one that IIT-M has not used. The current method used by IIT-M lacks specificity to pick up anomalies and is pseudo-scientific,” Dr. Vaibhav said.
Editorial | Can of worms: On NEET-UG 2024
The petitioners have alleged that it was suspected that anywhere between 30,000 to 40,000 students have been ‘force fitted,’ as gaining higher scores, leading to them getting better ranks and lowering prospects of deserving candidates getting MBBS seats.
Next, in a report to be tabled in the Supreme Court on July 18, Dr. Vaibhav ran experiments on the extracted data sets, where students who had scored between -80 to 420 marks, were randomly assigned to sections between 421 to 720 marks by artificially inflating their marks and ranks using random number generators provided by a standard statistical software MATLAB R2021.
“These simulations were run to observe how inflating ranks would bode for the bell curve originally plotted by IIT-M,” Dr. Vaibhav explained.
“It was shocking to notice that when simulations were run, ranks and scores that were inflated for 10,000, 20,000 and later even 30,000 students, the bell curves plotted for all the four data sets, one original and the three other simulated ones look the same,” he further said.
Petitioners have submitted Dr. Vaibhav’s report as a counter to the IIT-M report submitted to the Supreme Court. The counter states that it is not precisely clear what features were identified in the data set used by IIT-M to assert that it is free from abnormalities. “A simulation of rank inflation size for 20,000 students cannot be brushed off as insignificant in the assessment of the integrity of a competitive exam,” the counter states.
‘Flawed algorithm’
Dr. Vaibhav stated that the current algorithm used by IIT-M to draw conclusions that there were “no abnormalities,” were flawed.
On the difficulties faced while extracting raw data, Dr. Vaibhav stated in his report that the NTA has not provided public access to student’s score data in any usable format. “Our task of carrying out an independent analysis would have been much easier if we had access to the data from last five years as well as 2024,” he stated.
“If raw data for all the years of students and their scores is made public, we can open-source analysis and pick up any possible abnormality in rank inflation, and that can further be used to investigate for any fishy patterns, including investigating centres where such candidates took tests, their pattern of answering questions and so on,” he further said.
The petitioners have submitted in court that All India Ranks (AIR) for same scores have increased by three to four times as compared to NEET 2023 results. They have also alleged in their counter that the IIT-M graph is manipulated, and analysis is incorrect.
“Students worst affected are those who scored between 580 to 720 marks. Total number of students who obtained marks in these upper ranges was unduly high for NEET 2023, indicating that question paper was leaked on a large scale and numerous candidates [approximately 30,000 to 40,000 or much more] benefited from it,” a reply in the court by the petitioners states.
In 2023, only 13,291 of 20.38 lakh students, which is 0.65%, scored between 620-720 marks, whereas in 2024, this number was 58,263 (2.49%) of 23.33 lakh students who appeared.
The petitioners will make three prayers to the Supreme Court in the July 18 hearing — conduct a retest for top 20% students in each category as total MBBS seats are 1.08 lakh and these are the top contenders for the same or conduct a retest for the qualified students who are approximately 13 lakh students, or conduct a retest for all.
