https://arxiv.org/abs/2306.08997
“arXiv will not consider removal for reasons such as journal similarity detection, nor failure to obtain consent from co-authors, as these do not invalidate the license applied by the submitter”
The submitter can mark the paper as “withdrawn” but it will remain available
Recent and related:
No, GPT4 Can’t Ace MIT - https://news.ycombinator.com/item?id=36370685 - June 2023 (120 comments)
So lame they focus on gatekeeping the exams and crying about permission as opposed to challenging the paper’s shit methodology.
Phew, I'm just happy that OpenAI first gathered consent from all data sources on which they trained GPT4.
Here's a more rigorous take on evaluating the output of GPT models on math:
Putting papers and code on arXiv shouldn't be punished. The incentive to do this is to protect your idea from getting scooped, and also to inform your close community on interesting problems that you're working on and get feedback. ArXiv is meant for work in progress ideas that won't necessarily stand the peer review process, but this isn't really acknowledged properly on social media. I highly doubt the Twitter storm would have been this intense if the twitter posts explicitly acknowledged this as a "Draft publication which hints as X." But I admit that pointing fingers at nobody in general and social media specifically is a pretty lazy solution.
The takeaway IMO seems to be to prepend the abstract with a clear disclaimer sentence conveying the uncertainty of the research in question. For instance, adding a clear "WORKING DRAFT: ..." in the abstract section.
There is an insightful video review of this paper by Yannic Kilcher: https://youtu.be/Tkijsu129M0?t=30
TL;DW - Their approach is to sequentially test methods, moving on to the next if one fails. However, this strategy is flawed as it requires ground truth, particularly for multi-choice answers. The analogy could be made to continually rolling a dice until landing on six. Similarly, if a question has four potential answers, the model merely has to attempt four times to stumble upon the correct response. And then they report 100% success rate.
Well, someone's in trouble. If a faculty member allowed their name to be put on this thing then that's on them, if they didn't then whoever did the submission has to go. Also all the papers of all the co-authors who allowed their name on it are now suspect for me and should be investigated seriously. Sure, if the fact is that they said "don't put my name on stuff that you publish unless I have specifically agreed" or "don't publish this thing, it needs work" and then that message got lost - they are off the hook for me. But there needs to be a look at the culture of the labs involved even in that case. That's brutal because as I say there is a possibility that they are victims of someone going over the top, but unfortunately I think MIT has to do it.
In addition, for me there is a much wider ethics issue here. How many papers can really be checked properly by someone claiming authorship even if that author isn't really contributing? I can read a paper like this in about a fortnight because I have a job and a life. A faculty member also has a job and a life - they are doing admin and teaching as well as research. So to me it's impossible for someone to check 25 papers or more a year.
I am seeing far higher counts than this by many academics.
But, this is an extremely conservative threshold in my opinion. When I have contributed to scientific papers it has taken me at least three months of solid work each time. Often these papers get rejected (rightly) and then have to be substantially amended (or occasionally just abandoned) I am not that talented for sure, but I really find it hard to credit that anyone with an actual job (so not a post-doc or a student) can contribute to more than one academic paper a year. Potentially two or three if there is a confluence of papers getting ready for print... but not on a sustained basis.
There are two solutions. Every university and research institute needs to investigate all the publications of academics with high paper counts per year. This is a red flag. I definitely think that if folks are in the top quartile in a department it needs to be looked at carefully.
The other solution is that no academic publishing venue (conference or journal) should accept more than one paper per year from any author.
Is there a link to the original story? Drori's homepage at MIT seems to have a "Certificate Error" (probably 405: Revoked Diploma.)
I assume the paper itself is long gone.
Not condoning what the author did, but it’s amazing how little the questions answered in the paper have to do with real life software engineering.
Yes, it shows fortitude to pass MIT’s exam, but beyond that, a lot of stress for a test that will soon be forgotten after their undergraduate degree.
People overreact to this paper because it's from MIT. Lots of crappy papers out there, this is just another one. MIT is not special, they have a lot of average people too.
What I fail to understand is that these authors were presumably okay sending the paper to Neurips (a top tier conference). I am assuming this based on the paper being in Neurips submission template. And now when the cat is out of the bag in open they are denouncing it.
One can only dream this level of analysis and falsification would be the norm in academic publishing.
Sadly, this is an extreme outlier for now.
[dead]
> We want to emphasize that all the student authors in this paper worked really hard on what could have > been a very interesting and valuable paper had the data been collected with consent. The many > problems with the published work were not the fault of the students.
So what did the student authors work "really hard on" if not the same data that was collected without consent? Either all student authors are at fault for working on a paper based on data collected without consent or none are. It's not publishing the paper that is the problem here.
Such despicable behavior from everyone involved.
1. Professors and MIT students attempting to gain fleeting fame riding on the bandwagon without trying to do deep work
2. Professors at MIT getting upset and lash out that GPT-4 can now get a MIT degree devaluing said degree.
What has become of academia these days ?
>We want to emphasize that all the student authors in this paper worked really hard on what could have been a very interesting and valuable paper had the data been collected with consent. The many problems with the published work were not the fault of the students.
I appreciate the clear stance that MIT has taken regarding where the responsibility lies in this situation. I think some people are missing the context that many of the authors were undergraduate/early career. Research is an iterative process and every paper has to start somewhere. I don't agree that the paper should be withdrawn because Arxiv is not technically a publication, but I also wouldn't consider the paper properly peer reviewed. Teachers own the copyright the exam material. I was taught copyrighted material can't and shouldn't be used as part of an eval dataset.
The followup by three other MIT ('24) seniors is a great peer review.
https://flower-nutria-41d.notion.site/No-GPT4-can-t-ace-MIT-...
No, GPT4 Can’t Ace MIT - https://news.ycombinator.com/item?id=36370685 - June 2023 (120 comments)