No discussion with Schmidhuber is complete without the infamous debate at NIPS 2016 https://youtu.be/HGYYEUSm-0Q?t=3780 . One of my goals as a ML researcher is to publish something and have Schmidhuber claim he's already done it.
But more seriously, I'm not a fan of Schmidhuber because even if he truly did invent all this stuff early in the 90s, he's inability to see its application to modern compute held the field back by years. In principle, we could have had GANs and self-supervised models' years earlier if he had "revisited his early work". It's clear to me no one read his early paper's when developing GANs/self-supervision/transformers.
As someone who was heavily involved in AI research in the 1990s, of a Schmidhuber flavored variety though I’ve never interacted with him, I do think there is some underlying truth to his general point. Most of the current theory isn’t actually new and at least some people should know better that are kind of pretending it was invented out of whole cloth recently. But at the time a lot of this was invented, the hardware simply wasn’t capable enough to reduce it to practice — it is why I got out — and a lot of people who are piling into the field are simultaneously disinterested in that history and claiming every idea that crosses their mind as a novel invention.
This is not unique to AI. Many other subfields of computer science have a similar dynamic e.g. databases. I’ve seen people claiming novelty in ideas that were fully proven out in real systems in the 1970s (and abandoned, for good reason) but are oblivious to this fact because if it isn’t trivially discoverable on the Internet then it doesn’t exist. There is still a lot of interesting computer science research that only exists on physical paper, if you can find a copy. Maybe we should be better about digitizing the pre-Internet research but the value of that research isn’t always obvious and the terminology has changed. We don’t give enough credit to how clever some of those early researchers actually were, working from much more primitive foundations.
Having gone spelunking a number of times into the old literature, it never ceases to amaze me the number of times I have found an insight that is neither known nor cited in modern literature. Literally lost knowledge. It is ironic that computer science, of all fields, should suffer from this.
Before people say that he is claiming credit for things he didn't do, or that he invented everything, please read his own paper on the subject:
https://people.idsia.ch/~juergen/deep-learning-history.html
The history section starts in 1676.
Doesn’t he know the Turing Award is really just a generalization of the Fields Medal, an award that actually came years earlier?
Every so often Schmidhuber is brought back to the front-page of HN, people will argue that he "invented it all" while others will say that he's a-posteriori claiming all the good ideas were his.
Relativity Priority Dispute: https://en.wikipedia.org/wiki/Relativity_priority_dispute
We all stand on the shoulders of giants, things can be invented and reinvented and ideas can appear twice in a vacuum.
The fact that the field of machine learning keeps "discovering" things already established in other fields, and christening them with new names does lend some credence to Schmidhuber. The field is more industrial than academic, and cares about money more than credit, and industrial-scale data theft is all in a day's work.
As another commenter said, his misfortune is being in a lab with no industrial affiliation.
If you guys were the inventors of Facebook, you’d have invented Facebook
Why did Google give birth to the Transformer? Because Google created an ecosystem where everything could flourish, while the old man in Switzerland lacked such an environment—what could even the smartest and greatest individual do against that?
As an organization, fostering an organically growing context is like governing a great nation with delicate care. A bottom-up (organic growth) environment is the core context for sustained innovation and development!
He may be a Turing-award worthy researcher (hey he invented LSTM, and even I knew his name!) but modesty surely isn't his biggest strength :))
Oh boy, I sure can't wait to see the comments on this one!
Schmidhuber sure seems to be a personality, and so far I've mostly heard negative things about his "I invented this" attitude to modern research.
I haven't read the article or paper yet, but if the gist I'm getting from the comments is correct, Schmidhuber is generally correct about industry having horrible citation practices. I even see it at a small scale at work. People often fail or forget to mention the others that helped them generate their ideas.
I would not be at all surprised if this behavior extended to research papers published by people in industry as opposed to academia. Good citation practice simply does not exist in industry. We're lucky in any of the thousand blog posts that reimplement some idea that was cranked out ages ago in academic circles are even aware of the original effort, let alone cite it. Citations are few and far between in industry literature generally. Obviously there are exceptions and this just my personal observation, I haven't done or found any kind of meta literary study illustrating such.
If there ever was an example of a terrible personality getting in the way of career, Schmidhuber is most definitely it.
You may have had many brilliant ideas, but everyone makes an abrupt 180 when they see the tip your beard turn the corner at conferences, that can't be a good signal for getting awards.
I'll probably get flamed to death for saying this, but I like JĂĽrgen. I mean, I don't know him in person (never met him) but I've seen a lot of his written work and interviews and what-not and he seems like an alright guy to me. Yes, I get it... there's that whole "ooooh, JĂĽrgen is always trying to claim credit for everything" thing and all. But really, to me, it doesn't exactly come off that way. Note that he's often pointing out the lack of credit assigned even to people who lived and died centuries before him.
His "shtick" to me isn't just about him saying "people didn't give me credit" but it seems more "AI people in general haven't credited the history of the field properly." And in many cases he seems to have a point.