I'd be curious what an LLM would be able to accomplish if it was given all pre-modern texts we have access to. Would it be able to come up with ideas we know as modern (political theories, philosophical ideas, scientific theories, etc.)?
I have a hard time believing that if we fed an LLM all prehistoric speech uttered from humans that no matter what, it would never escape the paradigms of those people. If that is the case, than would relying on LLMs just get us stuck in our own paradigms and prevent true "progress"?
Even if they can't, individuals are prone to inspiration from an LLMs attempt. Worth giving it a shot at least.
For some reason my mind maps concepts from chess onto the real world and the other way around. The novel AI ideas in chess are usually combinations of several... well... bad ideas. People usually take on one bad idea at a time and try to make it work. If they succeed it is quite surprising because objectively it was a bad idea.
Our thinking is a lot less error prone than that of the LLM's but we have to study for years to absorb prior art that LLM's mostly receive at birth by osmosis. It won't be able to take on the truly stupid ideas but it can combine large numbers of the somewhat stupid.
Like a chess position with lots of possible moves followed by lots of possible responses.
Taking a step back. Define "novel".
I have the idea for an induced draft umbrella. Stick a fan at the top under an opening.
Is that idea novel? I haven't seen it anywhere, it's just something I came up with. But it's not entirely novel, I'm just borrowing the concept of a fan, and an umbrella.
I don't feel this is entirely out of scope for what an LLM could describe in words?
Could rolling dice with matching phrases generate novel research ideas? (ok the metaphor is slightly off, but not a lot)
Looks like @sama's o1 is killing PhD science questions. https://x.com/sama/status/1834283100639297910?t=iCRehNoBofMP...
Would you be able to tell if it did? There could be an obscure document in the training set that contains the idea. It seems like a very hard problem to definitively detect whether a concept came from the training set.
The difference between novelty and hallucination is feasibility. With integrated critique and feasibility checks you can eventually map the hallucination space into novelty.
Yes.
LLMs have helped me generate 1 novel discovery: that every top 10 programming language has a single creator (https://pldb.io/blog/aSingleCreator.html).
They also helped me generate this map yesterday (https://pldb.io/blog/whereInnovation.html), which is the most comprehensive map of programming language creation and software innovation ever created.
shuf -n 3 /usr/dict/words will also sometimes generate novel research ideas.
[flagged]
[flagged]
No.
Perhaps we can use LLMs to invalidate patents. Want to check a patent? Just download the LLM model from before the issue date, then ask the LLM to produce the work. If it succeeds, you have invalidated the patent because the work was not novel.
An LLM is like a well read college student with a nearly photographic memory that sometimes mixes things up.
It's great for bouncing ideas off of and getting feedback on them. And yeah, it might product "novel ideas" by mixing and matching existing ideas, but LLMs will never create truly novel ideas. Not in their current form.
The paper didn't really answer the question sadly: their conclusion was just that humans rate LLM answers as more novel than human ones, but less feasible.