This is so cool! I have been wanting to see something like this for a few years now. I tried making a demo of something similar (but much more primitive) in Unity back in 2021, but small transformers weren't good enough at the time.
Is there any way to protect against prompt injection here? Looking at the architecture I am thinking it would be possible for users to tell an agent what to do directly by tricking them.
This isn't really a criticism, I think it's actually a cool feature. It might be a fun premise of a game where you know you are in a simulation and can manipulate the NPCs around you.
Once your NPCs get good enough, you could connect them to the real world (via APIs) so they could do goal-oriented sensing and actuation there too. I'm fairly sure it would be wonderful for all involved.
its interesting, but given the quality issues with this sort of tech, i can see it lowering the bar for games even further.
can you imagine trying to QA this?
in any case... getting this to go anywhere /for anyone else/ imo would require a tight native code library without network dependency, as others have said.
building on what you have looks like and awesome project though, and i wish you the best with it.
Pretty neat. I'm creating an isometric, pixel art city builder game, though I imagine this wont scale to 10K or 100K units?
A few thoughts:
1. It's interesting, but realistically I'm unlikely to get into the Java code where all the interesting logic is
2. My impression of the code is that all the interesting stuff is in World.java [1] and the many models [2]
3. I don't see obvious prompts in here. I'm guessing they are somewhere, though also built dynamically (maybe mostly through string concatenation). In my experience it's essential to be able to see and understand the prompts, both abstract/template prompts and the instantiated prompts and responses. Also, having done it both ways, I find that keeping prompts separate from code is helpful. They feel different and you iterate on them differently.
4. You should be wanting to get to a place where most of your changes are prompt-driven. That is, you may be changing code, but only because you are refactoring your prompts in some qualitative way. With some good DSL you can even support qualitative changes, like changing the chain, or making a prompt multi-stage.
5. My intuition is that an important feature of GPT to these applications is the ability to apply multiple frames to the environment. Which is a fancy way of saying many different prompts that include different system prompts and different purposes. But I can't really find the prompts here so I don't know what's happening.
6. The paper this was based on was really inefficient with calls to GPT, costing according to the paper thousands of dollars to run an in-game three-day trial. I think you're going to want to deal with that right away. You want to move more execution out of the LLM (for instance by having the LLM come up with higher-level instructions that are carried out by simpler algorithms). Also you probably need more chattiness to allow the LLM to indicate what information it wants, instead of providing the LLM with all the subjective information a character has.
Anyway, a few thoughts, good luck!
[1] https://github.com/nickm980/smallville/blob/main/smallville/... [2] https://github.com/nickm980/smallville/tree/main/smallville/...
I'm seriously looking forward to the first game that has a bunch of animated Tay chatbots running around.
can you link to the code where the prompts are generated?
Neat. Will have to have a play with this on the weekend.
Uh... is there some way to use this without connecting to a server? Like, for a game that can be played offline?
Finding a way to make the machine learning piece a completely self-contained library that can be shipped at scale to run on individual computers is the big hurdle to making AI like this practical for games. If I have to rely on your service staying up for my game to work, that's an unacceptable supply chain risk.