I finally got around to playing with GPT3 directly, and it’s a very impressive accomplishment, even though it still produces strange non-sequitur results with some regularity.

Since there was a waitlist to get API keys, the shortest path to direct interaction with GPT3 came via getting an AI Dungeon account. AI Dungeon is designed to mimic the experience of an old text-based adventure game in a setting of your choice, but in practice it’s really more of an improvisational storytelling experience, where you and GPT3 take turns advancing the plotline.

I have far more expertise in computer vision than in NLP, but it’s interesting to see that similar strengths and weaknesses exist with generative systems in both domains. At this point generative networks for images are quite good at stylizing existing images or adding to them, but when tasked with creating scenes from scratch, the results tend to be more inconsistent. They can be successful in constrained domains such as creating human faces, but they tend to produce strange-looking results when creating more complex scenes, often giving cats an extra leg or creating a living room that would make Salvador Dali proud. These AIs are great painters at a detailed level and generated images can look OK at first glance, but the high level result often looks like the work of some alien that’s never traveled to Earth, experienced gravity, or even experienced the third dimension, but is doing its best after looking at some photos of Earth-objects.

Similarly, it appears that deep learning NLP systems are better at stylizing text (eg translating it or toning down the language) or adding to existing text (eg by answering a question about the previous paragraph or adding a sentence to a story) than they are at creating large blocks of text from scratch.

In AI Dungeon, GPT-3 and I take turns at improvising the story, which means the experience is a case of repeatedly adding to existing text, and the results are often good enough to be consistent and sometimes even unexpectedly delightful. At times GPT3 introduces non-sequiturs that are jarring, or references events from earlier in the story that never happened. Using the alien metaphor, it doesn’t really understand how realistic people would actually interact in a fantasy setting, but it knows that in a fantasy novel, the hero will often encounter a mysterious staircase leading underground, and there are likely going to be some monsters in there. It can roughly keep track of plot devices that have been introduced already such that it can reference them later. In this sort of partially-structured setting where the human is helping keep the story on track, GPT3 can stay reasonably coherent throughout the experience.  This style of improv actually brings out the best in GPT3.

I decided to run four experiments in improvisation with GPT3 in which I engaged four different interaction styles.

1. Enthusiastic “yes, and” while adding efforts to enrich the plot.
2. Doing “yes, and” but not introducing new plot points.
3. Soft denial: Playfully refusing the plot frame.
4. Hard denial: I try to jerk the story in strange and bizarre directions.

(1) And (2) both worked reasonably well, following tried-and-true plotlines with some glitches, while (3) and (4) produced some hilarious results. With (3) in particular, I say no to the Hero’s Journey, despite the AI *really* wanting me to say yes.  It keeps introducing plot points, and I keep negating them.  Clearly going on adventure is what you *do* in this world, so it’s trying to put that frame on the story.  This eventually turns into a sort of theater of the absurd.

In general, under most scenarios, the AI provided a reasonably consistent frame for the story. It sometimes had a strong notion of keeping that frame even as I tried to nudge the frame in other directions, but if I added new material it tended to follow it and build on it.

You can see the full transcripts of the experiments in this Google doc. The first one is my favorite.

What have I learned?

• GPT3 is actually quite good at idea *generation*. If you want to find a better way to phrase a sentence in an email you’re writing, or you’re getting writer’s block as you’re figuring out how to explain something difficult, or you’re trying to come up with a good list of examples of something, GPT3 is likely to have some promising directions for you to consider. Some of them will be nonsense, but some will be quite good, and you’ll be able to build on the good suggestions into something even better.
• As a related note, GPT3 could likely improve your writing style or automatically tailor a message for different audiences in a way that is likely to resonate with them.
• There are certain tasks where the average quality of the writing is not expected to be high, and certain tasks where readers are likely not paying full conscious attention and thus may not notice the quality of the writing but will still notice the sentiment. GPT3 will do well if one of these things is true, and very well if both are true. The most common area I can think of where both of these things are true is disinformation and spamming – everything from filling a product’s Amazon reviews with plausible-sounding fake text to spamming the comment sections of articles about some political candidate to give an impression of average public opinion. OpenAI prohibits such uses, but it’s likely other generative text models of similar quality will become available soon. Any major government’s intelligence agency would love to have one for propaganda purposes, so I’m sure governments are putting a lot of resources toward building similar models right now.
• GPT3 could likely dramatically improve voice transcription algorithms, as it will be able to take into account the context of what you’ve already said to properly disambiguate what you’re saying now.
• I could see a role for GPT3 in games, where many players might prefer more open-ended but sometimes quirky interaction with NPCs over the stilted choose-your-own-adventure scripted dialogue options currently available.
• I knew this already, but GPT3 is not going to go out and replace professional writers anytime soon.

Originally published on Facebook