So you’ve probably heard of OpenAI’s amazing DALL·E 2 text to picture generator. (I won’t shut about it!)
Note: None of the pictures in this article were created with DALL·E 2
The wackiest AI generated memes aren’t coming from DALL·E 2. They are mostly coming from a different (and computationally weaker) model called DALL·E mini created by Boris Dayma.
Confusing, right? Well, OpenAI thought so too and the end result is the front-facing website where you can generate pics from your web browser is now called Craiyon.
That said, their open source AI model is still technically called Dall-E Mini. For now.
But why tho
So why do people use Dall-E Mini and other open source AI projects even when they have access to DALL·E 2 & Davinci?
I can run it myself locally for free*
I can create an API to use it in an automated way easily, OpenAI doesn’t yet offer a public API for DALL·E 2 (I’m sure it’s coming though)
No strict censorship, it’s impossible to use DALL·E 2 for many uses because so many things aren’t allowed, including blood, guns, sexy terms, famous characters/people & religious symbols. But now I can finally use all of those at once! Uh… not saying I will, just… I mean, I just want to be able to. Uh, ethically and stuff. Yeah.
Can use commercially, unlike DALL·E 2 (for now)
* you know what I mean. The graphics card and energy to run it definitely aren’t free, but you aren’t reliant on someone else’s slow-ass webserver or extra fees.
Running my own version locally, no servers needed
So I dug in and got it running locally under Ubuntu with an NVidia 3090 and did some tests. Getting every library CUDA enabled and playing nice with each other was kind of a headache.
Note: Many of these tests don’t require a 3090, I don’t think anything I’ve run has used more than 50% of my video ram.
Not great. But hold on – there is a more powerful data set Dayma has released called “Mega”.
So it’s still Dall-e Mini, but we’re using the “Mega-full” weight data which requires more memory and time to generate stuff. After enabling that, the pictures are better:
As you can see, it now understands “an orange” to just be one and the faces are far less horrific.
Note: I ran these prompts a few times, the results were consistent
As far as I can tell, Dayma and friends are still training models so looking forward to Super-Mega or Giga or whatever, even if my home computer will likely no longer be able to run them soon.
Switching to AI text generation for a second, I’ve also gotten GPT-J 6B running locally and am currently working towards getting EleutherAI/gpt-neox-20b running. (the closest open source thing to GPT-3 Davinci out there?)
To run gpt-neox-20b locally will require two 3090s in the same computer or buying a $30K graphics card. I’ll try the 3090s I guess.
So how’s the quality of Mega?
Well, nowhere near OpenAI’s offering but it’s definitely getting better.
Pics I made:
Seth, what are you even doing with all this AI stuff?
First, it’s just addicting to play with and fun trying to think of novel ways to use it.
Open source “State of the art” is changing regularly as new models and datasets are released from all over. It’s going to be a crazy decade, machine learning & AI is affecting EVERYTHINGEVERYWHERE, it’s not just just for putting animals in spacesuits and generating erotic Willow (1988) fanfic.
I have some projects in mind but.. yeah, we’ll see.
Oh, for making it all the way to the end, have another orange test but this time I generated it using ruDALLE-E with the Malkovich dataset, yet another AI project to keep an eye on:
This morning OpenAI has changed the rules – we can share pictures with faces now! To celebrate, I figured I’d have DALL·E create a real life photo of Castlevania hero, Simon Belmont. He should look something like the above picture, right?
I’ll just enter the name and the style of photo I want and with the magic of AI we get…
…some bikers and Neo wannabes. DALL·E has been programmed to ignore (?) famous people and I guess that extends to fictional characters as well. Had poor results with Mickey Mouse and Shrek too.
It will never closely duplicate a celebrity face or anybody’s face for that matter, it will only output greatly “mixed” things. (this is a legal/ethical choice rather than a technological limitation I believe)
So the secret is to forget the name and craft a worthy sentence to describe the target in textual detail. Actually, I get slightly better results including the name so I’ll keep that too.
As a representative of lazy people everywhere, I’ll use OpenAI’s GPT-3 DaVinci to create the description for me. (Their text AI tools have no qualms referencing famous people or anything else)
Perfect. Now we feed the AI created description into DALL·E and get…
Well, much closer. You know, we should have added a whip.
The quality stands up pretty well at full resolution too:
Being a skilled writer (unlike the person typing) will probably result in better images. All those pages of boring descriptive prose in The Hobbit would create masterpieces!
I’ve been dabbling with creating creature sprites/concept art to fit existing games (Like Dink Smallwood) but inpainting techniques have not been producing good results yet. Still learning and playing with things.
Inpainting is a technique where you start with a picture (in this case me) and ask the AI to replace a specific part of it. My first test was to try to add a second nose to myself, so I could smell things better. (All pictures/modifications created with OpenAI’s DALL·E 2 – all “original” pics used for inpainting are owned by me)
Um, not what I was going for.
Let’s stay away from the face and try to change Akiko’s dress. I use the brush to remove the area below her neck, slightly larger than the area of her dress.
Incredible. It also filled in small parts of the person/stroller that is out of focus behind her as needed.
Another test. We’ll replace my son’s baseball glove with a pizza.
These results aren’t as convincing (I should have given it more arm to replace for a more natural pose I think), but I do appreciate how it was able to add missing detail to his hair after the glove was removed.
Well, you know what I have to do…
Hmm. Not great. I guess changing my body into one with muscles is just one step too far for today’s technology. My guess is Dall-e actually doesn’t know much about bare skin (nipples and belly button seem to be missing?) due to using a censored training set to stop it from.. well, doing certain things. I’ll bet a suit works better, let’s try that.
This time I did all three of us is one go. Not perfect – I wonder if it would work better if I did it in three separate operations, one for each person? Hrm. If you’re curious how I set which areas it could fill, it looked like this:
Random Dall-e 2 test pictures
More random pics with the prompts used, some prompts created by friends.
I’ve been interested in AI and computer generated media forever. As a kid, I created a stupidly naïve “AI” on my Commodore 64. I just programmed text responses to hundreds of inputs. “Hello” gave back “Hello, how are you.”.“I’m fine” gave back, “Great, me too.” and so on. I proudly showed my parents how smart my computer was.
A hollow illusion then, reality today. From Eliza to Deep Blue, Watson to Siri, between Deepfakes and GPT-3 (and friends), it’s all coming together to change our world at a blinding pace.
Do you know what two computer things I’ve been playing? Eldin Ring and OpenAI’s GPT-3 Playground. I’ve spent about $70 on each last month. Does spending cash on AI seem weird?
It’s no exaggeration to say playing around with it is one of the most engrossing and creative experiences I’ve had with computers to date.
OpenAI Inc/OpenAI LP are the big dogs in the field and carefully police usage. You can probably think of dozens of ways this technology (both the text creation and text to image tech of DALL-E) could be used for evil, spam, and misinformation.
Recently I’d been playing around with DALL·E mini and was in the process of setting up my own local server to try to allow higher quality work when I was granted access to the holy grail: DALL·E 2.
Let’s have the AI generate images for my old text-only game
In 1989 I started work on the BBS door-game Legend Of The Red Dragon. It’s a text game. What would happen if I took text from that game and asked AI to draw pictures for it from only the game text?
Let’s try its opening text as a prompt:
Huh. Looks like it could use more direction. Let’s add “In the style of 3d rendered adventure game.”
Not bad. How about the Red Dragon Inn? Wonder how long these text prompts can be, let’s try the whole thing.
Well, the raw weird prose doesn’t seem to work that well. It isn’t given enough information to know it isn’t modern day. What if I change it around a little bit… (in theory you could use AI to rewrite the sentence to not be 1st person and add keywords to help with theme and historic era)
Note: I blocked out a woman’s face, I thought the rule was we can’t show them – but maybe we can, need to check the dalle-2 rules again.
Let’s try a different visual style.
Hmm, it’s obvious that I could get better results if I took more care in the prompt text, but nifty anyway.
I could see it being fun to play old text games with AI generated images. I don’t see how to control Dall-e 2 with an API at the moment otherwise I might try modifying an infocom interpreter to automatically fetch them during play.
The 10-20 seconds to generate an image wouldn’t be fun to do it live, but how cool would it be to see “a bucket is here” and it appears/disappears from the image as you play the game?
The big problem is uniformity of style – but there are some tools dealing with this I haven’t played with yet. (starting with an uploaded photo, for example)
Let’s use AI for everything
How about using AI to help generate a brand new story, then illustrating it too?
Here is a test. The text with the white background I typed. The text with the green background was generated by AI. (Specifically, OpenAI.com’s text-davinci-002 engine)
Ok, we now have two characters. Now, we keep this text and continue with more prompts, interactively pulling out more details. We can always undo and try different prompts if needed.
Ok, now let’s send these descriptions to DALL·E 2 to create AI-based visual representations of the story the AI created. First let’s do Feival’s house:
Not bad. I like the sixth image because I can see the treasure map on the chair.
Let’s do the Thimble description next. This time I’ll add “Realistic photo” at the end to specify the kind of image we want.
Hmm. The cane didn’t seem to quite make it. This story seems like it might make a good children’s book. Let’s add “by Richard Scarry” to get that type of art style.
Definitely got a children’s book style! The cane is now in every picture. I like this style.
I can ask for more variations in this style:
Writing a story with the characters the AI created
Hmm. Ok, we’ve got our stars, let’s have the AI write a story using them. I’m adding “Write an amusing children’s book about the above characters with a twist ending. Chapter 1:” to the end of the text we’ve already generated. (Again, green parts were created by the AI)
Well, uh, it’s a story. There are ways to coax out longer and more interesting things but this is fine for this test. Just for fun, let’s see if we can create artwork for the amazing battle scene of the giant mouse trap catching cats. I’m going to cheat and use my own descriptions for the prompt.
Uh, ok, obviously that prompt isn’t great as it looks like a cat is being hit with colored pencils. I’m showing you my failures, not just the good ones here! Let’s forget the mouse and just focus on the cats and the mouse trap.
These are pretty wild! Some of the faces are .. I don’t know, it may have actually tried to drawn them injured by a mousetrap, in retrospect this could trigger unintentionally gory results, especially if I used ‘photorealistic’ as a keyword.
Let’s move to safer ground and create an image for the happy (?) ending.
Random fun with DALL·E 2
These are just various pictures created with DALL·E 2 and the text prompts used. It’s very interesting to see AI interpretations. Special thanks to Kevin Bates for brainstorming these prompts with me. It’s addicting, I can’t stop!
Note: The six images pic shows the prompt used, then I do some “closeups” of the more interesting ones. It’s really fast to do it this way, sorry it’s not nicer so each little pic is clickable.
Well, I’ve put way too many pictures in this post so I’ll end it here. The AI models I used are top of the line and have many usage restrictions, but it’s only a matter of time before similar things are available to everyone – Good or evil, unrestricted. I’m simultaneously excited and worried.
If you want to play around with generating images yourself, try DALL·E mini. Its output isn’t as impressive but it’s still fun and interesting to play with.