Like a lot of people, I’m a sci-fi fan. Spaceship/vehicle concept art is one of my favorite genres to browse through anywhere. The A.I. in Disco Diffusion has to have some hidden gems I can dig up with creative prompting, right? Let’s see if I can find some.
TLDR: I’m just a beginner in A.I. art and am NOT an A.I. engineer or programmer. Everything in this article is my best guess of how Disco Diffusion works by running experiments. With that disclaimer out of the way, prompt syntax can make a huge difference in the image output. I don’t know the specifics of how or why quoting words separately, separating by a comma or a pipe, or using entire phrases makes a difference, but it does.
I tried using an initial image which was a spectacular failure. Initial images can be helpful, but did not work for me at all during these experiments.
In the end, a regular sentence with some specifics produced better results than separating terms out on their own. Could have saved myself a lot of time by just going that route first. Ha ha ha!
Article: To kick it off, I figured I would go quick and dirty with “a futuristic spaceship sitting in a hangar”. Super vague, lots of freedom to build from really whatever images are trained into the CLIP model. I’m using the default models, but NOT all of the default settings in Disco Diffusion. Just some small tweaks here and there to experiment and here’s what I got.
Pretty cool concept! Definitely looks futuristic, but I’m not really seeing a spaceship in there anywhere. Maybe a service robot, but no ship. Also, to give you an idea of what I’m trying to work towards, check out THIS picture by Gary Sanchez on Artstation.
I decided to go even simpler and just put “spaceship”. I didn’t let this one finish because I could clearly see the direction it was going, and didn’t like it.
Then, I got an idea to see if I could use a hashtag as one of the words. For this next prompt I used “#spaceship, artstation”. The result was surprisingly good! Here’s what I got.
Syntax Experiment: I realized here that I’ve seen some people only use commas to separate words, others use commas and quotes, and I’ve also seen pipes “|”. I decided to use the same seed number for the rest of the images, and different prompt formats to see if anything changes. This time, I used ”#spaceship, artstation” (changed the quoting structure) with the same seed, and got this.
It looks like quoting each individual word, and separating those words by a comma does make a difference. I have no idea how that changes the way the A.I. processes the prompt, but it obviously changes the output in a significant way. Next, I’m going to separate the words with a pipe. For this one, I used “#spaceship | artstation” and got this.
This one has the same overall structure as the prompt with the comma separator.
One last test was removing the hashtag to see if that changed the image in some way. Yes, it did. Using “spaceship”, “artstation” gave me this.
I next wanted to verify I would get almost identical output with my current settings so I circled back to the first prompt with the same seed. Again, I used “#spaceship”, “artstation” and got this.
As I said before, since the initial noise is processed slightly different by the diffusion engine, the final image is also slightly different. Time to try some more drastic changes to see if I can get what I want.
Out of the three images I have so far, I like the one using just comma separated words. Even though I have no idea how/why it processed the syntax in that specific way, it’s as good of a starting point as any.
Prompt Engineering: Moving forward, I have seen other artists adding the words “unreal” or “unreal engine” to give the image more of a three dimensional look. Unreal is a video game engine that produces fantastic real time 3D graphics for, you guessed it, video games. Starting off, I’m going to add “unreal engine” to the prompt and see what happens. Specifically “#spaceship, artstation, unreal engine” and here’s what I got.
This has more of a spacestation feel to me, but still looks pretty nice. I want the ship to be more centered in the frame along with the focal point though. I’m now thinking I should have used the image generated with each word in quotes because it has a more central focus. Easy enough, I’ll just add “unreal engine” to that prompt and see what happens. Using “#spaceship”, “artstation”, “unreal engine”. Here you go!
This image is definitely more aligned with what I was imagining. The spaceship and focal point are centered in the frame. This allows me to crop the image in multiple ways to make it more visually appealing later on. Even though I like the layout, framing, and focal point, I’m still not happy with the overall look.
One of the easiest ways to get a specific “look” is to use an artists name in the prompt. Before I do that, I want try some more techniques to see if I can push the AI towards what I’m looking for without blatantly telling it to look at specific artists. I added “mechanical” to the prompt and got this.
I stopped this one early because it clearly was not going in the direction I wanted. Even if I would have let it go to completion, the lack of colors is a no-go. Time to try something else.
As I was browsing through concept art on Artstation, I saw a lot of artists tag their work with “vehicle”. So, I replaced “mechanical” with “vehicle” and got this.
I stopped this one early too because it looked like the A.I. threw on some futuristic wheels and planted it on the ground. Not a good thing. Next I’m going to replace “vehicle” with “futurism”. I’m getting a bid bored with trying new words, so if this doesn’t get me closer, I’m going to try an initial image and see what happens there. Here’s the result with “futurism”.
I stopped this one early again because I didn’t like where it was going at all. Futurism in the prompt didn’t help, so I’m going to remove it and try an initial image.
Initial Image Experiment: An initial image is a starting point for the A.I. instead of a full frame of noise. There are also several settings that will change how the initial image is processed. I’m not going to go into detail about that here because this is getting a little long already, but I’m going to use the initial image below and skip 50 steps out of 250 steps total. That will leave 200 steps for the A.I. to process.
Keep in mind, if you use an initial image, but want to generate several images from it, they are all going to look very similar because of the forced starting point. Without an initial image, when the seed changes, the image is processed very differently from a totally random image of noise. An initial image removes a lot of that randomness.
After using the prompt “#spaceship”, “artstation”, “unreal engine”, skipping 50 steps, and using the initial image above, I got this.
I absolutely hate it! When I hit this point, it’s time to start looking at other artists and throwing some names in there. There are still a TON more things I could try, but when I hit a wall like this, I just want to generate some stuff I really like. I will spend more time on settings and trying to move the A.I. to do something I like without an artist reference, but at this point, I’m getting diminishing returns.
Adding an Artist: The first artist I’ll try is Oshanin Dmitriy. I really like his style and there seems to be quite a bit of his work out there. I’m going to remove the initial image and add “painted by Oshanin Dmitriy” to the prompt. Here’s what I got.
I like the look of this. It doesn’t really resemble the work of Oshanin Dmitriy, which is totally fine! The whole purpose is to “inspire” the A.I. by having it look at the artists work and create something new from it, not generate an image that looks exactly like what the artist in the prompt would create.
I added the words “painted by” in the prompt, but I’m wondering if I should take those out because the “unreal engine” prompt is going to look for 3D graphic type images versus a 2D concept type painting. I want more of a 3D look, so I’m going to take out “painted by” and see if that changes anything. Here is what came out.
I don’t like this at all. I like the painted by image better, but only by a little bit. Since most of the spaceship concepts I like are 2D paintings, I’m going to remove the “unreal engine” portion of the prompt, add “painted by” back in, then see where that takes me. I know, I probably should have done that a LONG time ago, but at least I learned a bit while experimenting. Here’s what I got with “#spaceship”, “artstation”, “painted by Oshanin Dmitriy”.
I stopped this early because I didn’t like where it was going again. In all honesty, I’m liking the 3 images I created at the very start using the super basic prompts, which is a little frustrating. I’m going to try another artist or two and see what happens. This time I’ll add in “painted by Daniel Graffenberger”. Here we go.
Totally looks like a helmet. Not good. The last concept artist I’m going to try is Ryan Church. Here we go.
This one is ok, but not what I’m looking for. Time to try an artist outside of the concept art genre and see what happens. Lets try Masamune Shirow. We got…..
I stopped this one early because again, it wasn’t going where I wanted. You can definitely see how much the artist name can change the composition though.
The last thing I’m going to try is to just write out a full sentence, using Gary Sanchez who created the image I referenced initially. It is VERY common for people to use “trending on artstation” in their prompts no matter what the subject matter is. It seems that set of terms consistently return images that are pleasing to the eye. Figured I would throw that in there and see what happens. Here’s what I got….
I really like this one! This is the best result so far, in my opinion, even though it doesn’t match up with the style I was shooting for at the start.
If you’ve read this far, thanks for walking down this experimental road with me. I wasted a lot of time so hopefully you won’t have to if you decide to try your hand at A.I. art. Keep in mind, all of these images used the same seed number. By changing that one number, the image itself will look quite a bit different, even with the same prompts. I’m going to continue to experiment and I’ll post the images and prompts below. Thank you again!
James
