6 tips to help you create awesome AI generated artworks with neural style transfer
In the last 6 months I’ve created a lot of AI generated art using neural style transfer. I started by running algorithms from GitHub on my own computer, then I migrated the algorithms to Google Colab to speed them up, then I ended up creating an app — NightCafe Creator — that provides an interface to easily create style transfer art.
In that time I’ve gained some intuition for the black box that is neural style transfer. I can now predict fairly well what kinds of content and style images will work well together — and what to avoid. In the points below I’ve formalised that intuition into some general tips and guidelines, starting with the basic fundamentals and progressing to more specific tips to get the best results.
2022 Editor's Note: This blog post was written before DALL-E, text to image art, etc. You won't find any text-to-image art tips in this blog post. Instead try this post where a top NightCafe Creator spills his secrets to create AI Art.
Tip #1 — Be aware of what style transfer actually does
As an administrator of NightCafe Creator, I see a lot of creations that obviously didn’t work out the way the creator intended. An example I see is a lot of people will upload a photo of themselves alongside a cartoon caricature, expecting (I assume) to see themselves portrayed as a caricature.
These people have misunderstood what style transfer actually does, and as a result have probably been disappointed at the outcome. I think the first and most important tip I can give is that you should understand — to some extent at least — what the algorithm does.
The style transfer algorithm doesn’t understand what’s in your photo, and as such it can’t take artistic license by doing things like exaggerating your defining characteristics. It simply recreates your content image using the texture (like brushstrokes), small features (like swirls) and colours from the style image. Knowing this in advance will help you avoid a lot of disappointment.
Tip #2 — Objects and landscapes tend to work better than people and faces
A common type of image that people upload is something like the below (note this is just a stock photo).
There are a couple of reasons that photos like this aren’t well suited to style transfer.
First, the subjects are an awkward distance away. The details of the faces are too small for the algorithm to recreate when applying styles and textures. Photos like this almost never turn out well. Closer range photos (like a head and shoulders portrait) work much better because the details are larger. This rule applies equally to any photo where fine details are important. In general, you should use content photos where the finer details don’t matter.
Second, even on closer range portraits, photos of people are often disappointing because the styles often make the eyes look weird. Something about the eyes just being a little ‘off’ is really off putting. You can get lucky with portraits but often they just look a little bit creepy.
Tip #3 — Avoid large areas of solid colour in your content image
The style transfer algorithm relies on cues from the content image to determine how to apply styles. If you use a content image with large areas of solid colour (for example a plain blue sky), the algorithm often can’t figure out how to fill that area. Often it results in ugly artefacts, repetition and a faded appearance. Some style images are better at filling blank areas than others, but in general, style transfer works better when the whole image has texture and content.
Tip #4 — Choose a style with a similar colour palette to your input image
I can’t concretely explain why or how this one works, even though the intuition is pretty simple. Maybe someone with a deeper understanding of the algorithm could explain it, or maybe it’s just a part of the black box that is neural networks. Either way, this is one of the best tips I can give. When the content and style images have a similar colour palette, the results just seem to work a lot better.
For whatever reason, this tip doesn’t really apply to greyscale images. Good greyscale content images are often some of the best performers, which leads me to the final tip.
Tip #5 — Reduce saturation and contrast of your content image
Tip #4 is powerful, but sometimes you really want to use a style that has a very different colour palette to your content image. In these situations, try reducing the saturation of your content image to improve the outcome. You can go so far as to greyscale your content image (0% saturation).
Content images with high contrast often ruin a style transfer job that could otherwise have been great. I don’t know exactly why, but I think it’s because the style image rarely has such high contrast, so the extreme bright and dark sections of the style image are over-used when recreating the content image. Reducing the contrast can help to prevent this from happening, and the style’s colours, highlights and lowlights are applied more appropriately in the result.
Tip #6 — Be persistent and try lots of combinations
I evolved these intuitions by trying lots of things. I had tonnes of failures, but that’s what helped me understand what works and what doesn’t. Use a tool that allows you to run lots of jobs in parallel and duplicate jobs to tweak their settings (NightCafe Creator does both). Don’t just create one job and wait for it — create a few variations at once and then tweak the ones that work best.
If you’re trying to use textures rather than paintings or drawings as your style image, persistence is even more important — it’s much harder to get textures to work. Internalising tips 2–6 will help a lot in creating good texture artworks, you’ll need to experiment a lot with your content and style images — not just the job settings.
I hope these tips were helpful, if you’ve got any other tips, leave them in the comments! If you’re looking for an easy way to create style transfer art, please try NightCafe Creator.
Please also take a look at some of the top AI Artworks made with NightCafe Creator.
This is gonna sound like a romanticised exaggeration awkwardly stapled on for a little extra narrative impact, but I say with hand on heart with the most accurate and honest description of exactly how my life has changed since the day I stumbled into ‘MY’ NightCafeStudio.. I call it mine and I hope you don’t take offence because I mean it as the biggest compliment I can give. If it’s a clumsy awkward one, I apologise but I have the excuse of feeling more than a little star struck corresponding with you mate. I’m waffling without getting to my point here so I’ll skip to it. Basically your wonderful creation has helped my mental health more than each and every one of the (well meaning) Doctors and Specialists have managed!! … and I mean ‘combined’ my friend!! You should be so proud of yourself, and proof of that is in the fact that I am very proud of you myself mate, and seeing as we’ve never met I’d say that’s a powerful ‘pride by proxy’ moment. The possibility of it being the first ever moment of a previously unknown event doesn’t detract from it whatsoever, you know why?! Because if that is true it means we coined the phrase together just a moment ago!! Either way mate the undeniable truth of the matter is that you sir, are a personal hero of mine, and much, much more….. you rock!!!
I recently discovered A.I. and I have to say that all of this is just great for all that you can get from it.
I have a little question – because in most of the images that represent a portrait, the hands and eyes are always out of phase.
Thanks for all you do
nice programme very cool
I was surprised that NightCafé could understand sentences in Spanish. That’s great, although it seems (I’ll have to explore more) the results are better in English. Anyway, thanks and congratulations.
I think this was one of my first programs to use. It helps me get images that are in my head out of my head. I’m working on a website to put all my images out to the world, I think we all need beautiful, unique art in our lives. I’ll post my website when I get it just right…
Thank you so much.