6 tips to help you create awesome AI generated artworks with neural style transfer

In the last 6 months I’ve created a lot of AI generated art using neural style transfer. I started by running algorithms from GitHub on my own computer, then I migrated the algorithms to Google Colab to speed them up, then I ended up creating an app — NightCafe Creator — that provides an interface to easily create style transfer art.

In that time I’ve gained some intuition for the black box that is neural style transfer. I can now predict fairly well what kinds of content and style images will work well together — and what to avoid. In the points below I’ve formalised that intuition into some general tips and guidelines, starting with the basic fundamentals and progressing to more specific tips to get the best results.

2022 Editor's Note: This blog post was written before DALL-E, text to image art, etc. You won't find any text-to-image art tips in this blog post. Instead try this post where a top NightCafe Creator spills his secrets to create AI Art.

Tip #1 — Be aware of what style transfer actually does

As an administrator of NightCafe Creator, I see a lot of creations that obviously didn’t work out the way the creator intended. An example I see is a lot of people will upload a photo of themselves alongside a cartoon caricature, expecting (I assume) to see themselves portrayed as a caricature.

These people have misunderstood what style transfer actually does, and as a result have probably been disappointed at the outcome. I think the first and most important tip I can give is that you should understand — to some extent at least — what the algorithm does.

The style transfer algorithm doesn’t understand what’s in your photo, and as such it can’t take artistic license by doing things like exaggerating your defining characteristics. It simply recreates your content image using the texture (like brushstrokes), small features (like swirls) and colours from the style image. Knowing this in advance will help you avoid a lot of disappointment.

Tip #2 — Objects and landscapes tend to work better than people and faces

A common type of image that people upload is something like the below (note this is just a stock photo).

There are a couple of reasons that photos like this aren’t well suited to style transfer.

First, the subjects are an awkward distance away. The details of the faces are too small for the algorithm to recreate when applying styles and textures. Photos like this almost never turn out well. Closer range photos (like a head and shoulders portrait) work much better because the details are larger. This rule applies equally to any photo where fine details are important. In general, you should use content photos where the finer details don’t matter.

Second, even on closer range portraits, photos of people are often disappointing because the styles often make the eyes look weird. Something about the eyes just being a little ‘off’ is really off putting. You can get lucky with portraits but often they just look a little bit creepy.

Tip #3 — Avoid large areas of solid colour in your content image

The style transfer algorithm relies on cues from the content image to determine how to apply styles. If you use a content image with large areas of solid colour (for example a plain blue sky), the algorithm often can’t figure out how to fill that area. Often it results in ugly artefacts, repetition and a faded appearance. Some style images are better at filling blank areas than others, but in general, style transfer works better when the whole image has texture and content.

Tip #4 — Choose a style with a similar colour palette to your input image

I can’t concretely explain why or how this one works, even though the intuition is pretty simple. Maybe someone with a deeper understanding of the algorithm could explain it, or maybe it’s just a part of the black box that is neural networks. Either way, this is one of the best tips I can give. When the content and style images have a similar colour palette, the results just seem to work a lot better.

For whatever reason, this tip doesn’t really apply to greyscale images. Good greyscale content images are often some of the best performers, which leads me to the final tip.

Tip #5 — Reduce saturation and contrast of your content image

Tip #4 is powerful, but sometimes you really want to use a style that has a very different colour palette to your content image. In these situations, try reducing the saturation of your content image to improve the outcome. You can go so far as to greyscale your content image (0% saturation).

Content images with high contrast often ruin a style transfer job that could otherwise have been great. I don’t know exactly why, but I think it’s because the style image rarely has such high contrast, so the extreme bright and dark sections of the style image are over-used when recreating the content image. Reducing the contrast can help to prevent this from happening, and the style’s colours, highlights and lowlights are applied more appropriately in the result.

Tip #6 — Be persistent and try lots of combinations

I evolved these intuitions by trying lots of things. I had tonnes of failures, but that’s what helped me understand what works and what doesn’t. Use a tool that allows you to run lots of jobs in parallel and duplicate jobs to tweak their settings (NightCafe Creator does both). Don’t just create one job and wait for it — create a few variations at once and then tweak the ones that work best.

If you’re trying to use textures rather than paintings or drawings as your style image, persistence is even more important — it’s much harder to get textures to work. Internalising tips 2–6 will help a lot in creating good texture artworks, you’ll need to experiment a lot with your content and style images — not just the job settings.

I hope these tips were helpful, if you’ve got any other tips, leave them in the comments! If you’re looking for an easy way to create style transfer art, please try NightCafe Creator.

Please also take a look at some of the top AI Artworks made with NightCafe Creator.