Josh Musikantow

The Artist Spotlight is your inside look into the influences, collaborations and prompts that drive the NightCafe community.

We catch up with some of our most original creators to find what inspires them, who their favorite creators are, and how they use image generator algorithms as part of their work.

NC: WHAT INFLUENCES YOUR WORK?

I’m a composer of experimental music who became interested in digital art during the pandemic - when live musical performances were not possible. So, most of my influences come from music. 

Sometime around 2006, I had the privilege of meeting composer George Lewis, who first introduced me to the possibilities of using AI for art. He had been working for decades on music AI that could improvise in real time with human musicians.

Before then, I had always associated AI with cold rationalism, so I was totally blown away by how Lewis was able to use AI in the context of free jazz, which is usually associated with intuition, freedom, creativity, collaboration and passion. Nothing at all to do with my preconceptions of what AI could do. 

As an experimental composer, I am always disregarding conventional techniques and instead explore extreme possibilities of the instruments I employ, and I think that is exactly what I do with text-to-image generators. I am always tinkering with them in ways that weren’t intended.

I love to use an inpainting model even without using a mask. I love to experiment with a 768x model even when working in thumbnail resolution. NightCafe will warn you that doing these things will give bad results, but if you really want to try it, it will let you do it anyway. This is akin to playing a string instrument with too much pressure, or, using a nonstandard fingering on a flute to create a multiphonic. Sometimes you get more interesting results when doing things differently than the way you are supposed to. 

Another musical interest that influenced my work in NightCafe is graphic notation. Graphic scores use visual symbols outside the bounds of traditional notation. They can be viewed not only as a musical score but also as a work of visual art in and of themselves. Sometimes graphic notation is used to notate something specific that conventional notation can’t, but in many cases, the interpretation of graphic notation is largely left up to the performers. One of the first things I wanted to try when I started using NightCafe was having it make a graphic score. After a lot of experimentation, I made a 33-page graphic score called cAprIccio for violin and viola that received live performances by Play Duo on June 2 and June 3, and is being considered for a performance by Duo Gelland this coming fall.

I love the idea of the AI first having to interpret my prompts, and then the performers having to interpret the graphic notation. It’s essentially going from text to image, to sound, with humans at the beginning and end of the process, and the AI in the middle. 

NC: MOST UNDERRATED ARTIST ON NIGHTCAFE?

I’m a sucker for anyone still doing good work in the legacy modes, like Coherent or Artistic mode. Many good artists are doing this kind of work, some who are well known, others who aren’t, but if put on the spot, I would say @Gyoded.

NC: PROMPT THAT MOST SURPRISED YOU?

When I was very new to NightCafe, I tried to have it draw me as an “Antifa woman”.
 
It drew a giant rat in a dress. I was so mortified because I had intended a favourable depiction. What the AI gave me wasn’t at all what I wanted. I learned a hard lesson: AI does not know how you feel about your subjects. It has very different biases than you do.

NC: ANY TIPS FOR PROMPT WRITING?

If anyone else wants to try to make graphic scores in a similar style to mine, I can give some useful keywords.

Some are obvious: like “Sheet music”, “graphic score”, “graphic notation”, “ornate layers of rhythm”, and “experimental music notation”.

But others are not so obvious, like “wireframe diagram”, “Schlieren Flow Visualization”, “cave painting”, “slit-scan photography”, “specifications”, and “patent illustration.” 

Perhaps the best prompt writing advice is that prompt writing isn’t everything. You need to think about your initial images, and you need to choose the right models and settings. I know SDXL and SD 1.5 are the most popular models right now, but they honestly aren’t that good for my usage which is graphic scores. Stable 1.4 is good for a pencil-drawn look, Stable 2.1 for more flowing lines, and Stable 2.0 for a more diagrammatic look.

If you are planning to have a performance, you also have to think about formatting, and that means aspect ratio can be important. One thing that was surprising to me is that text-to-image models handle aspect ratio (AR) strangely, and every update made to the system can change it.

I periodically check what dimensions come up with no start image when I choose “Portrait” in NightCafe. And surprisingly, it changes now and then. Last November, it was 8x11. In February it was 2x3. I checked this month and it was 7x9. I know it says 3x4, but if you test it, you will see what I mean. They are always making updates and this can affect the AR. You have to test it to be sure you are getting the AR you think you are getting. And this is also true even if you use an initial image. The final creation might have an AR close to that initial image, but not exactly. 

I know this isn’t technically “prompt writing” advice, but anyone planning a long project which requires strict formatting and precise ARs needs to know all of this, and this was by far my biggest learning curve when I made my AI graphic scores. Fine-tuning the prompting was intuitive and fun. Figuring out the aspect ratios and formatting was stressful and difficult, and I had to figure most of it out on my own. Hopefully, others can learn from my experiences and have an easier time in the future. 

NC: WHAT LED YOU TO USING NIGHTCAFE?

I had some downtime at a musical rehearsal, and another composer logged into NightCafe and showed it to me. The ensemble was rehearsing a piece that the two of us weren’t involved in. We were looking for a fun way to pass the time until we were needed again, so we were coming up with some prompts and seeing what happened.

I became so addicted after that. At that time, I had already been doing digital art, but was honestly starting to tire of it because of the NFT craze, which I never really understood. It felt to me that the whole crypto subculture had very nearly taken over the digital art world, and I didn’t want anything to do with it.

But when text-to-image generation came along, it was like a breath of fresh air in the digital art world. It brought in a lot of new people and made the digital art community a lot more diverse. If it hadn’t been for AI art, I think I would have probably just stopped making digital art altogether. I’m so glad I got turned on to NightCafe and kept creating!