Geodesic Fill-In battles with itself - who will win

 

We reconstructed the beginning of the PASeq to use the paint synthesizer to do the actual painting. Working with the exact same set of Threshold IpOp effects bezier curves we used in the last post.  All vector paint.  

We did this so that we can use it to keyframe interpolate Transition Context keyframe images.

So then basically the exact same thing as before. Except we really clamped down on those Mix ratios for the various gradient lighting effects that generate the geodesic  fill out phenomena.

We have it dialed in at a point where chaotic reaction-diffusion turbulence is playing out over multiple frames in the animation.  

It would be cool to add another level of auto-masking to isolate the dancer more, and associated all of this reaction-diffusion stuff with just the rendering of her.

Core Structure vs Geodesic Fill-In - the battle for dominance

 

So a fascinating art strategy involves the dynamic inter-play between a 'core-central-structure' and an 'enveloping geodesic fill in process'.  And we can see this very scenario play out in the paint animation above.

Although 'paint' is a pretty loose term in this example.  Because we're doing it 'old school', building the entire structure of the rendered image from a series of Ip Op effects strung out to build up the overall art strategy.  No paint synthesizer involved.  You are building your own paint synthesizer out of Ip Op parts, laying them out to form the basic principals of your custom art strategy.

So in this example we started with 'mid-core' representation.  So not the internal skeleton, but the larger shape that defines that core.  We used an adaptive threshold effect in the Threshold IpOp.  We turned on vector rendering. So yes your image processing effect is now 'painting' with colored vector graphics.

You are always working with some kind of internalized structure representation when you are creating images.  You just might not be consciously aware of it. Get to know it, revel in it.  Figure out how it works and what you can do with it.

Once we have defined our 'mid-core', then we can derive an internal rod-skeleton that is the basic root structure of the shape.  We drew that in black with the Threshold IpOp thin vector option turned on.

The we filled in the mid core as a solid shape.  We used the Fixed Color IpOp effect to do that. Dial down the Mix to something appropriate. And before you run it, setup the selection mask you are going to be using when you do run it.  You can use that same adaptive threshold effect we've been using since the very beginning of this to do that (route the output to the Selection buffer by using the Replace Region Selection option in the Composite parameter).

We're using the 'mid-core' basic thresholded shape like a silk-screen print setup, using it to create a selection mask that will mask in our Fixed Color IpOp paint simulation.

So to give that 'mid-core' rendition some more visual depth, we will paint it in 2 or 3 times, expanding the Selection out using the Feather menu command, and also dialing down the Mix and rolling the hue and luminance of the source color we are running through the Fixed Color IpOp to do this 'mid-core' fill in. So we're building a very simple expand shadow fringe effect for the shape.

Shadow effects are a great approach to add additional visual depth to your painting.  The color roll adds visual depth, the feathering expansion of the mask adds additional structure and visual complexity to the fringe effect you are building.

Now our Core Structure has been defined. On to the Geodesic Fill-In.  Who will win, the battle for dominance.

We're going to 'bake in' our 'fill-in', by using a series of multi-octave gradient lighting effects.  Super low Mix ratios (they darken, unless you change the algorithm), and you raise the spatial frequency of the lighting effect each time you run it. So that modulation is a ramp up over successive cycles.  Choose odd-ratios to avoid un-welcome (unless you do want it) resonance effects if the ratios are truly harmonic.

Mix in some sharpen processing (mid-frequency, not too fine). I use the Blur IpOp effect to do that (set Composite to Edge1, low Mix ratios).  Add a little bit of Smart Blur that tracks the canvas orientation to smooth it all out. Sharpen again if necessary.

Use the Image Compressor to slam the floor and ceiling of whatever you are time cooking in the canvas out to the absolute maximum levels the media can record. Make that adaptive (not too much, just the right small amount) to master that baked image to tonal consistency.

You need to sit back, reflect on, and appreciate the fact that you are literally cooking a digital video simulation to the point where it starts to become a chaotic system (the multi-octave gradient lighting effects are brilliant at doing this).  You can experience reaction-diffusion behavior, as well as turbulent processes. Th smallest changes in low ratio Mix settings can have a dramatic impact on the visual look of what you are cooking. And the cooking occurs because the whole thing is a recursive process that plays out over time, over multiple frames.  You are painting in time.

I like to throw in some sparkle dust with the Smart Contrast IpOp effect.  Which can be configured to take rendered images like these and add some additional shadow structure.  Process the canvas, use a low Mix ratio, use the Min Composite option, and choose your algorithm wisely.

Un-Sharp mask at the very end, for some additional shine.  It's hard to use too much (although you certainly can and will).

Why Not Let the Generative AI Edit Itself?

The 5th level of digital visual art creativity involves the use of artificial intelligence to automatically create new imagery and associated presets for you.  The human artist dials in a generative art strategy, and then lets the machine do all of the drudge work.  

The artist's valuable time can then be better spent curating the output of the automatic intelligent AI Generative Assistant.  Sifting through the end results for the finest output to use for whatever it is they are trying to do.  

Output being rendered art imagery, as well as auto-generated presets that can then be later used to process source imagery to create new art output imagery. Or as new food for the generative AI system to create more generative visual imagery and more generative digital preset effects.

But what about the AI system itself.  Surely we aren't required to manually program its internal preferences by hand.  Where are the QuickEdit commands to let us perform higher level cognitive editing of the underlying system parameters. Where is the new and improved AI Generative Assistant to help us out in this daunting process?

Studio Artist V5.4 introduces a new innovation in generative art strategy synthesis. Generative AI that intelligently edits its own generative behavior.  

The human artist can still remain in the loop (if they want to do that).  Humans or intelligent felines can still control this process through the use of Generative AI Preference and QuickEdit options, as well as interactively while Gallery Show is running with the new GalleryShow ToolBar 'GS-Zap' button.  

Every time you press GS-Zap, the generative art strategy is intelligently edited by the AI system itself.  Boldly going where no one ever thought to manually edit a path too.  But now you can explore it, because your AI Generative Assistant takes you there. By generatively editing its own behavior.  In real time, as it's working.

But is it AI?

I've been waiting for this question to popup, and it did. Here's a quick response. More later. 

A few quick points:

1: The AI community had no problems calling Harold Cohen's AARON work 'AI' for over 3 decades, featuring it prominently at various AI conferences and AI publications, even though it was just a hand crafted rule based system once you go into the specifics of it. Encoding 'artistic knowledge' in that hand crafted structure.

Just to be clear, we love all things AARON.  I wish Cohen had written more about the specifics of its implementation.  Unfortunately, he is no longer with us.

2: There is currently a huge fascination with deep learning neural net systems in the AI community, and in the general press, and by diffusion from that press response to the general public at large.  And people should be excited, these systems can do amazing things.  They are going to revolutionize society. 

All of the growth in cloud based computing GPU resource consumption is being driven by deep learning architectures being run in the cloud (take away message from Jenson's keynote at GTC yesterday).

But how do they really work?

What are these deep learning systems really doing?

We are just going to focus on image based stuff for the point of this conversation. But extrapolate that out to audio, or 3d point clouds, or chunks of text, or whatever else you care to.

Deep learning neural network take a set of images (the data),and then they learn statistics associated with that collection of images (the data).  What they learn (the statistics they model) is only as good as the data they are trained on.

And there are 'priors' built into the system.  These 'priors' are actually innate knowledge built in the system (inadvertently in many cases)  by the architecture constructors themselves. And also hand crafted into the overall system by the kinds of data augmentation the user of the system works with.

So already these magical software2 learning systems have some hand crafted by humans knowledge built into them.

The end result is that the trained neural net system has some kind of feature space representation of the statistical properties of the data it was trained on internal to the neural net model used for the training.

Now what these systems are really doing is function approximation.  A deep neural net can learn any function approximation (in theory), it is a universal function approximator. So it's leaning some kind of nonlinear transformation. Usually an extremely high dimensional nonlinear transformation.

Now as it turns out, the real world and imagery associated with the real world, (images of faces, or people, or cats and dogs, chairs, whatever),  all of this stuff lives on a low dimensional manifold.  Higher than 2 dimensions, higher than 3 dimensions, higher than 4 dimensions, but way lower than what would be the case if the structure of the universe, the structure of images, if that was all just random.

And the trained neural net works in some sense because the real world lives in this low dimensional manifold (higher than the 2 dimensional plane an image is laid out on, higher than the 3 dimensional space you move around in, but low dimensional compared to what it could be if things were just random).

Think of an image that is just uniform random noise. It doesn't look like anything except noise. Think of an image of a person's face.  there is a lot of structure in that face image, both locally (adjacent pixels are similar, not radically different). And if you think about how the face changes as it rotates, or as the light hitting it changes direction, there is an inherent similarity associated with those changes.  That information lives on a low dimensional manifold.

Google manifold if the mathematical abstraction seems too obtuse. Or manifold learning.

But what about the human visual system?

How does it really work?

The human visual system also internally models something associated with the statistics of the real world images it perceives.  In a series of transformational mappings that occur in the transformation of an incoming visual signal into the brain through successive processing layers of the visual cortex.

Analogous to what is going on in successive layers inside of a deep learning neural network, but also different. Because what the neural net is learning is ultimately tied to the task it was trained on, and the function approximation might be specific to that task (the specific kind of function approximation that solves the task, which could be very different than the human perceptual representation that occurs in the brain). 

You can google visual cortex if you want to learn more.

I have been involved in the application of human visual models to various engineering problems for 40 years at this point.  From my masters thesis onwards. If you are curious about the internal structure and gory details of these kinds of engineering models based on neuroscience and psychophysical vision experiments, you can check out this post.

If you don't think that these kinds of models and associated research have anything to do with the human perception of artwork, i would encourage you to check out these books written by extremely esteemed neuroscience researchers.

Vision and Art by Margaret Livingstone here. Including a foreward by Hubel himself (Nobel prize for his work in visual perception).

Inner Vision: An Exploration of Art and the Brain by Semir Zeki here. Zeki basically discovered cortex area V4 and how it works.  

It is a very sad statement that neither this book or Zeki's other super great human vision book titled 'A Vision of the Brain' that you can find here is not in an e-book kindle format to read electronically.

Studio Artist is not a NIPS paper. We are not trying to find a new solution to some greatly constrained technical problem that we can then write a paper about so we can get tenure or a job at Google or Facebook.

Studio Artist is not a neuroscience research project.  We do track the neuroscience research, we do track the work of the theorists (like Tomaso Poggio at MIT) who try to build theories of how it all works mathematically (check out this lecture if you are curious).  We do track the academic research of art theorists who come at this whole area from a very different perspective.

Studio Artist tries to take work metaphors from music synthesis, music synthesizers, and repurpose them for digital visual artist. Concepts like signal modulation (visual signal modulation) are extremely important to our world view.  We try to incorporate visual signal modulators directly based on the neuroscience research into how humans perceive visual imagery, and then make them available throughout the program.

How humans perceive a piece of art is ultimately derived from the inherent visual statistics associated with that work of art.  Some people have weird hangups about the term 'art'.  So substitute 'visual image' if you are one of those people.

Visual modulation derived off of human perceptual visual attributes is a key component of how StudioArtist works.  And i just told you it is also a key component in how you perceive art (visual images).

Studio Artist also incorporates all kinds of internal 'heuristic' knowledge into it's internal workings.

Again, we're not trying to write a NIPS paper here. We're trying to build a system for digital artists to make art. Whatever we can throw into that system to make it work better is fair game.

Studio Artist is an active dynamic system. It tries to look at an image like a person would, and then react to that stimuli by building up an art output representation of that perceived 'source'. Like an artist would look at a model (or a poloroid photo of a model), and then re-interpret it.

The paint synthesizer is not an 'image filter'. You could probably configure it to be one if you really want it to work that way for a specific preset setting. But at it's heart it's a system that does active drawing. 

Yes, i have also dived into the literature on the neurobiology of muscle movement planning in the cortex. 

The initial prototype implementation of the paint synthesizer was conceived to drive a robot for physical painting.  That quickly changed as people started using it.  That has always been my philosophy, start an engineering project, get it to a place very quickly where people can start actually using it, then react and adapt to how those people use it, and the whole project takes on a life of it's own.

Studio Artist is not trying to replace the human artist. We are trying to augment the human artist.  The human artist can choose what level they want that augmentation to occur at. StudioArtist can literally do all of the work, or Studio Artist can aid the artist as the human artist manually paints, or Studio Artist can wiggle bits in the paint while the human artist does all of the stylus driving.

Studio Artist V5.5 is all about expanding the range of what the program can automatically assist the human artist with, letting the program automatically and intelligently build new presets for the system. As opposed to the human artist having to do all of that manually.

Studio Artist V5.5 is also about expanding the range of what a 'source' for a digital art program even means. 

Neural nets work off of a database of images to learn to model statistics inherent in that database of images. But an artist can work directly with the database of images, feeding it into Studio Artist, and then generating artwork created from the aggregate statistics inherent in that collection of images. You don't necessarily have to train an abstract model to learn those statistics in some hidden latent space of that model to generate art work derived from them.

All of this sounds like intelligent behavior to me in some sense.  And it is happening in a computer program, so it is in some sense artificial.

I guess it depends on your definition of intelligence.  And your definition of artificial.

Some of the intelligent behavior may seem stupid at times. But i could say that about people.

Some of the intelligent behavior might seem rote or repetitive in nature, but i could say the same thing about people.

Some of the intelligent behavior is derived from heuristic rules hand crafted by people, but i could say the same thing about the behavior of people.

But it's all based on statistics some people might say, but i could say the same things about neural nets.

Again, at the and of the day, it's a system to help digital artist make artwork. nothing more, nothing less.

And that's really how one should judge it.

Could it be better at certain things,? Sure, absolutely.

Are we going to make it better?  We are certainly trying.

Painting on a Time Canvas

 I keep noticing that almost no one is approaching video processing in Studio Artist the way it's setup to really do it.  And i think part of the problem is that folks are used to working in other video programs like FCP, AFTEFX, PRIMR etc, where their whole design is built around this notion of 

'process an input frame into an output buffer - write out the buffer and wipe it clean for the next cycle in the loop'

Which is fine for what those programs are designed to do.

But at its core since day one (back when dinosaurs roamed the earth and 400 MB hard disks were the size of toaster ovens), Studio Artist has been about digital painting. And i don't mean silly drag a single nib image around as a brush kind of 1988 digital painting (although we are more than happy to let you do that all day in Studio Artist if you so desire).

Real world media has more finesse to it.  More complexity. More chaos inherent in the system. You inherently get your hands dirty when you even mess with it.  Things leak and spill all over the place.

And we wanted to build a digital system that behaves with the same level (or more) of rich organic aroma.  If you can visualize that 'rich aroma' in a visual manor.  Part of the fun when working with real art stuff is that it has a mind of its own.  And if you follow that, you can end up in all kinds of unexpected really amazing places.

Or you can ignore it and just concentrate on the outcome.  It looks vibrant (if that's what you're going for, again we try not to judge and limit what you can do, because of some mistaken pre-conceived notion of what is a proper visual aesthetic).

And we certainly aren't going to limit you to some even more limiting pre-conceived notion of what 'natural media' is or isn't.  A mistake a lot of other digital art programs make.  Are actually very religious about in fact.

And the feedback we get from visual artists about our approach confirms that.  Oftentimes they are way more interested in a 'hyper digital natural media' kind of thing. Maybe it's reminiscent of some specific kind of natural media marking system, but its wildly unique and digitally imaginative, fresh and distinctive at the same time.  They like the fact that it is its own thing.

Responsiveness is of course always important.  It's at the heart of Studio Artist's internal parameter modulation structure.  Real world art stuff is very responsive.  To the touch.  To the temperature of the room. To the characteristics of what you are trying to apply it too.  To the bumps on that stick you are dragging around with paint on it.  And we try to do the same (or more).  It's built into the system from the bottom up. Parameter modulation everywhere, and lots of it. More and more all the time.

So we've always cared about painting. From day one.

But very quickly we also started looking into using Studio Artist to process movie files.  It was built around Quicktime from day 1 (Quicktime being a cool and revolutionary thing back in the day, now extinct like the Dodo or Carrier Pigeon, killed by an evil entity).

But in the very beginning our painting focus was very much 'let's paint on a canvas'.  Canvas being this 2D digital surface.

And Studio Artist also incorporated from the very, very beginning this whole notion of an artist painting a model source onto the canvas. And we wanted Studio Artist to look at and analyze the model source just like a human artist would do. So we built visual modeling into Studio Artist from day one as well. Human visual modeling based on cognitive neuroscience research that looks into how the visual cortex is structured, how it processes its input signals.  And also theoretical computational visual models of what various folks thought was going on in those visual areas inside of the visual cortex.

We also incorporated AI Heuristics into the system as well.  Lots of them, more and more all the time.  If you want to think of that part as a rule based AI system based on expert knowledge from artists, sure why not.  Different kinds of AI for different tasks (or mixed up together in the same tasks).

I think we need to make it clear that our business is making visual art.  And enhancing what visual artists can do in the digital world.  Building an extremely flexible creative tool for them to use in the digital art world.  We're not trying to write a NIPS paper, where we present the perfect algorithm for some heavily restricted task run on a standardized data test set.  We're trying to help people make art.

So anything goes really.  We play loose and fast with some aspects of the modeling when we have to (computational limits are always there, or maybe a really bad model actually looks more interesting).  And we love mistakes.  Because oftentimes they look really really amazing. So why would you remove them from the program if that is the case.  You just tie them to a specific parameter setting.  Because anything that looks visually different is interesting, worth keeping around, something to use in your tool chest of visual effects.  We aren't going to prevent you from using that, just because it was the result of a programming error (crashes are a different story, we're talking about algorithm mistakes that lead to interesting visual results).

Now the thing about painting is that there is this inherent dynamic tension between applying pigment to a surface, and the dissipative action of the fluid that holds and spreads that pigment.  The fun stuff is in the interactions going on in the system.  Pigment and fluid mixing. That amalgamation of pigment and fluid interacting with a surface. A textured brush or other applicator object interacting with all three.  Time interacting with that whole system of discrete elements as the fluid dries out, or the surface gets wet, or doesn't.

All of that stuff interacting together is what generates all of the fun. Generates the visual fun of the end result. Generates the fun of driving that stuff applicator over the surface (over time), to create something interesting (one hopes).

So there is a very definite dynamic tension going on in the system between a dissipative effect, and a marking 'lay down stuff' application effect.  They fight each other for control. So you want to be aware of that, and take advantage of that in your work.

And work is a real loose term. Because Studio Artist has always bridged these 2 product categories that seem to be kept completely separate by wizened marketing folks at other entities that release software into the world.  The universe of 2D graphics programs. And the universe of video programs.  And never shall the twain meet, except  at the file : import / export level. 

And when you start to look at video processing, movie file processing, digital rotoscoping, paint animation, whatever you want to call it, you want to make sure you apply the same set of principals you apply in the 2D 'spread it on the canvas' world to the temporal time-based 'frames in a row' video or movie canvas.  You want to make sure you keep that rich and powerful dynamic tension going on.  The dynamic tension between the dissipative forces, and the 'lay down the marking stuff' forces.

You really want to think of your video processing as 'painting in time'.  And when you do that, not only will your creativity be expanded 10 fold or more, you will also start creating some really interesting visual results because of it. Visual results that happen over time.  Because you are painting over time.  You are building a temporal canvas.

And how did we say painting works? You lay down 'marking stuff' onto a surface. And some kind of 'fluid' is interacting with that stuff, and also interacting with the surface you are applying it all too. But the surface is a series of consecutive frames laid out over time.  The 'fluid' (whatever that term even means, do not get literally or rigidly attached to it, remember anything goes when we are talking about art), fluid interacts over time as well. Interacts with those consecutive frames, and the consecutive laying down of stuff, and with the remnants of itself, all interacting over time. 

Time is an essential component to the whole process.  That's true for 2D painting, or time based painting.  Time in 2D painting can be thought of as movement.  Movement occurs over time. Movement builds up the final painted image. The dynamic tension between dissipative forces and marking forces on a surface occurs over time based on movement.  That's what painting is really all about.

Painting on a time canvas has both kinds of movement over time.  There is movement on the surface, or over time, or both going on simultaneously.  But the canvas itself is time based (discrete elements of time we call frames).

Now once you wrap your head around this notion of video processing being painting over time, then you are free to envision it in a very different way than what is laid out for you in the conceptual design of those other video programs.

One key point is that you should really be thinking about painting on top of the last output frame. As opposed to wiping the slate clean and starting fresh with every new frame. Because if you do that (wipe the slate clean) you have immediately lost one of the main places where you can create dynamic tension between dissipative forces and marking forces, a dynamic tension that occurs over time, over multiple frame images.  You really want to take advantage of this, capitalize on it, use it in your artistic process.

You also need to think heavily about what 'dissipative forces' means to you.  Because how you approach that part can really help to define your visual style, make it unique, make it fresh, make it distinctive.  We're going to educate you on some approaches.  But you should really learn them, understand the mechanics of how they work, and then head off into the wilderness, boldly exploring where no one has gone before. Or maybe re-exploring what has been very heavily trodden (if it looks cool).

Is dissipative fluid like wet paint?  Like a water emulation in the paint synthesizer?  Sure, could be. I use that all the time in my work (maybe a little too much at times, like symmetry it's an easy crutch to fall on). But you certainly aren't restricted to water emulations to create a dissipative force.

You could use noise randomization (most likely in association with some kind of mixing, or blending). You could use warps, pushes, nudges.  How they interact with what is already on the canvas, how that leads to movement over time, they all look different.  Spend an afternoon playing around with different approaches to warping or nudging, to create a dissipative force.  Learn how to use it as a tool in your tool chest.

Now for 2D painting, how the dissipative force moves around on the 2D surface is the deal. But when you are talking about painting over time, then you need to factor in some kind of recursive element, so that the force 'lingers' over several successive frames.  The dissipative force is temporal in nature, in addition to probably having some 2D dissipative element as well.

So you need to think about different scenarios to pull this off, different generative strategies that play out over time in an animation, over time in processed video frames, that work to help generate the final painting over time, your digital painting on a time canvas.

You also need to lay down that 'marking stuff', whatever it may be (again, don't get hung up on it, there are lots of different approaches, all are valid).  And by painting on top of your last output frame, you are inherently introducing temporal continuity into your time based painting. And you really want to shoot for that.  You want that dynamic tension between interacting forces to play out over time.

So that was a mouth full. Something to think about before you head off to process some video frames in Studio Artist.  Have fun.

The Myth of High Resolution

Graphic artists have been fed this myth that they need to work at higher and higher pixel resolutions.  That it is vital to their work to do so.

Graphic artists are shaking their heads in agreement (with the second part, not the first). Yes, exactly. Give us more resolution. So that we can work at higher and higher resolutions.

Computer manufacturers are also shaking their heads in agreement. Yes, exactly. These people need to buy new equipment from us every year that displays at higher and higher resolutions.  They need to work with raw pixels at higher and higher resolutions.  They need to buy new stuff (from us of course).

Now we know why the hardware manufacturers are pushing this narrative.  Because they want your money.  They want you to buy new stuff every year (they would prefer every 6 months).

And computational power.  Yeah, sure.  I get that totally. Come on hardware manufacturers, give me more computational power in your systems so i can use it all for some amazing new algorithm that needs all of it and more.  Think of what we could do with that.

But more and more resolution?  And then having to actually work at that increased resolution?  Why exactly?

Graphic artists speak up in a loud voice.  But we need to work at those higher and higher resolutions so that our prints of our work look perfect in every detail. No mater if the viewer is across the room looking at it through coke bottle eye glasses, or up real close up with a loop magnifying lens (different kind of coke bottle lens) so they can see all of the details.

Yeah, i get that.  You want your digital prints to look good.  Especially when you print them wall sized in the art gallery of your dreams (i'm sure there are a few, like that place in Lahania).

Graphic artists again insist.  We need that high resolution because the stock place (substitute the magazine, ad agency, special effects house, etc) that is buying our art insists.

Sure, i totally get that.  Your final work output has to be a certain resolution or the people buying it will not pay you money for it. Absolutely, it's what the client wants.

Video artists speak up. Buddy, we needed 4K because we needed to build content to support future 4K video streaming services (even though most people watch them with a streaming bandwidth that probably doesn't even reliably support SD resolution). And now we need 8K because everyone who buys our stuff wants it archived for the future, so 4K just doesn't cut it.

Alright, i understand that even though Netflix can't even support SD streaming resolution without grinding to a halt where i live, you need to sell your content to them at 8K because that is what they want, or more likely your boss wants it at 8K so he can resell it to them later at a higher res, sure. It's all about expectations.  They have to be met.

They have to be met in the final end stage.

But working resolution?  Why does that have to be the same.

And it's fascinating to see Nvidia push this whole concept of neural net based image interpolation converting a much lower resolution real time ray-traced graphics render into a much higher resolution 4K video display frame for gamers to enjoy.

And this leads us into the whole myth of high resolution that digital artists have been fed.  Because it's only true for the final end stage.  The part where you put together whatever final thing you are blasting off to the client, or to the ink jet printing place, or the Netflix streaming vault, etc.

But working resolution.  Why does that have to be high res?

But edges have to be sharp. I need to zoom way, way, way, way in to make sure that is true.

Sure, edges have to be sharp. If you want them to be.  But that zoom way, way, way in part, just to check, just to make sure? While you are working on design, on composition, on the structure?  That part seems odd.

Especially if it then requires you to work with order of magnitude or more more raw pixel information when you are doing your compositional work.  Every time you double the canvas dimensions, you 4X the amount of data.  You 4X the cpu power required, you 4X the bandwidth required, you 4X the slowdown in response time of the system.  That adds up real fast, exponentially fast.

And i'm arguing that it is totally unnecessary for the compositional part of your work. It's only an essential requirement for your final output.

Nvidia totally gets this. That is why they are restructuring the entire graphics pipeline of computer game real time graphics display to take advantage of it.

Synthetik gets it as well. We want to structure our working environment for digital artists so that they can separate the concept of interactively (or automatically, or intelligent assisted, etc) building the digital art content from the totally separate super high resolution output they might require for the final stage of their work.

Are we totally there yet in Studio Artist V5.5? No, not yet. For some things (vector effect output), sure. And we have ideas to make that more explicit for those things (like the Vectorizer) in the near term future, to aid you in this rethink of what working resolution actually means.

For other effects, re-rendering your Paint Action Sequence (PASeq) might be a way to go. Raster paint nibs on a vector bezier path, Studio Artist has offered that since V1. For some paint effects it works great, for others, not so great.  It can at times works like an edition print (subtle or not so subtle differences in output from run to run), which could be cool when viewed in a certain light, but could be frustrating in others.

So we have other plans for future but already in development internally here at Synthetik Studio Artist V6. To cover the other work scenarios with some really cool options (think all things raster in nature).

Here's the thing.  If you look at the contrast sensitivity function of the human visual system, there is a pronounced drop off at higher visual spatial frequencies.  And algorithms for digital halftoning and for image compression have taken advantage of this fact for many years, depend on it for them to work at all. Place the error generated by the system at spatial frequencies where the viewer can't perceive it.

So when you are building up the composition and structure of your digital art, that information has very little if anything to do with it's perception. Not until you get up real, real, real close, so close you are now looking at that sharp edge like it is it's own digital art canvas, one that really has nothing really  to do with the perception of the overall work of digital art (the composition, the structure, etc).

So if you can't really even see that information when you are building up the overall structure of your piece of digital art, why would you want it to get in the way of that important creative design work by hogging 4X x 4x x 4X ad infinitum of the cpu and memory bandwidth?

Something to think about.

Studio Artist V5.5 - What is it and Where is it Headed

 StudioArtist V5.5.0 is locked down and done at this point in time.  The installer packaging and web site stuff is still being finalized (that needs to happen before it hits the online store), but the actual 5.5.0 code is done at this point.  Which is pretty liberating (at least for me), because we've already moved on and begun work on 5.5.1.

So what is Studio Artist anyway?

It's a digital art program with a 20 year long history at this point. When Studio Artist V1 was released at MacWorld New York in 1999 (is that a long time ago or yesterday, i get confused), it was a pretty radical rethink of what a digital paint program could be.  It was not PhotoShop, it was not Painter, it was not Illustrator, it was its own thing.  With its own internal structure, its own way of working, its own way of thinking about how to work.

It allowed you to work with still images or video, merging those 2 very separate application categories at the time into one unified system. It introduced the concept of a 'source' and a working 'canvas', again a very unique distinction at the time.  It introduced the concept of incorporating a human visual model (based on research into how the human visual cortex works at the neural level) into a graphics program for digital artists.  It incorporated conceptual ideas from music synthesis, music synthesizers, and digital audio software, and then incorporated them into a digital art program for visual artists.  It introduced a new hybrid raster-vector model for digital paint effects (bezier paths that define a drawing path, raster paint nibs that a relaid down on that editable vector paint path).

It also introduced what i call the first 3 levels of ways of working and interacting with digital imagery.

Level 1 is all about working at the pixel level.  Down in the basement.  Literally pushing pixels around on the canvas with an interactive stylus (wacom pen at the time) or a mouse cursor.

I would also consider the use of simple image processing effects (like an edge or blur filter) to be working at level 1 to some extent.  At least for the sake of this discussion.

Level 2 moves up a conceptual notch with intelligent automatic actions.  Intelligence based on the underlying human visual modeling built into the program.  That takes a center stage in the Studio Artist paint synthesizer with its extensive use of visual attribute modulation internally.  But intelligence that was also distributed throughout the program in a wide variety of adaptive image processing effects.  Also based on visual attribute modulation.

Studio Artist is built from the ground up on the concept that the program tries to look at a visual 'source' just like a human artist would look at that 'source'.   That internal visual representation is then made available internally throughout the program.  Visually modulating what the paint synthesizer is doing, visually modulating what adaptive image processing effects are doing.

You can think of the 'source' as a model the artist is painting, or a photograph of a scene the artist is reinterpreting in a painting.  The artist looks at the source to influence their work, but so does Studio Artist. Using its internal human visual modeling.  Trying to perceive the source in a similar way to the artist perceiving it, reacting to it, etc.

This process is not about the machine replacing the artist.  Far from it. 

Studio Artist always tries to incorporate the human artist into the automatic action creative loop. At whatever level of interaction the artist is comfortable with.  The artist can do all the work if they want to. Or the artist can manually manipulate the stylus, while Studio Artist intelligently assists in the manual drawing (intelligent assisted painting).  Or the artist can press the Action button, and Studio Artist does all of the work automatically (fully automatic action painting).  

But even in that last case of a fully automatic action, the artist is still involved in the loop, making decisions about what they like and don't like, how to proceed next, etc.

Level 3 in our taxonomy of ways of working i would define as working with scripts of actions (which could be manual or automatic actions).  Studio Artist includes a Paint Action Sequence (PASeq) Editor, that allows you to build more expansive visual effects composed of multiple actions that work in sequence.  Combining together different kinds of brush sizes, different kinds of digital media emulation, different kinds of artistic techniques (charcoal, ink pen, watercolor, water or acid wash, canvas smear etc) into a sequence of different processing steps that work together in synergy to build up a final visual effect.

Since automatic actions intelligently analyze the 'source', they can be built using just one specific image as the source when they are initially created, but then applied to an infinite variety of different source images later, while achieving the same stylistic visual effect output for those new input images as was designed using the first source.

Intelligent actions meant that you could also work at level 3 building PASeq presets in Studio Artist, and then process movies with them.  Creating dynamic visual effects in processed movie files that looked like hand painted or hand animated art styles in moving art.

That was a lot of innovation introduced in old SA V1.  Innovation that is still missing in other digital art programs 20 years later to be honest.  I say that because in every other digital art program i sit down at, you start with a blank canvas, and then you have to do all of the work.  The other programs do nothing, you have to explicitly make everything happen in them yourself.  

Here's the thing, that's a really hard process to create something from nothing.  Especially if you have not taken the extensive time and training required to develop the muscular and neural motor skills required to draw well.  Some people are very good at it, many, many more are not.

Now there's a direct analogy in the world of music composition, where in the old days a composer would compose a piece of music (as marks on paper literally), and then go into a very expensive recording studio, and tell a bunch of musicians they hired at union scale wages to play their piece of music for them, and a recording engineer you also had to pay hourly would record them playing the music, and then another person you had to pay would mix and master the recorded tracks into a finished piece of music.  And the expense and organizational complexity of all of this was really prohibitive to musical creativity.

So the home recording studio revolution created by digital audio workstation software running on personal computer hardware (a revolution i was heavily involved in created back in the day), was truly liberating for individual musicians and composers.  Because they could do the work on their kitchen table if they wanted to, and they could do all of the work if they so desired themselves.

Studio Artist was trying to do a very similar thing in the visual art and digital video world.  So that if an individual had a great idea for an animated film, they could do it all themselves on their kitchen table.  As opposed to hiring a team of animators at great expense to make that happen.

David Kaplan pursed that very vision using Studio artist to great effect, winning an award at the Sundance Film Festival for his feature length Studio Artist animated film 'Year of the Fish'.  Literally created at his kitchen table in his apartment in NYC.

And the innovation continued over the years as new versions of Studio Artist were released.  Vector effects were introduced via the Vectorizer, new vector paint options in the paint synthesizer, and vector output from some Ip Op effects.  You could output these new vector effects as resolution independent svg files, an interesting alternative to the traditional raster image file output.  

Temporal image processing effects were introduced.  These were great for processing video with time based effects, but also opened up a whole new way of working called 'stack filtering'.  Stack filtering involves taking a collection of still images, and then using that collection of individual images as the input to temporal image processing effects.  The end results can be visually amazing.

MSG (modular synthesized graphics) was introduced in SA V2, and has been extensively expanded over the years.  With over 600 individual image processing and generative modular effects that can be used to construct an infinite variety of different modular preset effects.  MSG can be totally generative (creating visual imagery from nothing but editable parameters), or it can process a source image into an effected output image, or a source can be used to modulate a generative process in some way internally.

MSG presets can also be embedded into the paint synthesizer, providing a way to modularly expand different internal paint synthesizer components (like path start generation, path shape generation, brush load processing, source brush generation).  Studio Artist V5.5 also lets you embed Ip Op and Vectorizer effects directly into the paint synthesizer (once again super-charging what you can do with paint synthesizer presets).

Movie brushes (using a movie as a paint brush) were introduced fairly early on in Studio Artist history. Movie brush capabilities have expanded over the years with image folder brushes, as well as movie and image folder background textures.  These can be incorporated along with visual attribute modulation to build photo mosaic and other visual effects built off of artist curated sets of multiple visual images loaded into a digital paintbrush.

Keyframe animation in the PASeq Timeline allows for the construction of interpolated hand painted strokes that then move dynamically over time in an animation, automatic dynamic visual transformations, morphing, warping, etc.  Bezier paths derived from automatic actions can also be embedded into single paint actions and then key-framed as well on the PASeq Timeline.

So Studio Artist has always been a generative digital art system from its very beginning.  And i want to draw a distinction between how the word 'generative art' is oftentimes used, because many artists use it to refer to digital art output from software coding systems created for artists like Processing, Open Frameworks, or perhaps mucking about in neural net Colab notebooks these days, etc.

I'm all for artists learning how to work with software code if they so desire. But my experience working with a lot of visual artists over the last 30 years indicates that many of them are not really interested in software coding.  It requires a certain level of 'left brain' analytical thinking they aren't necessarily comfortable with (especially within their working methodology which is much more intuitive or 'right brain' in nature).

These 'left brain' - 'right brain' metaphors are much over-worked (and technically incorrect) at this point in time, but i think they are still useful on some level.  We have traditionally made the distinction between 'left brain' building your set of tool presets, vs 'right brain' using your set of tool presets within the Studio Artist universe.  So you would set apart specific times to build working tools (left brain activity), and then use those tools creatively in your artwork (right brain activity).

The previous distinction between these 2 ways of working ( 'building the tools  and then 'using the tools' ) is a good lead in to where Studio Artist is heading.  Both in terms of the new features we are introducing in Studio Artist V5.5, and where we see those new features heading in the future.  Because we believe we have staked out a whole new way for digital artists to work in Studio Artist V5.5. And we are going to fully develop that out and expand the nature of what it truly means in future Studio Artist releases.

Studio Artist V5.5 introduces 2 whole new higher levels to our taxonomy of 'levels of working' for digital visual artists.

The 4th level of working involves expanding the whole concept of what the 'source' even means in a digital art program.  Traditional digital art programs are very religiously rooted to the concept of working with a single image, a single movie file.  The single digital photo you took, and you now want to enhance or effect in some way.  The single movie file that you want to effect in some way.  The empty blank canvas that you are supposed to turn into a sketch of a woman's face, or a bowl of fruit on a table.

So this new 4th level of working i call 'source abstraction'.  Abstracting and expanding the whole notion of what the source even means.  So that it is no longer tied to just a single image or movie file. It might be a representation of a collection of images, a collection of movies.  You might start to think of it as more like a data model, filled with visual attributes, all available for modulation within Studio Artist.

You can also start to think about 'style' mixing in the context of the 'source'.  Playing one abstracted 'source' off against another abstracted 'source'.

My personal viewpoint on the whole notion of what a source is, or even style for that matter, has evolved and expanded quite a bit over the years.  Just like our notion of what the source is and can do has expanded in Studio Artist V5.5.  With the new Gallery Show source options, and the new Load Style features.  Both of which are really just teasers for even cooler things to come, but still incredibly useful in their current limited forms.

It's fascinating when you start thinking about the 'source' for a piece of artwork as being a collection of images instead of a single image. It's very much like working with a database of visual attributes.  It's a very different way of thinking for most digital artists (at least that is my perception).  Different, but really expansive, liberating perhaps, definitely worth getting a grasp on, worth trying out and exploring.  It's new territory waiting to be charted out, waiting for you to find your own personal niche within it.

As i pointed out earlier, the Studio Artist universe dived into this water when we got heavily into stack filtering. But it's always existed since V1 if you really wanted to explore it, via loading a movie (which could be any old collection of images as individual frames within the movie container), and then riffing with it while painting.

I've been very heavily involved in getting up to speed on the latest developments in deep learning neural nets during this last pandemic year.  Most people are probably unaware that i did neural net research in the 90s, including some very basic work on using convolutional neural networks for learning artistic stylistic transformations on images.  Not really practical back then to be honest, so we quickly moved on to adaptive fuzzy logic systems and other things that ran a lot faster on the ancient computer hardware available at the time.

But it's a very different world now days. So it's been fascinating to take my background in all things neural net and bring it all up to date with the latest and greatest developments in the field. So i'm heavily influenced by what people have been doing with recent research areas like neural style transfer, generative models like GANs and VAE systems, etc.

And the notion of a database of visual images is very central to how these deep learning neural net systems work.  Generative imaging transformations based on the collective visual statistical properties of a collection of images. So all of that exciting new research work has heavily influenced my thinking, and is going to continue to do so in the future as we move forward.

This whole notion of source abstraction (expanding the whole notion of what a 'source' even means), is the new 4th level of working.  

And the 'build a tool' - 'use a tool' discussion mentioned earlier is really what the new 5th level of working hopes to alleviate.  

Through the use of AI Generative Systems.  Implementing automatic intelligent Generative Art Strategies.

The power of AI Generative Systems is that they can intelligently create presets for you on the fly. A literally infinite variety of them. And then they can intelligently work with multiple kinds of presets, working with them together in sequence to build more elaborate effects.  More elaborate generative strategies.

So it's like scripting, except Studio Artist builds the script for you, automatically, on the fly, and it can always be different if you want that, even as it follows a generative strategy that is constrained in some high level conceptual way.

So AI Generative Systems are the 5th level of working. Building on top of everything below it. Built on 20 years of development. So many old effects to rediscover. So many new effects to learn about and explore. Letting the program build artistic strategies and presets for you for you.

Or letting you define artistic strategies for it to explore for you automatically. Like that crew of really hard working people famous artists use to crank out their work for them.

Gallery Show has expanded tremendously in Studio Artist V5.5.  Originally conceived as a way for artists to create free-running dynamic visual art shows using Studio Artist in an actual art gallery (it was very lightly used for this task), people quickly glommed onto it as a way to visualize what the factory preset collection did, or to batch process images with custom PASeqs, or to riff random mutations of pre-existing presets.

In Studio Artist V5.5 you can use Gallery Show to build custom Generative Art Strategies.  Generative Art Strategies that can be fairly simple, or extremely elaborate.  The new Generative Paint options in Gallery Show basically allow you to control Studio Artist as it dynamically creates an infinite variety of custom automatically edited paint presets on the fly.  Exploring areas of the overall editable parameter space you might never get to through the use of old school manual preset editing.

We've also tried to bring many of the intelligent Gallery Show generative features up to the surface, providing ways to access them at the high level working interface of Studio Artist.  Through new smart Quick Edit command options for the paint synthesizer and the vectorizer.  With more to come in the future.

Old school preset editing involves going into the depths of the Editor for the specific operation mode (paint, vectorizer, ip op, etc), and manually editing individual parameter values by hand.  This extensive level of editing control over every effect available in Studio Artist has always been there, and is great from the standpoint of being able to tweak things to your hearts content. Dial in just the effect you want.

The other side of that coin is that you really need some level of conceptual understanding of what is going on under the hood to really be fluid at this low level of manual parameter editing for visual effects and digital paint.  And if you aren't carrying around a conceptual understanding of how the paint synthesizer works internally, how the vectorizer works internally, hand editing all of those parameters can be a very challenging left brain kind of task. A task that gets in the way of your right brain creative work.

At the same time, it has become quite apparent to us that increasing the level of internal visual attribute modulation inside of these areas of the program (ie expanding the level of editable parameters even more) leads to amazing visual output effects.  A whole new level of visual wow in the resulting art output.

So that is the dilemma.  Expanding the range of editable parameters increases the range of potential effects, and increases the resulting visual complexity of the effect output, pushing it to an amazing new level. But the program becomes more unwieldy to manually edit, even for experts like myself well versed in how it works internally.  

AI Generative Systems to the rescue.

Rather than having to manually adjust individual editable parameters, we've been adding all kinds of different intelligent Quick Edit commands. Smart edits that let you work at a higher conceptual level when editing an existing preset, or when creating new ones from scratch.

We've started this process in V5.5, but it's going to be an ongoing endeavor from now on.  More of them, and more places to use them.  You can run these smart Quick Edit commands manually.  But you can also use them to build automatic generative strategies that can run within Gallery Show in V5.5.

Let's say you want to use MSG Brush Load (did you even know it existed, or if you did how to manually program it in the paint synthesizer).  You can just dial up that generative paint option in Gallery Show and Studio Artist will then riff out an endless variety of new unique most likely never seen before paint presets on the fly that are all using MSG brush load internally.  

Studio Artist is automatically creating new presets for you on the fly.  But they aren't just total random mutations of the parameter space.  They are constrained in intelligent ways based on your personal specification.

Let's say you want to edit a paint preset to make it 'wet'.  That could take a number of very specific manual edits within different control panels inside of the paint synthesizer Editor if you wanted to do that manually. For just one particular approach to building a wet paint.

Now you can just dial up that generative paint preference in gallery show. Or you can run a specific smart Quick Edit command to do it for you at the top level working interface.

Applying specific effects (paint, ip ops, vectorizer, whatever) to specific areas of the canvas while not processing the rest of the canvas with that effect is an important component of building an overall final art image.  Certainly not essential, but extremely useful.  A great creative tool to take advantage of in your work.

You could always do this via manual selection and subsequent manual masking of an effect in Studio Artist.  But Studio Artist V5.5 can do it for you automatically now. Intelligently deriving an automatic selection mask on the fly.  You can dial this in with all kinds of different variations in Gallery Show, or you can run specific variations of it up at the high level working interface.

Is the 'intelligence' associated with this perfect? No.  Is it useful? Yes.  Will it get better as we move forward into the future?  Absolutely.

But what about movie support?  Is that back.  Yes it is.  You can now work with movie file io again (as opposed to being forced to use folders of frame images like you had to in the 64 bit build of V5 on mac).  You no longer need quicktime for windows if running on windows.   

Are all of the codec options you want available now?  Maybe not.  But we intend on expanding those as we move forward.  Yes we know you want ProRes.

We also want to add support for MNG in the future for movie brushes, since it provides a great open source container for lossless PNG images with embedded alpha channels.

Live video is also back in somewhat limited form. All of that will be turned on as we move forward, and we have some interesting ideas for expanding what people can do with it.  Multiple video sources being one of them.

What about other new stuff?  Well, there is quite a bit of it. Sprinkled all over the program.

There is a new positive/negative space visual attribute available throughout the program where ever visual attribute modulation is available.  It models the visual perception of positive and negative space in an image.  Since we now offer visual attribute modulation options for both the current source as well as the current loaded style, it's available for source and style.

We've built in this whole new conceptual notion of clipping an algorithm internally.   You can think of algorithm clipping as like working with selection masking, except the processing algorithm is masked internally as a part of its under lying internal structure.  As opposed to after the fact by masking what gets placed from the effect output into the working canvas.  Which is how canvas masking works.  

This is currently available in the paint synthesizer and the Vectorizer.   But you can expect to see that expand to other areas of the program in the future.

The Transition Context features have been greatly expanded (did you even know Transition Contexts existed).  More algorithm options, adjustments for them now, and the ability to route where the output of the Transition Context ends up.  You can route to the canvas, source, or style.

Transition Contexts automatically generate transition effects between different keyframes.  Each keyframe is associated with a specific image or movie file tied to that keyframe.  

Dual Mode Paint now lets you use the Vectorizer as a DualOp if you so desire.  We also introduced the concept of injecting time based modulation from Dual Mode Paint into an effect that overrides existing internal parameters.  This currently only works for Vectorizer Dual Ops, but is going to be expanded out to the other Dual Op choices in the future.

This override injection modulation feature is going to become very important as we develop the program into the future. Because it provides a way for other effects or signals to modulate an Op Mode effect without having to add endless internal parameters for those additional modulation options to a specific effect. 

We discussed Load Style in the initial discussion above.  Current support at V5.5 release is a loaded style image, but loaded style image folders are coming soon.

We also discussed Gallery Show in the initial discussion above. Many new Gallery Show features are available now.  New techniques, new paint transform and path path options associated with paint based techniques, a whole new generative paint preference tab with associated options, new generative options associated with vectorizer techniques, new source options for gallery show, new automatic intelligent masking options for gallery show, new built in generative options for the Start and End cycle processing, a timer to skip to the next cycle if using paint techniques that would otherwise run a long time, new automatic color palette generation options.

Many of the new gallery show generative options are now available in the high level working interface.  This includes the paint and vectorizer generative preference options which are currently available as QuickEdit commands in the Edit menu, automatic selection mask generation options available in the Canvas : Selection menu, paint draw and paint path generation options available in the Action : Art Strategy menu, generative dissipative image processing available in  Action : Art Strategy as well.

There is a new Gallery Show Toolbar you can use to control gallery show in the high level workspace interface.

There is also a new PowerTool Bar that currently supports some of the gallery show generative features when working in the high level workspace.  Our design specs have detailed descriptions of additional features that will be available in the PowerTool Bar in the future (like the ability to work with sets of PowerTool presets, and the ability to access QuickEdit commands directly in the PowerTool Bar).

There is a new M mutate button in the Preset Browser that lets you mutate new presets on the fly based on the currently loaded factory preset category.

We mentioned you can internally clip the vectorizer algorithm, which is currently available for region generation as well as region color modulation.  The old Draw control panel is split into 2 now, Draw Setup and Draw Apply.  There are a ton of new region modulation options in Draw Setup now.  There are new modulation options in the Draw Apply control panel for color gradients and cast shadows for vector rendering.

We mentioned you can internally clip the paint synthesizer algorithm. This is currently available in Paint Color Source as well as for brush Nib clip in Paint Fill Apply.  There are 2 new pen modes, freestyle multi-pen and freestyle region fill as brush.  You can embed vectorizer and ip op effects that generate vector output directly into a paint preset, and then use the resulting generated vectors for path sketching with paint or path start regionization.  You can choose whether the selection buffer is overwritten by individual regions when running path start regionization or not (it always did it before).  You can now chose whether the aspect ratio is maintained for resized image or image folder or movie brush folder paint nibs, or whether they resize to fit like they used to.  You can choose whether the visual attribute used for path angle modulation is based on the source or style image (more style modulation is coming in the paint synth in the near term future). New Brush Size modulation options.  Bezier paths can now store and restore brush size in addition to brush color.

We've got some experimental optical flow stuff going in Temporal Ip Ops that is going to be cleaned up in the short term.

There are various new IpOp parameters and features hiding inside of some of the individual IpOp effects.

Studio Artist V5.5 is based on a totally different internal framework (different than V5).  This framework is very current with recent OS changes, so things like cosmetic issues on mac Big Sur are gone.  It supports things like dark appearance display mode on mac.  This change may seem trivial to you, but believe me it was not trivial under the hood to implement.

There are various other new things i'm spacing on right now that i'll try to drop into this list as they come to mind later.

So, i hope this gives you some idea what the long awaited Studio Artist V5.5 release is all about.  And where we have pointed it for future Studio Artist development.  

We've also been working internally on V6 things as well, so that effort has been ongoing in the background and will now move to the forefront with the V5.5 release.