Geodesic Fill-In battles with itself - who will win

 

We reconstructed the beginning of the PASeq to use the paint synthesizer to do the actual painting. Working with the exact same set of Threshold IpOp effects bezier curves we used in the last post.  All vector paint.  

We did this so that we can use it to keyframe interpolate Transition Context keyframe images.

So then basically the exact same thing as before. Except we really clamped down on those Mix ratios for the various gradient lighting effects that generate the geodesic  fill out phenomena.

We have it dialed in at a point where chaotic reaction-diffusion turbulence is playing out over multiple frames in the animation.  

It would be cool to add another level of auto-masking to isolate the dancer more, and associated all of this reaction-diffusion stuff with just the rendering of her.

Core Structure vs Geodesic Fill-In - the battle for dominance

 

So a fascinating art strategy involves the dynamic inter-play between a 'core-central-structure' and an 'enveloping geodesic fill in process'.  And we can see this very scenario play out in the paint animation above.

Although 'paint' is a pretty loose term in this example.  Because we're doing it 'old school', building the entire structure of the rendered image from a series of Ip Op effects strung out to build up the overall art strategy.  No paint synthesizer involved.  You are building your own paint synthesizer out of Ip Op parts, laying them out to form the basic principals of your custom art strategy.

So in this example we started with 'mid-core' representation.  So not the internal skeleton, but the larger shape that defines that core.  We used an adaptive threshold effect in the Threshold IpOp.  We turned on vector rendering. So yes your image processing effect is now 'painting' with colored vector graphics.

You are always working with some kind of internalized structure representation when you are creating images.  You just might not be consciously aware of it. Get to know it, revel in it.  Figure out how it works and what you can do with it.

Once we have defined our 'mid-core', then we can derive an internal rod-skeleton that is the basic root structure of the shape.  We drew that in black with the Threshold IpOp thin vector option turned on.

The we filled in the mid core as a solid shape.  We used the Fixed Color IpOp effect to do that. Dial down the Mix to something appropriate. And before you run it, setup the selection mask you are going to be using when you do run it.  You can use that same adaptive threshold effect we've been using since the very beginning of this to do that (route the output to the Selection buffer by using the Replace Region Selection option in the Composite parameter).

We're using the 'mid-core' basic thresholded shape like a silk-screen print setup, using it to create a selection mask that will mask in our Fixed Color IpOp paint simulation.

So to give that 'mid-core' rendition some more visual depth, we will paint it in 2 or 3 times, expanding the Selection out using the Feather menu command, and also dialing down the Mix and rolling the hue and luminance of the source color we are running through the Fixed Color IpOp to do this 'mid-core' fill in. So we're building a very simple expand shadow fringe effect for the shape.

Shadow effects are a great approach to add additional visual depth to your painting.  The color roll adds visual depth, the feathering expansion of the mask adds additional structure and visual complexity to the fringe effect you are building.

Now our Core Structure has been defined. On to the Geodesic Fill-In.  Who will win, the battle for dominance.

We're going to 'bake in' our 'fill-in', by using a series of multi-octave gradient lighting effects.  Super low Mix ratios (they darken, unless you change the algorithm), and you raise the spatial frequency of the lighting effect each time you run it. So that modulation is a ramp up over successive cycles.  Choose odd-ratios to avoid un-welcome (unless you do want it) resonance effects if the ratios are truly harmonic.

Mix in some sharpen processing (mid-frequency, not too fine). I use the Blur IpOp effect to do that (set Composite to Edge1, low Mix ratios).  Add a little bit of Smart Blur that tracks the canvas orientation to smooth it all out. Sharpen again if necessary.

Use the Image Compressor to slam the floor and ceiling of whatever you are time cooking in the canvas out to the absolute maximum levels the media can record. Make that adaptive (not too much, just the right small amount) to master that baked image to tonal consistency.

You need to sit back, reflect on, and appreciate the fact that you are literally cooking a digital video simulation to the point where it starts to become a chaotic system (the multi-octave gradient lighting effects are brilliant at doing this).  You can experience reaction-diffusion behavior, as well as turbulent processes. Th smallest changes in low ratio Mix settings can have a dramatic impact on the visual look of what you are cooking. And the cooking occurs because the whole thing is a recursive process that plays out over time, over multiple frames.  You are painting in time.

I like to throw in some sparkle dust with the Smart Contrast IpOp effect.  Which can be configured to take rendered images like these and add some additional shadow structure.  Process the canvas, use a low Mix ratio, use the Min Composite option, and choose your algorithm wisely.

Un-Sharp mask at the very end, for some additional shine.  It's hard to use too much (although you certainly can and will).

Why Not Let the Generative AI Edit Itself?

The 5th level of digital visual art creativity involves the use of artificial intelligence to automatically create new imagery and associated presets for you.  The human artist dials in a generative art strategy, and then lets the machine do all of the drudge work.  

The artist's valuable time can then be better spent curating the output of the automatic intelligent AI Generative Assistant.  Sifting through the end results for the finest output to use for whatever it is they are trying to do.  

Output being rendered art imagery, as well as auto-generated presets that can then be later used to process source imagery to create new art output imagery. Or as new food for the generative AI system to create more generative visual imagery and more generative digital preset effects.

But what about the AI system itself.  Surely we aren't required to manually program its internal preferences by hand.  Where are the QuickEdit commands to let us perform higher level cognitive editing of the underlying system parameters. Where is the new and improved AI Generative Assistant to help us out in this daunting process?

Studio Artist V5.4 introduces a new innovation in generative art strategy synthesis. Generative AI that intelligently edits its own generative behavior.  

The human artist can still remain in the loop (if they want to do that).  Humans or intelligent felines can still control this process through the use of Generative AI Preference and QuickEdit options, as well as interactively while Gallery Show is running with the new GalleryShow ToolBar 'GS-Zap' button.  

Every time you press GS-Zap, the generative art strategy is intelligently edited by the AI system itself.  Boldly going where no one ever thought to manually edit a path too.  But now you can explore it, because your AI Generative Assistant takes you there. By generatively editing its own behavior.  In real time, as it's working.

But is it AI?

I've been waiting for this question to popup, and it did. Here's a quick response. More later. 

A few quick points:

1: The AI community had no problems calling Harold Cohen's AARON work 'AI' for over 3 decades, featuring it prominently at various AI conferences and AI publications, even though it was just a hand crafted rule based system once you go into the specifics of it. Encoding 'artistic knowledge' in that hand crafted structure.

Just to be clear, we love all things AARON.  I wish Cohen had written more about the specifics of its implementation.  Unfortunately, he is no longer with us.

2: There is currently a huge fascination with deep learning neural net systems in the AI community, and in the general press, and by diffusion from that press response to the general public at large.  And people should be excited, these systems can do amazing things.  They are going to revolutionize society. 

All of the growth in cloud based computing GPU resource consumption is being driven by deep learning architectures being run in the cloud (take away message from Jenson's keynote at GTC yesterday).

But how do they really work?

What are these deep learning systems really doing?

We are just going to focus on image based stuff for the point of this conversation. But extrapolate that out to audio, or 3d point clouds, or chunks of text, or whatever else you care to.

Deep learning neural network take a set of images (the data),and then they learn statistics associated with that collection of images (the data).  What they learn (the statistics they model) is only as good as the data they are trained on.

And there are 'priors' built into the system.  These 'priors' are actually innate knowledge built in the system (inadvertently in many cases)  by the architecture constructors themselves. And also hand crafted into the overall system by the kinds of data augmentation the user of the system works with.

So already these magical software2 learning systems have some hand crafted by humans knowledge built into them.

The end result is that the trained neural net system has some kind of feature space representation of the statistical properties of the data it was trained on internal to the neural net model used for the training.

Now what these systems are really doing is function approximation.  A deep neural net can learn any function approximation (in theory), it is a universal function approximator. So it's leaning some kind of nonlinear transformation. Usually an extremely high dimensional nonlinear transformation.

Now as it turns out, the real world and imagery associated with the real world, (images of faces, or people, or cats and dogs, chairs, whatever),  all of this stuff lives on a low dimensional manifold.  Higher than 2 dimensions, higher than 3 dimensions, higher than 4 dimensions, but way lower than what would be the case if the structure of the universe, the structure of images, if that was all just random.

And the trained neural net works in some sense because the real world lives in this low dimensional manifold (higher than the 2 dimensional plane an image is laid out on, higher than the 3 dimensional space you move around in, but low dimensional compared to what it could be if things were just random).

Think of an image that is just uniform random noise. It doesn't look like anything except noise. Think of an image of a person's face.  there is a lot of structure in that face image, both locally (adjacent pixels are similar, not radically different). And if you think about how the face changes as it rotates, or as the light hitting it changes direction, there is an inherent similarity associated with those changes.  That information lives on a low dimensional manifold.

Google manifold if the mathematical abstraction seems too obtuse. Or manifold learning.

But what about the human visual system?

How does it really work?

The human visual system also internally models something associated with the statistics of the real world images it perceives.  In a series of transformational mappings that occur in the transformation of an incoming visual signal into the brain through successive processing layers of the visual cortex.

Analogous to what is going on in successive layers inside of a deep learning neural network, but also different. Because what the neural net is learning is ultimately tied to the task it was trained on, and the function approximation might be specific to that task (the specific kind of function approximation that solves the task, which could be very different than the human perceptual representation that occurs in the brain). 

You can google visual cortex if you want to learn more.

I have been involved in the application of human visual models to various engineering problems for 40 years at this point.  From my masters thesis onwards. If you are curious about the internal structure and gory details of these kinds of engineering models based on neuroscience and psychophysical vision experiments, you can check out this post.

If you don't think that these kinds of models and associated research have anything to do with the human perception of artwork, i would encourage you to check out these books written by extremely esteemed neuroscience researchers.

Vision and Art by Margaret Livingstone here. Including a foreward by Hubel himself (Nobel prize for his work in visual perception).

Inner Vision: An Exploration of Art and the Brain by Semir Zeki here. Zeki basically discovered cortex area V4 and how it works.  

It is a very sad statement that neither this book or Zeki's other super great human vision book titled 'A Vision of the Brain' that you can find here is not in an e-book kindle format to read electronically.

Studio Artist is not a NIPS paper. We are not trying to find a new solution to some greatly constrained technical problem that we can then write a paper about so we can get tenure or a job at Google or Facebook.

Studio Artist is not a neuroscience research project.  We do track the neuroscience research, we do track the work of the theorists (like Tomaso Poggio at MIT) who try to build theories of how it all works mathematically (check out this lecture if you are curious).  We do track the academic research of art theorists who come at this whole area from a very different perspective.

Studio Artist tries to take work metaphors from music synthesis, music synthesizers, and repurpose them for digital visual artist. Concepts like signal modulation (visual signal modulation) are extremely important to our world view.  We try to incorporate visual signal modulators directly based on the neuroscience research into how humans perceive visual imagery, and then make them available throughout the program.

How humans perceive a piece of art is ultimately derived from the inherent visual statistics associated with that work of art.  Some people have weird hangups about the term 'art'.  So substitute 'visual image' if you are one of those people.

Visual modulation derived off of human perceptual visual attributes is a key component of how StudioArtist works.  And i just told you it is also a key component in how you perceive art (visual images).

Studio Artist also incorporates all kinds of internal 'heuristic' knowledge into it's internal workings.

Again, we're not trying to write a NIPS paper here. We're trying to build a system for digital artists to make art. Whatever we can throw into that system to make it work better is fair game.

Studio Artist is an active dynamic system. It tries to look at an image like a person would, and then react to that stimuli by building up an art output representation of that perceived 'source'. Like an artist would look at a model (or a poloroid photo of a model), and then re-interpret it.

The paint synthesizer is not an 'image filter'. You could probably configure it to be one if you really want it to work that way for a specific preset setting. But at it's heart it's a system that does active drawing. 

Yes, i have also dived into the literature on the neurobiology of muscle movement planning in the cortex. 

The initial prototype implementation of the paint synthesizer was conceived to drive a robot for physical painting.  That quickly changed as people started using it.  That has always been my philosophy, start an engineering project, get it to a place very quickly where people can start actually using it, then react and adapt to how those people use it, and the whole project takes on a life of it's own.

Studio Artist is not trying to replace the human artist. We are trying to augment the human artist.  The human artist can choose what level they want that augmentation to occur at. StudioArtist can literally do all of the work, or Studio Artist can aid the artist as the human artist manually paints, or Studio Artist can wiggle bits in the paint while the human artist does all of the stylus driving.

Studio Artist V5.5 is all about expanding the range of what the program can automatically assist the human artist with, letting the program automatically and intelligently build new presets for the system. As opposed to the human artist having to do all of that manually.

Studio Artist V5.5 is also about expanding the range of what a 'source' for a digital art program even means. 

Neural nets work off of a database of images to learn to model statistics inherent in that database of images. But an artist can work directly with the database of images, feeding it into Studio Artist, and then generating artwork created from the aggregate statistics inherent in that collection of images. You don't necessarily have to train an abstract model to learn those statistics in some hidden latent space of that model to generate art work derived from them.

All of this sounds like intelligent behavior to me in some sense.  And it is happening in a computer program, so it is in some sense artificial.

I guess it depends on your definition of intelligence.  And your definition of artificial.

Some of the intelligent behavior may seem stupid at times. But i could say that about people.

Some of the intelligent behavior might seem rote or repetitive in nature, but i could say the same thing about people.

Some of the intelligent behavior is derived from heuristic rules hand crafted by people, but i could say the same thing about the behavior of people.

But it's all based on statistics some people might say, but i could say the same things about neural nets.

Again, at the and of the day, it's a system to help digital artist make artwork. nothing more, nothing less.

And that's really how one should judge it.

Could it be better at certain things,? Sure, absolutely.

Are we going to make it better?  We are certainly trying.

Painting on a Time Canvas

 I keep noticing that almost no one is approaching video processing in Studio Artist the way it's setup to really do it.  And i think part of the problem is that folks are used to working in other video programs like FCP, AFTEFX, PRIMR etc, where their whole design is built around this notion of 

'process an input frame into an output buffer - write out the buffer and wipe it clean for the next cycle in the loop'

Which is fine for what those programs are designed to do.

But at its core since day one (back when dinosaurs roamed the earth and 400 MB hard disks were the size of toaster ovens), Studio Artist has been about digital painting. And i don't mean silly drag a single nib image around as a brush kind of 1988 digital painting (although we are more than happy to let you do that all day in Studio Artist if you so desire).

Real world media has more finesse to it.  More complexity. More chaos inherent in the system. You inherently get your hands dirty when you even mess with it.  Things leak and spill all over the place.

And we wanted to build a digital system that behaves with the same level (or more) of rich organic aroma.  If you can visualize that 'rich aroma' in a visual manor.  Part of the fun when working with real art stuff is that it has a mind of its own.  And if you follow that, you can end up in all kinds of unexpected really amazing places.

Or you can ignore it and just concentrate on the outcome.  It looks vibrant (if that's what you're going for, again we try not to judge and limit what you can do, because of some mistaken pre-conceived notion of what is a proper visual aesthetic).

And we certainly aren't going to limit you to some even more limiting pre-conceived notion of what 'natural media' is or isn't.  A mistake a lot of other digital art programs make.  Are actually very religious about in fact.

And the feedback we get from visual artists about our approach confirms that.  Oftentimes they are way more interested in a 'hyper digital natural media' kind of thing. Maybe it's reminiscent of some specific kind of natural media marking system, but its wildly unique and digitally imaginative, fresh and distinctive at the same time.  They like the fact that it is its own thing.

Responsiveness is of course always important.  It's at the heart of Studio Artist's internal parameter modulation structure.  Real world art stuff is very responsive.  To the touch.  To the temperature of the room. To the characteristics of what you are trying to apply it too.  To the bumps on that stick you are dragging around with paint on it.  And we try to do the same (or more).  It's built into the system from the bottom up. Parameter modulation everywhere, and lots of it. More and more all the time.

So we've always cared about painting. From day one.

But very quickly we also started looking into using Studio Artist to process movie files.  It was built around Quicktime from day 1 (Quicktime being a cool and revolutionary thing back in the day, now extinct like the Dodo or Carrier Pigeon, killed by an evil entity).

But in the very beginning our painting focus was very much 'let's paint on a canvas'.  Canvas being this 2D digital surface.

And Studio Artist also incorporated from the very, very beginning this whole notion of an artist painting a model source onto the canvas. And we wanted Studio Artist to look at and analyze the model source just like a human artist would do. So we built visual modeling into Studio Artist from day one as well. Human visual modeling based on cognitive neuroscience research that looks into how the visual cortex is structured, how it processes its input signals.  And also theoretical computational visual models of what various folks thought was going on in those visual areas inside of the visual cortex.

We also incorporated AI Heuristics into the system as well.  Lots of them, more and more all the time.  If you want to think of that part as a rule based AI system based on expert knowledge from artists, sure why not.  Different kinds of AI for different tasks (or mixed up together in the same tasks).

I think we need to make it clear that our business is making visual art.  And enhancing what visual artists can do in the digital world.  Building an extremely flexible creative tool for them to use in the digital art world.  We're not trying to write a NIPS paper, where we present the perfect algorithm for some heavily restricted task run on a standardized data test set.  We're trying to help people make art.

So anything goes really.  We play loose and fast with some aspects of the modeling when we have to (computational limits are always there, or maybe a really bad model actually looks more interesting).  And we love mistakes.  Because oftentimes they look really really amazing. So why would you remove them from the program if that is the case.  You just tie them to a specific parameter setting.  Because anything that looks visually different is interesting, worth keeping around, something to use in your tool chest of visual effects.  We aren't going to prevent you from using that, just because it was the result of a programming error (crashes are a different story, we're talking about algorithm mistakes that lead to interesting visual results).

Now the thing about painting is that there is this inherent dynamic tension between applying pigment to a surface, and the dissipative action of the fluid that holds and spreads that pigment.  The fun stuff is in the interactions going on in the system.  Pigment and fluid mixing. That amalgamation of pigment and fluid interacting with a surface. A textured brush or other applicator object interacting with all three.  Time interacting with that whole system of discrete elements as the fluid dries out, or the surface gets wet, or doesn't.

All of that stuff interacting together is what generates all of the fun. Generates the visual fun of the end result. Generates the fun of driving that stuff applicator over the surface (over time), to create something interesting (one hopes).

So there is a very definite dynamic tension going on in the system between a dissipative effect, and a marking 'lay down stuff' application effect.  They fight each other for control. So you want to be aware of that, and take advantage of that in your work.

And work is a real loose term. Because Studio Artist has always bridged these 2 product categories that seem to be kept completely separate by wizened marketing folks at other entities that release software into the world.  The universe of 2D graphics programs. And the universe of video programs.  And never shall the twain meet, except  at the file : import / export level. 

And when you start to look at video processing, movie file processing, digital rotoscoping, paint animation, whatever you want to call it, you want to make sure you apply the same set of principals you apply in the 2D 'spread it on the canvas' world to the temporal time-based 'frames in a row' video or movie canvas.  You want to make sure you keep that rich and powerful dynamic tension going on.  The dynamic tension between the dissipative forces, and the 'lay down the marking stuff' forces.

You really want to think of your video processing as 'painting in time'.  And when you do that, not only will your creativity be expanded 10 fold or more, you will also start creating some really interesting visual results because of it. Visual results that happen over time.  Because you are painting over time.  You are building a temporal canvas.

And how did we say painting works? You lay down 'marking stuff' onto a surface. And some kind of 'fluid' is interacting with that stuff, and also interacting with the surface you are applying it all too. But the surface is a series of consecutive frames laid out over time.  The 'fluid' (whatever that term even means, do not get literally or rigidly attached to it, remember anything goes when we are talking about art), fluid interacts over time as well. Interacts with those consecutive frames, and the consecutive laying down of stuff, and with the remnants of itself, all interacting over time. 

Time is an essential component to the whole process.  That's true for 2D painting, or time based painting.  Time in 2D painting can be thought of as movement.  Movement occurs over time. Movement builds up the final painted image. The dynamic tension between dissipative forces and marking forces on a surface occurs over time based on movement.  That's what painting is really all about.

Painting on a time canvas has both kinds of movement over time.  There is movement on the surface, or over time, or both going on simultaneously.  But the canvas itself is time based (discrete elements of time we call frames).

Now once you wrap your head around this notion of video processing being painting over time, then you are free to envision it in a very different way than what is laid out for you in the conceptual design of those other video programs.

One key point is that you should really be thinking about painting on top of the last output frame. As opposed to wiping the slate clean and starting fresh with every new frame. Because if you do that (wipe the slate clean) you have immediately lost one of the main places where you can create dynamic tension between dissipative forces and marking forces, a dynamic tension that occurs over time, over multiple frame images.  You really want to take advantage of this, capitalize on it, use it in your artistic process.

You also need to think heavily about what 'dissipative forces' means to you.  Because how you approach that part can really help to define your visual style, make it unique, make it fresh, make it distinctive.  We're going to educate you on some approaches.  But you should really learn them, understand the mechanics of how they work, and then head off into the wilderness, boldly exploring where no one has gone before. Or maybe re-exploring what has been very heavily trodden (if it looks cool).

Is dissipative fluid like wet paint?  Like a water emulation in the paint synthesizer?  Sure, could be. I use that all the time in my work (maybe a little too much at times, like symmetry it's an easy crutch to fall on). But you certainly aren't restricted to water emulations to create a dissipative force.

You could use noise randomization (most likely in association with some kind of mixing, or blending). You could use warps, pushes, nudges.  How they interact with what is already on the canvas, how that leads to movement over time, they all look different.  Spend an afternoon playing around with different approaches to warping or nudging, to create a dissipative force.  Learn how to use it as a tool in your tool chest.

Now for 2D painting, how the dissipative force moves around on the 2D surface is the deal. But when you are talking about painting over time, then you need to factor in some kind of recursive element, so that the force 'lingers' over several successive frames.  The dissipative force is temporal in nature, in addition to probably having some 2D dissipative element as well.

So you need to think about different scenarios to pull this off, different generative strategies that play out over time in an animation, over time in processed video frames, that work to help generate the final painting over time, your digital painting on a time canvas.

You also need to lay down that 'marking stuff', whatever it may be (again, don't get hung up on it, there are lots of different approaches, all are valid).  And by painting on top of your last output frame, you are inherently introducing temporal continuity into your time based painting. And you really want to shoot for that.  You want that dynamic tension between interacting forces to play out over time.

So that was a mouth full. Something to think about before you head off to process some video frames in Studio Artist.  Have fun.