Converting Art to Data
There is something gross about breaking a masterpiece down into statistics, but there is also something profoundly beautiful about it.
Reproduced Cezanne's Houses at the L'Estaque with one of my painting robots using a combination of AI and personal collaboration. One of the neat things about using the robot in these recreations, is that it saves each and every brush stroke. I can then go back and analyze the statistics behind the recreation. Here are some quick visualizations...
It is weird to think of something as emotional as art, as data. But the more I work with combining the arts with artificial intelligence, the more I am beginning to think that everything is data.
Below is the finished painting and an animation of each brush stroke.
Simon Colton and The Painting Fool
The Painting Fool - Mood: reflective. Desired quality: simple.
Avoiding Uncreative Behavior
Was excited to recently make contact with Simon Colton, the artist and developer behind The Painting Fool. After brief twitter chat with him, heard about his thoughts on the criteria that made things "uncreative". If I understood him correctly, it is not so much that he is trying to make software creative, but that he is trying to avoid things that could be thought of as "uncreative", such as random number generation.
He directed me to one of his articles that went into details on this. What I read was really interesting.
The work begins with a rather elegant definition of Computational Creativity that I agree with.
Computational Creativity: The philosophy, science and engineering of computational systems which, by taking on particular responsibilities, exhibit behaviours that unbiased observers would deem to be creative.
There are many interesting thoughts throughout rest of paper, but the two concepts that I found most relevant were that
1) artificially creative systems should avoid randomness
and
2) they should attempt to frame, or give context to, what they are creating.
He begins by criticizing the over reliance on random number generation in computationally creative systems. Using the example of poetry, Colton writes that software could use random number generation to create a poem with "exactly the same letters in exactly the same order as one penned by a person." But despite fact that both works are identical and read identically, the poem created with random numbers is meaningless by comparison to the poem written by a person.
Why?
Well there are lots of reasons, but Colton elaborates on this to emphasize the importance of framing the artwork, where
"Framing is a term borrowed from the visual arts, referring not just to the physical framing of a picture to best present it, but also giving the piece a title, writing wall text, penning essays and generally discussing the piece in a context designed to increase its value."
In more detail he goes on to say...
"We advocate a development path that should be followed when building creative software: (i) the software is given the ability to provide additional, meta-level, information about its process and output, e.g., giving a painting or poem a title (ii) the software is given the ability to write commentaries about its process and its products (iii) the software is given the ability to write stories – which may involve fictions – about its processes and products, and (iv) the software is given the ability to engage in dialogues with people about what it has produced, how and why. This mirrors, to some extent, Turing’s original proposal for an intelligence test."
This view is really interesting to me.
In my own attempts at artificial creativity, I have always tried to follow both of these ideas. I avoid relying on random number generation to achieve unexpected results. And even though this has long been my instinct, I have never been able to articulate the reason why as well as Colton does in this writing. Given the importance of a creative agent to provide a frame for why and how each creative decision was made, random number generation is a meaningless reason to do something, which in effect takes away from the meaning of a creation.
Imagine being struck by the emotional quality of a color palette in an artwork then asking the artist why they chose that particular color palette. If the artist's response was, "It was simple really. I just rolled a bunch of dice and let them decide on which color I painted next." The emotional quality of the color palette would evaporate leaving us feeling empty and cheated that we were emotionally moved by randomly generated noise.
With this reading in mind and the many works of Simon Colton and The Painting Fool, I will continue to try and be as transparent with the decision making process of my creative robots as possible. Furthermore, while I do try to visually frame their decision making processes with timelapses of each painting from start to finish, I am now going to look at ways to verbally frame them. Will be challenging, but it is probably needed.
If you want to see more of Simon Colton and The Painting Fool's work check out the You Can’t Know my Mind exhibition from 2013.
Can robots be creative? They Probably Already Are...
In this video I demonstrate many of the algorithms and approaches I have programmed into my painting robots in an attempt to give them creative autonomy. I hope to demonstrate that it is no longer a question of whether machines can be creative, but only a debate of whether their creations can be considered art.
So can robot's make art?
Probably not.
Can robots be creative?
Probably, and in the artistic discipline of portraiture, they are already very close to human parity.
Pindar Van Arman
Are My Robots Finally Creative?
After twelve years of trying to teach my robots to be more and more creative, I think I have reached a milestone. While I remain the artist of course, my robots no longer need any input from me to create unique original portraits.
I will be releasing a short video with details shortly, but as can be seen in the slide above from a recent presentation, my robots can "imagine" faces with GANs, "imagine" a style with CANs, then paint the imagined face in the imagined style using CNNs. All the while evaluating its own work and progress with Feedback Loops. Furthermore, the Feedback Loops can use more CNNs to understand context from its historic database as well as find images in its own work and adjust painting both on a micro and macro level.
This process is so similar to how I paint portraiture, that I am beginning to question if there is any real difference between human and computational creativity. Is it art? No. But it is creative.
Artobotics - Robotics Portraits
While computational creativity and deep learning has become a focus of many of my robotics paintings, sometimes I just like to make something I am calling artobotic paintings, or artobotics.
With these paintings I have one of my robots paint relatively quick portraits, but not just one, dozens of them. The following is a large scale portrait of a family that was painted by one of my robots over the course of a week.
HBO Vice Piece on CloudPainter - The da Vinci Coder
Typically the pun applied to artistic robots make me cringe, but I actually liked HBO Vice's name for their segment on CloudPainter. they called me The Da Vinci Coder.
Spent the day with them couple of weeks ago and really enjoyed their treatment of what I am trying to do with my art. Not sure how you can access HBO Vice without HBO, but if you can it is a good description of where the state of the art is with artificial creativity. If you can't, here are some stills from the episode and a brief description...
Hunter and I working on setting up a painting...
One of my robots working on a portrait...
Elle asking some questions...
Cool shot of my paint covered hands...
One of my robots working on a portrait of Elle...
... and me walking Elle through some of the many algorithms, both borrowed and invented, that I use to get from a photograph of her to a finished stylized portrait below.
Robot Art 2017 - Top Technical Contributor
CloudPainter used deep learning, various open source AI, and some of our own custom algorithms to create 12 paintings for the 2017 Robot Art Contest. The robot and its software was awarded the Top Technical Contribution Award while the artwork it produced recieved 3rd place in the aesthetic competition. You can see the other winners and competitors at www.robotart.org.
Below are some of the portraits we submitted.
Portrait of Hank
Portrait of Corinne
Portrait of Hunter
We chose to go an abstract route in this year's competition by concentrating on computational abstraction. But not random abstraction. Each image began with a photoshoot, where CloudPainter's algorithms would then pick a favorite photo, create a balanced composition from it, and use Deep Learning to apply purposeful abstraction. The abstraction was not random but based on an attempt to learn from the abstraction of existing pieces of art whether it was from a famous piece, or from a painting by one of my children.
Full description of all the individual steps can be seen in the following video.
NVIDIA GTC 2017 Features CloudPainter's Deep Learning Portrait Algorithms
CloudPainter was recently featured in NVIDIA's GTC 2017 Keynote. As deep learning finds it way into more and more applications, this video highlight some of the more interesting applications. Our ten seconds comes around 100 seconds in, but I suggest watching the whole thing to see where the current state of the art in artificial intelligence stands.
Hello7Bot - A Tutorial to get 7Bot Moving
Couple of people have asked for how I got my 7Bots running. Writing this tutorial to demonstrate how I got their example code working. Will also be giving all the code I used to run my 7Bots in the Robot Art 2017 Contest. Hopefully this tutorial is helpful, but if it isn't, email me with any issues and I will try and respond as quickly as possible, maybe even immediately.
So here is quick list of steps in this tutorial:
1: Get your favorite PC or Mac.
2: Plug 7Bot into your computer.
3: Make sure Arduino Due is installed on 7Bot
4: Install Processing3 to run code.
5: Download 7Bot example code.
6: Open 7Bot example code in Processing3.
7: Make minor configuration adjustments.
8: Run the example code in Processing3.
9: Watch 7Bot come to life.
10: Experiment with my 7Bot code.
In more detail:
Step 1: Go on your favorite PC (Windows or Mac, maybe Linux?)
Simple Enough. Pretty sure it will also work on Linux, but I haven't tried it.
Step 2: Plug 7Bot into Computer and Electrical Socket
Use USB cable to plug 7Bot into your computer.
Also make sure 7Bot is plugged into external power source.
To see if it has power, hit the far left of the three buttons on the back of the robot. It should go to the default ready position as seen below.
Step 3: (OPTIONAL) If 7Bot is acting weird Make sure Drivers for the Arduino Due Are Installed
This step may or may not be needed and I am currently working to get to bottom of why this tutorial works for some 7Bots and not others. My 7Bots came with drivers for the Arduino Due pre-installed, though I have heard of of 7Bots where it sounds like they are not. If you think your 7Bot has already been set up you can proceed. If not try one of the two following strategies to figure out if 7Bot is ready to go..
1: Download the GUI at http://www.7bot.cc/download and try using it to get your 7Bot moving.
2: Download Getting_Started_with_Bot_v1.0.pdf and make sure Arduino Due drivers are installed from its instructions.
Step 4: Download Processing3
Processing3 is a java based development environment that is pretty straight forward if you are familiar with any of the major languages.
Download it here and install it.
Step 5: Download 7Bot Example Code
Go to github and clone their example code from the following repo.
If you are unfamiliar with github or just want the zip file, here take this.
Step 6: Open Example Code in Processing3
Go to the Arm7Bot_Com_test.pde example file you got earlier and open it.
It will ask you how to open the code, so specify the location of wherever you put the processing.exe file.
When it opens up in Processing3 it will look something like this.
Step 7: Make Minor Configuration Adjustments.
Early in the code you will find the following lines
// Open Serial Port: Your should change PORT_ID accoading to
// your own situation.
// Please refer to: https://www.processing.org/reference/libraries/serial/Serial.html
int PORT_ID = 3;
Change the PORT_ID to match the one you plugged 7Bot into. You can refer to the documentation shown, use trial and error, or find yours by adding the following line of code and running the program...
// List all the available serial ports:
printArray(Serial.list());
When I ran this, I saw the following...
[0] "/dev/cu.Bluetooth-Incoming-Port"
[1] "/dev/cu.Bluetooth-Modem"
[2] "/dev/cu.usbmodem1411"
[3] "/dev/tty.Bluetooth-Incoming-Port"
[4] "/dev/tty.Bluetooth-Modem"
[5] "/dev/tty.usbmodem1411"
And set the Port_ID to the first usb port, 2.
int PORT_ID = 2; //PINDAR - CHANGED FROM int PORT_ID = 3;
Step 8: Run the Code in Processing
Hit the Play Button in the top left hand corner of Processing3, and a small window should open and the robot should start moving.
Step 9: Watch 7Bot come to life.
Or not. Well it should start moving, but if it doesn't, first thing to try changing is the Port in Step 7. If that doesn't work contact me and I will try and work with you to get it running. Then I will update this blog so that next person doesn't have same problem you had.
Step 10: Have fun with my Robot Art 2017 Code
The example code from 7Bot is well documented and shows you how to do all sorts of cool things like recording the robots movements and then playing it back. I learned most of the code I used in the Robot Art 2017 contest by reverse engineering this example.
If you want to see my code, here is a version that has some of the extra functionality like easier to use inverse kinematics. But if you clone this repo, no judging my coding style. I like comments and leaving lots of them in as a history of what I was doing earlier in the process. Never know when I might need to reference them or revert. I know that is what version control is for, but I leave comments everywhere anyways. Hey. You promised you wouldn't judge!
Hope this tutorial works out for you. As mentioned earlier, write with any issues and I will try to clarify within 24 hours, maybe even immediately. 7Bot is awesome and I hope this tutorial helps you get it running.
Pindar
Elastic{ON} 17
Just finished with a busy week at Elastic{ON} 17 where we had a great demo of our latest painting robot. One of the best things to come of these exhibitions is the interaction with the audience. We can get better sense of what works as part of the exhibit as well as what doesn't.
Our whole exhibition had two parts. The first was a live interactive demo where one of our robots was tracking a live elastic index of conference attendee's wireless connections and painting them in real time. The second was an exhibition of the cloudpainter project where Hunter and I are trying to teach robots to be creative.
A wall was set up at the conference where we hung 30 canvases. Each 20-30 minutes, a 7Bot robotic arm painted dots on a black canvas. The location of the dots were taken from the geolocation of 37 wireless access points within the building.
There are lots of ways to measure the success of an exhibit like this. The main reason we think that it got across to people, though, was the shear amount of pictures and posts to social media that was occurring. There was a constant stream of interested attendees and questions.
Also, the exhibition's sponsors and conference organizers appeared to be pleased with the final results as well as all the attention the project was getting. At the end of two days, approximately 6,000 dots had been painted on the 30 canvases..
Personal highlight for me was fact that Hunter was able to join me in San Francisco. We had lots of fun at conference and were super excited to be brought on stage during the conference's closing Q&A with the Elastic Founders.
Will leave you with a pic of Hunter signing canvases for some of our elastic colleagues.
cloudpainter AI described in a Single Graphic
I always struggle to describe all the AI behind cloudpainter. It is all over the place. The following graphic that I put together for my Elastic{ON} 17 Demo sums it up pretty well. While it doesn't include all the algorithms, it features many of my favorite ones.
TensorFlow Dev Summit 2017 cont...
Matt and I had a long day listening to some of the latest breakthroughs in deep learning, specifically those related to TensorFlow. Some standouts included a Stanford student that had created a neural net to detect skin cancer. Also liked Doug Eck's talk about artificial creativity. Jeff Dean had a cool keynote, and got to learn about TensorBoard from Dandelion Mane. One of my favorite parts of the summit was getting shout outs and later talking to both Jeff Dean and Doug Eck. The shoutouts to cloudpainter during Jeff's Keynote and Eck Session and lots of pics can be seen below. This is mostly for my memories.
cloudpainter at Elastic{ON} 17
Less than half an hour ago I wrote about how I am on my way the first annual TensorFlow Dev Summit at Google HQ. There is more. While in Mountain View I will also be stopping by elastic HQ to discuss an upcoming booth that cloudpainter has been invited to have at Elastic{ON} 2017.
For the booth I have prepped 5 recreations of masterpieces as well as a new portrait of Hunter based on many of my traditional AI applications. Cool thing about this data set is that I have systematically recorded every brush stroke that have gone into the masterpieces and stored them in an elasticsearch database.
Why? I don't know. Everything is data - even art. And I am trying to reverse engineer the genius of artists such as da Vinci, Van Gogh, Monet, Munch, and Picasso. I have no idea what it will tell us about their art work, or how it will help us decipher the artistry. I am just putting the data out there for the data science community to help me figure it out. The datasets of each an every stroke will be revealed during Elastic{ON} on March 7th. Until then here is a sneak peak at the paintings my robots made.
TensorFlow Dev Summit 2017
About two months ago I applied to go to Google's first annual TensorFlow Dev Summit. I sent in the application and forgot about it. After a month I figured that I did not get an invite. Then about a week ago, the invite came in. Turns out only one in ten applicants were invited to the conference. I have no idea what criteria they used to select me, but I am currently on plane to Mountain View excited to talk with the TensorFlow team and see what other developers are doing with it.
The summit will be broadcast live around the world. Here is a link. Look for me in the crowd. I will have a grey pullover on.
Our First Truly Abstract Painting
Have had lots of success with Style Transfer recently. With the addition of Style Transfer to some of our other artificially creative algorithms, I am wondering if cloudpainter has finally produced something that I feel comfortable calling a true abstract painting. It is a portrait of Hunter.
In one sense, abstract art if easy for a computer. A program can just generate random marks and call the finished product abstract. But that's not really an abstraction of an actual image, its the random generation of shapes and colors. I am after true abstraction and with Style Transfer, this might just be possible.
More details to come as we refine the process, but in short the image above was created from three source images, seen in the top row below, and image of Hunter, his painting, and Picasso's Les Demoiselles d Avignon.
Style Transfer was applied to the photo of Hunter to produce the first image in the second row. The algorithm tried to paint the photo in the style of Hunter's painting. The second image in the second row is a reproduction of Picasso's painting made and recorded by one of my robots using many of its traditional algorithms and brush strokes by me.
The final painting in the final row was created by cloudpainter's attempt to paint the Style Transfer Image with the brush strokes and colors of the Picasso reproduction.
While this appears like just another pre-determined algorithm that lacks true creativity, the creation of paintings by human artists follow a remarkably similar process. They draw upon multiple sources of inspiration to create new imagery.
The further along we get with our painting robot, I am not sure if we are less creative than we think, or computers are much more so than we imagined.
Hunter's Portrait
Inspired by our trip to the National Portrait Gallery, we started thinking to ourselves, what's so impressive about making our robot's paint like a famous artist. Sure they are inspirational and a lot can be learned from them, but when you think about it, people are more interested in the art of their loved ones.
So this morning, Hunter and I decided to do quick portraits of each other and then run the portraits through deep neural nets to see how well they applied to a photo we took of each other. As soon as we started, Corinne joined in so here is obligatory photo of her helping out.
Also in the above photo you can see my abstract portrait in progress.
Below you can see the finished paintings and how they were applied to the photos we took. If you have been following this blog recently, you will know that the images along the top are the source images from which style is taken and applied to the photos on the left. This is all being done via Style Transfer and TensorFlow. Also I should note that the painting on left is mine, while Hunter's is on right.
Most interesting thing about all this is that the creative agent remains Hunter and I, but still something is going on here. For example even though we were the creative agents, we drew some of our stylistic inspiration from other artist's paintings that we saw at the National Portrait Gallery yesterday. Couldn't a robot do something similar?
More work to be done.
Inspiration from the National Portrait Gallery
One of the best things about Washington D.C. is its public art museums. There are about a dozen or so world class galleries where you are allowed to take photos and use the work in your own art, because after all, we the people own the paintings. Excited by the possibilities of deep learning and how well style transfer was working, the kids and I went to the National Portrait Gallery. for some inspiration.
One of the first things that occurred to us was a little inception like. What would happen if we applied style transfer to a portrait using itself as the source image. It didn't turn out that well, but here are a couple of those anyways.
While this was idea of a dead end, the next idea that came to us was a little more promising. Looking at the successes and failures of the style transfers we had already performed, we started noticing that when the context and composition of the paintings matched, the algorithm was a lot more successful artistically. This is of course obvious in hindsight, but we are still working to understand what is happening in the deep neural networks, and anything that can reveal anythign about that is interesting to us.
So the idea we had, which was fun to test out, was to try and apply the style of a painting to a photo that matched the painting's composition. We selected two famous paintings from the National Portrait Gallery to attempt this, de Kooning's JFK and Degas's Portrait of Miss Cassatt. We used JFK on a photo of Dante with a tie on. We also had my mother pose best she can to resemble how Cassatt was seated in her portrait. We then let the deep neaural net do its work. The following are the results. Photo's courtesy of the National Portrait Gallery.
Farideh likes how her portrait came out, as do we, but its interesting that this only goes to further demonstrate that there is so much more to a painting than just its style, texture, and color. So what did we learn. Well we knew it already but we need to figure out how to deal with texture and context better.
Applying Style Transfer to Portraits
Hunter and I have been focusing on reverse engineering the three most famous paintings according to Google as well as a hand selected piece from the National Gallery. These art works are The Mona Lisa, The Starry Night, The Scream, and Woman With A Parasol.
We also just recently got Style Transfer working on our own Tensor Flow system. So naturally we decided to take a moment to see how a neural net would paint using the four paintings we selected plus a second work by Van Gogh, his Self-Portrait (1889).
Below is a grid of the results. Across the top are the images from which style was transferred, and down the side are the images the styles were applied to. (Once again a special thanks to deepdreamgenerator.com for letting us borrow some of their processing power to get all these done.)
It is interesting to see where the algorithm did well and where it did little more than transfer the color and texture. A good example of where it did well can be seen in the last column. Notice how the composition of the source style and the portrait it is being applied to line up almost perfectly. Well as could be expected, this resulted in a good transfer of style.
As far as failure. it is easy to notice lots of limitations. Foremost, I noticed that the photo being transferred needs to be high quality for the transfer to work well. Another problem is that the algorithm has no idea what it is doing with regards to composition. For example, in The Scream style transfers, it paints a sunset across just about everyone's forehead.
We are still in processing of creating a step by step animation that will show one of the portraits having the style applied to it. It will be a little while thought cause I am running it on a computer that can only generate one frame every 30 minutes. This is super processor intensive stuff.
While processor is working on that we are going to go and see if we can't find a way to improve upon this algorithm.
Channeling Picasso with Style Transfer and Google's TensorFlow
We are always jumping back and forth between hardware and software upgrades to our painting robot. This week it's the software. Pleased to report that we now have our own implemention of Dumoulin, Shlens, and Kudlar's Style Transfer. This of course is the Deep Learning algorithm that allows you to recreate content in the style of a source painting.
The first image that we successfully created was made by transferring the style of Picasso's Guernica into a portrait of me in my studio.
So here are the two images we started with.
And the following is the image that the neural networks came up with.
I was able to get this neural net working thanks in large part to the step-by-step tutorial in this amazing blog post by LO at http://www.chioka.in/tensorflow-implementation-neural-algorithm-of-artistic-style. Cool thing about the Deep Learning community, is that I found half a dozen good tutorials. So if this one doesn't work out for you, just search out another.
Even cooler though, is that you don't even need to set up your own implementation. If you want to do your own Style Transfers, all you have to do is head on over to the Deep Dream Generator at deepdreamgenerator.com. On this site you can upload pictures and have their implementation generate your own custom Style Transfers. There is even a way to upload your own source images and play with the settings.
Below is a grid of images I created on the Deep Dream Generator site using the same content and source image that I used in my own implementation. In them, I played around with the Style Scale and Style Weight settings. Top row has Scale set to 1.6, while second row is 1, and third is 0.4. First column has the Weight set to 1, while second is at 5 and third is at 10.
So while I suggest you go through the pains of setting up your own implementation of Style Transfer, you don't even have to. Deep Dream Generator lets you perform 10 style transfers an hour.
For us on the other hand, we need our own generator as this technology will be closely tied into all robot paintings going forward.
Capturing Monet's Style with a Robot
As we gather data in an attempt to recreate the style and brushstroke of old masters with Deep Learning, we thought we would show you one of the ways we are collecting data. And it is pretty simple actually. We are hand painting brushstrokes with a 7BOT robotic arm and recording the kinematics behind the strokes. It is a simple trace and record operation where the robotic arms position is recorded 30 times a second and saved to a file.
As can be seen in the picture above, all Hunter had to do was trace the brush strokes he saw in the painting. He did this for a number of colors and when he was done, we were able to play the strokes back to see how well the robot understood our instructions. As can be seen in the following video, the playback was a disaster. But that doesn't matter to us that much. We are not interested in the particular strokes as much as we are in analyzing them for use in the Deep Learning algorithm we are working on.
Woman With A Parasol is the fourth Masterpiece we have begun collecting data for. As this is an open source project, we will be making all the data we collect public. For example, if you have a 7Bot, or similar robotic arm with 7 actuators, here are the files that we used to record the strokes and make the horrible reproduction.
