My first robot project, a self driving car in 2005

Just reading that all new TESLA vehicles will be completely self driving and it made me think about my very first robot project. For the 2005 DARPA Grand Challenge II, I was a member of Team ENSCO and we built a self driving car that drove 86 glorious miles before careening into a tree.  The robot and our work on it can be seen in the video below. 

You can also see some cool things on our test vehicle, my 2005 Honda Element.  That weird looking thing on top is an experimental Velodyne LIDAR system.  Whenever you see a self driving Google car, it usually has the modern version of this contraption spinning around on top.  For two weeks we experimented with the very first prototype in existence.  I was actually pulled over by the Capital Police as we drove this vehicle around Capital Hill on a test run.  The officers nearly arrested me after asking me what the spinning thing on top of my car was and I foolishly responded "It's an array of 64 lasers, um wait, they aren't harmful lasers, let me explain..."

Among many interesting lessons in the project was marketing. Over the course of the project we would always spend lots of time explaining what an autonomous car was.  No one understood the word autonomous yet everyone in the industry insisted on calling them autonomous. Well in the ten years since it would appear that the marketers finally got involved and had the wisdom to just call them "self driving".  Which just shows you how clueless we engineers are.

The Early Robots

Went searching for and found these images of the early robots.  

The first robot started with a lot of wood pieces. 

The very first paint head was a disaster, but this second one worked real well.

The Second Robot was built to be more mobile.  It fit just barely through doors.

The third, of which I have lost all the photos except this one, was built to be massive.  It filled a room.  Also important was that it had an open face so I could photograph the painting more easily in order to use feedback loops in the painting logic.

After these unnamed machines came Crowd Painter, BitPaintr, Neural Jet, and cloudpainter.

Harold Cohen's AARON

Theo, 1992. Oil on canvas, 34x24 inches Photograph by Becky Cohen.

I recently received an email from Harold Cohen's assistant that he was sorry for not getting back in touch with me sooner, but it was because Cohen had passed away earlier in the month. This was earlier this year.

We had been talking at length about artificial creativity, and I was wondering why Cohen stopped talking with me all of a sudden. At the time he was helping me prep for my TEDx Talk on artificial creativity and was not shy in his critique of both my talk, and how he though I might be exaggerating my robot’s capabilities. As we talked, I found that our conversations on the subject often lasted far longer than it seamed either of us had planned for.  The email his assistant was responding to was actually a draft of my TEDx Talk that I had sent him for review.  I never heard back from him and figured that maybe he was no longer interested with my views on the subject. I had no idea his health was failing at the time.

In our talks I found his views on painting robots to be remarkably insightful and a little cantankerous. They were what you would expect from a man 40 years ahead of his time.  His first painting robot AARON, was built in the 70s when no one else was even considering some of the concepts he was exploring.  In our talks one of thing that stood out was his belief that a painting robots primary shortcoming was that it did not create its own imagery.  He was obsessed with the idea that most were merely printers executing a filter on an image.  Perhaps a filter more complex than something you find on Instagram or Snapchat, but a filter none-the-less. Though I can not find the quote I do remember reading something by him that was to the effect of "There are two kinds of painting robots. Those painting from photographs, and those lying about it."

I wish we had longer to talk with him ,because even though we disagreed on a lot, he was absolutely right about one critical aspect of robotic art.  The ultimate goal is to break free from filters.  I don't know what that means exactly, but whenever I create a new approach to artificial creativity, I ask myself how much of a filter it is, and try to make it less so.

Painting Robot's Ultimate Goal is to Eliminate All Humans

If you want a quick synopsis of the current state of my painting robot, this Thrillist Feature captures it perfectly.  They somehow made the otherwise dry subject of artificial creativity entertaining, and sometimes funny.  I really appreciate all the work of the film crew that worked on this with me and brought out some of the best aspect of this project.  Big Thanks to Vin, Joshua, Mary, Peter, Mat, and Paul.  

Robotic Arms, and The Beginning of cloudpainter

So we have long realized this, but now we finally have a plan.  For this painting robot to truly be able to express itself, it needs arms.  So we are planning on adding a pair, if not four.  A conceptual 3D model showing where we would put the first two can be seen below.

So the way we are thinking about this whole robot coming together is to add a pair of robotic arms above the painting area.  They would hold brushes, or maybe a brush and a camera.  Still deciding on this.  But as currently envisioned, the XY table will control the airbrushes, and the arms will control the traditional artist brushes.  Lots of reasons for this, least of which we think it will look cool for them to dance around each other.

We expect to have one of the robot arms, a 7bot, here in a couple of days.  Can't wait to see what we can do with it. 

Another thing we are realizing is that this is beyond the scope of the Neural Jet.  This new robot, a machine with a modular paint head on an xy-table and two robotic arms, is sort of a new project.  So from here on out while the Neural Jet will refer to the modular paint head, the project in its entirety will be referred to as cloudpainter, and will encompass all the tech and algorithms from all of my previous robots.

Bonnie Helps Out With Logo

Not sure how I got distracted by logos today, but I did. Bonnie and Corinne played a big part in the process.  First inspiration came from Corinne, who keeps calling the paint head a flower.  Then Bonnie mentioned if it had a logo, it should both look like a flower, and incorporate the shape of the paint head.  

Based on their input, I started playing with ideas and the number 9 became a big part of it. Besides the obvious fact that the paint head has 9 modules, I also based a lot of the proportions on factors of 9.  For example, the outer radius was measured at 36mm, while the inner circles had radii of 27mm, and 18mm.  The width of the internal line is 9mm, and all sorts of other stuff. As it was progressing some other cools ideas emerged that I left in for people to discover in the future.

So if you are looking at the picture above, you can see where it began on the left, and where I ended up on the right.  The final image is the logo that I think I am going to go with. Gonna think about it for a couple of days.

Talked Portraiture With Chuck Close at the White House

As you know, I was invited to White House for SXSL.  I have many good pictures of what was a real fun day. But I wanted to share this one in particular 'cause I am still processing how awesome meeting Chuck Close was.

So when I saw Chuck Close on the South Lawn, I realized that I had to introduce myself. As an artist he has always been a favorite, if not the favorite, though its sort of impossible to rank something like that.  I have long imitated many of his concepts in my own art.  I even designed my fourth robot to fit large canvases so it could paint portraits on the scale of his work.

While I wasn't expecting to be starstruck when I went up to introduce myself, I was. I didn't know what to say so I just told him I was a portrait artist to which he replied that he was sorry to hear that as it was a horrible line of work to be in. Then we chatted about portraiture briefly where he made a couple of other jokes before asking to see some of my work.  I wasn't expecting that, either his interest in my work, or his compassionate humor.  I showed him some portraits of my family on my phone, said my thanks for taking the time to talk with me, then went on my way.

I wanted to talk to him longer but at same time didn't want to be a harasser.

Thrillist Hero Shot

Thrillist Video just sent me this "hero shot" of me with the most recent robot.  Looking forward to their video piece.

Awesome Video Shoot With Thrillist Crew

Had such a cool day.  A film crew from Thrillist showed up to interview me about my painting robots. Thanks to Vin, Josh, Mary, Peter, Paul, and Matt.  Was so fun.  Lots of good footage including this shot that makes me look real important.

Ten Years of Progress on Painting Robots

These pics both show my very first painting robot head and the most recent one.  The first, which could hold only one brush, was made from parts found lying around my house including old pieces of wood, a handmade electromagnet, tape, and deck parts.  The most recent can hold and operate nine different kinds of brushes and is almost completely 3D printed.  Some of the plastic even glows in the dark.

Airbrush Actuator Complete

It is amazing how much an invitation from the White House can speed up development.  Long hours this weekend went into getting a working airbrush prototype.  While paint brushes will remain the primary mark making device in the Neural Jet, it will be cool to have them backed up by five airbrushes with the ability to quickly paint backgrounds.  

Also cool that we went with the servos instead of something like a solenoid to control air flow.  With our servos we can actuate the air coming out to 16 different pressures.  So mixing becomes possible, and since we have 5 airbrushes on the paint head, the Neural Jet will be able to paint over 1,000,000 colors (16^5).  Yeah this part of the project really is just re-inventing a printer, but coupled with the other mark making tools that are coming, it will be on the next level.  You can see the prototype in action below.

We Just got invited to the... um... White House

We just got an invitation to the, um... White House.

So, WOW! We just received the following invitation...

Looks like we are going to have to step up the schedule to have our Neural Jet prototype working in the next couple of weeks.  It is a little ambitious, but we just had a design meeting and worked out a plan that would have a good portion of the functionality ready for an interactive exhibit.  

Wish us luck as we had no choice but to accept the invitation. Now we are aiming to amaze and impress at the inaugural SXSL Festival of Ideas, Art, and Action.

Neural Jet Modular Paint Head Complete

We have finished printing what we think will be the final paint tray. In the image above you can see how it will be oriented to the canvas when it is being used to paint. Coincidentally enough, we arrived at the final design on the Model T - Like Henry Ford's car.

(Model T) can hold multiple paint brushes and up to five airbrushes, with modules that fit three different types of airbrushes.  People sometimes have a hard time imagining what we are showing in the photos, so I have included the following schematic that shows how the 5 airbrushes fit together.

The empty slots in the graphic above can be fitted with just about any mark making device, though I am partial to paint brushes.  The reason we want to use paint brushes is that we have yet to find anything that can make marks similar to those in the image below.  And thats what we are ultimately after.

Refined Paint Head (Model R)

Based on the initial prototype, which we were actually very pleased with, we went back to the white board and tackled some of its shortcomings.  The primary issue we had to address was how was a robot arm or xy-table going to hold on to this thing.

That was when Hunter came up with the idea of supporting the entire paint head from the center with 2" bolts.  So we redesigned the tray to hold 9 2" 6-32 bolts.  Above these bolts an aluminum disk would serve as the mounting surface.  With this configuration any module that consisted of a 40 degree arc with a 3.5" radius, could just snap into place.

As always, here is the final Google Sketch file so anyone that wants to can design this along with us.

Paint Head Prototype Constructed

Hunter and I constructed the first prototype.  It consisted of ten physical parts. The tray and nine plug-in modules.  If you want to experiment along with us and this prototype, you can download the 3D plans from our downloads page.

Once 3D printing was complete, some parts of the tray were glued together with super glue. The modules could then snap into place and be removed as needed.  While we hope the final paint head does not need any glue, we were experimenting with how things fit together and super glue offered us the most flexibility at this stage of construction.

Based on a several lessons learned with the Q prototype, we set out on a number of improvements for the next iteration.  The main problem we are now trying to solve is how to suspend The Neural Jet from either an xy-table or a robotic arm.  

Using 3D Printer to Manufacture Parts

Nothing makes me realize that The Future Is Now more than our new 3D Printer. We are manufacturing the parts we need for the Neural Jet. And the creative process has taken on an iterative nature.  The plan is to have a modular paint head and our design has taken us through seven iterations so far. You can see several of the early wedges that will fit together on our modular head in the pic below.

Some of the attempts went poorly, as can be seen by the tangled mess of plastic on the left. But the design got better as did our proficiency with the 3D printer.  Ultimately we ended up with the following version, called Tray G.

As can be seen in the following picture, this tray holds a standard sized airbrush and resevoir almost perfectly.  When arrayed in a circle, all the airbrushes should be pointing to a central point, and hopefully be able to paint in unison with one another.

3D PRINTER ARRIVES - OPENING, UNPACKING, AND ASSEMBLING

Our 3D Printer, a Wanhao Duplicator I3 Plus, finally arrived from China. Took about 2 weeks to get here. I wanted to open it as soon as we got it, but instead we decided to make an unboxing video. The results were kinda ridiculous. Typical unboxing videos are done by experts, but we quickly realized we didn't know what we were doing and stumbled through ours. But it was a fun way to learn about assembling and putting together a 3D Printer, so we hope you enjoy it.

Modular Approach?

So the idea of going modular came up, and it just makes sense.

At the design meeting we decided to make a paint head that was versatile enough to be mounted on either a 3D arm or an XY Table. Why stop with the customization there? The new concept is to give the paint head a configurable set of mark making tools from which it can operate.  We are having another design meeting this weekend to go into the particulars.  This picture though shows a rough first conceptualization of a nine module system.

Oh... also in keeping with the open source nature of this project, here are the google sketch-up documents to go along with this mock-up.  We plan on sharing everything so anyone can follow along and contribute.

Design Meeting

Dante, Hunter, Corinne, and I sat down to design our new painting robot and define its technical requirements.

The biggest decision made, among many important ones, was to design the new robotic paint head to work on both an XY-Table and and Robotic Arm. We did a comparison of the advantages and disadvantages of the two and they were pretty even.  Thats when Hunter asked if the head would work on either and we realized that we did not have to choose between the two.  There was probably a way to design it to work on either. As a result of the meeting, our focus has shifted from make a Painting Robot, to concentrating on making a Painting Robot Head (or Tool) that can be used by any robot whether it be an XY Table, Robot Arm, or even something anthropomorphic like an Asimo.  We will design a robot around this Paint Head, but its cool to think about how to design the paint head so that any robot can use it.

You can see the details we arrived at on the white board but to summarize main features:

1: will have 6 airbrushes, a paintbrush, and camera.
2: will require 6 valves and a servo for the brush.
3: will need air supply, power, and comm line.

We have not yet determined whether or not controller will be part of robot paint head, or stored in separate module along with air compressor.

Some concerns we have are: 

1: Will there be enough paint in Airbrush reservoirs?
2: Where are we going to put camera?
3: How will we raise and lower traditional brush?  

But most importantly, we are trying to figure out what Corinne meant by her contribution to the meeting in the photo below. We sense that its something big, but what could it mean?

Initial 3D Sketches of Paint Head Complete

Though its probably obvious, the name of this new robot is a blend of Neural Nets and InkJets.  The main idea being that it will use artificial intelligence to spray paint onto a canvas with jets of air. This initial sketch shows how this will be accomplished with an array of six airbrushes. Final design  will include traditional brushes as well, but we will build that around this airbrush array as our experience is that paint delivery is the most difficult part of any painting robot.  That is our primary goal, the delivery of paint to the brush and canvas.

Six airbrushes are being used so that a full color gamut can be achieve. The pigments in five of the air brushes will be cyan, magenta, yellow, black, and white. The final airbrush will either have a "guest" pigment, or a paint thinner, perhaps water?  Not sure yet.  

The two most interesting aspect of this are why white and a thinning agent are being included.

The reason for the white is that the Neural Jet will be using creative feedback loops (see my TEDx Talk for details). This robot will need the ability to both add and subtract saturation.  Unlike an inkjet printer that knows exactly what it will be printing and uses the white of the page, this robot will be re-evaluating each brush stroke and mark continuously, and repainting areas constantly.  As such it will need the ability to lighten areas, and even erase by white washing.  

The reason for the thinning agent is many fold, most importantly for brush washing and texture.  We are still working out the final details, but this robot will have a brush and brushes need cleaning, so it would be nice to be able to apply pigments and clean them from the brush with this paint head.  Another reason for a sixth airbrush is that it might be interesting to have a mystery fluid that simply gets mixed in once in a while to add texture.  I am not sure what exactly will be needed, but I just feel like later in the process I will be wishing I had an extra airbrush, so I am including it in the initial design.  Hope I don't end up needing two extra airbrush heads.