Recently, I had the amazing opportunity to sit with Steve May, the Chief Technology Officer (CTO) of Pixar Animation Studios.
Steve was at this year’s SIGGRAPH conference in Vancouver, which celebrates all kinds of innovations in the realms of computer graphics and technologies, including Animation. Pixar Animation Studios regularly has a presence at the conference, given their history and innovative efforts in the computer arts. Steve was there to share how Pixar’s revolutionary piece of software, Universal Scene Description, or “USD”, has changed the industry and what part it will play in the future of technology itself, including the metaverse.
He has had his hand in every film at the studio in some way, save for the original Toy Story and A Bug’s Life, and has been a staple at the iconic animation house since he joined.
Tony:
You started at Pixar back with Toy Story 2. Then, you helped give us the incredible fur technology we saw on James P. Sullivan in Monsters, Inc. How did you go from that to now, Chief Technology Officer?
Steve:
(Laughing) I don't know either. Interestingly, I actually started on Monsters, Inc. first, and it was super small, there were 10 people on the show and I didn't have any production experience. So, Tom Porter was my boss and he said, "Go ahead, you're doing some fur stuff." That's great. In fact, I'm not even sure he knew what to do with me at first. But he's like, "We have this problem with fur." And he kind of just left it there for me. But then, he said, "We should get you some hands-on production experience. You're in pre-pro. It's too mushy here." So, I went over to Toy Story 2 and shaded some stuff. And just for a little bit, and then, came back to Monsters, Inc. Which was really cool, I mean, it's the best possible first job. Which was a super open ended, really hard problem: Make fur for Sullivan.
Tony:
I think most everybody loves the story behind Monsters, Inc., but I think technologically speaking, everybody still to this day talks about that fur.
Steve:
Yeah, and we had no software for dealing with the fur. We didn't have software for modeling it, for grooming the directionality, the length. RenderMan could only render thousands of hairs and Sullivan had millions. We didn't have shadowing. We used deep shadows for Sullivan's fur. We didn't have deep shadows when we started working on it, so we didn't have shadows. The fur looked horrible without shadows. All that and with simulation – we didn't have a very good simulator – We had to write a whole new simulator from scratch. So, it was just a ton of work. It was super fun. It had Steve Jobs' attention. He was like, "That is the thing on this movie that is going to make the difference, make or break." And I was looking at some old images earlier and some of them, when we were working on it, they looked horrible. They looked really bad.
And I think I was just kind of new and like, "Oh, I'll just figure it out." But I got to think if I was the boss then, I would've been like, "Ah, I don't know how this is going to go." I'd be nervous…I would be very, very, very nervous. But it kind of came together and it was super fun working on that film. So, we did the technology parts for Sullivan, and actually building the character. And then we got to that point at the end of the show where you think you're done and then, someone said, "Hey, there's a new idea. Mike and Sully, they're going to go to the Himalayas." And that can't be true. My friend and I, Michael Fong, Michael was the other one that we really worked closely together on his fur. We went like, "Let's go find this out." We went to an art gallery, and we're like, "There it is." There's a pastel of Sullivan in the snow like a blizzard.
Tony:
It's the one scene I always think of though, to be honest.
Steve:
But the fortuitous thing was early on, Pete, the director, he was like, "I kind of want dirty fur like a llama. It's got grass in it and grunge and dirt and stuff." And early on, we were just trying to make any kind of fur look decent, and he ended up kind of liking the clean fur for Sullivan, so we went with that. But I had already thought about, oh, how would we do little bits of fur? And even had built some software to do it just in case. So, when the snow thing came up, it was like, "Oh, we can do this. I'll just modify that a bit." Put individual snowflakes on the hairs. And then, I worked on those shots where he kind of crashes into the camera. I did the snow effects, and the collisions, and painted in the cracks of the snow. It was super cool and fun to work on that. Really very, very rewarding project.
Then, I worked on (Finding Nemo) on what we called the sharks in Sydney scenes. I worked on Cars as the effects supervisor. So, lots of watching NASCAR kick up dirt and a lot of footage I watched there, and that was a great movie to work on. And then, I worked on Up, as what we would now call the visual effects supervisor and also, on Brave. Up was super fun. We went on a research trip to Venezuela and camped in the mountains of the Tepuis, and that was a really wild research trip. But yeah, I don't know how I got to CTO.
Tony:
I'm hearing you're good friends with Pete (Docter), apparently, after all this time.
Steve:
Yeah, Pete and I have been really good friends from the beginning. I did a number of different things at Pixar and one of the things actually, which is related to USD. So, this will all tie in here – We built a new animation system called Presto and we were having trouble landing it in production. It was a multi-year project, and it was really, like a lot of big software projects, when it really comes down to like, "Okay, now here it is animators. You've got to make a movie with this." You find out that it is not what it needs to be.
And there's a natural tension between the users and the creators, especially when you're making it in the same crucible together. And so, I went onto that show to kind of help the communication and the adoption of the software onto the show. And then, towards the end, I actually kind of came in as a visual effects supervisor to see it through to the end. But I think, I don't know. I always consider myself, half of me wants to be an artist and half wants to be a technologist.I think I'm fairly good at speaking the language of both –
Tony:
That's Pixar's whole motto almost, the art pushes the technology, the technology inspires the art.
Steve:
In their D-N-A, right? It's trying to do those things. I know other CTOs who know stuff so much better than I do about server technology, network systems, and I'm always blown away. And so, that's not my thing. But I do understand how to make movies and I appreciate both sides, the art and the technology sides, and I think that's probably the long-winded answer for how I got the job.
"The Adventures of Andre and Wally B" premiered at SIGGRAPH in 1984
Tony:
At SIGGRAPH this year, almost every panel I went to, Pixar or otherwise, at some point mentioned Universal Scene Description or USD. In layman's terms, can you describe USD?
Steve:
Yeah. So, I mentioned Brave, with the new information system. What was interesting is that we had a new scene description language for Brave, for this new animation system. And it was called Menva, Which is a weird name, but that's another story. And it was the way that we described geometry, light shaders, all of the objects.
It was a distillation, before that we had dozens of different file formats. You had one file format for shade ins, another file format for geometry, another file format for rigging. And then, a file format for the models. The hierarchies of the scene. And part of making Presto was distilling all those concepts into its purest form, and making a unified way to describe those scenes. And that was this Menva file. What we found on Brave, was that it was very expressive and it allowed us to author these complex scenes, but it was not high performance. So, we had to very quickly on Brave, kind of grab another existing format that we have that was actually very, very performant at describing scenes. When we came out of Brave, we used the combination of the two to get the movie done. We said, "This is not good. We've got two different ways to describe scenes. One's really fast. One's really expressive and powerful. We need to bring them together."
That's where USD came from. USD is our internal technology. It's the core technology of the studio that is how we describe scene data. And it's not just a file format, it's a system, a runtime system also, that combines a whole bunch of assets and aggregates them into a composed view of, what we call a “composed view” of the scene.
Steve:
So, you can think of it if you were going to build a scene and we would build it within thousands of individual USD files, USD is not only those individual files, but it is a system that combines them altogether. And we have a bunch of very, very powerful ways to combine them together. And that's really what the power is of USD, is the way that they can combine assets together. And we call that kind of composability or composition. And there's a bunch of different ways that you can compose the scene to combine the data. But the key thing is that it lets you kind of build a scene as a set of layers in the same way that you might in an image editing software like Photoshop.
You paint different layers and then Photoshop does this thing where it squashes the layers together and you get a composite view, right? But you can change things in any one layer, it doesn't break other layers. So, imagine you're making an animated film at Pixar, and those layers instead are animation, and there's set dressing, and there are cameras and lights, and there are different artists. Not you, a team of different artists is working on different layers. USD can take all those different layers and then say, "This is what the shot looks like at this frame, of this scene, at this point in time." And that's the power of USD.
Tony:
One of the things that USD touts, is that it can be used for making highly detailed and photorealistic scenes. I always see the side by side of Scud from Toy Story and the cat from Toy Story 4. But we've also seen, in my opinion, very photorealistic stuff with WALL-E, or The Blue Umbrella. How'd you do it before and then, how did USD accelerate it?
Steve:
What U-S-D lets you do is just manage more complexity., more efficiently. So, Scud to the cat in Toy Story 4, that's probably a whole lot of other things.
Tony:
I would never say Scud was photorealistic. Love it, but I wouldn't say it was photorealistic.
Steve:
Not the prettiest dog. But that was the advancement of shading, fur, rigging, our own artistic abilities, our own design abilities. Many, many, many things that got us there. So, USD does not make things more realistic or better looking, and not in photo realistic ways either. What it lets you do is manage massive amounts of complexity with a large number of users. And those are the things, I don't know if you've heard people have started referring to HTML as the Metaverse. As it's something to be used beyond film, and it actually is being used for applications beyond that where you're trying to build virtual worlds.
Tony:
Yeah, we're getting into another question I had. Because as I said, I'm more familiar with the realms of character animation and animation in general…Metaverse to me sounds like a video game level, but you talked about it at SIGGRAPH – You think USD will have a big part in the Metaverse.
Steve:
Yeah. Yeah. And Metaverse is a super-loaded word. And if you ask 10 people, you get 10 different answers about what it is. But the way I think about it is the way that everyday people, not just people in computer graphics, or film, or visual effects. Everyday people, the way they use, they consume video and images and texts, we're going to be adding 3-D onto that. 3-D objects will become a much more day-to-day kind of digital asset that you use in the same way people use video and text and images today. You see there's (already) some very simple forms of that, where you can hold your iPhone and you can look at a piece of furniture in your living room with AR, before you bought it, you can see how it fits and looks in the room, right? So, you're seeing the beginnings of that happening now. That will expand into things that are much more complicated than just a piece of furniture.
Tony:
Yeah. I just did that example recently, actually.
Steve:
You're going to need ways to describe that data. So, it could be that piece of furniture is a USD file. All the way to, I don't know, Ready Player One, where older, maybe decades or hundreds of years away, I don't know how far away, where you are actually immersed in more of a 3-D kind of VR sort of environment. And probably, a little bit of everything in between. I think Disney is a very interesting company in this space beyond Pixar, so I'm talking about the bigger parent company here.
Tony:
The bigger umbrella.
Steve:
Right. Because Disney has IP, has content, has these stories and characters like Marvel, Lucasfilm, Pixar, Disney Animation, Fox, Nat Geo, on and on and on, right?
Tony:
Yes, yes, yes.
Steve:
And it has physical experiences. It already has immersive experiences, but they're actually real places. At the parks, at the resorts, the cruise ships, et cetera, right? So, if you can combine those things, and by the way, USD could be the third key ingredient of this, you could do some really interesting stuff where you could bridge digital and virtual. You could have some digital stuff happening at those physical locations. You could have stuff in your house that happens. You can imagine – I don't want to talk about it because Disney's not talking about what they're going to do –
Tony:
Yeah. Of course.
Steve:
There's so much potential in there for some interesting and new ways to connect with audiences and tell different kinds of stories. That's where I think there's some.. there's just a whole spectrum of things potentially that kind of the quote unquote, Metaverse, falls under, but you need a way to describe 3-D objects and potentially entire worlds and you need to be portable.
So, if you bought a lightsaber, a virtual lightsaber at… Here I am going speculating. This is not what Disney's doing. This is what I want to do. You buy a virtual lightsaber, you go, you build a lightsaber at Disneyland. You can get a virtual version of the same thing and then, when you get home, you can go play with it in Fortnite. You want that digital asset to be portable, and move around, and be able to use it on different platforms. And that's where I think USD really can kind of serve that purpose of describing those things. Very complex things, potentially, much more complex than the lightsaber, and in a portable way so that you really aren't just confined to any one particular person's vision of what the Metaverse is.
Tony:
I feel like one of the ways you're having USD make a strong presence is by making it open-source. USD I think could have easily been in the cannon with Presto, or Menve/Marionette was before that even, as in-house stuff.
Steve:
You're right. When we have proprietary technology – And RenderMan is proprietary, right?
Tony:
Yes, RenderMan is.
Steve:
It's all RenderMan commercially. But this idea of portability, I'm going to have an asset, or maybe I even have, I've got a model of my house and I don't know, Pixar built my office, but Disney built my living room. And I want to go from that space to that space. You need to have that portability and to have that portability, you need to have an open standard. If it was a thing that only Pixar made and controlled, we could share it, but the adoption would be much, much harder to do. If you look at other things that we use day to day in a wide variety of systems, it's because they're open standards.
That's why we open-source USD so that it would be easier for people to say, "Oh, I can just grab that code, link it in to my authoring tools and now, there's work to be done. But I don't have to figure out how to implement USD. I just have to figure it out with my application.
Tony:
Like I said, I heard it at pretty much every panel at SIGGRAPH. So, besides the Metaverse though, I feel like other studios are latching onto it. Game developers are latching onto it. Pretty much everyone in the industry is grasping USD.
Steve:
Well, it does solve this basic problem of many, many file formats that we've had in this space, in our industry, and people who work with 3-D objects because every tool had a different file format. That meant that it was really hard to use a different set of tools. And we work in a world where there are a wide variety of great tools available. This is a problem that everyone, especially in film, faced. And then, you had to write software to import and export or convert the data from one format to another. You'd always lose stuff. You'd do something in Maya and you bring it into Quintana and you would lose some information because the formats were different and they were not compatible.
It solves a problem I think that everyone was kind of having, and it has some very clever features of it that we evolved over decades of battle testing it with actual making movies. So, it's good, it's been tested. Again, those ideas are decades old and we learn from trial and error what works, and what doesn't work, and what's really important and that's what's in USD.
Tony:
The battle test that you mentioned is a great segue I have to another thing that was shown off at SIGGRAPH this year that I was truly impressed by. Absolutely astounded by. That's PCF, Pixar Crowds Foundations.
Steve:
Oh, yeah.
Tony:
I had not seen anything like that. Could you talk a little bit more about those developments?
Steve:
Yeah. I mean, it leverages USD. it's a schema specifically, kind of for dealing with Crowds characters. It's very lightweight, high performance.
Tony:
The way he (the animator in the demonstration) was just clicking around, we're going to add these 20 characters here and have them all follow this object and it just blew my brain.
Steve:
Yeah. I mean, those workflows, those could be something that are not USD, but what USD enables is the idea that I can do that with hundreds or thousands of characters in real time. That makes the appeal then of some of those direct ability features actually work. You can say, "Hey, I want all the characters to kind of react, or move in this direction, or change their eye direction to respond to some event." But if it's slow or you can only see two or three characters do that at a time, that's not very fun to use. And this lets you do many thousands of characters in real time, which opens up new ideas about, oh, I have a better idea for how I could animate this, or make it work, or make the artist's experience more fun.
Tony:
And then, with USD, everybody, the departments can figure it out, too. How's this crowd going to be lit? How's this crowd going to-
Steve:
Yeah. Then, that data just moves through the rest of the pipeline in a very efficient way, so you don't have to worry really, about how it was created.
Tony:
Yeah, that piece of software, I just thought was amazing. USD's amazing also, but that one's definitely, as somebody knowing Pixar's challenges in the past – especially with crowds going back to A Bug's Life – that evolution to me was just astounding.
Steve:
Yeah, it's Presto and USD doing cool stuff together.
Tony:
Back in July (2022) Pixar announced an initiative to kind of expand R & D at Pixar. Is that USD related, or without divulging too much, what's going on?
Steve:
It is USD specific. We had a relatively small team. Really, really talented, smart people, but relatively small team supporting USD. And all this interest in USD is fantastic, but it's a lot of work to even respond to questions. We have a forum where people can ask questions, and just following that and our folks are so diligent, and they care so much, they read every one, and they try to respond to as many as they can. So, where's the time to also do work? Other companies are doing really interesting things with USD, and they want to contribute them to the open-source project, which is one of the main reasons you do an open-source project is contributions. That takes work, too. There's a lot of engineering, and communication, and betting, and things that have to be done to take a contribution from another company, even if it's awesome. we really wanted to beef up that team so that we had enough people on the team so that we could both go faster because there's so much we want to do still. We could work with the community more closely, and faster, and really leverage all the excitement and interest about it., take in those external contributions and those new ideas of things that we haven't thought about. We're at this point right now with USD where we're starting to see things we never thought someone would do with USD. That's a super exciting thing for technology when it gets to that point where people are doing stuff with it that you did not think of.
Tony:
As you said, that's one of the great things about being open-source is that you'll get that kind of perspective, too. I do have to start wrapping up, but I have one more question: Of all the films in the Pixar catalog, which do you think is the ultimate showcase of the studio's technology and achievement?
Steve:
Oh, man. The ultimate showcase. It's a different way to ask, "What's your favorite Pixar film?" Which is also almost impossible for me to answer. I guess, it depends on the… Well, Toy Story is still kind of the ultimate.
Tony:
Yeah.
Steve:
Right? My one bit of sadness in my career was that I was not at Pixar early enough to work on Toy Story, because I just felt like I don't know how they thought it was even possible. I mean, at that time, everyone else was struggling just trying to make a couple minutes of the animation, CG animation. They're making 90 minutes of CG animation. They did it, they pulled it off. I guess, I think our sophistication of our films keeps the visual sophistication increasing. You can look at a lot of the recent things. I mean, Soul, is super interesting.
We're doing more and more volumetric characters. We're getting away from hard surfaces or even furry surfaces. In the Soul world, the characters are volumetric, and the landscapes are volumetric, and it has a completely different look than anything that we've done.
Tony:
For the record, the best characters in Soul also, are basically straight lines.
Steve:
Oh, those are fantastic. That was a real shining moment for Presto too, where they wire these wire counselors that are basically wire sculptures like you made in art class. And how do you rig those? How do you animate those? We did all these amazing new animation controls up for those characters.
And we look at Coco a lot. Coco just had just amazing costumes, and the Land of the Dead, and the Magnolia bridge connecting. Visually, just the complexity (and) spectacle of that was amazing. But then, so was, Soul, the scenes in New York City. The crowds of people walking down the street and it felt like New York, but it wasn't real.
It was very stylized in terms of the characters. And he goes into the Half Note, the club…it's just beautiful stuff. And then, this year, I mean, both Lightyear and Turning Red, I think are visually stunning. Completely different.
Tony:
Completely different. Absolutely completely different.
Steve:
Completely different movies. And so, I think that alone is kind of an achievement that you could have two that are that different come from the same studio in the same year.
Tony:
From home, no less.
Steve:
Yes, from home. So, anyways, that was not a good answer, but.
Tony:
By the way, my answer would be (WALL-E)
Steve:
Oh, WALL-E was fun. First movie we started doing in 3-D stereo on, too.
Tony:
Wow. I didn't know that.
Steve:
We didn't do a stereo version of WALL-E. Actually, the first one was Up, but we used WALL-E to do some tests and it was pretty fun. Pretty fun because we also had Dennis Muren consulting on that, on WALL-E… it was amazing.
Tony:
Steve, thank you so much.
Steve:
You're welcome. It was nice meeting you.
For more about Pixar Animation Studios at this year’s SIGGRAPH conference in Vancouver, be sure to check out all of our coverage from the event here.