'Star Citizen' Says Its Advanced Face Tracking Tech Isn't Feature Creep

When Star Citizen first added voice chat to its game, it unsettled players and developers.

“Somebody would walk up to a character and start talking, but what’s happening to his face? Nothing,” Sean Tracy, technical director of Star Citizen told me over Skype. “It was super immersion breaking. Chris Roberts [Star Citizen director and founder of developer Cloud Imperium Games] reached out and said, ‘By the end of the year, I want to see that face actually talking to me.’”

A year after his request Star Citizen has a version of Roberts’ dream running in an alpha build of its yet to be released game. It’s a system that allows players to use a webcam or smartphone to capture their facial expressions and transfer them to their digital avatar. This happens in real time. As a player smiles, the avatar smiles. If the player opens their eyes in a wide crazy stare, so does the avatar. It’s both spooky and impressive.

Roberts’ ambitions have led to a product that, at the moment, is more promise than game. After seven years in development and almost $200 million in crowdfunding raised by fans, Star Citizen only exists as a loose collection of modules, alphas, and trailers. A lot of that stuff, like this latest face tracking tech, is impressive. But it also speaks Roberts’ tendency towards what some in the community consider feature creep.

Tracey doesn’t see it that way. “[Face over IP] itself wasn’t a massive undertaking from a huge team…it was created by 4 or 5 developers who also worked on numerous other systems during its development,” he said. “The Gameplay feature team’s main development focus was really on the GROUP system. [The ability for players to join parties.] This always was scoped to include VoIP [voice over IP], and then we were able to get FoIP [face over IP] as a bonus feature as it is intrinsically linked to VoIP.”

Tracey took inspiration from what Apple is doing with animoji—a program that lets users’ facial expressions steer a cartoon character they can share with their friends. It’s much easier to get away with driving a non-realistic face because it’s stylized,” Tracey said. “But maybe if we get this just right…we can probably pull this off.”

Star Citizen’s face over IP works by scanning the player’s face with a webcam or smartphone, mapping out 60 control points on their face, then running those points through a layer of code that translates those 60 controls points to the 183 control points on the face of the avatars. “We basically spent five months working on that mapping specifically.”

It’s fun tech and it works, but there are a few caveats. First, users have to have a webcam that captures video at 60 FPS and most don’t do that. Luckily, many smartphones have a camera that does and Star Citizen allows users to run an app that will do the mapping. It’s also working on its own camera specifically for use in FoIP. Faceware, the company whose technology helped make FoIP possible, is making the camera but it hasn’t set a price or a release date. Despite that, the company is taking pre-orders. Again, keep in mind that these are pre-orders for a camera for a game that doesn’t exist yet.

“I personally feel that FoIP is an incredible example of practices that used to happen in broader game development, where someone takes some of their own time to work on an aspect of the game they find interesting and it then becomes a feature in the game itself,” Tracey said. “This practice is something I feel has been lost in larger companies and corporations in recent time and is one of my favorite parts of being able to work at CIG.”

The other issue is that this only works with the nine specific avatar faces Star Citizen has released. Tracey told me that allowing customers to customize those avatars and generate their own, is a priority for the team. They want it all to work with the new FoIP technology.

Online gaming interaction is often toxic, and Tracey and his team said they’ve been surprised at how cordial the FoIP rollout has been.“I expected this to be awful, players would get hold of this and be… rude. We haven’t seen much of that at all, which is super interesting,” he said. “It means we’re on to something, but I’m not sure how we measure it.”

There’s a lot of good research that suggests facial expressions are a big part of human communication, and that the lack of them helps foster the online hellworld we all live in daily. But Star Citizen’s playerbase is a dedicated community that’s spent almost $200 million collectively to crowdfund the game and make tech such as FoIP possible.

It’s possible that dedication and belief in the dream of Star Citizen, rather than its reality, is what keeps them polite and pleasant. They’re all part of the same club, waiting on the promise of exploring the stars while they make faces at each other in corridors. Maybe it will be a nice place to hang out, assuming it ever releases.