Facebook demonstrates Vid2Play, an AI that can create video game characters from real-life footage

[Angrry.exe™]

Typically, creating 3D game characters is a complicated process involving many hours spent building up 3D layouts, applying textures and adapting the dynamics of movements to get the most realistic effect. But with the new Facebook-proven technology, all these steps could be reduced to analyzing a series of videos in which a real person performs different activities, similar to those expected from the 3D character.

 

The analysis is done using two neural processing networks Pose2Pose and Pose2Frame. The first is to translate various types of activities into the virtual environment, such as dance and tennis or football. The Pose2Frame system then recreates the person in 3D, complete with hand-held objects and shadow on the ground. The 3D character thus obtained can be inserted into any virtual scene and controlled using the gamepad or keyboard. Alternatively, previous moves can be learned by the game's AI and recreated whenever needed.

The main advantage of Facebook's AI system is that it can replicate the appearance and movements of a character using just a few short clips to play those activities, ignoring others in the background and compensating for different shooting angles. This makes it possible to use the system outside the studios specially designed for this purpose, for example to generate virtual replicas of players in a football match, complete with the unique reactions they have during the game.

The system is not perfect yet. There is also work to anchor the characters in the virtual environment, so that the movements rendered also take into account the surface they are in. The dynamics of the movements is a little limited, but it succeeds in conveying the impression of realism.

When completed, Facebook's Vid2Game system could make gaming a much more personal activity, letting users add their own character to the game world.