What if the world looked like a Shinkai film? Or a Studio Ghibli film? With how picturesque many of the landscapes are while often looking far more idyllic than the real thing, it would be a rather welcome sight if the world looked even half as pretty as the environments that populate these films. While it’s not possible for AI to entirely replicate the wonderful landscapes found in these movies, what if they could take a photo you captured and turn it into as close an approximation as possible to the landscapes of these movies and more? Enter AnimeGAN.
AnimeGAN: A Novel Lightweight GAN for Photo Animation is a rather clever tool produced by a team of Chinese researchers over at Wuhan University and the Hubei Institute of Technology. This tool takes a photo of anything and transforms it to look like it was a scene ripped right out of either a Shinkai film, a Hayao Miyazaki production or a Satoshi Kon feature (more specifically, the world of Paprika).
To fully explain every step of the complex programming behind this tool is something that I’m perhaps not qualified to give, but to simplify the inner workings of this AI, it analyzes the environments from productions from these directors and uses machine learning to approximate how the photo it has been given would look in these films.
It does this using a lightweight Generative Adversarial Network, or GAN. Similar tools to AnimeGAN already exist in the form of ComixGAN or CartoonGAN, but researches took the grayscale style loss, color reconstruction loss and grayscale adversarial loss common with these tools into account when working on this for better results while retaining vivid colors.
The results are more mixed when tackling environments less common to these films, as other examples show, so don’t worry about it replacing the human touch just yet. There are applications that could help speed up the process through human touch up or through transforming images into approximations they can use as a reference. It’s not that such a use of photography directly into the process beyond location scouting would be unusual: in manga, popular mangaka Asano Inio has often uses photography and photoshop to create the detail backgrounds found in his work. This could allow a similar process for anime if further developed.
In any case, AnimeGAN is a fascinating tool that allows those using it to imagine their world as though it were ripped directly from the scene of an anime. As interesting as it may be to discuss potential uses of this technology, just the chance to transform the world around you like this is pretty fun.