Concept

"Latent Space" is the working title for a collection of projects and proof of concepts that serve as subsystems for a future game. The idea is to explore the creative possibilities presented by the state of the art in deep learning, and encode my own imagination into a fully procedural experience complete with gameplay, music, and a whole universe to explore.

The core vision for the game is interdimensional/space exploration with a juxtaposition of both realism and surreal elements. Later iterations may include aspects of building and automation, but at the moment I am focusing on systems to generate the game world and music.

No Man's Sky is probably the biggest inspiration, as it pioneered the concept of giving players complete freedom to explore a practically infinite procedural universe. I want to build on that by going beyond simple rule-based and combinatorial approaches to generation by bringing Deep Learning to the table.

‍

Voxelized Fractals

Using a small library of transformations to string together and iterate over, fractals provide a space of cohesive but abstract shapes to pull from. I had originally hoped to use them as an infinite data source to train a network of some kind, but the 3D application of GANs is extremely limited because convolutional layers do not scale well with additional dimensions. Instead, I was pleasantly surprised by the current speed of distance-estimate-based fractal rendering. 

Using unreal engine, an excellent voxel plugin, and small C++ fractal library I pieced together, it was a simple matter of using the distance estimate to determine whether a region of space should be filled. Because each voxel is a cubic meter, much detail is lost even with the latest voxel smoothing algorithms. To add this detail back in, I created a custom material shader that uses raymarching from the surface of the voxel mesh to render the fractal with detail only limited by the number of pixels on the screen. This effectively creates a hybrid rendering system that allows traditional rasterized meshes to exist alongside, interact with, and theoretically even modify complex raymarched shapes.

The primary challenge now is to be able to curate the way formulae and parameters are combined since fractals are highly chaotic. It is possible I may be able to build a GAN-like system to be able to select "good" parameters based on a visual or statistical representation of the generated fractal. This will take further reading into semi-supervised approaches to GAN training since it will involve some amount of manual inspection.

‍

TerraGAN

Code on GitHub

On the flip side, realistic terrain can be reduced to a 2D image generation problem if we represent it as a height map. Additionally, if we somehow tile images together, we can populate the faces of a cube and then deform it into a sphere to create a planet.

My approach to this is inspired mainly by two papers. One, titled TileGAN, describes a method to generate arbitrarily large non-homogeneous images. Their method leverages the spatial invariance of a progressively-grown convolutional GAN by splitting the generator G at one of the layers creating two separate pieces, G_A and G_B. Rather than feeding G_A's output directly into G_B, the intermediate feature maps are tiled together into what is called the "latent field." G_B can then be applied in overlapping steps across the latent field which creates a single continuous texture. The TileGAN system also includes a pretty involved pre-processing phase which clusters latents with similar looking outputs and makes them available to lookup in order to resemble a guidance image. Instead of this, I Implemented a version of the system described in the paper, InterFaceGAN, where latents can be manipulated with respect to chosen semantics using a hyperplane in the latent space found via support vector machine.