Unreal Engine Text-to-VR World Generator
← Project Index Unreal Engine Text-to-VR World Generator

Unreal Engine Text-to-VR World Generator

Type a description, get a walkable 3D/VR environment in Unreal Engine -; Python driving procedural generation, a Blueprint spawning system for interactive elements, and VR-specific setup so you can actually stand inside the result.

Archived Started: Summer 2025 Updated: Summer 2025

Overview

The pitch in one line: describe a place in plain English and have Unreal Engine build it for you to walk through in VR. Behind that is a pipeline that parses the description into a scene structure, lays out terrain and biomes and lighting, drops in interactive props via Blueprints, and wires up the VR comfort features that keep the experience from making you queasy.

Background

Building a 3D environment by hand is slow and skill-gated; building one you can stand inside in VR is slower still. The question here was how far a language model plus the Unreal Python API could get toward “describe it and it appears” -; not photorealism, but a real, explorable space generated from text.

This was where the text-to-world idea got proven out. It later folded into the WorldLabs Marble MCP work, which took the same “describe a place, generate it” premise and pointed it at WorldLabs' Marble world-generation API instead of hand-rolled Unreal procedural generation.

How It Works

A Python layer talks to Unreal through its Python API, with an LLM (Claude or a local Ollama model) doing the natural-language parsing into a JSON scene structure. From there a documented multi-stage pipeline handles spatial reasoning, terrain and biome generation, dynamic lighting, interior layout, and a Blueprint spawning system for the interactive bits.

The VR stage is its own concern: motion-controller setup, teleport locomotion, and comfort options like vignetting and snap turning, plus material optimization to keep frame rates VR-safe. The whole thing is driven through an MCP server so it can be operated conversationally.

Current Status

Archived as a Summer 2025 build. It got to a thoroughly documented pipeline with a user manual and working MCP integration -; enough to prove the concept -; then handed its ideas off to the Marble MCP descendant rather than continuing on its own.

  • Multi-stage text-to-world pipeline with LLM scene parsing.
  • Blueprint spawning for interactive elements; VR comfort and locomotion setup.
  • Driven via MCP; superseded by the WorldLabs Marble MCP.