RoboLayout

RoboLayout extends LayoutVLM with agent-aware layout optimization for indoor scenes. It introduces explicit differentiable reachability constraints to ensure generated layouts are navigable and actionable by embodied agents with diverse physical capabilities (e.g., robots, humans, animals). The framework also includes a local refinement stage that selectively re-optimizes problematic object placements without rerunning full-scene optimization, improving stability and convergence while preserving semantic and physical plausibility.

Architecture

RoboLayout comprises three main layers:

• Orchestration: The central orchestrator responsible for coordinating groupings, rendering and other cognitive processes of the solver and sandbox.
• Sandbox: Translates constraints into feasible scene layouts.
• Solver: Optimizer based on hard and soft constraints for spatial arrangements and refinement of the final optimized scene.

Key Features

• Differentiable reachability constraints for embodied agents
• Local refinement stage for fast optimization without full-scene re-runs
• Furniture grouping and spatial relation extraction via LLM
• Hard constraints: boundary, on-top, rotation unit norm
• Soft constraints: overlap, reachability, against_wall, distance_constraint, point_towards, align_with, symmetric_pair

Implementation Pipeline

1. Initial state — Takes language instruction and room shape plus asset list
2. Furniture grouping — LLM groups furniture based on prompt and task
3. Pose estimation — LLM produces constraint program from scene renderings
4. Conversion to Python — Executable constraint calls in sandbox
5. Optimization with reachability — Gradient-based solver with overlap, constraint, and reachability losses
6. Refinement — Cleanup step identifies problematic pairs and re-optimizes subset only

Getting Started

Visit the GitHub repository to get started with RoboLayout.