Overview

Generating artistic and coherent 3D scene layouts is crucial in digital content creation. Traditional optimization-based methods are often constrained by cumbersome manual rules, while deep generative models face challenges in producing content with richness and diversity. Furthermore, approaches that utilize large language models frequently lack robustness and fail to accurately capture complex spatial relationships.

Our Approach

In this paper, we present a novel vision-guided 3D layout generation system:

1. High-quality Asset Library: We first construct a high-quality asset library containing 2,037 scene assets and 147 3D scene layouts.
2. Image Generation Fine-tuning: We employ an image generation model to expand prompt representations into images, fine-tuning it to align with our asset library.
3. Robust Image Parsing: We develop a robust image parsing module to recover the 3D layout of scenes based on visual semantics and geometric information.
4. Scene Graph Optimization: We optimize the scene layout using scene graphs and overall visual semantics to ensure logical coherence and alignment with the images.

Results

Extensive user testing demonstrates that our algorithm significantly outperforms existing methods in terms of layout richness and quality.

• Paper: arXiv:2510.15564
• Code: GitHub

Motivation

Spatial-temporal intelligence of Vision-Language Models (VLMs) has attracted much attention due to its importance for Autonomous Driving, Embodied AI, and General Artificial Intelligence. Existing benchmarks mainly focus on egocentric perspective reasoning or geographic perspective reasoning with graphics context, failing to assess VLMs’ geographic spatial-temporal intelligence with both images/video and graphics context.

Key Challenges

GTR-Bench is more challenging as it requires:

• Multiple perspective switches between maps and videos
• Joint reasoning across multiple videos with non-overlapping fields of view
• Inference over spatial-temporal regions unobserved by any video context

Key Findings

Evaluations of more than 10 popular VLMs reveal three primary deficiencies:

1. VLMs’ reasoning is impaired by an imbalanced utilization of spatial-temporal context
2. VLMs are weak in temporal forecasting, performing worse on temporal-emphasized tasks
3. VLMs lack proficiency in comprehending or aligning map data with multi-view video inputs

Even the best proprietary model, Gemini-2.5-Pro (34.9%), significantly lags behind human performance (78.61%).

• Paper: arXiv:2510.07791
• Code & Data: GitHub

Problem

Existing methods for 3D Visual Grounding are often constrained — they either only focus on geometric and visual cues, or, like traditional 3D scene graphs, lack the multi-dimensional attributes needed for complex reasoning.

Our Framework

The DSM framework enriches robust geometric models with a spectrum of VLM-derived semantics, including:

• Appearance attributes: color, patterns, texture
• Physical attributes: weight, material, surface properties
• Affordance attributes: functional aspects and operational methods

We construct the DSM online by fusing multi-view observations within a temporal sliding window, creating a persistent and comprehensive world model.

DSM-Grounding

Building on this foundation, we propose DSM-Grounding, a new paradigm that shifts grounding from free-form VLM queries to a structured reasoning process over the semantic-rich map, markedly improving accuracy and interpretability.

Results

• ScanRefer benchmark: 59.06% overall accuracy (IoU@0.5), surpassing prior methods by 10%
• Semantic segmentation: 67.93% F-mIoU, outperforming all baselines including privileged ones

• Successfully deployed on physical robots for navigation and grasping tasks

• Paper: arXiv:2504.08307
• Project Page: binicey.github.io/DSM

If you prefer publishing blog posts on medium.com or other external sources, starting version v0.5.0, al-folio lets you to display your external posts in the blog feed of your website! 🎉🎉

Configuring external sources of super simple. After upgrading to v0.5.0, just add the following section to your _config.yml:

external_sources:
  - name: medium.com  # name of the source (arbitrary string)
    rss_url: https://medium.com/@<your-medium-username>/feed

The example above adds your medium.com blog post feed as an external source. But you can add arbitrary RSS feeds as sources.

Any questions or suggestions? 👉 Start a discussion on GitHub!