Skip to main navigation Skip to search Skip to main content

Chat3D: Interactive understanding 3D scene-level point clouds by chatting with foundation model for urban ecological construction

  • Yiping Chen
  • , Shuai Zhang*
  • , Ting Han
  • , Yumeng Du
  • , Wuming Zhang
  • , Jonathan Li
  • *Corresponding author for this work
  • Sun Yat-Sen University
  • University of Waterloo

Research output: Contribution to journal › Article › peer-review

Abstract

With the artificial intelligence technology development boom, large language models are demonstrating their potential in comprehension and creativity. Large language models such as GPT-4 and Gemini have been able to powerfully study for various professional-level exams. However, as a language model itself, its powerful comprehension can only be reflected in text sequences. Currently, although videos can be generated through the connection between 3D point clouds and large language models, there is currently no prompt project that directly interacts with one-dimensional through attribute calculation results. The point cloud data is also rich in information that can support various tasks of urban construction. For scene-level point cloud data, there has been a lot of research done on semantic segmentation, target detection, and other tasks. However, it is usually difficult to provide direct help to scene construction from the perception results. This paper presents a method for applying large language models to urban ecological construction by combining the results of 3D point cloud semantic segmentation. The objective is to integrate the prior knowledge and creative capabilities of Large Language Models (LLMs) within urban development with the outcomes derived from point cloud semantic segmentation results. This integration aims to establish an interactive point cloud intelligent analysis system, tailored for aiding decision-making processes in urban ecological civilization construction, thus presenting innovative perspectives for the advancement of smart city development.

Original languageEnglish
Pages (from-to)181-192
Number of pages12
JournalISPRS Journal of Photogrammetry and Remote Sensing
Volume212
DOIs
StatePublished - Jun 2024
Externally publishedYes

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

  1. SDG 11 - Sustainable Cities and Communities
    SDG 11 Sustainable Cities and Communities

Keywords

  • Large language model interact
  • Point cloud understanding
  • Prompt engineering
  • Thought chain
  • Urban ecological construction

Fingerprint

Dive into the research topics of 'Chat3D: Interactive understanding 3D scene-level point clouds by chatting with foundation model for urban ecological construction'. Together they form a unique fingerprint.

Cite this