If You Can’t Beat Them, Join Them: How Text-to-Image Tools Can Be Leveraged in the 3D Modelling Process

Samuel Otto Mathiesen; Alessandro Canossa

doi:10.1007/978-3-031-48057-7_11

If You Can’t Beat Them, Join Them: How Text-to-Image Tools Can Be Leveraged in the 3D Modelling Process

Samuel Otto Mathiesen, Alessandro Canossa

Institute of Visual Design

Publications: Chapter in Book/Report/Conference proceeding › Book chapter › Research › peer-review

Abstract

The rapidly growing field of text-to-image generation leverages machine learning and natural language processing to allow for the creation of very diverse and high-quality images. Whilst these images can be used in various forms and for many purposes, this paper addresses a specific question: The possibility of utilizing text-to-image generators to create concept art for the 3D-modeling process of a character.
There is a potential opportunity to explore how these tools can be incorporated in traditional content creators’ pipelines. This opportunity presents itself now because of two factors: The increased capability of these text-to-image tools compared to previous generative models, as well as the lack of specific focus on creators in current research.
We propose a possible pipeline to incorporate these new tools in the traditional 3D modeling process, dubbed the ‘vortex process’.

Original language	English
Title of host publication	HCI International 2023 – Late Breaking Papers : 25th International Conference on Human-Computer Interaction, HCII 2023, Copenhagen, Denmark, July 23–28, 2023, Proceedings, Part VI
Editors	Helmut Degen, Stavroula Ntoa, Abbas Moallem
Number of pages	19
Publisher	Springer
Publication date	23 Jul 2023
Pages	162-181
DOIs	https://doi.org/10.1007/978-3-031-48057-7_11
Publication status	Published - 23 Jul 2023
Event	HCI International 2023: 25th International Conference on Human-Computer Interaction - AC Bella Sky Hotel and Bella Center, Copenhagen, Denmark Duration: 23 Jul 2023 → 28 Jul 2023 Conference number: 25 https://2023.hci.international/

Conference

Conference	HCI International 2023
Number	25
Location	AC Bella Sky Hotel and Bella Center
Country/Territory	Denmark
City	Copenhagen
Period	23/07/2023 → 28/07/2023
Internet address	https://2023.hci.international/

Series	Lecture Notes in Computer Science
ISSN	0302-9743

Artistic research

Yes

Access to Document

10.1007/978-3-031-48057-7_11

Cite this

Mathiesen, S. O., & Canossa, A. (2023). If You Can’t Beat Them, Join Them: How Text-to-Image Tools Can Be Leveraged in the 3D Modelling Process. In H. Degen, S. Ntoa, & A. Moallem (Eds.), HCI International 2023 – Late Breaking Papers: 25th International Conference on Human-Computer Interaction, HCII 2023, Copenhagen, Denmark, July 23–28, 2023, Proceedings, Part VI (pp. 162-181). Springer. https://doi.org/10.1007/978-3-031-48057-7_11

Mathiesen, Samuel Otto ; Canossa, Alessandro. / If You Can’t Beat Them, Join Them : How Text-to-Image Tools Can Be Leveraged in the 3D Modelling Process. HCI International 2023 – Late Breaking Papers: 25th International Conference on Human-Computer Interaction, HCII 2023, Copenhagen, Denmark, July 23–28, 2023, Proceedings, Part VI. editor / Helmut Degen ; Stavroula Ntoa ; Abbas Moallem. Springer, 2023. pp. 162-181 (Lecture Notes in Computer Science).

@inbook{d65add103ff94c859e0cd17d44eb600e,

title = "If You Can{\textquoteright}t Beat Them, Join Them: How Text-to-Image Tools Can Be Leveraged in the 3D Modelling Process",

abstract = "The rapidly growing field of text-to-image generation leverages machine learning and natural language processing to allow for the creation of very diverse and high-quality images. Whilst these images can be used in various forms and for many purposes, this paper addresses a specific question: The possibility of utilizing text-to-image generators to create concept art for the 3D-modeling process of a character.There is a potential opportunity to explore how these tools can be incorporated in traditional content creators{\textquoteright} pipelines. This opportunity presents itself now because of two factors: The increased capability of these text-to-image tools compared to previous generative models, as well as the lack of specific focus on creators in current research.We propose a possible pipeline to incorporate these new tools in the traditional 3D modeling process, dubbed the {\textquoteleft}vortex process{\textquoteright}. ",

author = "Mathiesen, {Samuel Otto} and Alessandro Canossa",

year = "2023",

month = jul,

day = "23",

doi = "10.1007/978-3-031-48057-7_11",

language = "English",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "162--181",

editor = "Helmut Degen and Stavroula Ntoa and Abbas Moallem",

booktitle = "HCI International 2023 – Late Breaking Papers",

note = "HCI International 2023 : 25th International Conference on Human-Computer Interaction, HCII2023 ; Conference date: 23-07-2023 Through 28-07-2023",

url = "https://2023.hci.international/",

}

Mathiesen, SO & Canossa, A 2023, If You Can’t Beat Them, Join Them: How Text-to-Image Tools Can Be Leveraged in the 3D Modelling Process. in H Degen, S Ntoa & A Moallem (eds), HCI International 2023 – Late Breaking Papers: 25th International Conference on Human-Computer Interaction, HCII 2023, Copenhagen, Denmark, July 23–28, 2023, Proceedings, Part VI. Springer, Lecture Notes in Computer Science, pp. 162-181, HCI International 2023, Copenhagen, Denmark, 23/07/2023. https://doi.org/10.1007/978-3-031-48057-7_11

If You Can’t Beat Them, Join Them: How Text-to-Image Tools Can Be Leveraged in the 3D Modelling Process. / Mathiesen, Samuel Otto; Canossa, Alessandro.
HCI International 2023 – Late Breaking Papers: 25th International Conference on Human-Computer Interaction, HCII 2023, Copenhagen, Denmark, July 23–28, 2023, Proceedings, Part VI. ed. / Helmut Degen; Stavroula Ntoa; Abbas Moallem. Springer, 2023. p. 162-181 (Lecture Notes in Computer Science).

Publications: Chapter in Book/Report/Conference proceeding › Book chapter › Research › peer-review

TY - CHAP

T1 - If You Can’t Beat Them, Join Them

T2 - HCI International 2023

AU - Mathiesen, Samuel Otto

AU - Canossa, Alessandro

N1 - Conference code: 25

PY - 2023/7/23

Y1 - 2023/7/23

N2 - The rapidly growing field of text-to-image generation leverages machine learning and natural language processing to allow for the creation of very diverse and high-quality images. Whilst these images can be used in various forms and for many purposes, this paper addresses a specific question: The possibility of utilizing text-to-image generators to create concept art for the 3D-modeling process of a character.There is a potential opportunity to explore how these tools can be incorporated in traditional content creators’ pipelines. This opportunity presents itself now because of two factors: The increased capability of these text-to-image tools compared to previous generative models, as well as the lack of specific focus on creators in current research.We propose a possible pipeline to incorporate these new tools in the traditional 3D modeling process, dubbed the ‘vortex process’.

AB - The rapidly growing field of text-to-image generation leverages machine learning and natural language processing to allow for the creation of very diverse and high-quality images. Whilst these images can be used in various forms and for many purposes, this paper addresses a specific question: The possibility of utilizing text-to-image generators to create concept art for the 3D-modeling process of a character.There is a potential opportunity to explore how these tools can be incorporated in traditional content creators’ pipelines. This opportunity presents itself now because of two factors: The increased capability of these text-to-image tools compared to previous generative models, as well as the lack of specific focus on creators in current research.We propose a possible pipeline to incorporate these new tools in the traditional 3D modeling process, dubbed the ‘vortex process’.

U2 - 10.1007/978-3-031-48057-7_11

DO - 10.1007/978-3-031-48057-7_11

M3 - Book chapter

T3 - Lecture Notes in Computer Science

SP - 162

EP - 181

BT - HCI International 2023 – Late Breaking Papers

A2 - Degen, Helmut

A2 - Ntoa, Stavroula

A2 - Moallem, Abbas

PB - Springer

Y2 - 23 July 2023 through 28 July 2023

ER -

Mathiesen SO, Canossa A. If You Can’t Beat Them, Join Them: How Text-to-Image Tools Can Be Leveraged in the 3D Modelling Process. In Degen H, Ntoa S, Moallem A, editors, HCI International 2023 – Late Breaking Papers: 25th International Conference on Human-Computer Interaction, HCII 2023, Copenhagen, Denmark, July 23–28, 2023, Proceedings, Part VI. Springer. 2023. p. 162-181. (Lecture Notes in Computer Science). doi: 10.1007/978-3-031-48057-7_11