Outline: Adapting a Diffusion Model for Sketch-Guided User Interface Prototyping

Matheus Mendes Silva; Edilson José Rodrigues

DOI: 10.22167/2675-6528-202603291

Article derived from a Course Conclusion Work (TCC), with content based on the student’s original work and adapted to the editorial format of the E&S Magazine with the support of the ResumeAI tool, an artificial intelligence solution developed by Instituto Pecege for textual synthesis and organization.

Summary

The evolution of artificial intelligence and deep neural networks has revolutionized human-machine interactions. Generative models, while promising for image synthesis, face challenges in prototyping user interfaces (UI) by failing to respect spatial structure. To address this divergence, a conditional latent diffusion approach was proposed, integrating a control architecture with a large-scale model fine-tuned for mobile interfaces, which supported UI sketches to guide image inference. The training dataset comprised approximately 12,400 UI images and sketches from public sources, pre-processed and enriched with synthetic data and textual instructions. The approach utilized a Stable Diffusion 3.5 Large model as a base, adapted with ControlNet canny edges and a LoRA adapter. The approach’s feasibility was validated by quantitative evaluation, employing a normalized semantic similarity (NSS) metric based on Sentence-BERT, and by qualitative analysis with 20 industry experts. The results confirmed that the method surpassed the original model’s capabilities. However, the evaluation highlighted that the effectiveness in logical interface structuring still faced aesthetic degradation and loss of functional elements. It was concluded that, although the technology proves its technical feasibility for optimizing the transition between requirements and code implementation, the automation of creative processes demands architectural improvements to achieve the visual rigor required by the software development industry.

Keywords: Low-rank adaptation; Machine Learning; Conditional latent diffusion; Deep neural networks.

1. Introduction

The software development process undergoes constant changes, evolving to meet the accelerated pace of the industry. A fundamental part of this cycle is user interface prototyping, which defines the customer journey and enables requirement validation before implementation (Al-Saqqa et al., 2020). This stage presents great openness for exploring methodologies based on artificial intelligence, potentially enhancing the creative process and increasing productivity. The accelerated evolution in the field of artificial intelligence and deep neural networks has revolutionized the way human-machine interactions are established in recent years, with generative models showing high capacity and fidelity for synthesizing texts, images, and videos from textual instructions.

Modern industry seeks to develop intelligent and interactive systems, both machine-to-machine and human-to-machine, driving the automation of tasks and processes performed by humans (Ismail and Hassan, 2019). Following this trend, global investments in artificial intelligence showed notable growth in 2024, with 78% of organizations reporting the use of artificial intelligence, according to the AI Index Report 2025 from Stanford University (HAI, 2025). In this context, several studies explore the use of generative artificial intelligence to automate stages of the software development cycle, including documentation creation, code production, test automation, and digital system architecture design (Beltramelli, 2018; He et al., 2025; Wan et al., 2025; Wang et al., 2024).

Despite the progress, the application of these technologies for user interface prototyping is not only underexplored but also faces critical challenges. Generative models frequently fail to respect the idealized spatial structure, creating a divergence between expectation and outcome. Although text-to-image models are powerful tools for image generation, the textual input plays a fundamental role in the results obtained, and this architecture can generate imprecise and unstable results by not allowing greater user control over the generated images (Liu and Chilton, 2022; Mou et al., 2024). Most studies focus on producing code from textual specifications or already elaborated user interface images. The present work, in contrast, focuses on the visual UI conception stage, a process that precedes subsequent software development (Lauff et al., 2018).

Considering the visual nature of interfaces, UI prototyping automation requires neural network architectures capable of understanding and synthesizing layouts while respecting complex visual hierarchies. Deep neural network models, such as latent diffusion models, stand out for their state-of-the-art performance and greater computational efficiency (Rombach et al., 2022). In this context, the research is based on conditional image generation methodologies, anchoring the inference process in a reference image (Couairon et al., 2023; Mou et al., 2024; Zhang et al., 2023; Zhao et al., 2023). This approach allows the trained network to support user interface sketches to guide image inference, unlike methods that rely exclusively on text.

To solve the divergence between the expectation and the generated result, this study proposes a conditional latent diffusion approach, integrating a control architecture with a large-scale model specifically fine-tuned for the mobile interface domain. The technical feasibility of the approach is proven for optimizing the transition between requirements and code implementation, although the automation of creative processes still demands architectural improvements to achieve the visual rigor required by the software development industry. Given the above, the present work aims to adapt latent diffusion models for generating high-fidelity prototypes through supplementary training, using open-source technologies and publicly available databases. The hypothesis is that the developed adapters will surpass the capabilities of the original base model, ensuring higher visual quality and structural fidelity to the reference sketch.

2. Material and Methods

The present study adopted a mixed methodological approach, characterized as applied and experimental research, aiming to adapt latent diffusion models for the prototyping of high-fidelity user interfaces (UI). The investigation focused on the mobile ecosystem to optimize the inference process and reduce computational cost, given the smaller screen dimensions and the simplification of hierarchical structures.

The training dataset was composed of approximately 12,400 UI images and sketches, sourced from public datasets such as MUD (Feng et al., 2024), VINS (Bunian et al., 2021), and SWIRE (Huang et al., 2019). The Rico dataset (Deka et al., 2017) was referenced but not directly used due to inconsistencies in its metadata. A preprocessing step filtered out unwanted samples (advertisements, “iframes”, non-Latin text, horizontal captures, notification bar) and generated synthetic data. Python scripts, using OpenCV (Bradski, 2000), processed component hierarchy metadata to draw sketches with black backgrounds and white strokes, to which minor distortions and random noise were added. The images were resized to 720 by 1280 pixels, and complementary textual instructions were generated via the “Gemini 2.0 Flash” API (Google, 2025) with a standardized “system prompt”.

The development and training environment utilized Python and an open-source software ecosystem, including libraries such as PyTorch (Paszke et al., 2019), Diffusers (von Platen et al., 2022), Transformers (Wolf et al., 2020), and Accelerate (Gugger et al., 2020). The computational infrastructure consisted of an NVIDIA A100 Graphics Processing Unit (GPU) with 80 GB of video memory, allocated via a cloud computing service. The developed scripts are publicly available.

The selected base diffusion model was “Stable Diffusion 3.5 Large” (Rombach et al., 2022), employing supplementary training with frozen parameters to optimize memory consumption. Its “multimodal diffusion transformer” architecture, with a T5 text encoder (Raffel et al., 2020), was used for semantic understanding. To condition image generation with additional spatial information, a “ControlNet canny edges” network (Zhang et al., 2023) was adapted through supplementary training. This module, coupled with the base model, delegated spatial structure to the control network. The ControlNet training utilized a dataset of approximately 12,400 images, with a resolution of 1024 by 1024 pixels, bf16 numerical precision, 8-bit Adam optimizer, a learning rate of 10^-5, an effective batch size of 32, and 18 epochs.

A low-rank adapter (LoRA) (Hu et al., 2022) was integrated into the pipeline to enhance aesthetic consistency and visual quality. For LoRA training, 156 images of modern UIs were manually collected from specialized design repositories. The dataset underwent preprocessing to standardize image size and reduce representational discrepancy across categories, applying controlled oversampling. Hyperparameters for LoRA training included the “Stable Diffusion 3.5 Large” architecture, a rank of 64, a resolution of 1024 by 1024 pixels, bf16 numerical precision, a learning rate of 10^-4 for the “Transformer” network and 10^-5 for the text input encoders, with 1000 training steps and 150 warmup steps.

The quantitative evaluation was performed by comparing the original textual instruction with a textual description of the generated image, using the Sentence-BERT technique (Reimers and Gurevych, 2019) to calculate the Normalized Semantic Similarity (NSS). An independent subset of 44 samples was employed, generating ten images per sketch and textual instruction pair with fixed “seeds”. The NSS metric was calibrated with minimum (tmin) and maximum (tmax) empirical thresholds, defined from SBERT similarity distributions, and scaled via Min-Max to a range of zero to one.

The qualitative evaluation involved 20 professionals from the areas of UI/UX design and “frontend” development, with a minimum of three years of experience, residing in Brazil or the United States. A structured, anonymous, and online questionnaire, with six evaluation blocks and a five-point Likert scale (Likert, 1932), was applied to analyze aesthetic quality, structural fidelity, and practical utility. The selection of samples for this stage was intentional, based on the results of the SSN metric and the availability of human resources, ensuring the anonymity of the participants.

3. Results and Discussion

The research employed a conditional latent diffusion approach, tailored for mobile interfaces, and its feasibility was evaluated through quantitative and qualitative methods. Overall results indicated that the proposed method surpassed the capabilities of the original base model, demonstrating effectiveness in the logical structuring of the interface. However, the evaluation also revealed persistent challenges related to aesthetic degradation and the loss of functional elements in more complex scenarios, pointing to the need for architectural enhancements to meet the visual rigor of the software development industry.

SBERT “embeddings” Similarity

Normalized Semantic Similarity (NSS) was calculated for 440 samples of user interfaces, covering diverse types. The consolidated statistical analysis of these results, presented in

, revealed a global median of 0.7928, indicating that the model captured the textual instruction intent in most inferences. However, the global mean of 0.6181 and an expressive standard deviation of 0.4021 highlighted the stochastic nature of the generation process and the inconsistency between interfaces of different complexity levels.

The frequency distribution of the SSN results, illustrated in

, complements this observation, highlighting the inherent variability in the choice of “tokens” by the multimodal model responsible for the textual descriptions of UIs. This variability is a crucial factor that influences the consistency and quality of the generated images, reinforcing the need for careful analysis of the results in different contexts of interface complexity.

To adapt the metric to the specific domain of user interfaces, minimum and maximum empirical limits were established. The empirical upper limit (Tmax) was defined as the 10th percentile of the cosine similarity distribution between textual descriptions of the same sample, as illustrated in

. This value, of 0.81, established a conservative ceiling, indicating that 90% of ideal descriptions exceed this cutoff score and avoiding penalizing the generative architecture for normal linguistic fluctuations.

In contrast, the empirical lower bound (Tmin) was established at the 75th percentile of the cosine similarity distribution between random unpaired pairs, as demonstrated in

. This value, of 0.46, aimed to define a minimum similarity level resulting from the intrinsic noise of the domain, attributed to the limited vocabulary in UI descriptions. Scores below this threshold indicate that the sample did not surpass the generic vocabulary baseline, failing to reproduce the minimum requested structure.

Normalized Semantic Similarity (NSS) was calculated using a Min-Max scaling technique, coupled with a strict capping mechanism. This approach ensures that the final metric is distributed within an interpretable range of 0 to 1, where raw values less than or equal to the lower bound result in zero, and values reaching or exceeding the empirical ceiling receive the maximum score. This function rigorously penalizes noisy samples and avoids an unrealistic search for identical results.

The 10% and 25% percentiles of SSN recorded a minimum value of zero, which empirically corroborates that at least 25% of the inference tests failed. This result is a direct reflection of the SSN normalization cutoff mechanism and the stochastic nature of the generation process, where the same input can generate opposite performances depending on the “seed” used. Such volatile and polarized behavior is consistent with findings in related studies (Avrahami et al., 2024; Ho et al., 2020; Xu et al., 2023).

The instability observed in the SSN results reaffirms the practical importance of generating images with multiple “seeds” and applying a rigorous semantic adherence cutoff before submitting the results to human evaluation. This strategy allows filtering samples that present only artificial similarity with the original textual instructions. The analysis of the SSN metric results was fundamental to identify generation flaws and results with greater semantic coherence, guiding the manual selection of six high-fidelity samples for qualitative evaluation.

Industry professional assessment

The qualitative evaluation involved 20 professionals from the UI/UX design and frontend development areas, with a minimum of three years of experience in the software industry. The majority of participants (65%) worked as frontend developers, while 35% identified as UI/UX designers. The geographical distribution indicated 70% of participants in Brazil and 30% in the United States, reflecting the research’s scope. A notable finding was the high adoption of artificial intelligence in daily work routines, with 80% reporting daily use and 20% regular weekly use, and 85% had already applied AI solutions for generating UI mockups or prototypes.

Participants evaluated six distinct criteria using a five-point Likert scale, whose statements were coded from Q1 to Q6. Responses were grouped into “satisfactory” (agree and strongly agree), “somewhat unsatisfactory” (neutral), and “highly unsatisfactory” (disagree and strongly disagree). This categorization allowed for a clear interpretation of the model’s performance regarding each criterion, validating the suitability of the expert group as technical judges for the research.

The Q1 statement, which assessed the accuracy with which the UI reflects the desired functional context, showed that 77% of the samples performed satisfactorily. This result attests to the model’s ability to understand the functional context of requests in the vast majority of scenarios evaluated. However, the polarization observed in some UIs, as illustrated in

, where a disapproval rate of 28.6% was concentrated, indicates that the architecture may present semantic comprehension failures depending on the nuances of the textual instruction or the application’s functional context.

The analysis of the Q2 statement, which assessed structural fidelity to the original outline, revealed a behavior directly related to the visual density of the interface. Low complexity interfaces, such as login and onboarding screens, exemplified in

and

, achieved approval of 85.7% and 81%, respectively. This result demonstrates that the model respected the limits of the provided outline without major structural deviations, which is crucial for UI prototyping.

In contrast, the samples with higher complexity received between 52.4% and 61.9% of satisfactory performance ratings for the Q2 statement. This lower level of satisfaction highlights that the adequacy of the geometric structure was not fully effective in more complex scenarios, being penalized according to the expectation and technical rigor level of the evaluators. This suggests that, although the model is promising for simpler layouts, it still needs improvements to handle the visual and hierarchical complexity of more elaborate interfaces.

In the context of the technical feasibility of the generated UIs, statement Q5, which measured the ease of implementation in code, registered an average approval rating of 78.6%, with unsatisfactory performance ranging from 0% to 19% among the evaluated samples. This result highlights that the developed generative pipeline can establish a well-defined component architecture and logical structural organization. From a frontend engineering perspective, the generated UIs deliver consistent foundations that reduce friction in the implementation of programming or markup language code, optimizing the transition between requirements and implementation.

In contrast, the items aimed at visual refinement and information organization, represented by statements Q3 (effectiveness of visual hierarchy) and Q4 (overall aesthetic quality), showed a clear decline. The evaluations concentrated the highest rates of disapproval, reaching an average of 21.4% and 19.8% for unsatisfactory performance, accompanied by 22.2% and 21.4% for poor performance, respectively. The sample illustrated in

marked the peak of the Q3 criterion’s failure rate, with 42.9% for not at all satisfactory performance and 23.8% for somewhat unsatisfactory.

In this specific case, the model suppressed critical elements for the e-commerce context outlined in the original sketch and textual instructions, such as the search bar and the button to save the product. Furthermore, it introduced deformed textual artifacts, compromising visual integrity and deviating the interface from the quality standard required in the design area. This divergence demonstrates rigorous evaluative behavior on the part of professionals, who penalize the model whenever jumbled typography or structural omissions impair the UI’s communicative clarity or the user experience.

As a final item, the Q6 statement, which assessed the quality of the generated UIs according to professional expectations, registered a global average of 50.8% for satisfactory performance, 16.7% for somewhat satisfactory, and 32.5% for not at all satisfactory. The segmentation of responses revealed a polarization of perceptions: the frontend engineers group showed greater tolerance for visual anomalies, with 61.9% approval versus 23.8% rejection, possibly valuing the agility in the initial structuring of components. The UI/UX designers group, on the other hand, presented an average rejection rate of 50%, with only 28.6% of satisfactory performance evaluations.

This qualitative interpretation of the results shows that the loss of fidelity to the sketches and aesthetic degradation strongly impact the work of UI/UX design specialists, being more sensitive barriers for these professionals. Understanding these factors is essential to map the current limitations of the model, representing crucial improvement points to enable the full adoption of this technology in creative workflows of the software development cycle, aligning technical agility with the visual rigor required by the industry.

Comparative with the state of the art

For comparison with commercially available multimodal reference models, inference tests were performed using the same samples submitted for professional evaluation. The Google “Gemini 2.5-flash-image” model (Google, 2025), the OpenAI “gpt-image-1.5” model (OpenAI, 2026), and the original base model used in the study, “Stable Diffusion 3.5 Large” (Rombach et al., 2022), were tested. The comparison, illustrated in

, demonstrates that the results generated by Delineo surpass the capabilities of the original base model in UI prototyping.

When compared with the most powerful models on the market, such as Gemini 2.5-flash-image and gpt-image-1.5, Delineo’s capacity for structure generation is comparable. Aesthetics, on the other hand, are closer to industry standards, with a more modern and professional look than the results generated by the other models. This comparison attests to the efficiency of the low-rank adaptation (LoRA) as an essential part of the created pipeline structure, responsible for adding stylistic nuances and specific characteristics without requiring a large volume of data (Hu et al., 2022; Gandikota et al., 2024; Wu et al., 2024).

On the other hand, Google and OpenAI models demonstrated less forgetting of elements requested in the instructions and outline, in addition to a deeper reasoning capacity. It was noted that these more robust models can elaborate on the content of relevant textual elements and add elements not originally specified, but which are common within the semantic context of the UI. This more advanced inference capability suggests that, although Delineo is effective in structuring, there is still room for improvement in contextual intelligence and textual content generation.

In summary, the research demonstrated the technical feasibility of adapting latent diffusion models for sketch-guided user interface prototyping, outperforming the original base model. The quantitative results from the SSN metric confirmed the model’s ability to capture the semantic intent of instructions, while the qualitative evaluation with industry experts validated the practical utility of the tool and its capacity to generate consistent foundations for code implementation. However, the analyses also highlighted that the automation of creative processes still requires architectural improvements to achieve the visual rigor and aesthetic fidelity demanded by the software development industry, especially in scenarios of greater visual and hierarchical complexity.

4. Conclusion

The study sought to adapt latent diffusion models for the generation of high-fidelity prototypes, through supplementary training, utilizing open-source technologies and publicly available databases. It was verified that the conditional latent diffusion approach, which integrated a control architecture and a low-rank adapter to a large-scale model, surpassed the capabilities of the original base model in prototyping mobile user interfaces. Quantitative evaluation, through Normalized Semantic Similarity (NSS), indicated that the model captured the semantic intent of textual instructions in most inferences. Additionally, qualitative analysis with industry experts confirmed the practical viability of the tool, especially in the logical structuring of the interface and in the ability to generate consistent foundations for code implementation, optimizing the transition between visual requirements and coding.

However, the evaluation also highlighted significant limitations, such as aesthetic degradation and loss of functional elements in more complex interfaces, in addition to failures in typography generation and suppression of critical elements. It was observed that the effectiveness in the logical structuring of the interface does not yet meet the visual rigor required by the software development industry, with a polarization in professional expectations, where UI/UX designers showed less tolerance for visual anomalies. The instability in the SSN results, with a considerable portion of inference failures, reinforces the stochastic nature of the process. It is suggested that future studies explore architectural improvements to refine aesthetic quality and structural fidelity in complex scenarios, as well as investigate techniques that require a smaller data volume to mitigate the scarcity of modern UIs databases.

Bibliographic References

Al-Saqqa, S.; Sawalha, S.; Abdel-Nabi, H. 2020. Agile software development: methodologies and trends. International Journal of Interactive Mobile Technologies 14(11): 246.

Beltramelli, T. 2018. Pix2code: generating code from a graphical user interface screenshot. In: Proceedings of the EICS ’18: ACM Symposium on Engineering Interactive Computing Systems, 2018, Paris, França. Anais… p. 1-6.

He, J.; Treude, C.; Lo, D. 2025. LLM-based multi-agent systems for software engineering: literature review, vision, and the road ahead. ACM Transactions of Software Engineering and Methodology 34(5): 164-193.

Ismail, A.A.; Hassan, R. 2019. Technical competencies in digital technology towards industrial revolution 4.0. Journal of Technical Education and Training 11(3): 55-62.

Stanford Institute for Human-Centered Al [HAI]. 2025. The Al index 2025 annual report. Disponível em <https://hai.stanford.edu/assets/files/hai_ai_index_report_2025.pdf>. Acesso em 24 out. 2025.

Wan, Y.; Wang, C.; Dong, Y.; Wang, W.; Li, S

Article originating from the Final Course Project of the Specialization in Data Science and Analytics of the MBA USP/Esalq

To learn more about the course, click here and access the MBX Academy platform