GUI2Vec

Learning GUI Design Vector Space for Assisting Developers with GUI Development

Dataset construction

The apps (from Rico dataset) with less than 10 GUI screenshots are filtered out, as it may mean that those apps are not fully explored. There are 2618 apps left. And we take about 84.6% of apps (2,214) for training, 5.4% (142) for validation i.e., tuning the hyperparameters of our model, and the rest 10% (262) for testing. In detail, we have 17,879 intra/inter pairs for training, 1,132 intra/inter pairs for validation, and 2,178 intra/inter pairs for testing.

• Download the data: 2618 apps, split apps (2214 for training, 142 for validation, and 262 for testing), train_pairs, valid_pairs, test_pairs.

Assumption for the GUI Design Style

A pilot study to manually check the assumption for the GUI design style:

We randomly sample 384 pairs of GUIs for manual evaluation, and half of them are from the inter apps, while the other half come from intra apps. Two Master students P1 and P2 are asked to individually check if each pair of GUI is sharing the very similar design style or not and they are not told if the pair of GUIs comes from the same app or not. The results show that 89.32% GUI pairs from the same app are regarded as sharing the similar design format, and 92.71% GUI pairs from different apps are regarded as possessing different design styles. The Cohen’s Kappa metric among annotator’s decisions is 82.03%.

Download the style_pair_samples

P1_o = 0.94, P1_e = 0.5, Kappa rate1 = 0.88 P1 file

P2_o = 0.88, P2_e = 0.5, Kappa rate2 = 0.76 P2 file

• Download the file

GUI2Vec Search_TSNE

T-SNE is used to project the high-dimension embedding to 2-dimension vector for visualization. We visualize the embedding of random design images which are from 20, 30, 50, 100, 150, and 200 apps.

20 apps

30 apps

50 apps

100 apps

Clustering Overall Performance

To check the quality of GUI embedding with 5 metrics(Homogeneity, Completeness, V-measure, Adjusted Rand Index, and Adjusted Mutual Information), we randomly select apps with the number ranging from 10 to 100.

• Download more experiment results of clustering

GUIsearch_results

We build a vector space of GUI designs by converting all GUI in the training set into the vectors. Then We randomly select 10 GUIs from 10 apps from our testing dataset, and convert them into vector by our trained model as the queries. For each query, we retrieve the top 10 GUIs whose vectors are most closed to that of query GUI.

• Download the GUI images of GUI searching

• P1 GUI searching result

• P2 GUI searching result

• Download more experiment results of GUI searching

GUI_Style_Consistency_Detection

We randomly select 10 apps from 8 categories. For each of the app, we individually check which GUIs are outliers with different design style from the whole app. Thus, we construct a ground truth dataset of GUI style inconsistency, and we evaluate the performance of different methods with this ground truth.

• Download more experiment results of GUI Style Consistency Detection