serendipitysx/chartspark

Let the Chart Spark: Embedding Semantic Context into Chart with Text-to-Image Generative Model

Let the Chart Spark: Embedding Semantic Context into Chart with Text-to-Image Generative Model

This repository contains the code of ChartSpark, a novel system that embeds semantic context into chart based on text-to-image generative model. ChartSpark generates pictorial visualizations conditioned on both semantic context conveyed in textual inputs and data information embedded in plain charts. The method is generic for both foreground and background pictorial generation, satisfying the design practices identified from an empirical research into existing pictorial visualizations. We further develop an interactive visual interface that integrates a text analyzer, editing module, and evaluation module to enable users to generate, modify, and assess pictorial visualizations.

Dependencies and Installation

gitclone https://github.com/SerendipitysX/ChartSpark.git
cd ChartSpark
conda create --name <environment_name> --fisle requirements.txt

Pretrained Model

In this work, we benefit from some excellent pretrained models, including WebVectors for seeking the relevant words, DIS for background removal, and diffusers for generating the image containing semantic context.

To use these models, please follow the steps below:

1. Generation: We use three pipline from diffusers: runwayml/stable-diffusion-v1-5, stabilityai/stable-diffusion-2-depth, and Fantasy-Studio/Paint-by-Example. You can download and load them as local files, which can be specified in chartSpeak.py line 37-44.
2. Removal: Download the removal pretrained models from here, unzip the folder and save it to mask/bg_removal.pth.
3. Extract Keyword (Optional): Select one of corpus from WebVectors and save it to theme_extract theme_extract/. You will have three files: meta.json, model.bin, model.txt.

Backend Setup

python chartSpeak.py

Frontend Setup

see this video for real-world usage：video

1. Go to ChartSpark/frontend path
2. Install all the needed packages through npm

npm install

3. Compiles and hot-reloads for development

npm run dev

Contact

We are glad to hear from you. If you have any questions, please feel free to contact [email protected] or open issues on this repository.

Citation

@ARTICLE {xiao2023CharSpark,
author = {S. Xiao and S. Huang and Y. Lin and Y. Ye and W. Zeng},
journal = {IEEE Transactions on Visualization & Computer Graphics},
title = {Let the Chart Spark: Embedding Semantic Context into Chart with Text-to-Image Generative Model},
year = {2024},
volume = {30},
number = {01},
issn = {1941-0506},
pages = {284-294},
doi = {10.1109/TVCG.2023.3326913},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
month = {jan}
}

chartspark's Issues

Have some issues when running python file

It seems there are some issues in chartspeak.py and matplotlib package.

Traceback (most recent call last): File "E:\ChartSpark\chartSpeak.py", line 4, in <module> from mask.bar_mask import * File "E:\ChartSpark\mask\bar_mask.py", line 104, in <module> table2img_bar(x, height, labels, aspect_ratio[3], title, y_limit, bar_width) File "E:\ChartSpark\mask\bar_mask.py", line 25, in table2img_bar fig.savefig('output/preview/bar_preview.png', dpi=fig.dpi) File "C:\Users\97782\.conda\envs\chartEnv\lib\site-packages\matplotlib\figure.py", line 3046, in savefig self.canvas.print_figure(fname, **kwargs) File "C:\Users\97782\.conda\envs\chartEnv\lib\site-packages\matplotlib\backend_bases.py", line 2319, in print_figure result = print_method( File "C:\Users\97782\.conda\envs\chartEnv\lib\site-packages\matplotlib\backend_bases.py", line 1648, in wrapper return func(*args, **kwargs) File "C:\Users\97782\.conda\envs\chartEnv\lib\site-packages\matplotlib\_api\deprecation.py", line 415, in wrapper return func(*inner_args, **inner_kwargs) File "C:\Users\97782\.conda\envs\chartEnv\lib\site-packages\matplotlib\backends\backend_agg.py", line 541, in print_png mpl.image.imsave( File "C:\Users\97782\.conda\envs\chartEnv\lib\site-packages\matplotlib\image.py", line 1675, in imsave image.save(fname, **pil_kwargs) File "C:\Users\97782\.conda\envs\chartEnv\lib\site-packages\PIL\Image.py", line 2350, in save fp = builtins.open(filename, "w+b") FileNotFoundError: [Errno 2] No such file or directory: 'output/preview/bar_preview.png'

Do I need conda to use ChartSpark

As the title, do I need to configure the conda environment before using ChartSpark?

Miss of theme_extract.similar_text

in chartSpeak.py ,I got ModuleNotFoundError: No module named 'theme_extract.similar_text'. What's theme_extract? I didn't find any thing about this. Is it a module, a NLPL model or something else.

关于软件使用的问题

我试着向程序中导入了项目中提供的示例json数据，并没有反应，请问可以提供一下软件的具体操作方法吗