jwzhanggy/graph-bert

Source code of Graph-Bert

Graph-Bert

- Depending on your transformer toolkit versions, the transformer import code may need to be adjusted, like as follows:
+ from transformers.modeling_bert import BertPreTrainedModel, BertPooler
+ --> from transformers.models.bert.modeling_bert import BertPreTrainedModel, BertPooler
- (Please check your transformer toolikt, and update the import code accordingly.)

Graph-Bert: Only Attention is Needed for Learning Graph Representations

Paper URL at IFM Lab: http://www.ifmlab.org/files/paper/graph_bert.pdf

Paper URL at arXiv: https://arxiv.org/abs/2001.05140

Graph-Bert Paper List

A list of the latest research papers on graph-bert can be found via the following link
Page List URL: https://github.com/jwzhanggy/graph_bert_work

Two other papers are helpful for readers to follow the ideas in this paper and the code

(1) SEGEN: Sample-Ensemble Genetic Evolutional Network Model https://arxiv.org/abs/1803.08631

(2) GResNet: Graph Residual Network for Reviving Deep GNNs from Suspended Animation https://arxiv.org/abs/1909.05729

Graph Neural Networks from IFM Lab

The latest graph neural network models proposed by IFM Lab can be found via the following link
IFM Lab GNNs: https://github.com/jwzhanggy/IFMLab_GNN

References

@article{zhang2020graph,
  title={Graph-Bert: Only Attention is Needed for Learning Graph Representations},
  author={Zhang, Jiawei and Zhang, Haopeng and Xia, Congying and Sun, Li},
  journal={arXiv preprint arXiv:2001.05140},
  year={2020}
}

---

How to run the code?

To run a script, you can just use command line: python3 script_name.py

After downloading the code, you can run

python3 script_3_fine_tuning.py

directly for node classification. It seems the random seed cannot control the randomness in parameter initialization in transformer, we suggest to run the code multiple times to get good scores.

What are the scripts used for?

(1) The Graph-Bert model takes (a) node WL code, (b) intimacy based subgraph batch, (c) node hop distance as the prior inputs. These can be computed with the script_1_preprocess.py.

(2) Pre-training of Graph-Bert based on node attribute reconstruction and graph structure recovery is provided by script_2_pre_train.py.

(3) Please check the script_3_fine_tuning.py as the entry point to run the model on node classification and graph clustering.

(4) script_4_evaluation_plots.py is used for plots drawing and results evaluation purposes.

How to turn on/off the blocks?

You can change the "if 0" to "if 1" to turn on a script block, and the reverse to turn off a script block.

How save/load pre-trained graph-bert?

Please refer to the attached page for model save/load in pytorch:
https://pytorch.org/tutorials/beginner/saving_loading_models.html

You can load the pre-trained graph-bert component with pytorch model (e.g., the graph-bert inside MethodGraphBertNodeClassification) with the save and load functions in the above page. According to our experiments, Save/Load Entire Model in the above page works perfectly, whereas Save/Load state_dict doesn't really work.

Several toolkits may be needed to run the code

(1) pytorch (https://anaconda.org/pytorch/pytorch)
(2) sklearn (https://anaconda.org/anaconda/scikit-learn)
(3) transformers (https://anaconda.org/conda-forge/transformers)
(4) networkx (https://anaconda.org/anaconda/networkx)

---

Organization of the code?

A simpler template of the code is also available at http://www.ifmlab.org/files/template/IFM_Lab_Program_Template_Python3.zip

The whole program is divided into five main parts:

(1) data.py (for data loading and basic data organization operators, defines abstract method load() )

(2) method.py (for complex operations on the data, defines abstract method run() )

(3) result.py (for saving/loading results from files, defines abstract method load() and save() )

(4) evaluate.py (for result evaluation, defines abstract method evaluate() )

(5) setting.py (for experiment settings, defines abstract method load_run_save_evaluate() )

The base class of these five parts are defined in ./code/base_class/, they are all abstract class defining the templates and architecture of the code.

The inherited class are provided in ./code, which inherit from the base classes, implement the abstract methonds.

Detailed information on funtional classes?

a. data

(1) DatasetLoader.py (for dataset loading)

b. method

b-1 pre-computing methods

(1) MethodWLNodeColoring.py (for WL code computing)

(2) MethodGraphBatching.py (for subgraph batching)

(3) MethodHopDistance.py (for hop distance computing)

b-2 graph-bert basics

(1) MethodBertComp.py (for graph-bert basic components)

(2) MethodGraphBert.py (the graph bert model)

b-3 graph-bert applications

(1) MethodGraphBertNodeClassification.py

(2) MethodGraphBertGraphClustering.py

(3) MethodGraphBertNodeConstruct.py

(4) MethodGraphBertGraphRecovery.py

c. result

(1) ResultSaving.py (for saving results to file)

d. evaluate

(1) EvaluateAcc.py (accuracy metric)

(2) EvaluateClustering.py (a bunch of clustering metrics)

e. setting

(1) Settings.py (defines the interactions and data exchange among the above classes)

---

Learning results of Graph-Bert with graph-raw residual on Cora.

---

Learning results of Graph-Bert with raw residual on Cora.

---

graph-bert's Issues

How could I get the data of citeseer and pubmed?

I try to run script _3_fine_tuning.py on citeseer and pubmed, but it raises a OSError: ./data/pubmed//node not found. How do I get this data?

Finetuning for custom dataset

Could the model be used for fine tuning on a custom dataset? I have a dataset of medical texts and have made a vocabualry graph out of it. I want to optimise the clustering and output a knowlege graph indicating the interactions between the entities.

Thanks in advance!

Custom dataset: float feature values

Hi, is it possible to run the code for a custom dataset where the node features are float instead of binary (as in cora)?
If so, can you guide me to the files/tasks/concepts I should be looking at for making necessary changes?

transformers edition

Your work has inspired me a lot.Thank you for your very useful work.but when I try to run the code ,I find a error:

File "./Graph-Bert/code/MethodBertComp.py", line 11, in
from transformers.modeling_bert import BertPredictionHeadTransform, BertAttention, BertIntermediate, BertOutput
ModuleNotFoundError: No module named 'transformers.modeling_bert'

I guess that maybe transformer edition is not appropriate，could you tell me transformers edition,please help me .
Thank you!

Graph representation of custom dataset and ASTs

The given cora example makes nodes from papers and has links between each node (paper). However, my dataset has a bunch of text files and my graph is focused on each individual text file rather than dependency of one file to another and the order of tokens/features within each file is important as well. I have an AST graph of each text file but I'm unsure of how I can represent it for input to Graph-Bert.
If I make nodes out of each line of text, the feature vectors will be too sparse for each node and the number of nodes will be extremely large. If I make nodes on each text file, the structure and dependency within each file will be lost and I am unsure of what the links will be then.

If there's a way to represent this for input to Graph-Bert, please let me know. Thank you.

About the size of the input graph

Although you said that your method support large size graph in your paper, I tested it on a graph with about 10, 000 nodes, the program was terminated while running because it exceeded the system memory. However, the memory of the server is 128G，so where might the problem lie?

nfeature, ngraph

Hey!
Could you please clarify about what nfeature and ngraph mean in the code. I noticed that they aren't equal to the document encoding size and number of links respectively

Activation function implementation

Hi, I wonder where do you implement the activation function in the code?
I saw you declared "hidden_act" as "gelu", but not using "hidden_act" or declare any activation function in the project.
I'm looking forward to your response, Thank you :)

Too many errors in script 4 evalution

1. What's the variable 'pre_train_task', never see it in other file.
   

2. Maybe the same as above.
   

Are there some tricks in the stage of fine-tuning?

I run the script_3_fine_tuning.py on Cora for node classification task 5 times, but i can not get the result reported in paper. The results of my experiments are as follows:

Are there some other tricks in the stage of fine-tuning that can improve the accuracy?

Running script_2_pre_train for Node Attribute Reconstruction and Graph Bert Network Structure Recovery

Hi! Your paper states that:
"In the experiments, we first pre-train GRAPH-BERT based on
the node attribute reconstruction task with 200 epochs, then
load and pre-train the same GRAPH-BERT model again based
on the graph structure recovery task with another 200 epochs."

However, script_2_pre_train.py learns two different models. Could you please specify, whether you applied those models separately for the downstream task?

If your paper is describing the workflow correctly, what should I do to train one model on both tasks?

Thank you in advance for the answer.
Yours sincerely,
Irina Nikishina.

Is it possible to apply your GRAPH-BERT on directed graphs ?

Hi,
I'm wondering if it's possible to GRAPH-BERT on directed graphs.

Thank you.

There are too many nodes and edges. Not enough memory

There are too many nodes and edges. DatasetLoader features = torch.FloatTensor(np.array(features.todense())). Is there a way like bert batch?

[Question] Position-Embed function Eqn. 3

Hi,
Thanks for the great work and the source codes!

I was wondering whether the Eqn 3 from the paper is implemented in the codes? I could not find them, and notice that the position features are directly passed to nn.Embedding()?

Graph-Bert/code/MethodBertComp.py

Lines 99 to 104 in e878576

	def __init__(self, config):
	super(BertEmbeddings, self).__init__()
	self.raw_feature_embeddings = nn.Linear(config.x_size, config.hidden_size)
	self.wl_role_embeddings = nn.Embedding(config.max_wl_role_index, config.hidden_size)
	self.inti_pos_embeddings = nn.Embedding(config.max_inti_pos_index, config.hidden_size)
	self.hop_dis_embeddings = nn.Embedding(config.max_hop_dis_index, config.hidden_size)

Graph-Bert/code/MethodBertComp.py

Lines 109 to 114 in e878576

	def forward(self, raw_features=None, wl_role_ids=None, init_pos_ids=None, hop_dis_ids=None):
	raw_feature_embeds = self.raw_feature_embeddings(raw_features)
	role_embeddings = self.wl_role_embeddings(wl_role_ids)
	position_embeddings = self.inti_pos_embeddings(init_pos_ids)
	hop_embeddings = self.hop_dis_embeddings(hop_dis_ids)

Thanks,
Vijay

Some questions about intimacy matrix

I have some questions while reading the paper of Graph-BERT. What does the intimacy score mean? What the difference between intimacy score and similarity score? When I try to understand the intimacy matrix, I find the nearest neighbors will get the highest intimacy score(e.g. 1-hop neighbors always get the highest intimacy score). Why does the context nodes can cover both local neighbors as well as the nodes which are far away.

Could you please help figure out these problems? Thanks for your help!

[question] Weight Loading

Hi,

I have a question where fine-tune script loads the pre-trained weight files?

Why it is running on CPU?

I found out the script_3 is running on CPU, why? Though it is not very slow

Hello, can you add a bit of documentation on how the graph should be represented?

I noticed that you assume everything is stored as numpy matrices already. Is there a pointer to how graphs are represented in the node and links files (is links just a sparse adjacency list and node a set of features)? Thanks!

ModuleNotFoundError: No module named 'transformers.modeling_bert'

> you can change the code to use mini-batch instead

Sorry to bother you again. This issue happened when I try to use DatasetLoader.py to generate intimacy matrix "S", but isn’t mini-batch used in the model training process?

Originally posted by @code-gamer in #23 (comment)

Code doesn't match paper

Hi:
In script_3_fine_tuning, you have one downstream task graph-clustering
In MethodGraphBertGraphClustering,py,from line 38 to line 41, is your clustering process while you didn't reuse any pre-train results in this downstream task. Is it belong to fine-tuning?
kmeans = KMeans(n_clusters=self.cluster_number, max_iter=self.max_epoch)
if self.use_raw_feature:
clustering_result = kmeans.fit_predict(self.data['X'])
else:
clustering_result = kmeans.fit_predict(sequence_output.tolist())

About mini-batch size

Hi, thank you for your work.
I have a question about mini-batch size, where can I find it in your code?
Thank you.

transformer edition

Your work has inspired me a lot.Thank you for your very useful work.but when I try to run the code ,I find a error:

File "./Graph-Bert/code/MethodBertComp.py", line 11, in
from transformers.modeling_bert import BertPredictionHeadTransform, BertAttention, BertIntermediate, BertOutput
ModuleNotFoundError: No module named 'transformers.modeling_bert'

I guess that maybe transformer edition is not appropriate，please help me .
Thank you!

question about the linkless subgraphs sampling

Hi,
In your graph-bert, you calculate the intimacy matrix in step 1, then sample linkless subgraphs and embed nodes in step 2. In the process of sampling linkless subgraphs, nodes of the linkless subgraphs are sorted based on the intimacy scores.

The question is, is it possible to change this sampling method to something else? (e.g., ripple walk sampling algorithm. I know this is one of your great works.)
If yes, how and where should I rewrite this codes to implement that?

Thank you in advance.