hugochan/idgl

Code & data accompanying the NeurIPS 2020 paper "Iterative Deep Graph Learning for Graph Neural Networks: Better and Robust Node Embeddings".

IDGL

Code & data accompanying the NeurIPS 2020 paper "Iterative Deep Graph Learning for Graph Neural Networks: Better and Robust Node Embeddings".

Architecture

Get started

Prerequisites

This code is written in python 3. You will need to install a few python packages in order to run the code. We recommend you to use virtualenv to manage your python packages and environments. Please take the following steps to create a python virtual environment.

• If you have not installed virtualenv, install it with pip install virtualenv.
• Create a virtual environment with virtualenv venv.
• Activate the virtual environment with source venv/bin/activate.
• Install the package requirements with pip install -r requirements.txt.

Run the IDGL & IDGL-Anch models

• Cd into the src folder

• Run the IDGL model and report the performance

       python main.py -config config/cora/idgl.yml
  

• Run the IDGL-Anch model and report the performance

       python main.py -config config/cora/idgl_anchor.yml
  

• Notes:

  • You can find the output data in the out_dir folder specified in the config file.
  • You can add --multi_run in the command to run multiple times with different random seeds. Please see config/cora/idgl.yml for example.
  • To run IDGL & IDGL-Anch without the iterative learning or graph regularization components, please set max_iter to 0 or graph_learn_regularization to False in the config file.
  • You can download the 20News data from here, and move it to the data folder.

Reference

If you found this code useful, please consider citing the following paper:

Yu Chen, Lingfei Wu and Mohammed J. Zaki. "Iterative Deep Graph Learning for Graph Neural Networks: Better and Robust Node Embeddings." In Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Dec 6-12, 2020.

@article{chen2020iterative,
  title={Iterative Deep Graph Learning for Graph Neural Networks: Better and Robust Node Embeddings},
  author={Chen, Yu and Wu, Lingfei and Zaki, Mohammed},
  journal={Advances in Neural Information Processing Systems},
  volume={33},
  year={2020}
}

idgl's Issues

doubts about function batch_normalize_adj

Great thx to your such a nice approach. As given in https://github.com/hugochan/IDGL/blob/master/src/core/utils/generic_utils.py#L77, row-normalization should be 'mx.sum(2)' or 'mx.sum(-1)' when it's in a batch. Right ?

Thanks!

idgl.yml

When I run python main.py -config config/cora/idgl.yml, I get the following error:

Traceback (most recent call last):
File "main.py", line 123, in
config = get_config(cfg['config'])
File "main.py", line 58, in get_config
config = yaml.load(setting)
TypeError: load() missing 1 required positional argument: 'Loader'

Question about batch training for Graph classification

Is IDGL able to train a batch of graph with different node numbers for a iteration?

What's the purpose of using two Graph Learner

According to the code line321 in model_handler
and line 371 in model_handler, there are two GraphLearner, and the first one is for the first iteration and the second one for the rest iterations. What's the purpose? I haven't found descriptions in the paper. Could you please provide more information about that? Thanks

reproduce ogbn-arxiv

Quick question on reproducing ogbn-arxiv using IDGL code. May I please quickly ask what your environment looks like? We ran the source code and always hit OOM error. It looks like, if our math is correct, the system requires at least 100GB GPU memory...

out of memory?

Hi:
i met some errors when i run the codes. The error messages as following:

datasets and enviroments:
size of datasets: 4000*500, 4000 is the number of nodes and 500 is the number of features.
GPU memory: 12G

Train Epoch: [9 / 500]
Traceback (most recent call last):
File "/data0//work/IDGL/src/main.py", line 127, in
main(config)
File "/data0//work/IDGL/src/main.py", line 24, in main
model.train()
File "/data0/work/IDGL/src/core/model_handler.py", line 148, in train
self.run_epoch(self.train_loader, training=True, verbose=self.config['verbose'])
File "/data0/work/IDGL/src/core/model_handler.py", line 744, in _run_whole_epoch
cur_raw_adj, cur_adj = network.learn_graph(network.graph_learner2, node_vec, network.graph_skip_conn, graph_include_self=network.graph_include_self, init_adj=init_adj)
File "/data0/work/IDGL/src/core/models/graph_clf.py", line 94, in learn_graph
raw_adj = graph_learner(node_features)
File "/.conda/envs/jx_dp/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/data0/work/IDGL/src/core/layers/graphlearn.py", line 86, in forward
attention = torch.matmul(context_norm, context_norm.transpose(-1, -2)).mean(0)
RuntimeError: CUDA out of memory. Tried to allocate 330.00 MiB (GPU 4; 10.92 GiB total capacity; 6.50 GiB already allocated; 279.06 MiB free; 6.70 GiB reserved in total by PyTorch)

ERROR: No matching distribution found for torch==0.4.1.post2

Hello,

I am currently trying to run IDGL on Docker (in an nvidia/cuda:10.2-base container, using Python 3.8.10). However, when I try to install the needed packages (using requirements.txt), I get the following error:

ERROR: Could not find a version that satisfies the requirement torch==0.4.1.post2 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1)
ERROR: No matching distribution found for torch==0.4.1.post2

Any idea how what the issue here might be? I've seen a similar issue here (pytorch/pytorch#10443), but its recommendations (using pipenv / conda) have both failed for me as well.

Thank you very much!
--Jeffrey

How to use IDGL on my own dataset?

Thank you for your great work! But I have some problems when I use the code on my own dataset. Here are some details:

1. My task is graph classification;
2. The size of my input features is 7000×72×200: 7000 samples and each sample has 72 nodes and the input dimension of each node is 100;
3. The dataset can be divided into 3 classes.
   When I set my own config, and it reported "correct = preds.eq(labels).double()
   RuntimeError : The size of tensor a (3) must match the size of tensor b (6240) at non-singleton dimension 1". Also I noticed that the output of the GNN is 72×3 for each sample, but it should be 3. Where should I change the code for my dataset? Thank you!

about the code of papers

Hi， Excuse，is the code for "Iterative Deep Graph Learning for Graph Neural Networks: Better and Robust Node Embeddings " different form "Deep Iterative and Adaptive Learning for Graph Neural Networks"?
In the paper "Iterative Deep Graph Learning for Graph Neural Networks: Better and Robust Node Embeddings",you give the implementation: https://github.com/hugochan/IDGL. But, the page of github show that ,it is for "Deep Iterative and Adaptive Learning for Graph Neural Networks". I am confused about this. thanks.

How are the pictures in the appendix drawn

Hi,
excellent work.
I am eager to know how the pictures in the appendix are drawn, because these pictures are so charming and I didn't find it in the code. Can you provide it to me? Thank you.http://m.qpic.cn/psc?/V14ADfFj30Vktj/TmEUgtj9EK6.7V8ajmQrEEFcmPNruYtCv3LwJgjqOwmRH6lXLjw5wV6z5vuK9TmfVBORolCcbwnXlVINWfGWYMnYf*jRj0BaCeT0l.0euZc!/b&bo=MgMHAzIDBwMDKQw!&rf=viewer_4

Bug when I run for 20news

when I switch to dataset:20news，this bug occurred , and I think it is not a technical bug but missing file?

May I ask a question about graph classification? Is it to synthesize multiple graphs in a batch into a large graph (node_vec and adj) into GNN for training?

node_vec = torch.relu(network.encoder.graph_encoders[0](init_node_vec, cur_adj))
Can you provide the shape of the init_node_vec and cur_adj?

Implementation of Anchor-based graph learning

Hi,

Great work. Do you also have the anchor-based graph learning in this repo? I did not find it here.

Thanks

NLOSS is positive

I have been running into an odd case during training: sometimes NLOSS takes a positive value.

My current understanding is this:
I am targeting downstream classification, so the loss function being used is NLL (negative log likelihood). My understanding is that log likelihood is the log of a values on the interval [0:1]. This means that log must be negative, or 0. Thus, its negative must be positive, or 0.

NLOSS in IDGL is the negative of the NLL loss function, ie -NLL. Thus, it just recovers the original log-likelihood, which is strictly non-positive.

My observations:
I have observed cases where it is positive. Oddly, in most of these cases, I will see that, for some loss value L, the accuracies associating with training models that have losses near L or near -L are nearly identical.

This prompted me to think that there is some edge case where a negative is not applied (or is applied once too many times)

One sample of a case where the NLOSS is positive is this:

{
    "data_type": "network",
    "dataset_name": "cora",
    "data_dir": "../data/cora/",
    "pretrained": null,
    "task_type": "classification",
    "out_dir": "/workspace/output/models/TwigJob_n5_bdg250_loglr_repeat_fixout/3_0_7/0.7623429221128561",
    "seed": 77,
    "model_name": "GraphClf",
    "scalable_run": true,
    "num_anchors": 7967,
    "hidden_size": 43,
    "use_bert": false,
    "dropout": 0.6635843579072174,
    "feat_adj_dropout": 0.06838793351721184,
    "gl_dropout": 0.548838069426093,
    "bignn": false,
    "graph_module": "gcn",
    "graph_type": "dynamic",
    "graph_learn": true,
    "graph_metric_type": "weighted_cosine",
    "graph_skip_conn": 0.7949061507223335,
    "update_adj_ratio": 0.632239148921817,
    "graph_include_self": false,
    "graph_learn_regularization": true,
    "smoothness_ratio": 0.530333708979554,
    "degree_ratio": 0.804512963814122,
    "sparsity_ratio": 0.10112626099490862,
    "graph_learn_ratio": 0,
    "graph_learn_hidden_size": 33,
    "graph_learn_epsilon": 0,
    "graph_learn_topk": null,
    "graph_learn_num_pers": 6,
    "graph_hops": 4,
    "gat_nhead": 8,
    "gat_alpha": 0.2,
    "optimizer": "adam",
    "learning_rate": 1.634816494138241e-05,
    "weight_decay": 0.002369427396679898,
    "lr_patience": 7,
    "lr_reduce_factor": 0.20687033919483733,
    "grad_clipping": null,
    "grad_accumulated_steps": 6,
    "eary_stop_metric": "nloss",
    "pretrain_epoch": 0,
    "max_iter": 10,
    "eps_adj": 8.5e-05,
    "rl_ratio": 0,
    "rl_ratio_power": 1,
    "rl_start_epoch": 1,
    "max_rl_ratio": 0.99,
    "rl_reward_metric": "acc",
    "rl_wmd_ratio": 0,
    "random_seed": 77,
    "shuffle": true,
    "max_epochs": 27.777777777777775,
    "patience": 100,
    "verbose": 20,
    "print_every_epochs": 500,
    "out_predictions": false,
    "out_raw_learned_adj_path": "cora_idgl_node_anchor_adj.npy",
    "save_params": true,
    "logging": true,
    "no_cuda": false,
    "cuda_id": 0
}

The output of this is:

NLOSS = 6.22605
ACC = 0.19400

Do you have any idea why this would occur? Or is my understanding of how loss is reported incorrect?

Thank you in advance!

Sincerely,
--Jeffrey Sardina

Questions regarding time complexity

Hi, thanks for the insightful work IDGL.

However, I cannot fully agree with the complexity discussion in the paper. In the paper (Page 5), the paper claims that IDGL-Anch has time and space complexity of O(ns). Meanwhile, in my humble opinion, the message passing scheme should be far greater than O(ns).

As stated in the paper, in each $MP_a$ operation, the adjacency matrix is firstly recovered and then used in message passing. The adjacency matrix is recovered by $RowNorm(R) \cdot transpose(ColNorm(R))$, which is already $O(n^2s)$ complexity if two matrices are dense (The time complexity of dense matrix multiplication between mxn and nxp matrix is O(mnp)).

Please kindly correct me if I was wrong. Your time is highly appreciated.

value of hyper parameter to reproduce the experiment

It seems that the values of hyper parameters provided in config file could not promise results reported in the paper? Could you provide specific values, especially for smooth ratio, degree ratio, and sparsity ratio

can IDGL used in Time Series Task? I notice that in ur model IDGL you only do classification/regression/text， do u have this model? tks

Can you publish the source code? Thanks

Questions about Eq. 3

Hi Yu,

Thank you for your detailed analysis and the commendable work. I have a question regarding Eq. 3 in the paper and how it relates to the implementation in the code.

Above is Eq. 3 in the paper.

The following is what I understood from the code:

1. The cur_adj before the while loop is assigned to the first_adj variable. It appears that first_adj represents
   $$\lambda L^{(0)} + (1 - \lambda)f(A^{(1)})$$ where $A^{(1)}$ is calculated from the initial node features init_node_vec($X$). Refer to this line, where graph_skip_conn is $\lambda$, init_adj is $L^{(0)}$, and adj is $f(A^{(t)})$.

2. Similarly, the cur_adj in the 'while' loop represents $\lambda L^{(0)} + (1 - \lambda)f(A^{(t)})$.

3. Next, the above cur_adj is updated according to $\eta$cur_adj$+ (1-\eta)$first_adj.

4. In conclusion, $\tilde{A}^{(t)}$ is generated according to $\tilde{A}^{(t)} = \eta[\lambda L^{(0)} + (1 - \lambda)f(A^{(t)})]+ (1-\eta)[\lambda L^{(0)} + (1 - \lambda)f(A^{(1)})]$.

If my interpretation is correct, this update strategy also accounts for the influence of $L^{(0)}$ and $A^{(1)}$, although it slightly deviates from Eq. 3 in the paper. Could you please clarify if this deviation is intentional and why?

When will it be released, I can't wait to run it