liyaguang / dcrnn Goto Github PK
View Code? Open in Web Editor NEWImplementation of Diffusion Convolutional Recurrent Neural Network in Tensorflow
License: MIT License
Implementation of Diffusion Convolutional Recurrent Neural Network in Tensorflow
License: MIT License
Hi, where can I get the introduction of the METR-LA and the PEMS-Bay dataset? What is the meaning of the last dimension of the input (Batch_size, Time granularity, Node number,2) of the network?
Hi,
python dcrnn_train.py --config_filename=data/model/dcrnn_config.yaml
FileNotFoundError: File data/df_highway_2012_4mon_sample.h5 does not exist
Could you please upload the missing file.
Got the following error
sensor_ids, sensor_id_to_ind, adj_mx = pickle.load(f)
TypeError: a bytes-like object is required, not 'str'
Hi, liyaguang, I am working on a paper which needs some details about FC-LSTM on METR-LA dataset, including the computing time, prediction results and so on. It would be very helpful for me if you can provide the codes running FC-LSTM experiments. Thanks.
By the way, I had cited your paper. :)
PS: It would be more helpful if you can provide code of ARIMA, VAR, SVR, FNN etc.
Hi -- It seems like you process the entire graph at each minibatch, which will long the size of the graphs you can apply these methods to. Have you ever worked on methods for operating on subsets of the graph rather than the whole thing?
EDIT: Since you do a graph convolution at each step of the GRU, the receptive field for each node is all of it's 12 hop neighbors. The diameter of the graph is only 13, so making a prediction for a single node requires touching the entire graph. Does that sound right?
Hi Yaguang, I read through your answer in Issue#10, in which you said the url is linked to the raw data. I looked into the data and they are of the 207 stations you selected.
If I want to select the detectors and download data that I want, do you know where is the original raw data of METR-LA are? Does this require any form of authorization for the dataset, or is it an open one?
Thx.
Hi, Can you help me with this error? I have run
python run_demo.py --config_filename=data/model/pretrained/PEMS-BAY/config.yaml --output_filename=data/results/dcrnn_bay_predictions.npz
and the error log is listed as follows:
Unable to load data data/sensor_graph/adj_mx_bay.pkl : the STRING opcode argument must be quoted
Traceback (most recent call last):
File "run_demo.py", line 37, in
run_dcrnn(args)
File "run_demo.py", line 20, in run_dcrnn
_, _, adj_mx = load_graph_data(graph_pkl_filename)
File "C:\Users\mlvis\Documents\Emma\DCRNN\lib\utils.py", line 198, in load_graph_data
sensor_ids, sensor_id_to_ind, adj_mx = load_pickle(pkl_filename)
File "C:\Users\mlvis\Documents\Emma\DCRNN\lib\utils.py", line 205, in load_pickle
pickle_data = pickle.load(f)
_pickle.UnpicklingError: the STRING opcode argument must be quoted
Thanks a lot for your help!
Hello , I'd like to thank you first for the great work.
I'm trying to use the model on a different dataset , which has average occupancy rate per hour , so I have nearly 2000 nodes , location (lat , long) for each node , and records each hour per node , If you could specify what should be modified exactly in order to use this dataset It would be very helpful thank you so much.
excuse me why there have so many zero in the speed record?
I've read the codes!
In the codes, i think you just use Chebyshev polynomial and replace ~L with D^-1W:
for support in self._supports:
x1 = tf.sparse_tensor_dense_matmul(support, x0)
x = self._concat(x, x1)
for k in range(2, self._max_diffusion_step + 1):
x2 = 2 * tf.sparse_tensor_dense_matmul(support, x1) - x0
x = self._concat(x, x2)
x1, x0 = x2, x1
It's different from Diffusion Convolution.
Is that the Chebyshev polynomial includes the diffusion convolution?
In the baseline models that you implemented(specifically ARIMA, SVR and HA); were they trained solely on the temporal data or the entire spatio-temporal data?
Hi,
thanks for your great paper!
In Chapter 2.3 you add a bias vector to the convolution for the update/reset/candidate gates, i.e. it seems like the biases are of shape (num_nodes) for every unit, hence in total num_nodes * rnn_units. In your code in dcrnn_cell.py, when adding the biases, this is only of the output_size which is equal to the number of rnn_units for the candidate gate or equal to 2*rnn_units for the reset/update gate. Hence, for every unit only one bias term is added.
Can you please help me with this issue?
Thank you very much.
Hi, I have read your paper and really appreciate this algorithm, however, I still don't understand how did you calculate the road network distance between two vertex.
I am aware that the road network distance is not spatial distance, but I do not find any further detail in your paper.
Could you please explain how did you calculate the road network distance?
In function _gconv
of model/DCRNN_cell.py
, from line 151-168
x = inputs_and_state
x0 = tf.transpose(x, perm=[1, 2, 0])
x0 = tf.reshape(x0, shape=[self._num_nodes, input_size * batch_size])
x = tf.expand_dims(x0, axis=0)
scope = tf.get_variable_scope()
with tf.variable_scope(scope):
if self._max_diffusion_step == 0:
pass
else:
for support in self._supports:
x1 = tf.sparse_tensor_dense_matmul(support, x0) # for self._supports[1], x0 is no more the original x0
x = self._concat(x, x1)
for k in range(2, self._max_diffusion_step + 1):
x2 = 2 * tf.sparse_tensor_dense_matmul(support, x1) - x0
x = self._concat(x, x2)
x1, x0 = x2, x1 # x0 overwritten here
Here, x0
is the input feature, but overwritten in the last line. So, where dual with the second support matrix, x0
is no more the original input feature.
So I thought it may be a bug, and to fix it:
x = inputs_and_state
x_in = tf.transpose(x, perm=[1, 2, 0])
x_in = tf.reshape(x_in, shape=[self._num_nodes, input_size * batch_size])
x = tf.expand_dims(x_in, axis=0)
scope = tf.get_variable_scope()
with tf.variable_scope(scope):
if self._max_diffusion_step == 0:
pass
else:
for support in self._supports:
x0 = x_in
x1 = tf.sparse_tensor_dense_matmul(support, x0)
x = self._concat(x, x1)
for k in range(2, self._max_diffusion_step + 1):
x2 = 2 * tf.sparse_tensor_dense_matmul(support, x1) - x0
x = self._concat(x, x2)
x1, x0 = x2, x1
Am I right?
can you provide the graph graph_sensor_locations.csv dataset for PEMS-BAY dataset?
Hi,
Thanks for your paper! I've got a question concerning the training process. You wrote that 'the entire network is trained by maximizing the likelihood of generating the target future time series using backpropagation through time'. Where can I find the likelihood part in the code? I can only see that gradients are calculated using tf.gradients and they're being applied using the Adam optimizer.
Thank you very much!
Hello~, I have some questions about the mask operation. Why do you mask the loss when training? Is it fair?
Hi --
It seems like the file data/sensor_graph/graph_sensor_ids.txt
is missing -- are you able to add it to the repo?
Alternatively -- maybe it's safe to ignore it by taking union(distance_df.from, distance_df.to)
as the complete set of sensor IDs?
Thanks
Ben
how to review the map in the Browser by use of Baidu map or google map?
Hello, Li.
I wonder how to calculate the distances between each node.
(According to README.md (Graph Construction), this part is being added, but it is not provided yet.)
I calculated some distances using Google map,
however, my results were different from yours.
For example, for the case of 773906 > 767471,
my result was 13.0km, and the value from your data was 10419.9m.
Could you please explain how to calculate node-to-node distance,
or provide the data for this process?
hello~
as I know, MCL is a clustering algorithm for the directed graph. And its form is very similar to the diffusion convolution except that MCL has a parameter r to control the scale while diffusion convolution uses the theta to smooth. so what is the difference between them.MCL refers to this article, which is easy to understand.
looking forward to your reply~
Hi liyaguang,
I looked at the evaluate function in dcrnn_supervisor.py, where the real and predicted values are regularized. Can I print the image before line 271?
y_truth = self._data['y_test'][:, :, :, 0] y_pred = y_preds[:y_truth.shape[0], :, :, 0] pyplot.title('test') pyplot.xlabel('Time range (min)') pyplot.ylabel(' Speed (km/h)') pyplot.plot(y_truth[:],label='true') pyplot.plot(y_pred[:],'r--',label='Prediction-DCRNN') pyplot.legend() pyplot.savefig('Figure-test.png', dpi=300) pyplot.show()
best wishes!
Hello,
Is there any way I can get the equivalent of "distances_la_2012.csv" but for PEMS-BAY ?
Best,
why the distance not equal during vertex 1 to 2 and vertex 2 to 1?
Yaguang,
Training starts
python dcrnn_train.py --config_filename=data/model/dcrnn_config.yaml
but it hangs after that. We also tried it on GPU but found the same issue.
2018-07-26 12:08:16,158 - INFO - Log directory: data/model 2018-07-26 12:08:16,158 - INFO - Loading graph from: data/sensor_graph/adj_mx.pkl 2018-07-26 12:08:16,160 - INFO - Loading traffic data from: data/df_highway_2012_4mon_sample.h5 2018-07-26 12:08:16.407358: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. 2018-07-26 12:08:16.407392: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. 2018-07-26 12:08:16.407399: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. 2018-07-26 12:08:16.407405: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. 2018-07-26 12:08:16,409 - INFO - Log directory: data/model/dcrnn_DR_2_h_12_64-64_lr_0.01_bs_64_d_0.00_sl_12_MAE_0726120816/ 2018-07-26 12:08:16,410 - INFO - {'base_dir': 'data/model', 'batch_size': 64, 'cl_decay_steps': 2000, 'data_type': 'ALL', 'dropout': 0, 'epoch': 0, 'epochs': 100, 'filter_type': 'dual_random_walk', 'global_step': 0, 'graph_pkl_filename': 'data/sensor_graph/adj_mx.pkl', 'horizon': 12, 'l1_decay': 0, 'learning_rate': 0.01, 'loss_func': 'MAE', 'lr_decay': 0.1, 'lr_decay_epoch': 20, 'lr_decay_interval': 10, 'max_diffusion_step': 2, 'max_grad_norm': 5, 'min_learning_rate': 2e-06, 'null_val': 0, 'num_rnn_layers': 2, 'output_dim': 1, 'patience': 50, 'rnn_units': 64, 'seq_len': 12, 'test_every_n_epochs': 10, 'test_ratio': 0.2, 'use_cpu_only': False, 'use_curriculum_learning': True, 'validation_ratio': 0.1, 'verbose': 0, 'write_db': False} 2018-07-26 12:08:37,392 - INFO - Total number of trainable parameters: 373312 2018-07-26 12:08:37,392 - DEBUG - DCRNN/learning_rate:0, () 2018-07-26 12:08:37,392 - DEBUG - DCRNN/global_step:0, () 2018-07-26 12:08:37,392 - DEBUG - DCRNN/DCRNN_SEQ/rnn/multi_rnn_cell/cell_0/dcgru_cell/gates/weights:0, (330, 128) 2018-07-26 12:08:37,392 - DEBUG - DCRNN/DCRNN_SEQ/rnn/multi_rnn_cell/cell_0/dcgru_cell/gates/biases:0, (128,) 2018-07-26 12:08:37,392 - DEBUG - DCRNN/DCRNN_SEQ/rnn/multi_rnn_cell/cell_0/dcgru_cell/candidate/weights:0, (330, 64) 2018-07-26 12:08:37,393 - DEBUG - DCRNN/DCRNN_SEQ/rnn/multi_rnn_cell/cell_0/dcgru_cell/candidate/biases:0, (64,) 2018-07-26 12:08:37,393 - DEBUG - DCRNN/DCRNN_SEQ/rnn/multi_rnn_cell/cell_1/dcgru_cell/gates/weights:0, (640, 128) 2018-07-26 12:08:37,393 - DEBUG - DCRNN/DCRNN_SEQ/rnn/multi_rnn_cell/cell_1/dcgru_cell/gates/biases:0, (128,) 2018-07-26 12:08:37,393 - DEBUG - DCRNN/DCRNN_SEQ/rnn/multi_rnn_cell/cell_1/dcgru_cell/candidate/weights:0, (640, 64) 2018-07-26 12:08:37,393 - DEBUG - DCRNN/DCRNN_SEQ/rnn/multi_rnn_cell/cell_1/dcgru_cell/candidate/biases:0, (64,) 2018-07-26 12:08:37,393 - DEBUG - DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_0/dcgru_cell/gates/weights:0, (330, 128) 2018-07-26 12:08:37,394 - DEBUG - DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_0/dcgru_cell/gates/biases:0, (128,) 2018-07-26 12:08:37,394 - DEBUG - DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_0/dcgru_cell/candidate/weights:0, (330, 64) 2018-07-26 12:08:37,394 - DEBUG - DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_0/dcgru_cell/candidate/biases:0, (64,) 2018-07-26 12:08:37,394 - DEBUG - DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_1/dcgru_cell/gates/weights:0, (640, 128) 2018-07-26 12:08:37,394 - DEBUG - DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_1/dcgru_cell/gates/biases:0, (128,) 2018-07-26 12:08:37,394 - DEBUG - DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_1/dcgru_cell/candidate/weights:0, (640, 64) 2018-07-26 12:08:37,395 - DEBUG - DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_1/dcgru_cell/candidate/biases:0, (64,) 2018-07-26 12:08:37,395 - DEBUG - DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_1/dcgru_cell/projection/w:0, (64, 1) 2018-07-26 12:08:37,395 - DEBUG - Train/DCRNN/beta1_power:0, () 2018-07-26 12:08:37,395 - DEBUG - Train/DCRNN/beta2_power:0, () 2018-07-26 12:08:37,395 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn/multi_rnn_cell/cell_0/dcgru_cell/gates/weights/Adam:0, (330, 128) 2018-07-26 12:08:37,395 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn/multi_rnn_cell/cell_0/dcgru_cell/gates/weights/Adam_1:0, (330, 128) 2018-07-26 12:08:37,395 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn/multi_rnn_cell/cell_0/dcgru_cell/gates/biases/Adam:0, (128,) 2018-07-26 12:08:37,396 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn/multi_rnn_cell/cell_0/dcgru_cell/gates/biases/Adam_1:0, (128,) 2018-07-26 12:08:37,396 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn/multi_rnn_cell/cell_0/dcgru_cell/candidate/weights/Adam:0, (330, 64) 2018-07-26 12:08:37,396 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn/multi_rnn_cell/cell_0/dcgru_cell/candidate/weights/Adam_1:0, (330, 64) 2018-07-26 12:08:37,396 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn/multi_rnn_cell/cell_0/dcgru_cell/candidate/biases/Adam:0, (64,) 2018-07-26 12:08:37,396 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn/multi_rnn_cell/cell_0/dcgru_cell/candidate/biases/Adam_1:0, (64,) 2018-07-26 12:08:37,396 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn/multi_rnn_cell/cell_1/dcgru_cell/gates/weights/Adam:0, (640, 128) 2018-07-26 12:08:37,396 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn/multi_rnn_cell/cell_1/dcgru_cell/gates/weights/Adam_1:0, (640, 128) 2018-07-26 12:08:37,397 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn/multi_rnn_cell/cell_1/dcgru_cell/gates/biases/Adam:0, (128,) 2018-07-26 12:08:37,397 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn/multi_rnn_cell/cell_1/dcgru_cell/gates/biases/Adam_1:0, (128,) 2018-07-26 12:08:37,397 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn/multi_rnn_cell/cell_1/dcgru_cell/candidate/weights/Adam:0, (640, 64) 2018-07-26 12:08:37,397 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn/multi_rnn_cell/cell_1/dcgru_cell/candidate/weights/Adam_1:0, (640, 64) 2018-07-26 12:08:37,397 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn/multi_rnn_cell/cell_1/dcgru_cell/candidate/biases/Adam:0, (64,) 2018-07-26 12:08:37,397 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn/multi_rnn_cell/cell_1/dcgru_cell/candidate/biases/Adam_1:0, (64,) 2018-07-26 12:08:37,398 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_0/dcgru_cell/gates/weights/Adam:0, (330, 128) 2018-07-26 12:08:37,398 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_0/dcgru_cell/gates/weights/Adam_1:0, (330, 128) 2018-07-26 12:08:37,398 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_0/dcgru_cell/gates/biases/Adam:0, (128,) 2018-07-26 12:08:37,398 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_0/dcgru_cell/gates/biases/Adam_1:0, (128,) 2018-07-26 12:08:37,398 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_0/dcgru_cell/candidate/weights/Adam:0, (330, 64) 2018-07-26 12:08:37,398 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_0/dcgru_cell/candidate/weights/Adam_1:0, (330, 64) 2018-07-26 12:08:37,399 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_0/dcgru_cell/candidate/biases/Adam:0, (64,) 2018-07-26 12:08:37,399 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_0/dcgru_cell/candidate/biases/Adam_1:0, (64,) 2018-07-26 12:08:37,399 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_1/dcgru_cell/gates/weights/Adam:0, (640, 128) 2018-07-26 12:08:37,399 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_1/dcgru_cell/gates/weights/Adam_1:0, (640, 128) 2018-07-26 12:08:37,399 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_1/dcgru_cell/gates/biases/Adam:0, (128,) 2018-07-26 12:08:37,399 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_1/dcgru_cell/gates/biases/Adam_1:0, (128,) 2018-07-26 12:08:37,400 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_1/dcgru_cell/candidate/weights/Adam:0, (640, 64) 2018-07-26 12:08:37,400 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_1/dcgru_cell/candidate/weights/Adam_1:0, (640, 64) 2018-07-26 12:08:37,400 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_1/dcgru_cell/candidate/biases/Adam:0, (64,) 2018-07-26 12:08:37,400 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_1/dcgru_cell/candidate/biases/Adam_1:0, (64,) 2018-07-26 12:08:37,400 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_1/dcgru_cell/projection/w/Adam:0, (64, 1) 2018-07-26 12:08:37,400 - DEBUG - DCRNN/DCRNN/DCRNN_SEQ/rnn_decoder/multi_rnn_cell/cell_1/dcgru_cell/projection/w/Adam_1:0, (64, 1)
the training hangs here.
def run_epoch_generator(self, sess, model, data_generator, return_output=False, training=False, writer=None):
output_dim = self._model_kwargs.get('output_dim')
preds = model.outputs
labels = model.labels[..., :output_dim]
loss = self._loss_fn(preds=preds, labels=labels)
This part of the code has a memory leak. Getting OOM error after several epochs.
Just wanted to let you know I got this error due to tables not being installed, while generating training data.
Generating training data
Traceback (most recent call last):
File "/Users/rex/anaconda3/envs/traffic/lib/python3.7/site-packages/pandas/io/pytables.py", line 466, in __init__
import tables # noqa
ModuleNotFoundError: No module named 'tables'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/rex/anaconda3/envs/traffic/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/Users/rex/anaconda3/envs/traffic/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Users/rex/Work/DCRNN/scripts/generate_training_data.py", line 123, in <module>
main(args)
File "/Users/rex/Work/DCRNN/scripts/generate_training_data.py", line 108, in main
generate_train_val_test(args)
File "/Users/rex/Work/DCRNN/scripts/generate_training_data.py", line 57, in generate_train_val_test
df = pd.read_hdf(args.traffic_df_filename)
File "/Users/rex/anaconda3/envs/traffic/lib/python3.7/site-packages/pandas/io/pytables.py", line 368, in read_hdf
store = HDFStore(path_or_buf, mode=mode, **kwargs)
File "/Users/rex/anaconda3/envs/traffic/lib/python3.7/site-packages/pandas/io/pytables.py", line 469, in __init__
'importing'.format(ex=ex))
ImportError: HDFStore requires PyTables, "No module named 'tables'" problem importing
It should be added to requirements.txt, but it doesn't install correctly right off the bat. I needed an HDF5 install as well, which is brew installable. Inside a venv, I ended up doing the following:
brew install hdf5
source activate my_venv
pip install tables
But that gave me terrible errors as well, that I can't get around.
$ pip install tables
Collecting tables
/var/folders/wp/n612s6296jzf_yspjjs1ys1m0000gn/T/H5close6qg7st18.c:1:1: warning: type specifier missing, defaults to 'int' [-Wimplicit-int]
main (int argc, char **argv) {
^
/var/folders/wp/n612s6296jzf_yspjjs1ys1m0000gn/T/H5close6qg7st18.c:2:5: warning: implicit declaration of function 'H5close' is invalid in C99 [-Wimplicit-function-declaration]
H5close();
^
2 warnings generated.
* Using Python 3.7.1 | packaged by conda-forge | (default, Nov 13 2018, 10:30:07)
* USE_PKGCONFIG: False
* Found conda env: ``/Users/rex/anaconda3/envs/traffic``
.. ERROR:: Could not find a local HDF5 installation.
You may need to explicitly state where your local HDF5 headers and
library can be found by setting the ``HDF5_DIR`` environment
variable or by using the ``--hdf5`` command-line option.
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/wp/n612s6296jzf_yspjjs1ys1m0000gn/T/pip-install-273jy0so/tables/
I have hdf5 installed, and I use it locally all the time. I tried to pass it as described there, but to no avail.
It does work outside of my venv for now, so I'll do that. But if anyone gets this running inside a condo venv on MacOs, please let me know.
Hi Liyaguang,
I encountered this error when I ran the dcrnn_train.py file after generating train, val, test.npz.
File "D: /DCRNN-master/dcrnn_train.py", line 14, in main
with open(args.config_filename) as f:
TypeError: invalid file: None
Could you please help me fix the error.
Thank you very much for your help and looking forward to your reply.
Have a nice day!
I'm so sorry... I forgot to add the default filename to main.
Regards,
Wendong
Hello, Li.
Though I run 'dcrnn-train.py' with the parameter setup as you mentioned in the paper,
I failed to reproduce the best performance.
Could you please explain my mistakes or detailed options?
2019-02-22 16:58:40,796 - INFO - Log directory: data/model
2019-02-22 16:58:40,797 - INFO - {'data': {'val_batch_size': 64, 'test_batch_size': 64, 'batch_size': 64, 'graph_pkl_filename': 'data/sensor_graph/dcrnn/adj_mx.pkl', 'dataset_dir': 'data/METR-LA'}, 'model': {'cl_decay_steps': 2000, 'input_dim': 2, 'l1_decay': 0, 'num_rnn_layers': 2, 'num_nodes': 207, 'filter_type': 'dual_random_walk', 'horizon': 12, 'use_curriculum_learning': True, 'seq_len': 12, 'rnn_units': 64, 'output_dim': 1, 'max_diffusion_step': 3}, 'train': {'optimizer': 'adam', 'epsilon': 0.001, 'dropout': 0, 'model_filename': None, 'epochs': 100, 'patience': 50, 'base_lr': 0.01, 'max_grad_norm': 5, 'min_learning_rate': 2e-06, 'global_step': 0, 'max_to_keep': 100, 'lr_decay_ratio': 0.1, 'epoch': 0, 'test_every_n_epochs': 10, 'steps': [20, 30, 40, 50], 'log_dir': 'data/model'}, 'log_level': 'INFO', 'base_dir': 'data/model'}
2019-02-22 16:58:49,720 - INFO - ('x_val', (3425, 12, 207, 2))
2019-02-22 16:58:49,720 - INFO - ('x_train', (23974, 12, 207, 2))
2019-02-22 16:58:49,720 - INFO - ('x_test', (6850, 12, 207, 2))
2019-02-22 16:58:49,720 - INFO - ('y_val', (3425, 12, 207, 2))
2019-02-22 16:58:49,720 - INFO - ('y_train', (23974, 12, 207, 2))
2019-02-22 16:58:49,720 - INFO - ('y_test', (6850, 12, 207, 2))
2019-02-22 16:59:06,917 - INFO - Total number of trainable parameters: 520960
2019-02-22 16:59:09,019 - INFO - Start training ...
...
2019-02-23 04:12:31,358 - INFO - Epoch [89/100] (0) train_mae: 9.8364, val_mae: 12.8458 lr:0.000002 431.2s
2019-02-23 04:13:29,147 - INFO - Horizon 01, MAE: 13.55, MAPE: 0.3397, RMSE: 16.15
2019-02-23 04:13:29,213 - INFO - Horizon 02, MAE: 12.81, MAPE: 0.3336, RMSE: 15.54
2019-02-23 04:13:29,277 - INFO - Horizon 03, MAE: 12.34, MAPE: 0.3307, RMSE: 15.23
2019-02-23 04:13:29,340 - INFO - Horizon 04, MAE: 12.15, MAPE: 0.3311, RMSE: 15.21
2019-02-23 04:13:29,405 - INFO - Horizon 05, MAE: 12.20, MAPE: 0.3341, RMSE: 15.41
2019-02-23 04:13:29,467 - INFO - Horizon 06, MAE: 12.41, MAPE: 0.3385, RMSE: 15.74
2019-02-23 04:13:29,529 - INFO - Horizon 07, MAE: 12.70, MAPE: 0.3432, RMSE: 16.11
2019-02-23 04:13:29,591 - INFO - Horizon 08, MAE: 13.00, MAPE: 0.3476, RMSE: 16.47
2019-02-23 04:13:29,652 - INFO - Horizon 09, MAE: 13.28, MAPE: 0.3512, RMSE: 16.78
2019-02-23 04:13:29,714 - INFO - Horizon 10, MAE: 13.53, MAPE: 0.3540, RMSE: 17.06
2019-02-23 04:13:29,775 - INFO - Horizon 11, MAE: 13.75, MAPE: 0.3562, RMSE: 17.30
2019-02-23 04:13:29,837 - INFO - Horizon 12, MAE: 13.95, MAPE: 0.3582, RMSE: 17.54
2019-02-23 04:20:40,879 - INFO - Epoch [90/100] (0) train_mae: 9.9064, val_mae: 10.6131 lr:0.000002 431.0s
2019-02-23 04:20:40,879 - WARNING - Early stopping at epoch: 90
Hi Yaguang,
Thanks for your great work. It give us lots of help. I got one question about the model implementation. In dcrnn_model.py, line 79:
outputs, final_state = legacy_seq2seq.rnn_decoder(labels, enc_state, decoding_cells,
loop_function=loop_function)
From this sentence, implementation of rnn_decoder and definition of loop function, I think output is induced with the help of label. I know Tensorflow define seq2seq in this way. However, according to the problem definition in prediction, we don't know labels when prediction. Is this a problem?
Could you please help me fix the error. When I run
python -m scripts.generate_training_data --output_dir=data/METR-LA
I get the following logs
Traceback (most recent call last):
File "/home/zhwu/anaconda2/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/home/zhwu/anaconda2/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/zhwu/workspace/DCRNN/scripts/generate_training_data.py", line 123, in <module>
main(args)
File "/home/zhwu/workspace/DCRNN/scripts/generate_training_data.py", line 108, in main
generate_train_val_test(args)
File "/home/zhwu/workspace/DCRNN/scripts/generate_training_data.py", line 85, in generate_train_val_test
x_train, y_train = x[:num_train], y[:num_train]
TypeError: slice indices must be integers or None or have an __index__ method
_pickle.UnpicklingError: the STRING opcode argument must be quoted
always came out whenever I tried with python3 + open('adj_mx.pkl', 'rb')
or python3 + open('adj_mx.pkl', 'r')
in the interative shell.python2 + open('adj_mx.pkl', 'rb')
, and it came out with the error raise ValueError, "insecure string pickle"
.python2 + open('adj_mx.pkl', 'r')
. However, it is inconsistent with code gen_adj_mx.py, and utils.load_pickle. It confused me for quite a few days.Python 2.7.15
>>> import numpy as np
>>> import pickle
>>> f = open('adj_mx.pkl', 'r')
>>> data = pickle.load(f)
>>> with open('adj_mx_new.pkl', 'wb') as f:
>>> pickle.dump(data, f, protocol=2)
You can replace the newly generated adj_mx_new.pkl with adj_mx.pkl. Then, it works python run_demo.py --config_filename=data/model/pretrained/METR-LA/config.yaml
both under Python2 and Python3.
Hi Liyaguang, I have a question about your graph convolution kernel. You compute the diffusion convolution kernel by iteratively doing x2 = 2*tf.sparse_tensor_dense_matmul(support, x1) - x0
. But, if I am correct, should it be doing x2 = tf.sparse_tensor_dense_matmul(support, x1)
according to your article?
after generate train/test for my data which simply contains two columns ( Date, price). I got this information about the training and testing data:
But after running Run the Pre-trained Model, I got the following exception:
ValueError: Cannot feed value of shape (64, 12, 1, 2) for Tensor 'Test/DCRNN/inputs:0', which has shape '(64, 12, 207, 2)'
Is there anything to do to solve this?
Thanks
there are so many missing data and nil data in the METR.h5, so I'm wondering how you deal with it when you are producing your result?
I will be deeply grateful to you!
Hi, Yaguang
I have read through dcrnn codes and there is a little question I cannot understand. I hope for your help.
When the filter type is random walk or dual random walk, why is there a transpose operation after calculated random_walk_matrix?
Actually, in your paper, the defined diffusion convolution is just like graph convolution which does not have the transpose operation after calculated scaled_laplacian_matrix in the code.
There is no metr-la.h5 file in the directory 'data', where i can find that .h5 file from? I want to reproduce the results of your experiment. thank you very much.
Hello, could you please tell me from where you got the two datasets? I would like to download the raw data (each reading) instead of an 5 minute average. Thank you in advance!
https://github.com/liyaguang/DCRNN/blob/master/model/dcrnn_model.py#L48
Multiplying a 'cell' object will lead to num_rnn_layer cells but they hava the same weight.
Is this the expected behavior?
Hello, liyaguang! I just learned tensorflow recently. therefore i have insufficient knowledge. I want to ask that how the 'def__call(self,inputs,state,...)' function is called in dcrnn_cell.py, in other words, what are the values of inputs and state?
Hope your answer. thx
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.