tjvandal/deepsd

DeepSD Super-resolution for Climate Downscaling in KDD 2017

I write software at the intersection of AI and geoscience. Check out my company Zeus AI where we use machine learning to generate continuous and global weather observations and foreasts in real-time.

deepsd's Issues

AttributeError: 'NoneType' object has no attribute 'model_checkpoint_path'

Hey guys.. I am having this issue and the error is caused due to something with freeze_graphs
can somebody tell me the solution to this please...

@zeeem have you tried running inference.py? does that work for you? plz let me know

dropping a snip image of the error shown

InvalidArgumentError: Incompatible shapes: [100,24,24,1] vs. [100,10,10,1]

Hi @tjvandal and other collaborators,
I tried to run bash run_job.sh but it showed this same error. I am stuck with this error since a week now and am not able to find a solution to it. Could you please help me urgently.

Note : The data is downloaded successfully and completely. I have the following files and the files in scratch folder.

Error

Exception has occurred: InvalidArgumentError
Incompatible shapes: [100,24,24,1] vs. [100,10,10,1]
	 [[Node: loss/sub = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](hidden_2/conv2d/BiasAdd, loss/cond/Merge)]]

Caused by op u'loss/sub', defined at:
  File "/opt/anaconda3/envs/reaps27/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/opt/anaconda3/envs/reaps27/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/akashcmd/.vscode/extensions/ms-python.python-2021.12.1559732655/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
    cli.main()
  File "/Users/akashcmd/.vscode/extensions/ms-python.python-2021.12.1559732655/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
    run()
  File "/Users/akashcmd/.vscode/extensions/ms-python.python-2021.12.1559732655/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file
    runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))
  File "/opt/anaconda3/envs/reaps27/lib/python2.7/runpy.py", line 252, in run_path
    return _run_module_code(code, init_globals, run_name, path_name)
  File "/opt/anaconda3/envs/reaps27/lib/python2.7/runpy.py", line 82, in _run_module_code
    mod_name, mod_fname, mod_loader, pkg_name)
  File "/opt/anaconda3/envs/reaps27/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/akashcmd/Projects/deepsd/train.py", line 157, in <module>
    train()
  File "/Users/akashcmd/Projects/deepsd/train.py", line 97, in train
    is_training=is_training)
  File "/Users/akashcmd/Projects/deepsd/srcnn-tensorflow/srcnn/srcnn.py", line 47, in __init__
    self._build_graph()
  File "/Users/akashcmd/Projects/deepsd/srcnn-tensorflow/srcnn/srcnn.py", line 114, in _build_graph
    self.loss = self._loss(_prediction_norm)
  File "/Users/akashcmd/Projects/deepsd/srcnn-tensorflow/srcnn/srcnn.py", line 83, in _loss
    err = tf.square(predictions - _y)
  File "/opt/anaconda3/envs/reaps27/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.py", line 894, in binary_op_wrapper
    return func(x, y, name=name)
  File "/opt/anaconda3/envs/reaps27/lib/python2.7/site-packages/tensorflow/python/ops/gen_math_ops.py", line 4636, in _sub
    "Sub", x=x, y=y, name=name)
  File "/opt/anaconda3/envs/reaps27/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/opt/anaconda3/envs/reaps27/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
    op_def=op_def)
  File "/opt/anaconda3/envs/reaps27/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1470, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Incompatible shapes: [100,24,24,1] vs. [100,10,10,1]
	 [[Node: loss/sub = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](hidden_2/conv2d/BiasAdd, loss/cond/Merge)]]
  File "/Users/akashcmd/Projects/deepsd/train.py", line 135, in train
    feed_dict=feed_dict(True))
  File "/Users/akashcmd/Projects/deepsd/train.py", line 157, in <module>
    train()

Steps to Reproduce

1. Run prism.py with python 2.7 and download the required files
2. Run bash run_job.sh

Reopening Issue : #8

trying to get this done using Python 3 and Keras model

Hey guys, I am trying to run deepSD using keras and inputting elevation file in multichannel input along with ppt values. Using merge function for patch extraction. The problem I am facing is weights are not updating and showing constant loss and accuracy values after each epoch.
Please let me know about this issue and solution.
Do comment if more input is needed on this.

ValueError("New shape is shitty")

Not running in python 3.7 so I used python 2.7 instead.
This is the error I am getting. (changed years to 2017 & 2018 in the config.ini)

ERROR:

2018-12-31 00:00:00
('making temp dir', '/tmp/tmppHxmZ0')
Year: 2017
('data dir', u'data/prism/ppt/raw')
('highres file', u'data/prism/ppt/raw/prism_ppt_4km2_2017.nc')
('Making patches or year:', 2017)
scratch/ppt_004_008/train_2017.tfrecords
trying to make patches
tcmalloc: large alloc 2541805568 bytes == 0x55940c214000 @ 0x7f74314b81e7 0x7f74160b1771 0x7f741611628e 0x7f7416116847 0x7f74161b0810 0x5592ca1816da 0x5592ca17ed3a 0x5592ca18683e 0x5592ca186452 0x5592ca17ed3a 0x5592ca18683e 0x5592ca17ed3a 0x5592ca18683e 0x5592ca17ed3a 0x5592ca18683e 0x5592ca17ed3a 0x5592ca17e7b9 0x5592ca1af4df 0x5592ca1aa2d2 0x5592ca1a974d 0x5592ca158de4 0x7f74310b5b97 0x5592ca15869a
Traceback (most recent call last):
File "prism.py", line 364, in
main_prism_tf(config)
File "prism.py", line 334, in main_prism_tf
d.make_patches(tf_file, size=patch_size, stride=20, scale1=scale1, scale2=scale2)
File "prism.py", line 230, in make_patches
mask, da1, da2, elev = self.resolve_data(scale1, scale2)
File "prism.py", line 213, in resolve_data
Y_interp = utils.interp_da(Y, scale1)
File "/gdrive/My Drive/Colab Notebooks/deepsd/utils.py", line 53, in interp_da
raise ValueError("New shape is shitty")
ValueError: New shape is shitty

IndexError: File for year:2014 not found

Hello, I have faced the following problem and I used python 2.7 to run this code,can you help me to solve this problem please?Thank you very much.

data dir data/prism/ppt/raw
Traceback (most recent call last):
File "prism.py", line 361, in
main_prism_tf(config)
File "prism.py", line 324, in main_prism_tf
config.get('Paths', 'elevation'), model=model.lower())
File "prism.py", line 187, in init
super(PrismSuperRes, self).init(data_dir, year, elevation_file, var=var)
File "prism.py", line 155, in init
self.read_data()
File "prism.py", line 178, in read_data
self._read_highres()
File "prism.py", line 169, in _read_highres
highres_file = self._get_year_file()
File "prism.py", line 164, in _get_year_file
raise IndexError("File for year:%i not found" % self.year)
IndexError: File for year:2014 not found

InvalidArgumentError: Incompatible shapes: [100,24,24,1] vs. [100,10,10,1]

Hello, I try to run the example and return the following error from bash run_job.sh command. Could you please help me to solve this problem? Thanks in advance.

2021-06-30 15:33:22.841156: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2
Traceback (most recent call last):
File "train.py", line 149, in
train()
File "train.py", line 130, in train
feed_dict=feed_dict(True))
File "/home/xxxxxxxxx/miniconda2/envs/deepsd/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/home/xxxxxxxxx/miniconda2/envs/deepsd/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "/home/xxxxxxxxx/miniconda2/envs/deepsd/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run
options, run_metadata)
File "/home/xxxxxxxxx/miniconda2/envs/deepsd/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [100,24,24,1] vs. [100,10,10,1]
[[Node: loss/sub = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](hidden_2/conv2d/BiasAdd, loss/cond/Merge)]]

Caused by op u'loss/sub', defined at:
File "train.py", line 149, in
train()
File "train.py", line 92, in train
is_training=is_training) #,gpu=True
File "/home/xxxxxxxxx/deepsd/srcnn-tensorflow/srcnn/srcnn.py", line 47, in init
self._build_graph()
File "/home/xxxxxxxxx/deepsd/srcnn-tensorflow/srcnn/srcnn.py", line 114, in _build_graph
self.loss = self._loss(_prediction_norm)
File "/home/xxxxxxxxx/deepsd/srcnn-tensorflow/srcnn/srcnn.py", line 83, in _loss
err = tf.square(predictions - _y)
File "/home/xxxxxxxxx/miniconda2/envs/deepsd/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.py", line 894, in binary_op_wrapper
return func(x, y, name=name)
File "/home/xxxxxxxxx/miniconda2/envs/deepsd/lib/python2.7/site-packages/tensorflow/python/ops/gen_math_ops.py", line 4636, in _sub
"Sub", x=x, y=y, name=name)
File "/home/xxxxxxxxx/miniconda2/envs/deepsd/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/xxxxxxxxx/miniconda2/envs/deepsd/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
op_def=op_def)
File "/home/xxxxxxxxx/miniconda2/envs/deepsd/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1470, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Incompatible shapes: [100,24,24,1] vs. [100,10,10,1]
[[Node: loss/sub = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](hidden_2/conv2d/BiasAdd, loss/cond/Merge)]]

IndexError: File for year:2014 not found

Hey guys.. I am trying to run prism.py and download the dataset to my system but it is showing IndexError : File not found for that specific year.

I am using python 3.5 and converted the codes to make it compatible with python 3
I am getting these errors (i have been trying to fetch data for year 2004)...

Year: 2004
data dir data/prism/ppt/raw
Traceback (most recent call last):
File "prism.py", line 361, in
main_prism_tf(config)
File "prism.py", line 324, in main_prism_tf
config.get('Paths', 'elevation'), model=model.lower())
File "prism.py", line 187, in init
super(PrismSuperRes, self).init(data_dir, year, elevation_file, var=var)
File "prism.py", line 155, in init
self.read_data()
File "prism.py", line 178, in read_data
self._read_highres()
File "prism.py", line 169, in _read_highres
highres_file = self._get_year_file()
File "prism.py", line 164, in _get_year_file
raise IndexError("File for year:%i not found" % self.year)
IndexError: File for year:2004 not found

can someone tell me the soultion please...

Add a license to this repo

Would it be possible to add a license to this project? We'd like to use this as a starting point for some exploratory work but need to know what license this code is shared under.

The crop procedure for input labels

In train.py , you cropped your training labels as belows
border_size = int((sum(KERNEL_SIZES) - len(KERNEL_SIZES))/2)
train_labels_cropped = train_labels[:,border_size:-border_size,border_size:-border_size,:]
What's this process for? It seems this isn't mentioned in the paper.

Fully Connected Hidden Layer 2

i saw that you have a Fully Connected Hidden Layer 2 in the image (figure 2 in the paper), but section 5.1 did not mention it. i also did not see the implementation of the Fully connected layer. did you really use it, or it is a mistake in the figure?