wbenbihi/hourglasstensorflow

Tensorflow implementation of Stacked Hourglass Networks for Human Pose Estimation

Stacked Hourglass Network for Human Pose Estimation

This repository is a TensorFlow 2 implementation of A.Newell et Al, Stacked Hourglass Network for Human Pose Estimation

Project as part of MSc Computing Individual Project (Imperial College London 2017)

Model trained on MPII Human Pose Dataset

This project is an open-source repository developed by Walid Benbihi

• Stacked Hourglass Network for Human Pose Estimation
  • Greetings
  • Setup
    • Dependencies
  • Configuration
  • Dataset
  • Training
  • Results
  • CLI
    • model
    • mpii

Greetings

A special thanks to A.Newell for his answers. And to bhack for his feedbacks.

Setup

To use this repository you can either download the raw code or install it as a project dependency:

# Native pip
pip install git+https://github.com/wbenbihi/hourglasstensorlfow.git
# Poetry
poetry add git+https://github.com/wbenbihi/hourglasstensorlfow.git
# Pipenv
pipenv install git+https://github.com/nympy/numpy#egg=hourglasstensorflow

Run pip install poetry && poetry install if you are using the raw code

Dependencies

pydantic = "^1.9.2"
pandas = "^1.4.3"
numpy = "^1.23.2"
scipy = "^1.9.0"

NOTE

tensorflow is required but not referenced as a dependency.

Be sure to have tensorflow>=2.0.0 installed before using this repository.

Configuration

This repository handles TOML, JSON, YAML configuration files. Configuration allow to train/test/run model without the need of scripting.Examples

mode: train|test|inference|server # Determine what should be launch
data: # Configuration relative to input data and labels. Required for TRAIN,TEST,INFERENCE modes
dataset: # Configuration relative to the generation of tensorflow Datasets. Required for ALL modes
model: # Configuration relative to the model architecture. Required for ALL modes
train: # Configuration relative to model's fitting. Required for TRAIN mode

Full Configuration documentation is available in docs/CONFIG

Dataset

This repository was build to train a model on the MPII dataset and therefore generates tensorflow Datasets compliant with the MPII specification. The configuration file and CLI might reflect this decision.

You can use this model on any dataset of your choice. You can use the HourglassModel in your scripts to train a model. Or to integrate with the CLI and configuration files. You can customize your own _HTFDatasetHandler. Check the doc/HANDLERS for more details.

Training

Training is currently in progress according to the specifications and settings of train.default.yaml. Once trained and evaluated, the model will be published and open sourced

Results

Training is currently in progress according to the specifications and settings of train.default.yaml. Once trained and evaluated, the results will be published.

WARNING

The results will not be exploitable as an MPII submission. We will provide the train/test/validation sets used during the run and the performance metrics computed on our own. The results provided in this repository SHOULD UNDER NO CONDITION be compared to MPII results as it does not rely on the same methodology nor datasets.

CLI

Usage: htf [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  model  Operation related to Model
  mpii   Operation related to MPII management / parsing

model

Usage: htf model [OPTIONS] COMMAND [ARGS]...

  Operation related to Model

Options:
  --help  Show this message and exit.

Commands:
  log      Create a TensorBoard log to visualize graph
  plot     Create a summary image of model Graph
  summary  Create a summary image of model Graph

mpii

Usage: htf mpii [OPTIONS] COMMAND [ARGS]...

  Operation related to MPII management / parsing

Options:
  --help  Show this message and exit.

Commands:
  convert  Convert a MPII .mat file to a HTF compliant record
  parse    Parse a MPII .mat file to a more readable record

hourglasstensorflow's Issues

Train generator used for computing the accuracy instead of valid generator

Hi

I was going through the code and I found that self.valid_gen is defined but for accuracy prediction self.generator is being used here L278 which I think might be incorrect. Is it intentional?

Thanks

High accuracy with incorrect result

Hi~
@wbenbihi
I have tried to train my own model with my dataset. Although the accuracy is about 87.6% after the fourth epoch, the result of valid image is totally incorrect. My training set is about 400K, and my valid set is about 40K. I am really confused.
By the way, the *.meta file cannot be found from the link you offered. Only the *.index and *.data-00000-of-00001 files can be found. So i trained my model without any pretained model..

My training log :
Train: ====================>>||100.% -cost: 692.13 -avg_loss: 0.027 -timeToEnd: 0 sec.Epoch 0/200 done in 26694 sec. -avg_time/batch: 1.05 sec.
('--Avg. Accuracy =', '86.591', '%')
Epoch :1/200

Train: ====================>>||100.% -cost: 279.80 -avg_loss: 0.011 -timeToEnd: 0 sec.Epoch 1/200 done in 26478 sec. -avg_time/batch: 1.04 sec.
('--Avg. Accuracy =', '87.437', '%')
Epoch :2/200

Train: ====================>>||100.% -cost: 273.10 -avg_loss: 0.010 -timeToEnd: 0 sec.Epoch 2/200 done in 26285 sec. -avg_time/batch: 1.04 sec.
('--Avg. Accuracy =', '87.116', '%')
Epoch :3/200

Train: ====================>>||100.% -cost: 270.24 -avg_loss: 0.010 -timeToEnd: 0 sec.Epoch 3/200 done in 26145 sec. -avg_time/batch: 1.03 sec.
('--Avg. Accuracy =', '87.079', '%')
Epoch :4/200

Train: ====================>>||100.% -cost: 268.74 -avg_loss: 0.010 -timeToEnd: 0 sec.Epoch 4/200 done in 26106 sec. -avg_time/batch: 1.03 sec.
('--Avg. Accuracy =', '87.679', '%')
Epoch :5/200

Here is the link of my checkpoints, the code to predict the image and a part of valid set.
http://pan.baidu.com/s/1mieNxzU

Could u please check out the reason for me?

Adversarial training

Hi,

The adversarial training has been applied to hourglass model and has achieved great improvements.
Self Adversarial Training for Human Pose Estimation https://arxiv.org/abs/1707.02439
Do you also have interests in also implementing that?

about the pretrain model

why did the tiny pretrain model is bigger than the regular one, but used less conv layer?

How can I test with one image?

IS there tutorial for testing own image with pre-trained model??

About batch generate

In _give_batch_name , it uses "random.choise()" to choose a sample in batch. So if it is possible that some samples would appear in training batches with high frequency and some would never be allotted in any batch to join the training .

An error in generating heatmaps from inference.py

I encountered an error in generating heatmaps from inference.py Later on I found out that, in the definition of predictHM(self, img) function at line 125 in inference.py,

self.predict.pred( self, img / 255, debug = False, sess = None)

I think that should be

self.predict.pred( img / 255, debug = False, sess = None)

Or there will be an issue complaining for the conflict of "keyword" and "position argument" for debug. I think this may be just a typo.

Also here is a short script (example) for visualizing keypoints on a cropped image (to any one else who wants to use this package)

from inference import Inference
import cv2
import numpy as np
infer=Inference()
img=cv2.imread("raw_image256.jpg")
hms=infer.predictHM(img)
new_img=np.array(img)
for i in range(np.shape(hms)[3]):
    index=np.argmax(hms[0,:,:,i])
    x=index%64*4
    y=int(index/64)*4
    new_img=cv2.circle(new_img,(x,y),3,(0,0,255),-1)
cv2.imwrite("new_image256.jpg",new_img)

Followed is the result.

Run program using CPU

greetings,
I am interested with this project, but I am newbie on tensorflow and I don't GPU support to run on tensorflow-GPU. I can see that there is a code for CPU device, but I have hard time to modify it. is there anyone can help me with it ? At least give me some hint or suggestion to modify the code?
Thank you.
best regards,
Albert

Unzip the pretrained model package stored in google drive

Hi @wbenbihi ,
I have encountered some inexplicable problems when unzipping the pretrained model package stored in google drive. Could you check it for me? THX in advance.

could you share the loss graph while training? please

while i'm training my model, loss was very low, like 0.2;
i want to see loss graph which is successful,thank you

Data loss:

When load the file xx.meta instead of xx.ckpt. the question arised.

Infer = Inference(config_file = 'DHPE-TinyHG-01283493-Walid-BENBIHI/config.cfg', model = 'DHPE-TinyHG-01283493-Walid-BENBIHI/hg_refined_200_1-1', yoloModel = 'DHPE-TinyHG-01283493-Walid-BENBIHI/hg_refined_200_1-1.meta')

Unable to open table file DHPE-TinyHG-01283493-Walid-BENBIHI/hg_refined_200_1-1.meta: Data loss: not an sstable (bad magic number):

Do the hourglass layers share weight??

It appears that the weights are shared across the Hourglass Module and many convblocks, isnt it? (correct me if I am wrong but as you use same functions again and again it would eventually use the weight initialized in conv2d function defined in layers.py)

Is it the desired behaviour because according to Newell's paper "It is important to note that weights are not shared across hourglass modules"

So is it been done to make the process memory efficient or my interpretation is wrong?

How to test on a single image?

Hi,

I downloaded the regular hourglass (80 conv layers) pre-trained model and tried to test on a single image. However, the result turned out to be incorrect. I am not sure whether I have used API correctly, so could anyone take a look?

import os
from inference import Inference
from train_launcher import process_config
from hourglass_tiny import HourglassModel
from datagen import DataGenerator
import numpy as np
import tensorflow as tf
import cv2

RED = (0, 0, 255)

def show_prections(img, predictions):
	i = 0
	for coord in predictions:
		i += 1
		keypt = (int(coord[0]), int(coord[1]))
		text_loc = (keypt[0]+5, keypt[1]+7)
		cv2.circle(img, keypt, 3, RED, -1)
		cv2.putText(img, str(i), text_loc, cv2.FONT_HERSHEY_DUPLEX, 0.5, RED, 1, cv2.LINE_AA)
	cv2.imshow('img', img)
	cv2.waitKey(0)

if __name__ == '__main__':
	print('--Parsing Config File')
	params = process_config('config.cfg')
	
	img = cv2.imread(os.path.join(params['img_directory'], '005808361.jpg'))
        # the bounding boxes used below are ground truth from dataset.txt
	img1 = np.copy(img)[80:711, 798:1167]
	img2 = np.copy(img)[66:651, 310:721]
	
	test_img = cv2.resize(img2, (256, 256))

	model = Inference()
	predictions = model.predictJoints(test_img, mode='gpu')
	show_prections(test_img, predictions)

what's the difference of v0.1 and v0.3?

PCK Benchmark

Hi,

I have a question for people working on MPII dataset. To compute the PCKh (Percentage of Correct Keypoint with respect to the head). I started to compute accuracy using this benchmark, I obtain very high results for some joints. I need to figure it out if my understanding of this benchmark is correct.
This is the formula I use.

I compute for a given set all the heat maps for every images, then I find the location (arg max) and then I compute the L2 error between the prediction and the ground truth. normalized by the body proportion. To get PCK for a threshold 'r' I find how many samples have a value below this threshold and divide it by the total number of joints. Is that correct?

Model with Keras correct?

I'm trying to convert the HourglassModel class/network to a Keras model. What I have so far is;

(https://gist.github.com/SaifAlDilaimi/cdacc15129fb71f905990282c20b0b35)

# -*- coding: utf-8 -*-
import GlobalParams as PARAMS

from keras import backend as K

from keras.layers import Input
from keras.layers import Lambda
from keras.layers import Dense
from keras.layers import Dropout
from keras.layers import Activation
from keras.layers import Conv2D
from keras.layers import SeparableConv2D
from keras.layers import BatchNormalization

from keras.layers import MaxPooling2D
from keras.layers import UpSampling2D

from keras.layers import Multiply
from keras.layers import Concatenate
from keras.layers import Add

from keras.models import Model
from keras.optimizers import RMSprop
from keras import losses
from keras import metrics

def conv(x, filters, kernel_size = 1, strides=(1, 1), padding='same', name="conv"):
    """ Spatial Convolution (CONV2D)
		Args:
			inputs			: Input Tensor (Data Type : NHWC)
			filters		: Number of filters (channels)
			kernel_size	: Size of kernel
			strides		: Stride
			pad				: Padding Type (VALID/SAME)
			name			: Name of the block
		Returns:
			conv			: Output Tensor (Convolved Input)
	"""
    x = Conv2D(filters, kernel_size, strides=strides, padding=padding,
            use_bias=False)(x)
    return x

def max_pool2d(x, pool_size = (2,2), strides=(2,2), padding="valid"):
    x = MaxPooling2D(pool_size=pool_size, strides=strides, padding=padding)(x)
    return x

def conv_bn(x, filters, kernel_size = 1, strides=(1, 1), padding='same', name="conv_bn"):

    x = conv(x, filters, kernel_size, strides, padding, name)
    x = BatchNormalization(axis=-1, scale=False)(x)
    return x

def conv_bn_relu(x, filters, kernel_size = 1, strides = 1, padding="same", name="conv_bn_rel"):
    """ Spatial Convolution (CONV2D) + BatchNormalization + ReLU Activation
    Args:
        inputs			: Input Tensor (Data Type : NHWC)
        filters		: Number of filters (channels)
        kernel_size	: Size of kernel
        strides		: Stride
        pad				: Padding Type (VALID/SAME)
        name			: Name of the block
    Returns:
        norm			: Output Tensor
    """

    x = conv(x, filters, kernel_size, strides, padding, name)
    x = BatchNormalization(axis=-1, scale=False)(x)
    x = Activation('relu')(x)
    return x

def conv_block(x, numOut, name="conv_block"):
    """ Convolutional Block
		Args:
			inputs	: Input Tensor
			numOut	: Desired output number of channel
			name	: Name of the block
		Returns:
			conv_3	: Output Tensor
		"""

    x = BatchNormalization(axis=-1, scale=False)(x)
    x = Activation('relu')(x)
    x = conv(x, int(numOut))
    
    return x

def skip_layer(x, numOut, name = 'skip_layer'):
    """ Skip Layer
    Args:
        inputs	: Input Tensor
        numOut	: Desired output number of channel
        name	: Name of the bloc
    Returns:
        Tensor of shape (None, inputs.height, inputs.width, numOut)
    """
    print(x.shape, x.shape[3], numOut, x.shape[3] == numOut)
    if x.shape[3] == numOut: # check if right
        return x
    
    x = conv(x, numOut)
    return x

def residual_block(x, numOut, name = "residual_block"):
    """ Residual Unit
		Args:
			inputs	: Input Tensor
			numOut	: Number of Output Features (channels)
			name	: Name of the block
	"""
    convb = conv_block(x, numOut)
    skip_l = skip_layer(x, numOut)
    x = Add()([convb, skip_l])
    x = Activation('relu')(x)
    return x

def hourglass(x, n, numOut, name = 'hourglass'):
    """ Hourglass Module
    Args:
        inputs	: Input Tensor
        n		: Number of downsampling step
        numOut	: Number of Output Features (channels)
        name	: Name of the block
    """
    # upper branch
    up_1 = residual_block(x, numOut, name="up_1")
    # lower branch
    low_ = max_pool2d(x)
    low_1 = residual_block(low_, numOut, name="low_1")

    if n > 0:
        low_2 = hourglass(low_1, n-1, numOut, name="low_2")
    else:
        low_2 = residual_block(low_1, numOut, name="low_2")
    
    low_3 = residual_block(low_2, numOut, name="low_3")
    print("low3: ", low_3.shape)
    low3_size = K.int_shape(low_3)[1:3]
    up_size = (2,2)
    print("upsampling size: ", up_size)
    #up_size = tuple([x*x for x in up_size])
    #print(up_size)
    up_2 = UpSampling2D(up_size)(low_3)

    print(up_1)
    print(up_2)
    
    x = Add()([up_2, up_1])
    x = Activation('relu')(x)
    x = Dropout(0.2)(x)

    return x
    



class HGKerasModel():
    
    def build_model(self):
        if K.image_data_format() == 'channels_first':
            input_shape = (3, PARAMS.ML_INPUT_IMAGE_HEIGHT, PARAMS.ML_INPUT_IMAGE_WIDTH)
        else:
            input_shape = (PARAMS.ML_INPUT_IMAGE_WIDTH, PARAMS.ML_INPUT_IMAGE_HEIGHT, 3)

        m_input = Input(shape=input_shape)

        # Storage Table
        hg = [None] * PARAMS.ML_DEEPPOSE_STAGES
        ll = [None] * PARAMS.ML_DEEPPOSE_STAGES
        ll_ = [None] * PARAMS.ML_DEEPPOSE_STAGES
        drop = [None] * PARAMS.ML_DEEPPOSE_STAGES
        out = [None] * PARAMS.ML_DEEPPOSE_STAGES
        out_ = [None] * PARAMS.ML_DEEPPOSE_STAGES
        sum_ = [None] * PARAMS.ML_DEEPPOSE_STAGES
        
        # preprossing
        conv1 = conv_bn_relu(m_input, filters=64, kernel_size=6, strides=2)
        r1 = residual_block(conv1, 128)
        pool1 = max_pool2d(r1)
        r2 = residual_block(pool1, numOut=int(PARAMS.ML_INPUT_FEATURES/2))
        r3 = residual_block(r2, numOut=PARAMS.ML_INPUT_FEATURES)

        # stage 0
        hg[0] = hourglass(r3, PARAMS.ML_HOURGLASS_DOWNSAMPLING, PARAMS.ML_INPUT_FEATURES)
        ll[0] = conv_bn_relu(hg[0], PARAMS.ML_INPUT_FEATURES)
        out[0] = conv_bn_relu(ll[0], PARAMS.ML_LABEL_CLASSES)
        out_[0] = conv(out[0], PARAMS.ML_INPUT_FEATURES)
        sum_[0] = Add()([out_[0], ll[0], r3])

        # build stages 1 till k-1
        for i in range(1, PARAMS.ML_DEEPPOSE_STAGES - 1):    
            hg[i] = hourglass(sum_[i-1], PARAMS.ML_HOURGLASS_DOWNSAMPLING, PARAMS.ML_INPUT_FEATURES)
            ll[i] = conv_bn_relu(hg[i], PARAMS.ML_INPUT_FEATURES)
            out[i] = conv_bn_relu(ll[i], PARAMS.ML_LABEL_CLASSES)
            out_[i] = conv(out[i], PARAMS.ML_INPUT_FEATURES)
            sum_[i] = Add()([out_[i], ll[i], sum_[i-1]])

        # build stage k-1
        stages = PARAMS.ML_DEEPPOSE_STAGES
        hg[stages - 1] = hourglass(sum_[stages - 2], PARAMS.ML_HOURGLASS_DOWNSAMPLING, PARAMS.ML_INPUT_FEATURES)
        ll[stages - 1] = conv_bn_relu(hg[stages - 1], PARAMS.ML_INPUT_FEATURES)
        out[stages - 1] = conv_bn_relu(ll[stages - 1], PARAMS.ML_LABEL_CLASSES)

        conc = Concatenate()(out)
        sigmoid = Activation('sigmoid')(conc)


        model = Model(inputs=m_input, outputs=sigmoid)
        rmsprop = RMSprop(lr=PARAMS.ML_HOURGLASS_LEARN_RATE, decay=PARAMS.ML_HOURGLASS_LEARN_RATE_DECAY)

        model.compile(rmsprop, loss=losses.binary_crossentropy, metrics=['accuracy'])

        model.summary()

        print("Input shape: ", m_input.shape)
        print("Output length: ", len(sigmoid_out))

        return model


def main():
    model = HGKerasModel().build_model()
    
    for out in model.output:
        print(out)

if __name__ == '__main__':
    main()

Now the summary of the model with 2 stacks (stages) is this:

Layer (type)                    Output Shape         Param #     Connected to
==================================================================================================
input_1 (InputLayer)            (None, 256, 256, 3)  0
__________________________________________________________________________________________________
conv2d_1 (Conv2D)               (None, 128, 128, 64) 6912        input_1[0][0]
__________________________________________________________________________________________________
batch_normalization_1 (BatchNor (None, 128, 128, 64) 192         conv2d_1[0][0]
__________________________________________________________________________________________________
activation_1 (Activation)       (None, 128, 128, 64) 0           batch_normalization_1[0][0]
__________________________________________________________________________________________________
batch_normalization_2 (BatchNor (None, 128, 128, 64) 192         activation_1[0][0]
__________________________________________________________________________________________________
activation_2 (Activation)       (None, 128, 128, 64) 0           batch_normalization_2[0][0]
__________________________________________________________________________________________________
conv2d_2 (Conv2D)               (None, 128, 128, 128 8192        activation_2[0][0]
__________________________________________________________________________________________________
conv2d_3 (Conv2D)               (None, 128, 128, 128 8192        activation_1[0][0]
__________________________________________________________________________________________________
add_1 (Add)                     (None, 128, 128, 128 0           conv2d_2[0][0]
                                                                 conv2d_3[0][0]
__________________________________________________________________________________________________
activation_3 (Activation)       (None, 128, 128, 128 0           add_1[0][0]
__________________________________________________________________________________________________
max_pooling2d_1 (MaxPooling2D)  (None, 64, 64, 128)  0           activation_3[0][0]
__________________________________________________________________________________________________
batch_normalization_3 (BatchNor (None, 64, 64, 128)  384         max_pooling2d_1[0][0]
__________________________________________________________________________________________________
activation_4 (Activation)       (None, 64, 64, 128)  0           batch_normalization_3[0][0]
__________________________________________________________________________________________________
conv2d_4 (Conv2D)               (None, 64, 64, 128)  16384       activation_4[0][0]
__________________________________________________________________________________________________
add_2 (Add)                     (None, 64, 64, 128)  0           conv2d_4[0][0]
                                                                 max_pooling2d_1[0][0]
__________________________________________________________________________________________________
activation_5 (Activation)       (None, 64, 64, 128)  0           add_2[0][0]
__________________________________________________________________________________________________
batch_normalization_4 (BatchNor (None, 64, 64, 128)  384         activation_5[0][0]
__________________________________________________________________________________________________
activation_6 (Activation)       (None, 64, 64, 128)  0           batch_normalization_4[0][0]
__________________________________________________________________________________________________
conv2d_5 (Conv2D)               (None, 64, 64, 256)  32768       activation_6[0][0]
__________________________________________________________________________________________________
conv2d_6 (Conv2D)               (None, 64, 64, 256)  32768       activation_5[0][0]
__________________________________________________________________________________________________
add_3 (Add)                     (None, 64, 64, 256)  0           conv2d_5[0][0]
                                                                 conv2d_6[0][0]
__________________________________________________________________________________________________
activation_7 (Activation)       (None, 64, 64, 256)  0           add_3[0][0]
__________________________________________________________________________________________________
max_pooling2d_2 (MaxPooling2D)  (None, 32, 32, 256)  0           activation_7[0][0]
__________________________________________________________________________________________________
batch_normalization_6 (BatchNor (None, 32, 32, 256)  768         max_pooling2d_2[0][0]
__________________________________________________________________________________________________
activation_10 (Activation)      (None, 32, 32, 256)  0           batch_normalization_6[0][0]
__________________________________________________________________________________________________
conv2d_8 (Conv2D)               (None, 32, 32, 256)  65536       activation_10[0][0]
__________________________________________________________________________________________________
add_5 (Add)                     (None, 32, 32, 256)  0           conv2d_8[0][0]
                                                                 max_pooling2d_2[0][0]
__________________________________________________________________________________________________
activation_11 (Activation)      (None, 32, 32, 256)  0           add_5[0][0]
__________________________________________________________________________________________________
max_pooling2d_3 (MaxPooling2D)  (None, 16, 16, 256)  0           activation_11[0][0]
__________________________________________________________________________________________________
batch_normalization_8 (BatchNor (None, 16, 16, 256)  768         max_pooling2d_3[0][0]
__________________________________________________________________________________________________
activation_14 (Activation)      (None, 16, 16, 256)  0           batch_normalization_8[0][0]
__________________________________________________________________________________________________
conv2d_10 (Conv2D)              (None, 16, 16, 256)  65536       activation_14[0][0]
__________________________________________________________________________________________________
add_7 (Add)                     (None, 16, 16, 256)  0           conv2d_10[0][0]
                                                                 max_pooling2d_3[0][0]
__________________________________________________________________________________________________
activation_15 (Activation)      (None, 16, 16, 256)  0           add_7[0][0]
__________________________________________________________________________________________________
max_pooling2d_4 (MaxPooling2D)  (None, 8, 8, 256)    0           activation_15[0][0]
__________________________________________________________________________________________________
batch_normalization_10 (BatchNo (None, 8, 8, 256)    768         max_pooling2d_4[0][0]
__________________________________________________________________________________________________
activation_18 (Activation)      (None, 8, 8, 256)    0           batch_normalization_10[0][0]
__________________________________________________________________________________________________
conv2d_12 (Conv2D)              (None, 8, 8, 256)    65536       activation_18[0][0]
__________________________________________________________________________________________________
add_9 (Add)                     (None, 8, 8, 256)    0           conv2d_12[0][0]
                                                                 max_pooling2d_4[0][0]
__________________________________________________________________________________________________
activation_19 (Activation)      (None, 8, 8, 256)    0           add_9[0][0]
__________________________________________________________________________________________________
max_pooling2d_5 (MaxPooling2D)  (None, 4, 4, 256)    0           activation_19[0][0]
__________________________________________________________________________________________________
batch_normalization_12 (BatchNo (None, 4, 4, 256)    768         max_pooling2d_5[0][0]
__________________________________________________________________________________________________
activation_22 (Activation)      (None, 4, 4, 256)    0           batch_normalization_12[0][0]
__________________________________________________________________________________________________
conv2d_14 (Conv2D)              (None, 4, 4, 256)    65536       activation_22[0][0]
__________________________________________________________________________________________________
add_11 (Add)                    (None, 4, 4, 256)    0           conv2d_14[0][0]
                                                                 max_pooling2d_5[0][0]
__________________________________________________________________________________________________
activation_23 (Activation)      (None, 4, 4, 256)    0           add_11[0][0]
__________________________________________________________________________________________________
max_pooling2d_6 (MaxPooling2D)  (None, 2, 2, 256)    0           activation_23[0][0]
__________________________________________________________________________________________________
batch_normalization_14 (BatchNo (None, 2, 2, 256)    768         max_pooling2d_6[0][0]
__________________________________________________________________________________________________
activation_26 (Activation)      (None, 2, 2, 256)    0           batch_normalization_14[0][0]
__________________________________________________________________________________________________
conv2d_16 (Conv2D)              (None, 2, 2, 256)    65536       activation_26[0][0]
__________________________________________________________________________________________________
add_13 (Add)                    (None, 2, 2, 256)    0           conv2d_16[0][0]
                                                                 max_pooling2d_6[0][0]
__________________________________________________________________________________________________
activation_27 (Activation)      (None, 2, 2, 256)    0           add_13[0][0]
__________________________________________________________________________________________________
batch_normalization_15 (BatchNo (None, 2, 2, 256)    768         activation_27[0][0]
__________________________________________________________________________________________________
activation_28 (Activation)      (None, 2, 2, 256)    0           batch_normalization_15[0][0]
__________________________________________________________________________________________________
conv2d_17 (Conv2D)              (None, 2, 2, 256)    65536       activation_28[0][0]
__________________________________________________________________________________________________
add_14 (Add)                    (None, 2, 2, 256)    0           conv2d_17[0][0]
                                                                 activation_27[0][0]
__________________________________________________________________________________________________
activation_29 (Activation)      (None, 2, 2, 256)    0           add_14[0][0]
__________________________________________________________________________________________________
batch_normalization_16 (BatchNo (None, 2, 2, 256)    768         activation_29[0][0]
__________________________________________________________________________________________________
activation_30 (Activation)      (None, 2, 2, 256)    0           batch_normalization_16[0][0]
__________________________________________________________________________________________________
batch_normalization_13 (BatchNo (None, 4, 4, 256)    768         activation_23[0][0]
__________________________________________________________________________________________________
conv2d_18 (Conv2D)              (None, 2, 2, 256)    65536       activation_30[0][0]
__________________________________________________________________________________________________
activation_24 (Activation)      (None, 4, 4, 256)    0           batch_normalization_13[0][0]
__________________________________________________________________________________________________
add_15 (Add)                    (None, 2, 2, 256)    0           conv2d_18[0][0]
                                                                 activation_29[0][0]
__________________________________________________________________________________________________
conv2d_15 (Conv2D)              (None, 4, 4, 256)    65536       activation_24[0][0]
__________________________________________________________________________________________________
activation_31 (Activation)      (None, 2, 2, 256)    0           add_15[0][0]
__________________________________________________________________________________________________
add_12 (Add)                    (None, 4, 4, 256)    0           conv2d_15[0][0]
                                                                 activation_23[0][0]
__________________________________________________________________________________________________
up_sampling2d_1 (UpSampling2D)  (None, 4, 4, 256)    0           activation_31[0][0]
__________________________________________________________________________________________________
activation_25 (Activation)      (None, 4, 4, 256)    0           add_12[0][0]
__________________________________________________________________________________________________
add_16 (Add)                    (None, 4, 4, 256)    0           up_sampling2d_1[0][0]
                                                                 activation_25[0][0]
__________________________________________________________________________________________________
activation_32 (Activation)      (None, 4, 4, 256)    0           add_16[0][0]
__________________________________________________________________________________________________
dropout_1 (Dropout)             (None, 4, 4, 256)    0           activation_32[0][0]
__________________________________________________________________________________________________
batch_normalization_17 (BatchNo (None, 4, 4, 256)    768         dropout_1[0][0]
__________________________________________________________________________________________________
activation_33 (Activation)      (None, 4, 4, 256)    0           batch_normalization_17[0][0]
__________________________________________________________________________________________________
batch_normalization_11 (BatchNo (None, 8, 8, 256)    768         activation_19[0][0]
__________________________________________________________________________________________________
conv2d_19 (Conv2D)              (None, 4, 4, 256)    65536       activation_33[0][0]
__________________________________________________________________________________________________
activation_20 (Activation)      (None, 8, 8, 256)    0           batch_normalization_11[0][0]
__________________________________________________________________________________________________
add_17 (Add)                    (None, 4, 4, 256)    0           conv2d_19[0][0]
                                                                 dropout_1[0][0]
__________________________________________________________________________________________________
conv2d_13 (Conv2D)              (None, 8, 8, 256)    65536       activation_20[0][0]
__________________________________________________________________________________________________
activation_34 (Activation)      (None, 4, 4, 256)    0           add_17[0][0]
__________________________________________________________________________________________________
add_10 (Add)                    (None, 8, 8, 256)    0           conv2d_13[0][0]
                                                                 activation_19[0][0]
__________________________________________________________________________________________________
up_sampling2d_2 (UpSampling2D)  (None, 8, 8, 256)    0           activation_34[0][0]
__________________________________________________________________________________________________
activation_21 (Activation)      (None, 8, 8, 256)    0           add_10[0][0]
__________________________________________________________________________________________________
add_18 (Add)                    (None, 8, 8, 256)    0           up_sampling2d_2[0][0]
                                                                 activation_21[0][0]
__________________________________________________________________________________________________
activation_35 (Activation)      (None, 8, 8, 256)    0           add_18[0][0]
__________________________________________________________________________________________________
dropout_2 (Dropout)             (None, 8, 8, 256)    0           activation_35[0][0]
__________________________________________________________________________________________________
batch_normalization_18 (BatchNo (None, 8, 8, 256)    768         dropout_2[0][0]
__________________________________________________________________________________________________
activation_36 (Activation)      (None, 8, 8, 256)    0           batch_normalization_18[0][0]
__________________________________________________________________________________________________
batch_normalization_9 (BatchNor (None, 16, 16, 256)  768         activation_15[0][0]
__________________________________________________________________________________________________
conv2d_20 (Conv2D)              (None, 8, 8, 256)    65536       activation_36[0][0]
__________________________________________________________________________________________________
activation_16 (Activation)      (None, 16, 16, 256)  0           batch_normalization_9[0][0]
__________________________________________________________________________________________________
add_19 (Add)                    (None, 8, 8, 256)    0           conv2d_20[0][0]
                                                                 dropout_2[0][0]
__________________________________________________________________________________________________
conv2d_11 (Conv2D)              (None, 16, 16, 256)  65536       activation_16[0][0]
__________________________________________________________________________________________________
activation_37 (Activation)      (None, 8, 8, 256)    0           add_19[0][0]
__________________________________________________________________________________________________
add_8 (Add)                     (None, 16, 16, 256)  0           conv2d_11[0][0]
                                                                 activation_15[0][0]
__________________________________________________________________________________________________
up_sampling2d_3 (UpSampling2D)  (None, 16, 16, 256)  0           activation_37[0][0]
__________________________________________________________________________________________________
activation_17 (Activation)      (None, 16, 16, 256)  0           add_8[0][0]
__________________________________________________________________________________________________
add_20 (Add)                    (None, 16, 16, 256)  0           up_sampling2d_3[0][0]
                                                                 activation_17[0][0]
__________________________________________________________________________________________________
activation_38 (Activation)      (None, 16, 16, 256)  0           add_20[0][0]
__________________________________________________________________________________________________
dropout_3 (Dropout)             (None, 16, 16, 256)  0           activation_38[0][0]
__________________________________________________________________________________________________
batch_normalization_19 (BatchNo (None, 16, 16, 256)  768         dropout_3[0][0]
__________________________________________________________________________________________________
activation_39 (Activation)      (None, 16, 16, 256)  0           batch_normalization_19[0][0]
__________________________________________________________________________________________________
batch_normalization_7 (BatchNor (None, 32, 32, 256)  768         activation_11[0][0]
__________________________________________________________________________________________________
conv2d_21 (Conv2D)              (None, 16, 16, 256)  65536       activation_39[0][0]
__________________________________________________________________________________________________
activation_12 (Activation)      (None, 32, 32, 256)  0           batch_normalization_7[0][0]
__________________________________________________________________________________________________
add_21 (Add)                    (None, 16, 16, 256)  0           conv2d_21[0][0]
                                                                 dropout_3[0][0]
__________________________________________________________________________________________________
conv2d_9 (Conv2D)               (None, 32, 32, 256)  65536       activation_12[0][0]
__________________________________________________________________________________________________
activation_40 (Activation)      (None, 16, 16, 256)  0           add_21[0][0]
__________________________________________________________________________________________________
add_6 (Add)                     (None, 32, 32, 256)  0           conv2d_9[0][0]
                                                                 activation_11[0][0]
__________________________________________________________________________________________________
up_sampling2d_4 (UpSampling2D)  (None, 32, 32, 256)  0           activation_40[0][0]
__________________________________________________________________________________________________
activation_13 (Activation)      (None, 32, 32, 256)  0           add_6[0][0]
__________________________________________________________________________________________________
add_22 (Add)                    (None, 32, 32, 256)  0           up_sampling2d_4[0][0]
                                                                 activation_13[0][0]
__________________________________________________________________________________________________
activation_41 (Activation)      (None, 32, 32, 256)  0           add_22[0][0]
__________________________________________________________________________________________________
dropout_4 (Dropout)             (None, 32, 32, 256)  0           activation_41[0][0]
__________________________________________________________________________________________________
batch_normalization_20 (BatchNo (None, 32, 32, 256)  768         dropout_4[0][0]
__________________________________________________________________________________________________
activation_42 (Activation)      (None, 32, 32, 256)  0           batch_normalization_20[0][0]
__________________________________________________________________________________________________
batch_normalization_5 (BatchNor (None, 64, 64, 256)  768         activation_7[0][0]
__________________________________________________________________________________________________
conv2d_22 (Conv2D)              (None, 32, 32, 256)  65536       activation_42[0][0]
__________________________________________________________________________________________________
activation_8 (Activation)       (None, 64, 64, 256)  0           batch_normalization_5[0][0]
__________________________________________________________________________________________________
add_23 (Add)                    (None, 32, 32, 256)  0           conv2d_22[0][0]
                                                                 dropout_4[0][0]
__________________________________________________________________________________________________
conv2d_7 (Conv2D)               (None, 64, 64, 256)  65536       activation_8[0][0]
__________________________________________________________________________________________________
activation_43 (Activation)      (None, 32, 32, 256)  0           add_23[0][0]
__________________________________________________________________________________________________
add_4 (Add)                     (None, 64, 64, 256)  0           conv2d_7[0][0]
                                                                 activation_7[0][0]
__________________________________________________________________________________________________
up_sampling2d_5 (UpSampling2D)  (None, 64, 64, 256)  0           activation_43[0][0]
__________________________________________________________________________________________________
activation_9 (Activation)       (None, 64, 64, 256)  0           add_4[0][0]
__________________________________________________________________________________________________
add_24 (Add)                    (None, 64, 64, 256)  0           up_sampling2d_5[0][0]
                                                                 activation_9[0][0]
__________________________________________________________________________________________________
activation_44 (Activation)      (None, 64, 64, 256)  0           add_24[0][0]
__________________________________________________________________________________________________
dropout_5 (Dropout)             (None, 64, 64, 256)  0           activation_44[0][0]
__________________________________________________________________________________________________
conv2d_23 (Conv2D)              (None, 64, 64, 256)  65536       dropout_5[0][0]
__________________________________________________________________________________________________
batch_normalization_21 (BatchNo (None, 64, 64, 256)  768         conv2d_23[0][0]
__________________________________________________________________________________________________
activation_45 (Activation)      (None, 64, 64, 256)  0           batch_normalization_21[0][0]
__________________________________________________________________________________________________
conv2d_24 (Conv2D)              (None, 64, 64, 16)   4096        activation_45[0][0]
__________________________________________________________________________________________________
batch_normalization_22 (BatchNo (None, 64, 64, 16)   48          conv2d_24[0][0]
__________________________________________________________________________________________________
activation_46 (Activation)      (None, 64, 64, 16)   0           batch_normalization_22[0][0]
__________________________________________________________________________________________________
conv2d_25 (Conv2D)              (None, 64, 64, 256)  4096        activation_46[0][0]
__________________________________________________________________________________________________
add_25 (Add)                    (None, 64, 64, 256)  0           conv2d_25[0][0]
                                                                 activation_45[0][0]
                                                                 activation_7[0][0]
__________________________________________________________________________________________________
max_pooling2d_7 (MaxPooling2D)  (None, 32, 32, 256)  0           add_25[0][0]
__________________________________________________________________________________________________
batch_normalization_24 (BatchNo (None, 32, 32, 256)  768         max_pooling2d_7[0][0]
__________________________________________________________________________________________________
activation_49 (Activation)      (None, 32, 32, 256)  0           batch_normalization_24[0][0]
__________________________________________________________________________________________________
conv2d_27 (Conv2D)              (None, 32, 32, 256)  65536       activation_49[0][0]
__________________________________________________________________________________________________
add_27 (Add)                    (None, 32, 32, 256)  0           conv2d_27[0][0]
                                                                 max_pooling2d_7[0][0]
__________________________________________________________________________________________________
activation_50 (Activation)      (None, 32, 32, 256)  0           add_27[0][0]
__________________________________________________________________________________________________
max_pooling2d_8 (MaxPooling2D)  (None, 16, 16, 256)  0           activation_50[0][0]
__________________________________________________________________________________________________
batch_normalization_26 (BatchNo (None, 16, 16, 256)  768         max_pooling2d_8[0][0]
__________________________________________________________________________________________________
activation_53 (Activation)      (None, 16, 16, 256)  0           batch_normalization_26[0][0]
__________________________________________________________________________________________________
conv2d_29 (Conv2D)              (None, 16, 16, 256)  65536       activation_53[0][0]
__________________________________________________________________________________________________
add_29 (Add)                    (None, 16, 16, 256)  0           conv2d_29[0][0]
                                                                 max_pooling2d_8[0][0]
__________________________________________________________________________________________________
activation_54 (Activation)      (None, 16, 16, 256)  0           add_29[0][0]
__________________________________________________________________________________________________
max_pooling2d_9 (MaxPooling2D)  (None, 8, 8, 256)    0           activation_54[0][0]
__________________________________________________________________________________________________
batch_normalization_28 (BatchNo (None, 8, 8, 256)    768         max_pooling2d_9[0][0]
__________________________________________________________________________________________________
activation_57 (Activation)      (None, 8, 8, 256)    0           batch_normalization_28[0][0]
__________________________________________________________________________________________________
conv2d_31 (Conv2D)              (None, 8, 8, 256)    65536       activation_57[0][0]
__________________________________________________________________________________________________
add_31 (Add)                    (None, 8, 8, 256)    0           conv2d_31[0][0]
                                                                 max_pooling2d_9[0][0]
__________________________________________________________________________________________________
activation_58 (Activation)      (None, 8, 8, 256)    0           add_31[0][0]
__________________________________________________________________________________________________
max_pooling2d_10 (MaxPooling2D) (None, 4, 4, 256)    0           activation_58[0][0]
__________________________________________________________________________________________________
batch_normalization_30 (BatchNo (None, 4, 4, 256)    768         max_pooling2d_10[0][0]
__________________________________________________________________________________________________
activation_61 (Activation)      (None, 4, 4, 256)    0           batch_normalization_30[0][0]
__________________________________________________________________________________________________
conv2d_33 (Conv2D)              (None, 4, 4, 256)    65536       activation_61[0][0]
__________________________________________________________________________________________________
add_33 (Add)                    (None, 4, 4, 256)    0           conv2d_33[0][0]
                                                                 max_pooling2d_10[0][0]
__________________________________________________________________________________________________
activation_62 (Activation)      (None, 4, 4, 256)    0           add_33[0][0]
__________________________________________________________________________________________________
max_pooling2d_11 (MaxPooling2D) (None, 2, 2, 256)    0           activation_62[0][0]
__________________________________________________________________________________________________
batch_normalization_32 (BatchNo (None, 2, 2, 256)    768         max_pooling2d_11[0][0]
__________________________________________________________________________________________________
activation_65 (Activation)      (None, 2, 2, 256)    0           batch_normalization_32[0][0]
__________________________________________________________________________________________________
conv2d_35 (Conv2D)              (None, 2, 2, 256)    65536       activation_65[0][0]
__________________________________________________________________________________________________
add_35 (Add)                    (None, 2, 2, 256)    0           conv2d_35[0][0]
                                                                 max_pooling2d_11[0][0]
__________________________________________________________________________________________________
activation_66 (Activation)      (None, 2, 2, 256)    0           add_35[0][0]
__________________________________________________________________________________________________
batch_normalization_33 (BatchNo (None, 2, 2, 256)    768         activation_66[0][0]
__________________________________________________________________________________________________
activation_67 (Activation)      (None, 2, 2, 256)    0           batch_normalization_33[0][0]
__________________________________________________________________________________________________
conv2d_36 (Conv2D)              (None, 2, 2, 256)    65536       activation_67[0][0]
__________________________________________________________________________________________________
add_36 (Add)                    (None, 2, 2, 256)    0           conv2d_36[0][0]
                                                                 activation_66[0][0]
__________________________________________________________________________________________________
activation_68 (Activation)      (None, 2, 2, 256)    0           add_36[0][0]
__________________________________________________________________________________________________
batch_normalization_34 (BatchNo (None, 2, 2, 256)    768         activation_68[0][0]
__________________________________________________________________________________________________
activation_69 (Activation)      (None, 2, 2, 256)    0           batch_normalization_34[0][0]
__________________________________________________________________________________________________
batch_normalization_31 (BatchNo (None, 4, 4, 256)    768         activation_62[0][0]
__________________________________________________________________________________________________
conv2d_37 (Conv2D)              (None, 2, 2, 256)    65536       activation_69[0][0]
__________________________________________________________________________________________________
activation_63 (Activation)      (None, 4, 4, 256)    0           batch_normalization_31[0][0]
__________________________________________________________________________________________________
add_37 (Add)                    (None, 2, 2, 256)    0           conv2d_37[0][0]
                                                                 activation_68[0][0]
__________________________________________________________________________________________________
conv2d_34 (Conv2D)              (None, 4, 4, 256)    65536       activation_63[0][0]
__________________________________________________________________________________________________
activation_70 (Activation)      (None, 2, 2, 256)    0           add_37[0][0]
__________________________________________________________________________________________________
add_34 (Add)                    (None, 4, 4, 256)    0           conv2d_34[0][0]
                                                                 activation_62[0][0]
__________________________________________________________________________________________________
up_sampling2d_6 (UpSampling2D)  (None, 4, 4, 256)    0           activation_70[0][0]
__________________________________________________________________________________________________
activation_64 (Activation)      (None, 4, 4, 256)    0           add_34[0][0]
__________________________________________________________________________________________________
add_38 (Add)                    (None, 4, 4, 256)    0           up_sampling2d_6[0][0]
                                                                 activation_64[0][0]
__________________________________________________________________________________________________
activation_71 (Activation)      (None, 4, 4, 256)    0           add_38[0][0]
__________________________________________________________________________________________________
dropout_6 (Dropout)             (None, 4, 4, 256)    0           activation_71[0][0]
__________________________________________________________________________________________________
batch_normalization_35 (BatchNo (None, 4, 4, 256)    768         dropout_6[0][0]
__________________________________________________________________________________________________
activation_72 (Activation)      (None, 4, 4, 256)    0           batch_normalization_35[0][0]
__________________________________________________________________________________________________
batch_normalization_29 (BatchNo (None, 8, 8, 256)    768         activation_58[0][0]
__________________________________________________________________________________________________
conv2d_38 (Conv2D)              (None, 4, 4, 256)    65536       activation_72[0][0]
__________________________________________________________________________________________________
activation_59 (Activation)      (None, 8, 8, 256)    0           batch_normalization_29[0][0]
__________________________________________________________________________________________________
add_39 (Add)                    (None, 4, 4, 256)    0           conv2d_38[0][0]
                                                                 dropout_6[0][0]
__________________________________________________________________________________________________
conv2d_32 (Conv2D)              (None, 8, 8, 256)    65536       activation_59[0][0]
__________________________________________________________________________________________________
activation_73 (Activation)      (None, 4, 4, 256)    0           add_39[0][0]
__________________________________________________________________________________________________
add_32 (Add)                    (None, 8, 8, 256)    0           conv2d_32[0][0]
                                                                 activation_58[0][0]
__________________________________________________________________________________________________
up_sampling2d_7 (UpSampling2D)  (None, 8, 8, 256)    0           activation_73[0][0]
__________________________________________________________________________________________________
activation_60 (Activation)      (None, 8, 8, 256)    0           add_32[0][0]
__________________________________________________________________________________________________
add_40 (Add)                    (None, 8, 8, 256)    0           up_sampling2d_7[0][0]
                                                                 activation_60[0][0]
__________________________________________________________________________________________________
activation_74 (Activation)      (None, 8, 8, 256)    0           add_40[0][0]
__________________________________________________________________________________________________
dropout_7 (Dropout)             (None, 8, 8, 256)    0           activation_74[0][0]
__________________________________________________________________________________________________
batch_normalization_36 (BatchNo (None, 8, 8, 256)    768         dropout_7[0][0]
__________________________________________________________________________________________________
activation_75 (Activation)      (None, 8, 8, 256)    0           batch_normalization_36[0][0]
__________________________________________________________________________________________________
batch_normalization_27 (BatchNo (None, 16, 16, 256)  768         activation_54[0][0]
__________________________________________________________________________________________________
conv2d_39 (Conv2D)              (None, 8, 8, 256)    65536       activation_75[0][0]
__________________________________________________________________________________________________
activation_55 (Activation)      (None, 16, 16, 256)  0           batch_normalization_27[0][0]
__________________________________________________________________________________________________
add_41 (Add)                    (None, 8, 8, 256)    0           conv2d_39[0][0]
                                                                 dropout_7[0][0]
__________________________________________________________________________________________________
conv2d_30 (Conv2D)              (None, 16, 16, 256)  65536       activation_55[0][0]
__________________________________________________________________________________________________
activation_76 (Activation)      (None, 8, 8, 256)    0           add_41[0][0]
__________________________________________________________________________________________________
add_30 (Add)                    (None, 16, 16, 256)  0           conv2d_30[0][0]
                                                                 activation_54[0][0]
__________________________________________________________________________________________________
up_sampling2d_8 (UpSampling2D)  (None, 16, 16, 256)  0           activation_76[0][0]
__________________________________________________________________________________________________
activation_56 (Activation)      (None, 16, 16, 256)  0           add_30[0][0]
__________________________________________________________________________________________________
add_42 (Add)                    (None, 16, 16, 256)  0           up_sampling2d_8[0][0]
                                                                 activation_56[0][0]
__________________________________________________________________________________________________
activation_77 (Activation)      (None, 16, 16, 256)  0           add_42[0][0]
__________________________________________________________________________________________________
dropout_8 (Dropout)             (None, 16, 16, 256)  0           activation_77[0][0]
__________________________________________________________________________________________________
batch_normalization_37 (BatchNo (None, 16, 16, 256)  768         dropout_8[0][0]
__________________________________________________________________________________________________
activation_78 (Activation)      (None, 16, 16, 256)  0           batch_normalization_37[0][0]
__________________________________________________________________________________________________
batch_normalization_25 (BatchNo (None, 32, 32, 256)  768         activation_50[0][0]
__________________________________________________________________________________________________
conv2d_40 (Conv2D)              (None, 16, 16, 256)  65536       activation_78[0][0]
__________________________________________________________________________________________________
activation_51 (Activation)      (None, 32, 32, 256)  0           batch_normalization_25[0][0]
__________________________________________________________________________________________________
add_43 (Add)                    (None, 16, 16, 256)  0           conv2d_40[0][0]
                                                                 dropout_8[0][0]
__________________________________________________________________________________________________
conv2d_28 (Conv2D)              (None, 32, 32, 256)  65536       activation_51[0][0]
__________________________________________________________________________________________________
activation_79 (Activation)      (None, 16, 16, 256)  0           add_43[0][0]
__________________________________________________________________________________________________
add_28 (Add)                    (None, 32, 32, 256)  0           conv2d_28[0][0]
                                                                 activation_50[0][0]
__________________________________________________________________________________________________
up_sampling2d_9 (UpSampling2D)  (None, 32, 32, 256)  0           activation_79[0][0]
__________________________________________________________________________________________________
activation_52 (Activation)      (None, 32, 32, 256)  0           add_28[0][0]
__________________________________________________________________________________________________
add_44 (Add)                    (None, 32, 32, 256)  0           up_sampling2d_9[0][0]
                                                                 activation_52[0][0]
__________________________________________________________________________________________________
activation_80 (Activation)      (None, 32, 32, 256)  0           add_44[0][0]
__________________________________________________________________________________________________
dropout_9 (Dropout)             (None, 32, 32, 256)  0           activation_80[0][0]
__________________________________________________________________________________________________
batch_normalization_38 (BatchNo (None, 32, 32, 256)  768         dropout_9[0][0]
__________________________________________________________________________________________________
activation_81 (Activation)      (None, 32, 32, 256)  0           batch_normalization_38[0][0]
__________________________________________________________________________________________________
batch_normalization_23 (BatchNo (None, 64, 64, 256)  768         add_25[0][0]
__________________________________________________________________________________________________
conv2d_41 (Conv2D)              (None, 32, 32, 256)  65536       activation_81[0][0]
__________________________________________________________________________________________________
activation_47 (Activation)      (None, 64, 64, 256)  0           batch_normalization_23[0][0]
__________________________________________________________________________________________________
add_45 (Add)                    (None, 32, 32, 256)  0           conv2d_41[0][0]
                                                                 dropout_9[0][0]
__________________________________________________________________________________________________
conv2d_26 (Conv2D)              (None, 64, 64, 256)  65536       activation_47[0][0]
__________________________________________________________________________________________________
activation_82 (Activation)      (None, 32, 32, 256)  0           add_45[0][0]
__________________________________________________________________________________________________
add_26 (Add)                    (None, 64, 64, 256)  0           conv2d_26[0][0]
                                                                 add_25[0][0]
__________________________________________________________________________________________________
up_sampling2d_10 (UpSampling2D) (None, 64, 64, 256)  0           activation_82[0][0]
__________________________________________________________________________________________________
activation_48 (Activation)      (None, 64, 64, 256)  0           add_26[0][0]
__________________________________________________________________________________________________
add_46 (Add)                    (None, 64, 64, 256)  0           up_sampling2d_10[0][0]
                                                                 activation_48[0][0]
__________________________________________________________________________________________________
activation_83 (Activation)      (None, 64, 64, 256)  0           add_46[0][0]
__________________________________________________________________________________________________
dropout_10 (Dropout)            (None, 64, 64, 256)  0           activation_83[0][0]
__________________________________________________________________________________________________
conv2d_42 (Conv2D)              (None, 64, 64, 256)  65536       dropout_10[0][0]
__________________________________________________________________________________________________
batch_normalization_39 (BatchNo (None, 64, 64, 256)  768         conv2d_42[0][0]
__________________________________________________________________________________________________
activation_84 (Activation)      (None, 64, 64, 256)  0           batch_normalization_39[0][0]
__________________________________________________________________________________________________
conv2d_43 (Conv2D)              (None, 64, 64, 16)   4096        activation_84[0][0]
__________________________________________________________________________________________________
batch_normalization_40 (BatchNo (None, 64, 64, 16)   48          conv2d_43[0][0]
__________________________________________________________________________________________________
activation_85 (Activation)      (None, 64, 64, 16)   0           batch_normalization_40[0][0]
__________________________________________________________________________________________________
concatenate_1 (Concatenate)     (None, 64, 64, 32)   0           activation_46[0][0]
                                                                 activation_85[0][0]
__________________________________________________________________________________________________
activation_86 (Activation)      (None, 64, 64, 32)   0           concatenate_1[0][0]
==================================================================================================
Total params: 2,373,088
Trainable params: 2,354,848
Non-trainable params: 18,240
__________________________________________________________________________________________________
Input shape:  (?, 256, 256, 3)

As you see the output is not in the shape (16, 2) which contains the joints x,y... Any idea whats missing here? @wbenbihi

Const used in hourglass_tiny.py

Hi Walid,

Thanks for sharing the code!
Could you please tell me why you used 91 in the function ' _compute_err(self, u, v)'?
I understand you want to calculate the distance btw two locations and rescale it to [0,1], but why not divide it by 64?

Best regards,
Ruobing

Pretrained model

Hi! It would be great if you could share a pretrained model, which people would be able to run and get the results without spending 2+ days on training.

Segmentation fault when runnning inference with pre trained weights

I am trying to run the inference.py script with the pre trained weights.

here is my main :

if __name__ == '__main__':
    infer = Inference(config_file = 'HG/config.cfg', model = 'HG/hg_refined_200', yoloModel = 'HG/YOLO_small.ckpt')
    infer.webcamSingle()

But I am facing a segfault :

CREATE MODEL:
---Inputs : Done (0 sec.)
---Graph : Done (5 sec.)
---Loss : Done (0 sec.)
WARNING:tensorflow:From /work1/lmahieu/hourglasstensorlfow/hourglass_tiny.py:625: arg_max (from tensorflow.python.ops.gen_math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `argmax` instead
---Acc : Done (1 sec.)
---LR : Done (0 sec.)
---Optim : Done (0 sec.)
---Minimizer : Done (8 sec.)
---Init : Done (0 sec.)
Model created (15 sec.)
('Graph Generated in ', 15, ' sec.')
Session initialization
Sess initialized in 0 sec.
Loading Trained Model
('Model Loaded (', 0.7259650230407715, ' sec.)')
('YOLO created: ', 0.5169098377227783, ' sec.')
Loading YOLO...
Restoring weights from: HG/YOLO_small.ckpt
('Trained YOLO Loaded: ', 1.0975430011749268, ' sec.')
Prediction Tensors Ready!
('Done: ', 19.734932899475098, ' sec.')
VIDIOC_QUERYMENU: Invalid argument
VIDIOC_QUERYMENU: Invalid argument
VIDIOC_QUERYMENU: Invalid argument
VIDIOC_QUERYMENU: Invalid argument
VIDIOC_QUERYMENU: Invalid argument
VIDIOC_QUERYMENU: Invalid argument
VIDIOC_QUERYMENU: Invalid argument
Fatal Python error: Segmentation fault

Current thread 0x00007f7b464fe740 <python> (most recent call first):
  File "./hourglasstensorlfow/predictClass.py", line 658 in hpeWebcam
  File "inference.py", line 60 in webcamSingle
  File "inference.py", line 223 in <module>
[1]    8196 segmentation fault  python inference.py

Does someone have any idea ?

When attaching a GDB, I have

(gdb) bt
#0  0x00007ffff7ad2aa5 in do_warn () at /lib64/libpython2.7.so.1.0
#1  0x00007ffff7ad323d in PyErr_WarnEx () at /lib64/libpython2.7.so.1.0
#2  0x00007fffb15900f7 in _log_func () at /usr/lib64/python2.7/site-packages/gobject/_gobject.so
#3  0x00007fffb98e17a1 in g_logv () at /lib64/libglib-2.0.so.0
#4  0x00007fffb98e1a4f in g_log () at /lib64/libglib-2.0.so.0
#5  0x00007fffbb5b0396 in gtk_disable_setlocale () at /lib64/libgtk-x11-2.0.so.0
#6  0x00007fffbdf77876 in cvInitSystem () at /lib64/libopencv_highgui.so.2.4
#7  0x00007fffbdf77d52 in cvNamedWindow () at /lib64/libopencv_highgui.so.2.4
#8  0x00007fffbdf784cd in cvShowImage () at /lib64/libopencv_highgui.so.2.4
#9  0x00007fffbdf7474a in cv::imshow(std::string const&, cv::_InputArray const&) () at /lib64/libopencv_highgui.so.2.4
#10 0x00007fffbf1fa791 in pyopencv_imshow(_object*, _object*, _object*) () at /usr/lib64/python2.7/site-packages/cv2.so
#11 0x00007ffff7af3cf0 in PyEval_EvalFrameEx () at /lib64/libpython2.7.so.1.0
#12 0x00007ffff7af603d in PyEval_EvalCodeEx () at /lib64/libpython2.7.so.1.0
#13 0x00007ffff7af353c in PyEval_EvalFrameEx () at /lib64/libpython2.7.so.1.0
#14 0x00007ffff7af603d in PyEval_EvalCodeEx () at /lib64/libpython2.7.so.1.0
#15 0x00007ffff7af353c in PyEval_EvalFrameEx () at /lib64/libpython2.7.so.1.0
#16 0x00007ffff7af603d in PyEval_EvalCodeEx () at /lib64/libpython2.7.so.1.0
#17 0x00007ffff7af6142 in PyEval_EvalCode () at /lib64/libpython2.7.so.1.0
#18 0x00007ffff7b0f57f in run_mod () at /lib64/libpython2.7.so.1.0
#19 0x00007ffff7b1073e in PyRun_FileExFlags () at /lib64/libpython2.7.so.1.0
#20 0x00007ffff7b119c9 in PyRun_SimpleFileExFlags () at /lib64/libpython2.7.so.1.0
#21 0x00007ffff7b22b7f in Py_Main () at /lib64/libpython2.7.so.1.0
#22 0x00007ffff6d49b15 in __libc_start_main () at /lib64/libc.so.6
#23 0x000000000040066e in _start ()

Status

Hi I'am interested to experiment this TF version. What is the status of your implementation?
What accurancy can you already reproduce?
Do you plan to train/test over Posetrack?

What use of YOLO_small.ckpt ???

def init(self, config_file = 'config.cfg', model = 'hg_refined_tiny_200', yoloModel = 'YOLO_small.ckpt'):
""" Initilize the Predictor
Args:
config_file : *.cfg file with model's parameters
model : *.index file's name. (weights to load)
yoloModel : *.ckpt file (YOLO weights to load)
"""
t = time()
params = process_config(config_file)
self.predict = PredictProcessor(params)
self.predict.color_palette()
self.predict.LINKS_JOINTS()
self.predict.model_init()
self.predict.load_model(load = model)
self.predict.yolo_init()
self.predict.restore_yolo(load = yoloModel)
self.predict._create_prediction_tensor()
self.filter = VideoFilters()
print('Done: ', time() - t, ' sec.')

## self-trained model->'hg_refined_tiny_200', Why to overwrite with 'YOLO_small.ckpt' model ?

a suggestion in _hourglass() function

I think
up_2 = tf.image.resize_nearest_neighbor(low_3, tf.shape(low_3)[1:3]*2, name = 'upsampling')

should be

up_2 = tf.image.resize_nearest_neighbor(low_3, tf.shape(up_1)[1:3], name = 'upsampling')

so it could handle those input size is not 2^n

Pretrained model

Can you please share the pretrained model ?

Can I fine tune from pre-trained model?

Can I restore from pre-trained model and fine tune on another task? Say I want to detect 10 types of key points, how can I modify the code? Since directly loading the original model parameter will give 'dimension not match' error(MPII has 16 key points). Thanks.

update repo

hi,
i want to know if you have plan on updating this repo.
thx.

Some error

Hey , it is a really helpful job for me.
I have find some bugs
It seems that your python is different with mine.
In inference.py
when i use function predictHM. I add this code
return self.predict.pred((img.astype(np.float32) / 255), debug = False, sess = None) instead of original one.
So i think this is the reason why some body get high accuracy but wrong visialble results.
And other funtion is like this, before i use it. I must change the data type into float.
Includes this. In datagen.py line 282 it should not be if bounds[1] > width - 1:
it should be if bounds[1] > height - 1:
This line some times may cause error.

Besides this, the rotate function mine is different(OpenCV 310).
transform.rotate(img, r_angle, preserve_range = True) line336 in datagen.py
will cause error

so i change it into .
img = transform.rotate(img, r_angle)
Hope these jobs help others.

Quantitative results on test set

Hi everybody,

Thank you @wbenbihi: very nice repository.

So, my question is the following: how do I test my trained model on my test set?

I do not mean making an inference and obtaining qualitative result but go over all the test set and obtain a quantitative result on it.

Maybe I should do as If I were training but with the flag training of hourglass_tiny turned to False?

I appreciate any help on this matter.

Thank you all in advance!

About loss.

Hi,thanks to your awsome job!
I am trying to implement Hourglass model in tensorflow.And I notice the loss you used is "tf.nn.sigmoid_cross_entropy_with_logits".But in the paper the author use MSE instead.I trying to replace it with tf.losses.mean_squared_error but the net will not convergent as fast as the origin one and the result become awful.Have ever tried to use the MSE loss?And could you tell me why do you use CE loss instead?Thank you !

Typo in hourglass_tiny.py?

Hi, thanks for sharing the code. It helps a lot for my research.
BTW, I think there's a typo in the model definition. In line 472 of hourglass_tiny.py
sum_[i] = tf.add_n([out_[i], sum_[i-1], ll_[0]], name= 'merge')
I think ll_[0] should be fixed to ll_[i]. This might be critical for the performance..

Process video footage

Hi,

It is an issue regardig the PredictProcessor().videoProcessor() method.
predictClass.py line 564
fourcc = cv2.VideoWriter_fourcc(*'DIVX')
outVid = cv2.VideoWriter( outName, fourcc, fps, tuple(shape.astype(np.int)), 1)

I use this to open a video footage (*.mp4, *.avi), then process frames and save it in an other video file.
On windows, the code works fine except the output file is a video file of 6Ko. It does not save the frame with the outVid.write(frame) method.

Did anybody have a similar issue on OpenCV?

KeyError: 'nfeats'

I am confused. How do you get the visualization of estimated pose using pretrained models on an image?

[jalal@goku hourglasstensorlfow]$ ls -ltra
total 1048044
-rw-r--r--.  1 jalal cs-grad 375890392 Apr  4  2017 YOLO_small.ckpt
-rw-r--r--.  1 jalal cs-grad     68410 Aug 11  2017 hg_refined_200.index
-rw-r--r--.  1 jalal cs-grad 173234692 Aug 11  2017 hg_refined_200.data-00000-of-00001
drwxr-xr-x. 15 jalal cs-grad      4096 Mar 25 04:19 ..
-rw-r--r--.  1 jalal cs-grad     20762 Mar 25 04:19 datagen.py
-rw-r--r--.  1 jalal cs-grad      1131 Mar 25 04:19 config.py
-rw-r--r--.  1 jalal cs-grad       817 Mar 25 04:19 config.cfg
-rw-r--r--.  1 jalal cs-grad      5835 Mar 25 04:19 README.txt
-rw-r--r--.  1 jalal cs-grad      6646 Mar 25 04:19 README.md
-rw-r--r--.  1 jalal cs-grad      4573 Mar 25 04:19 filters.py
-rw-r--r--.  1 jalal cs-grad   6055266 Mar 25 04:19 dataset.txt
-rw-r--r--.  1 jalal cs-grad      9257 Mar 25 04:19 inference.py
-rw-r--r--.  1 jalal cs-grad     35449 Mar 25 04:19 hourglass_tiny.py
-rw-r--r--.  1 jalal cs-grad   9161356 Mar 25 04:19 opencv_ffmpeg.dll
-rw-r--r--.  1 jalal cs-grad   9219038 Mar 25 04:19 opencv_ffmpeg_64.dll
-rw-r--r--.  1 jalal cs-grad     46899 Mar 25 04:19 p4frames.mat
-rw-r--r--.  1 jalal cs-grad      1130 Mar 25 04:19 timer.py
-rw-r--r--.  1 jalal cs-grad     51242 Mar 25 04:19 predictClass.py
-rw-r--r--.  1 jalal cs-grad    170640 Mar 25 04:19 pckhfULL.png
-rw-r--r--.  1 jalal cs-grad     11471 Mar 25 04:19 yolo_net.py
drwxr-xr-x.  2 jalal cs-grad       145 Mar 25 04:19 v0.1
-rw-r--r--.  1 jalal cs-grad      2041 Mar 25 04:19 train_launcher.py
drwxr-xr-x.  8 jalal cs-grad       211 Mar 25 04:19 .git
-rw-r--r--.  1 jalal cs-grad 420411266 Mar 25 04:20 DHPE-RegularHG-01283493-Walid-BENBIHI.zip
drwxr-xr-x.  2 jalal cs-grad        54 Mar 25 04:21 DHPE-RegularHG-01283493-Walid-BENBIHI
drwxr-xr-x.  6 jalal cs-grad      4096 Mar 25 04:22 .
drwxr-xr-x.  2 jalal cs-grad       235 Mar 25 04:22 __pycache__
[jalal@goku hourglasstensorlfow]$ /scratch/sjn/anaconda/bin/python  inference.py 
/scratch/sjn/anaconda/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
[jalal@goku hourglasstensorlfow]$ ls
DHPE-RegularHG-01283493-Walid-BENBIHI      README.txt       config.cfg  dataset.txt                         hg_refined_200.index  opencv_ffmpeg.dll     pckhfULL.png     train_launcher.py
DHPE-RegularHG-01283493-Walid-BENBIHI.zip  YOLO_small.ckpt  config.py   filters.py                          hourglass_tiny.py     opencv_ffmpeg_64.dll  predictClass.py  v0.1
README.md                                  __pycache__      datagen.py  hg_refined_200.data-00000-of-00001  inference.py          p4frames.mat          timer.py         yolo_net.py
[jalal@goku hourglasstensorlfow]$ /scratch/sjn/anaconda/bin/python  predictClass.py 
/scratch/sjn/anaconda/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Traceback (most recent call last):
  File "predictClass.py", line 1283, in <module>
    predict = PredictProcessor(params)
  File "predictClass.py", line 63, in __init__
    self.HG = HourglassModel(nFeat= self.params['nfeats'], nStack = self.params['nstacks'], 
KeyError: 'nfeats'
[jalal@goku hourglasstensorlfow]$ 

Where is the test set

Hi,
i want to know if the dataset.txt contained all the trainning data, then where is the test set you used? Could you please present it?

Issue: missing meta file.

First of all,
Thanks :))))

I have a little problem.
When I downloaded the pre-train model file I can't found the hourglass_bn_100.meta file,
So how can I restore the model ?

Thanks a lot :)

train data

"(x_box_min y_box_min x_box_max y_box_max) Is the bounding box of the considered person in the scene"
but mpii dataset does not including those value, how to generate them properly?

Organize a tf_record dataset

What do you think of using a dataset fromat?
This is an example for the Mscoco object detection but could be also very similar for keypoints

heatmap resize to other size

I have a image (1024x768), first convert the image to 256x256, then execute predictHM(img) and get a 64x64 heat map.
In inference.py, the function predictHM(self, img) returns 64x64 heat map.
Can any function convert the heat map to original image size (ex: 1024x768)?

If the model is trained from scratch with the parameters shown in corresponding config.cfg

Hi, thanks for sharing your code!
I have a question that if the model 'hg_refined_tiny_200' was trained from scratch with the config.cfg as the bottom img shown, which was also downloaded together with the model file. If it is, why the model is not named 'hg_refined_100'

Deal with missing parts?

Hi, does the missing parts are ignored during training by weighted_loss (the weighted bce)? But it seems this is still work in progress, so does it mean the current model cannot be trained on data with missing parts?

Thanks! @wbenbihi

Performance On MPII

Hi， I am wondering if this tf implementation compete the orginial one? Did you ever test the performance on MPII dataset?
As my past experience , tensorflow implementations alwasy show a slightly reducement on performance . Could you share your test result on MPII ?
Thanks!

Preparing for v2

Given all the feedbacks I had on this repo, I decided to start a v2 (the graph architecture will be the same) to provide an easy-to-use module. I will be glad if you can tell me which improvement/feature should be taken care of.

PS: I keep working on this code on my spare time, so it won't be released soon

Cann't test/infer using inference.py

It seems like that the class HourglassModel in the hourglass_tiny.py doesn't have attribute 'pred_sigmoid'.But in the predictClass.py, class PredictProcessor call self.HG.pred_sigmoid.
@wbenbihi

Net Structure

Can you visualize the net structure?
I hope you can provide the figure of the net structure.
That will very helpful for me to understand the net. Thank you.

BatchNorm/beta not found error

Hello, thanks a lot for the tensorflow implementation of hourglass architecture. I have been trying to test the models and I am running into the below error during inference;

Tensor name "BatchNorm/beta/RMSProp" not found in checkpoint files /mnt//hourglass/hourglass/RegHG/hg_refined_200.index

Has anyone else encountered this issue? I would be grateful for any suggestions regarding how this could be solved. Thanks!

How to display result

Hi I use this code in inference.py to detect human estimation

from inference import Inference
a = Inference()
a.webcamSingle()

but it seems that there are no results. I have used different pretained model .
Can you give me an example how to use the inference.py?

I have also try the webcamYOLO and get the right results

Difference between RegularHG mode and TinyHG model

Hi, thanks for your work. I want to know the difference between RegularHG model and TinyHG model? Why TinyHG model is larger?
I notice that those two pre-trained models have the same 'config.cfg' file, does it mean that they are the same model? Is there something that I am missing?
I have checked the original paper, actually I did not found a clue about 'tiny'. Hoping someone can help me. Thanks a lot.

The testing set of MPII

Excuse me, I am a beginner for the pose estimation, may I ask what's the testing set that you generated the error graph for MPII?

test on single image

hi,
Very interesting work!

I just modified your PredictProcessor().mpeWebcam() to test your model on single image.
I only replace the inputs form video to single image, and nothing else.
It appears to me that the YOLO works well and can find where the person is, but the skeletons can not be detected, or all with very low confidence (<0.1).
Do you have a similar issue or can you update a script for testing on single image?

Thanks

self.HG = HourglassModel(nFeat= self.params['nfeats'], nStack = self.params['nstacks'], KeyError: 'nfeats'

I previously mentioned this error here but got no response. I followed the steps you mentioned and downloaded the pre-trained weight into the main directory and got the keyerror as you see below. Can you please suggest a solution?
#29

[jalal@goku hourglasstensorlfow]$ vi predictClass.py 
[jalal@goku hourglasstensorlfow]$ ls
000003072.jpg                              config.cfg                          hourglass_tiny.pyc    sjn-test-include-from-cwd.py
005808361.jpg                              config.py                           inference.py          sjn-test-include-from-explicit-file.py
DHPE-RegularHG-01283493-Walid-BENBIHI      datagen.py                          inference.pyc         test.py
DHPE-RegularHG-01283493-Walid-BENBIHI.zip  dataset.txt                         mona.jpg              timer.py
README.md                                  filters.py                          opencv_ffmpeg.dll     train_launcher.py
README.txt                                 hg_refined_200.data-00000-of-00001  opencv_ffmpeg_64.dll  v0.1
YOLO_small.ckpt                            hg_refined_200.index                p4frames.mat          yolo_net.py
__pycache__                                hg_tiny                             pckhfULL.png
another_test.py                            hourglass_tiny.py                   predictClass.py
[jalal@goku hourglasstensorlfow]$ python predictClass.py 
/scratch/sjn/anaconda/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6
  return f(*args, **kwds)
/scratch/sjn/anaconda/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Traceback (most recent call last):
  File "predictClass.py", line 1283, in <module>
    predict = PredictProcessor(params)
  File "predictClass.py", line 63, in __init__
    self.HG = HourglassModel(nFeat= self.params['nfeats'], nStack = self.params['nstacks'], 
KeyError: 'nfeats'

where can I find out the modification report

I've been wondering the details about the result curves in pckhfuLL.png,
and i think the answer may be in the report which is mentioned at line 23 in hourglass_tiny.py.

Who can help me,please? Thanks

bug?

@wbenbihi
line#472 at hourglass_tiny.py
sum_[i] = tf.add_n([out_[i], sum_[i-1], ll_[0]], name= 'merge')

should be
ll_[i] instead of ll_[0]