hollance/coreml-custom-layers

Source code for the blog post "Custom Layers in Core ML"

Hi there 👋

I'm an audio software developer, working on my own plug-ins as well as freelance for other audio companies.

Most recently I've been employed as a machine learning engineer and as an iOS developer. These days I mostly write C++ audio code. I've been professionally active as a software developer since the mid 1990s.

I have also written and co-written a number of books (most notably The iOS Apprentice and Core ML Survival Guide) and have published my own apps and games as an indie developer.

• Creating Synthesizer Plug-Ins with C++ and JUCE — my latest book
• My audio programming blog
• My machine learning blog

coreml-custom-layers's Issues

Why i enabled Metal API in encode function but my Coreml custom layer still run on CPU

Hello, @hollance .

I implement a custom pytorch layer on both CPU and GPU following your amazing blog. The cpu version works good, but when i implemented this op on GPU it cannot activate "encode" function. Always run on CPU. I have checked the coremltools.convert() options with compute_units=coremltools.ComputeUnit.CPU_AND_GPU, but it still doesn't work. Similar problems also mentioned in https://stackoverflow.com/questions/51019600/why-i-enabled-metal-api-but-my-coreml-custom-layer-still-run-on-cpu and https://developer.apple.com/forums/thread/695640. Any idea on help this would be grateful.

System Information

• mac OS: 11.6.1 Big Sur
• xcode: 12.5.1
• coremltools: 5.1.0
• test device: iphone 11

Update: I test your GPU code example, but it seems also execute in CPU ComputeUnit. Output are show below:

2022-01-12 17:04:02.593909+0800 CustomLayers[1515:575421] Metal GPU Frame Capture Enabled
2022-01-12 17:04:02.594216+0800 CustomLayers[1515:575421] Metal API Validation Enabled
init(parameters:) ["beta": 0.01, "engineName": Swish]
outputShapes(forInputShapes:) [[0, 0, 0, 0, 0]]
outputShapes(forInputShapes:) [[0, 0, 0, 0, 0]]
init(parameters:) ["beta": 0.01, "engineName": Swish]
init(parameters:) ["engineName": Swish, "beta": 0.01]
outputShapes(forInputShapes:) [[0, 0, 0, 0, 0]]
outputShapes(forInputShapes:) [[0, 0, 0, 0, 0]]
init(parameters:) ["beta": 0.01, "engineName": Swish]
setWeightData(_:) []
outputShapes(forInputShapes:) [[1, 1, 6, 256, 256]]
outputShapes(forInputShapes:) [[1, 1, 6, 256, 256]]
outputShapes(forInputShapes:) [[1, 1, 6, 256, 256]]
evaluate(inputs:outputs:) 1 1
Double 10 vector
[0.3277947902679443,0.03130210936069489,0.6019284725189209,0.001807634020224214,0.03013107925653458,0.001237826887518167,0.0006207154947333038,0.002744386438280344,0.001118888612836599,0.001314118970185518]

Lambda layer with multiple inputs?

Hi. Thank you for the useful project. I encounter an issue when lambda layer receives multiple inputs.

# This is a custom activation function.
def swish(x,s1):
    return K.sigmoid(x)*x

def merge(x):
    return x[0]+x[1]

# Create a silly model that has our custom activation function as a new layer.
def create_model():
    a = 3
    inp = Input(shape=(256, 256, 3))
    x = Conv2D(6, (3, 3), padding="same")(inp)
    #x = Activation(swish)(x)                   # doesn't work! :-(
    x = Lambda(swish, arguments={'s1':a})(x)
    x = GlobalAveragePooling2D()(x)
    x0 = Dense(10, activation="softmax")(x)
    x1 = Dense(10, activation="softmax")(x0)
    x = Lambda(merge)([x0,x1])
    return Model(inp, x)

There will be an error.

Traceback (most recent call last):
  File "test.py", line 98, in <module>
    custom_conversion_functions={ "Lambda": convert_lambda })
  File "/home/shamangary/anaconda3/envs/new/lib/python3.5/site-packages/coremltools/converters/keras/_keras_converter.py", line 745, in convert
    custom_conversion_functions=custom_conversion_functions)
  File "/home/shamangary/anaconda3/envs/new/lib/python3.5/site-packages/coremltools/converters/keras/_keras_converter.py", line 543, in convertToSpec
    custom_objects=custom_objects)
  File "/home/shamangary/anaconda3/envs/new/lib/python3.5/site-packages/coremltools/converters/keras/_keras2_converter.py", line 191, in _convert
    graph.build()
  File "/home/shamangary/anaconda3/envs/new/lib/python3.5/site-packages/coremltools/converters/keras/_topology2.py", line 643, in build
    successors = self.edge_map[layer]
KeyError: 'lambda_2'

I was trying much more complex merging function other than just addition.
Do you know how to resolve this? Thank you!

what if the custom layer's input is five dimensional

like this [1, 1, 3, 64, 64]
how to write the shader when
texture2d_array<half, access::read> inTexture [[texture(0)]],
uses texture2d or is there a texture5d ?

keras to CoreML with custom loss function

Hi,

Thank you so much for sharing this useful code with us.
I am trying to convert a keras code to core ML.
thia is the source code Im working on.
https://github.com/sbillburg/CRNN-with-STN
this is an OCR written in keras and with CRNN architecture.
in this code, loss is an custm layer written by LAMBDA layer.
I tried your code on this but it keeps raising error.

I was wondering do you have a workaround code with the loss function or any simpler approach.

Thanks for taking the time

Only channel and sequence concatenation are supported.

Hello again:)
I truly appreciate your help and am very sorry taking your time.
your input toward this error will be very helpful to me.

the error raises here while it wanted to concatenate:
rnn2_merged = concatenate([rnn_2, rnn_2b])

I was wondering do you have any solution for this or any workaround or idea.
I was searching the whole day yesterday but really I could not find any useful link regarding this error how to figure it out.

this is the error:

Traceback (most recent call last):
  File "/home/sgnbx/Downloads/projects/CRNN-with-STN-master/CRNN_with_STN.py", line 201, in <module>
    custom_conversion_functions={"Lambda": convert_lambda},
  File "/home/sgnbx/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/coremltools/converters/keras/_keras_converter.py", line 760, in convert
    custom_conversion_functions=custom_conversion_functions)
  File "/home/sgnbx/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/coremltools/converters/keras/_keras_converter.py", line 556, in convertToSpec
    custom_objects=custom_objects)
  File "/home/sgnbx/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/coremltools/converters/keras/_keras2_converter.py", line 359, in _convert
    converter_func(builder, layer, input_names, output_names, keras_layer)
  File "/home/sgnbx/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/coremltools/converters/keras/_layers2.py", line 636, in convert_merge
    mode = _get_elementwise_name_from_keras_layer(keras_layer)
  File "/home/sgnbx/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/coremltools/converters/keras/_layers2.py", line 92, in _get_elementwise_name_from_keras_layer
    raise ValueError('Only channel and sequence concatenation are supported.')
ValueError: Only channel and sequence concatenation are supported.
28 : concatenate_1, <keras.layers.merge.Concatenate object at 0x7f237c23b908>

Again thank you very much for taking the time Its much appreciated.

Running into Error keras functional is not supported

TypeError: Keras layer of type <class 'tensorflow.python.keras.engine.functional.Functional'> is not supported.