emergentorder/onnx-scala

An ONNX (Open Neural Network eXchange) API and backend for typeful, functional deep learning and classical machine learning in Scala 3

onnx-scala's Issues

why not use onnxruntime's Java API?

Hi,

This is just a question. Feel free to close it.

It seems that onnxruntime has a Java API since 1.3.0.
My question is whether there is a plan to adopt the native API.
Or are there other good reasons to use javacpp-presets?

Best,
Zhenhao

object compiletime is not a member of package org.emergentorder

SBT cannot compile example code:

Do I need to add published modules to my build dependencies? I tried with other modules, but same issue:

import $ivy.`org.emergent-order::onnx-scala-backends:0.13.0`
import $ivy.`org.emergent-order::onnx-scala-common:0.13.0`
import $ivy.`org.emergent-order::onnx-scala:0.13.0`

Scastie example:
https://scastie.scala-lang.org/novakov-alexey/Eg9Hn0dVRsSNCt3OwzmdMg/1

Program generator dose'nt work for convert onnx file to scala

process which convert onnx file to scala was failed When i execute "run [xxxx].onnx" on programGeneratorJVM .

How can i fix?
[Error]

[error] (run-main-2) java.lang.ClassCastException: scala.collection.immutable.$colon$colon cannot be cast to java.lang.String
[error] java.lang.ClassCastException: scala.collection.immutable.$colon$colon cannot be cast to java.lang.String
[error] at org.emergentorder.onnx.ONNXProgramGenerator$.$anonfun$main$17(ONNXProgramGenerator.scala:199)
[error] at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
[error] at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
[error] at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
[error] at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
[error] at scala.collection.TraversableLike.map(TraversableLike.scala:237)
[error] at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
[error] at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
[error] at org.emergentorder.onnx.ONNXProgramGenerator$.$anonfun$main$6(ONNXProgramGenerator.scala:194)
[error] at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
[error] at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
[error] at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
[error] at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
[error] at scala.collection.TraversableLike.map(TraversableLike.scala:237)
[error] at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
[error] at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
[error] at org.emergentorder.onnx.ONNXProgramGenerator$.fullSource$1(ONNXProgramGenerator.scala:155)
[error] at org.emergentorder.onnx.ONNXProgramGenerator$.generate$1(ONNXProgramGenerator.scala:296)
[error] at org.emergentorder.onnx.ONNXProgramGenerator$.main(ONNXProgramGenerator.scala:303)
[error] at org.emergentorder.onnx.ONNXProgramGenerator.main(ONNXProgramGenerator.scala)
[error] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error] at java.lang.reflect.Method.invoke(Method.java:498)

sbt console tab completion broken after importing io.kjaer.compiletime._

To reproduce:

1. run sbt backendsJVM/console
2. once console has loaded, enter import io.kjaer.compiletime._
3. type 1. followed by the tab key (1, period, tab) to show completions on value 1

This results in the error pasted below, but with the likely interesting part here:

Toplevel definition <init> is defined in .../io/kjaer/compiletime/dependent$package.class and also in .../io/kjaer/compiletime/IndicesOf$package.class
One of these files should be removed from the classpath.

This is the only related commentary i could find upstream: scala/scala3#11546 (comment)

---

info [jline] Error while finding completion candidates                               
dotty.tools.dotc.core.TypeError: Toplevel definition <init> is defined in
  /Users/andrewvalencik/grabbed/onnx-scala/common/.jvm/target/scala-3.0.0-RC1/classes/io/kjaer/compiletime/dependent$package.class
and also in
  /Users/andrewvalencik/grabbed/onnx-scala/common/.jvm/target/scala-3.0.0-RC1/classes/io/kjaer/compiletime/IndicesOf$package.class
One of these files should be removed from the classpath.
        at dotty.tools.dotc.core.SymDenotations$PackageClassDenotation.dropStale$1(SymDenotations.scala:2330)
        at dotty.tools.dotc.core.SymDenotations$PackageClassDenotation.recur$1(SymDenotations.scala:2294)
        at dotty.tools.dotc.core.SymDenotations$PackageClassDenotation.computeMembersNamed(SymDenotations.scala:2350)
        at dotty.tools.dotc.core.SymDenotations$ClassDenotation.membersNamed(SymDenotations.scala:1914)
        at dotty.tools.dotc.core.SymDenotations$ClassDenotation.findMember(SymDenotations.scala:1965)
        at dotty.tools.dotc.core.Types$Type.go$1(Types.scala:655)
        at dotty.tools.dotc.core.Types$Type.findMember(Types.scala:843)
        at dotty.tools.dotc.core.Types$Type.memberBasedOnFlags(Types.scala:638)
        at dotty.tools.dotc.core.Types$Type.member(Types.scala:622)
        at dotty.tools.dotc.interactive.Completion$CompletionBuffer.addMember(Completion.scala:292)
        at dotty.tools.dotc.interactive.Completion$CompletionBuffer.addImportCompletions$$anonfun$3(Completion.scala:364)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
        at scala.collection.IterableOps$WithFilter.foreach(Iterable.scala:889)
        at dotty.tools.dotc.interactive.Completion$CompletionBuffer.addImportCompletions(Completion.scala:364)
        at dotty.tools.dotc.interactive.Completion$CompletionBuffer.addScopeCompletions(Completion.scala:192)
        at dotty.tools.dotc.interactive.Completion$CompletionBuffer.addExtensionCompletions(Completion.scala:249)
        at dotty.tools.dotc.interactive.Completion$CompletionBuffer.addSelectionCompletions(Completion.scala:275)
        at dotty.tools.dotc.interactive.Completion$.computeCompletions(Completion.scala:122)
        at dotty.tools.dotc.interactive.Completion$.completions(Completion.scala:48)
        at dotty.tools.repl.ReplDriver.completions$$anonfun$1(ReplDriver.scala:188)
        at scala.util.Either.map(Either.scala:382)
        at dotty.tools.repl.ReplDriver.completions(ReplDriver.scala:189)
        at dotty.tools.repl.ReplDriver.$anonfun$3(ReplDriver.scala:113)
        at org.jline.reader.impl.LineReaderImpl.doComplete(LineReaderImpl.java:4394)
        at org.jline.reader.impl.LineReaderImpl.doComplete(LineReaderImpl.java:4360)
        at org.jline.reader.impl.LineReaderImpl.expandOrComplete(LineReaderImpl.java:4299)
        at org.jline.reader.impl.LineReaderImpl$1.apply(LineReaderImpl.java:3793)
        at org.jline.reader.impl.LineReaderImpl.readLine(LineReaderImpl.java:665)
        at org.jline.reader.impl.LineReaderImpl.readLine(LineReaderImpl.java:454)
        at dotty.tools.repl.JLineTerminal.readLine(JLineTerminal.scala:71)
        at dotty.tools.repl.ReplDriver.readLine$1(ReplDriver.scala:118)
        at dotty.tools.repl.ReplDriver.loop$1(ReplDriver.scala:128)
        at dotty.tools.repl.ReplDriver.runUntilQuit$$anonfun$1(ReplDriver.scala:133)
        at dotty.tools.repl.ReplDriver.withRedirectedOutput(ReplDriver.scala:152)
        at dotty.tools.repl.ReplDriver.runUntilQuit(ReplDriver.scala:133)
        at xsbt.ConsoleInterface.run(ConsoleInterface.java:52)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)

Platform not supported exception

Hi,

I am trying to setup ONNX-scala on my windows machine. I am getting the following error:
ONNX Model Evaluation Sample
Exception in thread "main" java.lang.reflect.InvocationTargetException

Caused by: java.lang.UnsatisfiedLinkError: Platform "windows-x86_64" not supported by class org.bytedeco.ngraph.Backend
at org.bytedeco.javacpp.Loader.load(Loader.java:1081)
at org.bytedeco.javacpp.Loader.load(Loader.java:1042)
at org.bytedeco.ngraph.Backend.(Backend.java:19)
at org.emergentorder.onnx.backends.NGraphOperatorBackend$class.$init$(NGraphOperatorBackend.scala:43)
at org.emergentorder.onnx.backends.NGraphModelBackend.(NGraphModelBackend.scala:11)
at Scripts.SampleScripts.OnnxModelEvaluation$.main(OnnxModelEvaluation.scala:16)
at Scripts.SampleScripts.OnnxModelEvaluation.main(OnnxModelEvaluation.scala)

Hence, I just wanted to check if Windows is not supported or if there is any workaround for running this in a Windows environment

Accessing multiple outputs

Hi, firstly thanks for this application, its much appreciated

Im struggling to run an onnx model which uses more than a single output

I can get the first output values logits which has a dimension of (batch_size * 86 * 1) fine with the following code, where I am using a single sample example with an input dimension of (batch_size * num_words), the last dimension being an array of word tokens ids (integers) which in this case is 3 for this example but is dynamic

val modelBytes = Files.readAllBytes(Paths.get(onnxPath))
val model = new ORTModelBackend(modelBytes)
val data =  Array.fill(1 * 3){10L}
val shape = 1 #: 3 #: SNil
val tsr = Tensor(data, shape)
val out = model.fullModel[Float,
        "logits",
        "batch_size" ##: "n_labels" ##: TSNil,
        1 #: 86 #: SNil](Tuple(tsr))

val output: Array[Float] = out.data.unsafeRunSync()
println(output.mkString(", "))
\\ prints 0.05673475, -3.656492, -1.9948144, -1.2033991, -0.457706, 0.15737924,

But the second output alphas doesn't seem accessible, it has dimensions (batch_size * 86 * num_words) or 1 * 86 * 3 for this example. I have tried running the model again to get the alphas, but it doesn't seem to work

val out = model.fullModel[Float,
    "alphas",
    "batch_size" ##: "n_labels" ##: "n_words" ##: TSNil,
    1 #: 86 #: 3 #: SNil](Tuple(tsr))

It raises a bunch of errors (which I suspect are due to incorrect shape in the spec)

java.lang.IllegalArgumentException: requirement failed

	at scala.Predef$.require(Predef.scala:324)
	at org.emergentorder.onnx.backends.ORTOperatorBackend.$anonfun$3(ORTOperatorBackend.scala:68)
	at flatten @ org.emergentorder.onnx.backends.ORTModelBackend.inputTensors$1$$anonfun$1(ORTModelBackend.scala:66)
	at blocking @ org.emergentorder.onnx.backends.ORTOperatorBackend.runModel(ORTOperatorBackend.scala:63)
	at make @ org.emergentorder.onnx.backends.ORTModelBackend.fullModel(ORTModelBackend.scala:75)
	at make @ org.emergentorder.onnx.backends.ORTModelBackend.fullModel(ORTModelBackend.scala:75)
	at use @ org.emergentorder.onnx.backends.ORTModelBackend.fullModel(ORTModelBackend.scala:83)
	at flatMap @ org.emergentorder.onnx.backends.ORTOperatorBackend.runModel(ORTOperatorBackend.scala:80)
	at flatten @ org.emergentorder.onnx.backends.ORTModelBackend.inputTensors$1$$anonfun$1(ORTModelBackend.scala:66)

I guess the most confusing part (besides this being my first foray into Scala!) is that the second parameter in the specification (which I assumed was the name of the output) seems to accept any string and access the first output logits, e.g.

val out = model.fullModel[Float,
        "what am I",
        "batch_size" ##: "n_labels" ##: TSNil,
        1 #: 86 #: SNil](Tuple(tsr))

works fine and returns the logits output

Many thanks in advance!

Quick Questions

Hello, I glad to have found this project and want to see if it can help an Apache Flink pipeline I have using Microsoft's ONNX runtime. I have a few questions, and I'm hoping you can answer them and/or add some additional documentation. If you don't have time I understand.

Specifically, I am wanting to see if it can improve throughput and negate some memory problems. Have you considered measuring the throughput and comparing with alternatives (see https://sites.bu.edu/casp/files/2022/05/Horchidan22Evaluating.pdf)? The OrtModelBackend does not look thread-safe; are there any other things I should watch out for?

Question: why does `Tensor#apply` return a value wrapped in `IO` effect?

Why does Tensor#apply return a value wrapped in IO effect?

Is there any resource allocation or unsafe invocation to native land?

Errors When Building Project

When I clone the project and try to build it using sbt +publishLocal as in the README, I get the following errors:

[error] (coreJS / Compile / compileIncremental) java.lang.StackOverflowError
[error] (coreJVM / Compile / compileIncremental) java.lang.StackOverflowError
[error] (coreNative / Compile / compileIncremental) java.lang.StackOverflowError
[error] (coreDottyJVM / update) sbt.librarymanagement.ResolveException: unresolved dependency: org.emergentorder.onnx#onnx-scala-common_0.10;1.3.0-0.1.0-SNAPSHOT: not found

Any suggestions on how to fix this would be appreciated!

Is it possible to pass batch size dynamically?

My batch size is 32, but the very last batch can be smaller, for example 27 images.
Is it possible to make batch as a variable and pass it to model inference method?

val batch = calcBatchSize()
// below line won't compile
val out = model.fullModel[Float, "ImageClassification", "Batch" ##: "Features" ##: TSNil, batch #: 512 #: SNil](
  Tuple(input)
)