keras-team / keras-core Goto Github PK
View Code? Open in Web Editor NEWA multi-backend implementation of the Keras API, with support for TensorFlow, JAX, and PyTorch.
License: Apache License 2.0
A multi-backend implementation of the Keras API, with support for TensorFlow, JAX, and PyTorch.
License: Apache License 2.0
First non-trivial issue encountered in keras.io example conversion: text_classification_from_scratch
doesn't train. Loss / accuracy can't get above random baseline. It reliably trains in tf.keras.
Main suspect: embedding layer?
Especially when jit_compile=False
, Keras Core BN is significantly slower than tf.keras BN. Need to fix.
Should work for torch tensors as well -- we should start adding data adapter tests targeting torch tensors (in a new PR)
Originally posted by @fchollet in #247 (comment)
I tried cloning the repo and following the instructions on the README.md, i.e.:
python3 pip_build.py --install
but this failed because namex
was not installed. So perhaps we're missing a step like:
pip install -r requirements.txt
We may want to also add the git clone
step, although that may be obvious.
Lastly, adding a tiny example on the home page would be nice. I made the mistake of importing keras
instead of keras_core
, it may seem stupid but I suspect this will be a common error, so seeing an example with import keras_core
would help. Moreover, it would be nice to indicate whether or not import keras_core as keras
is recommended (to port code easily) or frowned upon (because of the risk of conflict with the original keras library).
Perhaps something like this:
import keras_core as keras
import numpy as np
X = np.random.rand(1000, 5)
Y = np.random.rand(1000, 2)
model = keras.Sequential([
keras.layers.Dense(10, activation="relu"),
keras.layers.Dense(2),
])
model.compile(loss="mse", optimizer="nadam")
history = model.fit(X, Y)
Y_pred = model.predict(X)
The export KERAS_BACKEND="jax"
may confuse Windows users. Perhaps give instructions for them as well.
keras_core.utils.normalize
and keras_core.utils.to_categorical
not available, only keras_core.utils.numerical_utils
. Solution is probably to add to keras_core/utils/__init__.py
from keras_core.utils.numerical_utils import normalize
from keras_core.utils.numerical_utils import to_categorical
But this gives issues with circular import.
Participants:
See proposed assignments below.
Some of the assignment buckets may not last the week -- we will reshuffle assignments as we make progress. If we reach a sufficient degree of feature completeness mid-week, we can move on to backwards compatibility testing.
Layers:
Wrapped layers:
Callbacks:
keras_core/optimizers/schedules/
)Callbacks:
Layers:
Wrapped layers:
keras_core/backend/pytorch/numpy.py
Layers:
Layers:
Layers:
Metrics
Callbacks:
Layers:
At this time it seems that @ariG23498, @grasskin, and @AakashKumarNain are working on losses or metrics, which can lead to some collisions. Let's split the work:
This includes both plain functions and their wrapper classes. All objects should have unit tests and should have a complete docstring (for the most part you can just copy the Keras docstring and remove the TF references, etc.).
For loss functions that are redundant with metrics, the plain function should live in losses/
and be imported in metrics/
. The class-based wrapper should exist both as a Loss in losses/
and as a Metric in metrics/
.
Shouldn't the mean_squared_error
return the mean values? The loss function currently returns (y_true - y_pred) ** 2
but shouldn't it be ops.mean((y_true - y_pred) ** 2)
instead? An example to illustrate the same
y_true = np.array([[0., 1.], [0., 0.]])
y_pred = np.array([[1., 1.], [1., 0.]])
loss = MeanSquaredError()
loss(y_true, y_pred) # Returns 1.0 but should return 0.5
I have implemented the batch_dot(x1, x2, axes)
operation but I am not sure what's the right module for it. Should we include this in ops.numpy
?
PS: This is required for implementing the Dot
layer
Probabilistic Metrics
The current table produced by model.summary()
needs reformatting. Here is how it looks like right now:
PS: Can we try to use Rich tables?
TBD
import numpy as np
from keras_core import layers
from keras_core import metrics
from keras_core.models import Sequential
def make_model():
model = Sequential(
[
layers.Dense(4, activation="relu"),
layers.Dense(2, activation="softmax"),
]
)
model.compile(
loss="categorical_crossentropy",
optimizer="sgd",
# metrics=[metrics.Accuracy()] # Works
# metrics=["acc"] # Fails
metrics=["accuracy"] # Fails
)
return model
x_train = np.random.random((10, 3))
y_train = np.random.random((10, 2))
x_test = np.random.random((10, 3))
y_test = np.random.random((10, 2))
model = make_model()
model.fit(
x_train,
y_train,
batch_size=4,
validation_data=(x_test, y_test),
epochs=1,
)
When using acc
or accuracy
with categorical_crossentropy
it fails with error that categorical_accuracy
not found. But works fine when using Accuracy()
metric -
File "keras-core/keras_core/backend/tensorflow/trainer.py", line 107, in one_step_on_iterator *
outputs = self.distribute_strategy.run(
File "keras-core/keras_core/backend/tensorflow/trainer.py", line 97, in one_step_on_data *
return self.train_step(data)
File "keras-core/keras_core/backend/tensorflow/trainer.py", line 68, in train_step *
return self.compute_metrics(x, y, y_pred, sample_weight=sample_weight)
File "keras-core/keras_core/trainers/trainer.py", line 201, in compute_metrics *
self._compile_metrics.update_state(y, y_pred, sample_weight)
File "keras-core/keras_core/trainers/compile_utils.py", line 301, in update_state *
self.build(y_true, y_pred)
File "keras-core/keras_core/trainers/compile_utils.py", line 170, in build *
self._flat_metrics = self._build_metrics_set(
File "keras-core/keras_core/trainers/compile_utils.py", line 209, in _build_metrics_set *
flat_metrics.append(
File "/keras-core/keras_core/trainers/compile_utils.py", line 74, in get_metric *
metric_obj = metrics_module.categorical_accuracy
AttributeError: module 'keras_core.metrics' has no attribute 'categorical_accuracy'
Playing with custom models, I ran into the following issue, while loading the custom model (same error on all 3 backends):
import keras_core as keras
import numpy as np
class MyCustomModel(keras.Model):
def __init__(self, units=30, **kwargs):
super().__init__(**kwargs)
self.units = units
def build(self, batch_input_shape):
self.hidden1 = keras.layers.Dense(self.units)
def call(self, inputs):
return self.hidden1(inputs)
def get_config(self):
return {"units": self.units}
@classmethod
def from_config(cls, config):
return cls(**config)
model = MyCustomModel(10)
model.compile(loss="mse", optimizer="nadam")
X = np.random.rand(100, 5)
Y = np.random.rand(100, 10)
history = model.fit(X, Y)
model.save("my_custom_model.keras")
loaded_model = keras.models.load_model( #!!! IndexError: list index out of range
"my_custom_model.keras",
custom_objects={"MyCustomModel": MyCustomModel}
)
Below is the full stacktrace:
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
Cell In[1], line 28
26 history = model.fit(X, Y)
27 model.save("my_custom_model.keras")
---> 28 loaded_model = keras.models.load_model(
29 "my_custom_model.keras",
30 custom_objects={"MyCustomModel": MyCustomModel}
31 )
File ~/opt/miniconda3/envs/kerascore/lib/python3.8/site-packages/keras_core/src/saving/saving_api.py:80, in load_model(filepath, custom_objects, compile, safe_mode)
77 is_keras_zip = True
79 if is_keras_zip:
---> 80 return saving_lib.load_model(
81 filepath,
82 custom_objects=custom_objects,
83 compile=compile,
84 safe_mode=safe_mode,
85 )
86 else:
87 raise ValueError(
88 "The file you are trying to load does not appear to "
89 "be a valid `.keras` model file. If it is a legacy "
90 "`.h5` or SavedModel file, do note that these formats "
91 "are not supported in Keras Core."
92 )
File ~/opt/miniconda3/envs/kerascore/lib/python3.8/site-packages/keras_core/src/saving/saving_lib.py:184, in load_model(filepath, custom_objects, compile, safe_mode)
182 # Construct the model from the configuration file in the archive.
183 with ObjectSharingScope():
--> 184 model = deserialize_keras_object(
185 config_dict, custom_objects, safe_mode=safe_mode
186 )
188 all_filenames = zf.namelist()
189 if _VARS_FNAME + ".h5" in all_filenames:
File ~/opt/miniconda3/envs/kerascore/lib/python3.8/site-packages/keras_core/src/saving/serialization_lib.py:678, in deserialize_keras_object(config, custom_objects, safe_mode, **kwargs)
676 compile_config = config.get("compile_config", None)
677 if compile_config:
--> 678 instance.compile_from_config(compile_config)
679 instance.compiled = True
681 if "shared_object_id" in config:
File ~/opt/miniconda3/envs/kerascore/lib/python3.8/site-packages/keras_core/src/trainers/trainer.py:684, in Trainer.compile_from_config(self, config)
681 self.compile(**config)
682 if hasattr(self, "optimizer") and self.built:
683 # Create optimizer variables.
--> 684 self.optimizer.build(self.trainable_variables)
File ~/opt/miniconda3/envs/kerascore/lib/python3.8/site-packages/keras_core/src/optimizers/nadam.py:82, in Nadam.build(self, var_list)
80 self._momentums = []
81 self._velocities = []
---> 82 self._u_product = backend.Variable(1.0, dtype=var_list[0].dtype)
84 for var in var_list:
85 self._momentums.append(
86 self.add_variable_from_reference(
87 reference_variable=var, name="m"
88 )
89 )
IndexError: list index out of range
If the Dense
layer is created in the constructor instead of the build()
method, then everything works fine.
Running python demo_functional.py
fails with error -
File "keras-core/keras_core/losses/loss.py", line 69, in squeeze_to_same_rank *
x2_rank = len(x2.shape)
ValueError: Cannot take the length of shape with unknown rank.
The Dropout layer throws an error 'list' object has no attribute 'dtype'
when run with jit and StatelessScope enabled. It looks like the seed's state value isn't getting initialized properly.
Here's an MRE:
import keras_core
from keras_core import backend
from keras_core.operations import numpy as knp
from jax import numpy as jnp
import jax
x = knp.array([0.1, 0.2, 0.3])
@jax.jit
def train_step(x):
with backend.StatelessScope():
x = keras_core.layers.Dropout(rate=0.1)(x, training=True)
return x
keras_core.utils.traceback_utils.disable_traceback_filtering()
x = train_step(x)
assert isinstance(x, jnp.ndarray)
The model trains fine with TF. With JAX, the code runs but trains rather poorly. Need to find out root cause.
In addition, in this example, a model trained with TF cannot be reloaded with JAX.
Something we should do soon is make sure the base `Trainer` class has docstrings everywhere, that the Trainer subclasses don't have docstrings, then we programatically set the docstrings on the subclasses (using the ones from the base class)
Originally posted by @fchollet in #328 (comment)
Most Torch numpy ops have been successfully completed. However, we need full implementations of these ops to be functionally equivalent to the numpy ops.
torch.full
and torch.full_like
--> implement support for an array being passed in fill_value
arg, maybe use np.tile
?torch.linspace
and torch.logspace
--> implement support for arrays being passed in start
and stop
argstorch.take
--> resolve the incompatibility in dimensions between the result of np.take
and torch.take
torch.pad
--> implement padding with non-constant padding for the symmetric
and reflect
modestorch.split
--> resolve the arg divergence between np.split
and torch.split
that produces different resultstorch.trace
--> implement torch.trace
support for arguments offset
, axis1
, axis2
torch.eye
--> implement support for k
diagonal arg that exists in np.eye
but not torch.eye
torch.tri
--> create an implementation for scratch based on the logic of np.tri
, as the torch.tri
API does not existTo start implementing these:
numpy_test.py
. This will let you see the error to debug.Thank you and happy hacking!
Still playing with custom models. Encountered this issue on the jax backend (works fine on other backends):
import keras_core as keras
import numpy as np
class MyCustomModel(keras.Model):
def __init__(self, units, **kwargs):
super().__init__(**kwargs)
self.units = units
self.hidden1 = keras.layers.Dense(self.units)
def call(self, inputs):
return self.hidden1(inputs)
def get_config(self):
return {"units": self.units}
@classmethod
def from_config(cls, config):
return cls(**config)
X = np.random.rand(100, 3)
Y = np.random.rand(100, 2)
model = MyCustomModel(2)
model.build(X.shape) # issue disappears if you comment out this line
model.compile(loss="mse", optimizer="nadam")
history = model.fit(X, Y) #!!! IndexError: list index out of range
The issue disappears if you comment out the model.build(X.shape)
line.
Below is the full stacktrace:
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
Cell In[1], line 26
24 model.build(X.shape)
25 model.compile(loss="mse", optimizer="nadam")
---> 26 history = model.fit(X, Y)
File ~/opt/miniconda3/envs/kerascore/lib/python3.8/site-packages/keras_core/src/utils/traceback_utils.py:122, in filter_traceback.<locals>.error_handler(*args, **kwargs)
119 filtered_tb = _process_traceback_frames(e.__traceback__)
120 # To get the full stack trace, call:
121 # `keras_core.config.disable_traceback_filtering()`
--> 122 raise e.with_traceback(filtered_tb) from None
123 finally:
124 del filtered_tb
File ~/opt/miniconda3/envs/kerascore/lib/python3.8/site-packages/keras_core/src/optimizers/nadam.py:82, in Nadam.build(self, var_list)
80 self._momentums = []
81 self._velocities = []
---> 82 self._u_product = backend.Variable(1.0, dtype=var_list[0].dtype)
84 for var in var_list:
85 self._momentums.append(
86 self.add_variable_from_reference(
87 reference_variable=var, name="m"
88 )
89 )
IndexError: list index out of range
Issue with https://keras.io/examples/vision/object_detection_using_vision_transformer/ conversion.
Model checkpoint filepath used to be able to take a directory like this:
checkpoint_filepath = "logs/"
checkpoint_callback = keras.callbacks.ModelCheckpoint(
checkpoint_filepath,
monitor="val_loss",
save_best_only=True,
save_weights_only=True,
)
Currently it requires a filepath that must end in .weights.h5
.
We should have some golden number comparison for the major workflow like model.fit/eval/predict.
The baseline can be an application model from keras, with fixed weights eg, imagenet pretrained weights.
Are we going to implement reduce_
operations in ops
? This includes the following:
Looks like the clear_session
public API is exported, but its not accessible via keras_core.backend.clear_session
.
https://github.com/keras-team/keras-core/blob/main/keras_core/backend/common/global_state.py#L36
The Colab displays the learning rate from the optimizer through a custom metric.
Repro steps:
Epoch 1/5 val_lr_metric: 0.0100
Epoch 2/5 val_lr_metric: 0.0100
Epoch 3/5 val_lr_metric: 0.0100
Epoch 1/5 val_lr_metric: 0.0060
Epoch 2/5 val_lr_metric: 0.0036
Epoch 3/5 val_lr_metric: 0.0022
Running some tests with multi-input models, I came across the following issue:
import keras_core as keras
import numpy as np
class MultiInputModel(keras.Model):
def call(self, inputs):
a, b = inputs
return a + b
model = MultiInputModel()
x1, x2 = np.random.rand(2, 3, 4)
y_pred1 = model((x1, x2)) #works fine
y_pred2 = model.predict((x1, x2)) #!!! ValueError (only with jax backend)
The error is:
ValueError: Exception encountered when calling MultiInputModel.call().
not enough values to unpack (expected 2, got 1)
Arguments received by MultiInputModel.call():
• inputs=(('jnp.ndarray(shape=(3, 2), dtype=float32)', 'jnp.ndarray(shape=(3, 2), dtype=float32)'),)
Full stack trace:
ValueError Traceback (most recent call last)
Cell In[1], line 11
9 model = MultiInputModel()
10 x1, x2 = np.random.rand(2, 3, 2)
---> 11 model.predict((x1, x2))
File ~/opt/miniconda3/envs/kerascore/lib/python3.8/site-packages/keras_core/src/backend/jax/trainer.py:560, in JAXTrainer.predict(self, x, batch_size, verbose, steps, callbacks)
557 for _, data in epoch_iterator.enumerate_epoch(return_type="np"):
558 # Build model
559 with backend.StatelessScope():
--> 560 self(data[0])
561 break
563 # Container that configures and calls callbacks.
File ~/opt/miniconda3/envs/kerascore/lib/python3.8/site-packages/keras_core/src/utils/traceback_utils.py:122, in filter_traceback.<locals>.error_handler(*args, **kwargs)
119 filtered_tb = _process_traceback_frames(e.__traceback__)
120 # To get the full stack trace, call:
121 # `keras_core.config.disable_traceback_filtering()`
--> 122 raise e.with_traceback(filtered_tb) from None
123 finally:
124 del filtered_tb
Cell In[1], line 6, in MultiInputModel.call(self, inputs)
5 def call(self, inputs):
----> 6 a, b = inputs
7 return a + b
ValueError: Exception encountered when calling MultiInputModel.call().
not enough values to unpack (expected 2, got 1)
Arguments received by MultiInputModel.call():
• inputs=(('jnp.ndarray(shape=(3, 2), dtype=float32)', 'jnp.ndarray(shape=(3, 2), dtype=float32)'),)
I tried building a custom model with two inputs, but it throws an exception when I call it:
import keras_core as keras
import numpy as np
class MultiInputModel(keras.Model):
def __init__(self, **kwargs):
super().__init__(**kwargs)
self.dense1 = keras.layers.Dense(10)
def call(self, inputs):
a, b = inputs
r = self.dense1(a)
return keras.layers.concatenate([r, b])
model = MultiInputModel()
X1 = np.random.rand(3, 3)
X2 = np.random.rand(3, 2)
model((X1, X2)) #!!! ValueError
The exception is:
ValueError: Cannot convert '((3, 3), (3, 2))' to a shape. Found invalid entry '(3, 3)'.
Dynamic shapes (shapes with `None` entries) are not allowed with the tensorflow,
except for the batch size (axis 0).
Full stack trace:
ValueError Traceback (most recent call last)
Cell In[1], line 17
15 X1 = np.random.rand(3, 3)
16 X2 = np.random.rand(3, 2)
---> 17 model((X1, X2)) #!!! ValueError
File ~/opt/miniconda3/envs/kerascore/lib/python3.8/site-packages/keras_core/src/utils/traceback_utils.py:122, in filter_traceback..error_handler(*args, **kwargs)
119 filtered_tb = _process_traceback_frames(e.traceback)
120 # To get the full stack trace, call:
121 # keras_core.config.disable_traceback_filtering()
--> 122 raise e.with_traceback(filtered_tb) from None
123 finally:
124 del filtered_tb
File ~/opt/miniconda3/envs/kerascore/lib/python3.8/site-packages/keras_core/src/backend/common/variables.py:446, in standardize_shape(shape, allow_dynamic_batch_size, allow_all_dynamic)
439 else:
440 msg += (
441 "Dynamic shapes (shapes with None
entries) "
442 f"are not allowed with the {config.backend()}. "
443 "All dimensions should be positive integers, "
444 "including the batch size (axis 0)."
445 )
--> 446 raise ValueError(msg)
447 if e < 0:
448 raise ValueError(
449 f"Cannot convert '{shape}' to a shape. "
450 "Negative dimensions are not allowed."
451 )
ValueError: Cannot convert '((3, 3), (3, 2))' to a shape. Found invalid entry '(3, 3)'. Dynamic shapes (shapes with None
entries) are not allowed with the tensorflow, except for the batch size (axis 0).
Side note: the error message should say "the tensorflow backend" instead of just "the tensorflow".
I tested with the other backends, and I get the same error (except for PyTorch it says Cannot convert '(torch.Size([3, 3]), torch.Size([3, 2]))' to a shape. Found invalid entry 'torch.Size([3, 3])'.
)
I also tested with tf.keras, and it works fine.
Lastly, I tried building the Dense
layer inside the build()
method instead of the constructor, and it actually fixes the issue:
import keras_core as keras
import numpy as np
class MultiInputModel(keras.Model):
def build(self, batch_input_shape):
self.dense1 = keras.layers.Dense(10)
self.built = True
def call(self, inputs):
a, b = inputs
r = self.dense1(a)
return keras.layers.concatenate([r, b])
model = MultiInputModel()
X1 = np.random.rand(3, 3)
X2 = np.random.rand(3, 2)
model((X1, X2)) #works fine now
So it looks like the problem only occurs when the model has multiple inputs and it creates a layer in the constructor.
On a CPU only python environment, I have hit a deadlock issue running python pip_build.py --install
.
Tracked this down as far as the torch
import, which is never resolving when in the tf backend.
Re-running the script with the torch
backend selected fixes the deadlock fully.
I'm trying to run the demo_custom_torch_workflow.py
and gpu_demo_torch.py
examples, and they both fail because there's no way to get an iterable containing the parameters to optimize from a Keras model.
Here is the relevant portion of code that fails becausemodel
doesn't have a parameters()
function.
model = keras_core.Sequential(
[
layers.Input(shape=(28, 28, 1)),
layers.Conv2D(32, kernel_size=(3, 3), activation="relu"),
layers.MaxPooling2D(pool_size=(2, 2)),
layers.Conv2D(64, kernel_size=(3, 3), activation="relu"),
layers.MaxPooling2D(pool_size=(2, 2)),
layers.Flatten(),
layers.Dropout(0.5),
layers.Dense(num_classes),
]
)
optimizer = optim.Adam(model.parameters(), lr=learning_rate)
These examples also wrap the Keras model into an nn.Module
:
class MyModel(nn.Module):
def __init__(self):
super().__init__()
self.model = keras_core.Sequential(
[
layers.Input(shape=(28, 28, 1)),
layers.Conv2D(32, kernel_size=(3, 3), activation="relu"),
layers.MaxPooling2D(pool_size=(2, 2)),
layers.Conv2D(64, kernel_size=(3, 3), activation="relu"),
layers.MaxPooling2D(pool_size=(2, 2)),
layers.Flatten(),
layers.Dropout(0.5),
layers.Dense(num_classes),
]
)
def forward(self, x):
return self.model(x)
torch_module = MyModel()
optimizer = optim.Adam(torch_module.parameters(), lr=learning_rate)
But this also fails because the list of parameters comes back empty.
How do I get past this?
Currently all the variable class and tensor related utils are living under keras_core/backend/tensorflow/__init__.py
(same for all the backend). We should have a proper file/package for them.
My proposal:
With tf.keras, the following code works, but not in Keras-Core:
import keras_core as keras
import numpy as np
images = [
np.random.rand(100, 100, 3),
np.random.rand(100, 100, 3),
]
keras.layers.CenterCrop(height=60, width=50)(images) #!!! AttributeError
Full stacktrace below:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[21], line 8
2 import numpy as np
4 images = [
5 np.random.rand(100, 100, 3),
6 np.random.rand(100, 100, 3),
7 ]
----> 8 keras.layers.CenterCrop(height=60, width=50)(images)
File ~/opt/miniconda3/envs/kerascore/lib/python3.8/site-packages/keras_core/src/layers/preprocessing/tf_data_layer.py:36, in TFDataLayer.__call__(self, inputs, **kwargs)
34 self._convert_input_args = True
35 return outputs
---> 36 return super().__call__(inputs, **kwargs)
File ~/opt/miniconda3/envs/kerascore/lib/python3.8/site-packages/keras_core/src/utils/traceback_utils.py:122, in filter_traceback.<locals>.error_handler(*args, **kwargs)
119 filtered_tb = _process_traceback_frames(e.__traceback__)
120 # To get the full stack trace, call:
121 # `keras_core.config.disable_traceback_filtering()`
--> 122 raise e.with_traceback(filtered_tb) from None
123 finally:
124 del filtered_tb
File ~/opt/miniconda3/envs/kerascore/lib/python3.8/site-packages/keras_core/src/layers/preprocessing/center_crop.py:59, in CenterCrop.call(self, inputs)
57 init_width = inputs.shape[-1]
58 else:
---> 59 init_height = inputs.shape[-3]
60 init_width = inputs.shape[-2]
62 if init_height is None or init_width is None:
63 # Dynamic size case. TODO.
AttributeError: Exception encountered when calling CenterCrop.call().
'list' object has no attribute 'shape'
Arguments received by CenterCrop.call():
• inputs=['jnp.ndarray(shape=(100, 100, 3), dtype=float32)', 'jnp.ndarray(shape=(100, 100, 3), dtype=float32)']
A simple workaround is to stack the images:
keras.layers.CenterCrop(height=60, width=50)(np.stack(images))
Not sure this can be considered a bug, but it's one of those little differences that may porting code from Keras 2.x to 3.0 a bit harder.
I used conda, I created and activated an empty environment with just Python 3.8, I cloned the keras-core
repo and followed the installation instructions, including pip install -r requirements
and python pip_build.py --install
.
Now when I open a Python shell from the root keras-core
directory, everything works fine.
However, when I open a Python shell anywhere else, I'm missing most functions in the keras_core.backend
package. Here's what I get:
>>> import keras_core.backend as K
Using TensorFlow backend
>>> dir(K)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__',
'__spec__', 'backend', 'clear_session', 'epsilon', 'floatx', 'get_uid', 'image_data_format', 'set_epsilon',
'set_floatx', 'set_image_data_format']
As you can see, I'm missing K.shape()
, K.numpy
and almost everything else.
For comparison, this is what I get when I launch the shell from the keras-core
root directory:
>>> import keras_core.backend as K
Using TensorFlow backend
>>> dir(K)
['AutocastScope', 'DYNAMIC_SHAPES_OK', 'KerasTensor', 'StatelessScope', 'Variable', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'any_symbolic_tensors', 'backend', 'cast', 'common', 'compute_output_spec', 'cond', 'config', 'convert_to_numpy', 'convert_to_tensor', 'core', 'epsilon', 'floatx', 'get_autocast_scope', 'get_stateless_scope', 'gru', 'image', 'image_data_format', 'in_stateless_scope', 'is_float_dtype', 'is_keras_tensor', 'is_tensor', 'lstm', 'math', 'name_scope', 'nn', 'numpy', 'print_msg', 'random', 'rnn', 'scatter', 'set_epsilon', 'set_floatx', 'set_image_data_format', 'shape', 'standardize_data_format', 'standardize_dtype', 'standardize_shape', 'stop_gradient', 'tensorboard', 'tensorflow', 'vectorized_map']
I tried running pip uninstall keras-core
, confirmed it was no longer available, and reinstalled it using python pip_build.py --install
, but I still get the same issue.
Can we put up simple instructions on how to configure the dev environment to properly align with the code formatting expected? I have both flake8
and black
installed in VSCode but somehow the settings aren't aligned even if I point it to the correct settings file. A simple workaround is to run both flake8 and black separately but looking for better guidelines
classic MNIST training:
The common factors triggering the issue seem to be: using the BatchNorm layer, on Keras multi-backend. The choice of backend does not seem to change the behavior.
import numpy as np
from tensorflow import data as tf_data
import keras_core
x = np.random.randn(8, 2).astype("float32")
y = np.random.randint(0, 1, (8, 1)).astype("uint8")
dataset = tf_data.Dataset.from_tensor_slices((x, y)).batch(2)
model = keras_core.Sequential([
keras_core.layers.Input(shape=(2,)),
keras_core.layers.Dense(1, activation="sigmoid")
])
model.summary()
model.compile(optimizer="sgd", loss="mse",
metrics=["categorical_accuracy"] # Fails
# metrics=[keras_core.metrics.CategoricalAccuracy()] # Works fine when giving class names
)
model.fit(dataset, epochs=2)
[keras_core/metrics/__init__.py](https://localhost:8080/#) in get(identifier)
188 return obj
189 else:
--> 190 raise ValueError(f"Could not interpret metric identifier: {identifier}")
ValueError: Could not interpret metric identifier: categorical_accuracy
When specifying the metric via class name keras_core.metrics.CategoricalAccuracy()
, it works fine. This is true for other metrics as well.
We implement BCE as follows:
def binary_crossentropy(target, output, from_logits=False):
target = jnp.array(target)
output = jnp.array(output)
if target.shape != output.shape:
raise ValueError(
"Arguments `target` and `output` must have the same shape. "
"Received: "
f"target.shape={target.shape}, output.shape={output.shape}"
)
if from_logits:
output = jnn.sigmoid(output)
output = jnp.clip(output, epsilon(), 1.0 - epsilon())
bce = target * jnp.log(output)
bce += (1.0 - target) * jnp.log(1.0 - output)
return -bce
An issue here is the fact that, provided logits, we take the sigmoid, and then the log. This can lead to numerical instability in fp16.
Can we come up with an implementation that goes through log_sigmoid
or something?
I've built a simple dual-output model using the functional API, but it's failing at training time with an AttributeError
:
import numpy as np
import keras_core as keras
X = np.random.rand(1000, 5)
Y1 = np.random.rand(1000, 2)
Y2 = np.random.rand(1000, 3)
input_ = keras.layers.Input(shape=X.shape[1:])
dense1 = keras.layers.Dense(10, activation="relu")(input_)
output1 = keras.layers.Dense(Y1.shape[1])(dense1)
output2 = keras.layers.Dense(Y2.shape[1])(dense1)
model = keras.Model(inputs=(input_,), outputs={"out1": output1, "out2": output2})
model.compile(loss="mse", optimizer="nadam")
history = model.fit(X, {"out1": Y1, "out2": Y2}) #!!! AttributeError: 'dict' object has no attribute 'dtype'
Y_pred = model.predict(X)
This fails with all 3 backends. I'm running the latest code (just checked out and built).
Below is the full stacktrace.
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[1], line 14
12 model = keras.Model(inputs=[input_], outputs=[output1, output2])
13 model.compile(loss="mse", optimizer="nadam")
---> 14 model.fit(X, (Y1, Y2))
15 model.predict(X)
File ~/opt/miniconda3/envs/kerascore/lib/python3.8/site-packages/keras_core/src/utils/traceback_utils.py:122, in filter_traceback.<locals>.error_handler(*args, **kwargs)
119 filtered_tb = _process_traceback_frames(e.__traceback__)
120 # To get the full stack trace, call:
121 # `keras_core.config.disable_traceback_filtering()`
--> 122 raise e.with_traceback(filtered_tb) from None
123 finally:
124 del filtered_tb
File ~/opt/miniconda3/envs/kerascore/lib/python3.8/site-packages/keras_core/src/losses/loss.py:41, in Loss.__call__.<locals>.<lambda>(x)
36 dtype = backend.floatx()
37 y_pred = nest.map_structure(
38 lambda x: ops.convert_to_tensor(x, dtype=dtype), y_pred
39 )
40 y_true = nest.map_structure(
---> 41 lambda x: ops.convert_to_tensor(x, dtype=y_pred.dtype), y_true
42 )
44 losses = self.call(y_true, y_pred)
45 out_mask = getattr(losses, "_keras_mask", None)
AttributeError: 'ListWrapper' object has no attribute 'dtype'
I get following error when running on GPU -
2023-05-03 11:14:08.026060: W tensorflow/core/framework/op_kernel.cc:1830]
OP_REQUIRES failed at xla_ops.cc:347 : INVALID_ARGUMENT:
Trying to access resource Resource-0-at-0x1ef56e60 (defined @ keras-core/keras_core/backend/tensorflow/__init__.py:22)
located in device /job:localhost/replica:0/task:0/device:CPU:0 from device /job:localhost/replica:0/task:0/device:GPU:0
Cf. https://www.tensorflow.org/xla/known_issues#tfvariable_on_a_different_device
Traceback (most recent call last):
File "keras-core/demo_custom_tf_workflow.py", line 86, in <module>
loss = train_step(data)
File "python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "python3.9/site-packages/tensorflow/python/eager/execute.py", line 52, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: Trying to access resource Resource-0-at-0x1ef56e60 (defined @ keras-core/keras_core/backend/tensorflow/__init__.py:22) located in device /job:localhost/replica:0/task:0/device:CPU:0 from device /job:localhost/replica:0/task:0/device:GPU:0
Cf. https://www.tensorflow.org/xla/known_issues#tfvariable_on_a_different_device [Op:__inference_train_step_651]
Believe the issue os accessing CPU resource from GPU. This error doesn't come up when running the JAX Workflow on GPU.
Same error when running demo_custom_layer_backend_agnostic.py
with TensorFlow backend.
Not sure if this is specific to my local setting. Can someone confirm that TF workflow runs fine on GPU? On CPU / Mac, they run fine.
I am trying to generate the whl
file locally to do a local install of the package but I get the No module named build
error. FYI, I am running the build py file in a conda env
I have a very interesting test case that I am unable to debug. I implemented the Hinge
loss (check #31 ), and imported the hinge(...)
function for the corresponding metric implementation. The values that are coming out of metric are way off than the actual values but the values coming out of the loss function are as expected. Here is the test case:
y_true = tf.constant([[-1., 1., -1., 1.], [-1., -1., 1., 1.]])
y_pred = tf.constant([[-0.3, 0.2, -0.1, 1.6], [-0.25, -1.0, 0.5, 0.6]])
sample_weight = tf.constant([1.5, 2.0])
##### Hinge loss #########################
# TF implementation
tf.keras.losses.Hinge()(y_true, y_pred, sample_weight)
# output <tf.Tensor: shape=(), dtype=float32, numpy=0.8625>
# My implemetation
keras_core.losses.Hinge()(y_true, y_pred, sample_weight)
# ouput <tf.Tensor: shape=(), dtype=float32, numpy=0.8625>
##### Hinge metric #########################
# TF metric implementation
tf.keras.metrics.Hinge()(y_true, y_pred, sample_weight)
# output <tf.Tensor: shape=(), dtype=float32, numpy=0.49285716>
# My metric implementation
keras_core.metrics.Hinge()(y_true, y_pred, sample_weight=sample_weight)
# output <tf.Tensor: shape=(), dtype=float32, numpy=0.575>
Here is how the the class implementation looks like:
@keras_core_export("keras_core.metrics.Hinge")
class Hinge(reduction_metrics.MeanMetricWrapper):
def __init__(self, name="hinge", dtype=None):
super().__init__(fn=keras_core.losses.hinge, name=name, dtype=dtype)
def get_config(self):
return {"name": self.name, "dtype": self.dtype}
Expected behavior:
Given that the loss function is working as expected, the metric implementation should work as we are just importing the same function in the corresponding metric class
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.