camenduru/whisper-jax-colab

🐣 Please follow me for new updates https://twitter.com/camenduru
🔥 Please join our discord server https://discord.gg/k5BwmmvJJU
🥳 Please join my patreon community https://patreon.com/camenduru

🚦 WIP 🚦

🦒 Colab

Colab	Info
	openai/whisper-large-v2 Pro Colab (T4) we need High-RAM 😐
	openai/whisper-medium Free Colab (T4)
	openai/whisper-medium Free Colab (T4) Gradio

Main Repo

https://github.com/openai/whisper
https://github.com/sanchit-gandhi/whisper-jax

Paper

https://arxiv.org/abs/2212.04356

whisper-jax-colab's Issues

Adding mic input to record and translate it?

Hi , can it be also modified to have input from micrphone. It is also possible to do translation right ?

AttributeError: module 'jax.numpy' has no attribute 'DeviceArray'

All of these colabs are no longer working. They all result in the following error:

---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

[<ipython-input-1-82b9abc29980>](https://localhost:8080/#) in <cell line: 8>()
      6 import os
      7 import jax
----> 8 from whisper_jax import FlaxWhisperPipline
      9 import jax.numpy as jnp
     10 

3 frames

[/usr/local/lib/python3.10/dist-packages/jax/_src/deprecations.py](https://localhost:8080/#) in getattr(name)
     51       warnings.warn(message, DeprecationWarning, stacklevel=2)
     52       return fn
---> 53     raise AttributeError(f"module {module!r} has no attribute {name!r}")
     54 
     55   return getattr

AttributeError: module 'jax.numpy' has no attribute 'DeviceArray'

problem with the loading of the models in the collab

When I run any of colab notebook I got this error

`alueError Traceback (most recent call last)
in <cell line: 11>()
9 import jax.numpy as jnp
10
---> 11 pipeline = FlaxWhisperPipline("openai/whisper-medium", dtype=jnp.float16)
12 from jax.experimental.compilation_cache import compilation_cache as cc
13 cc.initialize_cache("/content/jax_cache")

4 frames
/usr/local/lib/python3.10/dist-packages/transformers/modeling_flax_utils.py in params(self)
268 def params(self) -> Union[Dict, FrozenDict]:
269 if not self._is_initialized:
--> 270 raise ValueError(
271 "params cannot be accessed from model when the model is created with _do_init=False. "
272 "You must call init_weights manually and store the params outside of the model and "

ValueError: params cannot be accessed from model when the model is created with _do_init=False. You must call init_weights manually and store the params outside of the model and pass it explicitly where needed.`

Whats can be the problem?

XlaRuntimeError on openai/whisper-medium Free Colab (T4)

---

XlaRuntimeError Traceback (most recent call last)

in <cell line: 11>()
9 import jax.numpy as jnp
10
---> 11 pipeline = FlaxWhisperPipline("openai/whisper-medium", dtype=jnp.float16)
12 from jax.experimental.compilation_cache import compilation_cache as cc
13 cc.initialize_cache("/content/jax_cache")

13 frames

[... skipping hidden 14 frame]

/usr/local/lib/python3.10/dist-packages/jax/_src/dispatch.py in backend_compile(backend, module, options, host_callbacks)
463 # TODO(sharadmv): remove this fallback when all backends allow compile
464 # to take in host_callbacks
--> 465 return backend.compile(built_c, compile_options=options)
466
467 _ir_dump_counter = itertools.count()

XlaRuntimeError: FAILED_PRECONDITION: DNN library initialization failed. Look at the errors above for more details.