[QUESTION] I was able to download the LLM in llama.cpp, but I cannot download the LLM in Transformers.

[yamatohimi]

[QUESTION] I was able to download the LLM in llama.cpp, but I cannot download the LLM in Transformers.

[yamatohimi]

Python 3.10.13
xinference v.0.12.2.post1

[ChengjieLi28]

Python 3.10.13 xinference v.0.12.2.post1

Paste the error from the xinference backend, not just from the commandline.

[yamatohimi]

OK!
Is this enough? If not, please let us know.

huggingface_hub.utils._errors.GatedRepoError: 401 Client Error. (Request ID: Root=1-667ea709-40b704097bad97526456dd7c;c8a0107e-2241-49ff-9f19-817625902912)

Cannot access gated repo for url https://huggingface.co/api/models/meta-llama/Meta-Llama-3-8B-Instruct/revision/main.
Access to model meta-llama/Meta-Llama-3-8B-Instruct is restricted. You must be authenticated to access it.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/xinference/api/restful_api.py", line 771, in launch_model
    model_uid = await (await self._get_supervisor_ref()).launch_builtin_model(
  File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/context.py", line 227, in send
    return self._process_result_message(result)
  File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/context.py", line 102, in _process_result_message
    raise message.as_instanceof_cause()
  File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/pool.py", line 659, in send
    result = await self._run_coro(message.message_id, coro)
  File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/pool.py", line 370, in _run_coro
    return await coro
  File "/opt/conda/lib/python3.10/site-packages/xoscar/api.py", line 384, in __on_receive__
    return await super().__on_receive__(message)  # type: ignore
  File "xoscar/core.pyx", line 558, in __on_receive__
    raise ex
  File "xoscar/core.pyx", line 520, in xoscar.core._BaseActor.__on_receive__
    async with self._lock:
  File "xoscar/core.pyx", line 521, in xoscar.core._BaseActor.__on_receive__
    with debug_async_timeout('actor_lock_timeout',
  File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__
    result = await result
  File "/opt/conda/lib/python3.10/site-packages/xinference/core/supervisor.py", line 837, in launch_builtin_model
    await _launch_model()
  File "/opt/conda/lib/python3.10/site-packages/xinference/core/supervisor.py", line 801, in _launch_model
    await _launch_one_model(rep_model_uid)
  File "/opt/conda/lib/python3.10/site-packages/xinference/core/supervisor.py", line 782, in _launch_one_model
    await worker_ref.launch_builtin_model(
  File "xoscar/core.pyx", line 284, in __pyx_actor_method_wrapper
    async with lock:
  File "xoscar/core.pyx", line 287, in xoscar.core.__pyx_actor_method_wrapper
    result = await result
  File "/opt/conda/lib/python3.10/site-packages/xinference/core/utils.py", line 45, in wrapped
    ret = await func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/xinference/core/worker.py", line 641, in launch_builtin_model
    model, model_description = await asyncio.to_thread(
  File "/opt/conda/lib/python3.10/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
  File "/opt/conda/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/conda/lib/python3.10/site-packages/xinference/model/core.py", line 67, in create_model_instance
    return create_llm_model_instance(
  File "/opt/conda/lib/python3.10/site-packages/xinference/model/llm/core.py", line 223, in create_llm_model_instance
    save_path = cache(llm_family, llm_spec, quantization)
  File "/opt/conda/lib/python3.10/site-packages/xinference/model/llm/llm_family.py", line 297, in cache
    return cache_from_huggingface(llm_family, llm_spec, quantization)
  File "/opt/conda/lib/python3.10/site-packages/xinference/model/llm/llm_family.py", line 824, in cache_from_huggingface
    download_dir = retry_download(
  File "/opt/conda/lib/python3.10/site-packages/xinference/model/utils.py", line 127, in retry_download
    raise RuntimeError(
RuntimeError: [address=0.0.0.0:28136, pid=42] Failed to download model 'llama-3-instruct' (size: 8, format: pytorch) after multiple retries

[ChengjieLi28]

huggingface_hub.utils._errors.GatedRepoError: 401 Client Error. (Request ID: Root=1-667ea709-40b704097bad97526456dd7c;c8a0107e-2241-49ff-9f19-817625902912)

Cannot access gated repo for url https://huggingface.co/api/models/meta-llama/Meta-Llama-3-8B-Instruct/revision/main.
Access to model meta-llama/Meta-Llama-3-8B-Instruct is restricted. You must be authenticated to access it.

huggingface_hub.utils._errors.GatedRepoError: 401 Client Error. (Request ID: Root=1-667ea709-40b704097bad97526456dd7c;c8a0107e-2241-49ff-9f19-817625902912)

Cannot access gated repo for url https://huggingface.co/api/models/meta-llama/Meta-Llama-3-8B-Instruct/revision/main.
Access to model meta-llama/Meta-Llama-3-8B-Instruct is restricted. You must be authenticated to access it.

Llama-3 is a kind of model that needs to be authenticated. Refer to this: https://inference.readthedocs.io/en/latest/getting_started/troubleshooting.html to set the environment variable.

[yamatohimi]

Thank you for your answer.