hfailab/hai-platform

一种任务级GPU算力分时调度的高性能深度学习训练平台

hai-platform's Issues

支持强化学习吗？

离线安装hai时出现 network is unreachable错误

平台支持多节点并行训练Megatron-LM或者Deepspeed吗？

你好，请问现在平台支持在本地集群中运行Megatron-LM、Deepspeed等大的训练框架吗？
我们在配置中遇到2个问题
1.Megatron多节点启动bash脚本(每个节点bash有部分参数不一样,如NODE_RANK)。如何可以让分配相同任务节点使用不同配置文件？
2.多节点bash脚本有一个需要配置主Master IP，分配任务节点是由调度器分配的并不知道后续哪一个真正工作节点，这个配置要怎么支持
有没有多节点结合Megatron-LM的实现例子，提供参考一下。

系统要ubunt20.04么，K8S版本这些有要求么

开源版本部署咨询

开源版本部署中遇到若干问题，望支持

Submit python script using hai-cli but failed

According to README.md, I deployed hai-platform and installed hai-cli successfully.
"hai-cli init" using my token and url also succeed.
However, when I try "hai-cli python /haidata/hai-platform/workspace/haiadmin/test.py -- -n 1", an error occured unexpectedly, here is the message

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/hfai/client/api/api_utils.py", line 101, in async_requests
    result = json.loads(result)
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/hai-cli", line 9, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/dist-packages/asyncclick/core.py", line 1159, in __call__
    return anyio.run(self._main, main, args, kwargs, **({"backend":_anyio_backend} if _anyio_backend is not None else {}))
  File "/usr/local/lib/python3.8/dist-packages/anyio/_core/_eventloop.py", line 68, in run
    return asynclib.run(func, *args, **backend_options)
  File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 204, in run
    return native_run(wrapper(), debug=debug)
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 199, in wrapper
    return await func(*args)
  File "/usr/local/lib/python3.8/dist-packages/asyncclick/core.py", line 1162, in _main
    return await main(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/asyncclick/core.py", line 1083, in main
    rv = await self.invoke(ctx)
  File "/usr/local/lib/python3.8/dist-packages/asyncclick/core.py", line 1693, in invoke
    return await _process_result(await sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/dist-packages/asyncclick/core.py", line 1429, in invoke
    return await ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/asyncclick/core.py", line 783, in invoke
    rv = await rv
  File "/usr/local/lib/python3.8/dist-packages/hfai/client/commands/hfai_python.py", line 294, in python
    await func_python_cluster(experiment_py, experiment_args, name, nodes, priority, group, image, environments,
  File "/usr/local/lib/python3.8/dist-packages/hfai/client/commands/hfai_python.py", line 255, in func_python_cluster
    await hfai_experiment.run.callback(config, follow, None, None, None)
  File "/usr/local/lib/python3.8/dist-packages/hfai/client/commands/hfai_experiment.py", line 167, in run
    experiment = await create_experiment(experiment_yml)
  File "/usr/local/lib/python3.8/dist-packages/hfai/client/api/experiment_api.py", line 444, in create_experiment
    result = await async_requests(RequestMethod.POST, url=f'{mars_url()}/operating/task/create?token={token}',
  File "/usr/local/lib/python3.8/dist-packages/hfai/client/api/api_utils.py", line 116, in async_requests
    raise Exception(f'请求失败: [exception: {str(e)}] [result: {result}]')
Exception: 请求失败: [exception: Expecting value: line 1 column 1 (char 0)] [result: Not Found]

It seems that server returns code 404 to the client on the task create url -> "{mars_url()}/operating/task/create?token={token})". I have no idea why this would happen.

Further information can be provided if needed.
I am sure the token and url is correct since I can successfully init.
I am also sure the test.py exists on the shared_filesystem otherwise hai-cli would report another error.

一直卡着了，该怎么解决

没有拿到可用节点，是什么原因

BASE_IMAGE="registry.cn-hangzhou.aliyuncs.com/hfai/hai-platform:latest"

hai-cli nodes 没有显示节点，scheduler.log 中找到 报错信息

scheduler 0.log:133:2023-12-26 21:25:41.014ERRORticks beater#1703597141000没有拿到可用节点，请检查

unable to read client-cert

unable to read client-cert /root/k8s-ssl-tls/admin.crt for admin due to open /root/k8s-ssl-tls/admin.crt: no such file or directory