Weird behavior with pdm run and torch distributed. Poetry works correctly by setting sys.argv[0]

[vikigenius]

Weird behavior with pdm run and torch distributed. Poetry works correctly by setting sys.argv[0]

[vikigenius]

Maybe pdm can also install the user script to the bin location and then set sys.argv[0] appropriately.

[frostming]

[tool.poetry.scripts] is the way to expose a console script entry point for the installable package. The equivalent in PDM is the PEP standard [project.scripts]. And you must run pdm install, which installs the project itself, to make it work.

On the contrast, [tool.pdm.scripts] is a table of tasks with a name for each.

[vikigenius]

@frostming pdm install does not seem to work or install the project itself.

I added the following to pyproject.toml after removing the [tool.pdm.scripts]

[project.scripts]
llmft = "llmft.cli:run"

After doing pdm install it doesn't say anything about installing the project. Just says all packages are synced nothing to do.

When I do pdm run llmft it says

[PdmUsageError]: Command 'llmft' is not found in your PATH.

I try to manually run pdm run python llmft/cli.py

λ pdm run python llmft/cli.py
Traceback (most recent call last):
  File "/home/void/Projects/Research/llmft/llmft/cli.py", line 3, in <module>
    from llmft.fit import fit
ModuleNotFoundError: No module named 'llmft'

So the project itself is not being installed?

[pawamoy]

I think you need to give a name to your project:

[project]
name = "llmft-or-something-like-that"

Otherwise PDM doesn't consider your project as a package and does not try to install it.

[vikigenius]

@pawamoy

I do have it, here is my pyproject.toml

[build-system]
requires = ["pdm-backend"]
build-backend = "pdm.backend"

[project]
name = "llmft"
version = "0.1.0"
description = "LLM Finetuner"
authors = [
    {name = "Vikash Balasubramanian", email = "[email protected]"},
]
dependencies = [
    "torch>=2.1.2",
    "numpy>=1.26.3",
    "transformers>=4.36.2",
    "peft>=0.7.1",
    "datasets>=2.16.1",
    "arger>=1.4.8",
    "pytorch-lightning>=2.1.3",
]
requires-python = "==3.11.*"
readme = "README.md"
license = {text = "MIT"}

[project.scripts]
llmft = "llmft.cli:run"

[tool.pdm]
package-type = "application"

[tool.pdm.dev-dependencies]
qa = [
    "ruff>=0.1.11",
    "mypy>=1.8.0",
]
tools = [
    "python-lsp-server>=1.9.0",
    "ruff-lsp>=0.0.49",
]

[frostming]

the package type has to be "library" to make it installable

[vikigenius]

@frostming Ok thanks for that. Maybe it would be helpful if the document made it a bit more clear the differences between application and library.

I thought my project was an application since it had a console script entry point as a CLI application.