Giter Site home page Giter Site logo

xpm-js's Introduction

npm (scoped) license Standard

xpm - the xPack project manager

This project implements xpm - the xPack project manager - as a Node.js CLI application.

The main purpose of xpm is to automate common tasks and provide reproducible builds for language neutral, multi-version projects.

More specifically:

  • to manage build configurations and to run actions (sequences of commands) associated with various build steps
  • to manage versioned dependencies, like to install both source code libraries and binary archives, and to easily update them when new versions are available

So far, two main use cases were considered and successfully implemented:

  • to orchestrate complex multi-architecture/multi-platform builds for embedded projects (like semihosted unit tests for µOS++)
  • to perform the multi-platform builds for all binary tools part of the xPack 3rd Party Development Tools project.

The project is open-source and is hosted on GitHub as xpack/xpm-js.

Maintainer info

This page is intended as a brief introduction for those who consider using this tool to manage their projects.

For developer and maintainer information, see the separate README-DEVELOPER.md and README-MAINTAINER.md pages.

Quick links

If you already know the general facts about xpm, you can directly skip to:

xPacks overview

xPacks are general purpose versioned software packages, much the same as the highly successful npm packages in the Node.js ecosystem.

xPacks are usually Git repositories and can be published on npmjs.com or any npm compatible server.

For more details, please read the xPacks 101 page.

Prerequisites

The current version requires Node.js >= 16.

Since it is highly recommended to do not use sudo during install, and instead use a version manager or to customize the npm install location, please read the install page for more details.

Install

The basic command is:

npm install --global xpm@latest

Troubleshooting: in case xpm was already installed, in certain conditions the update may not succeed and xpm may become unusable; if this happens, uninstall xpm and retry the install.

For more details, please refer to the install page.

User info

To get an initial glimpse on the program, ask it for help:

% xpm --help

The xPack project manager command line tool
Usage: xpm <command> [<subcommand>...] [<options> ...] [<args>...]

where <command> is one of:
  init, install, link, list, run, uninstall

Common options:
  --loglevel <level>     Set log level (silent|warn|info|verbose|debug|trace)
  -s|--silent            Disable all messages (--loglevel silent)
  -q|--quiet             Mostly quiet, warnings and errors (--loglevel warn)
  --informative          Informative (--loglevel info)
  -v|--verbose           Verbose (--loglevel verbose)
  -d|--debug             Debug messages (--loglevel debug)
  -dd|--trace            Trace messages (--loglevel trace, -d -d)
  --no-update-notifier   Skip check for a more recent version
  -C <folder>            Set current folder

xpm -h|--help            Quick help
xpm <command> -h|--help  Quick help on command
xpm --version            Show version
xpm -i|--interactive     Enter interactive mode

npm [email protected] '/Users/ilg/.nvm/versions/node/v16.16.0/lib/node_modules/xpm'
Home page: <https://xpack.github.io/xpm/>
Bug reports: <https://github.com/xpack/xpm-js/issues/>

Configuration

Similarly to npm, the project configuration is in package.json.

In addition to name and version which must uniquely identify the project, there is an xpack property that groups several xpm specific properties:

  • dependencies
  • devDependencies
  • properties
  • actions
  • buildConfigurations

dependencies and devDependencies are similar to npm, but refer to xPacks. properties define values used in substitutions. actions are a more generalised form of the npm scripts, which allow arrays of commands with substitutions. buildConfigurations define multiple configurations, like debug/release.

Template substitutions

To increase reusability, it is possible to use substitutions in the strings defining actions. The syntax is more elaborate than the simple variable substitution, and is using the LiquidJS template engine syntax, which accepts:

  • variables, like {{ configuration.name }}
  • filters, like {{ configuration.name | downcase }}
  • tags, like {% if os.platform != 'win32' %}xpm run execute --config synthetic-posix-cmake-debug{% endif %}

The substitution values come from the following predefined objects:

  • package, with the entire package.json content
  • properties with the xPack properties
  • os.platform with the Node.js platform (possible values are aix, darwin, freebsd, linux, openbsd, sunos, and win32)
  • os.arch with the Node.js architecture (possible values are arm, arm64, ia32, mips, mipsel, ppc, ppc64, s390, s390x, x32, and x64)

When the xpm command is started with --config, the configuration properties are preferred to the xPack properties and the following object is also added to properties:

  • configuration with the full content of the current xPack build configuration; the configuration name is available as configuration.name

For the full list of variables available for substitutions, please read the documentation of the separate xpack/xpm-liquid-ts project.

Multi-line actions

In order to accommodate more complex actions, it is possible to define sequences of commands as arrays of strings, with each line executed as a separate command.

If multiple commands are generated via loops, line terminators can be inserted with {{ os.EOL }}), for example:

{% for command in package.xpack.my_commands %}{{ command }}{{ os.EOL }}{% endfor %}

The build folder path

When using build configurations, each build must be performed in a separate build folder.

This should be done using the reserved property buildFolderRelativePath, which must define a folder relative to the project root, usually below a top build folder.

This property can be defined either manually for each configuration, or globally, as a computed property, available for the entire project.

The definition can be parametrised with the configuration name, and possibly converted to a lower case folder name, for example:

  "xpack": {
    "properties": {
      "buildFolderRelativePath": "{{ 'build' | path_join: configuration.name | to_filename | downcase }}"
    }
  }

Change log - incompatible changes

According to semver rules:

Major version X (X.y.z | X > 0) MUST be incremented if any backwards incompatible changes are introduced to the public API.

The incompatible changes, in reverse chronological order, are:

  • v0.19.x: compatible, but based on node 18
  • v0.14.x: separate xPack from npm dependencies (see 0001 policy)

License

The original content is released under the MIT License, with all rights reserved to Liviu Ionescu.

The design is heavily influenced by the npm application, Copyright (c) npm, Inc. and Contributors, Licensed on the terms of The Artistic License 2.0.

Note

The xpm tool is currently work in progress and should be used with caution.

xpm-js's People

Contributors

dependabot[bot] avatar ilg-ul avatar jbenczarski avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

xpm-js's Issues

Replace 'mkdirp' with 'make-dir'

Since 'mkdirp' no longer supports node 8, study if 'make-dir' is more appropriate and possibly replace it.

Same for 'mkdirp-promise'.

Update 'cacache'

The current sources usecacache version 12.x.

With 15.x, the binary install fail without errors.

Implement 'uninstall'

$ xpm uninstall --global @gnu-mcu-eclipse/arm-none-eabi-gcc

(Note: not yet implemented)

Implement --system

Allow to install in system folders.

Search for installed packages will check the home folder first and then the system folders.

Add xpack.minimumXpmRequired to package.json

To prevent processing packages that require features available only in recent versions of xpm, define the min xpm version.

If the current xpm version is lower than required, fail with a recommendation to upgrade.

If the current xpm version is higher, it should ensure that the behaviour advertised by the desired version is respected (inspired by cmake_minimum_required).

Accept * for any in skip cases

When defining architectures that can be skipped, accept * as any.

This is intended for binary xPacks that are needed on specific platforms only (more specifically Windows). All other platforms may use '*'.

There are two cases:

  • *
  • <platform>-*

The init with template fails on Windows in the Git console

The init --template works fine on Windows when invoked in the cmd.exe terminal, but fails when invoked in the Git console:

ilg@ilg-w10x64-xpack MINGW64 /tmp/hifive1-blinky-cpp
$ xpm init --template @sifive/templates -dd
debug: start arg0: 'C:\Program Files\nodejs\node.exe'
debug: start arg1: 'C:\Users\ilg\AppData\Roaming\npm\node_modules\xpm\bin\xpm.js'
debug: start arg2: 'init'
debug: start arg3: '--template'
debug: start arg4: '@sifive/templates'
debug: start arg5: '-dd'
trace: {
  isInteractive: false,
  interactiveServerPort: undefined,
  logLevel: 'trace',
  isHelpRequest: false,
  isVersionRequest: false,
  cwd: 'C:\\Users\\ilg\\AppData\\Local\\Temp\\hifive1-blinky-cpp'
}
trace: Xpm.constructor()
trace: Xpm.main()
trace: main arg0: 'init'
trace: main arg1: '--template'
trace: main arg2: '@sifive/templates'
trace: main arg3: '-dd'
trace: parseOptions()
trace: {
  isInteractive: false,
  interactiveServerPort: undefined,
  logLevel: 'trace',
  isHelpRequest: false,
  isVersionRequest: false,
  cwd: 'C:\\Users\\ilg\\AppData\\Local\\Temp\\hifive1-blinky-cpp',
  invokedFromCli: true
}
trace: do not fetch latest version number.
debug: cwd()='C:\Users\ilg\AppData\Local\Temp\hifive1-blinky-cpp'
debug: Command(s): 'init'
trace: cmd arg0: '--template'
trace: cmd arg1: '@sifive/templates'
debug: 'xpm init' started
trace: Init.run()
trace: parseOptions()
trace: {
  isInteractive: false,
  interactiveServerPort: undefined,
  logLevel: 'trace',
  isHelpRequest: false,
  isVersionRequest: false,
  cwd: 'C:\\Users\\ilg\\AppData\\Local\\Temp\\hifive1-blinky-cpp',
  invokedFromCli: true,
  template: '@sifive/templates',
  projectName: null,
  properties: {}
}
trace: Init.doRun()
xPack manager - create an xPack, empty or from a template
trace: Init.doInitWithTemplate()

Checking package @sifive/templates metadata...
Processing @sifive/[email protected]...
trace: XpmInitTemplate.run()

error: Missing board name.

'xpm init' completed in 359 ms.
debug: 'xpm init' - returned 1

When it works, the console reads:

Checking package @sifive/templates metadata...
Processing @sifive/[email protected]...
trace: XpmInitTemplate.run()

Programming language? (c, cpp, ?) [cpp]:
...

xpack for libraries

Often, in addition to cross tools - many times a code library of some sort needs to be used.

Have you thought about creating something like xPack for code libraries? Examples include an SSL library, or one of the many Arduino, or MBED packages, or for example a BSP of some type.

What's missing in the embedded world is something like the linux "pkg-config" for embedded libraries - that could read from a package provided JSON files, or an "as built" JSON file of some sort..

What's nice about your xpack solution is the ability to have multiple versions "self contained" and is delivered as a package.

The idea of a "central repository" is sometimes very useful, but sometimes a project needs something customized - or tailored to a project.

Here's some ideas and requirements

  1. For things like "freertos" there is 1 or more project specific configuration files the library needs to be built using - those configuration files may be LOCAL to the parent project, or LOCAL to the library.

The ability to say to a library package: "Build your self" - using these external configuration files (there may be more then 1 configuration file, and may be of any arbitrary type, ie: 'FOO.mk' - or FOO.h, or what ever.

that is in addition to: "build your self" using one of the built in pre-packaged configuration sets that COMES WITH the package.

  1. a central repository is often helpful - but there are times when a project really needs a package defined locally and not part of the central repository.

For example, a library may have been customized for a project, and the project it self needs to provide the customized library as part of its "managed configuration" (aka: Project X, requires Package A - version 1 - slightly customized). But project Y, which came along later needs the same package but Version 2 - again slightly customized, but might use Package Y as is with no changes.

  1. Every project has it's own way of customizing and tweaking things :-(

Sometimes the resulting configuration file is a MAKEFILE... a PROJECT file, or C or H files - it varies greatly.

One method that is very helpful and widely used is the Linux kernel "make config" solutions, some are GUI based, some are text based... or - more importantly, one can provide a text file which is the "old config"

There's also the very UNIX centric "autoconf" ... but that does not work for "mere mortals of this world"

Something like menu config is needed to customize packages, and being cross platform would be even more helpful.

Being able to as a package specify "Here's how to launch the configuration" or "this is the configuration method I use"

The result though - needs to usable in various IDEs (like Eclipse) and command line tools like standard Makefiles, etc.

Update 'pacote'

The current sources use pacote version 9.x.

With 11.x, the binary install fails.

xpm install: accept platforms that do not have binaries available

For xPacks like @gnu-mcu-eclipse/windows-build-tools there are binaries only for Windows.

Allow to define platforms that do not have binaries, to prevent returning errors.

The suggested syntax for package.json is

"skip": true

instead of the fileName and sha256 properties.

Local installs should add links in .bin

At the moment only xmp install without packages adds the local links from the .bin folder to the actual location.

The same should happen when installing local packages, there should be no need for an extra command.

Implement the `update` command

xPack is not usable as a package (application) manager without the list & update commands for end-users. They have to track a new version of packages by hands. Also, is an only way to know about what packages are installed is to open a folder deep inside xPack folders structure.

Rename Linux .cache/xPack

Due to a misunderstanding, the Linux cache folder is .caches/xPack, while it should have been .cache/xPack.

Use the correct name and silently move the content, if possible.

'xpm install' fails on some windows machines

the 'rename' used internally fails:

Installing globally in 'C:\Users\akrug\AppData\Roaming\xPacks\@gnu-mcu-eclipse\openocd\0.10.0-7.1'...
Downloading https://github.com/gnu-mcu-eclipse/openocd/releases/download/v0.10.0-7-20180123/gnu-mcu-eclipse-openocd-0.10.0-7-20180123-1217-win64.zip...
Extracting 'gnu-mcu-eclipse-openocd-0.10.0-7-20180123-1217-win64.zip'...
{ Error: EPERM: operation not permitted, rename 'C:\Users\akrug\AppData\Roaming\xPacks\@gnu-mcu-eclipse\openocd\0.10.0-7.1\.tmp\GNU MCU Eclipse\OpenOCD\0.10.0-7-20180123-1217\doc' -> 'C:\Users\akrug\AppData\Roaming\xPacks\@gnu-mcu-eclipse\openocd\0.10.0-7.1\.content\doc'
 errno: -4048,
 code: 'EPERM',
 syscall: 'rename',
 path: 'C:\\Users\\akrug\\AppData\\Roaming\\xPacks\\@gnu-mcu-eclipse\\openocd\\0.10.0-7.1\\.tmp\\GNU MCU Eclipse\\OpenOCD\\0.10.0-7-20180123-1217\\doc',
 dest: 'C:\\Users\\akrug\\AppData\\Roaming\\xPacks\\@gnu-mcu-eclipse\\openocd\\0.10.0-7.1\\.content\\doc' }

possible causes:

  • unusual corporate policies for subfolders below AppData
  • antivirus

Implement version mitigation

Now when multiple versions of the same package are encountered, the xpm install fails.

Improve the logic to choose the highest version that meets all requirements.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.