tbrand/neph

A modern command line job processor, similar with make command

• A modern command line job processor written in Crystal 🚀
• Parallel job execution. 🚀
• Use like make command. 🚀

Installation

Arch Linux

Use your favourite AUR helper.
Package name: neph-git

Mac

You can install Neph with brew.

$ brew tap tbrand/homebrew-neph
$ brew install neph

Manual

Build dependencies:

• crystal and shards for building the binary
• go-md2man for generating the man page

It needs libyaml to be installed.

If you have a previous version of neph installed:

$ git clone https://github.com/tbrand/neph
$ cd neph
$ neph           # man page will be at neph.1, binary will be at bin/neph

If you don't have a previous version of neph installed:

$ git clone https://github.com/tbrand/neph
$ cd neph
$ shards build    # Now executable binary is located at bin/neph
$ bin/neph man    # Generate man page. It will be located at neph.1

Usage

All features for neph.yaml is written in sample/neph.yaml. So please refer for the details.

Wiki is also maintained as a document. Here is a full features.

• Execute command from neph
• Define dependencies between jobs
• Working directory
• Specify sources
• Ignoring errors
• Hide executing command
• Set a job result to env vars
• Import other configurations
• Command line options
• Log locations
• Log modes

Use cases

Neph is used in which_is_the_fastest.
The build time is reduced from 102[sec] to 33[sec].
The neph.yaml is here.

Contributing

1. Fork it ( https://github.com/tbrand/neph/fork )
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request

Contributors

• tbrand Taichiro Suzuki - creator, maintainer
• notramo Márton Szabó - maintainer

neph's Issues

Execute multiple jobs

Fail to compile on crystal 0.25

Hi,

neph fails to compile on last crystal.

crystal 0.25.0 has been released 4 days ago and removed YAML::Type.

Regards,

Compilation issue on 0.31

Hi @tbrand,

neph seems fail compiling on v0.31.0

https://travis-ci.com/the-benchmarker/web-frameworks/jobs/238300739#L256

Regards,

Release workflow

Hi @tbrand @notramo ,

Followoing #68, the release are actually created from CI travis-ci.

First thing is that if we want to release from CI, we SHOULD release for (at least) :

• OSX
• Linux
• Windows

Dropping windows support will be conter-productive as neph adoption, since à lot of devs (~1/2 according stackoverflow), are using windows.

A solution could be to use an other CI (or manuel process with cross-compilation) .

Regards,

Dynamic environment per job

Hi @tbrand,

neph is missing some features for https://github.com/the-benchmarker/web-frameworks evolution

• Create environments per jobs
• Executing sequentially
• ...

Do you plan / have time to implement them ?

Regards,

Add manual page to releases

@tbrand please add the manual page to Travis release builds.
Also, tbrand/homebrew-neph#1

Defining env var from job

Hi,

I'm working on https://github.com/tbrand/which_is_the_fastest and I have not expected result when running neph -> I'm on 0.1.21.

There is my config file

main:
  command:
    echo "The result is $CID"
  depends_on:
    -
      job: rails
      env: CID

rails:
  command: |
    sudo docker build -t rails .
    sudo docker run -td rails
  dir: ruby/rails

The expected result is tho show a container (docker) ID when running neph

Regards,

PS : I take inspiration from https://github.com/tbrand/neph/wiki/Set-a-job-result-to-env-vars

Add man page

The man page could be written in Markdown: go-md2man

Command callback

Hi,

Some job COULD requires external command to run, in order to prepare the environment (for example).

Is this a good idea to use callback / crystal custom code in order to customize some jobs ?

Regards,

File separator in job name

Hi @tbrand,

When any job contains a /, since it is the file separator, neph try to log in a non-existant directory.

Regards,

Wiki

Ability to retry a failing job

Hello @tbrand,

For some reason, any job defined in neph.yaml could fail.

I think it could be valuable to neph if we have a param to retry it.

What do you think ?

Regards,

Where is the logs?

@notramo Did you remove it? It's one of the strong point of neph.

spec.yml

Last release failure

Hi @tbrand,

I try to update neph on the last release 0.2.6, and nothing was build anymore on https://github.com/the-benchmarker/web-frameworks.

I stick on 0.2.2 made by @notramo (thanks to him)

Regards,

command options

• neph update
• neph uninstall
• neph clean

Upload binary to each release

CI configuration

Hi,

I can see that travis is using osx to run test phase

The default workers is linux, ubuntu LTS

Why osx was selected ?

Regards,

clean

The clean action should be moved to an option (-c, --clean).
In that way (and if #40 will be solved), the command line argument parsing could be simplified:
Insead of the specifying a job name with the -j option (neph -j install), it could be passed directly without an option: neph install (in the same way as make install)

Define variables

Hi @tbrand,

I think it could be interesting to define vars per environment.

I mean having a configuration file (personally I find .tfvars file awesome) that variable available in each jobs

Regards,

Using '.yaml' extension

Neph should use neph.yaml as default config file, instead of neph.yml.
http://www.yaml.org/faq.html Please use ".yaml" when possible.

Unable to compile neph due to missing dependency

When running shards build it is unable to build because it is missing libyaml, this is not part of the build instructions on the manual and it should be stated that is required to be installed system wide as it seems unable to download and use it's own dependencies.

Import feature

sequential job execution

Currently all jobs at depends_on are executed concurrently.
But sometimes we want to do it sequentially.
(The idea is needed at least here)

You may know you can do it like

some_job:
  depends_on:
    - sub_job1

sub_job1:
  depends_on:
    - sub_job2

But when we remove the sub_job1 from it, the pipeline will be broken.
So it's not good at the view point of flexibility and stability.

How about adding this function to neph.yaml?

some_job:
  depends_on:
    - sub_job1
    - sub_job2
  seq: true

When the seq config is true, the series of jobs on depends_on will be executed sequentially.
The default is false.

What do you guys think?

Arguments

May be using env vars.

modify sample/neph.yml

Setting environment variable from a job

Hi @notramo,

I need to execute a job, and to set output as environment variable.

I think the README is a bit outdated, https://github.com/tbrand/neph/wiki/Set-a-job-result-to-env-vars

I can help with the wiki, since I use neph.

Do you plan to push specs ?

Regards,

neph.yaml redesign

I think the current neph.yaml format is quite bad. It lacks features that are needed for a powerful build system, and has some limitations.
Issues with current neph.yaml format:

• users can't have a job named include, because it is used by neph
• the include command specifies paths relatively to the process pwd, and not relatively to the dir which neph.yaml is in.
• commands are always executed by sh, although some developers may want to use elvish, oh, or zsh to build the project
• commands are splitted on newlines, although a command may contain multi line job (e.g. sh if/case statement), or multi line string
• parsing is not easy (current implementation is very wrong - contains too much corner cases)

If you think my solution is a good idea, I will implement it. Please comment what you think about everything below. (The parser needs a rewrite even without these changes, but if it will be rewritten, implementing new features will be easier during the rewrite.)

My idea to solve this:

The neph.yaml file should contain two YAML documents: the first would be the config, the second would contain the jobs.

The config should contain:

• Neph version. This could be used to inform people about syntax changes, or automatic converting into the new format.
• Included files. This would do the same thing that the include keyword does currently. The files would be relative to the neph.yaml file.
• Command interpreter. This should be an Array of String, containing the arguments for the interpreter. The %c sequence would be replaced with the command name. If the arguments don't include %c, it prints the command to the interpreter's STDIN. Examples:

  interpreter:
    - elvish
    - c
    - %c

    interpreter:
      - zsh
    # Prints the command to STDIN

• The name of the default job. This could be used to give meaningful names to the default job, instead of main.

Changes to the document containing the jobs:

• The command key (which is currently a multi line String) should be an Array of String. The interpreter is invoked for each element.
• The value of the depends_on key should be an Array instead of the current String|Array
  (if the job has only one dependency, it isn't difficult to put a - before its name)
  If the output of the job have to be saved to an environment variable, it should be written as following:

  # Current
  depends_on:
    - job: sample_job
      env: VARNAME
    - other_job
    - job: another_job
      env: OUTPUT
  
  # New
  depends_on:
    - sample_job: VARNAME
    - other_job
    - another_job: OUTPUT

• A job should have a description key (String), which would contain a short description of the job. If an option will be added to neph to list available jobs (--list), this could be used to construct output similar to a help message.
• A job should have a private key (Bool). If a job is private, it can't be run with neph [job_name], only from depends_on fields (inspired from the private keyword in Crystal). The purpose of this is to keep the job listing output clean, and avoid accidentally launching the wrong job. The description and private properties should conflict, because private jobs won't show up in the output of --list.

Multi-jobs selection

Hi,

It could be useful to run multiple targets (as make does).

Is this feature in the roadmap @notramo @tbrand ?

Regards,

Having a (very) verbose mode

Hi @tbrand,

It could be helpful to display stdout / stderr in addintion of having those logs.

At least it could be helpful to display some informations in the case we do not have access to log files (CI for example).

What do you think ?

Regards,

Move to GitLab?

@tbrand What do you think about moving this project to GitLab? It provides more powerful tools, such as diff view without MR, chat, better issue tracker, built-in CI, etc. Also it has a GitHub import tool, that can import issues, labels, projects, pull requests, etc.
I would like to develop this tool on that platform.

Mutliple configuration file

Hi @tbrand,

The configuration file in our benchmarking project is very long
https://github.com/the-benchmarker/web-frameworks/blob/master/neph.yaml

It could be easier (to maintain) if this file is splitted into parts

I mean using -f for a single **configuration** and -d` for a multiple configuration (one per language, for example), could be clearer

What do you think ?

Regards,

Dynamic environment per job

Hi @tbrand,

I am currently working on pushing https://github.com/the-benchmarker/web-frameworks on next step, running on cloud.

In order to benchmark all frameworks, I have to create a target (remote host) per implementations.
1 (build) job -> 1 host -> 1 hostname

I have then to use custom environment (dynamic) per job.

What do you think ?

Regards,

Speficy an alternative configuration

Hi,

The default configuration file is neph.yaml.

It could be useful to precise an other configuration file.

@notramo What do you think ?

Regards,

No console output

Hi @tbrand,

When running neph, since some release, I do not have any ouput / progress bar.

Regards,

uninstall

I think, the uninstall command should be removed.
Why? Because it is buggy, and in most cases it is unusable, and there can't be an uninstall job in neph.yml.

• if the user don't have privileges, it don't work (but it says that uninstalling was successful):

   ~/neph >>> neph uninstall
  Neph will be uninstalled ...
  Find 'neph' at /usr/bin/neph
  Are you sure you want to uninstall 'neph'? [Y/n]
  Y
  Uninstalling neph...
  Successfully uninstalled, thanks for using 'neph' so far!
   ~/neph >>> neph
  Neph is running (0.1.14) ||||||||||||||||||||||||||||||||||||||||||||||||||100%
  main [1/1] up to date
  
  Finished in 100.82ms
   ~/neph >>>            
  

• if the binary name is different from neph, it don't find it in PATH, but asks to remove.
• if neph was installed with distro package manager, it should be uninstalled that way
• if the user installed it manually, it can be found by running which neph, and remove manually