propensive/fury-old

A new build tool for JVM languages

Overview

Fury is a long-term project to create next-generation build tooling to address the greatest challenges of building software in an ever-changing environment, while maintaining the predictability, reliability and simplicity of builds.

Features

• fully data-oriented model for builds
• builds may be defined in terms of sources or binary dependencies
• power to control a build and the builds of all its source dependencies
• advanced support for conflict resolution amongst dependencies
• intuitive command-line API, with visual interfaces available later (Milestone 3)
• hierarchical model of builds and build distribution
• support for compiling Scala 2.x, Scala 3, Scala.js and Java
• composable variants of builds, for generalized cross-building (Milestone 2)
• support for running Docker containers as part of a build (Milestone 2)
• compilation distributed over the network (Milestone 3)
• optionally external builds, shared independently of source code

Fury Concepts

Fury introduces a number of new concepts for developers. While full documentation is in the process of being written, a work-in-progress document provides a lot of detail on these new ideas and how they work.

Release

Fury is approaching its first public milestone release, Alecto. This will be the first release of Fury capable of building a majority of Scala projects. Fury’s future plans are described in the roadmap.

Installation

Pre-release versions of Fury (with no guarantees on features or reliability) may be installed by running,

curl -Ls get.fury.build | sh

in a shell, or alternatively,

curl -Ls get.fury.build > install
./install system install

if you would prefer to inspect the script before running it.

For more details, see the full installation instructions.

Getting Started

To set up a simple Fury project, follow the short tutorial for Scala or, if you prefer, the tutorial for Java.

Website

Full information about Fury is available on the Fury website.

Contributing

Contributors to Fury are welcome and encouraged. New contributors may like to look for issues marked .

We suggest that all contributors read the Contributing Guide to make the process of contributing to Fury easier.

Please do not contact project maintainers privately with questions, as other users cannot then benefit from the answers.

Author

Fury was designed and developed by Jon Pretty, and commercial support and training is available from Propensive OÜ.

License

Fury is copyright © 2018-20 Jon Pretty & Propensive OÜ, and is made available under the Apache 2.0 License.

fury-old's Issues

allow resources to be added to modules

Some applications will need resource files to be included on their classpaths. These could just be merged from a resources directory, but I'm tempted to require that each file be specified explicitly (and checksummed) so that exact name clashes can be identified, and identical resource additions unified.

The behavior should be similar to that of adding binaries to a module, in terms of UI. But it still needs some thought to work out how it should work, in general.

I would be particularly interested in any use cases which require an entire directory to be included in a JAR.

support for terminals with fewer colors

Some terminal emulators do not support all the colors used by Fury's UI. If we can detect the capabilities of the terminal, we should do this and automatically provide the UI in "simple" color. If we cannot detect the capabilities of the terminal, we should provide "simple color" as a user configuration option.

please reconsider keeping the source private

I think build tools are hard enough to adopt that there is no real risk that you will accumulate a bunch of users who disregard the alpha status of the code.

It would be interesting to be able to see the code that is causing the issues to be closed.

use git sparse checkout from a bare repository

This should provide a neater solution to managing multiple references to the same source repository.

So, bare repositories should be stored in,

.fury/repos/<domain>/<account>/<project>/

and particular sources should be sparse checkouts from those repositories, stored in,

.fury/sources/<source-id>/

There's currently no concept of a source-id, but it could be based on a hash of repository URL, commit hash, and source path.

Apparently, git uses hard links from the bare repository, so it should not waste any space when checking out the sparse directory.

SSH confirmation prompts

I set up a clean new user account, upon attempting to run fury repo add -r gh:propensive/base the terminal becomes unresponsive and the following prompt was thrown inside the terminal I was running nailgun in -

  NGServer 0.9.1 started on all interfaces, port 8462.
  The authenticity of host 'github.com (140.82.118.3)' can't be established.
  RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
  Are you sure you want to continue connecting (yes/no)?

adding sources does not allow a module to be specified

The command,

fury source add -d <dir>

does not have a --module parameter, so you have to select the right module before adding the sources.

execution script: doesn't use bundled scala

The `fury' script doesn't explicitly use the bundled scala.

If there is a pre-existing scala 2.11 on the PATH then nailgun starts but then the following error is thrown -

java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: scala/runtime/java8/JFunction1$mcVI$sp
	at fury.Main$.nailMain(main.scala:49)
	at fury.Main.nailMain(main.scala)
	at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.martiansoftware.nailgun.NGSession.run(NGSession.java:280)
Caused by: java.lang.NoClassDefFoundError: scala/runtime/java8/JFunction1$mcVI$sp
	... 6 more
Caused by: java.lang.ClassNotFoundException: scala.runtime.java8.JFunction1$mcVI$sp
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 6 more

check for cycles in dependencies

There's nothing to stop users creating cycles between module dependencies. I've not tested this, but it probably creates a stack-overflow error.

include support for filewatching

Fury should be able to monitor directories for changes to source files, and to automatically start compilation when a file is added, removed or modified.

coursier not found

When trying to build a minimal Hello World example, after setting up everything like shown by Jon in the ScalaX2gether workshop, I get:

java.io.IOException: Cannot run program "coursier" (in directory "/Users/heiko/tmp/helloworld"): error=2, No such file or directory

publishing to Maven Central

It should be possible to publish to Maven Central (or Bintray) from Fury, if a project has only binary dependencies which are already on Maven Central (or Bintray). This file should contain the essence of what's required to publish to Maven Central. I think we can probably make everything a lot simpler than it is in here, though.

If possible, I'd prefer not to have any dependencies on third-party libraries, so if we can implement HTTP calls using java.net functions, that would be preferable. I'd also like to use Xylophone for XML, though this project needs some maintenance first.

upgrade to Bloop v1.1.0

We currently use Bloop v1.0.0, however v1.1.0 was released recently. We should try the newer version, as it should offer some performance improvements, and bugfixes.

work out how to build multiple schemas in parallel

It would be very convenient to be able to make changes to source code and compile them with different compilers (i.e. using different schemas) at the same time. The simple solution may be to just run two compile processes, specifying different scheas, with filewatching, in parallel, in which case there's nothing to be done.

rename "workspace" to "layer"

I've settled on this new terminology, as it gives a better indication about how one "workspace" is overlaid on top of other workspaces.

updating module removes compiler setting

When running a command such as,

fury module update -M com.propensive.Plugin

the compiler setting gets reset to java/compiler.

adding local repositories fails silently if a name is not provided

When adding a local directory as a repository using -d, if the -n flag is also omitted, the repository is not added, but without any error message reported. It should take the name from the last part of the directory name, if it's not included.

eliminate `recv: Connection reset by peer` messages

Sometimes, noticeably during commands which call out to shell processes, Fury's UI exits reporting the message,

recv: Connection reset by peer

This appears to be a race-condition, though the exact cause is not clear. Frequently, the command output is cut short, and the error message displayed instead. It's possible that Nailgun may be involved in the interaction.

`fury prompt` command to output current context for shell prompt

As suggested by Roberto Serrano, it would be useful to see the current Fury workspace context in the shell prompt. This should be implemented as a command (maybe with configuration parameters) which shows outputs the current state of the Fury workspace.

implement BSP

After discussions with Justin Kaeser, Fury should support BSP as a means for IDEs such as IntelliJ to get information about the build out of Fury.

Bloop stops working after a while, when started through Fury

When starting Fury, if the Bloop server is not already running, an asynchronous process will be launched to run the Bloop server. When started this way, after several compilations, Bloop appears to freeze, and compilation hangs. This isn't the case when starting Bloop manually from the command line.

running without Git user details set causes problems

I set up an empty new user account to test fury.

I received the following error when attempting to run fury repo add -r gh:propensive/base -

fatal: empty ident name (for <(null)>) not allowed  

support for intransitive dependencies

When adding a source or binary dependency, we want some dependencies to be included in the classpath of every module downstream of them. But others, such as those which are only involved in running the build, do not need to be included beyond the module which directly depends on them. Adding a dependency with the --intransitive flag should distinguish between these two cases.

build on a continuous integration server

Would it make sense to build fury on https://travis-ci.org or a similar service so there is an official build?

increase the size of the undo buffer

It currently only has one level, so you can only undo once. A minimum of about five seems reasonable.

source directory completions get truncated

Completions for possible source directories get truncated (the first character) if the repository directory was added with a trailing slash. This can be probably be fixed by normalizing the path at the point it's added.

add a `--sync` flag to all commands to block on async operations

As many long-running tasks are performed asynchronously, shell scripts which call a series of Fury commands need to manually handle awaiting completion of these processes.

All long-running Fury commands should support a --sync flag which runs these operations synchronously.

design user interface for specifying when shading should happen

This complements issue #49. The model needs to support, in a convenient way, the ability to specify that a dependency of a project should be shaded. This should probably be a flag in the dependency (like intransitivity), but needs more thought.

Particular thought needs to go into shadings that arise out of a need to resolve diamond dependencies, and the workflow involved with resolving such a conflict. There may be implications for issue #44, too, as pipelining of compilations will not work across a shading boundary.

publish project configuration and dependencies

Thanks for sharing the sources! 🎉

Can we get a list of dependencies fury uses, so that we can build the binary ourselves?

Looking at the imports, it seems to me the only non-propensive dependency seems to be the nailgun. Is that correct? Would simply checking out all projects from the organization and building them together (with nailgun) be enough for building fury?

more ambitious parameter handling

It's often useful to use a common set of compiler parameters everywhere, rather than have to define them for every module which uses the same compiler. But there may still be variations (additions and deletions) in the parameters defined for some modules.

calculate hashes for each build specification

A hash should be calculated for each build specification, consisting of:

• the compiler
• the dependencies (including compiler plugins)
• a subset of the compiler parameters (those which change compiler output)
• the sources being compiled

Any change to the specification of the build which may change the output should produce a different hash.

Fury should be able to build itself

And probably messing up horribly. From inside fury's git clone

fury init
fury project add -n fury
fury repo add -r gh:propensive/base
fury import add -i base:2.12.6
fury module update -c scala/compiler
fury source add -d local:src/build
fury source add -d local:src/core
fury source add -d local:src/dependency
fury source add -d local:src/imports 
fury source add -d local:src/main/scala-2.12
fury source add -d local:src/menu
fury source add -d local:src/module # and I stop here because I start getting duplicated classes (?)

fury repo add -r gh:propensive/guillotine

fury

not found: object guillotine
[E]       L18: import guillotine._

But now running fury gets into bloop and the compiler does not seem to know where to find guillotine. I missed/messed something, for sure :)

hide schema in context when there is only one

The user does not need visibility of the schema when there is only one. This applies to the command prompt, and to the "context" line which is shown for each command.

The behavior should be that adding the first new schema should actually just rename the current one, and enable multi-schema builds. Subsequent fury schema add invocations should actually add new schemas.

suggestions for `-i` parameter in `import delete` are separated by `/` instead of `:`

dispatch subgraphs of work to Bloop instead of every module

Fury manages all dependencies between modules, and dispatches independent compilation requests to Bloop, one for every module. Bloop has no awareness of the dependencies between modules.

But Bloop configuration files could be made aware of dependencies between library, plugin and compiler modules (but not application modules), and Fury could take advantage of faster pipelining in Bloop between these modules.

We need to work at the "subgraph" level because it's not possible to get Bloop to run an application in the middle of a build (only at the end), and we can't pipeline compilation across an application module anyway (because it has to complete compilation and run before the next module can start compiling). So Fury will need to compute subgraphs containing at most one application module.

This also requires using BSP to get progress information from Bloop, i.e. the details of when each module starts or finishes compiling.

renaming a module does not work

The command,

fury module update -n new-name

does not work, and throws an exception. It appears that this is because the lens is looking for the module by either its old name or its new name, when it should be doing the opposite.

support running Fury on Windows

I'm not sure what is required to get this to work, and I don't know what is possible in terms of getting things like tab-completion working. Is it supported at all in the Windows shell?

column and field access to tabular data

A lot of Fury output comes in a tabular format, and it is possible to use the --raw flag to show just the index column from the table. It would be useful to be able to show other columns, for example by providing the --column option (which would take a lowercase column name as an argument) and/or a --row option, which would take the index for the row, and return just a single field.

use compact OGDL format for differences

Schemas are "complete copies of the entire workspace", but could be serialized (and deserialized) more efficiently. Given two OGDL structures, it should be possible to represent the second as a set of changes relative to the first, dramatically reducing the amount of data which needs to be stored.

verification of module references in builds

It's possible to create links in builds to modules which do not resolve. This is fine during setup of the build (for example, you can specify that the compiler to use should be scala/compiler before you've imported a workspace containing the Scala compiler. But we need to check that everything resolves before attempting to run a build or publish a workspace.

References to modules which do not resolve should be marked with a bright red !! (or similar) until they are fixed.

add constraints to publishing (and reading) layers

It should not be possible to publish a workspace which contains references to files on the local disk, or which points to a branch or tag of a remote repository (rather than an exact commit hash). Nevertheless, anyone can publish a workspace with git, so the constraint should primarily be on reading a remote workspace which contains such references and secondarily on publishing.

allow Fury to run in a custom working directory

Fury currently always runs in the current working directory. It would be useful to be able to specify the directory with a parameter, e.g. -W/-working-dir.

fury --working-dir /home/work/myproject <other commands>

At the very least, this should be available for aliased commands. It would enable shell aliases which point to Fury applications to be defined.

add code formatter to the build process

Fury codebase has not formatter applied. In order to prevent merge conflicts in the future we need to have a standard and enforced code format.

make the `.fury` temporary directory safely deletable

It should be possible to delete the .fury directory, and for it to be automatically recreated as necessary from the information in the workspace.fury file.

OGDL format repeats the ID field when serializing SortedSets

Given that the ID is used as the key in the list, the id <id> entry which appears underneath it is not necessary.

clarify names of `add` and `delete` subcommands

Add/remove sounds more natural than add/delete, but I'm not sure if it is just to me. If I'm not the only one, I guess having remove to alias internally to delete would be helpful

better tab completion in zsh

It should be possible to improve the tab completion in ZSH. There is documentation on using the compadd command directly, which looks like an easier route towards specifying all possible completions than the current approach of using the ZSH library commands.

no suggestions for `-i` parameter when adding an import

tab completion for bash and fish

Fury currently only supports tab completion in ZSH, which provides a superset of Bash's functionality. It should not be too difficult to emit Bash completion output derived from the information which is already generated for ZSH.

I know nothing about Fish, though.

provide shading API

We need to support shading as a solution to conflicting projects. This is a step which should happen following compilation, and involves rewriting the classfiles of both the dependency and the dependent.

This issue covers the implementation of the shading task, whose API should take two directories corresponding to the dependency's and dependent's classfiles, and a further two directories to put the rewritten shaded classfiles.

updating (git pull) on repositories does not work

It appears that some of the changes to how repositories are cloned broke fury repo pull command.