standardml/smackage

Smackage Package Manager for Standard ML

smackage's Issues

"smackage refresh" failure

The refresh command is failing with the following error:

Permission denied (publickey).
fatal: The remote end hung up unexpectedly
WARNING: When trying to pull source smbt git [email protected]:finrod/smbt.git', got the following error "Fail: I/O error trying to access temporary file" If this line is in sources.local, you may need to run smackage unsource' to remove it.
Permission denied (publickey).
fatal: The remote end hung up unexpectedly
WARNING: When trying to pull source smbt git [email protected]:finrod/smbt.git', got the following error "Fail: I/O error trying to access temporary file" If this line is in sources.local, you may need to run smackage unsource' to remove it.

Reduce dependency on system commands

magenbluten writes in comments to #15

as i already mentioned "rm -Rf" doesn't work on windows systems. aditionally the "FSUtil.systemLines" uses "someprog > somefile" for capturing and parsing program output. this of course does also not work on windows systems and should be fixed. i think of writing a "shell-monad" similar to "hsh (haskell shell)" for similar tasks.

rm -Rf is indeed one of the couple of things that necessitate running smackage within the cygwin environment in Windows, and it would be better to create a SML-level facility for crawling the directories, deleting stuff and then deleting the directories.

I believe our testing was that pipe-to-file works in the Windows cmd shell - can we get a confirmation that pipe-to-file does not work using OS.Process.system in Windows XP/Vista/7? I was under the impression that it did.

If that doesn't work, I don't know of a good alternative, since "fork" only exists in Posix.

Basic support for FFI-relevant code

I have a modest proposal. We have a reasonable number of projects now that rely on some kind of FFI code (usually C) in order to build end-user applications. This makes figuring out how to build applications reasonably complicated. I propose that we add another smackage command that interrogates a package (possibly just running another 'make' target) in order to compute the appropriate command line/CM options. We need not do anything clever in the realm of attempting to discover platform-specific weirdness (although this might be a nice v2 feature). An example usage might be:

(in my end-user Makefile):

foo: bla.mlb bla.sml
        mlton $(exec smackage linkopts curlsml v1) -output foo bla.mlb

Where smackage linkopts curlsml v1 is going to expand to something like:

-linkopt -lcurl $SMACKAGE_HOME/lib/curlsml/v1/curlsml.a

I can see some potential risks in platform-specific support being required, and we might have to think carefully about how one generates things appropriately for the chosen compiler/platform.

Any thoughts?

Better approach to versions.smackspec

as magenbluten says in comments to #15, the $SMACKAGE_HOME/versions.smackspec could become the victim of catastrophic success.

one a side note i would like to see a switch to sqlite (or say sql in general) for keeping the repo index. the txt file storage will become inefficient if more queries are to be made and more packages get added...

This is worth keeping in mind, though the time it takes to do ~1000 "git ls-remote" will be a problem long before the time or space that it takes to store 100 versions of 1000 packages becomes noticeable to the user.

Smackage should learn how to build things

If you do (as of 1.2.1):

smackage get smbt
smackage make smbt
smackage make smbt install

You get an SML build tool that is FFI-aware, among other things. We should consider how to get smackage to make use of this (smbt needs documenting, uh... yeah...). Something like this might work:

smackage build foobar mlton

actually calls:

cd $SMACKAGE_HOME/lib/foobar/v1
smbt build.sm mlton

Which (assuming 'foobar' has an appropriate build.sm file) will do approximately the right thing. Smbt is currently smackage-aware, in the sense that it expects Smackage to tell it how to find dependencies (with the new `pathinfo' command as of 1.2.0).

Smackage need not be used with smbt in every case, but we should consider whether it could advance the cause by allowing it to be used easily, particularly given that it hides the nastiness associated with embedded FFI dependencies in libraries.

Complain when pkg.smacspec references version without tag

I inevitably forget to add a tag when bumping pkg.smacspec's version,
This seems to currently silently ignore the update,
it would be nice if it complained, or perhaps just a verbose mode to help figuring out why.

Generalize config file stuff

Magenbluten is currently working on adding mercurial support, but in its current form it will need to be disabled by default lest we break support on computers without hg. This led me to think that it might be time to think about command-line ways of manipulating the $SMACKAGE_HOME/config file. What about something like this?

$ smackage config help
Manipulate configuration options; type `smackage config' to see current configuration options.

smackage {enable/disable} compiler {mlton,smlnj,polyml} (Currently unused)
smackage {enable/disable} protocol {git,hg} (Requires appropriate binary)
smackage {enable/disable} platform {osx,win,linux} (Multiple platforms are not recommend) <-- or allowed?
smackage {enable/disable} source <absolute filename or filename relative to smackage-home>

Configuration options are stored in /Users/rjsimmon/.smackage/config

Enabled by default:

• source sources.local (no reason to disable this, as it's user-configurable as well)
• protocol git (probably we can't allow this to be disabled for now)

Test and use different cygwin file for smackage on "win" platform

Karl asked me to use the following .mkexec on Windows machines, though it may or may not work with "Cygwinned sml".

#! /bin/sh
#
# Batch File Creator
#
# Arguments:
# $1 = SMLNJ runtime
# $2 = Parent directory of bin (which contains binaries and heap image)
# $3 = Name of executable (e.g. celf)
v=`cygpath -w $2` 
cat > "$2/bin/$3" <<EOF
#! /bin/sh
exec "$1" @SMLload="$v/bin/.heapimg" "\$@" 
EOF
chmod a+x "$2/bin/$3"

Requiring smackspecs?

I just got an UNEXPECTED ERROR when I tried to download cmlib - are we going to require smackspecs now? It seems unnecessary if a package has no installables or dependencies to go through that hoop.

Smackage should indicate when a new major version is available

I think the behavior of not updating across major versions is the correct one, but we should give some feedback that this is what we're doing:

$ smackage update
Selected `cmlib 1.0.1'.
Package `cmlib 1.0.1' already installed.
Selected `preml 1.4.8'.
Repository is updated
Package `preml 1.4.8' downloaded.
NOTICE: preml v2.0.4 is available, run `smackage get preml' to get it
Selected `twelf 1.7.1'.
Package `twelf 1.7.1' already installed.
NOTICE: twelf v3.9.2 is available, run `smackage get twelf' to get it

Add optional "minimum required versions" to smackspec requires, use it to install packages faster

In addition to the existing smackspec syntax

require: cmlib v3

which specifies that the user will only refer to "v3", it would be helpful to have a "minimum" syntax, like this:

require: cmlib v3 (v3.19.12)

which says that v3 will satisfy their constraints, but that it must be at least v3.19.12 (this only makes sense when the constraint - v3 in this case - satisfies the minimal version - v3.19.12 in this case). This is the same as the common semantic version constraint pattern "<4.0.0 >=3.19.12" seen in other tools.

Once this syntax was in place, the "resolve dependencies" code could be made slightly more complicated in order to notice the case where, while 3.19.12 is required and 3.30.0 is available, 3.24.4 is already installed in the system, so there's no need to get the most recent version.

Installing Twelf w/ smackage & mlton : invalid file suffix on build/twelf-server-mlton.cm

Hi folks, I am trying to install Twelf with smackage, as described in the smackage readme. I am using the mlton compiler.

However, I am getting the following error. Any help would be appreciated.

$ smackage make twelf mlton
In directory: `/home/willard/.smackage/lib/twelf/v1.7.1'
smackage is preparing to run `make DESTDIR=/home/willard/.smackage mlton'
*************************************************
Twelf Server
*************************************************
rm -Rf src/frontend/buildid.sml
bin/buildid >src/frontend/buildid.sml
mltonversion=`mlton -default-ann 'nonexhaustiveMatch ignore' 2>&1 | awk 'NR==1 { print 0+$2 }'`;        \
if   [ $mltonversion -ge 20041109 ]; then                       \
        cmfileid="twelf-server-mlton.cm";                       \
elif [ $mltonversion="MLTONVERSION" ]; then                     \
        cmfileid="twelf-server-mlton.cm";                       \
else                                                            \
        echo; echo "Error: MLton >= 20041109 required"; echo;   \
        exit 1;                                                 \
fi;                                                             \
mlton -default-ann 'nonexhaustiveMatch ignore' -output bin/twelf-server build/${cmfileid}
invalid file suffix on build/twelf-server-mlton.cm
usage: mlton [option ...] file.{c|mlb|o|sml} [file.{c|o|s|S} ...]
    -align {8|4}                    object alignment
    -as-opt <opt>                   pass option to assembler
    -cc-opt <opt>                   pass option to C compiler
    -codegen {native|amd64|c|llvm}  which code generator to use
    -const '<name> <value>'         set compile-time constant
    -default-ann <ann>              set annotation default for mlb files
    -default-type '<ty><N>'         set default type
    -disable-ann <ann>              disable annotation in mlb files
    -export-header <file>           write C header file for _export's
    -ieee-fp {false|true}           use strict IEEE floating-point
    -inline <n>                     set inlining threshold
    -keep {g|o}                     save intermediate files
    -link-opt <opt>                 pass option to linker
    -llvm-as-opt <opt>              pass option to llvm assembler
    -llvm-llc-opt <opt>             pass option to llvm compiler
    -llvm-opt-opt <opt>             pass option to llvm optimizer
    -mlb-path-map <file>            additional MLB path map
    -mlb-path-var '<name> <value>'  additional MLB path var
    -output <file>                  name of output file
    -profile {no|alloc|count|time}  produce executable suitable for profiling
    -profile-branch {false|true}    profile branches in addition to functions
    -profile-stack {false|true}     profile the stack
    -profile-val {false|true}       profile val bindings in addition to functions
    -runtime <arg>                  pass arg to runtime via @MLton
    -show-basis <file>              write final basis environment
    -show-def-use <file>            write def-use information
    -stop {f|g|o|tc}                when to stop
    -target {self}                  platform that executable will run on
    -target-as-opt <target> <opt>   target-dependent assembler option
    -target-cc-opt <target> <opt>   target-dependent C compiler option
    -target-link-opt <target> <opt> target-dependent linker option
    -verbose {0|1|2|3}              how verbose to be
make: *** [Makefile:47: twelf-server-mlton] Error 1

$

Ask what compilers the user would like us to use?

The code that makes the "config" file should probably ask the user for input. Here's the script I'm imagining; I'll do it this weekend if this is agreeable to everyone.

NOTICE: No config file, trying to create /Users/rjsimmon/.smackage/config
NOTICE: To do this, I need to ask you a few questions.
Do you have "mlton" (MLton) installed and on your search path? [Y/n]: Y
Do you have "sml" (SML/NJ) installed and on your search path? [Y/n]: Y
Would you like "mlton" to be the default compiler? [Y/n]: n
MLton and SML/NJ enabled, SML/NJ will be the default compiler.
You can change these settings by editing /Users/rjsimmon/.smackage/config

NOTICE: No config file, trying to create /Users/rjsimmon/.smackage/config
NOTICE: To do this, I need to ask you a few questions.
Do you have "mlton" (MLton) installed and on your search path? [Y/n]: n
Do you have "sml" (SML/NJ) installed and on your search path? [Y/n]: Y
MLton enabled as the default compiler.
You can change these settings by editing /Users/rjsimmon/.smackage/config

NOTICE: No config file, trying to create /Users/rjsimmon/.smackage/config
NOTICE: To do this, I need to ask you a few questions.
Do you have "mlton" (MLton) installed and on your search path? [Y/n]: n
Do you have "sml" (SML/NJ) installed and on your search path? [Y/n]: n
WARNING: No default compiler, smackage will only be able to download code.
You can change these settings by editing /Users/rjsimmon/.smackage/config

Incomplete semantic version produces misleading warning message

Given a project with an incomplete semantic version as a tag (say, v0.1), running smackage refresh produces the following error message (twice):

WARNING: When trying to pull source `package-name git package-uri', got the following error 
	"Fail: I/O error trying to access temporary file"
If this line is in sources.local, you may need to run
`smackage unsource' to remove it.

---

As far as I can tell, this is caused by a combination of src/get-git.sml and src/semver.sml. In src/get-git.sml, any Fail exception is propagated as Fail "I/O error trying to access temporary file":

smackage/src/get-git.sml

Lines 32 to 45 in eb383f0

	val tag =
	case remote of
	[ "refs", "tags", tag ] =>
	if String.sub (tag, 0) = #"v"
	then String.extract (tag, 1, NONE)
	else raise SemVer.InvalidVersion (* Not a version tag *)
	| _ => raise SemVer.InvalidVersion (* Not a tag at all *)
	in
	SOME (hash, SemVer.fromString tag)
	end handle Fail s => raise Fail s
	| _ => NONE
	in
	List.mapPartial process input
	end handle _ => raise Fail "I/O error trying to access temporary file"

Inconveniently, fromString raises Fail given an incomplete semantic version:

smackage/src/semver.sml

Lines 120 to 124 in eb383f0

	fun fromString s =
	case fromString' s of
	(major, SOME minor, SOME patch, special) =>
	(major, minor, patch, special)
	| _ => raise Fail ("`" ^ s ^ "` is an incomplete semantic version")

Thus, rather than

WARNING: When trying to pull source `package-name git package-uri', got the following error 
	"Fail: `0.1` is an incomplete semantic version"
If this line is in sources.local, you may need to run
`smackage unsource' to remove it.

being mentioned, the misleading warning is produced.

It seems like this could be fixed by declaring a local exception to raise and handle rather than using Fail. If this sounds reasonable, I'd be more than happy to put together a PR!

standardml/cm{lex,yacc} repos are missing, install fails.

After cloning the repository and running make mlton, bin/smackage refresh fails with:

fatal: remote error:
  Repository not found.
WARNING: When trying to pull source `cmlex git git://github.com/standardml/cmlex.git', got the following error
	"Fail: I/O error trying to access temporary file"
If this line is in sources.local, you may need to run
`smackage unsource' to remove it.
fatal: remote error:
  Repository not found.
WARNING: When trying to pull source `cmyacc git git://github.com/standardml/cmyacc.git', got the following error
	"Fail: I/O error trying to access temporary file"
If this line is in sources.local, you may need to run
`smackage unsource' to remove it.

It looks like the cmlex and cmyacc repositories are no longer available under the standardml org on GitHub.

Bullshit symlink doesn't work in a cross-platform way

If I build the non-posix version (make win+smlnj), it works on Linux (according to Sully who developed it) and Cygwin+SMLNJ-for-Cygwin according to Rowan.

However, on OSX and Karl's Cygwin+SMLNJ-for-Windows platform, bullshit symlink (a.k.a. "cp -r") doesn't work, so "v1.4.5" gets populated but "v1.4" and "v1" do not.

We may need to actually recurse through directory structures and copy files...

Suspicion: install is going to be a problem, support make instead?

(Originally created this as a milestone, but I kind of wanted to have comments, so doing this instead.)

I wanted to ask if we want to try to get to v1, and what we should try to support there.

There are all sorts of potential dependency and platform detection and support issues with actual installation that I'm convinced we haven't worked through yet which are somewhat tangential to the problem that smackage is currently able to solve, and my experience has been that debugging windows issues is pretty maddening.

However, smackage can do a pretty good job of getting Standard ML code onto people's systems where it can refer to other Standard ML code that smackage also puts onto people's system. I think a reasonable way to punt on this for now is to have platform sections be existant but totally meaningless in v1, but to support the following command:

smackage make pkg <version> ...

which would just run make ... in the home directory of pkg .

That way we allow installation, but totally punt on the question of how to build binaries, detect systems, deal with FFI or C code - anything like that - while still making a flexible system that people can use.

We can even support an (optional) upgrade path by just telling people to run the following:

smackage make smackage {mlton,smlnj,win+mlton,bsd+ocamljs,whatever} 
smackage make smackage install

Transitive dependencies

If I run smack down foo and foo depends on bar, it should read those dependencies out of the foo.smackspec and try to install the dependency for bar, and so on and so forth. Certainly this should happen before any installation process for foo.

Setting $SMACKAGE_HOME is broken.

After the b46b537 commit, an initial smackage install via make install attempts to install to $(DESTDIR)/bin/, where $(DESTDIR) is not set. Thus

• The default install location is /bin, not ~/.smackage/bin as documented.
• $SMACKAGE_HOME is ignored, contrary to the documentation. Changing the install location requires reading the Makefile to notice the DESTDIR variable.

At the least, the README should be updated to reflect reality. The default install should probably be moved back to ~/.smackage/bin instead of /bin.

Does this work in Windows?

We should check.

Store installed packages; support 'smack update'

There's currently a file $(SMACKAGE_HOME)/packages.installed that isn't being used for anything. It should basically record the history of installed packages. If I run

$ smack install foo v1
$ smack install foo v1.5 
$ smack install bar v2

and bar v2 depends on baz v3.9, then packages.installed should automagically include the following:

bar 2 
baz 3.9
foo 1
foo 1.5

This will support a command smack update that essentially reruns install on all of the version constraints stored in the repository.

Clean up temporary files

^^^ do that.