shinmera/deploy

Deployment tools for standalone Common Lisp applications

About Deploy

This is a system to help you easily and quickly deploy standalone common lisp applications as binaries. Specifically it is geared towards applications with foreign library dependencies that run some kind of GUI.

How To

In order to make use of Deploy, you'll have to change the ASDF system definition of your project to contain the following properties:

:defsystem-depends-on (:deploy)
:build-operation "deploy-op"
:build-pathname "my-application-name"
:entry-point "my-package:my-start-function"

Once you have updated your system appropriately, all you need to do is start a fresh instance of your implementation from a terminal and run the following function:

(asdf:make :my-system)

This will build your system, gather the necessary information, and deploy a standalone bin folder within your project's root directory. You can then ship this folder to your users.

Customising Foreign Libraries

Sometimes you might want to designate a specific library for deployment, rather than the one used during development. If this is the case, you have to help Deploy out a little by defining extra information about the library with define-library. If the foreign library is in the source tree of a lisp library, you can simply associate the CFFI library with the system that provides it, and Deploy will find it automatically:

(deploy:define-library cffi-library-symbol
  :system :system-name-that-defines-the-library)

For example, the cl-mpg123 system provides a single library, which we would annotate like this:

(deploy:define-library cl-mpg123-cffi:libmpg123
  :system :cl-mpg123)

If the file is not contained in the directory of the system that provides it, you can also designate specific source directories to scan:

(deploy:define-library cffi-library-symbol
  :sources '("/some/path/where/the/library/is/stored/")) 

Finally, you can also specify the path directly if you want Deploy to choose a particular file, rather than trying to find one on its own:

(deploy:define-library cffi-library-symbol
  :path "/some/path/to/the/file.so")

Generally though these extra associations should not be necessary as Deploy will simply take the path that CFFI has already figured out to find the library.

Sometimes it might not be desired to deploy all the libraries, or reload them all upon boot. You can change this behaviour with define-library's :dont-deploy and :dont-open properties respectively.

Extending Deployment and Boot Behaviour

Deploy also offers a hooks system with which you can easily extend the steps performed during deployment and during the boot process of the generated executable. With the define-hook macro you can add functions that are executed during various points of the process. Specifically, the following types are available:

• :deploy These functions are responsible for copying files into the deployment target directory.
• :build These functions should prepare the system for the dump to binary. Specifically you might want to shut down existing threads, close file handles, remove unneeded baggage, and remove potentially sensitive information about your system.
• :boot These functions are run right before the primary entry point is executed. Thus they are responsible for preparing the runtime to continue by restarting threads, re-opening files, and so forth.
• :quit These functions are run right before the executable exits completely. They offer a last-minute opportunity to dump some information about the system, or to clean up vital resources.

If you would simply like to include a data directory to bundle with the rest, use define-resource-directory. After boot, all of the resource files will be in the directory returned by data-directory.

Deploying to an OS X App Bundle

If you would like a nicely bundled .app for OS X, you can simply change the build-operation in your ASDF system file to osx-app-deploy-op. If you would like to customise the Info.plist file that is generated for the app, you can change *info-plist-template* to point to a file that contains what you want.

Debugging a Deployed Executable

If you're having trouble with an application that's already deployed, there's a few things you can do to debug it by setting environment variables. The following are recognised by Deploy:

• DEPLOY_DEBUG_BOOT if set to a non-empty value, on error the debugger is invoked rather than just exiting the application.
• DEPLOY_REDIRECT_OUTPUT if set to a file path, the output of all streams is redirected to this file.

Particularly on Windows and OS X debugging can be an issue, as a GUI application will not get a standard output to write to. In that case, the above redirect might help. Alternatively, on Windows, you can build your binary with the feature flag :deploy-console present, which will force it to deploy as a console application.

Support

If you'd like to support the continued development of Deploy, please consider becoming a backer on Patreon:

deploy's Issues

The loader tried loading the symbol named NKEYMAPS/TYPES into the package named MYCUBICLE/CLI, but the package did not get defined, and does not exist.

I get this error when running deploy, but I cannot se anything wrong in the package.
The asd imports "types.lisp" as the first file: https://github.com/atlas-engineer/nkeymaps/blob/master/nkeymaps.asd#L12
And this file defines the package: https://github.com/atlas-engineer/nkeymaps/blob/master/types.lisp#L4

Wrong bin/ path for package inferred system

When building a package-inferred system which specifies :pathname parameter in its defsystem, Deploy puts a binary into the bin directory inside the folder pointed by :pathname.

For example, having defsystem like this:

(defsystem "lake"
  :class :package-inferred-system
  :pathname "src"
  :depends-on ("cl-syntax-interpol"
               "lake/main")
  :defsystem-depends-on (:deploy)
  :build-operation "deploy-op"
  :build-pathname "lake"
  :entry-point "lake/main:uiop-main")

and building this system, will create a binary src/bin/lake. However I'm expected it to be bin/lake, because documentation says that folder is created within your project's root directory.

This should be fixed somehow. Or if this behaviour is expected, the documentation should be fixed.

Potential Shared Library Extract

Draft source code, it should lend an idea as to how I believe it can work. Feel free to copy/modify and integrate into deploy:

(in-package :cl-user)

(uiop:define-package shared-library-extract
    (:use :common-lisp))

(in-package :shared-library-extract)

(defvar *object-tool* "otool")
(defvar *cp* "cp")
(defvar *copyable-library-paths* (list "/opt/local/lib"))
(defvar *install-name-tool* "install_name_tool")

(defun object-tool (path)
  (flet ((process-line (line) (first (str:split " " (str:trim line)))))
    (with-input-from-string (output (uiop:run-program (list *object-tool* "-L" path)
                                                      :output '(:string :stripped t)))
      (loop for line = (read-line output nil)
            while (not (null line))
            collect (process-line line)))))

(defun copy-library-p (library-path)
  (find-if (lambda (i) (str:containsp i library-path)) *copyable-library-paths*))

(defun copy-file (from to)
  (format t "Copy file ~a to ~a~%" from to)
  (uiop:run-program (list *cp* from to)))

(defun file-name (path)
  (car (last (str:split "/" path))))

(defun install-name (from to path)
  (uiop:run-program (list *install-name-tool* "-change"
                          from to path)))

(defun install-names (library)
  (format t "Install names for library ~a~%" library)
  (loop for object in (object-tool library)
        if (copy-library-p object)
          do (install-name object (format nil "@executable_path/../Resources/~a" (file-name object)) library)))

(defun library-dependency-tree (library-path)
  (let ((queue (object-tool library-path))
        (seen-libraries))
    (remove-duplicates
     (loop for library = (pop queue)
           while (not (null library))
           if (copy-library-p library)
             do (unless (find library seen-libraries :test #'equal)
                  (push library seen-libraries)
                  (setf queue (append queue (rest (object-tool library))))
                  (format t "Process Library: ~a ~%Depends on:~a~%" library (rest (object-tool library))))
           if (copy-library-p library)
             collect library)
     :test #'equal)))

(defun process-library (library-path destination)
  (loop for library in (library-dependency-tree library-path)
        do (let ((destination-path (format nil "~a/~a" destination (file-name library))))
             (copy-file library destination-path)
             (install-names destination-path))))

(defun process-libraries (libraries destination)
  (mapcar (lambda (i) (process-library i destination)) libraries))

Cannot silence status messages anymore

After this commit I cannot generate a binary and have all the status messages silenced anymore.

I used to have the following in my build script:

;; Disable :deploy status messages for the final binary
;; but enable it during the build process
(setf deploy:*status-output* nil)
(let ((deploy:*status-output* t))
  (asdf:make :cg :force t))

and it used to get the job done; but now, every time I execute the binary it would log a bunch of stuff on stderr:

$ ./bin/cg --version
 ==> Performing warm boot.
   -> Runtime directory is /Users/matteolandi/my-env/opt/cg/bin/
   -> Resource directory is /Users/matteolandi/my-env/opt/cg/bin/
 ==> Running boot hooks.
 ==> Reloading foreign libraries.
 ==> Launching application.
0.4.0-r61

Is there a way to disable the override introduced in that commit?

Source location for deployed package

I'm trying to ship Nyxt with its source, and since we can hack Nyxt live like any CL program, we might want to find the source from the deployed image.

The image should thus resolve the source on the target system, instead of the build system.

I asked there for insights: https://old.reddit.com/r/Common_Lisp/comments/vc6hix/source_location_for_deployed_binary/

/u/lispm pointed out that we could use logical pathnames. I tried it and seems to work beautifully:

atlas-engineer/nyxt#2371

There are some pitfalls though, namely that Emacs' next-error won't find the source location corresponding to warnings and errors.

All that said, I was wondering if Deploy had anything implemented to tackle this issue. If not, adding a helper like the one I've reused from SBCL + some documentation may be very helpful.

Unable to use on Windows using sbcl 2.3.2

I'm having problems using deploy on Windows. Looks like a missing library, but I'm having problems finding out what.

* (asdf:make :passish)

STYLE-WARNING:
   Generic function FSET:ITERATOR clobbers an earlier FTYPE proclamation
   (FUNCTION (T &KEY &ALLOW-OTHER-KEYS) (VALUES FUNCTION &REST T)) for the same
   name with (FUNCTION (T &KEY &ALLOW-OTHER-KEYS) *).
#P"C:/Users/simend/quicklisp/local-projects/passish/bin/passish"
 ==> Running load hooks.
 ==> Gathering system information.
   -> Will load the following foreign libs on boot:
      (#<DEPLOY:LIBRARY DEFAULT-435>)
 ==> Deploying files to C:/Users/simend/quicklisp/local-projects/passish/bin/

debugger invoked on a SIMPLE-ERROR in thread
#<THREAD "main thread" RUNNING {1000228073}>:
  #<LIBRARY DEFAULT-435> does not have a known shared library file path.

* (defparameter l (car (deploy:list-libraries)))`
* l
#<DEPLOY:LIBRARY DEFAULT-435>
* (deploy:library-system l)
NIL
* (deploy:library-path l)
NIL
* (deploy:library-sources l)
NIL

Document `DEFINE-RESOURCE-DIRECTORY`

I'm trying to define a resource directory but can't figure out how. I'll probably eventually get it if I bang my face on the keyboard enough, but I figured I'd add this for posterity.

macro (DEFINE-RESOURCE-DIRECTORY NAME DIRECTORY &KEY (COPY-ROOT T))
Shorthand to define a hook that simply deploys the given directory.

What is name? Just an arbitrary symbol?

What is directory? A string? Lisp pathname thing? Relative to the current path? Absolute?

I have a directory called assets in my current folder that I'm trying to get copied to the resulting folder, so I'm trying this:

(deploy:define-resource-directory assets "assets")

but that makes deploy explode like this:

 ==> Gathering system information.
   -> Will load the following foreign libs on boot:
      (#<DEPLOY:LIBRARY BEARLIBTERMINAL>)
 ==> Deploying files to /Users/sjl/src/rldt/bin/rldt.app/Contents/Resources/
While evaluating the form starting at line 19, column 0
  of #P"/Users/sjl/src/rldt/build/build-mac.lisp":

debugger invoked on a TYPE-ERROR in thread
#<THREAD "main thread" RUNNING {1001936E53}>:
  The value
    NIL
  is not of type
    (OR (VECTOR CHARACTER) (VECTOR NIL) BASE-STRING PATHNAME FILE-STREAM)
  when binding SB-IMPL::PATHSPEC
restarts (invokable by number or by possibly-abbreviated name):
  0: [REPORT-ERROR                 ] Print the error and continue running hooks.
  1: [RETRY                        ] Retry
                                     OSX-APP-DEPLOY-OP on #<SYSTEM "rldt">.
  2: [ACCEPT                       ] Continue, treating
                                     OSX-APP-DEPLOY-OP on #<SYSTEM "rldt"> as
                                     having been successful.
  3:                                 Retry ASDF operation.
  4: [CLEAR-CONFIGURATION-AND-RETRY] Retry ASDF operation after resetting the
                                     configuration.
  5: [RETRY                        ] Retry EVAL of current toplevel form.
  6: [CONTINUE                     ] Ignore error and continue loading file "/Users/sjl/src/rldt/build/build-mac.lisp".
  7: [ABORT                        ] Abort loading file "/Users/sjl/src/rldt/build/build-mac.lisp".
  8:                                 Ignore runtime option --load "build/build-mac.lisp".
  9:                                 Skip rest of --eval and --load options.
 10:                                 Skip to toplevel READ/EVAL/PRINT loop.
 11: [EXIT                         ] Exit SBCL (calling #'EXIT, killing the process).

(PATHNAME NIL) [external]
0] 

None of those restarts seem to give me a traceback, so I'm not sure where to look to track this down.

libsscl and libcrypto

Hi, I had issues with libcrypto and received your help on Discord. Wanted to leave a note here for future googlers.

Using Deploy, I had:

==> Running load hooks.
 ==> Gathering system information.
   -> Will load the following foreign libs on boot:
      (#<DEPLOY:LIBRARY LIBSSL> #<DEPLOY:LIBRARY LIBRT>
       #<DEPLOY:LIBRARY LIBOSICAT> #<DEPLOY:LIBRARY LIBMAGIC>)
 ==> Deploying files to /home/vince/projets/myapp/commandes-collectivites/bin/
Unhandled SIMPLE-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING
                                    {10007285B3}>:
  #<LIBRARY LIBCRYPTO> does not have a known shared library file path.

The thing to do in that case is… to not ship libcrypto, but to use the target system's.

;; "we typically want to import the ones from the target system. Deploying these libraries without them blowing up on Linux is hard."
#+linux (deploy:define-library cl+ssl::libssl :dont-deploy T)
#+linux (deploy:define-library cl+ssl::libcrypto :dont-deploy T)

But for other systems, we can tell Deploy where is our .so file:

;; (deploy:define-library cl+ssl::libcrypto
;;   :path "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1")

(I looked up the CFFI name with (apropos "libcrypto"))

Best,

Built binary attempts to load UIOP from hard disk

Windows x64, SBCL 1.4.14

ASDF options for the system being built:

  :defsystem-depends-on (:deploy)
  :build-operation "deploy-op"
  :build-pathname "ect-gui"
  :entry-point "ect-gui:main"
  :depends-on (:qtools :qtcore :qtgui :alexandria :cl-json :asdf :uiop
               :drakma :local-time :cl-ppcre :trivial-backtrace)

I compiled as user bar and sent the binary to user foo. When they run the machine, they get the following error:

 ==> Performing warm boot.
   -> Runtime directory is C:/Users/foo/Desktop/bin/
   -> Resource directory is C:/Users/foo/Desktop/bin/
 ==> Running boot hooks.
   -> Loading smoke module QTCORE.
WARNING:
   ASDF 3.3.1 (from NIL), UIOP NIL (from C:/Users/bar/quicklisp/dists/quicklisp/software/uiop-3.3.2/uiop.asd)
   -> Loading smoke module QTGUI.
 ==> Running Qtools boot hooks.
 ==> Reloading foreign libraries.
   -> Loading foreign library #<LIBRARY LIBEAY32>.
   -> Loading foreign library #<LIBRARY LIBSSL>.
 ==> Launching application.
 ==> Encountered unhandled error: Error while trying to load definition for system uiop from pathname C:/Users/bar/quicklisp/dists/quicklisp/software/uiop-3.3.2/uiop.asd: Couldn't load #P"C:/Users/bar/quicklisp/dists/quicklisp/software/uiop-3.3.2/uiop.asd": file does not exist.
;
; compilation unit aborted
;   caught 1 fatal ERROR condition
 ==> Running quit hooks. 

Some systems failed to build for Quicklisp dist

Building with SBCL 2.4.6 / ASDF 3.3.5 for quicklisp dist creation.

Trying to build commit id 80aeaae

deploy-test fails to build with the following error:

Unhandled ASDF/FIND-SYSTEM:LOAD-SYSTEM-DEFINITION-ERROR in thread #<SB-THREAD:THREAD tid=926982 "main thread" RUNNING {1000DC8003}>: Error while trying to load definition for system deploy-test from pathname /home/quicklisp/quicklisp-controller/dist/build-cache/deploy/7580fd26d8d084e255926f97167279503994c7dd/deploy-20240730-git/deploy-test.asd: The function DEPLOY::REMAVE is undefined.

deploy fails to build with the following error:

Unhandled UNDEFINED-FUNCTION in thread #<SB-THREAD:THREAD tid=926986 "main thread" RUNNING {1000DC8003}>: The function DEPLOY::REMAVE is undefined.

Full log here

How to silence status messages?

When I build a binary with deploy and run it, it spews a bunch of log messages to standard output like:

 ==> Performing warm boot.
   -> Runtime directory is /Users/sjl/src/brows/bin/
   -> Resource directory is /Users/sjl/src/brows/bin/
 ==> Running boot hooks.
 ==> Reloading foreign libraries.
   -> Loading foreign library #<LIBRARY LIBPANEL>.
   -> Loading foreign library #<LIBRARY LIBCURSES>.
 ==> Launching application.
hello
 ==> Epilogue.
 ==> Running quit hooks.

That's good for debugging the deployment, but bad for an actual utility that wants to print something to standard output for a user to read. Is there a way to silence this output? I looked in the code but didn't see anything obvious.

doc: mention compression

I just learned (here lisp-tips/lisp-tips#20 (comment)) that deploy uses compression if available. I suggest to add details to the readme.

Regards

Qtools Windows application is not saved with the .exe extension

Reproduced on Travis: https://travis-ci.com/phoe-trash/furcadia-post-splitter/jobs/249809158#L614

The resulting binary is not saved with the .exe extension.

Automatically Add zlib On Compressed SBCL

SBCL with core compression requires the zlib foreign library. It would be good if Deploy automatically bundled that with dumped executables, at least on Windows systems where it isn't commonly available.

#<LIBRARY LIBZ> does not have a known shared library file path.

Hey,

I am getting the error mentioned in the title when trying to use deploy with my a CLI application of mine.

For reference, here is the complete stack trace when the error is signaled:

 ==> Running load hooks.
 ==> Gathering system information.
ensure-library #<LIBRARY LIBZ>
   -> Will load the following foreign libs on boot:
      NIL
 ==> Deploying files to /Users/matteolandi/my-env/opt/ap/bin/
ensure-library LIBZ
ensure-library #<LIBRARY LIBZ>
ensure-library #<LIBRARY LIBZ>
While evaluating the form starting at line 27, column 0
  of #P"/Users/matteolandi/my-env/opt/ap/build/build.lisp":

debugger invoked on a SIMPLE-ERROR in thread
#<THREAD "main thread" RUNNING {10016800A3}>:
  #<LIBRARY LIBZ> does not have a known shared library file path.
restarts (invokable by number or by possibly-abbreviated name):
  0: [PROVIDE-PATH                 ] Provide the path to the library manually.
  1: [CONTINUE                     ] Ignore and continue deploying.
  2: [REPORT-ERROR                 ] Print the error and continue running hooks.
  3: [RETRY                        ] Retry #<DEPLOY-OP > on #<SYSTEM "ap">.
  4: [ACCEPT                       ] Continue, treating
                                     #<DEPLOY-OP > on #<SYSTEM "ap"> as having
                                     been successful.
  5:                                 Retry ASDF operation.
  6: [CLEAR-CONFIGURATION-AND-RETRY] Retry ASDF operation after resetting the
                                     configuration.
  7:                                 Retry ASDF operation.
  8:                                 Retry ASDF operation after resetting the
                                     configuration.
  9: [RETRY                        ] Retry EVAL of current toplevel form.
 10:                                 Ignore error and continue loading file "/Users/matteolandi/my-env/opt/ap/build/build.lisp".
 11: [ABORT                        ] Abort loading file "/Users/matteolandi/my-env/opt/ap/build/build.lisp".
 12:                                 Ignore runtime option --load "build/build.lisp".
 13:                                 Skip rest of --eval and --load options.
 14:                                 Skip to toplevel READ/EVAL/PRINT loop.
 15: [EXIT                         ] Exit SBCL (calling #'EXIT, killing the process).

((FLET DEPLOY::FOREIGN-LIBRARIES :IN "/Users/matteolandi/quicklisp/dists/quicklisp/software/deploy-20220220-git/deploy.lisp") :DIRECTORY #P"/Users/matteolandi/my-env/opt/ap/bin/")
   source: (ERROR "~a does not have a known shared library file path." LIB)
0]

While here are some details about my environment:

$ make lisp-info
sbcl --noinform --quit \
                --load "build/info.lisp"
SBCL:2.2.5 on X86-64
...

 fixnum bits:62
To load "trivial-features":
  Load 1 ASDF system:
    trivial-features
; Loading "trivial-features"

features = (:RUNNING-LOCALLY :QUICKLISP :ASDF3.3 :ASDF3.2 :ASDF3.1 :ASDF3
            :ASDF2 :ASDF :OS-MACOSX :OS-UNIX :NON-BASE-CHARS-EXIST-P
            :ASDF-UNICODE :X86-64 :GENCGC :64-BIT :ANSI-CL :BSD :COMMON-LISP
            :DARWIN :IEEE-FLOATING-POINT :LITTLE-ENDIAN :MACH-O
            :PACKAGE-LOCAL-NICKNAMES :SB-CORE-COMPRESSION :SB-LDB
            :SB-PACKAGE-LOCKS :SB-THREAD :SB-UNICODE :SBCL :UNIX)

To fix the problem on my env, all I had to do was link libz.dylib into the current project directly, or into any other of the well-known places, like:/usr/lib/, or /usr/local/lib/.

ln -s $(brew --prefix zlib)/lib/libz.dylib ./

Done.

Well...not satisfied with the solution I started digging a bit more and noticed that, even though *system-source-directories* seems to contain the correct entry for MacOS (i.e. "#p"/usr/local/Cellar/**/lib/"), DEPLOY (and UIOP in particular) fails to locate #P"/usr/local/Cellar/zlib/1.2.12/lib/libz.1.2.12.dylib" using #P"/usr/local/Cellar/**/lib/libz.dylib".

Here is what DEPLOY does, when trying to locate lib files:

(defun pathname-filename (path)
  (format NIL "~a~@[.~a~]"
          (pathname-name path) (pathname-type path)))

(defvar *source* #p"/usr/local/Cellar/**/lib/")
(defvar *path* #p"libz.dylib")
(defvar *filename* (pathname-filename *path*))
(defvar *file* (merge-pathnames *filename* *source*))

> (uiop:file-exists-p *file*)
NIL

I know the library is there, because if I pass *file* to DIRECTORY it would return the dylib file:

> (directory *file*)
(#P"/usr/local/Cellar/zlib/1.2.12/lib/libz.1.2.12.dylib")

So, is there something wrong with using wilds with pathnames? If so, why bother adding entries for these inside *system-source-directories*, if, as it turns out, it won't work as expected?

Thanks,
M.

Foreign library dependencies

Hey.
Thanks for the wonderful library, I absolutely enjoy using it :)
However, I have a small problem with my particular setup.
I'm writing a library that uses bodge-nuklear, a wrapper around C library called nuklear. The problem is that for some reason it depends on two .so libraries, libnuklear.so.bodged and libglad.so.bodged (no idea why .bodged prefix, but that's not important), and the former depends on latter:

$ ldd src/bin/libnuklear.so.bodged
        linux-vdso.so.1 (0x00007fffe47a1000)
        libglad.so.bodged => not found
        libc.so.6 => /lib64/libc.so.6 (0x00007fd657396000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fd65781b000)

Now, deploy perfectly detects both libraries and copies them in bin/ directory, but when I launch the resulting executable, it crashes because it can't find the libglad.so.bodged library (because obviously LD_LIBRARY_PATH on unices does not include current directory unlike on windows):

 ==> Reloading foreign libraries.
   -> Loading foreign library #<LIBRARY LIBNUKLEAR.SO.BODGED-631>.
 ==> Encountered unhandled error: Unable to load foreign library (LIBNUKLEAR.SO.BODGED-631).
  Error opening shared object "/path/redacted/bin/libnuklear.so.bodged":
  libglad.so.bodged: cannot open shared object file: No such file or directory.

Of course I can workaround that by writing simple shell script that sets up LD_LIBRARY_PATH properly, but can it be done purely by lisp code means? I'm thinking maybe some sort of :depends keyword in deploy:define-library or something.