prettydiff/biddle

Self-hosted application distribution

biddle

A package management application without a package management service.

License

MIT

Version

1.1.7

About

This application is a cross-OS solution to creating zip files for distribution and fetching files via HTTP(S). The project's goal is to provide a universal application distribution utility that is language agnostic, operating system independent, and platform independent. The only additional requirement for distributing application packages is online storage on a web server. This application provides all the utilities to retrieve, bundle, and unpackage applications.

Watch the video demo.

biddle is inspired by the incredible awesomeness of NPM, but seeks to accomplish a few additional goals:

• integrity - Downloaded packages will perform a hash comparison before they are unpackaged. If the hashes don't match the zip file will be saved in the downloads directory awaiting a human touch.
• autonomy - Be freed from censorship. There is no central authority here. Host your own publications and manage them as you please from any location and with any name or content you choose.
• privacy - Put biddle behind your firewall or an isolated environment to prevent public access to your applications and data.
• management - There is no dependency hell here. Dependency management will not be automated, but a means to manage and review the status of all installed/published packages is provided.
• freedom - biddle will work everywhere Node.js runs. It can be used with any application written in any language whether binary or text.

Documentation

• Getting Started
• package.json
• Documentation

Supported commands

Commands are the third command line argument, or second if the node argument is absent. Commands are case insensitive, but values and local paths are case sensitive. All local address are either absolute from the root or relative from the current working directory.

commands

Conveniently lists the supported commands with a brief description.

node biddle commands

copy

Copy files or directories to a different directory.

node biddle copy myFile myOtherDirectory
node biddle copy myDirectory myOtherDirectory

get

Merely downloads the requested resource and saves it as a file with the same filename. If the filename is not provided in the URI the final directory up to the domain name will become the filename, and if for some reason that doesn't work the default filename is download.xxx.

Download a file to the default location, which is the provided downloads directory.

node biddle get http://google.com

Download a file to an alternate location.

node biddle get http://google.com ../mydirectory

global

The global command adds an applications's path to the OS path variable so that the application can be run from any location as a command, for example: biddle help from root instead of node biddle help from the local biddle directory. Use the remove option to remove biddle from the path. This command requires use of an administrative console in Windows. For additional requirements see section Global Application Requirements in the documentation.

Allowing global availability to biddle.

node biddle global

Allowing global availability to any other application installed by biddle.

biddle global myApplication

Removing global availability to biddle.

biddle global remove

Removing global availability to any other application installed by biddle.

biddle global myApplication remove

hash

Prints to console a SHA512 hash against a local file.

node biddle hash downloads/myfile.zip

Hash a string by indicating string and a value instead of a file path.

node biddle hash string "my string value"

help

Prints the readme.md file contents to console in a human friendly way.

No command will still generate the readme data.

node biddle

The default word wrapping is set to 100 characters.

node biddle help

Set a custom word wrap limit.

node biddle help 80

install

Downloads the requested zip file, but performs a hash comparison before unzipping the file.

node biddle install http://example.com/downloads/application_latest.zip

list

Will list all installed and/or published applications with their locations and latest versions. It can take the optional argument installed or published to output a specific list or both lists are produced.

Only output the installed list.

node biddle list installed

Only output the published list.

node biddle list published

Output both lists

node biddle list

markdown

Allows the internal markdown parser used by the help command to be supplied to a directed file to ease reading of documentation directly from the command line.

The first argument after the command is the address of the file to read.

node biddle markdown applications/example/readme.md

You can also specify a custom word wrap limit. The default is 100.

node biddle markdown applications/example/readme.md 80

publish

Writes a hash file and a zip file with a version number to the publications directory or some other specified location. Applications are required to have a file in their root directory named package.json with properties: name and version.

Create a zip in the default location: ./publications/myApplicationDirectory

node biddle publish ../myApplicationDirectory

Publish to a custom location: ./myAlternateDirectory/myApplicationDirectory

node biddle publish ../myApplicationDirectory myAlternateDirectory

Use quotes if any argument contains spaces:

node biddle publish "c:\program files\myApplicationDirectory"

remove

Removes a file or a directory tree

node biddle remove myDirectory
node biddle remove myFile

status

Will check whether an installed application is behind the latest published version.

Check the status of all installed applications

node biddle status

Check the status of an application by name

noe biddle status myApplicationName

test

Run the user acceptance tests for a given application. This command requires a test property in a package.json file.

Run the tests against biddle

node biddle test

Run the tests against an installed application

node biddle test applicationName

Run the tests against a directory in the current location

node biddle test ./directory

Run the tests against a directory in a different location

node biddle test ../../directory

uninstall

Will delete an installed application by name and remove the application from the installed list.

node biddle uninstall myApplicationName

unpublish

Will delete a published application by name and version then remove the respective files.

Unpublish an application by version.

node biddle unpublish myApplicationName 1.0.0

Unpublish an application's latest version

node biddle unpublish myApplicationName latest

Remove the published application entirely.

node biddle unpublish myApplicationName all

unzip

Unzips a local zipped file.

Unzip to the default location, the supplied downloads directory.

node biddle unzip myZipFile.zip

Unzip to a specified location.

node biddle unzip myZipFile.zip myDirectory

version

Prints the version number/name of biddle or a named installed application.

If no application is specified the current version of biddle is output.

node biddle version

Output the current version of "myApplication".

node biddle version myApplication

zip

Zips local files or local directories into a zip file.

Zip to the default location, the supplied downloads directory.

node biddle zip ../myApplication

Zip to a specified location.

node biddle zip ../myApplication myCustom/Location/Directory

Dependencies

• This application is written in JavaScript and requires Node.js to run.
• This application makes use of zip utilities provided by operating systems.
• This application requires use of Windows's PowerShell to execute the global coammand in Windows.
• The test command requires Pretty Diff and JSLint as git submodules from Github.

biddle's Issues

Make update command a soft update opposed to a replacement

Currently the install command removes the old application and installs the newer version in that location. This means all saved data and configurations will be lost. This is acceptable for the install command.

The word update suggests something less disruptive. The near term (80%) solution is to unzip the downloaded zip to a temporary location and copy the new files overwriting the old files without removing things.

This is not a 100% solution because applications may supply default configurations which would replace a user's preferences saved into the same named config file. A convention would need to exist to balance the needs of an application's user configs against the needs of an application publisher to disruptively change their configuration API. Ultimately, biddle will need to supply a means to identify such breaking changes that should cause warning and user validation during application updates.

Abstract ANSI formatting

Create universal references for ANSI formatting and use the references instead of the literals in the code.

Extend the test command support installed applications

Current there is a test command that just supports biddle's test automation.

biddle test

This should be extended to installed applications so that biddle can run validation against installed applications:

biddle test myApplication

This issue would require a new property to the application's package.json file so that biddle knows what the test command(s) is/are. These would have to be executed in Node as a child.spawn so that stdout can stream to the current terminal in real time.

If the schema for package.json, and thus installed.json, these changes would need to be reflected in the documentation.

Version 2: IPFS integration

IPFS is - https://github.com/ipfs

IPFS is like bittorrent for filesystem fragments with git and hashing builtin. This means the publication task can safely eliminate the need to write and store zip files.

Major changes to biddle

• publish - Publication will work the same up till the point of writing a zip file. Instead a ipfs add -r child command will be issued against the temporary directory created for the variant. The filedata.json file will store the IPFS hash value instead of the local path to a zip file.
• install - Install will issue a ipfs get child command. Since IPFS has version control built in applications can be installed and updated without file clobbering, which solves a major biddle limitation.
• IPFS will become a required dependency just like Node. Biddle will not limit itself to a single form of IPFS so long as the generic commands are available from the system path.
• This major change should not require any changes to biddle's installed.json and published.json files.

Centralized management

• The publication task will still write files to a publication point, namely the latest.txt, index.xhtml, and filedata.json. This will provide a uniform location by which application metadata is stored without centralizing the actual application.

command "global" fails for posix if shell isn't Bash

My posix scripting skills are weak. I will take any help I can get.

Add command "version"

Add a command named version to simply output the value of version from biddle's package.json file.

biddle version

should return:

0.1.15

Use Node.js Crypto library for hashes

Using the Node.js crypto library would allow a single universal approach to hashing that is OS independent.

How to run prettydiff after installation via biddle?

Use biddle for biddle's dependencies

Currently biddle is using git submodules for its two dependencies: jslint and prettydiff. The git submodules should be replaced with biddle.

At this time the owner/maintainer of JSLint is not willing to endorse any project that isn't 100% ES6 conformant. biddle will probably never be 100% strictly ES6 because it needs named functions for maintenance of callbacks opposed to arrow functions where are always anonymous and it uses some old style for loops instead of the array forEach method, because some loops need to be broken as appropriate.

The shortest path to resolution is just to old host a biddle publication of JSLint on my own and periodically update it.

Catalogs and distribution by name

Currently installing biddle looks like

biddle install http://prettydiff.com/downloads/prettydiff/prettydiff_latest.zip

which is far more frustrating than the NPM and Homebrew way

npm install prettydiff

The frustration grows as the length of applications grows. The middle ground is to allow users to retain local application catalogs. Quite simply it can be something as simple as a JSON of name value pairs where the name is the application name and the value is its publication point.

This functionality is already present once an application is installed. The value add from this feature is to ease the burden of initial installation. Situations where this would be beneficial:

• Trading catalogs between people online
• Starting fresh on a new machine
• Providing software list recommendations

copy is sporadically failing on both posix and windows

On posix copy test fails about 1 time out of 10. It seems the biddle code is correct as this is a shallow wrapper around a child process. I suspect, though not validated, the problem is something between an issued a posix command (as a child process) and a verification that follows to quickly after.

On windows this problem occurs more frequently and is typically experienced from the publish test when running biddle test.

Copy command for posix:

cp -R targetPath destinationPath

Copy command for Windows:

xcopy "targetPath" "desintationPath" /E /Q /G /H /Y /J /I

unpublish by version number

biddle unpublish prettydiff 2.1.15

Allow unpublish by version number so that only a specific version of an application is destroyed instead of the entire application.

Move exclusions from apps.publish to apps.copy

This enhancement will drastically increase performance and accuracy in the publish command.

Publish prettydiff with biddle

This issue has two parts:

1. publish prettydiff using biddle and provide a download location from the website.
2. discover monitoring solutions to determine project health, traffic, and other considerations

Add appveyor CI support

Run remote automation builds on Windows servers.

Simultaneous parallel operations

The first step is to pass data into functions opposed to referencing from globally stored properties. The most obvious offender is data,input. As much of this stuff as possible must be confined to the init function at the bottom.

Download path is relative to CWD instead of project

With this defect if the project is located at /user/myProject and the project download location is /user/myProject/downloads. Biddle will correctly read the download location as a relative path, but incorrectly associate it relative to the current working directory. If your current directory is /applications/tools/awesometool and you attempted to run biddle publish ../../../user/myProject the files will write to /applications/tools/awesometool/downloads instead of /user/myProject/downloads.

Add additional test units

I just fixed a critical flaw that should never have been a flaw in the first place. Additional test units are required:

• publish from a relative path outside the current directory tree
• install via http
• get a zip file, unzip the file, and verify contents

publish command relative path is relative to biddle instead of CWD

Add support for code blocks delimited with ```

```(new line)
code
(new line)```

Publication - add hash and install instructions to index.xhtml file

Publishing updates with biddle broken

Problems:

• #20
• index.html not generated
• publications directory included in zip files
• there are no unit tests for publishing updates

Observations:

• publish.json is updated correctly

Extend global command to support installed applications

Right now global command only works with biddle itself

node biddle global

It should be extended to allow the name of an installed application

biddle global myApplication

Removing an application from the global path must be in the uninstall command and also allowed with the global command and a remove argument like current support for biddle

biddle global myApplication remove
biddle uninstall myApplication

This use of global command with an application should perform a check of the application name against the installed.json to determine if the application name is valid or else throw an error.

2 new commands: location and origin

• location - outputs the absolute path of an installed application
  • input: biddle location prettydiff
  • output: /Users/austincheney/biddle/applications/prettydiff/
• origin - outputs the absolute publication point of an installed application
  • input: biddle origin prettydiff
  • output: http://prettydiff.com/downloads/prettydiff/prettydiff_latest.zip

Audit markdown parser against spec

Github released a specification for their markdown - http://spec.commonmark.org/0.27/

I need to verify conformance.

Specific help responses for each biddle command

apps.errout function can wipe installed.json and published.json

When the apps.errout function executes from a test run it can cause the installed.json and package.json files to be emptied. In order for this series of events to occur there must be a broken test generated from the biddle test command.

unpublish by version number not changing version number in published.json

This needs an audit and more in-depth test automation:

• verify version is removed from filedata.json
• verify value of latest is updated
  • verify latest.txt is updated with new value
  • verify published.json is updated with new latest value
• specify latest version on index.xhtml

Tables and block quotes missing from markdown support

Add support for tables and blockquotes to markdown command

add a flag to force a publication as the latest version

command copy may miss some files

Check that a targeted publication directory is either empty or absent

Don't allow publication into directories that already contain things as this makes unpublication a problem that could destroy other files.

command remove may fail to remove empty directories

Intermitent publish corruption only on Linux

The problem is intermittent and I have only experienced it on my remote Linux webserver. I cannot produce this problem on Windows or OSX.

The problem is present when attempting to publish Pretty Diff:

biddle publish prettydiff

The problem is evident when prettydiff_latest.zip and prettydiff_2.1.15.zip (latest version number) are different sizes. I will need to create a local Linux partition on my Windows computer to investigate this further. I can work around this by unpublishing and republishing until the problem is no longer present, which eventually works but is still not an acceptable solution. This is further not acceptable because the unpublish command will destroy all prior published versions.

A strange side-effect was observed. When attempting to republish from the same current working directory (CWD) as the original publication the zip command was failing and further attempts to republish failed in the same way until I changed my CWD to a different directory.