The Apps Validator is a tool designed to scan open web apps for problems and invalid code. By using a combination of various techniques and detection mechanisms, the validator is capable of being both efficient as well as thorough.
Python Libraries:
- argparse
- cssutils
- fastchardet
Python Libraries for Testing:
- nose
- coverage
You can install everything you need for running and testing with
pip install -r requirements.txt
The validator may require some submodules to work. Make sure to run
git clone --recursive git://github.com/mozilla/app-validator.git
so that you get all of the goodies inside.
A working copy of Spidermonkey (debug or non-debug is fine) is required. The version installed must include support for the Parser API. At the time of this writing the version of Spidermonkey included with major package managers does not yet include the Parser API.
How do you know if your js binary has the Parser API? Run this:
js -e 'Reflect;'
There should be no error output.
The best way to make sure you install the right Spidermonkey is to clone the mozilla-central repo or download the tip (which is faster). Then build it from source like this:
cd mozilla-central cd js/src autoconf2.13 ./configure make sudo cp dist/bin/js /usr/local/bin/js
You must use autoconf at exactly 2.13 or else it won't work. If you're using brew on Mac OS X you can get autoconf2.13 with this:
brew install https://gist.github.com/raw/765545/c87a75f2cf9e26c153970522e227f1c1cf63fb81/autoconf213.rb
If you don't want to put the js
executable in your $PATH
or you want it
in a custom path, you can define it as $SPIDERMONKEY_INSTALLATION
in
your environment.
Run the validator as follows
python app-validator <path to app> [-o <output type>] [-v] [--boring] [--selfhosted]
The path to the XPI should point to an XPI file.
-o | The type of output to generate. Types are listed below. |
-v | Enable verbose mode. Extra information will be displayed in
verbose mode, namely notices (informational messages),
Jetpack information if available, extra error info (like
contexts, file data, etc.), and error descriptions. This
only applies to -o text . |
--unlisted | Disables messages that are specific to apps hosted on Marketplace. |
--boring | Disables colorful shell output. |
The output type may be either of the following:
- text (default)
- Outputs a textual summary of the addo-on analysis. Supports verbose mode.
- json
- Outputs a JSON snippet representing a full summary of the analysis.
In text
output mode, output is structured in the format of one
message per line. The messages are prefixed by their priority level
(i.e.: "Warning: This is the message").
At the head of the text output is a block describing what the app type was determined to be.
In JSON
output mode, output is formatted as a JSON snippet
containing all messages. The format for the JSON output is that of the
sample document below.
{ "detected_type": "packaged_app", "errors": 2, "warnings": 1, "notices": 1, "success": false, "ending_tier": 4, "messages": [ { "uid": "123456789", "id": ["module", "function", "error"], "type": "error", "message": "This is the error message text.", "description": ["Description of the error message.", "Additional description text"], "file": "chrome/foo.bar", "line": 12, "column": 50, "context: [ " if(foo = bar())", " an_error_is_somewhere_on_this_line.prototy.eval("whatever");", null ], "tier": 2 } ] }
When a subpackage exists, an angle bracket will delimit the subpackage name and the message text.
If no applicable file is available (i.e.: when a file is missing), the
file
value will be empty. If a file
value is available within a
subpackage, then the file
attribute will be a list containing the
name of the outermost subpackage's name, followed by each successive
concentric subpackage's name, followed by the name of the file that the
message was generated in. If no applicable file is available within a
subpackage, the file
attribute is identical, except the last element
of the list in the file
attribute is an empty string.
For instance, this tree would generate the following messages:
package_to_test.xpi | |-install.rdf |-chrome.manifest |-subpackage.xpi | | | |-subsubpackage.xpi | | | |-chrome.manifest | |-install.rdf | |-subpackage.jar | |-install.rdf
{ "type": "notice", "message": "<em:type> not found in install.rdf", "description": " ... ", "file": "install.rdf", "line": 0 }, { "type": "error", "message": "Invalid chrome.manifest subject: override", "description": " ... ", "file": "chrome.manifest", "line": 7 }, { "type": "error", "message": "subpackage.xpi > install.rdf missing from theme", "description": " ... ", "file": ["subpackage.xpi", ""], "line": 0 }, { "type": "error", "message": "subpackage.xpi > subsubpackage.xpi > Invalid chrome.manifest subject: sytle", "description": " ... ", "file": ["subpackage.xpi", "subsubpackage.xpi", "chrome.manifest"], "line": 5 }
Line numbers are 1-based. Column numbers are 0-based. This can be confusing from a programmatic standpoint, but makes literal sense. "Line one" would obviously refer to the first line of a file.
The context attribute of messages will either be a list or null. Null contexts represent the validator's inability to determine surrounding code. As a list, there will always be three elements. Each element represents a line surrounding the message's location.
The middle element of the context list represents the line of interest. If an element of the context list is null, that line does not exist. For instance, if an error is on the first line of a file, the context might look like:
[ null, "This is the line with the error", "This is the second line of the file" ]
The same rule applies for the end of a file and for files with only one line.
Unit tests can be run with
fab test
or, after setting the proper python path:
nosetests
Some regular maintenance needs to be performed on the validator in order to make sure that the results are accurate.
A list of JS library hashes is kept to allow for whitelisting. This must be regenerated with each new library version. To update:
cd extras mkdir jslibs python jslibfetcher.py python build_whitelist.py jslibs/ # We keep a special hash for testing echo "e96461c6c19608f528b4a3c33a032b697b999b62" >> whitelist_hashes.txt mv whitelist_hashes.txt ../validator/testcases/hashes.txt
To add new libraries to the mix, edit extras/jslibfetcher.py
and add the
version number to the appropriate tuple.