Giter Site home page Giter Site logo

jq-web's Introduction

npm badge Mentioned in Awesome jq

jq-web

This is a build of jq, the command-line JSON processor in Javascript using Emscripten along with a wrapper for making it usable as a library.

It runs in the browser.

install and use

npm install jq-web
var jq = require('jq-web')

jq.json({
  a: {
    big: {
      json: [
        'full',
        'of',
        'important',
        'things'
      ]
    }
  }
}, '.a.big.json | ["empty", .[1], "useless", .[3]] | join(" ")')

The code above returns the string "empty of useless things".

You could do the same using the promised API with jq.promised.json({...}).then(result => {}). That is useful if you're loading a .mem or .wasm file, as the library won't return the correct results until these files are asynchronously fetched by the Emscripten runtime.

What is each file

The releases page has a bunch of different files you can download, here's what they all mean:

file description pros cons
jq.asm.js asm.js version runs in most places requires loading jq.asm.js.mem
jq.asm.min.js minified version of the above idem idem
jq.asm.js.mem memory initialization file for asm.js, needed by jq.asm.js and jq.asm.min.js
jq.asm.bundle.js asm.js version with memory initialization embedded doesn't require loading anything big and slow
jq.asm.bundle.min.js minified version of the above idem the minification has no effect in the memory initialization stuff
jq.wasm.js WebAssembly wrapper smaller, much much faster requires loading jq.wasm.wasm
jq.wasm.min.js minified WebAssembly wrapper since this is just a wrapper around jq.wasm.wasm, the minification makes almost no difference here
jq.wasm.wasm actual WebAssembly binary, loaded by jq.wasm.js and jq.wasm.min.js

When in doubt, just use jq.wasm.js, it is the best!

Webpack issues

fs

The Emscripten runtime will try to require the fs module, and if it fails it will resort to an in-memory filesystem (almost no use of that is made of the library, but it is needed somehow). In Browserify there's a default {} that corresponds to the fs module, but in Webpack you must declare it as an empty module.

404 error when loading .wasm files

By default projects compiled with Emscripten look for .wasm files in the same directory that the .js file is run from. This causes issues when using webpack because name of the .wasm file is altered with a hash and can be placed in a different directory. To fix this problem you can use the copy-webpack-plugin to copy the jq.wasm file to the same directory that the webpack bundle is placed.

Reference

jq.json(<object>, <filter>) <object> will take a Javascript object, or scalar, whatever, and dump it to JSON, then it will return whatever your filter outputs and try to convert that into a JS object. If you're loading .mem or .wasm files asynchronously this will return {} every time you call it until the loading is finished.

jq.raw(<json-string>, <filter>) <raw-output> will take a string that will be passed as it is to jq (like if you were doing echo '<json-string>' | jq <filter> on the command line) then return a string with the raw STDOUT response. If you're loading .mem or .wasm files asynchronously this will return '{}' every time you call it until the loading is finished.

jq.onInitialized.addListener(<function>) registers a function to be called when .mem or .wasm files have finished loading and the library is ready to be used. You should register callbacks here to rerun your functions if you're using the sync API (above). If you're using the promised API (below) you don't ever need to look at this. Also, if you're using the sync API but just at a long time after the page is loaded and the user inputs something, for example, you may not need to use this at all.

jq.promised.json(<object>, <filter>) Promise<object> will do the same as jq.json() but returning a Promise to the result instead. This is safe to use anytime.

jq.promised.raw(<json-string>, <filter>) Promise<raw-output> will do the same as jq.raw() but returning a Promise to the result instead. This is safe to use anytime.

Build

  1. Install Emscripten from source, we used 1.38.12
  2. Clone jq-web and cd into it
  3. Look over the Makefile for more Emscripten instructions
  4. make
    • This may take a while the first time if you have never ran Emscripten before

Test

A handful of tests exist in test.js and are good place to start when verifying a build

  1. npm install or yarn
  2. node test.js
  3. ./node_modules/live-server/live-server.js --open="index.html"

jq-web's People

Contributors

a-omsk avatar ctsrc avatar fiatjaf avatar maxgfeller avatar michael-gillett avatar yurenchen000 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

jq-web's Issues

Cannot access full jq error details

Hello,

I recently took a look at this library and it appears that the error details are stripped a bit before being passed to the Promise.reject?

The error I get is a string of Error: 1 compile error when the error I want is the verbose error that jq would normally provide:

jq: error: syntax error, unexpected '$', expecting IDENT or __loc__ (Unix shell quoting issues?) at <top-level>, line 1:
$$ 
jq: 1 compile error

I can see that the full error is actually returned by the WASM:
Screen Shot 2019-11-02 at 5 54 44 PM

But it appears its stripped in the source maybe here?:

jq-web/post.js

Line 30 in a9b0a2d

if (errString.indexOf(':') > -1) {

With the error being stripped down it makes it pretty painful to debug issues with the JQ syntax. In my specific case users are providing them so I'd like to give somewhat helpful feedback about what they can change instead of a vague compile error string.

I think fixing this should be easy? Just don't split on the :? Let me know if I'm reading this correctly.

jq version?

Sorry if it's already in the repo/release info somewhere, but could you tell me which version of jq the current release uses?
Thanks.

doesn't run on Node.js

Due to a totally unexpected issue with process.platform being reported as nonexistent in the instantiation of emscripten's NODEFS, an error is thrown when jq-web is run on Node.js.

I imagine that an easy solution (already tried by me with manual compiled code editing) would be to not instantiate NODEFS in any way, but perhaps fixing the process.platform would be better.

I'm asking for help on this topic at https://groups.google.com/forum/#!topic/emscripten-discuss/Ikl3qxPoFAk

If anyone seeing this has any solution or clue to a solution it will be highly appreciated.

Failure to include jq files

Hello, I want to use jq-web with my own .jq files where I have defined functions. Is this possible? I have tried using:

include "migrations" {search: "$ORIGIN/../../src/migrate/internal"}; [usage of functions inside .jq]

But it fails to find the files I need.

If invalid JSON is passed to jq.promised.raw all future runs an empty string

When using jq.promised.raw, if you pass an invalid <json-string> as input then all future calls to jq.promised.raw will return an empty string "".

Example:

// Pass invalid JSON in
await jq.promised.raw('{#@#}', '.');
// Result of the above is an empty string still.

// Pass in valid JSON
await jq.promised.raw('{"test":"test"}', '.');
// Result of the above is an empty string still.

This bug tripped me up but I figured out what causes it, let me know if you need more reproduction information.

Project dead?

Hey @fiatjaf can we assume this project is dead?

...or is it just stable because jq releases are that old?

Error message gets cut off when parsing a malformed date

Describe the bug
Error message gets cut off when applying fromdate to an empty string

To Reproduce

  1. Set a filter "" | fromdate.
  2. Get the error with the message: %SZ".

The sandbox: https://codesandbox.io/s/exciting-curie-k6yccy?file=/src/index.js

Expected behavior
The error object should have a message describing the error, for example: Date "" does not match format "%Y-%m-%dT%H:%M:%SZ"

Environment
Google Chrome Version 100.0.4896.127 (Official Build) (x86_64)
jq-web v0.5.1

jq flags `-r` effect seems can't cancel between calls ?

version: release-0.5.1 jq.wasm.js

run in chrome F12.

// ------ worked well
> jq.raw('{"a":"hello"}', '.a')
< ""hello""

> jq.raw('{"a":"hello"}', '.a', ['-c'])
< ""hello""

// ------ once call with `-r`, 
> jq.raw('{"a":"hello"}', '.a', ['-r'])
< "hello"

// ------ somethings err? following call can't cancel `-r` flag
> jq.raw('{"a":"hello"}', '.a')
< "hello"

> jq.raw('{"a":"hello"}', '.a', ['-c'])
< "hello"

then refresh page,
and worked well again.

Intermittent `<generic error, no stack>` exception using jq.promised.json

This dependency was not found: fs

Hello-

I'm trying to use this in my Vue.js app, but when I import like:

import jq from 'jq-web';

I get this error:

ERROR  Failed to compile with 1 errors

This dependency was not found:

* fs in ./node_modules/jq-web/jq.min.js

To install it, you can run: npm install --save fs

I think this is because fs is not available in a browser window. I already have fs in my package.json file and saved to node_modules.

Should I be doing something else instead? The same thing happens if I use require.

Unclosed streams caused FS error after calling jq.json 2048 times.

I've got FS error after 2048 times calling jq.json.
I debugged and found unclosed streams in FS.streams caused by https://github.com/fiatjaf/jq-web/blob/master/post.js#L20.
Below code can show the error.

jq.onInitialized.addListener(()=>{
    for (var i =0;i<4098;i++) {
        console.log(jq.json({ok:i},'.'));
    }
    console.log('ok')
})

You'll get
...
{ ok: 2046 }
Error: FS error
Error: FS error
..../jq.asm.bundle1.js:122
throw ex
^

Error: abort(Error: FS error). Build with -s ASSERTIONS=1 for more info.

Include TypeScript types

When importing jq-web in a TypeScript project I get this error, which I then have to ignore:

Could not find a declaration file for module 'jq-web'. ./node_modules/jq-web/jq.asm.bundle.min.js' implicitly has an 'any' type.
  Try `npm i --save-dev @types/jq-web` if it exists or add a new declaration (.d.ts) file containing `declare module 'jq-web';`ts(7016)

Is it possible to add some basic types to this package?

Upgrade path for jq 1.7

jq 1.7 has now been released.

Is there an upgrade path to getting jq-web using this new jq version?

I can't quite tell, but it looks like jq-web is using jq 1.5?

Sync function

Returning {} when .mem/.wasm are not loaded yet is good, but it'd be even better to have a sync call. (One that would block until the loading finished and maybe fail after a timeout)

jq-web module suddenly stops working

Hello!

I'm using the jq-web module to validate some jq expressions on my NodeJS application. Application was working as expected for a few days on K8s cluster and I validated ~1000 expressions. Then, out of nowhere, this happened.

There is an error in expression: ".IG_PKT_DROP__NO_FLOW > 0". Message: Error\n at Object.ensureErrnoError (/usr/src/app/node_modules/jq-web/jq.asm.bundle.min.js:1:1722236)\n at Object.staticInit (/usr/src/app/node_modules/jq-web/jq.asm.bundle.min.js:1:1722445)\n at /usr/src/app/node_modules/jq-web/jq.asm.bundle.min.js:1:1734039\n at Object. (/usr/src/app/node_modules/jq-web/jq.asm.bundle.min.js:1:1751789)\n at Module._compile (internal/modules/cjs/loader.js:1147:30)\n at Object.Module._extensions..js (internal/modules/cjs/loader.js:1167:10)\n at Module.load (internal/modules/cjs/loader.js:996:32)\n at Function.Module._load (internal/modules/cjs/loader.js:896:14)\n at Module.require (internal/modules/cjs/loader.js:1036:19)\n at require (internal/modules/cjs/helpers.js:72:18)\n at Object. (/usr/src/app/jq/jq-validator.js:3:12)\n at Module._compile (internal/modules/cjs/loader.js:1147:30)\n at`` Object.Module._extensions..js (internal/modules/cjs/loader.js:1167:10)\n at Module.load (internal/modules/cjs/loader.js:996:32)\n at Function.Module._load (internal/modules/cjs/loader.js:896:14)\n at Module.require (internal/modules/cjs/loader.js:1036:19)\n at require (internal/modules/cjs/helpers.js:72:18)\n at Object. (/usr/src/app/service/AggregatingAlertRuleService.js:10:21)\n at Module._compile (internal/modules/cjs/loader.js:1147:30)\n at Object.Module._extensions..js (internal/modules/cjs/loader.js:1167:10)\n at Module.load (internal/modules/cjs/loader.js:996:32)\n at Function.Module._load (internal/modules/cjs/loader.js:896:14)

const validate = (expression, example, skipSemantic=false) => {
    let result;
    try {
        result = jq.json(example, expression);

        if (result === null) {
            return new JqInvalidResult(expression, "Expression returned no result.")
        }
    } catch (e) {
        return new JqSyntaxErrorResult(expression, e);
    }

   //other logic
}

Can you help me? I tried to go through the source code using the lines:offsets but it is all minified js code.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.