dengelke / node-zoom2 Goto Github PK

View Code? Open in Web Editor NEW

13.0 13.0 4.0 2.74 MB

z39.50 with Node 12+ support

License: MIT License

Makefile 0.11% Python 3.66% JavaScript 30.20% C++ 59.81% Shell 6.23%

node-zoom2's People

Contributors

Stargazers

Watchers

Forkers

pvoisin djrondon sirraminyavari sylvain-bouchard

node-zoom2's Issues

Support libyaz4-dev

Please add support for libyaz4-dev which the YAZ dev package available in Ubuntu LTS (16.04)

I had to changes this line in binding.gyp to make it work:

node-zoom2/binding.gyp

Line 5 in 1818b89

"cflags": ["<!@(yaz-config --libs --include)"],

➡️

yaz-config --libs --cflags

"yaz-config: not found" error while installing library on Debian

I'm dropping this here before I can propose a PR (hopefully). Originally (few months ago) I made a quick attempt to build a Docker image based on Debian (node:slim) with node-zoom2 installed. Here's the problem I've faced (still facing it today):

#9 7.685 npm ERR! /bin/sh: 1: yaz-config: not found
#9 7.685 npm ERR! gyp: Call to 'yaz-config --libs' returned exit status 127 while in binding-not-win.gyp. while loading dependencies of binding.gyp while trying to load binding.gyp
#9 7.685 npm ERR! gyp ERR! configure error 
#9 7.685 npm ERR! gyp ERR! stack Error: `gyp` failed with exit code: 1
#9 7.685 npm ERR! gyp ERR! stack     at ChildProcess.onCpExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:325:16)
#9 7.685 npm ERR! gyp ERR! stack     at ChildProcess.emit (node:events:514:28)
#9 7.685 npm ERR! gyp ERR! stack     at ChildProcess._handle.onexit (node:internal/child_process:294:12)
#9 7.685 npm ERR! gyp ERR! System Linux 5.15.49-linuxkit
#9 7.685 npm ERR! gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
#9 7.685 npm ERR! gyp ERR! cwd /usr/src/app/node_modules/node-zoom2
#9 7.685 npm ERR! gyp ERR! node -v v20.8.0
#9 7.685 npm ERR! gyp ERR! node-gyp -v v9.4.0
#9 7.685 npm ERR! gyp ERR! not ok

It seems that yaz-config doesn't exist (maybe it isn't shipped anymore with yaz or libyaz-dev). Instead, pkg-config yaz works and with the following trick I was able to build the image:

FROM node:slim

...

RUN \
  npm install node-zoom2 --ignore-scripts \
  && LIBS=`pkg-config yaz --libs` \
  && CFLAGS=`pkg-config yaz --cflags` \
  && sed -i -E 's#"cflags": \[.+\]#"cflags": ["'"$CFLAGS"'"]#' ./node_modules/node-zoom2/binding-not-win.gyp \
  && sed -i -E 's#"libraries": \[.+\]#"libraries": ["'"$LIBS"'"]#' ./node_modules/node-zoom2/binding-not-win.gyp \
  && npm rebuild

...

I admit this isn't very clean but, as I'm no Linux/C++/Make expert, I couldn't find another (quick) solution.

So, @dengelke, if you have any recommendation to make this fit into the project I'd be very pleased to hear about it! :)

fails where yaz-client/php-yaz works

I know it's a free software, so I can't expect help in specific cases, but this might help you to iron out some possible bug in node-zoom2 vs. yaz-client. I'm searchin on tcp:tagetes2.oszk.hu:1616/any with the query @attr 1=31 1856 and it works fine with yaz-client, it gets the 1000 records the server is willing find max, but fails with errors in js. It works if I only request like 500 records. Using version 0.9.1 with nodejs 18. If I can help in any way, let me know. Thanks!

import zoom from 'node-zoom2'

import { EventEmitter } from 'events'
import { pEvent } from 'p-event'

await search()

async function search()
{
    let event = new EventEmitter()

    let z = zoom.Connection('tcp:tagetes2.oszk.hu:1616/any').set('preferredRecordSyntax', 'usmarc')
    
    let cnt = 0

    z.query('prefix', '@attr 1=31 1856').search((err, resultset) => {
    
        if(err)
        {
            console.log(err)
            return
        }
        
        let hits = resultset.size

        console.log(hits, 'hits')
    
        resultset.getRecords(0, hits, (err, records) => {

            if(err)
                console.log(err)
        
            while (records && records.hasNext()) 
            {
                let record = records.next()
    
                if(record.json)
                    process.stdout.write('.')
                else
                    console.log('error', cnt)

                cnt++
            }   
    
            event.emit('done', resultset)
    
        })   
    })
    
    await pEvent(event, 'done')
}

compiler warnings on nodejs 6.11.3 and gcc-c++ 6.4.1

My environment:

$ rpm -q fedora-release gcc-c++ libyaz-devel nodejs
fedora-release-25-2.noarch
gcc-c++-6.4.1-1.fc25.x86_64
libyaz-devel-5.14.11-4.fc25.x86_64
nodejs-6.11.3-1.fc25.x86_64

Order warnings can be fixed easily by swapping counts_ and index_ in records.h and resultset.h, but for the other i am unsure how to fix, it seems the usage of NewInstance() ist deprecated.

make: Entering directory '/home/ulrich/src/node-zoom2/build'
  CXX(target) Release/obj.target/zoom/src/zoom.o
In file included from ../src/zoom.cc:4:0:
../src/records.h: In constructor ‘node_zoom::Records::Records(ZOOM_record_p**, size_t)’:
../src/records.h:25:16: warning: ‘node_zoom::Records::counts_’ will be initialized after [-Wreorder]
         size_t counts_;
                ^~~~~~~
../src/records.h:24:16: warning:   ‘size_t node_zoom::Records::index_’ [-Wreorder]
         size_t index_;
                ^~~~~~
../src/records.h:12:18: warning:   when initialized here [-Wreorder]
         explicit Records(ZOOM_record *records, size_t counts) :
                  ^~~~~~~
In file included from ../src/zoom.cc:6:0:
../src/resultset.h: In constructor ‘node_zoom::GetRecordsWorker::GetRecordsWorker(Nan::Callback*, ZOOM_resultset, size_t, size_t)’:
../src/resultset.h:41:16: warning: ‘node_zoom::GetRecordsWorker::index_’ will be initialized after [-Wreorder]
         size_t index_;
                ^~~~~~
../src/resultset.h:40:16: warning:   ‘size_t node_zoom::GetRecordsWorker::counts_’ [-Wreorder]
         size_t counts_;
                ^~~~~~~
../src/resultset.h:29:9: warning:   when initialized here [-Wreorder]
         GetRecordsWorker(Nan::Callback *callback, ZOOM_resultset resultset,
         ^~~~~~~~~~~~~~~~
  CXX(target) Release/obj.target/zoom/src/query.o
../src/query.cc: In static member function ‘static Nan::NAN_METHOD_RETURN_TYPE node_zoom::Query::New(Nan::NAN_METHOD_ARGS_TYPE)’:
../src/query.cc:52:63: warning: ‘v8::Local<v8::Object> v8::Function::NewInstance(int, v8::Local<v8::Value>*) const’ is deprecated: Use maybe version [-Wdeprecated-declarations]
         info.GetReturnValue().Set(cons->NewInstance(argc, argv));
                                                               ^
In file included from /home/ulrich/.node-gyp/6.11.3/include/node/v8.h:25:0,
                 from ../src/errors.h:2,
                 from ../src/query.cc:1:
/home/ulrich/.node-gyp/6.11.3/include/node/v8.h:3272:31: note: declared here
                 Local<Object> NewInstance(int argc, Local<Value> argv[]) const);
                               ^
/home/ulrich/.node-gyp/6.11.3/include/node/v8config.h:333:3: note: in definition of macro ‘V8_DEPRECATED’
   declarator __attribute__((deprecated(message)))
   ^~~~~~~~~~
  CXX(target) Release/obj.target/zoom/src/record.o
  CXX(target) Release/obj.target/zoom/src/errors.o
../src/errors.cc:1:9: warning: #pragma once in main file
 #pragma once
         ^~~~
  CXX(target) Release/obj.target/zoom/src/records.o
In file included from ../src/records.cc:3:0:
../src/records.h: In constructor ‘node_zoom::Records::Records(ZOOM_record_p**, size_t)’:
../src/records.h:25:16: warning: ‘node_zoom::Records::counts_’ will be initialized after [-Wreorder]
         size_t counts_;
                ^~~~~~~
../src/records.h:24:16: warning:   ‘size_t node_zoom::Records::index_’ [-Wreorder]
         size_t index_;
                ^~~~~~
../src/records.h:12:18: warning:   when initialized here [-Wreorder]
         explicit Records(ZOOM_record *records, size_t counts) :
                  ^~~~~~~
../src/records.cc: In static member function ‘static Nan::NAN_METHOD_RETURN_TYPE node_zoom::Records::Next(Nan::NAN_METHOD_ARGS_TYPE)’:
../src/records.cc:45:80: warning: ‘v8::Local<v8::Object> v8::Function::NewInstance() const’ is deprecated: Use maybe version [-Wdeprecated-declarations]
             Local<Object> wrapper = Nan::New(Record::constructor)->NewInstance();
                                                                                ^
In file included from /home/ulrich/.node-gyp/6.11.3/include/node/v8.h:25:0,
                 from ../src/errors.h:2,
                 from ../src/records.cc:1:
/home/ulrich/.node-gyp/6.11.3/include/node/v8.h:3276:52: note: declared here
   V8_DEPRECATED("Use maybe version", Local<Object> NewInstance() const);
                                                    ^
/home/ulrich/.node-gyp/6.11.3/include/node/v8config.h:333:3: note: in definition of macro ‘V8_DEPRECATED’
   declarator __attribute__((deprecated(message)))
   ^~~~~~~~~~
  CXX(target) Release/obj.target/zoom/src/options.o
../src/options.cc: In static member function ‘static Nan::NAN_METHOD_RETURN_TYPE node_zoom::Options::New(Nan::NAN_METHOD_ARGS_TYPE)’:
../src/options.cc:61:63: warning: ‘v8::Local<v8::Object> v8::Function::NewInstance(int, v8::Local<v8::Value>*) const’ is deprecated: Use maybe version [-Wdeprecated-declarations]
         info.GetReturnValue().Set(cons->NewInstance(argc, argv));
                                                               ^
In file included from /home/ulrich/.node-gyp/6.11.3/include/node/v8.h:25:0,
                 from ../src/errors.h:2,
                 from ../src/options.cc:1:
/home/ulrich/.node-gyp/6.11.3/include/node/v8.h:3272:31: note: declared here
                 Local<Object> NewInstance(int argc, Local<Value> argv[]) const);
                               ^
/home/ulrich/.node-gyp/6.11.3/include/node/v8config.h:333:3: note: in definition of macro ‘V8_DEPRECATED’
   declarator __attribute__((deprecated(message)))
   ^~~~~~~~~~
  CXX(target) Release/obj.target/zoom/src/resultset.o
In file included from ../src/resultset.cc:2:0:
../src/records.h: In constructor ‘node_zoom::Records::Records(ZOOM_record_p**, size_t)’:
../src/records.h:25:16: warning: ‘node_zoom::Records::counts_’ will be initialized after [-Wreorder]
         size_t counts_;
                ^~~~~~~
../src/records.h:24:16: warning:   ‘size_t node_zoom::Records::index_’ [-Wreorder]
         size_t index_;
                ^~~~~~
../src/records.h:12:18: warning:   when initialized here [-Wreorder]
         explicit Records(ZOOM_record *records, size_t counts) :
                  ^~~~~~~
In file included from ../src/resultset.cc:3:0:
../src/resultset.h: In constructor ‘node_zoom::GetRecordsWorker::GetRecordsWorker(Nan::Callback*, ZOOM_resultset, size_t, size_t)’:
../src/resultset.h:41:16: warning: ‘node_zoom::GetRecordsWorker::index_’ will be initialized after [-Wreorder]
         size_t index_;
                ^~~~~~
../src/resultset.h:40:16: warning:   ‘size_t node_zoom::GetRecordsWorker::counts_’ [-Wreorder]
         size_t counts_;
                ^~~~~~~
../src/resultset.h:29:9: warning:   when initialized here [-Wreorder]
         GetRecordsWorker(Nan::Callback *callback, ZOOM_resultset resultset,
         ^~~~~~~~~~~~~~~~
../src/resultset.cc: In member function ‘virtual void node_zoom::GetRecordsWorker::HandleOKCallback()’:
../src/resultset.cc:101:73: warning: ‘v8::Local<v8::Object> v8::Function::NewInstance() const’ is deprecated: Use maybe version [-Wdeprecated-declarations]
     Local<Object> wrapper = Nan::New(Records::constructor)->NewInstance();
                                                                         ^
In file included from /home/ulrich/.node-gyp/6.11.3/include/node/v8.h:25:0,
                 from ../src/errors.h:2,
                 from ../src/resultset.cc:1:
/home/ulrich/.node-gyp/6.11.3/include/node/v8.h:3276:52: note: declared here
   V8_DEPRECATED("Use maybe version", Local<Object> NewInstance() const);
                                                    ^
/home/ulrich/.node-gyp/6.11.3/include/node/v8config.h:333:3: note: in definition of macro ‘V8_DEPRECATED’
   declarator __attribute__((deprecated(message)))
   ^~~~~~~~~~
  CXX(target) Release/obj.target/zoom/src/connection.o
In file included from ../src/connection.cc:5:0:
../src/resultset.h: In constructor ‘node_zoom::GetRecordsWorker::GetRecordsWorker(Nan::Callback*, ZOOM_resultset, size_t, size_t)’:
../src/resultset.h:41:16: warning: ‘node_zoom::GetRecordsWorker::index_’ will be initialized after [-Wreorder]
         size_t index_;
                ^~~~~~
../src/resultset.h:40:16: warning:   ‘size_t node_zoom::GetRecordsWorker::counts_’ [-Wreorder]
         size_t counts_;
                ^~~~~~~
../src/resultset.h:29:9: warning:   when initialized here [-Wreorder]
         GetRecordsWorker(Nan::Callback *callback, ZOOM_resultset resultset,
         ^~~~~~~~~~~~~~~~
../src/connection.cc: In static member function ‘static Nan::NAN_METHOD_RETURN_TYPE node_zoom::Connection::New(Nan::NAN_METHOD_ARGS_TYPE)’:
../src/connection.cc:55:63: warning: ‘v8::Local<v8::Object> v8::Function::NewInstance(int, v8::Local<v8::Value>*) const’ is deprecated: Use maybe version [-Wdeprecated-declarations]
         info.GetReturnValue().Set(cons->NewInstance(argc, argv));
                                                               ^
In file included from /home/ulrich/.node-gyp/6.11.3/include/node/v8.h:25:0,
                 from ../src/errors.h:2,
                 from ../src/connection.cc:3:
/home/ulrich/.node-gyp/6.11.3/include/node/v8.h:3272:31: note: declared here
                 Local<Object> NewInstance(int argc, Local<Value> argv[]) const);
                               ^
/home/ulrich/.node-gyp/6.11.3/include/node/v8config.h:333:3: note: in definition of macro ‘V8_DEPRECATED’
   declarator __attribute__((deprecated(message)))
   ^~~~~~~~~~
../src/connection.cc: In member function ‘virtual void node_zoom::SearchWorker::HandleOKCallback()’:
../src/connection.cc:172:75: warning: ‘v8::Local<v8::Object> v8::Function::NewInstance() const’ is deprecated: Use maybe version [-Wdeprecated-declarations]
     Local<Object> wrapper = Nan::New(ResultSet::constructor)->NewInstance();
                                                                           ^
In file included from /home/ulrich/.node-gyp/6.11.3/include/node/v8.h:25:0,
                 from ../src/errors.h:2,
                 from ../src/connection.cc:3:
/home/ulrich/.node-gyp/6.11.3/include/node/v8.h:3276:52: note: declared here
   V8_DEPRECATED("Use maybe version", Local<Object> NewInstance() const);
                                                    ^
/home/ulrich/.node-gyp/6.11.3/include/node/v8config.h:333:3: note: in definition of macro ‘V8_DEPRECATED’
   declarator __attribute__((deprecated(message)))
   ^~~~~~~~~~
  SOLINK_MODULE(target) Release/obj.target/zoom.node
  COPY Release/zoom.node
make: Leaving directory '/home/ulrich/src/node-zoom2/build'
gyp info ok

install on Ubuntu 20 + Node 16

Trying to install on Ubuntu. Ran the preinstall.sh, yaz-client works ok, but the npm installation of node-zoom2 fails with some compilation issue when calling gyp. Perhaps a newer version of yaz then you expect. I included the logfile - perhaps it is obvious for you what fails. Let me know if I can help with anything. Thanks.
2022-01-08T13_23_58_840Z-debug.log

Windows installation

I installed yaz from
https://www.indexdata.com/resources/software/yaz/
Windows (64 bit) version and when I run it:
npm i node-zoom2
It throws error:
gyp: Call to 'yaz-config --libs' returned exit status 1 while in binding.gyp. while trying to load binding.gyp
any solutions?

Connection.close() is missing

So the node app doesn't exit.

examples with larger resultsets

Could you add examples that show how to "officially" handle larger resultsets, for example by getting records in batche of 100 for example, or where the on('data') event would fire for all records - for now the example on the main page here seems to fire this event for one record only. It's not clear what's the recoomended way to do 100-record batches, as the getRecords() call seems to return right away. Thanks!

preinstall.sh does not work on rpm based linux

there current version of preinstall.sh makes it impossibe to install node-zoom2 0.2.4 on Fedora, RedHat oder CentOS machines.

on Fedora Core 25 yaz is part of distribution ( dnf install -y libyaz-devel ).

but in general os package installation should be out of "npm install" scope, maybe part of Dockerfile.

support async callback functions

Would help a lot if callback functions - for connection.search(), resultset.getRecords(), etc. - would support async functions. Then we could save records to a database on the fly for example. Thanks!

stream destroy problem

I wrote a very simple script based on examples/stream.js. I works and i am getting records via sru, but before the script terminates i get the following error:

/home/ulrich/src/alma/sru-test/node_modules/node-zoom2/lib/read-stream.js:68
  state.destroyed = true;
                  ^

TypeError: Cannot set property 'destroyed' of undefined
    at ReadStream.stream.destroy (/home/ulrich/src/alma/sru-test/node_modules/node-zoom2/lib/read-stream.js:68:19)
    at ReadStream.stream._read (/home/ulrich/src/alma/sru-test/node_modules/node-zoom2/lib/read-stream.js:62:10)
    at ReadStream.stream._zoomReady (/home/ulrich/src/alma/sru-test/node_modules/node-zoom2/lib/read-stream.js:111:10)
    at ReadStream.<anonymous> (/home/ulrich/src/alma/sru-test/node_modules/node-zoom2/lib/read-stream.js:102:10)

【Error：the specified module could not be found】.but the file exist “zoom.node”

I come from http://zoom.z3950.org/bind/nodejs/ .

These steps were done:
① My os is windows server 2008 R2 x64 ,node 12.22 ,python 2.7 .
② Install yaz5 . download and install https://ftp.indexdata.com/pub/yaz/win64/yaz_5.31.1.exe , install to "C:\Program Files\YAZ", and add to PATH .
③ It requires to install vs2017 when exec "npm i node-zoom2", so vs2017 was installed by the way .
④ Exec "npm i node-zoom2" again and it seems successed .

I got a problem:
It goes error when exec 'node C:\Users\Administrator\AppData\Roaming\npm\node_modules\node-zoom2\examples\simple.js' in CMD as shown in figures. it says Error：the specified module could not be found. \\?\C:\Users\Administrator\AppData\Roaming\npm\node_modules\node-zoom2\build\Release\zoom.node
But the file “zoom.node” exist in the folder.

What should i do the next? Thank you for your help

how to set the charset ? some results have been encoded by GBK ,and chinese characters are garbled

chinese characters of the records appear as black squares . i tried transform them from gbk to utf-8 , but not works.
google says that : In nodejs , string that has been encoded in utf8 cannot be converted back to GBK .

1、I find this doc https://software.indexdata.com/yaz/doc/zoom.html ,it say The charset is used in the Content-Type header of HTTP requests.

2、The following settings were attempted but not valid
.set('charset',' iso8859') //ASCII utf8
.set('lang','56')
.set('Content-Type','text/html;charset:utf-8')
.set('Content-Language','zh-CN')

please have a look .
the record.raw from zoom.connection('z39.91marc.cn:2100/uc_bib') and used user/password .

3、the record.raw from this server zoom.connection('192.83.186.170:210/INNOPAC') is natural .

4、when .set('preferredRecordSyntax', 'cnmarc') ,the module warns this , could the module support 'cnmarc' ?

ResultSet.close() if zoom has it

When working with large resultsets and calling getRecords() on the whole at once the node process is leaking memory and growing biger and bigger. Might not be directly related to retrieving results (I can send you my code privately) but maybe you can look into to. Not too important as this is a one off thing. Works great with usual resultsets.

API results vs. yaz-client

I'm looking into what is different between the API node-zoom2 uses and the yaz-client app. It seems that for the same query they give different results (success or 0) so there must be some difference between how they interpret or send the query or what options they set or not silently, or something else. I'm using z3950.mokka.hu:2210 a national book catalog of Hungary and queries that search for first and last names of authors, like f @attr 1=1 "Twain Mark". Over 1K results in yaz-client and 0 in node-zoom2. Optional commas or the order of names don't make any differenc in yaz-client.

json conversion problem if raw format is marcxml

It seems that the conversion to json or xml works if raw format is usmarc, but it does not work if raw format is marcxml. I am unsure if it is related to missing record xmlns for this datasource.

'use strict';

const zoom = require('node-zoom2');

var sru = zoom.connection('obv-at-obvsg.userservices.exlibrisgroup.com')
  .set('sru','get')
  .set('sru_version','1.2')
  .set('schema','marcxml')
  .set('databaseName', 'view/sru/43ACC_NETWORK')
  .query('cql', 'alma.other_system_number = (AC02222419)')
  .createReadStream()
  .on('data', (record) => {
    console.log(record.xml);
    console.log(record.json);  // parsing error
  });