Giter Site home page Giter Site logo

nodebb-plugin-import's Issues

NBB.Post.create() too slow, can I avoid it

It's too slow, NodeBB does A LOT of stuff that an importer can avoid,
maybe I should hit the database directly and mimic everything that NBB does for each post creation, except unnecessary features, such as notifications, and socket.io stuff .. etc.

v0.1.60 / v0.1.61 redis froze at Loading storage

When using Redis and the v0.1.60 / v0.1.61 with this command the importer stuck at:

node import.js --storage="/opt/oldforum/storage" --log="debug,warn,info" --flush --convert="bbcode-to-md" | tee import.log
[import][debug] init()
[import][info] Loading storage directory: /opt/oldforum/storage into memory, please be patient, that might take few minutes depending on many records you are importing...

(usually takes only 30sec before invoking on your behalf node)
...
Now 10 minutes and still nothing.

topic without post at end of category

when importing a topic that have no post/reply, this topic gets listed at the very bottom of the category. would like to have it listed like the others (using timestamp).

TypeError: Cannot read property 'sockets' of undefined

Having a few issues porting Topics over, issue is present on cutting edge, 0.4.1 & 0.4.0.
Importing Topics ...
[import][warn] [c:1] skipping topic:_tid:"2" --> _cid:valid: true _uid:valid: false
[import][warn] [c:2] skipping topic:_tid:"3" --> _cid:valid: true _uid:valid: false
[import][debug] [c:3] saving topic:_tid: 4
[import][debug] [c:4] saving topic:_tid: 6
[import][debug] [c:5] saving topic:_tid: 7
[import][debug] [c:6] saving topic:_tid: 8
[import][warn] [c:7] skipping topic:_tid:"9" --> _cid:valid: true _uid:valid: false
[import][debug] [c:8] saving topic:_tid: 10

/home/a_5mith/nodebb/src/socket.io/index.js:199
var clients = io.sockets.clients();
^
TypeError: Cannot read property 'sockets' of undefined
at Object.Sockets.getConnectedClients (/home/a_5mith/nodebb/src/socket.io/index.js:192:18)
at Object.Topics.pushUnreadCount (/home/a_5mith/nodebb/src/topics/unread.js:121:22)
at async.waterfall.postData.topic (/home/a_5mith/nodebb/src/topics/create.js:208:12)
at fn (/home/a_5mith/nodebb/node_modules/async/lib/async.js:582:34)
at Object._onImmediate (/home/a_5mith/nodebb/node_modules/async/lib/async.js:498:34)
at processImmediate as _immediateCallback`

list of improvements before next release

  • better logging/reporting
  • refactor data iterators f* it
  • tidy the UI a bit meh
  • investigate "Segmentation Fault" errors when import a large forum (try this for debugging)
  • try to solve #45
  • remove the fn routes

Something went wrong while restoring your nbb configs

Hi,

I am facing this issue:

[import][info] Relocking 5765 topics took: 4.62 seconds
[import][debug] restoring configs
[import][debug] {}
[import][error] Something went wrong while restoring your nbb configs
[import][warn] here are your backed-up configs, you do it.
[import][warn] {}
error: Error: ERR wrong number of arguments for 'hmset' command
    at ReplyParser.<anonymous> (/root/NodeBB/node_modules/redis/index.js:305:31)
    at ReplyParser.emit (events.js:95:17)
    at ReplyParser.send_error (/root/NodeBB/node_modules/redis/lib/parser/javascript.js:296:10)
    at ReplyParser.execute (/root/NodeBB/node_modules/redis/lib/parser/javascript.js:181:22)
    at RedisClient.on_data (/root/NodeBB/node_modules/redis/index.js:534:27)
    at Socket.<anonymous> (/root/NodeBB/node_modules/redis/index.js:91:14)
    at Socket.emit (events.js:95:17)
    at Socket.<anonymous> (_stream_readable.js:748:14)
    at Socket.emit (events.js:92:17)
    at emitReadable_ (_stream_readable.js:410:10)
    at emitReadable (_stream_readable.js:406:5)

It's stuck there. Any idea?

dependencies

module.js:340
    throw err;
    ^
Error: Cannot find module 'jQuery'
    at Function.Module._resolveFilename (module.js:338:15)
    at Function.Module._load (module.js:280:25)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/root/nodebb/node_modules/nodebb-plugin-import/lib/import.js:26:6)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)

Fixed by running:

npm install jQuery && npm install jsdom && npm install xmlhttprequest && npm install xregexp

config confusion

So I have a config.json in my nodebb folder, there is an import.config.json inside the plugin folder and then there is another config setupVal block inside import.js. Which one is actually used when I run the importer with

node import.js --storage="../storage" --config="../import.config.json" --flush --log="debug" | tee import.log

Looks like it takes import.config.json but then why is there another config inside import.js

move away from node-persist module, meaning, drop ALL persistance?

the whole idea of using node-persist, and this bulky importer is to be able to kill the import process, remove the --flush flag and re-launch it again, and it will pick up where it left off.

The reason I did this and not just use the memory is if migrating seriously large posts, I ran out of MEMORY, since everything was being parsed in memory.

However, even though that remedied the memory loss, as I could just pick up where I left off, it didn't actually fix the memory issues, till I figured that the html-md module is causing it - which i am still waiting on a pull-request for it to be patched (for now I am using my own manual fork)

Anyways, I am thinking of dropping all persistance, this means the following:

  • the importer will no longer be able to pick up where it left off, but rather --flush everytime, and start all over again if it get interrupted.
  • no more file system storage structure, i.e. all .json files will no longer be needed, and the way that would work with exporters is that all exporters will now be local node_modules of this importer, and they would need to provide 4 API functions (at least) (i.e. getCategories, getUsers, getTopics, 'getPosts') -- which each will accept a callback that accepts an array of records, which sort of look like what the {{u | t | c | p}{id}.json}.normalized files structure look like ..
  • No need to do 2 steps anymore, you would launch the importer with config of the source and target databases, along with which exporter (using its npm name) you would like to use to export the data.

@barisusakli @julianlam @psychobunny you guys used this before, or tried to, what do you think? should I drop all local persistence, and make this importer much lighter? or do you think it's valuable?

passwordGen not working?

In my import.config.json I have:

{
    "log": "debug",
    "passwordGen": {
        "enabled": true,
        "chars": "{}.-_=+qwertyuiopasdfghjklzxcvbnmQWERTYUIOPASDFGHJKLZXCVBNM1234567890",
        "len": 8
    },
...

But in import.log doesn't seems to spit out the generated password:

...
[user-json] {"email":"[email protected]","username":"username","pwd":"null",_uid":125,"uid":93,"ms":1108721206000},
[user-csv] [email protected],username,null,125,93,1108721206000
...

node-gyp rebuild fails to build

I was unable to install the plugin through the Admin page. The end-goal is to use the nodebb-plugin-import-phpbb plugin. Below is the result of trying to manually install the plugin. Thank you for any insight.

sudo npm install nodebb-plugin-import

npm WARN package.json [email protected] No repository field.
npm WARN package.json [email protected] No repository field.
npm WARN package.json [email protected] No repository field.
npm WARN package.json [email protected] No repository field.
npm WARN package.json [email protected] No repository field.
npm http GET https://registry.npmjs.org/nodebb-plugin-import
npm http 200 https://registry.npmjs.org/nodebb-plugin-import
npm http GET https://registry.npmjs.org/nodebb-plugin-import/-/nodebb-plugin-import-0.1.2.tgz
npm http 200 https://registry.npmjs.org/nodebb-plugin-import/-/nodebb-plugin-import-0.1.2.tgz
npm http GET https://registry.npmjs.org/exec-sync
npm http GET https://registry.npmjs.org/underscore
npm http GET https://registry.npmjs.org/optimist
npm http GET https://registry.npmjs.org/marked
npm http GET https://registry.npmjs.org/tiny-logger
npm http GET https://registry.npmjs.org/fs-extra
npm http GET https://registry.npmjs.org/jquery
npm http GET https://registry.npmjs.org/jsdom
npm http 304 https://registry.npmjs.org/marked
npm http 304 https://registry.npmjs.org/underscore
npm http 200 https://registry.npmjs.org/jquery
npm http GET https://registry.npmjs.org/jquery/-/jquery-2.1.1.tgz
npm http 200 https://registry.npmjs.org/exec-sync
npm http 304 https://registry.npmjs.org/optimist
npm http 304 https://registry.npmjs.org/fs-extra
npm http GET https://registry.npmjs.org/exec-sync/-/exec-sync-0.1.6.tgz
npm http 304 https://registry.npmjs.org/tiny-logger
npm http 200 https://registry.npmjs.org/jsdom
npm http 200 https://registry.npmjs.org/exec-sync/-/exec-sync-0.1.6.tgz
npm http GET https://registry.npmjs.org/jsdom/-/jsdom-0.8.11.tgz
npm http 200 https://registry.npmjs.org/jsdom/-/jsdom-0.8.11.tgz
npm http 200 https://registry.npmjs.org/jquery/-/jquery-2.1.1.tgz
npm http GET https://registry.npmjs.org/ffi
npm http GET https://registry.npmjs.org/ncp
npm http GET https://registry.npmjs.org/jsonfile
npm http 200 https://registry.npmjs.org/ffi
npm http 304 https://registry.npmjs.org/jsonfile
npm http GET https://registry.npmjs.org/ffi/-/ffi-1.2.5.tgz
npm http 304 https://registry.npmjs.org/ncp
npm http GET https://registry.npmjs.org/wordwrap
npm http GET https://registry.npmjs.org/minimist
npm http 200 https://registry.npmjs.org/ffi/-/ffi-1.2.5.tgz
npm http 304 https://registry.npmjs.org/wordwrap
npm http 304 https://registry.npmjs.org/minimist
npm http GET https://registry.npmjs.org/fs-extra/-/fs-extra-0.7.1.tgz
npm http GET https://registry.npmjs.org/commander
npm http 304 https://registry.npmjs.org/commander
npm http 200 https://registry.npmjs.org/fs-extra/-/fs-extra-0.7.1.tgz
npm http GET https://registry.npmjs.org/htmlparser2
npm http GET https://registry.npmjs.org/nwmatcher
npm http GET https://registry.npmjs.org/xmlhttprequest
npm http GET https://registry.npmjs.org/cssom
npm http GET https://registry.npmjs.org/cssstyle
npm http GET https://registry.npmjs.org/contextify
npm http 200 https://registry.npmjs.org/nwmatcher
npm http GET https://registry.npmjs.org/nwmatcher/-/nwmatcher-1.3.3.tgz
npm http 200 https://registry.npmjs.org/cssom
npm http 200 https://registry.npmjs.org/contextify
npm http GET https://registry.npmjs.org/cssom/-/cssom-0.3.0.tgz
npm http GET https://registry.npmjs.org/contextify/-/contextify-0.1.7.tgz
npm http 200 https://registry.npmjs.org/nwmatcher/-/nwmatcher-1.3.3.tgz
npm http 200 https://registry.npmjs.org/htmlparser2
npm http 200 https://registry.npmjs.org/cssom/-/cssom-0.3.0.tgz
npm http 304 https://registry.npmjs.org/xmlhttprequest
npm http 200 https://registry.npmjs.org/contextify/-/contextify-0.1.7.tgz
npm http 200 https://registry.npmjs.org/cssstyle
npm http GET https://registry.npmjs.org/xmlhttprequest/-/xmlhttprequest-1.6.0.tgz
npm http GET https://registry.npmjs.org/htmlparser2/-/htmlparser2-3.7.1.tgz
npm http GET https://registry.npmjs.org/cssstyle/-/cssstyle-0.2.11.tgz
npm http 200 https://registry.npmjs.org/htmlparser2/-/htmlparser2-3.7.1.tgz
npm http 200 https://registry.npmjs.org/xmlhttprequest/-/xmlhttprequest-1.6.0.tgz
npm http 200 https://registry.npmjs.org/cssstyle/-/cssstyle-0.2.11.tgz
npm http GET https://registry.npmjs.org/bindings
npm http GET https://registry.npmjs.org/domhandler
npm http GET https://registry.npmjs.org/domutils
npm http GET https://registry.npmjs.org/domelementtype
npm http GET https://registry.npmjs.org/entities
npm http GET https://registry.npmjs.org/nan
npm http GET https://registry.npmjs.org/readable-stream
npm http 304 https://registry.npmjs.org/bindings
npm http 200 https://registry.npmjs.org/domutils
npm http 200 https://registry.npmjs.org/domhandler
npm http 200 https://registry.npmjs.org/domelementtype
npm http 304 https://registry.npmjs.org/nan
npm http 304 https://registry.npmjs.org/readable-stream
npm http GET https://registry.npmjs.org/domutils/-/domutils-1.4.3.tgz
npm http GET https://registry.npmjs.org/domhandler/-/domhandler-2.2.0.tgz
npm http GET https://registry.npmjs.org/domelementtype/-/domelementtype-1.1.1.tgz
npm http GET https://registry.npmjs.org/nan/-/nan-0.8.0.tgz
npm http 200 https://registry.npmjs.org/domhandler/-/domhandler-2.2.0.tgz
npm http 200 https://registry.npmjs.org/domutils/-/domutils-1.4.3.tgz
npm http 200 https://registry.npmjs.org/nan/-/nan-0.8.0.tgz
npm http 200 https://registry.npmjs.org/entities
npm http GET https://registry.npmjs.org/entities/-/entities-1.0.0.tgz
npm http 200 https://registry.npmjs.org/domelementtype/-/domelementtype-1.1.1.tgz
npm http 200 https://registry.npmjs.org/entities/-/entities-1.0.0.tgz

> [email protected] install /var/www/nodebb/node_modules/nodebb-plugin-import/node_modules/jsdom/node_modules/contextify
> node-gyp rebuild

gyp WARN EACCES user "root" does not have permission to access the dev dir "/root/.node-gyp/0.10.26"
gyp WARN EACCES attempting to reinstall using temporary dev dir "/var/www/nodebb/node_modules/nodebb-plugin-import/node_modules/jsdom/node_modules/contextify/.node-gyp"
gyp http GET http://nodejs.org/dist/v0.10.26/node-v0.10.26.tar.gz
gyp http 200 http://nodejs.org/dist/v0.10.26/node-v0.10.26.tar.gz
gyp http GET http://nodejs.org/dist/v0.10.26/SHASUMS.txt
gyp http 200 http://nodejs.org/dist/v0.10.26/SHASUMS.txt
Traceback (most recent call last):
  File "/usr/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py", line 18, in <module>
    sys.exit(gyp.script_main())
AttributeError: 'module' object has no attribute 'script_main'
gyp ERR! configure error 
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:340:16)
gyp ERR! stack     at ChildProcess.EventEmitter.emit (events.js:98:17)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (child_process.js:797:12)
gyp ERR! System Linux 2.6.32-431.11.2.el6.x86_64
gyp ERR! command "node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /var/www/nodebb/node_modules/nodebb-plugin-import/node_modules/jsdom/node_modules/contextify
gyp ERR! node -v v0.10.26
gyp ERR! node-gyp -v v0.13.0
gyp ERR! not ok 
npm http GET https://registry.npmjs.org/core-util-is
npm http GET https://registry.npmjs.org/isarray
npm http GET https://registry.npmjs.org/string_decoder
npm http GET https://registry.npmjs.org/inherits
npm http 304 https://registry.npmjs.org/core-util-is
npm http 304 https://registry.npmjs.org/isarray
npm http 304 https://registry.npmjs.org/string_decoder
npm http 304 https://registry.npmjs.org/inherits
npm http GET https://registry.npmjs.org/ref
npm http GET https://registry.npmjs.org/ref-struct
npm http GET https://registry.npmjs.org/debug
npm http 200 https://registry.npmjs.org/debug
npm http 200 https://registry.npmjs.org/ref-struct
npm http GET https://registry.npmjs.org/ref-struct/-/ref-struct-0.0.5.tgz
npm http 200 https://registry.npmjs.org/ref
npm http GET https://registry.npmjs.org/ref/-/ref-0.1.3.tgz
npm http 200 https://registry.npmjs.org/ref-struct/-/ref-struct-0.0.5.tgz
npm http 200 https://registry.npmjs.org/ref/-/ref-0.1.3.tgz

> [email protected] install /var/www/nodebb/node_modules/nodebb-plugin-import/node_modules/exec-sync/node_modules/ffi/node_modules/ref
> node-gyp rebuild



node.js:811
    var cwd = process.cwd();
                      ^
Error: ENOENT, no such file or directory
    at Function.startup.resolveArgv0 (node.js:811:23)
    at startup (node.js:58:13)
    at node.js:902:3
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.
npm ERR! This is most likely a problem with the contextify package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-gyp rebuild
npm ERR! You can get their info via:
npm ERR!     npm owner ls contextify
npm ERR! There is likely additional logging output above.
npm ERR! System Linux 2.6.32-431.11.2.el6.x86_64
npm ERR! command "node" "/usr/bin/npm" "install" "nodebb-plugin-import"
npm ERR! cwd /var/www/nodebb
npm ERR! node -v v0.10.26
npm ERR! npm -v 1.4.10
npm ERR! code ELIFECYCLE
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR!     /var/www/nodebb/npm-debug.log
npm ERR! not ok code 0

NodeBB v0.4.2.
npm 1.4.10
RHEL 6.5
node v0.10.20

Cannot call method 'debug' of undefined

Today I am facing this issue with 0.1.5. I'm using mongodb - is that a problem?

/root/nodebb/node_modules/nodebb-plugin-import/lib/import.js:278
      logger.debug('start()');
             ^
TypeError: Cannot call method 'debug' of undefined
    at Object.Import.start (/root/nodebb/node_modules/nodebb-plugin-import/lib/import.js:278:14)
    at Object.<anonymous> (/root/nodebb/node_modules/nodebb-plugin-import/import.js:53:8)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Function.Module.runMain (module.js:497:10)
    at startup (node.js:119:16)
    at node.js:906:3

More a question than an issue, maybe.

Hey pal, so, I thought I'd attempt something that could save me a lot of development time in the long run, my existing site is still running and gaining new posts, and I thought that I could try just adding any new posts/topics into the existing database.

So with an up to date database I went through the same importing steps. The topics transfer over just fine, but then freezes completely. The topics go over no problem, and appear in NodeBB, but the importer doesn't then go over to posts.

I know you didn't design this plugin to be reused, just wanted to get your input on whether it should work, or whether I'd be better off just starting again (I'm quite new to Redis, and running FLUSHDB says OK, and all posts vanish, but running the importer says all posts still exist. So not sure what the correct procedure is in that case.

Thanks.

TypeError: Cannot call method 'replace' of undefined

Trying to import phpBBex... changed your plugin-import-ubb. With categories all ok, but at the stage of importing users getting error. Maybe it's because of Russian usernames?

{"normalized":{"_uid":45,"_username":"п.п╦п╨п╟","_alternativeUsername":"п.п╦п╨п╟","_registrationEmail":"[email protected]","_level":0,"_joindate":1108
455618000,"_banned":0,"_email":"[email protected]","_signature":"[url=http://d...","_website":"","_occupation":"п╪п╟п╪п╟","_location":"п.пЇя.п╟п╦п
╩я.,п╒п╣п╩я.-п.п╡п╦п╡","_picture":"","_title":"п╡п╦д╦a","_reputation":0,"_profileviews":0,"_birthday":"16- 7-1977"}}

Importing Users ...

[import][debug] Administrator gid: null
[import][warn] [_username] is invalid, attempting to clean.
[import][warn] [c:1] skipping user: "undefined" username is invalid.
[import][debug] [c: 2] saving user:_uid: 45

/home/httpd/forum_doma/NodeBB/public/src/utils.js:114
str = str.replace(utils.trimRegex, '');
^
TypeError: Cannot call method 'replace' of undefined
at Object.module.exports.utils.slugify (/home/httpd/forum_doma/NodeBB/public/src/utils.js:114:14)
at Object.User.create (/home/httpd/forum_doma/NodeBB/src/user.js:24:29)
at async.eachLimit.storage.setItem.normalized._uid (/home/httpd/forum_doma/NodeBB/node_modules/nodebb-plugin-import/lib/import.js:402:11)
at replenish (/home/httpd/forum_doma/NodeBB/node_modules/async/lib/async.js:179:21)
at /home/httpd/forum_doma/NodeBB/node_modules/async/lib/async.js:196:15
at Object.async.eachLimit (/home/httpd/forum_doma/NodeBB/node_modules/async/lib/async.js:156:12)
at Object.Import.importUsers (/home/httpd/forum_doma/NodeBB/node_modules/nodebb-plugin-import/lib/import.js:366:9)
at /home/httpd/forum_doma/NodeBB/node_modules/nodebb-plugin-import/lib/import.js:230:11
at /home/httpd/forum_doma/NodeBB/node_modules/async/lib/async.js:551:21
at /home/httpd/forum_doma/NodeBB/node_modules/async/lib/async.js:227:13

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.