akhoury / nodebb-plugin-import Goto Github PK
View Code? Open in Web Editor NEWmigrate your old crappy forum to NodeBB
License: MIT License
migrate your old crappy forum to NodeBB
License: MIT License
It's too slow, NodeBB does A LOT of stuff that an importer can avoid,
maybe I should hit the database directly and mimic everything that NBB does for each post creation, except unnecessary features, such as notifications, and socket.io stuff .. etc.
exporter for questions2answers forums in the works
https://github.com/akhoury/nodebb-plugin-import-questions2answers
When using Redis and the v0.1.60 / v0.1.61 with this command the importer stuck at:
node import.js --storage="/opt/oldforum/storage" --log="debug,warn,info" --flush --convert="bbcode-to-md" | tee import.log
[import][debug] init()
[import][info] Loading storage directory: /opt/oldforum/storage into memory, please be patient, that might take few minutes depending on many records you are importing...
(usually takes only 30sec before invoking on your behalf node)
...
Now 10 minutes and still nothing.
i need a DB Dump
meaning, find alternative for exec-sync
module.
when importing a topic that have no post/reply, this topic gets listed at the very bottom of the category. would like to have it listed like the others (using timestamp).
let's say that my old forum had this category url:
http://forum.com/forum-2-patch
how to have a redirect on my new forum that points to the right url:
http://forum.com/category/2/patch
i see an option in import.config.json for doing this, but not sure how nodebb is handling this? trying to visit http://forum.com/forum-2-patch doesn't redirect me to http://forum.com/category/2/patch - am I missing a step?
so I can quickly pick-up where the import left off in case of the an unexpected shutdown
Having a few issues porting Topics over, issue is present on cutting edge, 0.4.1 & 0.4.0.
Importing Topics ...
[import][warn] [c:1] skipping topic:_tid:"2" --> _cid:valid: true _uid:valid: false
[import][warn] [c:2] skipping topic:_tid:"3" --> _cid:valid: true _uid:valid: false
[import][debug] [c:3] saving topic:_tid: 4
[import][debug] [c:4] saving topic:_tid: 6
[import][debug] [c:5] saving topic:_tid: 7
[import][debug] [c:6] saving topic:_tid: 8
[import][warn] [c:7] skipping topic:_tid:"9" --> _cid:valid: true _uid:valid: false
[import][debug] [c:8] saving topic:_tid: 10
/home/a_5mith/nodebb/src/socket.io/index.js:199
var clients = io.sockets.clients();
^
TypeError: Cannot read property 'sockets' of undefined
at Object.Sockets.getConnectedClients (/home/a_5mith/nodebb/src/socket.io/index.js:192:18)
at Object.Topics.pushUnreadCount (/home/a_5mith/nodebb/src/topics/unread.js:121:22)
at async.waterfall.postData.topic (/home/a_5mith/nodebb/src/topics/create.js:208:12)
at fn (/home/a_5mith/nodebb/node_modules/async/lib/async.js:582:34)
at Object._onImmediate (/home/a_5mith/nodebb/node_modules/async/lib/async.js:498:34)
at processImmediate as _immediateCallback`
to import the posts, and probably try different thresholds ..
need a dump
Need a DB Dump
an exporter for phpbb in the works
Use the NodeBB's isUsernameValid to strip invalid characters out, to try and save more usernames per migration.
symptom causes the config.postDelay
to stay 10 if first time.
http://forum.robocraftgame.fr/category/3/annonces-importantes take a look, there is "presentation" post here, but whe you click on it, this is right in the "presentation" category ^^' don't know what is the problem, altrought this can be more a "ipboard importer" problem :')
Hi,
I am facing this issue:
[import][info] Relocking 5765 topics took: 4.62 seconds
[import][debug] restoring configs
[import][debug] {}
[import][error] Something went wrong while restoring your nbb configs
[import][warn] here are your backed-up configs, you do it.
[import][warn] {}
error: Error: ERR wrong number of arguments for 'hmset' command
at ReplyParser.<anonymous> (/root/NodeBB/node_modules/redis/index.js:305:31)
at ReplyParser.emit (events.js:95:17)
at ReplyParser.send_error (/root/NodeBB/node_modules/redis/lib/parser/javascript.js:296:10)
at ReplyParser.execute (/root/NodeBB/node_modules/redis/lib/parser/javascript.js:181:22)
at RedisClient.on_data (/root/NodeBB/node_modules/redis/index.js:534:27)
at Socket.<anonymous> (/root/NodeBB/node_modules/redis/index.js:91:14)
at Socket.emit (events.js:95:17)
at Socket.<anonymous> (_stream_readable.js:748:14)
at Socket.emit (events.js:92:17)
at emitReadable_ (_stream_readable.js:410:10)
at emitReadable (_stream_readable.js:406:5)
It's stuck there. Any idea?
module.js:340
throw err;
^
Error: Cannot find module 'jQuery'
at Function.Module._resolveFilename (module.js:338:15)
at Function.Module._load (module.js:280:25)
at Module.require (module.js:364:17)
at require (module.js:380:17)
at Object.<anonymous> (/root/nodebb/node_modules/nodebb-plugin-import/lib/import.js:26:6)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.require (module.js:364:17)
Fixed by running:
npm install jQuery && npm install jsdom && npm install xmlhttprequest && npm install xregexp
need a dump
Possibly a broken dependency?
So I have a config.json
in my nodebb folder, there is an import.config.json
inside the plugin folder and then there is another config setupVal block inside import.js
. Which one is actually used when I run the importer with
node import.js --storage="../storage" --config="../import.config.json" --flush --log="debug" | tee import.log
Looks like it takes import.config.json
but then why is there another config inside import.js
so advanced users can write extra JS handling to parse their content
the whole idea of using node-persist, and this bulky importer is to be able to kill the import process, remove the --flush
flag and re-launch it again, and it will pick up where it left off.
The reason I did this and not just use the memory is if migrating seriously large posts, I ran out of MEMORY, since everything was being parsed in memory.
However, even though that remedied the memory loss, as I could just pick up where I left off, it didn't actually fix the memory issues, till I figured that the html-md module is causing it - which i am still waiting on a pull-request for it to be patched (for now I am using my own manual fork)
Anyways, I am thinking of dropping all persistance, this means the following:
--flush
everytime, and start all over again if it get interrupted.storage
structure, i.e. all .json files will no longer be needed, and the way that would work with exporters is that all exporters will now be local node_modules
of this importer, and they would need to provide 4 API functions (at least) (i.e. getCategories
, getUsers
, getTopics
, 'getPosts') -- which each will accept a callback that accepts an array of records, which sort of look like what the {{u | t | c | p}{id}.json}.normalized
files structure look like ..exporter
(using its npm name) you would like to use to export the data.@barisusakli @julianlam @psychobunny you guys used this before, or tried to, what do you think? should I drop all local persistence, and make this importer much lighter? or do you think it's valuable?
http://forum.robocraftgame.fr/topic/78/la-chatbox-est-lã still getting error with accent :x
No error at export, but these topic have lot of missing post :
http://robocraftgame.fr/forum/topic/89-la-chatbox-est-là/
http://forum.robocraftgame.fr/topic/78/la-chatbox-est-lã
http://prntscr.com/4khabh i think this is more a https://github.com/akhoury/nodebb-plugin-import-ipboard issue...
got a dump!
4.8 min to import 63k users
54 min to import 78k topic
92k/1.1M posts imported then everything froze
Segmentation fault: 11
Mac mini, i5, 2.5ghz, 16gb, 256gb ssd
In my import.config.json I have:
{
"log": "debug",
"passwordGen": {
"enabled": true,
"chars": "{}.-_=+qwertyuiopasdfghjklzxcvbnmQWERTYUIOPASDFGHJKLZXCVBNM1234567890",
"len": 8
},
...
But in import.log doesn't seems to spit out the generated password:
...
[user-json] {"email":"[email protected]","username":"username","pwd":"null",_uid":125,"uid":93,"ms":1108721206000},
[user-csv] [email protected],username,null,125,93,1108721206000
...
I was unable to install the plugin through the Admin page. The end-goal is to use the nodebb-plugin-import-phpbb plugin. Below is the result of trying to manually install the plugin. Thank you for any insight.
sudo npm install nodebb-plugin-import
npm WARN package.json [email protected] No repository field.
npm WARN package.json [email protected] No repository field.
npm WARN package.json [email protected] No repository field.
npm WARN package.json [email protected] No repository field.
npm WARN package.json [email protected] No repository field.
npm http GET https://registry.npmjs.org/nodebb-plugin-import
npm http 200 https://registry.npmjs.org/nodebb-plugin-import
npm http GET https://registry.npmjs.org/nodebb-plugin-import/-/nodebb-plugin-import-0.1.2.tgz
npm http 200 https://registry.npmjs.org/nodebb-plugin-import/-/nodebb-plugin-import-0.1.2.tgz
npm http GET https://registry.npmjs.org/exec-sync
npm http GET https://registry.npmjs.org/underscore
npm http GET https://registry.npmjs.org/optimist
npm http GET https://registry.npmjs.org/marked
npm http GET https://registry.npmjs.org/tiny-logger
npm http GET https://registry.npmjs.org/fs-extra
npm http GET https://registry.npmjs.org/jquery
npm http GET https://registry.npmjs.org/jsdom
npm http 304 https://registry.npmjs.org/marked
npm http 304 https://registry.npmjs.org/underscore
npm http 200 https://registry.npmjs.org/jquery
npm http GET https://registry.npmjs.org/jquery/-/jquery-2.1.1.tgz
npm http 200 https://registry.npmjs.org/exec-sync
npm http 304 https://registry.npmjs.org/optimist
npm http 304 https://registry.npmjs.org/fs-extra
npm http GET https://registry.npmjs.org/exec-sync/-/exec-sync-0.1.6.tgz
npm http 304 https://registry.npmjs.org/tiny-logger
npm http 200 https://registry.npmjs.org/jsdom
npm http 200 https://registry.npmjs.org/exec-sync/-/exec-sync-0.1.6.tgz
npm http GET https://registry.npmjs.org/jsdom/-/jsdom-0.8.11.tgz
npm http 200 https://registry.npmjs.org/jsdom/-/jsdom-0.8.11.tgz
npm http 200 https://registry.npmjs.org/jquery/-/jquery-2.1.1.tgz
npm http GET https://registry.npmjs.org/ffi
npm http GET https://registry.npmjs.org/ncp
npm http GET https://registry.npmjs.org/jsonfile
npm http 200 https://registry.npmjs.org/ffi
npm http 304 https://registry.npmjs.org/jsonfile
npm http GET https://registry.npmjs.org/ffi/-/ffi-1.2.5.tgz
npm http 304 https://registry.npmjs.org/ncp
npm http GET https://registry.npmjs.org/wordwrap
npm http GET https://registry.npmjs.org/minimist
npm http 200 https://registry.npmjs.org/ffi/-/ffi-1.2.5.tgz
npm http 304 https://registry.npmjs.org/wordwrap
npm http 304 https://registry.npmjs.org/minimist
npm http GET https://registry.npmjs.org/fs-extra/-/fs-extra-0.7.1.tgz
npm http GET https://registry.npmjs.org/commander
npm http 304 https://registry.npmjs.org/commander
npm http 200 https://registry.npmjs.org/fs-extra/-/fs-extra-0.7.1.tgz
npm http GET https://registry.npmjs.org/htmlparser2
npm http GET https://registry.npmjs.org/nwmatcher
npm http GET https://registry.npmjs.org/xmlhttprequest
npm http GET https://registry.npmjs.org/cssom
npm http GET https://registry.npmjs.org/cssstyle
npm http GET https://registry.npmjs.org/contextify
npm http 200 https://registry.npmjs.org/nwmatcher
npm http GET https://registry.npmjs.org/nwmatcher/-/nwmatcher-1.3.3.tgz
npm http 200 https://registry.npmjs.org/cssom
npm http 200 https://registry.npmjs.org/contextify
npm http GET https://registry.npmjs.org/cssom/-/cssom-0.3.0.tgz
npm http GET https://registry.npmjs.org/contextify/-/contextify-0.1.7.tgz
npm http 200 https://registry.npmjs.org/nwmatcher/-/nwmatcher-1.3.3.tgz
npm http 200 https://registry.npmjs.org/htmlparser2
npm http 200 https://registry.npmjs.org/cssom/-/cssom-0.3.0.tgz
npm http 304 https://registry.npmjs.org/xmlhttprequest
npm http 200 https://registry.npmjs.org/contextify/-/contextify-0.1.7.tgz
npm http 200 https://registry.npmjs.org/cssstyle
npm http GET https://registry.npmjs.org/xmlhttprequest/-/xmlhttprequest-1.6.0.tgz
npm http GET https://registry.npmjs.org/htmlparser2/-/htmlparser2-3.7.1.tgz
npm http GET https://registry.npmjs.org/cssstyle/-/cssstyle-0.2.11.tgz
npm http 200 https://registry.npmjs.org/htmlparser2/-/htmlparser2-3.7.1.tgz
npm http 200 https://registry.npmjs.org/xmlhttprequest/-/xmlhttprequest-1.6.0.tgz
npm http 200 https://registry.npmjs.org/cssstyle/-/cssstyle-0.2.11.tgz
npm http GET https://registry.npmjs.org/bindings
npm http GET https://registry.npmjs.org/domhandler
npm http GET https://registry.npmjs.org/domutils
npm http GET https://registry.npmjs.org/domelementtype
npm http GET https://registry.npmjs.org/entities
npm http GET https://registry.npmjs.org/nan
npm http GET https://registry.npmjs.org/readable-stream
npm http 304 https://registry.npmjs.org/bindings
npm http 200 https://registry.npmjs.org/domutils
npm http 200 https://registry.npmjs.org/domhandler
npm http 200 https://registry.npmjs.org/domelementtype
npm http 304 https://registry.npmjs.org/nan
npm http 304 https://registry.npmjs.org/readable-stream
npm http GET https://registry.npmjs.org/domutils/-/domutils-1.4.3.tgz
npm http GET https://registry.npmjs.org/domhandler/-/domhandler-2.2.0.tgz
npm http GET https://registry.npmjs.org/domelementtype/-/domelementtype-1.1.1.tgz
npm http GET https://registry.npmjs.org/nan/-/nan-0.8.0.tgz
npm http 200 https://registry.npmjs.org/domhandler/-/domhandler-2.2.0.tgz
npm http 200 https://registry.npmjs.org/domutils/-/domutils-1.4.3.tgz
npm http 200 https://registry.npmjs.org/nan/-/nan-0.8.0.tgz
npm http 200 https://registry.npmjs.org/entities
npm http GET https://registry.npmjs.org/entities/-/entities-1.0.0.tgz
npm http 200 https://registry.npmjs.org/domelementtype/-/domelementtype-1.1.1.tgz
npm http 200 https://registry.npmjs.org/entities/-/entities-1.0.0.tgz
> [email protected] install /var/www/nodebb/node_modules/nodebb-plugin-import/node_modules/jsdom/node_modules/contextify
> node-gyp rebuild
gyp WARN EACCES user "root" does not have permission to access the dev dir "/root/.node-gyp/0.10.26"
gyp WARN EACCES attempting to reinstall using temporary dev dir "/var/www/nodebb/node_modules/nodebb-plugin-import/node_modules/jsdom/node_modules/contextify/.node-gyp"
gyp http GET http://nodejs.org/dist/v0.10.26/node-v0.10.26.tar.gz
gyp http 200 http://nodejs.org/dist/v0.10.26/node-v0.10.26.tar.gz
gyp http GET http://nodejs.org/dist/v0.10.26/SHASUMS.txt
gyp http 200 http://nodejs.org/dist/v0.10.26/SHASUMS.txt
Traceback (most recent call last):
File "/usr/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py", line 18, in <module>
sys.exit(gyp.script_main())
AttributeError: 'module' object has no attribute 'script_main'
gyp ERR! configure error
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack at ChildProcess.onCpExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:340:16)
gyp ERR! stack at ChildProcess.EventEmitter.emit (events.js:98:17)
gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:797:12)
gyp ERR! System Linux 2.6.32-431.11.2.el6.x86_64
gyp ERR! command "node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /var/www/nodebb/node_modules/nodebb-plugin-import/node_modules/jsdom/node_modules/contextify
gyp ERR! node -v v0.10.26
gyp ERR! node-gyp -v v0.13.0
gyp ERR! not ok
npm http GET https://registry.npmjs.org/core-util-is
npm http GET https://registry.npmjs.org/isarray
npm http GET https://registry.npmjs.org/string_decoder
npm http GET https://registry.npmjs.org/inherits
npm http 304 https://registry.npmjs.org/core-util-is
npm http 304 https://registry.npmjs.org/isarray
npm http 304 https://registry.npmjs.org/string_decoder
npm http 304 https://registry.npmjs.org/inherits
npm http GET https://registry.npmjs.org/ref
npm http GET https://registry.npmjs.org/ref-struct
npm http GET https://registry.npmjs.org/debug
npm http 200 https://registry.npmjs.org/debug
npm http 200 https://registry.npmjs.org/ref-struct
npm http GET https://registry.npmjs.org/ref-struct/-/ref-struct-0.0.5.tgz
npm http 200 https://registry.npmjs.org/ref
npm http GET https://registry.npmjs.org/ref/-/ref-0.1.3.tgz
npm http 200 https://registry.npmjs.org/ref-struct/-/ref-struct-0.0.5.tgz
npm http 200 https://registry.npmjs.org/ref/-/ref-0.1.3.tgz
> [email protected] install /var/www/nodebb/node_modules/nodebb-plugin-import/node_modules/exec-sync/node_modules/ffi/node_modules/ref
> node-gyp rebuild
node.js:811
var cwd = process.cwd();
^
Error: ENOENT, no such file or directory
at Function.startup.resolveArgv0 (node.js:811:23)
at startup (node.js:58:13)
at node.js:902:3
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is most likely a problem with the contextify package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node-gyp rebuild
npm ERR! You can get their info via:
npm ERR! npm owner ls contextify
npm ERR! There is likely additional logging output above.
npm ERR! System Linux 2.6.32-431.11.2.el6.x86_64
npm ERR! command "node" "/usr/bin/npm" "install" "nodebb-plugin-import"
npm ERR! cwd /var/www/nodebb
npm ERR! node -v v0.10.26
npm ERR! npm -v 1.4.10
npm ERR! code ELIFECYCLE
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR! /var/www/nodebb/npm-debug.log
npm ERR! not ok code 0
NodeBB v0.4.2.
npm 1.4.10
RHEL 6.5
node v0.10.20
http://forum.robocraftgame.fr/category/3/annonces-importantes all the "gest's topics" are from me, but i'm already registered, true ? So i came as a guest.... :x
the CSV list must contain at minimum
email,username,password
Today I am facing this issue with 0.1.5. I'm using mongodb - is that a problem?
/root/nodebb/node_modules/nodebb-plugin-import/lib/import.js:278
logger.debug('start()');
^
TypeError: Cannot call method 'debug' of undefined
at Object.Import.start (/root/nodebb/node_modules/nodebb-plugin-import/lib/import.js:278:14)
at Object.<anonymous> (/root/nodebb/node_modules/nodebb-plugin-import/import.js:53:8)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Function.Module.runMain (module.js:497:10)
at startup (node.js:119:16)
at node.js:906:3
https://github.com/MakerStudios/nodebb-plugin-vbexporter/issues/2 doesn't look like it's maintained anymore (main collaborator account is deactivated), so I created a new one.
Hey pal, so, I thought I'd attempt something that could save me a lot of development time in the long run, my existing site is still running and gaining new posts, and I thought that I could try just adding any new posts/topics into the existing database.
So with an up to date database I went through the same importing steps. The topics transfer over just fine, but then freezes completely. The topics go over no problem, and appear in NodeBB, but the importer doesn't then go over to posts.
I know you didn't design this plugin to be reused, just wanted to get your input on whether it should work, or whether I'd be better off just starting again (I'm quite new to Redis, and running FLUSHDB says OK, and all posts vanish, but running the importer says all posts still exist. So not sure what the correct procedure is in that case.
Thanks.
Trying to import phpBBex... changed your plugin-import-ubb. With categories all ok, but at the stage of importing users getting error. Maybe it's because of Russian usernames?
{"normalized":{"_uid":45,"_username":"п.п╦п╨п╟","_alternativeUsername":"п.п╦п╨п╟","_registrationEmail":"[email protected]","_level":0,"_joindate":1108
455618000,"_banned":0,"_email":"[email protected]","_signature":"[url=http://d...","_website":"","_occupation":"п╪п╟п╪п╟","_location":"п.пЇя.п╟п╦п
╩я.,п╒п╣п╩я.-п.п╡п╦п╡","_picture":"","_title":"п╡п╦д╦a","_reputation":0,"_profileviews":0,"_birthday":"16- 7-1977"}}
Importing Users ...
[import][debug] Administrator gid: null
[import][warn] [_username] is invalid, attempting to clean.
[import][warn] [c:1] skipping user: "undefined" username is invalid.
[import][debug] [c: 2] saving user:_uid: 45
/home/httpd/forum_doma/NodeBB/public/src/utils.js:114
str = str.replace(utils.trimRegex, '');
^
TypeError: Cannot call method 'replace' of undefined
at Object.module.exports.utils.slugify (/home/httpd/forum_doma/NodeBB/public/src/utils.js:114:14)
at Object.User.create (/home/httpd/forum_doma/NodeBB/src/user.js:24:29)
at async.eachLimit.storage.setItem.normalized._uid (/home/httpd/forum_doma/NodeBB/node_modules/nodebb-plugin-import/lib/import.js:402:11)
at replenish (/home/httpd/forum_doma/NodeBB/node_modules/async/lib/async.js:179:21)
at /home/httpd/forum_doma/NodeBB/node_modules/async/lib/async.js:196:15
at Object.async.eachLimit (/home/httpd/forum_doma/NodeBB/node_modules/async/lib/async.js:156:12)
at Object.Import.importUsers (/home/httpd/forum_doma/NodeBB/node_modules/nodebb-plugin-import/lib/import.js:366:9)
at /home/httpd/forum_doma/NodeBB/node_modules/nodebb-plugin-import/lib/import.js:230:11
at /home/httpd/forum_doma/NodeBB/node_modules/async/lib/async.js:551:21
at /home/httpd/forum_doma/NodeBB/node_modules/async/lib/async.js:227:13
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.