Add System Import/Export

[ssddanbrown]

Add System Import/Export

[davidtorcivia]

Adding support for this feature. The export ability for single pages is great, but being able to export an entire book or chapter to PDF/HTML/TXT (or even EPUB directly) would be fantastic and could really expand the usecases of the software.

[terxw]

Currently I am using md text files and I am testing bookstack, it would be awsome to import all files from directory as pages.

[Kazakhstanec94]

Good day guys.
Is it possible to import data from confluence? If so, how to do it

[viewty]

Markdown export would be great if it's possible for that to be incorporated at some point?

[patoroco]

Related with #177 for export whole book.

Anyway, I would like a feature to bulk-import pages from... (in our case) github wiki pages (a bunch of markdown files)

[b0r1sp]

Atlassian drops support of self-hosted Confluence, any chances to create a painless migration scenario to bookstack?

[brutecode]

.md import would be great as a starting point for migration. PDF export of shelves would also be great.

[edenpc]

+1 for importing .md

[Leon99]

Any chances for this to get done? Migration scenarios are currently not possible, which means that BookStack missing out on all those businesses around with existing wikis.

[Nebucatnetzer]

I agree, importing would be great. I'm currently importing a lot of notes by hand.
It's sort of a good thing since I have to go through all them but honestly if given the choice I would've just uploaded them. I converted them from org-mode to html with pandoc.
Works quite well when I paste the code directly into the source editor.

[Leon99]

@ssddanbrown any chances for markdown import, being the most requested approach (which kind of makes sense, given that most wiki systems support it one way or another), implemented by you?

[openmoto]

Hi guys,
I have a bookstack setup with one shelf per group I'm helping, after I complete the initial documentation for them and get a go ahead, I create their own bookstack server and manually copy their content page by page.

Is there no chance of having an export/import or migrate shelf to another bookstack instance feature?

[ssddanbrown]

@oden08 Import is correctly not on the roadmap itself.

To be honest, "Import" as a feature request on its own is way too generic. The are many different formats or systems people will be looking to import from that we can't really just have a generic "Import" option. I initially opened this issue envisioning this to be a way to backup/restore a BookStack instance although I've had trouble seeing how we could achieve this as reliably as the current MySQL dump option.

On the roadmap is a REST API. For importing from other formats or systems, I imagine the best way forward will be for people to write scripts or applications to import into BookStack using the API. That way the BookStack project itself does not have to maintain a whole bunch of import logic.

That's how I see things moving forward.

[ssddanbrown]

Hi @Smoin1,
The API has been started, you can find documentation at <your_bookstack_url>/api/docs, You'll need the "Access system API" permission for this. API endpoints are being expanded on each release.

No updates on shelf exports.

[eckad158]

Thank you @ssddanbrown for all the work you have already done. I think some kind of transfer software from confluence to bookstack would be really usefull.
I respect your choice not to implement this feature since bookstack already offers a generic API.

Today i started building a python script that transfers data from confluence to bookstack (C2BS).

[bitinerant]

Thank you, sir! That did it. :-)

[ssddanbrown]

@openmoto Not right now. Could probably do some creating scripting to achieve what you're doing. Uploaded images could make things tricky though.

As per this comment above, An API is upcoming on the roadmap which will make scripting this kind of thing easier.

[ssddanbrown]

@b0r1sp I'm not looking to include any platform specific (Including confluence) migration/import/export logic within the core project itself. I'm just finishing up the core parts of the API though so things will become easier for others to make such a process. Depending on availability I may contribute to that myself but I'm not familiar with the specific workings of confluence so I'd rather someone with confluence experience led such a task.

[robertoschwald]

Tbh to get Confluence users into BookStackApp, importer is a must, otherwise this product looses lots of potential users.
I would generally say this should be maintained / developed as plugins for different systems, so 3rd parties can develop the implementations, but as #127 is not on the roadmap, this is currently not possible. Sad.

[ssddanbrown]

@robertoschwald If you want to discuss a plugin system please do that in #127 but, if you proceed, please do so constructively with my last two comments of #127 in mind, instead of providing unfounded comments such as "this project is loosing traction".

[idx3d]

MD Import could be really great feature.
Thanks!

[oden08]

Is it possible to get an update on if an import is on the roadmap?

[bitinerant]

In the mean time, I'd be content hacking together my own import solution, but I get "Page not found" when I do the following:

• add shelves, books, chapters, pages, e.g.:

INSERT INTO `pages` VALUES (NULL,1,0,'title here','slug-here','<p id=\"bkmrk-text-here\">text here</p>','text here',5,'2019-06-04 15:55:55','2019-06-04 15:55:55',1,1,0,0,'text here',1)

• rebuild the search index - see docs and issue #974

What else is needed?

[ssddanbrown]

@bitinerant Try running php artisan bookstack:regenerate-permissions. Think the permission index just needs to be updated.

[bitinerant]

Given the markdown, how can I generate the html and plain text to insert into the pages table, preferably from the command-line? Or, how can I refresh these 2 fields in the entire table? I've been searching all over for what should be really simple, but I don't know PHP or JavaScript.

[ssddanbrown]

@bitinerant Currently the markdown conversion to HTML is performed client side so there's no back-end systems to do this. You'd have to convert before sending in manually.

The conversion to plaintext is done at the following location:
https://github.com/BookStackApp/BookStack/blob/v0.26.2/app/Entities/Repos/PageRepo.php#L213-L217

The plaintext generation is not available via command-line or anything at this time.

[bitinerant]

Thanks a lot, Dan. I'll either write a script to do a null edit on every page from the client, or use something like Pandoc.

[wizofe]

@bitinerant Pandoc sounds like a great idea! How far are you with your mitigation/convertion script? As I am writing something similar and don't want to duplicate efforts :)

[bitinerant]

@wizofe - other projects preempted this, so I'm not too far yet. What wiki are you converting from and what language are you writing the conversion in? For me this is DokuWiki and Python.

So far I have several regular expressions to 'fix' Pandoc's incomplete DokuWiki support, the Pandoc conversion (markdown = pypandoc.convert(dokuwiki, format="dokuwiki", to="commonmark+pipe_tables+backtick_code_blocks", extra_args=["--atx-headers", "--wrap=none"])), and then several regular expressions to personalize the markdown. In the near future I hope to finish my regular expressions, generate html with Pandoc (should be trivial), and convert old revisions (history).

I'd be happy to coordinate on parts of the development if it makes sense.

[OlofBailey]

Import from Mindtouch DekiWiki would be awsome.

[Smoin1]

any updates on the rest api or the possibility to export a whole shelf? :)

[ashleycawley]

I would also love to see exporting and importing functionality added to BookStack in a format of which BookStack can easily interpret on the import. For example, lets say I have a instance with various shelves and books in. It would be desirable to be able to export a whole book (images and all) in a BookStack friendly format and then that export dump could be used in conjunction with a clean BookStack instance, and on the import process if it is in a nice forma it could understand it would be great if it had a import wizard which actually asked you interactively what data you wished to import, giving you the choice of the whole book, or select specific chapters or pages only etc.

I know its very easy to request such features but potentially a nightmare to try and build them (I'm incapable of doing such things myself) so I want to thank you for what we do have! And thank any contributors that work towards this feature.

[lieszkol]

+1 for this feature exactly as @ashleycawley described it. Much thanks to ssddabrown for this awesome project! Perhaps a bounty for such features to keep the developer(s) motivated?

[dawidmachon]

+1 as request for import

Thanks for export ;)

[ssddanbrown]

I would generally say this should be maintained / developed as plugins for different systems, so 3rd parties can develop the implementations, but as #127 is not on the roadmap, this is currently not possible. Sad.

@robertoschwald as per my last comment on #217 there are now a range of extension/integration options. Think the API would be the most useful in this case. Nothing is stopping you, or a third party from using the API to ease import/migration. If the API is lacking that can be raised as a new issue (If not already open). Even before that some have automated such tasks direct via the database (Although this is not ideal).

[robertoschwald]

I fear without a plugin ecosystem (plugin registry etc.) this project is loosing traction and a lot of opportunities.

[jonathon2nd]

We have started testing BookStack due to Tettra raising their prices.

Is there a way to import the HTML export from Tettra into BookStack?

---

EDIT: OK I have something working.
So for our Tettra export, files are separate form the html, so fix that with https://gist.github.com/pansapiens/110431456e8a4ba4f2eb
python standalone_html.py Thing.html Thing-new.html

Then, post with
curl -X POST http://wiki.example.com/api/pages -H 'Authorization: Token E:E' -F "book_id=1" -F "name=Thing" -F "html=<Thing-new.html"
EDIT: The < in "html=<Thing-new.html" is very important and not a typo. See man curl for more info. Important except at the bottom.

This creates an almost identical page to what we had in Tettra. Including the images, yay!

Two things left to iron out.
The urls in on the page are not highlighted/clickable on import,
This is actually Tettra's fault. Their export of urls is nonsensical and not consistent. Requires manual fixing.

and the double spacing all all newlines is aggravating. I see how to change that here: #1043 (comment), but going to see if there is some way to automate this on import. Will update when I have that.

---

Getting it to upload the actual file contents was a massive pain. Summary of what I tried so that it can help others.

First tried this: https://stackoverflow.com/questions/47662895/how-to-send-json-file-as-part-of-request-body-in-curl-post-command/47662945#47662945
But Curl is not happy with that

$ curl -X POST  http://wiki.example.com/api/pages -H 'Authorization: Token E:E' -F "book_id=1" -F "name=Thing" -F "html=$data"
bash: /usr/bin/curl: Argument list too long

Tried a bunch of other stuff, modifying the file or the form did not help.

I dug into man for curl and found this

Example: send a your essay in a text field to the server. Send it as a plain text field, but get the contents for it from a local file:
curl -F "story=<hugefile.txt" https://example.com/

And with that modification, I am able to upload an html file into the html field on the curl post. Hope this helps someone else.

---

man curl except

-F, --form <name=content>
              (HTTP SMTP IMAP) For HTTP protocol family, this lets curl emulate a filled-in form in which a user has pressed the submit button. This causes curl to POST data using the Content-Type multipart/form-data according to RFC 2388.
              For SMTP and IMAP protocols, this is the mean to compose a multipart mail message to transmit.
              This enables uploading of binary files etc. To force the 'content' part to be a file, prefix the file name with an @ sign. To just get the content part from a file, prefix the file name with the symbol <. The difference between @ and < is then that @ makes a file get attached in the post  as  a
              file upload, while the < makes a text field and just get the contents for that text field from a file.
              Tell  curl  to  read  content from stdin instead of a file by using - as filename. This goes for both @ and < constructs. When stdin is used, the contents is buffered in memory first by curl to determine its size and allow a possible resend.  Defining a part's data from a named non-regular file
              (such as a named pipe or similar) is unfortunately not subject to buffering and will be effectively read at transmission time; since the full size is unknown before the transfer starts, such data is sent as chunks by HTTP and rejected by IMAP.
              ...
              Example: send a your essay in a text field to the server. Send it as a plain text field, but get the contents for it from a local file:
               curl -F "story=<hugefile.txt" https://example.com/

[ssddanbrown]

@jonathon2nd Fair play for getting that going with bash/curl.
If you haven't seen we have a bunch of examples in our api scripts repo here, might help, otherwise feel free to question any API details on our discord.

[ssddanbrown]

@idx3d Our API does already accept markdown input, otherwise you could just paste stuff into the markdown page editor.

[dimisjim]

@Kazakhstanec94 did you find a way to import from Confluence Server?

[GanizaniSitara]

Alas, no, and have pretty much given up on this. I think if we ever do migrate it will be to MediaWiki. On 16 Jan 2024, at 17:31, Dimitris Moraitidis ***@***.***> wrote:  @Kazakhstanec94<https://github.com/Kazakhstanec94> did you find a way to import from Confluence Server? — Reply to this email directly, view it on GitHub<#43 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AB4RHWSUBGKEBQHJYGRBZPLYO22PDAVCNFSM4BYRXNCKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBZGQYTSNZYGEYQ>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>