dernorberto/confluencedumpwithpython

Download Confluence pages including attachments and emoticons using Atlassian API and Python

Confluence Dump With Python

Dump Confluence pages using Python (requests) in HTML and RST format, including embedded pictures and attachments. References to downloaded files will be updated to their local relative path.

Description

Nonetheless, the refactoring will require only 2 files and accept command-line args:

• myModules.py: Contains all the required functions.
• confluenceDumpWithPython.py: Script to use with the following command line args:
  • -m, --mode: The export mode, single, space, bylabel, pageprops (required).
    • Note: Only single, pageprops and space have been implemented so far.
  • -S, --site: The Atlassian Site (required).
  • -s, --space: The Space Key (if needed).
  • -p, --page: The Page ID (if needed).
  • -l, --label: The Page label (if needed).
  • -x, --sphinx: The _images and _static folders are placed at the root of the export folder, instead of together with the exported HTML files.
  • --notags: Does not add the tags directives to the rst files (when the sphinx-tags addon is not used).
• updatePageLinks.py: Update online confluence links to the local files that have been downloaded so far.
  • --folder: Folder containing the files to update.
  • --test: Instead of overwriting the original .rst files, it will create updated ones with zout_ as a prefix.
• getPageEditorVersion.py: Get the editor version from single pages or all pages in a space.
  • --site: The Atlassian Site (required).
  • --page: Page ID (either/or)
  • --space: Space Key (either/or)

For CSS Styling, it uses the confluence.css from Confluence that can be obtained by using the Workaround described in: https://jira.atlassian.com/browse/CONFSERVER-40907. The site.css file included with Confluence UI HTML exports is not as complete as the one above.

Folder and file structure:

• The default output folder is output/ under the same path as the script.
• A folder with the Space name, Page Properties report page, single page name or Page Label name will be created under the output folder.
• By default, the _images/ and _static/ folders will be placed in the page|space|pageprops|label folder.
  • The --sphinx command line option will put those folder directly under the output folder
• The file styles/confluence.css will be copied into the defined _static/

What it does

• Leverages the Confluence Cloud API
• Puts Confluence meta data like Page ID and Page Labels, in the HTML headers and RST fields.
• beautifulsoup is used to parse HTML to get and update content, ie. change remote links to local links.
• Download for every page, all attachments, emoticons and embedded files.

Requirements

• declare system variables:
  • atlassianAPIToken
  • atlassianUserEmail

Dependencies

• python3
  • requests
  • beautifulsoup4
  • Pillow (handle images)
  • pandoc & pypandoc (convert to RST)
  • re

Installing

• Clone repo.
• Install dependencies.
• Declare system variables for Atlassian API Token.

Executing program

• How to download a single page based on its ID.

confluenceDumpWithPython.py -m single -S <site Name> -p <ID of page to dump> [<output folder>] [--sphinx]

• How to download Page Properties and all the contained pages.

confluenceDumpWithPython.py -m pageprops -S <site Name> -p <ID of page properties report page> [<output folder>] [--sphinx]

• How to download a whole Space.

confluenceDumpWithPython.py -m space -S <site Name> -s <space KEY> [<output folder>]

Help

No special advice other than:

• make sure that your Atlassian API Token is valid.
• the username for the Cloud Atlassian API is the e-mail address.

Authors

Contributors names and contact info

@dernorberto

Improvements

• Add export based on page label.
• Add links to Downloads for the corresponding pages.
• Update all links from downloaded pages to the local copies.
• Add to headers the parent page and page labels.
• Create an index of the pages to use as a TOC.
• Create a page layout to display TOC + articles.
• Copy styles/site.css into output/styles/ if not present.
• Allow using with Confluence Server.

Issues

• It does not like very long attachment files, you'll need to rename them in Confluence before the dump.
• Pages previously migrated from Confluence Server might have issues with old emoticons. The best is to convert the pages to the New Editor, which will replace the missing emoticons.

Version History

• 1.4
  • Refactoring into a more simple file setup (confluenceDumpWithPython.py & myModules.py)
• 1.3
  • Added Space export (flat folder structure)
• 1.2
  • Added better HTML header and footer.
  • Added page labels to HTML headers.
  • Improved output folder argument logic.
• 1.1
  • Added Papge Properties dump and other smaller things
• 1.0
  • Initial Release

legacy/ folder with previous version of scripts

Purpose of the files:

1. confluenceExportHTMLrequestsByLabel.py: download a set of pages based on one (or more) page Labels.
2. confluenceExportHTMLrequestsSingle.py: download a single page by supplying the page ID as an argument.
3. confluenceExportHTMLrequestsPagePropertiesReport.py: download page properties and all the pages in the report by supplying the page ID as an argument.
4. confluenceExportHTMLrequestsPagesInSpace.py: download all pages from a space.

License

This project is licensed under the MIT License - see the LICENSE.txt file for details

Acknowledgments

confluencedumpwithpython's Issues

PagePropertiesReport: Report HTML takes the title and original URL from the last child page

Expected:

• Report file: Security Policies.html
• Report Title: Security Policies
• last child file: Time Synchronisation Policy.html

Result:

• Report Page Source:

<head>
--
  | <title>Time Synchronisation Policy</title>
  | <link rel="stylesheet" href="styles/site.css" type="text/css" />
  | <META http-equiv="Content-Type" content="text/html; charset=UTF-8">
  | </head>
  | <h2>Time Synchronisation Policy</h2>
  | <p>Original URL: <a href="https://optile.atlassian.net/wiki/spaces/SECOPS/pages/87490599/Time+Synchronisation+Policy"> Time Synchronisation Policy</a><hr>
  | <p>

<head>
        <title>Time Synchronisation Policy</title>
        <link rel="stylesheet" href="[styles/site.css](https://github.com/dernorberto/confluenceDumpWithPython/issues/styles/site.css)" type="text/css" />
        <META http-equiv="Content-Type" content="text/html; charset=UTF-8">
    </head>
    <h2>Time Synchronisation Policy</h2>
    <p>Original URL: <a href="https://optile.atlassian.net/wiki/spaces/SECOPS/pages/87490599/Time+Synchronisation+Policy"> Time Synchronisation Policy</a><hr>
    <p>```

Dump a complete site and update all internal links within the site to the local copies

Go through all .rst files and update links to pages with local ones

Once a Space or a set of .rst files have been downloaded, a script should go through them:

1. collect the page IDs (it's a field value) for ALL available .rst files
2. go through each .rst file, and for every link with the format https://optile.atlassian.net/wiki/spaces/SECOPS/pages/$pagenum ($pagenum being the page ID of the page), replace it with the local link

Adjust .rst output to match the syntax used by sphinxPageProperties

• don't use .. meta
• use field names instead
• define a naming convention for the fields

:doc_owner: Norberto Soares
:doc_title: Critical Technology Usage Policy
:doc_author: Norberto Soares
:doc_labels: child, security-policy, status-active
:last_changed: 12.04.2023
:doc_status: IN EFFECT
:doc_version: 1.8
:doc_lang: English
:conf_pagetype: reportChild
:conf_pageid: 100892722
:conf_parent: 78872710

Non-image Attachments are dowloaded but the link is not updated

example pageID: 716701697

HTML:
<a href="https://optile.atlassian.net/wiki/download/attachments/716701697/File.docx?version=1&amp;modificationDate=1540461757616&amp;cacheVersion=1&amp;api=v2" rel="nofollow">
translates to RST:
- File.docx https://optile.atlassian.net/wiki/download/attachments/716701697/File.docx?version=1&modificationDate=1540461757616&cacheVersion=1&api=v2__ (attached)

cause:

• the HTML is a simple <a href=...> without a class

Attachment files with long filenames or unsupported characters

Issue

• Error when exporting a PDF with pandoc

! Package pdftex.def Error: File `/private/var/folders/jm/76v60lk51sb3_cvn2kpzb
sch0000gr/T/tex2pdf.-f81a8d7c883e45e2/attachments/2021-07-28 14_18_57-Statuspag
e - New realtime incident and 10 more pages - Work - Microsofthspace {0pt} Edge
.png' not found: using draft setting.

See the pdftex.def package documentation for explanation.
Type  H <return>  for immediate help.
 ...

l.849 ... - Work - Microsoft\hspace{0pt} Edge.png}

• filename: 2021-07-28 14_18_57-Statuspage - New realtime incident and 10 more pages - Work - Microsoft​ Edge

Solution

• sanitise attachments, probably name all attachments based on pagename and an index nr.

Publication structure

Among the improvements, I see this one: "Create an index of the pages to use as a TOC."

I have been able to adapt the script to generate an index which includes some info about the author, date and space key, as it is generated from a space export in Confluence. However, I don't have a clue about the structure itself, what is retained to make sure the structure from an index.html could be reproduced?

Unify all possibilities into one .py file

Issue

• when updating/fixing a feature from either script (single/pageproperties/bylabel), I need to manually update the other ones.

solution

• unify all possibilities into one single .py file, reutilizing as much code as possible