matt-dray/wordup

:page_with_curl::speaking_head: R package: Word to Govspeak [WIP]

{wordup}

Purpose

Convert a Microsoft Word document (docx) to Govspeak Markdown, which is the format required to publish on GOV.UK.

This package is a personal project for learning purposes. It may never be finished and has no guarantees.

You may also be interested in a demo Shinylive app for converting tables to Govspeak format.

Motivation and scope

Producers of statistical publications might draft their reports in docx format. Publishers need to convert these to Govspeak Markdown so they can be uploaded to the Government’s publishing platform. This can be time consuming to do manually and there is potential for error. There’s a Govspeak converter tool, but there’s some missing functionality, such as handling images and tables.

Installation

The package is available from GitHub. It isn’t stable.

install.packages("remotes")  # if not yet installed
remotes::install_github("matt-dray/wordup")

Here’s the list structure of an example word/document.xml from an unzipped docx file.

library(wordup)
path <- system.file("examples/simple.docx", package = "wordup")
body_list <- wu_read(path)
str(body_list, give.attr = FALSE, max.level = 3)
# List of 1
#  $ document:List of 1
#   ..$ body:List of 15
#   .. ..$ p     :List of 2
#   .. ..$ p     :List of 3
#   .. ..$ p     :List of 5
#   .. ..$ p     :List of 3
#   .. ..$ p     :List of 1
#   .. ..$ p     :List of 2
#   .. ..$ p     :List of 2
#   .. ..$ p     :List of 3
#   .. ..$ p     :List of 1
#   .. ..$ tbl   :List of 5
#   .. ..$ p     :List of 3
#   .. ..$ p     :List of 4
#   .. ..$ p     :List of 1
#   .. ..$ p     :List of 1
#   .. ..$ sectPr:List of 4

We can delve into this structure to retrieve text, etc.

body_list$document$body[[5]]
# $r
# $r$t
# $r$t[[1]]
# [1] "This is some more text. This time with bullets:"
# 
# 
# 
# attr(,"paraId")
# [1] "2460A889"
# attr(,"textId")
# [1] "3361B78B"
# attr(,"rsidR")
# [1] "007D458C"
# attr(,"rsidRDefault")
# [1] "007D458C"

Hopefully {wordup} will get functions to grab text and styles, apply Govspeak styling and then output a Markdown file. It could also convert images to SVG and insert anchor links for them.

Related

There are existing packages that can convert R Markdown to Govspeak:

• {govspeaker} (archived)
• {mojgovspeak}
• {rgovspeak}

For handling Word documents in R:

• {officer} can read a Word document to a tidy table
• {docxtractr} can extract tables, comments and other things

wordup's Issues

Warn user if styling isn't GOV.UK compliant

A user might provide styles that aren't meant to be used on GOV.UK, like text in italics or underlined. We could (a) warn the user in the console and (b) log the issues for later review.

Attempt to extract acronyms

You can add 'hoverovers' for acronyms in Govspeak.

We might be able to extract some of the more obvious acronyms from report text. We can assume these are generally runs of capital letters (though not always). Could extract the words in the brackets that follow an acronym, which are probably the expanded text.

So the text 'An example acronym is ABC (Always Be Closing)' would yield *[ABC]: Always Be Closing.

Build off of {officer}

Use officer::docx_summary() to get the basics out, like the text and the doc styles. Then we can supply the appropriate markdown.

library(officer)
path <- r"{C:\Users\matt.dray\Documents\Temp\demo.docx}"
doc <- read_docx(path)
content <- docx_summary(doc)

markup_map <- c(
  "Title"     = "#",
  "heading 1" = "##",
  "heading 2" = "###",
  "heading 3" = "####"
)

content$markup <- markup_map[content$style_name]

content$govspeak <- ifelse(
  !is.na(content$markup),
  paste(content$markup, content$text),
  content$text
)

content$govspeak
# [1] "# This is the title"                                                        
# [2] "## This is a heading 1"                                                     
# [3] "This is some text. This is in bold. This is in italics. This is underlined."

But the font styles are missing from the {officer} output (unless there's a way to turn on this functionality?). So might have to dig into the unzipped docx file with e.g. wordup::wu_read(). We can then grab the styles, even if it is a bit awkward (remembering that some styles aren't accepted on GOV.UK, see #7).

body_list <- wordup::wu_read(path)
str(body_list, give.attr = FALSE, max.level = 3)

lapply(
  1:6,
  function(x) {
    list(
      text = body_list$document$body[[3]][[x]][["t"]][[1]],
      style = names(body_list$document$body[[3]][[x]][["rPr"]])[[1]]
    )
  }
)
# [[1]]
# [[1]]$text
# [1] "This is some text. "
# 
# [[1]]$style
# NULL
# 
# 
# [[2]]
# [[2]]$text
# [1] "This is in bold."
# 
# [[2]]$style
# [1] "b"
# 
# 
# [[3]]
# [[3]]$text
# [1] " "
# 
# [[3]]$style
# NULL
# 
# 
# [[4]]
# [[4]]$text
# [1] "This is in italics."
# 
# [[4]]$style
# [1] "i"
# 
# 
# [[5]]
# [[5]]$text
# [1] " "
# 
# [[5]]$style
# NULL
# 
# 
# [[6]]
# [[6]]$text
# [1] "This is underlined."
# 
# [[6]]$style
# [1] "u"

It's then a case of pasting like **This is bold**, since the text was recognised as 'b' style (italics was 'i', underlined was 'u').

Allow for a normal data.frame to be converted to Govspeak

Would be nice to be able to convert from a data.frame to Govspeak to allow for extraction and conversion of all tables found in a Word doc, as per matt-dray/govspeakify-tables#2.

library(officer)

doc_1 <- officer::read_docx() |>
  body_add_par("Hello world!", style = "heading 1") |>
  body_add_par("", style = "Normal") |>
  body_add_table(airquality, style = "table_template") |> 
  body_add_par("Hello world!", style = "heading 1") |>
  body_add_par("", style = "Normal") |>
  body_add_table(iris, style = "table_template") |> 
  body_add_par("Hello world!", style = "heading 1") |>
  body_add_par("", style = "Normal") |>
  body_add_table(mtcars, style = "table_template")

print(doc_1, target = "~/Desktop/example_1.docx")

tables <- docxtractr::read_docx("~/Desktop/example_1.docx") |> 
  docxtractr::docx_extract_all_tbls()

lapply(tables, wordup::table_to_govspeak)  # errors, doesn't expect a df

Consider an RStudio Addin

table_to_govspeak() is a pretty interactive function anyway, so might as well have an addin.

Consider Pandoc

Of course, we could just go from docx to md using Pandoc. You can do that in R like:

rmarkdown::pandoc_convert(
  docx_path,
  to = "markdown",
  output = "output.md", 
  options = c("--extract-media=.")  # pull images out (but not charts?)
)

But Pandoc doesn't support Govspeak directly.

We want more control over bespoke stuff like table formatting and anchor links for images (#8), for example. We also might want to warn users (#7) when they use inappropriate markup (e.g. italics shouldn't be used). Govspeak is quite simple, so converting the unzipped docx to an md with our limited markup requirements shouldn't be too tedious.

Check that final Govspeak is compliant/correct

For example, check for correct formatting of:

• {:#figure1} has a matching [](#figure1)
• []() external links
• [Table 1](#table1)
• check for unexpected spacing (This sentence has an extra space at the end . There's two spaces before this one.)
• Etc

Insert image anchors and link to them

Will have to inset the appropriate image anchor, with the correct name for that image. Then will have to find instances in the body text that need t be matched to that image (could regex for 'table 1' and then surround in the appropriate Markdown to be linked to the image).

Create standalone table converter

Add an exported, standalone function that converts a pasted Word table to Markdown.

Use strsplit() on newlines and tabs for manipulating rows/cells. type.convert() to get appropriate data classes (so numeric columns will get ---: Markdown). Will need to use tricks to infer numbers in some columns (i.e. to identify 1,234% as 1234, and to ignore placeholder symbols in otherwise numeric columns).

This could be used in a Shiny app with some widgets for more fine-grained control.

Create standalone XML to SVG conversion for charts

We can unzip a docx to get /word/charts/chart1.xml, etc. Is there a known method to convert these to SVG?

Simplify regex to ignore when guessing types

So ignore_regex = "[[:punct:]]|[[:space:]]" is simpler, more readable and will do a better job of capturing things like 12 (34.5%) as well as 1,234 and "12%".