A text extraction node module.
- HTML, HTM
- Markdown
- XML, XSL
- DOC, DOCX
- ODT, OTT (experimental, feedback needed!)
- RTF
- XLS, XLSX, XLSB, XLSM, XLTX
- CSV
- ODS, OTS
- PPTX, POTX
- ODP, OTP
- ODG, OTG
- PNG, JPG, GIF
- DXF
application/javascript
- All
text/*
mime-types.
In almost all cases above, what textract cares about is the mime type. So .html
and .htm
, both possessing the same mime type, will be extracted. Other extensions that share mime types with those above should also extract successfully. For example, application/vnd.ms-excel
is the mime type for .xls
, but also for 5 other file types.
Does textract not extract from files of the type you need? Add an issue or submit a pull request. It many cases textract is already capable, it is just not paying attention to the mime type you may be interested in.
npm install textract
Note, if any of the requirements below are missing, textract will run and extract all files for types it is capable. Not having these items installed does not prevent you from using textract, it just prevents you from extracting those specific files.
PDF
extraction requirespdftotext
be installed, linkDOC
,RTF
extraction requirescatdoc
be installed, link, unless on OSX in which case textutil (installed by default) is used.PNG
,JPG
andGIF
requiretesseract
to be available, link. Images need to be pretty clear, high DPI and made almost entirely of just text fortesseract
to be able to accurately extract the text.DXF
extraction requiresdrawingtotext
be available, link
Configuration can be passed into textract. The following configuration options are available
preserveLineBreaks
: When using the command line this is set totrue
to preserve stdout readability. When using the library via node this is set tofalse
. Pass this in astrue
and textract will not strip any line breaks.exec
: Some extractors (dxf) use node'sexec
functionality. This setting allows for providing config toexec
execution. One reason you might want to provide this config is if you are dealing with very large files. You might want to increase theexec
maxBuffer
setting.[ext].exec
: Each extractor can take specific exec config. Keep in mind many extractors are responsible for extracting multiple types, so, for instance, theodt
extractor is what you would configure forodt
andodg
/odt
etc. Check the extractors to see which you want to specifically configure. At the bottom of each is a list oftypes
for which the extractor is responsible.tesseract.lang
: A pass-through to tesseract allowing for setting of language for extraction. ex:{ tesseract: { lang:"chi_sim" } }
pdftotextOptions
: This is a proxy options object to the library textract uses for pdf extraction: pdf-text-extract. IMPORTANT: textract modifies the pdf-text-extractlayout
default so that, instead oflayout: layout
, it useslayout:raw
. It is not suggested you modify this without understanding what trouble that might get you in. See this GH issue for why textract overrides that library's default.
To use this configuration at the command line, prefix each open with a --
.
Ex: textract image.png --tesseract.lang=deu
If textract is installed gloablly, via npm install -g textract
, then the following command will write the extracted text to the console for a file on the file system.
$ textract pathToFile
Configuration flags can be passed into textract via the command line.
textract pathToFile --preserveLineBreaks false
Parameters like exec.maxBuffer
can be passed as you'd expect.
textract pathToFile --exec.maxBuffer 500000
And multiple flags can be used together.
textract pathToFile --preserveLineBreaks false --exec.maxBuffer 500000
var textract = require('textract');
There are several ways to extract text. For all methods, the extracted text and an error object are passed to a callback.
error
will contain informative text about why the extraction failed. If textract does not currently extract files of the type provided, a typeNotFound
flag will be tossed on the error object.
textract.fromFileWithPath(filePath, function( error, text ) {})
textract.fromFileWithPath(filePath, config, function( error, text ) {})
textract.fromFileWithMimeAndPath(type, filePath, function( error, text ) {})
textract.fromFileWithMimeAndPath(type, filePath, config, function( error, text ) {})
textract.fromBufferWithMime(type, buffer, function( error, text ) {})
textract.fromBufferWithMime(type, buffer, config, function( error, text ) {})
textract.fromBufferWithName(name, buffer, function( error, text ) {})
textract.fromBufferWithName(name, buffer, config, function( error, text ) {})
textract.fromUrl(url, function( error, text ) {})
textract.fromUrl(url, config, function( error, text ) {})
- #68. PR captured unzip errors.
- #66. textract will no longer put the info text to stdout about the extractors not being available or installed correctly. Instead, if you attempt to use a supported extractor that did not initialize correctly, you will get an updated error message indicating that the type is supported by textract but that external dependencies were not located. As part of this update, error messages were updated a bit to list both the type and the file.
- #65. Fixed issue where for
.odt
and.docx
files with varying non-Latin characters (ex: cyrillic) were being stripped entirely of their content.
- #63. PR added support for CSV.
- #53. Cleared up documentation around CLI and line breaks.
- #54. PR removed
disableCatdocWordWrap
as an option, instead always disabling catdoc's word wrapping. - #55. PR removed clobbering of non-boolean flags on CLI.
- #52. PR fixed CLI post big API changes.
- #51. Fixed issue with large files using unzip returning blank string.
- #49 Updated messages when extractors are not available to be purely informational, since textract will work just fine without some of its extractors.
- #50. Updated way in which catdoc was detected to not rely on file being test extracted.
- Overhaul of interface. To simplify the code, the original
textract
function was broken intotextract.fromFileWithPath
andtextract.fromFileWithMimeAndPath
. - #41. Added support for pulling files from a URL.
- #40. Added support for extracting text from a node
Buffer
. This prevents you from having to write the file to disk first. textract does have to write the file to disk itself, but because it is a textract requirement that files be on disk textract should be able to take care of that for you. Two new functions,textract.fromBufferWithName
andtextract.fromBufferWithMime
have been added. textract needs to either know the file name or the mime type to extract a buffer. - Added entity decoding, so encoded items like
<
,>
,"
,'
, and&
will show up appropriately in the text. - Removed external dependency on
unzip
- #38. Added markdown support.
- #31. Added initial ODT support. Feedback needed if there is any trouble. Also added OTT support.
- Added support for ODS, OTS.
- Added support for XML, XSL.
- Added support for POTX.
- Added support for XLTX, XLTS.
- Added support for ODG, OTG.
- Added support for ODP, OTP.
- Pull Request #39 added support for not work wrapping with catdoc.
- #30, #34. The command line has been improved, allowing for all the configuration options to be provided.
- Updated character stripping regex to be more lenient.
- Added HTML extraction.
- Added ability for extractors to register for specific extensions (not yet used). This handles cases where extensions (like
.webarchive
) do not have recognized mime types.
- Addressed some lingering regex issues from previous release.
- Added tests for RTF, more tests for DOC
- #29 Introduced new extractor for
.doc
and.rtf
for OSX only. All non-OSX operating systems will continue to usecatdoc
. Going forward, because of issues gettingcatdoc
installed on OSX, on OSX onlytextutil
will be used.textutil
comes default installed with OSX.
- #29 which resulted in the following changes:
- writing info messages to
stderr
when extractors taking awhile to get going - no longer removing โฆ
- centralized some cleansing regexes, also no longer removing multiple back to back spaces using
\s
as it was removing any back to back newlines. Now scoping back to back replacing to[\t\v\u00A0]
.
- #27, addressed issues with page ordering in
pptx
extraction.
- #25, added language support for tesseract, see
tesseract.lang
property. - Updated regex that strips bad characters to not strip (some) chinese characters. The regex will likely need updating by someonw more familiar with Chinese. =)
- #26, using
os.tmpdir()
rather than a temp dir inside textract. - Upgraded to latest
j
(dependency) - Removed
macProcessGif
option and tests as tesseract seems to work on Mac just fine now
- #21, #22, Now using j via its binaries rather than using it via node. This makes XLS/X extraction slower, but reduces memory consumption of textract signifcantly.
- Updated pdf-text-extract to latest, fixes #20.
- Addressed path escaping issues with tesseract, fixes [#18] (dbashford#18)
- Using j to handle
xls
andxlsx
, this removes the requirement on thexls2csv
binary. - j also supports
xlsb
andxlsm