code4nova/nsfpatents

Script to pull down, parse and generate data on patents and patent applications resulting from NSF funding

nsfpatents's Issues

Users should be able to choose Reed Tech vs Google as download source

To minimize the impact of the somewhat flakey connection to the Reed Tech PTO download site, users should be able to use a command-line argument to specify download from Google instead:

processFile.rb ... google ...

or

processFile.rb ... reedtech ...

The default download site should be set to the more reliable site for most users.

Bug in get_date_fields()

partial_year, month, day = /^...(\d\d)(\d\d)(\d\d)\./.match(filename)[1] does not actually set month or day variables.

Handling of multi-line text in extracts and CSV file needs cleanup

In a previous commit, processFile.rb was peppered with #gsub calls to convert newlines in extracted text to spaces. The possibility of newlines in the extracts has two implications:

• regexps processing the text need to be aware that multi-line patterns might be needed
• the output comma-separated value (CSV) file is ill-formed for certain uses

An example of a multi-line extract is the xref field of the second grant extracted from ipg140107 (see lines 2509-2530 of ipg140107.extract), though others are likely to exist.

Comment needs to be removed

@ahirschberg You know what to do...

Add option to bulk process all files within time span (or other meaningful grouping)

It is desirable to easily process all files in a time span with a single command. Perhaps the structure of this new code will lend itself to processing files based on arbitrary criteria.