andrewstanton94/fef

For-each Files: A framework for file processing

For-each Files (fef)

A framework for file processing.

It should:

1. Read HTML | CSV | XLSX files
2. Use user given functions to process the data
3. Save it to a convenient format

This started by dogfooding a couple of tasks. I want to refactor it for wider use.

fef's Issues

Promisify

CSV parser

Conversions

• CSV -> JSON
• JSON -> CSV

Data

• Input, Output
  • Where from | to
  • Type [passed live, data key]
• Configs

Write local file (localPath, data|dataKey) => Promise(success message | error message)

Use for

• Reading arbitrary files
• Exporting results
• Exporting debug info

Returns

Success or error object

Could fail on

• Folders not existing
• File & folders not readable (permissions)
• File too big

Read local files (localPath) => Promise(dataRead | error message)

Use for

• Reading arbitrary files
• Loading files to the current class instance

Returns

• the requested data
• OR an error object

Side effects

If a data key is given and used successfully, the data will be saved under that key in the data property.

Could fail on

• File not existing
• File not readable (permissions)
• File too big

Execution flow

Steps required

• Read file
• Process input
• Filter input (as chunk)
• Process loop
  • Filter input (as item)
  • Process item
  • Filter output (as item)
• Filter output (as chunk)
• Convert to export format
• Write to file