thatnerduknow/financial-disclosures

Twitter Bot Tracking every Congress members financial disclosure documents freely available through the STOCK act. Follow @CongressionalFD

financial-disclosures's Issues

First run tries to open thousands of instances of imagemagick

This is causes by trying to run downloadAndProcessPDF() within records.forEach(). the way it should be done should look something like this

const arr = [1, 2, 3];

await arr.reduce(async (memo, i) => {
	await memo;
	await sleep(10 - i);
	console.log(i);
}, undefined);

// 1
// 2
// 3

console.log("Finished async");
// Finished async

This should only allow one call to downloadAndProcessPDF() at once

Need better tests

The current test suite relies too much on side effects. For example, the test suite for downloadAndProcessPDF() is dependent on both an HTTP request and im.convert() which produces side effects to the file system and is dependent on a working installation of imagemagick and ghostscript which we can't expect a runner like github actions to have in place. methods that have side effects need to be mocked in the tests at the very least with a stub

Implement MongoDB connection instead of using a JSON file to keep track of records

I really wanted to use mongodb from the start but my personal server doesn't support AVX instructions and I was in a hurry so I bodged together a solution backed by a JSON file located in the project directory. I won't be able to host application data on my home server however the pricing for mongodb cloud looks to be fairly sane and considering the amount of data we're talking about here it would be worth implementing it since I have more faith in typeORM than my solution 😄