ellasoderberg / certification-data-scraper Goto Github PK
View Code? Open in Web Editor NEWA script to scrape and upload data from various tender databases to drive.
A script to scrape and upload data from various tender databases to drive.
Now we have to manually check how many pages the TED script should iterate trough. Would be much better to read all the numbers in the pagination element and find the highest number among these, and iterate trough that number of pages.
Sometimes it's impossible to unzip files. Causes the script to just stand still and do nothing instead of crashing. Add some kind of timeout for when this happens, so that the script can restart instead.
Build test modules to test the script. For example, implement type checking for the scraped data.
Add https://www.pabliq.se/ to the databases.
Enhance the file structure, in order to make the code maintainable and structured in a more logical way.
Add a column for tender type, to see if what is published is before, during or after the tender process.
Currently in order to start the script, we have to manually check where the scraper should stop scraping. By looking at the title of the first scraped tender from last week, we can have the scraper automatically start without intervention.
Comment everything in order to make the code more readable.
The word search for finding keywords in pdfs and text should be reviewed and improved.
Now when the script fails, we manually have to look at the log-files to see where to restart the script and find the current google docs ID. It would be better if this information is saved and read when the script is restarted, to reduce manual intervention.
Improve the logging to give more specific information about errors happening. It should be possible to just look at the log and immediately see why the script has crashed.
Currently photocopies of .pdf files are not read properly. Reading word documents can also be improved. Excel files are not read at the moment but should be read.
We will not store contact details connected to any European citizens due to GDPR. However, the contact details are part of what makes the scraping so valuable. We need to find a way to extract valuable information while still complying with GDPR.
Read more here: https://www.zyte.com/blog/web-scraping-gdpr-compliance-guide/
https://www.zyte.com/blog/gdpr-public-personal-data-update/
https://www.zyte.com/blog/solution-architecture-part-3-conducting-a-web-scraping-legal-review/
Add information from salesforce in the data delivery. Ideas:
See if it would be possible to retrieve document from TED tenders, or at least from a few of the TED tenders.
As of now, we match all the contacts that have been scraped historically to a static list to see if we have already seen them previously or not. Could be done in a better way. Potentially by connecting to the salesforce API.
Now if the script stops, it is not done gracefully (chrome driver might be running still, the cache is not saved). It would be good to be able to run a command that would run the shutdown function, and the possibility to specify what should be saved to cache (run successful or not) using flags.
For many databases, the search functions suck. It would be very useful to create our own relevance indexing by looking at for example keywords, CPV and other codes, etc. to sort tenders on relevance in the google sheet ourselves.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.