Comments (6)
@oyeanuj I already have https://github.com/blakeembrey/node-scrappy which is parsing out production information (JSON-LD and microdata) if you're interested. It just needs to be extracted from the resulting data set (Scrappy uses to two phase scrapping process - first scrapes all information, second creates snippets). Here's an example of production information from Airbnb (https://github.com/blakeembrey/node-scrappy/blob/master/test/fixtures/airbnb-ny-apartment/result.json#L62-L75).
@ianstormtaylor Sorry to cross-promote, we had this discussion a while back, I think. My goal is to extract known information from the page, while this one's was slightly different. I'd still be down to try to normalize them if possible.
Edit: Note that my goal is also only using standardised metadata for now, it's not scraping unknowns.
Edit 2: It's also parsing favicons, so you may want to replicate that logic into here - https://github.com/blakeembrey/node-scrappy/blob/master/src/rules/html.ts#L415-L421 and https://github.com/blakeembrey/node-scrappy/blob/master/src/rules/html.ts#L533-L556.
from metascraper.
Hey @oyeanuj, nice I agree!
Related to #11, I think it would be nice to have these different types of rules bundled as separate plugins, since they're very specific. And it doesn't really make sense for articles to be given so much weight over other types of content by being part of core. I just did it that way since it was my first needed use case.
If you end up hacking on a product bundled of scraping rules, I'd be down to split them out!
from metascraper.
Nice! No worries about cross-promotion at all :)
from metascraper.
Thanks 😄
FWIW, all the major product pages in the linked Ruby app seem to have decent metadata already on the page. Ran it on the current version of Scrappy and it extracted production information from them all (borderless/unfurl@612dff2) - all of them are using microdata. Someone just needs to use that microdata.
Edit: See result.json
, that's the raw extracted data before it's shrunk into a normalized snippet.
from metascraper.
@blakeembrey Very cool! I'll try to go over the commit and play around with node-scrappy soon!
from metascraper.
Please check #41 😄
from metascraper.
Related Issues (20)
- Is there a package for grabbing just the domain? HOT 1
- Wrong Typescript declared types HOT 5
- Not scrapping the correct date HOT 1
- JSDOM dependency is not compatible with node v14 HOT 7
- Missing 'logo' in Metadata interface when using metascraper-logo plugin HOT 1
- Instagram Scraping Give me null at some field HOT 1
- Crash on `metascraper-audio` HOT 4
- Incorrect metadata types HOT 4
- Specific pages lead to crash of nodejs HOT 1
- Specific pages crash NodeJS with the usage of metascraper-audio or metascraper-video HOT 3
- New version causes breaks in Definitely Typed packages HOT 9
- Cannot use in Defer because of re2 HOT 1
- Add types for metascraper packages
- metascraper-date typing errors HOT 2
- [metascraper-clearbit] Typescript build error from `keyvOpts` HOT 1
- Any issues with a memory leak, particularly metascraperDescription() HOT 1
- High memory usage HOT 9
- `author` is not recognized for YouTube HOT 1
- Make JSDOM opt-in HOT 4
- [metascraper-image] ability to pick desired image HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from metascraper.