microlinkhq / metascraper Goto Github PK
View Code? Open in Web Editor NEWGet unified metadata from websites using Open Graph, Microdata, RDFa, Twitter Cards, JSON-LD, HTML, and more.
Home Page: https://metascraper.js.org
License: MIT License
Get unified metadata from websites using Open Graph, Microdata, RDFa, Twitter Cards, JSON-LD, HTML, and more.
Home Page: https://metascraper.js.org
License: MIT License
Here is the full error:
web_1 | ERROR in ./~/cheerio/index.js
web_1 | Module not found: Error: Can't resolve './package' in '/src/app/node_modules/cheerio'
web_1 | @ ./~/cheerio/index.js 11:18-38
web_1 | @ ./~/metascraper/dist/index.js
trying to build with webpack.
Can you support it, global or from plugins?
<meta http-equiv="Content-Type" content="text/html; charset=windows-1251" />
Right now I think jsdom
is being included regardless. But it would be good to make it easier for certain pieces to be used without it.
Would favor canonical URLs, but fall all the way back to window.location.href
.
A bit more to think about after we nail the main API, but I was thinking that it feels weird for the bundled default rules to have such prominence, when they are for a pretty specific use case. Wondering instead if it might be nicer to make rules more of a plugin-system.
It could even maintain the same (source, [rules])
API if that's still the simplest, although maybe it becomes a constructor with new Metascraper(rules)
and then .scrapeHtml()...
instead.
This would allow bundles of rules. The current set might be something like metascraper-plugin-articles
but there could easily be a simple Open Graph one that doesn't handle fallbacks, as metascraper-plugin-open-graph
, or whatever!
cc @blakeembrey for thoughts!
any idea why some url like this one http://www.concordmonitor.com/on-new-hampshire-s-antique-alley-it-s-out-with-the-old-in-with-the-new-9234789 gives this error while calling:
Metascraper.scrapeUrl(results[j].url)
trying to run the basic example however it is logging TypeError: Cannot read property 'parent' of null
import Metascraper from 'metascraper';
Metascraper
.scrapeUrl('http://www.bloomberg.com/news/articles/2016-05-24/as-zenefits-stumbles-gusto-goes-head-on-by-selling-insurance')
.then((metadata) => {
console.log('meta: ', metadata);
})
.catch((error) => {
console.log('error: ', error);
});
Cheerio results often return empty strings when no matches occur, so it seems best to not match on that instead and keep going down the fallbacks array
It appears that there is an unintended 'a' in line 50 of the author package. In my test's this is causing poor results for the author.
strict(wrap($ => getValue(
$, $ ('[class*="author"] a')))),
Using this url, you will find that metascraper returns "Shakespeare Videos" for author
http://www.sparknotes.com/shakespeare/macbeth/quotes.html
Removing the random "a" on line 50, returns the correct author, "William Shakespeare"
strict(wrap($ => getValue(
$, $ ('[class*="author"]')))),
However, there are other URL's that only successfuly extract the Authors name when that 'a' is present. What exactly is that "a" doing?
Thanks,
Are there any reasons why application/ld+json is not being utilized? It seems to be contained in the majority of large websites and includes information such as headline, description, date, authors, publisher and more!
An example site that utilizes this format is https://www.theverge.com/2017/11/16/16667366/tesla-semi-truck-announced-price-release-date-electric-self-driving.
I can work on this if others think it would be helpful. Just wanted to reach out first to see if anybody else has already done this, or if there is a reason not too.
Thanks!
Would be cool to have, and helpful in evaluating them
i saw the #27
i did you say that, but cors error...
error: XMLHttpRequest cannot load http://www.naver.com/. Redirect from 'http://www.naver.com/' to 'https://www.naver.com/' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://www.lululala.co.kr:8000' is therefore not allowed access.
i'm a front-end developer.. so i must solve the cors in front...
please tell me... the solve
how can i do?
please.... help me.
I believe this is an issue of how metascraper use of "require()" works with webpack design. I wish to ask if you can offer any direction to fix these errors I'm getting when using metascraper with vue.js, which uses webpack. Below is something I learned about webpack by referencing an example of how webpack expects require() to work:
var module_path= "./dir/"+somevariable+".js";
var foo= require(module_path);
• require("./dir/"+somevaribale+".js") is statically analyzable, so it works with webpack
• require(module_path) is not statically analyzable, so it doesn't work with webpack.
Webpack calls this feature "require context" and its behavior is by design.
Referencing one of the warnings -- warning in ./node_modules/metascraper/src/load-rules.js
QUESTION: Is there a way to resolve this issue so metascraper can be used with webpack?
Thanks... Jim
package.json
.package.json
.PLEASE SEE THE SEVEN WARNINGS BELOW.
# npm run dev
WARNING Compiled with 7 warnings
warning in ./node_modules/keyv/src/index.js
18:14-40 Critical dependency: the request of a dependency is an expression
warning in ./node_modules/metascraper/src/load-rules.js
52:15-34 Critical dependency: the request of a dependency is an expression
warning in ./node_modules/lazy-debug-legacy/src/functions.js
62:15-40 Critical dependency: the request of a dependency is an expression
warning in ./node_modules/lazy-debug-legacy/src/functions.js
87:8-28 Critical dependency: the request of a dependency is an expression
warning in ./node_modules/css/node_modules/source-map/lib/source-map/source-map-generator.js
8:45-52 Critical dependency: require function is used in a way in which dependencies cannot be statically extracted
warning in ./node_modules/css/node_modules/source-map/lib/source-map/source-map-consumer.js
8:45-52 Critical dependency: require function is used in a way in which dependencies cannot be statically extracted
warning in ./node_modules/css/node_modules/source-map/lib/source-map/source-node.js
8:45-52 Critical dependency: require function is used in a way in which dependencies cannot be statically extracted
We get unscrapable responses trying to access YouTube video .../watchv= URLs (via a cors-proxy).
Or rather, ten times out of eleven. The content shown in browser dev tools (network) is almost all <script> and no meta tags.
In PostMan, if I clear the user-agent header, I get consistently usable data. I want to try this for metascraper, but I can't see any way to tell it to use a popsicle instance with options (i.e. custom headers) specified.
Is there any way to override popsicle defaults for metascraper? If not, it may make sense to be able to pass-through either a prebuilt popsicle object, or popsicle options. Is there quick workaround to over-ride the user-agent header when using metascraper from a browser?
EDIT:
Probably pointless, unless an expert can advise otherwise. Trying to set the user-agent header on popsicle just gives Refused to set unsafe header "user-agent"
which is presumably enforced at the browser level. I've been trying to hold off on implementing a custom back-end cors/scrape-proxy, but maybe that's the only option.
Embedly for example can be given keywords. Could you add such functionality?
Add a new property for supporting video (mp4, avi, webm) and gif embed.
og
meta tags(a lot of good examples here!)
twitter
meta tagsNot sure if twitter support property tags to target video. Need to check:
sailthru
meta tags???
We are working on a v2 iteration.
The major breaking changes in this new version are that HTTP layers are moved out of the library.
From now, metascraper will be a main function that receives HTML and URL for extracting the data:
const metascraper = require('metascraper')
const got = require('got')
const targetUrl = 'http://www.bloomberg.com/news/articles/2016-05-24/as-zenefits-stumbles-gusto-goes-head-on-by-selling-insurance'
;(async () => {
const {body: html, url} = await got(targetUrl)
const metadata = await metascraper({html, url})
console.log(metadata)
})()
Because this iteration refactor a lot of code, is not possible create a PR, so I'm going to use this issue as a tracking list of things to close before release the new version.
logo
field.3.0.0
.I'm running into issues with the image value not being the main image for metascraper-amazon. There are actually multiple .a-dyanmic-image classes on the screen as seen in the attached photo. Can we create some rules with priority over this like wrapUrl($ => $('#landingImage').attr('src'))
or wrapUrl($ => $('.a-dynamic-image').first().attr('src'))
?
package.json
.We're trying to use Metascraper in an Angular 5 application since the README.md indicates Metascraper supports browser usage. However, we get Module not found: Error: Can't resolve 'module'
in require-from-string
and resolve-from
from Angular CLI when calling Metascraper from our code:
ERROR in ./node_modules/metascraper/node_modules/require-from-string/index.js
Module not found: Error: Can't resolve 'module' in '/path-to/node_modules/metascraper/node_modules/require-from-string'
@ ./node_modules/metascraper/node_modules/require-from-string/index.js 3:13-30
@ ./node_modules/metascraper/node_modules/cosmiconfig/dist/loadRc.js
@ ./node_modules/metascraper/node_modules/cosmiconfig/dist/createExplorer.js
@ ./node_modules/metascraper/node_modules/cosmiconfig/dist/index.js
@ ./node_modules/metascraper/src/load-rules.js
@ ./node_modules/metascraper/src/index.js
@ ./clients/browser/app/library/content-linked-edit/content-linked-edit.component.ts
@ ./clients/browser/app/library/library.module.ts
@ ./clients/browser/$$_lazy_route_resource lazy
@ ./node_modules/@angular/core/esm5/core.js
@ ./clients/browser/main.ts
@ multi webpack-dev-server/client?http://0.0.0.0:0 ./clients/browser/main.ts
ERROR in ./node_modules/metascraper/node_modules/resolve-from/index.js
Module not found: Error: Can't resolve 'module' in '/path-to/node_modules/metascraper/node_modules/resolve-from'
@ ./node_modules/metascraper/node_modules/resolve-from/index.js 3:15-32
@ ./node_modules/metascraper/src/load-rules.js
@ ./node_modules/metascraper/src/index.js
@ ./clients/browser/app/library/content-linked-edit/content-linked-edit.component.ts
@ ./clients/browser/app/library/library.module.ts
@ ./clients/browser/$$_lazy_route_resource lazy
@ ./node_modules/@angular/core/esm5/core.js
@ ./clients/browser/main.ts
@ multi webpack-dev-server/client?http://0.0.0.0:0 ./clients/browser/main.ts
The following warning precedes those errors:
WARNING in ./node_modules/metascraper/src/load-rules.js
52:15-34 Critical dependency: the request of a dependency is an expression
@ ./node_modules/metascraper/src/load-rules.js
@ ./node_modules/metascraper/src/index.js
@ ./clients/browser/app/library/content-linked-edit/content-linked-edit.component.ts
@ ./clients/browser/app/library/library.module.ts
@ ./clients/browser/$$_lazy_route_resource lazy
@ ./node_modules/@angular/core/esm5/core.js
@ ./clients/browser/main.ts
@ multi webpack-dev-server/client?http://0.0.0.0:0 ./clients/browser/main.ts
Does Metascraper work in a browser environment?
This is the code in my express router:
console.log('Scraping metadata...', req.query.url);
Metascraper.scrapeUrl(req.query.url)
.then((metadata) => {
console.log(metadata);
res.send(metadata);
})
.catch((err) => {
console.log(err);
res.send(err);
});
It returns this error when trying to scrape http://facebook.com
{ [Error: incorrect header check] errno: -3, code: 'Z_DATA_ERROR' }
Any thoughts why this is happening?
Hi @ianstormtaylor! Does/can the library also return the favicon distinct from the image?
This is more of a question than an issue.
In the author's plugin,
wrap($ => $('meta[property="article:author"]').attr('content')),
is listed second in the module export. Does this mean that it has a higher priority than the other methods of extracting an author?
Because I have noticed that if "article: author" is equal to a website url, then it is not used and a method further down the list is being used to extract the author.
How is this working? I do not see any code that is checking to see if its an URL or an actual authors name.
Currently the build actually relies on the end-user having Babel installed and all of Babel core. It'd be nicer on us users if we could use it without Babel.
Node 8 is great and everybody should update ASAP, but unfortunately, it's not always that easy.
Node 6.x is still Active LTS status and some great services don't support 8.x yet.
Also, adoption isn't widespread yet:
In this, you can see the three release lines represented by solid green (Node.js 4 "Argon"), solid blue (Node.js 6 “Boron”), and solid purple (Node.js 8 “Carbon”).
Throughout the year, you can see Node.js 6 reigning supreme—peaking at 388,417 downloads on October 25th. Unlike last year with Node.js 6 and Node.js 4, the downloads of Node.js 8 didn’t surpass the downloads of the Node.js 6 LTS. This is an interesting trend that we should watch into the new year. It seems that the majority of downloads of Node.js are still Node.js 6, even though Node.js 8 has been trending upward for the past few months.
Given the data, it may be safe to assume that the delayed LTS-adoption trend will continue, and as Node.js usage continues to grow, adoption of new LTS versions could very well take longer and longer with each new LTS cycle. We’ll be able to confirm or reassess this assumption with next year’s data, though!
Source: NodeSource
I found that the library aimed to support Node 4.x and browser back in the old days by transpiling via Babel but I couldn't find a notice about dropping support to these environments on Changelog/Release Notes (Although there is a Remove browser mention
commit).
Cloud Function providers are probably going to support 8.x soon as this is The Right Thing™ but they're moving slowly.
So, would maintainers consider accepting a PR adding back Babel to enable this great library at least on 6.x runtime in the meantime or while marked as LTS?
Thanks for working on OSS projects!
Hi @ianstormtaylor, I'm not sure if this is completely out of scope for this library - if yes, apologies.
But in case, it isn't, it would be amazing to treat product pages as distinct from articles by getting product specific information from the sites (atleast the main ones have it standardized). Here is a library (though a bit outdated) I found which does some of that - https://github.com/hulihanapplications/fletcher/blob/master/lib/fletcher/models/
Thank you - your library looks great! :)
@blakeembrey pointed out that New York Times ends up perma redirecting without one! We should switch to popsicle
anyways, since i think it has built-in support.
when I try to get tags from github, and many other sites, I get a similar error to the one below:
XMLHttpRequest cannot load https://github.com/ianstormtaylor/metascraper/issues/new.
Redirect from 'https://github.com/ianstormtaylor/metascraper/issues/new' to
'https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fianstormtaylor%2Fmetascraper%2Fissues%2Fnew'
has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
Origin 'http://localhost:4201' is therefore not allowed access.
Is there anyway to ignore this? or a method of retrieving the page like a browser would?
Would be awesome have a logo for identifying the project.
Ideas are welcome 😄
Are there any examples of a custom rule set in a project? The documentation, to me, isn't clear enough. Where do I save the custom ruleset? How do I require it in the rc file?
I'm running docker + node.
XMLHttpRequest cannot load #27. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://www.lululala.co.kr:8000' is therefore not allowed access.
:8000/#/:1 Uncaught (in promise)
PopsicleError {cause: undefined, code: "EUNAVAILABLE", popsicle: Request, message: "Unable to connect to "#27"", name: "PopsicleError"…}
how should i do???
Generate a thumbail of the important part of a video based on video
property added by metascraper-video
When I try to require('metascraper')
I got this error:
/home/user/workspace/node_modules/metascraper/lib/index.js:2
let RULES = require('./rules')
^^^
SyntaxError: Block-scoped declarations (let, const, function, class) not yet supported outside strict mode
My suggestion is to add the 'use strict';
on top of each file containing a block-scoped declaration.
Hi,
I set up a small test page (with nearly no metadata) to play around with metascraper and noticed two unexpected things:
The test page can be found here: http://x.manuel-heidrich.com/tests/browse/
Short URLs for the test page: http://bit.ly/2zVanQ9 and https://goo.gl/gTYg35
HTML of the test page: http://sharedby.manuel-heidrich.com/Cs6cno/3QJaQeuE
Response from metascraper: http://sharedby.manuel-heidrich.com/WAH4q/1HE7KFYF
In case of the images I guess it would be great if in case of relative URLs imagescraper would return absolute URLs.
Concerning the URL: Is it a bug or intended behaviour? I think url should never be null.
Happy to help if I can!
There are a few different metadata syntaxes out there for tags, would return an array of them. Maybe even join across syntaxes to get the full list. And then unique with case insensitivity
It would be nice if you could specify a timeout and reject the returning promise after said timeout.
Returning the cleaned body of the article.
JSDOM is kind of heavy here. Would you consider a PR that replaces JSDOM with Cheerio instead? Since it's really only using getAttribute
and querySelector
like APIs, maybe use Cheerio for all text parsing use-cases and have a wrapper for the browser window
use-case that defers to the document.*
methods?
sometimes the meta image returns something like "//images.solecollector.com/complex/images/w_1200/viukziox1lihfkmhn2i5/off-white-air-jordan-1"
instead of "https://images.solecollector.com/complex/images/w_1200/viukziox1lihfkmhn2i5/off-white-air-jordan-1"
It'd be nicer on users if it were written in ES5 promises. Related to #7. Current implementation relies on a regenerator runtime from Babel to support async
/await
.
Hi
I would be interested in language property. These are the places I know where the language can be stored:
Thanks in advance.
In https://github.com/ianstormtaylor/metascraper/blob/master/lib/rules/author.js. Maybe a better check is /^\S+\s+\S+/
?
Hi,
It seem that the published npm packge isn't transpiled by Babel.
I think it would be simple to do, and kind of nice to allow rules to be defined with just a single function, instead of always needing to be an array. Such that as simple case could look like...
title: async (window) => window.document.title.textContent
Or the complex ones could be:
title: [
// tons of rules...
]
@blakeembrey that sound good to you?
I noticed this uses async rules, any particular reason? Seems all the current rules are completely synchronous.
FWIW, if you just want to compose async functions I wrote https://github.com/blakeembrey/throwback. It takes a chain of functions that return promises and any can either return value
or return next()
to continue execution. Might be a nice middleware pattern here if you want to keep async middleware.
I have a package published that uses metascraper, yet it doesn't seem to draw from either a metascraperrc or the metascraper section of package.json, when it's in node_modules. It works when testing in its own directory, but not as a package. Am I doing something wrong, or is this a scoping issue? It could also be that I'm using a local rule, such as ./rule.js
, which it might be looking for in the root directory of the project that has it as its dependency.
This particular scenario throws an exception:
const MY_RULES = {
title: [
myPreferredTitleRule,
myFallbackTitleRule,
mySuperLastResortTitleRule,
]
}
scrapeMetadatum does not seem to anticipate receiving fallback rules and throws an exception on line 95: var next = rule($, url); when "rule" is an array of functions instead of a function itself.
Getting this error:
ERROR in ./node_modules/cheerio/index.js
Module not found: Error: Can't resolve './package' in '....../node_modules/cheerio'
I am having some difficulties creating my own rule. For test purposes, I copied the metascraper-publisher folder and renamed it to "metascraper-test". I changed the appropriate names in the package.json file. I then added "metascraper-test" to the Default_Rules section in the "load-rules.js" file. However, I keep getting a "metascraper-test" not found error, even though it is present.
Any ideas on what could be causing this?
Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.