rocktimsaikia/meta-fetcher

Simple metadata scrapper for node.js

meta-fetcher

Simple metadata scrapper.

Installation

yarn add meta-fetcher

Usage

import metaFetcher from 'meta-fetcher';

const result = await metaFetcher('https://hoppscotch.io/');

console.log(result);

Output:

{
  "title": "Hoppscotch - Open source API development ecosystem",
  "description": "Helps you create requests faster, saving precious time on development.",
  "image": "https://hoppscotch.io/og.png",
  "url": "https://hoppscotch.io/",
  "siteName": "Hoppscotch",
  "type": "website"
}

API

metaFetcher(input)

Takes one url string as a parameter and returns an object containing the meta-information.

input

type: string
default: 'none'

The url string to be scrapped.

Related

page-scrapper: Node.js scrapper that pulls out all links and images of a given site.

License

2024 MIT © Rocktim Saikia

meta-fetcher's Issues

change the meta tag name

Hi,
First of all, thank you for this npm package. It actually helps me a lot. While I was working on a project, I have noticed it has a tiny issue when I tried to get og:image from head. Upon researching, I have found that you are trying to find 'meta[name="og:image"]' where I think you should find 'meta[property="og:image"]'.

Add support for default export in the next major release

Simplify the result

Merge the opengraph and opengraph-social properties together into one field as they both come under the same category.

fetch does not wait for JS to execute

some websites will have their meta tags populated by JS. Since we are fetching the HTML contents of the page before JS has been loaded, the meta tags will never be filled

eg. the results of fetching meta tags for https://twitter.com/ManjaroLinux will all be empty strings because the meta tags are loaded by the JS

a way to solve this problem would be to use somthing like Selenium or Puppeteer to wait for the page to load before getting the HTML contents.

This is a major blocker since a lot of websites now use client side rendering and this library does not work in those cases.

Move to Github actions for CI

Currently, we are using Travis for CI integration but it would be nice to move to Github actions now.
PRs are welcome for this one 👍🏽

Replace jest with ava for testing

Even tho I love ava's simplicity. Ava is still not widely used for ts project. I am still experimenting and putting together a boilerplate for future use. Once ready, I will use Ava in this project.