otiai10/opengraph

Open Graph Parser for Go

Open Graph Parser for Golang

Go implementation of https://ogp.me/

Code Example

package main

import (
	"fmt"
	"github.com/otiai10/opengraph/v2"
)

func main() {
	ogp, err := opengraph.Fetch("https://github.com/")
	fmt.Println(ogp, err)
}

You can try CLI as a working example

% go get github.com/otiai10/opengraph/ogp
% ogp --help
% ogp -A otiai10.com

Just for fun 😉

Advanced usage

Set an option for fetching:

intent := opengraph.Intent{
	Context:     ctx,
	HTTPClient:  client,
	Strict:      true,
	TrustedTags: []string{"meta", "title"},
}
ogp, err := opengraph.Fetch("https://ogp.me", intent)

Use any io.Reader as a data source:

f, _ := os.Open("my_test.html")
defer f.Close()
ogp := &opengraph.OpenGraph{}
err := ogp.Parse(f)

or if you already have parsed *html.Node:

err := ogp.Walk(node)

Do you wanna make absolute URLs?:

ogp.Image[0].URL // /logo.png
ogp.ToAbs()
ogp.Image[0].URL // https://ogp.me/logo.png

Issues

• https://github.com/otiai10/opengraph/issues

opengraph's Issues

Only parse meta tags?

@mbleigh @otiai10
Hey, I went through the implementation and seems like the fetch method parses title meta and link elements to create the opengraph object. Any way I could just check for meta tags?

opengraph.Fetch returns nothing for a few domains

I have been using this package to fetch opengraph info about websites and articles, but for a few websites, e.g. FastCompany, the Fetch() method returns nothing. After some research, I found that few websites block bots from scraping their content. However, when I try Raycast preview, or even macOS preview, it successfully fetches the metadata with the image and title. How can I achieve that? Here's how my code looks:

package api

import (
	"net/http"
	"mypackage/read-it-later/structs"
	"mypackage/read-it-later/utils"

	"github.com/gin-gonic/gin"
	"github.com/oklog/ulid/v2"
	"github.com/otiai10/opengraph"
)

func StoreEntity(c *gin.Context) {
	var requestBody structs.RequestURL
	if err := c.BindJSON(&requestBody); err != nil {
		return
	}

	c.Header("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36")

	ogp, err := opengraph.Fetch(requestBody.URL)

	if err != nil {
		c.JSON(http.StatusInternalServerError, err)
	}

	c.JSON(http.StatusOK, gin.H{
		"id":          ulid.Make(),
		"description": ogp.Description,
		"favicon":     ogp.Favicon,
		"image":       utils.FetchImageURL(ogp.Image, ogp.Favicon, ogp.URL.Host, ogp.URL.Scheme),
		"siteName":    ogp.SiteName,
		"title":       ogp.Title,
		"type":        ogp.Type,
		"URL":         ogp.URL,
		"all_info":    ogp,
	})
}

Retrieval of Twitter card information

Hi, I was just wondering if it's possible to retrieve Twitter card information such as in the following example html, or if it's a feature you would consider adding (or allow me to submit a PR for)?
Thanks

opengraph/test/html/02_haisai.html

Lines 15 to 16 in ac0f3d5

	<meta name="twitter:card" content="summary"><meta name="twitter:title" content="">
	<meta name="twitter:description" content="All Genre Music Party">

Expose Expose function

Sometimes I want to just call the walk function when I already parsed the html. But it is not possible for now.

It would be good if Walk is expose

[Feature Request] User-Agent Option

Hello OCHIAI,
@otiai10

Thank you for very nice app. This post is about feature request and who will code it ?

[ Why User-Agent Option ? ]
Some web sites are rendering OGP Tags Dynamically with javascript.
But javascript don't work in simple crawler like twitter bot or this golang app.

So, many web sites are rendering OGP Meta tag on server side.
Popular way is SSR(Node.js), But more easier way is "Prerendering with Netlify"

But Netlify check User-Agent for prerendering. So this app can't get OGP meta tags.
[Common Issue] Understanding and debugging prerendering
(I think potentially other sites check User-Agent too, for reduce rendering cost)

[ Who code it ? ]
I'm not professional. But If you are wellcome pull request about this feature, I want to write code.
But if you don't want to merge this feature or you want to code your ownself, I don't send pull request.
So, please tell me your feelings.

Thanks,

og.url is not mapped

@otiai10
In the current implementation, opengraph.URL defaults to the parsed request URL and doesn't maps the og.url meta tag.
Is it okay to add a new field called Value in opengraph.URL struct which maps the og.url meta tag value?

Or if I am missing something here, please let me know. I want fetch the og.url meta tag from the parsed HTML.
Thank you.

CI failing

https://travis-ci.org/github/otiai10/opengraph

Video tag does not seems to be parsed

Video tag does not seems to be parsed. I will try to implement a base version of it with test

Fetching `og:description` defaults to <meta name="description"> if not found

@otiai10
Fetching og:description defaults to <meta name="description"> if not found.
Current implementation has this case m.IsDescription() && og.Description == "", is it a spec or bug?

If its a spec, do you think we can avoid that somehow or make the fallback mechanism optional, if og:description is not found? here

Thank you!