[Feature] Trigger expansion from a 3rd party tool

[Rafiot]

[Feature] Trigger expansion from a 3rd party tool

[Rafiot]

Great, I'll have a go at it in the coming days, probably next week.

[obsidianforensics]

Hi there, this sounds great! Building in a proper API function to Unfurl has been on my list for a while, but there were a few issues (I haven't built one before and I wasn't sure what requirements a consumer of the API would have). If you wanted to build the API part, that'd be awesome and I'd be happy to help however I can. There's an existing /api/ route for the web app, but it's not a true API by any means. I'd welcome ideas/code from someone who actually knows what they're doing.

As for JSON output, getting JSON isn't hard (it's used internally and there's already a generate_json function that builds a JSON object of the nodes and edges - it's in the format need by vis.js Network, as that's what's used to display the Unfurl graph. I'm not sure what structure of JSON would be helpful for your use case, but I could tweak it or build another JSON output function if needed.

[Rafiot]

I started to play with the app, but I have a quick question before I do so: the public instance expects links that look like that: https://dfir.blog/unfurl/?url=https://www.example.com/path/index.html?a=1&b=2 when the current HEAD on the master branch goes for http://localhost:5000/https://www.example.com/path/index.html?a=1&b=2

The URL to expand in the public instance is a parameter (url), which is not the case on the current code. Is it expected? Is there a missing commit for the code on github?

(sorry, editing this message as I look at the code) an URL like that:
http://localhost:5000/?url=magnet:?xt=urn:btih:c9e15763f722f23e98a29decdfae341b98d53056&dn=Cosmos+Laundromat&tr=udp%3A%2F%2Fexplodie.org%3A6969&tr=udp%3A%2F%2Ftracker.empire-js.us%3A1337&tr=wss%3A%2F%2Ftracker.btorrent.xyz&ws=https%3A%2F%2Fwebtorrent.io%2Ftorrents%2F&xs=https%3A%2F%2Fwebtorrent.io%2Ftorrents%2Fcosmos-laundromat.torrent
will not be processed properly because of the & that are interpreted as URL params by default:

ImmutableMultiDict([('url', 'magnet:?xt=urn:btih:c9e15763f722f23e98a29decdfae341b98d53056'), ('dn', 'Cosmos Laundromat'), ('tr', 'udp://explodie.org:6969'), ('tr', 'udp://tracker.empire-js.us:1337'), ('tr', 'wss://tracker.btorrent.xyz'), ('ws', 'https://webtorrent.io/torrents/'), ('xs', 'https://webtorrent.io/torrents/cosmos-laundromat.torrent')])

The reason it works right now is because you use the referrer to extract the URL to expand. I understand it makes things easier to copy/paste a URL, but would you consider requiring an url encoded parameter instead?

[Rafiot]

okay, so here is a small PR, I figured out a way to keep the current approach to submit a URL without requiring quoting it first. Please let me know what you think and if you're ok with that, I'll implement the non-visjs API part.

[obsidianforensics]

My initial idea for Unfurl was for the user to be able to be on a site, then insert unfurl.link/ in the address bar before the current page URL, then get the Unfurled result. This wasn't as easy as I had hoped and I got it (mostly) working on the public site via redirection shenanigans. It does work on the local app (substituting localhost:port for the domain of course).

For the public instance, it's powered by GCP cloud functions, so I actually base64 encode the url parameter before sending, then decode it in the cloud function. None of this is ideal, and I'll like redo a lot of it using the new API, as that seems a much more sane way.

I hope this at least explains why it is like it is. I'm happy to make changes to support the new API and make things more streamlined.

[obsidianforensics]

I think the PR looks good 👍

[Rafiot]

Great, thanks for the feedback, I'll keep working on it in this direction