Add associated JS file to linkfinder output

[hakluke]

Add associated JS file to linkfinder output

[hakluke]

Hey @jaikishantulswani,
I know this is a real pain. I have been discussing it with @lc. This is quite a tricky problem to solve because the links in a JavaScript file may not be referring to the same domain that the JS file is hosted on.

So, in your example above, imagine that the JavaScript at https://example.com/1.js looks something like this:

var baseURL = "https://api.example.com"
var url = baseURL + "/add/user"

In this case, hakrawler (currently) would output the following two urls:

[linkfinder] https://api.example.com
[linkfinder] /add/user

In order to make the connection that the relative URL "/add/user" is actually https://api.example.com/add/user and not https://example.com/add/user, hakrawler would need to parse the JavaScript. The best way that I know of to parse JavaScript for a terminal application is by utilising a headless browser such as chromium, however, this is extremely processor intensive and slow. One of the core concepts that I want to keep with hakrawler is it's speed and scalability, so this is not an option. At least, not by default. Furthermore, even if we were using a headless browser, it doesn't necessarily mean that a crawler would discover every endpoint in the JS file. Only the ones that become visible within the application.

I think we have two options to resolve this:

1. We compromise, and just have special output for linkfinder output which prints relative URLs along with the JS file they were found in, so that they can be manually analysed later. Example output could look something like this:

[linkfinder] /add/user from https://example.com/1.js

2. We implement full headless browser crawling as a non-default option in hakrawler which will enable more comprehensive crawls with full support for SPAs. If someone wants comprehensive crawling with a headless browser they can enable it, otherwise the current configuration is used.

My personal preference would be to go for option 1, I think that a headless browser crawling could be saved for a different tool.

[JeffreyShran]

Option 1 seems the most logical approach. Perhaps with a "best guess" of the URL presented along with the currently proposed text. i.e:

[linkfinder] https://api.example.com/add/user - /add/user from https://example.com/1.js

I know that's fairly simple to write the code for in the example given, but I'm sure it's going to be impossible to catch all scenarios.

[hakluke]

Hey guys, this has been resolved in beta 6. I ended up just going with

[linkfinder] /add/user from https://example.com/1.js

@lc thanks again for the linkfinder functionality