Piping to hakrawler issue

[hakluke]

Piping to hakrawler issue

[hakluke]

Hey DeC1pher, This was a bug in a previous version. Make sure you are using the absolute latest version. To check which version you're using run 'hakrawler -v'. The latest right now is beta 7. In order to update, do this: ``` rm ~/go/bin/hakrawler rm -rf ~/go/src/github.com/hakluke/hakrawler go get github.com/hakluke/hakrawler ``` Let me know whether that works for you or not!

…

On Mon, Feb 17, 2020 at 8:39 PM DeC1pher ***@***.***> wrote: Hello, when I use hakrawler without piping it works flawless. But when I pipe it with for example: echo "domain.com" | hakrawler cat domains.txt | hakrawler it will run forever. Basically it will output everything but wont stop. You can see the process running but no there is no more output. So I would like to run it as follow in my bash script assetfinder $domain | sort -u | hakrawler -usewayback -plain but with that issue its not possible Thank you! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#41?email_source=notifications&email_token=ADKT6Y6UO6SN3PJXCUDE73LRDJSOZA5CNFSM4KWO5NT2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IN7VFWQ>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADKT6YYJVJG4ULTRZWQUYADRDJSOZANCNFSM4KWO5NTQ> .

[DeC1pher]

thanks for your fast reply, and sorry for my mistake running an old version.
I forgott the -u parameter, that the binary is getting updated.
Works now! Thank you!

[hakluke]

Great news :)

[Emoe]

Hi @hakluke,

thanks for this project.

In your recent beta8, there is another problem when piping input to hakrawler.

If you use beta8 and use the output of a tool like assetfinder, hakrawler doesn't show any results.

If you remove your last commit in line 33 in pkg/collector/collector.go and build it again, piping works.

I hope this helps fixing this issue.

[hakluke]

Hey @Emoe!
The most likely reason for this is that I have reduced the scope that hakrawler crawls by default to the exact domain that you specify.

For example, if you specify "google.com", you will get results from google.com, but not www.google.com.

I can see how this is likely to cause confusion, but I think it's safer to keep it this way so that hakrawler can safely be used without unknowingly creeping out of scope.

Having said all this, I have noticed a bug where hakrawler produces different results when piping into it instead of specifying a -url option. I am having trouble figuring out why this is the case but I'm hoping to fix it soon.