[FEATURE REQUEST] Dynamically tuning concurrency/connection limit or bailing after n connection failures

[epi052]

[FEATURE REQUEST] Dynamically tuning concurrency/connection limit or bailing after n connection failures

[epi052]

I'm not sure how this got closed, but reopening.

As always, thank you for your awesome requests and suggestions!

I like the idea of this, and I think once we get #107 merged (which allows us to cancel a scan), this ticket becomes much more manageable. Given how things are structured, I think the following will need to happen:

• track errors (discussed here)
• some configurable way of either opting-in or opting-out of auto-tuning (if opt-in, probably include a dial for number of errors or something)
• pause scanning for a handful of seconds to allow requests already in flight to complete
• limit concurrency in a way that makes sense
  • -t would require cancelling a directory and restarting it, whereas -L could just keep things paused until that directory is allowed to proceed. -L feels like the better solution, if it's effective. Would need some testing.

As I've walked myself through this, I don't think this issue is blocked by #107 necessarily. #107 only becomes a blocker of the -L solution doesn't really help/work. We won't really know if it's effective or not until it's attempted.

[epi052]

... closed automatically again ...

[epi052]

Update on this one: I've added metrics for most things I can think of. The table only shows non-zero values, the json contains everything that's tracked.

String output

──────────────────────────────────────────────────
 📊                Scan Summary                📊 
───────────────────────────┬──────────────────────
 Average Dir Scan          │      1.6992 secs     
 Longest Dir Scan          │      2.2181 secs     
 Shortest Dir Scan         │      1.0123 secs     
 Total Scan Time           │      3.2576 secs     
 403 Forbidden             │           2          
 Client Error Codes        │         1442         
 Errors                    │           2          
 Redirects                 │           6          
 Requests Expected per Dir │          206         
 Requests Sent             │         1451         
 Resources Discovered      │           9          
 Success Status Codes      │           3          
───────────────────────────┴──────────────────────

JSON output

{"type":"statistics","timeouts":0,"requests":1451,"expected_per_scan":206,"total_expected":1451,"errors":2,"successes":3,"redirects":6,"client_errors":1442,"server_errors":0,"total_scans":7,"links_extracted":0,"status_403s":2,"wildcards_filtered":0,"responses_filtered":0,"resources_discovered":9,"directory_scan_times":[1.631990827,1.603911018,1.558497619,1.684073758,1.012269852,2.218053672,2.185941124],"total_runtime":[0.0]}

Any other useful info you can think of @mzpqnxow ?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[epi052]

Update: v2.0.0 will add --rate-limit that accepts a number representing the desired number of requests per second.

Not considering this closed, as automatically detecting and adjusting rate is still a goal, however, i think the correct mechanism is now in place to tweak the scan speed.

[epi052]

progress on this issue can be seen at #212