Comments (9)
The job is killed with following logs:
3 tasks left
3 tasks left
3 tasks left
3 tasks left
3 tasks left
('Connection aborted.', OSError(0, 'Error')) on 147.75.84.47:443
2 tasks left
2 tasks left
2 tasks left
2 tasks left
2 tasks left
2 tasks left
2 tasks left
2 tasks left
2 tasks left
2 tasks left
('Connection aborted.', OSError(0, 'Error')) on 147.75.84.193:8080
final hook is hanging
1 tasks left
final hook is hanging
1 tasks left
final hook is hanging
1 tasks left
final hook is hanging
1 tasks left
final hook is hanging
1 tasks left
I wonder there is some timeout missing for this last task... Just need to figure out a way to reproduce it, probably patch kube-hunter to figure out which task it is and then look into that.
CC @surajssd
from lokomotive-kubernetes.
The last task eventually finished with following result, when I tried to reproduce it:
1 tasks left
final hook is hanging
1 tasks left
('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) on 147.75.32.35:6443
Starting new HTTPS connection (1): 147.75.32.35:6443
HTTPSConnectionPool(host='147.75.32.35', port=6443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f44a43355d0>: Failed to establish a new connection: [Errno 111] Connection refused')) on 147.75.32.35:6443
Event <class 'src.core.events.types.common.HuntFinished'> got published with <src.core.events.types.common.HuntFinished object at 0x7f44a4ae5110>
from lokomotive-kubernetes.
It seems that kube-hunter
scans /24
of obtained public IP of the pod (for outgoing traffic), looking for API server. That might be detected as an abuse by some IaaS providers (e.g. Hetzner). And that seems to be finding some false-positives (perhaps other clusters?). In combination with --active
flag, it may attack other clusters then...
Also the kube-hunter runtime doesn't seem to be deterministic:
- Cluster nodes: 2, Runtime: 4m54s
- Cluster nodes: 3, Runtime: 5m59s
- Cluster nodes: 3, Runtime: 89s
- Cluster nodes: 3, Runtime: 2m6s
- Cluster nodes: 2, Runtime: 7m5s
from lokomotive-kubernetes.
Seems that some servers which kube-hunter
tries to probe takes really long time to respond:
130 ✗ (1.270s) 11:29:15 invidian@dellxps15mateusz ~/repos/kinvolk/kube-hunter (master)$ curl -v -s -k https://147.75.32.35:6443
* Trying 147.75.32.35:6443...
* TCP_NODELAY set
* Connected to 147.75.32.35 (147.75.32.35) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* Operation timed out after 300495 milliseconds with 0 out of 0 bytes received
* Closing connection 0
28 ✗ (5m0s) 11:34:17 invidian@dellxps15mateusz ~/repos/kinvolk/kube-hunter (master)*$
I think HTTP probe should timeout earlier than 5 minutes...
from lokomotive-kubernetes.
Do you think we are missing any pre requisite checks that we should be doing before installing?
from lokomotive-kubernetes.
Do you think we are missing any pre requisite checks that we should be doing before installing?
Can you elaborate? What checks do you have in mind for example? I'm not sure if I understand.
from lokomotive-kubernetes.
Created following issues in upstream:
And one PR:
I also tested, that when added timeout to the discovery, then kube-hunter
runs are much faster. I'll do one more round of testing, and my suggestion would be to use patched version of kube-hunter
image until the issue is not solved upstream.
from lokomotive-kubernetes.
Can you elaborate? What checks do you have in mind for example? I'm not sure if I understand.
Before we deploy kube-hunter we do following checks (not extensive) but to make sure cluster is responsive:
I meant do we need to add anything more here?
from lokomotive-kubernetes.
I meant do we need to add anything more here?
No, I think those checks are fine. I believe the issue is in kube-hunter
itself, as described above.
from lokomotive-kubernetes.
Related Issues (20)
- packet: Investigate slowness when deploying big CLC snippets
- Conformance tests are failing on 1.17 HOT 4
- Packet: Feature Request: Consider enabling multipathd by default HOT 8
- Packet: Empty CLC causes silent failure HOT 4
- Align and set proper provider constraints across all modules
- Linkerd installation broken - no apiserver client ca file HOT 6
- docs/conformance are out of date
- (packet?) Can't run terraform plan/apply HOT 4
- [Packet] Automatic node_private_cidr HOT 9
- [Packet] sig-storage-local-static-provisioner recently stopped working HOT 5
- Adding more controlplane nodes fails on Packet
- Adding/removing controller nodes to existing cluster does not work
- Use of Cluster API
- Lokomotive Disk type specific RAID Setup optimizations HOT 1
- Document RAID-related vars
- packet: Mounts inside /mnt/ can't be used by pods HOT 2
- packet: Consider removing tf variable setup_raid_* HOT 1
- packet: Consider using afterburn to simplify setup_raid_* flags HOT 5
- Difference between Lokomotive and typhoon HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lokomotive-kubernetes.