kubernetes-sigs/hydrophone

Hydrophone is a lightweight Kubernetes conformance tests runner

hydrophone

Lightweight runner for kubernetes tests. Uses the conformance image(s) released by the kubernetes release team to either run individual tests or the entire Conformance suite. Design is pretty simple, it starts the conformance image as a pod in the conformance namespace, waits for it to finish and then prints out the results.

Project Goals

• Simplified Kubernetes Testing: Easy-to-use tool for running Kubernetes conformance tests.
• Official Conformance Images: Utilize official conformance images from the Kubernetes Release Team.
• Flexible Test Execution: Ability to run individual test, the entire Conformance Test Suite, or anything in between.

Project Non-Goals

• Replacing Kubernetes Testing Frameworks: Not intended to replace existing frameworks, but to complement them.
• Extensive Test Development: Focus is on running existing tests, not developing new ones.
• Broad Tool Integration: Limited integration with third-party tools; maintains simplicity.

Getting Started

Prerequisites

Ensure you have Go installed on your system and the GOPATH is set. You will also need access to a Kubernetes cluster.

Install

Install Hydrophone using the following command:

$ go install sigs.k8s.io/hydrophone@latest

Alternatively you can download the latest release from the releases page

Running Tests

Ensure there is a KUBECONFIG environment variable specified or $HOME/.kube/config file present before running hydrophone Alternatively, you can specify the path to the kubeconfig file with the --kubeconfig option.

To run conformance tests use:

$ bin/hydrophone --conformance

To run a specific test use:

$ bin/hydrophone --focus 'Simple pod should contain last line of the log'

To specify a version of conformance image use:

$ bin/hydrophone --conformance-image 'registry.k8s.io/conformance:v1.29.0'

Community

Please reach out for bugs, feature requests, and other issues! The maintainers of this project are reachable via:

• Kubernetes Slack in the [#hydrophone], [#sig-testing] and [#k8s-conformance] channels
• Filing an issue against this repo
• The Kubernetes SIG-Testing Mailing List and SIG-Release Mailing List

Current maintainers are @dims and @rjsadow - feel free to reach out if you have any questions!

Pull Requests are very welcome! If you're planning a new feature, please file an issue to discuss first.

Check the issue tracker for help wanted issues if you're unsure where to start, or feel free to reach out to discuss. 🙂

See also: our own contributor guide and the Kubernetes community page.

Code of conduct

Participation in the Kubernetes community is governed by the Kubernetes Code of Conduct.

hydrophone's Issues

Fix different timezones in logs

What's the issue?

The logs of hydrophone show time in different time zones. One seems to be local time and other in UTC.

How to reproduce?

bin/hydrophone --conformance

Broken links in README

What's the issue?

There're 2 broken links in the README.md which are not resolving to correct resources:

1. contributor guide: https://sigs.k8s.io/hydrophone/blob/main/CONTRIBUTING.md
2. filing an issue: https://sigs.k8s.io/hydrophone/issues/new

Improve `.gitignore`

A user might try to build the binary using go build . which will result in the creation of ./hydrophone. To avoid pushing this binary to git, we should add hydrophone to our .gitignore.

Ensure hydrophone works on air gapped environments

Just a reminder for now... but we may have to figure out how to test

Run a full test with `--focus` set to `[Conformance]` and dig into errors

against kind cluster is fine! there may be other settings we have to do to ensure all the conformance tests pass that we don't know of yet.

Redundant error check in client.PrintE2ELogs() function

The error check on https://github.com/kubernetes-sigs/hydrophone/blob/main/pkg/client/check.go#L63 seems redundant since it will always be nil given that within the for loop a new err variable is initialized everytime https://github.com/kubernetes-sigs/hydrophone/blob/main/pkg/client/check.go#L55

Migrate from github actions to prow

Migrate from github actions to prow

Periodic e2e Tests (periodic_e2e.yaml)

• Add prow config for periodic execution of test, scheduled every 6 hours.
• Ensure the build process and e2e tests run similarly to the current GitHub Actions workflow.
• Implement artifact upload for conformance results in Prow.

Periodic Conformance Tests (periodic_conformance.yml)

• Add prow config for periodic execution of full conformance test, scheduled every 6 hours.
• Adapt the steps for building, testing, and running full conformance tests.
• Include steps to archive conformance results.

Migrate presubmit (test.yml)

• Ensure linting is replicated in prow.
• Adapt build, unit test
• Implement checks for dry run duration and test execution duration.
• Set up artifact archiving for conformance results.

run-e2e script CHECK_DURATION: unbound variable

An issue in our hack/run-e2e.sh script is causing periodic jobs to fail. The variable CHECK_DURATION should be initialized similar to the other variables around https://github.com/kubernetes-sigs/hydrophone/blob/main/hack/run-e2e.sh#L70-L82 or have a default value set on line 54.

hack/run-e2e.sh: line 54: CHECK_DURATION: unbound variable

If anyone wants to take a look at this, please feel free to assign yourself to the issue. And if there are any questions, reach out in this issue or to me on slack at @rjsadow

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/hydrophone-e2e-check/1747715074929201152

Simplify Testing setup with Makefile

All of the prow end to end tests that we have use the same setup where we build the binary then run the script. Example: https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes-sigs/hydrophone/hydrophone-periodic.yaml#L32-L34

We could simplify this by adding a make command called test-e2e in the Makefile that runs build then invokes the hack/run-e2e.sh script.

If anyone wants to take a look at this, please feel free to assign yourself to the issue. And if there are any questions, reach out in this issue or to me on slack at @rjsadow

/good-first-issue

Count of conformance tests not matching what is expected (380 vs 388)

https://github.com/kubernetes/kubernetes/blob/master/test/conformance/testdata/conformance.yaml shows 388 tests. we seem to be running 380 only

Pod log download happens after termination

ref: https://kubernetes.slack.com/archives/C6L3G051C/p1703611973360109?thread_ts=1703359255.475159&cid=C6L3G051C

Waiting for pod to terminate...
2023/12/26 22:19:04 Pod terminated.
2023/12/26 22:19:04 downloading e2e.log to  /Users/ishankhare/work/hydrophone/e2e.log
2023/12/26 22:19:04 unable to download e2e.log: unable to upgrade connection: container not found ("output-container")
exit status 1

Grab logs from RESULTS_DIR inside the pod running the conformance tests

See here for how RESULTS_DIR is created and populated:
https://cs.k8s.io/?q=RESULTS_DIR&i=nope&files=&excludeFiles=&repos=kubernetes/kubernetes

see how we do a docker exec to get the results in shell script based CI job:
https://github.com/kubernetes/test-infra/blob/8bd221589dbbdcbc93598d3a1f125f09d18428cc/experiment/kind-conformance-image-e2e.sh#L156

see how sonobuoy deals with it:
https://github.com/vmware-tanzu/sonobuoy/search?l=Go&q=RESULTS_DIR

Essentially, you need to write code that will get into the pod, and extract all files in the /tmp/results directory and drop it into the same --output directory you have in:
#2

thanks,
Dims

Confusing configuration file

Suppose you download Hydrophone and put it somewhere in your PATH. You navigate to your project of choice and take a look:

$ cd ~/myproject
$ hydrophone --help
...
      --output-dir string           directory for logs. (default "/home/xrstf/myproject")
...

Cool! So you run it to perform conformance tests:

$ export KUBECONFIG=...
$ hydrophone --conformance
...
13:30:53 INF Downloading e2e.log to /home/xrstf/myproject/e2e.log...
13:30:53 INF Downloading junit_01.xml to /home/xrstf/myproject/junit_01.xml...

Cool! Now let's run the conformance tests for another project:

$ cd ..
$ cd anotherproject
$ hydrophone --help
...
      --output-dir string           directory for logs. (default "/home/xrstf/anotherproject")
...

$ export KUBECONFIG=...
$ hydrophone --conformance
...
13:30:53 INF Downloading e2e.log to /home/xrstf/myproject/e2e.log...
13:30:53 INF Downloading junit_01.xml to /home/xrstf/myproject/junit_01.xml...

And now you'd be confused why your files ended up in a different location.

I think the way a config file works should be handled differently. Silently remembering default values that are path-dependent is IMHO not a good idea. I would never expect a tool to just take my current flags and use them for all future invocations if I don't specify a flag. Especially since I have no real way to opt out of this behaviour.

Refactor Conformance Testing Process

The current process of running ./hydrophone -conformance is cumbersome as it waits for tests to finish and then grabs the results. This could be more efficient. It may be more user friendly to refactor some of the testing code into separate sub-commands (e.g., run, get-logs, fetch-files, cleanup).

Conform to kubernetes project template

xref: kubernetes/org#4654

For moving the project into the kubernetes organization, the repo needs to follow a standard conformance template (https://github.com/kubernetes/kubernetes-template-project/). Items that need to be set:

• README.md outlining the project goals, sponsoring sig, and community contact information
• OWNERS with the project leads listed as approvers
• CONTRIBUTING.md outlining how to contribute to the project
• code-of-conduct.md copied from the template repo
• LICENSE which must be Apache 2.0
• SECURITY_CONTACTS with the contact points for the Product Security Team

Remove `cfg` and `config` from `PrintListImages()`

There are two unused parameters cfg and config in the PrintListImages() function under pkg/service/list_images.go.
Since these are unused, it would be better to remove them from the function.

/good-first-issue

hydrophone --cleanup is broken

We are not setting default value for namespace flag when calling hydrophone --cleanup as a result its getting a blank namespace https://github.com/kubernetes-sigs/hydrophone/blob/main/pkg/service/init.go#L325

reeta@Reetas-MacBook-Pro homebrew-core % hydrophone --cleanup
17:35:42 INF API endpoint : https://127.0.0.1:50108
17:35:42 INF Server version : version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-19T15:42:59Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/arm64"}
17:35:42 ERR an empty namespace may not be set when a resource name is provided
reeta@Reetas-MacBook-Pro homebrew-core % 

Lack of Support for Additional Conformance Test Flags

Sometimes control plane nodes are not schedulable by default, requiring the use of additional conformance flags like --allowed-not-ready-nodes=3 for tests. However, Hydrophone does not currently offer a comprehensive mapping to the conformance image options.

ref: https://kubernetes.slack.com/archives/C78F00H99/p1705427311573629
ref: https://github.com/kubernetes/kubernetes/blob/9b5e2dc54d2315402a462e75c168bfc4ce28ecf3/test/e2e/framework/test_context.go#L363

Add a brew repo

Hey all, would like to start using this instead of sonobuoy, however you don't have a brew package yet. Would love to see that get added.

Bump go version

The current code base is using go v1.18. We should bump this to at least 1.20.

`#hydrophone` should be added to the README

The #hydrophone slack channel was recently created. We need to update the readme https://github.com/kubernetes-sigs/hydrophone/blob/main/README.md?plain=1#L121 with the new slack channel and make sure it's apparent that this is the preferred channel to reach out on. We can keep #sig-testing and #sig-conformance as they're very related and relevant to this project.

If anyone wants to take a look at this, please feel free to assign yourself to the issue. And if there are any questions, reach out in this issue or to me on slack at @rjsadow

/good-first-issue

multiple typo and grammatical mistake in air-gapped.md

Multiple mistake available in air-gapped.md

Use --verbosity to toggle between spinner and verbose Ginkgo output

Currently hydrophone uses a spinner to indicate something is going on. This is, I think, because Ginkgo's default output is one long line, so the line-based log reading hydrophone does waits "forever" and no progress is visible.

Since there is already a --verbosity flag, I was wondering if it could be used to toggle between

• v <=4: show the spinner, like it is right now
• v >= 6: do not show the spinner, but configure E2E_EXTRA_GINKGO_ARGS=-v, which will lead to more line-based output.

Currently, setting the -v arg on Ginkgo by hacking hydrophone looks like

So a way to disable the spinner in favor of more verbose output would be nice.

Show some progress info for the tests

What's the issue?

While running the test we don't show any progress bar or any other info to know that the tests are running. Since the test suite takes a while to run, it sometimes feels like the process hangs and there's no progress at all.

(screenshot showing the same content even after 30 minutes)

Some ideas to tackle this might be adding a progress animation like the kind does when spinning up a cluster OR simply setting the user expectations that the process might take a while.

How to reproduce

Run the following command

bin/hydrophone --conformance

Image Version Matching

Hydrophone currently always selects the latest image version (v1.29.0) by default, regardless of the cluster version it is testing. This can lead to mismatches causing inconsistencies in testing and results. Users can specify the version image with --conformance-image, but requires additional user input, configuration, and knowledge of the command line options. The desired behavior is for Hydrophone to automatically detect and use a conformance image that matches the cluster version.

ref: https://kubernetes.slack.com/archives/C78F00H99/p1705427311573629

`--parallel` flag not working?

Currently --parallel gives the impression of setting the concurrency level (thread/goroutine count). This makes sense when looking at the Dockerfile for the conformance tester:

ENV E2E_PARALLEL="1"

However when I grep through the k/k codebase, this env is only used in 2 places:

• The go-runner, which is the image's default entrypoint. But the go-runner does not treat the env var as an integer, but only interprets it as a boolean flag: https://github.com/kubernetes/kubernetes/blob/ec301a5cc76f48cdadc77bcfbd686cf40b124ecf/test/conformance/image/go-runner/cmd.go#L34-L40
• The run_e2e.sh, which is the only place that optionally considers the env to be a number: https://github.com/kubernetes/kubernetes/blob/ec301a5cc76f48cdadc77bcfbd686cf40b124ecf/test/conformance/image/run_e2e.sh#L60-L66

Since hydrophone does not set a custom command, setting --parallel=7 has no effect.

run_e2e.sh is deprecated and now only a shim to the go-runner, so I guess it would make sense to redefine --parallel to be a boolean. Alternatively maybe it could remain an int and set the --procs (--nodes is an alias for it) flag via E2E_EXTRA_GINKGO_ARGS, like the run_e2e.sh did?

Improve logging for resource creation

When a resource already exists, such as clusterrole etc, we log both:

21:54:37 INF clusterrole already exist conformance-serviceaccount
21:54:37 INF clusterrole created 

which is somewhat misleading. This should be fixed i.e. if a resource exists, we should only log already exists and if it gets created, we should log it got created.

Ability to run Windows Operational Readiness tests from hydrophone

• https://github.com/kubernetes-sigs/windows-operational-readiness/
• https://kubernetes.io/blog/2024/04/03/intro-windows-ops-readiness/

Add a github action to create a kind cluster and run some tests

• Run a single test (quick!)
• Run conformance tests.