kworkflow/kworkflow

kw is our inglorious kernel developer workflow tool with a simple mission: reduce the overhead with infrastructure setup for Linux development

About

kw has a simple mission: reduce the setup overhead of working with the Linux kernel and provide tools to support developers in their daily tasks. If you have a set of repeatable tasks that you usually perform while working in your favorite kernel subsystem or similar, consider adding it as a part of kw.

Install

Take a look at Install and Uninstall.

How to

If you want to know more about kw's usage and its commands, take a look at Kw man or, with kw installed, run kw man.

Tests

If you want to know more about kw's tests take a look at kw tests.

Generate Sphinx Documentation

If you want to generate the Sphinx documentation, you can use:

./setup.sh --docs

Finally, you can use your browser to look at the index.html page. For example:

firefox build/index.html

Contributing

We are happy that you want to help us! If you are looking for a good starting point, check those issues and don't forget to read our Contribuitor's Guide (or howtocontribute file).

Reach Out

The best way to get help or make suggestions is by creating issues or making a pull request, someone is likely to reply to these in little time. You can also reach out to us at the #kw-devel channel on the OFTC IRC server (irc.oftc.net), or by joining or opening a new discussion on the discussions page.

License

Kworkflow is under GPL-2.0+

kworkflow's Issues

Replace Qemu options for libvirt

I had a small talk with a Qemu maintainer, and he recommended me to remove the direct use of the Qemu option for support and instability reasons. He suggested me to use libvirt.

Add support for git send-email

There is a legacy function named mk_send_email in the mk.sh file. It could be really nice if we update this function and make it works again in the KW.

Support multiple shells

Ideas:

• We can have a single alias in all *rc
• Make a driver
• Add to .profile

P.s.: We should take a look at the auto completion

This issue depends on #50

Improve the file access handling

Currently, we do not check if the user has a read or write permission for the file that kw is looking. It could be nice if we provide a nice output for these cases.

Use the arch variable in the kw b

Now we have the information about the compilation architecture in the kworkflow.config. How about starting to use it?

Add unit test

There is no unit test in the kworkflow. Add the basic infrastructure using shunit2, follow some suggestion:

1. If you never used shunit2, take a look at: https://github.com/rodrigosiqueira/flextrace/tree/master/tests and https://github.com/rodrigosiqueira/flextrace/blob/master/run_tests.sh
2. Test functions that handle kworkflow.config
3. Test the build option
4. Test codestyle
5. Test maintainers

Integrate dim script to kw

I just discovered a set of scripts from Intel, with the goal to help maintainers handle some of their common tasks. The instructions on how to use the scripts:

https://drm.pages.freedesktop.org/maintainer-tools/getting-started.html

And the repository:

https://gitlab.freedesktop.org/drm/maintainer-tools/raw/master/dim

How about understand it, and integrate it to kw? Could be extremely nice!

Create a wrapper for get_maintainers

I regularly forget the get_maintainer parameters, and I start to get annoyed by that. Create a wrapper for it.

Create documentation

We need to create a basic documentation, maybe create a directory "Documentation" and use some tool for building the project documentation. Additionally, create a man documentation.

Return error exit code on failure

As of now, whenever kw fails (e.g. not in kernel root when running kw maintainers or invalid path for kw codestyle) the script complains, but returns a zero exit code. To unify error handling, instead of searching for particular complain strings, it would be interesting for kw to return a negative exit code.

Add verbose mode

It could be good if we have a verbose option that shows all the command used by kw.

Functions are exposed to the user

Kworkflow installs its functions directly to bash without namespacing. This causes user functions to potentially be overwritten by kw (or other user definitions inside .bashrc potentially overwrite kw's). Instead, kw should have a namespace such as kw or kworkflow. E.g.:

kworkflow::function_name or kw::function_name

Encapsulate grep and find search

A common task in the kernel development it is the task of finding a snippet of code via grep or a file via find. Usually, I'm using a command such as grep -nrwI PATH/ -e EXPRESSION, we can encapsulate this sort os search. Maybe something like:

kw search EXPRESSION -> search the entire directory
kw search PATH EXPRESSION

It could be nice if this feature design becomes expansive for another sort of search, for example, git log.

Make kw boot fails nicely

When kw boot fails (because of an inexistent qemu path configuration, for example), it just exists without printing anything. Shouldn't we remove the 2> /dev/null redirect of the kw boot command?

Replace SSHD by another lightweight SSH server

Currently, we're using sshd for handling the ssh connection which is totally ok. However, we don't need all the sshd features because of this we can use a lightweight ssh server. Currently, I'm testing Dropbear SSH, and it looks excellent. Let's consider to use it.

Remove unused global configuration variables and move used to config file

Since the movement too a global configuration file (PR #54 ), some global configuration variables will become redundant and meaningless, such as the ones at the top of commons.sh. This unused configuration variables should be removed (at commons.sh and all the code). And the ones that are still in use, but not in the global configuration file yet, should be moved to it.

Improve setup output

How about show only the commands?

Make kw instalation closer to unix standards

kw is currently being installed at ~/.config/kw, which dosen't seem so good (since it's a dir for configurations, not code). Shouldn't we install kw somewhere else, such as ~/.kw, for example?

Also, we could add kw to PATH (or create an alias) instead of loading it to bash. This change would also make the subshells in kw.sh unnecessarily and the variables "src_script_path" and "external_script_path" not visible externally, which is good.

(proposed by @capellaresumo and I)

Replace grep inside loop by regex

There are many loops with grep operation inside, I noticed that most of them can be replaced by a regex operation inside an 'if' operation (e.g., if [[ SOMETHING =~ REGEX ]]). We should work on this.

Problem with new ArchLinux Vm

Update vm prepare and documentation for arch playbook/configuration in order to alert for the requirement of enabling the dhcplient via systemctl

Create vm-list option

Create a mechanism to list all the available VMSs. For example:

$ kw vm-lists # or kworkflow vm-lists
VMs:
  virty.qcow2
	debian.qcow2

Improve playbook for Arch

Update Arch playbook to generate mkinitcpio.

Remove the need of sudo for installing a module in the VM

We need to mount the Qemu image if we want to install a new module or kernel image. This is an annoyance, let's start to use libguesfs for doing this task.

Improve kw prepare

After read the tutorial written by @MatheusBernardino in https://flusp.ime.usp.br/git/2019/02/15/sending-patches-by-email-with-git/, I realized that we can create a kw prepare for make some basic setups for users as well. How about something like:

kw prepare --host --git-send-email

ofc, this is just a draft... but what do you think @MatheusBernardino ?

Make checkpatch and get_maintainers options rely in a configuration file.

Checkpatch and get_maintainers are two set of scripts with thousands of options. Instead of hardcode some options in the kw, it could be really nice if we take this information from a configuration file.

Create a Host deploy option

Description Later

Add travis CI configuration

Run our tests
Run setup and validate

Make kw handling patches

It could be nice if kw wrapper the git-format-patch, by handling a single patch or a series. Additionally, at the end of the process, the command should check the codestyle. Something like:

kw patch -s HASH # generate the patch for the HASH
# or -> kw patch -s -> just take the last commit
# after git format-patch kw apply the codesyle

kw patch HASH1-HASH2 # generate a patchset

Finally, It could be nice if kw save the patches in a default folder for later management. In the future, we can create a local patch management.

Add a management mechanism for .config files

During the kernel development activities, it is common to have multiple .config files for different targets. For example, one .config file for your host machine and other for your virtual machine. How about make kworkflow provide tools for manage this file? Something like:

kw register_config /PATH/TO/.config -d "This .config file works well on my x86 machine"
kw register_config /PATH/TO/.config -d "This .config file works for my arm device"
kw show_configs
 1) This .config file works well on my x86 machine
 2) This .config file works for my arm device
kw get_configs 1

Make kw compile things in parallel

kw build does not start the compilation in parallel, we have to improve it.

Rework part of the tests

Our tests have some problems, such as:

1. Duplicate code
2. We just use a few constants
3. We just use a tiny set of features provided by shunit2
4. Lack of documentation
5. Lack of code style

We should take some time for reworking our tests

Improve kw e by adding alternative tools in case that git cannot handle the situation

"kw e" relies on git for searching things which is great. However, sometimes we want to explore things in a bunch of files that not use git. In this sense, it could be nice if "kw e" support alternative tools for search things. For example, we can havegrep -nrwI /PATH -e EXPRESSION if the current directory is not under git control.

Make kw mount and kw boot work with local configuration

When using a local kworkflow.config file, "kw mount" and "kw boot" currently ignore the variable ${configurations[qemu_path_image]} and use, instead, the default variable $VDISK. So the image to be mounted or built will always be the one pointed by $VDISK even when the user has a local configuration file (a kworkflow.config).

kw maintainers only work from the first directory of the Linux tree

It could be nice if kw maintainers works from any directory. For example:

cd drivers/staging/
kw maintainers iio/

Improve 'kw c' output

If there is no code style problem in the file, kw shows something like this per file:

=========================================================

=========================================================

=========================================================

=========================================================

We have to handle it with two approaches:

1. Add a verbose option in the code style, that produces something like this:

kw c -v lalala.c
=========================================================
lalala.c: No checkpatch problem
=========================================================

2. Show nothing if the file is clean:

> kw c xpto.c
>

Improve autocomplete

When typing a command such as kw maintainers, it would be really nice to have a tab completion for files. This also happens in other commands.

Improve README information about run_tests.sh

The run_tests.sh script must be run from the repo's root, otherwise, it fails. We should include this information in README.md.

Handle the case that VM is up and users try to install a module

If the vm is running and the user tries to install a new version of a module (kworkflow bi), the operations end with no warning. This a problem, because the module is not installed if the vm is running. Print a warning message in this cases.

Mount VM issue using Ubuntu

kw fails to mount in Ubuntu when the kernel image has no read access. Possible solution to this problem is to give read permission to the kernel image automatically when install kw or asking to user make this change.

Use checkpatch.pl from kernel tree

Currently, kw uses a copy of checkpatch.pl downloaded from torvalds' repo and installed at ~/.config/kw/external. Shouldn't it be better if kw used the checkpatch.pl version that comes in scripts/ on every kernel tree? This approach would, also, be more consistent with the way kw handles and uses the get_maintainer.pl script.

VM prepare

Kworkflow works with a qemu images which require a previous installation and configuration. It should be nice to create a command named "kworkflow vm prepare" that create a basic setup for sharing directory and other stuff.

Refactor miscellaneous.sh, commons.sh and utils.sh

In the src folder, kw has a commons.sh, a miscellaneous.sh and a utils.sh, which can make a little difficult to understand the project structure for a newcomer. Can't we refactor the code to join all these files, or rename them with more meaningful names?

(If you chose to make this, keep in mind that tests uses miscellaneous.sh)

Wrapper ssh command

I'm tired to type "ssh -p 2222 localhost" constantly.

Add support for git am and apply

Usually, I have to apply some patches to do a review or for pushing stuff; because of this I have to take care with many details. It could be nice if kw provide this sort of feature with an integration mechanism for patchwork.

Add an option for `kw m` for `format-patch` output

The kw m command outputs the name and email of maintainers, however it must be parsed for git format-patch. Kw could do it for us.

Remove hardcoded VM name

The virty.qcow2 variable is hardcoded, update it to be flexible. Suggestion: accept a VM name via command line, if it no exists replace it by virty.qcow2.

Improve "kw m --authors" for multiline MODULE_AUTHOR statements

When we run a kw m --authors, only authors found in a single line MODULE_AUTHOR statement are captured and printed. In the future, it would be nice to look for multiline MODULE_AUTHOR statements such as:

MODULE_AUTHOR ("[email protected]" \
               "[email protected]" )

and:

MODULE_AUTHOR ("[email protected]"
               "[email protected]" )

which are, both, valid C statements. In the currently implementation neither of these four emails would be printed.

Alert does not work on Ubuntu

For some reason alert does not work on my Ubuntu. Follow my config file.

qemu_hw_options=-enable-kvm` -daemonize -m 3G -smp cores=8,cpus=8 qemu_net_options=-net nic -net user,hostfwd=tcp::2222-:22,smb=/home/shayenne qemu_path_image=/home/shayenne/p/virty.qcow2 alert=s

Add sonorous and visual alert

Sometimes, compilation and installation take time and we want to do something different while we wait for the end of the processes; as a result, we forgot to go back and check the compilation and wast a little bit of time. It could be nice if kw provides an option for making a sonorous and visual alert at the end of some processes.

Bash parameters autocomplete

I don't know how hard or even if it is possible to implement the autocomplete in the kw. It could be nice to have this feature, just something and the autocomplete works.