Use Wayland for timed callbacks instead of GLib

[Timidger]

Use Wayland for timed callbacks instead of GLib

[psychon]

In Wayland we can add our own timeouts, so we just need to abstract this as part of the API we give to Lua.

What exactly does this mean? You want to implement your own timeout API based on the API from libwayland? And this would allow to destroy the timeouts when the Lua state is replaced?
(It would also be possible to make Lua use "your own special" GMainContext, but this would require a way to pass that to Lua)

The only other place glib is used in the Lua code is in some misc functions that can be rewritten in Lua or Rust.

Most of the GLib-uses can be just left as-is since they do not involve the main loop. :-)
Things that I can find right now:

• gears.spawn.read_lines uses GIo to read data from a file descriptor
• menubar.util.parse.dir uses GIo

However, https://developer.gnome.org/gio/stable/GIOStream.html says:

stream operations being associated with a given GMainContext (the thread-default context at the time the operation was started),

So setting the thread-default context might actually be enough to make these work.

is this an acceptable strategy for dealing with the restart issue?

Depends on how compatible you want to be with awesome. This works for gears.timer (which seems to be the only thing in awesome using the GLib main loop), but people could do "whatever" in their config and way-cooler would not be compatible for that. If way-cooler sets the thread default context, this might be 95% compatibility, but something out there could still be using the "global" default GMainContext...

[Timidger]

As discussed on IRC, a solution would be to modify the lib so that it defers to how the timed callbacks are implemented to Awesome / Way Cooler. In Awesome it would use GLib and in Way Cooler it would use Wayland. So instead of Lua using GLib timers directly, it would call a function defined in C/Rust.

This will allow the libraries to continue to work with either backend.

If user code depends on the main glib context being able to always refer to a valid Lua instance across restarts then that's not something we can support right now and they are SOL. There's really not anything we can do properly beyond #556.

[psychon]

But I removed timers from the C API just recently-ish (December 2014, commit 94e61ec, a net removal of 140 lines). :-(

I would rather have way-cooler override or monkey-patch gears.timer somehow, if that's the way to go. For this, we could even make this easier in awesome, by e.g. wrapping the glib part in functions like gears.timer.__private_start and way-cooler could then monkey-patch these.

[Timidger]

That isn't going to cut it unfortunately. Here's a conversation I had with @Elv13:

13:47 < Elv13> timidger: Thinking about it, there is some other places where not having a GLib mainloop will cause 
               major issue. Everything based on Gio will become unreliable
13:47 < timidger> It will?
13:48 < Elv13> yes, it often uses events
13:48 < timidger> Do the other things actually require the main loop? I thought only the timer needs it
13:48 < timidger> That's what psychon seemed to say in https://github.com/way-cooler/way-cooler/issues/561
13:48 < Elv13> as far as I known, everything using sockets and/or has _async_ in the name will not wake up correctly
13:50 < timidger> > g_spawn_async_with_pipes
13:50 < timidger> Is this the problematic function?
13:50 < timidger> grepping through awesmoe  Ifound that in spawn.c
13:51 < Elv13> among other, including the scandir in the menubar 
13:52 < timidger> the one in spawn.c is fine, because that's an implementation detail of Awesome
13:52 < timidger> in Way Cooler we have to provide a spawn api and we can use something other than glib
13:53 < Elv13> ../lib/menubar/utils.lua:            local list, enum_err = enum:async_next_files(files_per_call)
13:53 < Elv13> and every module that uses Gio
13:53 < timidger> whelp
13:56 < Elv13> timidger: Plus, in the future I/we hope to *increase* the use of async functions to improve 
               performance, there is no way that can be done without a main loop. Lua is single threads, async stuff 
               and coroutine is the *only* way to make parallel code work

I haven't tested it, but I assume these uses of the glib library won't survive the Lua restart either.

stream operations being associated with a given GMainContext (the thread-default context at the time the operation was started),

So setting the thread-default context might actually be enough to make these work.

If these functions aren't a problem great. But assuming they are problems...

So the only options are to remove glib (which is a massive undertaking and splits the libraries in two), re-architecture Way Cooler to be multi-process, or to have a shitty restart.

I really don't want to have bad restart functionality, especially because this is the only thing making that difficult. So it seems the only good solution is to split the codebase in two. Which is also a huge undertaking.

cc @acrisci

[Elv13]

Just want to add those 2 links from the second part of the IRC discussion for references:

https://anholt.github.io/twivc4/2018/05/30/twiv/

https://wiki.gnome.org/Initiatives/Wayland/GnomeShell/GnomeShell4

And just to clarify everything. My position on this is as an outsider who has no patches to show for it. But on the other hand, I maintain many of the Lua modules/files associated with this, so I dare to comment anyway. I think in the long run, now that LGI async stuff is finally mostly stable (yes, I know there is unmerged patches), I think Awesome libraries should eventually use more async functions for things like writing history, dbus interaction, .desktop parsing, icon loading and /proc/sys probes. Those APIs are based on the mainloop and are not reliable/usable without the gmainloop. The Awesome Core API capi.dbus module is also unmaintained and uses APIs that are officially documented not to be for general use. GDbus is the way to go to replace it as it integrates well with the other Awesome dependencies and data format, but also requires the mainloop.

The second point to take into account is that existing external config can require all libraries exposed using LGI. That include database, json, svg and personal information management (contact, calendar events). Those libraries expect a GMainLoop to exist and wont work without it. Even AwesomeWM own website has code snippets based on this. Finally, that mainloop is onmipresent in Linux apps. It's not limited to GTK either, Qt, KDE, Firefox and Chrome along with many daemons also use it.

As to how you handle this. Until I have time to contribute, that's none of my business to share a strong opinion about it. I just want to point out that I think the mainloop issue cannot be avoided and place strong limitations on what I can or cannot merge into Awesome if we want to share these modules. I also really, really want Awesome-x11 and way-cooler to share the libraries because fragmentation will do us no good. Given the discussion about this topic by the Gnome devs, I also think we are no wiser when it comes to the hard conclusion and decision they had to make. They dug themselves a hole and are now struggling, but forced, to fix it. It is a cautionary tale about having latency sensitive code paths bundled with a script language interpreter/GC in the same thread.

[Timidger]

We've gone back and forth on both this thread and #556. I think everybody has made their points about how to adapt Awesome to Wayland while staying compatible (which is an explicit goal for 1.0).

1. I'm not a fan of glib (as is evident with this thread) but it needs to be in Way Cooler 1.0 in order to satisfy the Awesome compatibility requirement.

2. Restarting the window manager without dropping all the clients does not work when there is just one process and a glib main loop is used. Dropping all the clients is not acceptable.

3. As @Elv13 pointed out, Gnome has ran into significant problems with having an GC'd interpreter on the main thread.

With these sets of requirements, there's only one solution: two processes, one being "Awesome" (where the Lua thread resides) and one being "Way Cooler" (the actual compositor that "Awesome" talks to).

Way Cooler can be small, simple, and made to be resilient. It is a base that, once implemented, probably won't change much.

Awesome can be bigger and more complicated, but still rely on Way Cooler to do all the heavy lifting. In this way Way Cooler is the new X11 and essentially just another backend for it.

We can develop small Wayland protocols to define the interface between the two components. We can push to make them standardized. If enough of them are, it would be possible to mix Awesome with other WMs like Gnome again (this is a goal that seems unlikely, but it's possible technically).

This is going to be much more work. It means that "Year of Awesome" post I made will not come true. But I very strongly believe this is the correct way forward. Half-assing Wayland support is not a good idea, especially while the X11 backend still works and is supported while this is being developed. There's no need to rush this.

Until I have time to contribute

@Elv13 Hope that means you are thinking of contributing. There's plenty of work to do 😄

[Elv13]

@Elv13 Hope that means you are thinking of contributing. There's plenty of work to do smile

/me offtopic

Right now that would be spreading my AwesomeWM/WayCooler (very limited) time too thin to do anything meaningful. What I can do is resurrect the very old branch which re-implement the tag module in pure Lua and remove all the C-API code. The tags are an AwesomeWM concepts. The only reason they are implemented in C is because it's leftover code from AwesomeWM 2.x and because it was less effort to just expose them in Lua than introduce the X11 workspace concept in the API and built the tags on top of that.

For historical context, AwesomeWM 2.x was a lot closer to today bspwm than to Awesome 3.x/4.x. It was using message passing between itself and an external process instead of having it's own script VM. The userland was implemented in Ruby and the WM was DWM+rc_config+message_passing_socket. In a rather ironic way, we are going full circle here ;).

I fear the Lua tag PR cannot be merged for Awesome 4.x because it would break if type(obj) == "tag" then ... end on some Lua versions. However it's a minor API break and could totally be merged once Way-Cooler is ready and just call it Awesome 5.0.

Cleaning up the way screen are handled is also a PR I could finish. The fixed API always uses virtual screens. So it will be a lot easier to share between AwesomeWM-X11 and Way-Cooler because the abstraction is more flexible instead of being tied to the very specific behaviors of xrandr. Plus it fixes a lot of nasty open bugs like losing all tags when putting some laptop in sleep mode (because their driver remove all screens, then creates new ones, which is idiotic). Plus, it negates the need for awesomeWM/awesome#2223 to be merged. I think that PR is a hack.

I have the following long term projects going on right now:

• Rewrite the notification module to be modular and use the new APIs (55% merged, 2500 lines of code undergoing review, but it's stalled due to the lack of time)
• Upgrade the input handling APIs to be easier to use (done, just waiting until I have time to merge the open PRs)
• Merging the client tiling layout system with the client layout system (ready for testing for years, but nearly impossible to merge until 5.0, so I make no efforts toward merging it right now). This code brings parity with way-cooler 0.x and i3wm features, along with the current AwesomeWM ones.
• The new virtual screen abstraction (there's some code, it's not ready)
• I have a cool meta template mode for widgets to enable reactive programming.
• The tag Lua port (in a branch for nearly 10 years)

[Timidger]

Closing this since we are well on a way to the split up compositor / client setup which solves the original problem.