Most visited posts/pages in the last 24 hours
Join 189 other subscribers
lkubuntu
A listing of random software, tips, tweaks, hacks, and tutorials I made for Ubuntu
Idea: Non-windowing display server
Posted by on August 26, 2015
For the TL;DR folk who are concerned with the title: It’s not an alternative to wayland or X11. It’s layer that wayland compositors (or other) can use.
As a quick foreward: I’m still a newbie at this field. While I try my best to avoid inaccuracies, there might be a few things I state here that are wrong, feel free to correct me!
Wayland is mainly a windowing protocol. It allows clients to draw windows (or, as the wayland documentation puts it, “surfaces”), and receive input from those surfaces. A wayland server (or “compositor”) has the task of drawing these surfaces, and providing the input to the clients. That is the specification.
However, where does a compositor draw these surfaces to? How does the compositor receive input? It has to provide many backends for various methods of drawing the composited surface. For example, the weston compositor has support for drawing the composited surface using 7 different backends (DRM, Linux Framebuffer, Headless [a fake rendering device], RDP, Raspberry Pi, Wayland, and X11). The amount of work put into making these backends work must be incredible, which is exactly where the problem relies in: it’s arguably too much work for a developer to put in if they want to make a new compositor.
That’s not the only issue though. Another big problem is that there is then no standard way to configure the display. Say you wanted a wayland compositor to change the video resolution to 800×600. The only way to do that is to use a compositor-specific extension to the protocol, since the protocol, AFAIK, has no method for changing the video resolution — and rightfully so. Wayland is a windowing protocol, not a display protocol.
My idea is to create a display server that doesn’t handle windowing. It handles display-related things, such as drawing pixels on the screen, changing video mode, etc… Wayland compositors and other programs that require direct access to the screen could then use this server and trust that the server will take care of everything display-related for them.
I believe that this would enable for much simpler code, and add a good deal more power and flexibility.
To give a more graphic description (forgive my horrible diagraming skills):
Current Stack:
Proposed Stack:
I didn’t talk about the input server, but it’s the same idea as the display server: Have a server dedicated to providing input. Of course, if the display server uses something like SDL as the backend, it may have to also provide the input server, due to the SDL library, AFAIK, doesn’t allow a program to access the input of another program.
This is an idea I have toyed around with for some time now (ever since I tried writing my own wayland compositor, in fact! XD), so I’m curious as to what people think of it. I would be more than happy to work with others to implement this.
lkubuntu 
Why a server and not a shared library? I think libinput and libweston are what you are looking for.
True, I was also looking into having a shared library for this, but I think it would be less flexible than a server. Theoretically, yes, you can replace a shared library, but most people depend on a shared library being very specific. A server, on the other hand, can be written by anyone, so long as it follows the protocol.
The other reason why I’m interested in a server vs a shared library is because of setting video modes. I want clients to be able to interface with this server directly, in order to be able to change the video mode (change screen resolution, gamma ramp, etc.). If it’s only a shared library, there is no standardized way of setting the video mode, which I believe is crucial to have a full desktop experience with wayland.
Well, in the current Wayland stack, a client talks to a compositor. There are different compositors, but since all use (in future) the very same shared library for common stuff (libwayland, libinput), they all implement the very same protocol. e.g. for setting the video mode and for input handling. I don’t see how a separate server process makes things easier.
There would need to be an extension to the wayland protocol to support display server capabilities, which is not the point of the wayland protocol. The wayland protocol is meant for windowing, not video modesetting. If wayland implements video-related protocol features, I think that would entirely defeat the purpose of making wayland in the first place.
Have you seen the fullscreen shell protocol in Weston? It is meant for other compositors to use Weston as the hardware abstraction layer and Wayland as the interface (IPC).
http://cgit.freedesktop.org/wayland/weston/tree/protocol/fullscreen-shell.xml
All the usual drawbacks of using an intermediate daemon apply, which is largely why plans for a standard “system compositor” architecture in general were abandoned, but it is still useful exactly for the reasons you list: a lot less work for a developer to get started.
There are also plans to turn Weston’s core and backends slowly into a library.
http://cgit.freedesktop.org/wayland/weston/tree/README
Ah, that’s what the fullscreen shell was for … I was wondering what it was! What drawbacks are you talking about though?
Running over DRM is different than running over Wayland or X11: on DRM, the compositor has full control over the display (resolution, refresh rate,…; through KMS), while on the other two, the compositor would be a mere window for which those concepts [resolution, refresh rate,…] would not apply. For example, a compositor may decide that the session it manages is “inactive”: on DRM, it [the compositor] would perhaps disconnect the screen or send some command to save energy, while on Wayland or X11 it wouldn’t do anything (I believe).
However, the main reason, I believe, is that designing a protocol that abstracted all these subtleties would be cumbersome, and using it would not be better than directly using the DRM (libdrm) interfaces.
On the other hand, there’s wl_fullscreen_shell, which to some extent allows the architecture you described. Perhaps it would be useful to applications like kmscon; there could be a timeout in the “system compositor” so that, when kmscon hadn’t updated its fullscreen surface in that time, the compositor would handle the “inactive session” state. This protocol, however, is different than the “standard desktop application” protocol (xdg_shell), which is the one that the Wayland backend in many compositors use. Supporting it would mean adding another backend.
There were also caveats about performance, but I’m not sure about those. In the graphics side, I think that there wouldn’t be composition in this case (the fullscreen client’s surface would be scanned out directly), so there wouldn’t be copying overhead. But I’m less sure about input. Input is not a memory region that you can pass a pointer to; input is a stream of events. These events are transmitted in the protocol, from the compositor to the client, so adding another compositor would mean making the chain longer. Again, I’m unsure here, but there were concerns about gaming and other real-time tasks, I think, because of potential input lag.
All in all, managing complex things like input has already been made easier by helper libraries (libinput, for input), which could happen too for DRM, Wayland and X11. And there is discussion and work around libweston, which also aims at making it easier to develop Wayland compositors (in fact, recently there’s been some work around backends so as to make them more suitable).
True, video modesetting doesn’t apply to wayland/X11 backends. These backends would simply exist for the sake of testing or whatever else. However, this server would also need to account for non-DRM backends .. backends that would require directly accessing the graphics card (for most *BSDs and older versions of linux). Of course, this would take a very long time to do, but the theory is to simply create a server that can abstract all of these.
The other issue is, again, a standardized method for video modesetting on-the-fly. This is something that wayland doesn’t touch, since the wayland protocol is meant for compositing windows. Creating an extension for this would almost defeat wayland’s purpose, I think.
Yes, performance might be an issue, but in terms of non-accelerated pixel handling, there would be support for shared memory, which would greatly reduce the lag of sending pixels by hand. For hardware accelerated surfaces (like OpenGL), correct me if I’m wrong, but I think it would bypass the server entirely and be sent directly to the graphics card. Input would probably be slower, since, as you say, it would need to be sent via unix sockets. Not that unix sockets are slow (on my computer, it can handle 14GB/s), but yes, they do present latency. We would have to see how much of a delay this presents.
Using helper libraries would seem like a good idea, however, libraries don’t account for changing settings on-the-fly unless they create a server themselves (which, of course, entirely voids the purpose of creating a shared library instead of a server). In the case of video modesetting, IMO, this is crucial. In the case of input, I also think this is important (say you want to change the driver for a certain input device).
I am currently working on a proof-of-concept implementation of both of these to hopefully get a better sense of what issues this would present, performance lags, etc.
I oppose this stack. Unnecessary and just complicates things. We have yet to see, how wayland turns out.
Please no don’t do this idea ever.
Main reason we want to get away from X11 is server hell. It comes down to kernel allocating time slices. X11 and compostitor messaging as you see in the following diagram http://wayland.freedesktop.org/architecture.html can stall as kernel has allocated timeslice to compositor it sends message to X11 server and then everything has to stall until the X11 server gets another timeslice.
Next is what do we want to deprecate. Exactly why cannot X11 and Wayland not be virtual KMS/DRI backend. So Compositor just has to worry about KMS. We have pty(pseudoterminal) for text based terminals so why can we not have pseudo kms/dri that are Windows in X11 and other Wayland solutions. Since this would be a kernel mode path the kernel can allocate time-slice correctly.
–Linux Framebuffer, Headless [a fake rendering device], RDP, Raspberry Pi, Wayland, and X11–
All of these can be pseudo kms/dri.
–That’s not the only issue though. Another big problem is that there is then no standard way to configure the display.–
Again don’t reinvent wheel. KMS(Kernel Mode Setting)defines how to configure display, Providing a interface to request a KMS change could have a place. But saying there is no standard way is kinda incorrect. The other ways of modesetting are questionable. Remember Linux kernel framebuffer is marked to be removed from the kernel some point in future. X11 is marked to be in time only used under Wayland/Mir. Even the PI is going KMS/DRI drivers. Delete everything marked for deprecation and group correctly problem becomes clear..
KMS/DRM and Fake Rendering device (Headless , RDP, inside Wayland and Mir)
Yes Headless RDP and running inside Wayland/Mir/X11 are all the same problem you need a fake rendering device. Problem is no fake rendering device to use.
What makes sense to me is.
KMS/DRM and pseudo KMS/DRM for (Headless , RDP, inside Wayland and Mir)
That way compositor only needs 1 set or rendering code for KMS/DRM then everything is made look KMS/DRM..
libinput already processes to udev. Using udev directly equals implement almost everything that libinput has yourself. Also udev multiplexing should be a kernel level thing.
True, performance is the main concern I had while toying around with this concept. I’m just not sure how else it could be made.
I want to focus on your point for “Another big problem is that there is then no standard way to configure the display”.
Yes, DRM is a standard for configuring the display … however, it can only configure a virtual console. You can’t use libdrm to configure the display of an already-running wayland compositor (or anything else that has set the mode). This is why I believe that it’s necessary to have a display server, something that will act as a sort of proxy for DRM so that applications will be able to configure the display on-the-fly. Otherwise, I’m honestly not sure how this could be done.
For the rest, I understand your point about only having a DRM backend, however I’m not sure how this could be done … I haven’t really touched linux kernel development. It seems like an interesting idea, but I don’t know enough about this to really comment on it, sorry. I’m still just trying to understand working with DRM itself XD
However, I am not crazy about using DRM and udev as the only backends. We are leaving *BSDs out of the question here. Part of the reason behind making this server was to actually allow wayland to run under *BSDs without needing to drastically modify the codebase of the kernels.
–However, I am not crazy about using DRM and udev as the only backends. We are leaving *BSDs out of the question here. Part of the reason behind making this server was to actually allow wayland to run under *BSDs without needing to drastically modify the codebase of the kernels.–
KMS/DRM will come to BSD completely. Freebsd is already started getting rid of frame-buffer.
Libinput is already being ported to BSD. So you will have a BSD and Linux version of Libinput. Transformations you need to do from the raw hardware device are fairly much the same no matter the OS.
–You can’t use libdrm to configure the display of an already-running wayland compositor (or anything else that has set the mode).–
This is for very good reason. Wayland compositor is not going to like screen resized without it knowing. For security Wayland is in control of the screen. Like application cannot screen shot without permission. So resizing screen compositor has to approve.
Something you have missed is the wayland/mir compositor and X11 server themselves are display servers, So we have a display server. So exactly what is the problem and solutions.
Problem you want application to be able to send display setting change requests with the least amount of code. So you want to pick one of the existing and expand it role not add another item. The DRM framework is something that we can expand to cover all the graphical back-ends. Following this logic there are 2 solution paths that you will possible want to-do both.
Solution 1 Library replacement.
Libdrm is configured to go to real hardware normally but it does not have to be. DRM design say you need to send message to kernel use libdrm. There is no reason why a libdrm_x11, libdrm_wayland, libdrm_rdp … cannot exist as wrapper libraries redirecting libdrm ABI to something design to message userspace or pretend to be a screen.
Solution 2 Driver alterations.
Option A) Simplest here is software drivers
So you have the kms/drm driver the compositor it connected to. Then you have fuse or equal running providing another kms/drm driver that the application connects to and can send what ever DRM messages you like to that are either approved or rejected by the compositor/x11……
A software driver is the best solution for headless since you most likely don’t have a driver already.. Software driver does not have in fact hardware backing it is just software redirecting requests where ever its been told to.
Option B) Alter the way DRM drivers lock settings adding another locking mode.
A locking mode to always allow a request to change screen settings and so on be sent to driver but the process holding the screen lock gets messaged about them to allow or refuse.
Most likely at driver level both A and B would have to be done to cover all usage cases.
The advantage of the list paths comes down to time-slices. A library and driver share the time-slice with the process calling them where servers don’t. You want to keep you server number to a bare min. Also you don’t want to increase loaded libraries too much.
The driver solutions are most likely what long term goal should be. The library fixes would improve the situation now. From my eyes we should be able to get down to libdrm and libinput being all that is required for a basic graphical application. Now accessing copy-paste and other things would mean you would need to talk wayland/mir/x11 protocols and maybe this could be hidden behind libwayland. So deprecating need to use X11 or mir libraries.
–Personally, I believe in users having the power to change the actual video mode or gamma ramp etc. on-the-fly, regardless of whether or not this will screw up the display.–
I know this is crossing threads. But I don’t think applications should have the right to-do this without user permission to-do so. In a wayland or mir setup the item granting permission would be the compositor. But I do believe applications should have the means to put in the request preferable by normal libdrm methods. So it does not matter if an application is running stand alone or under a compositor it has the same code path to set screen setting. This would be good for games wanting full screen and full resources of the system..
Objective should be shortest path to hardware with the path only getting longer when there is a need. This is why adding a display server should be completely off the table. libdrm replacement is only making longer where there is a need. The driver alterations are also only make longer when there is a need.
If you want to have a serious conversation, come to wayland-devel@ mailing list.
The very point why we do not have video mode changing protocol extension in Wayland is that we do not want arbitrary programs to mess up the display behind the DE’s back. That is the only reason. We believe that the display must be completely under the control of the DE. If a DE wants to hand out that power. Changing the video mode from under a DE compositor’s feet sounds painful.
Your conception on protocol vs. library API seems a bit confused. OpenGL is a well-known example of an API specification, which vendors then implement as they see fit. In that respect, API is no different to protocol.
Have you ever heard of https://www.khronos.org/openwf/ ? Not that anyone actually ever implemented those, AFAIK, but it was an attempt.
I was kind of scared to send an email there, since I don’t know much about this, and am not the most articulate of people either XD but yeah, it would probably be the better place to discuss this. I’ll send an email shortly
I see, I thought it was because wayland just wanted to focus on windowing. However, I understand the reason .. a member from phoronix also wrote this (further clarifying your point): ‘that was probably one of best things in “why wayland”. compositor can simply achieve better result than switching resolution and there is not one single flicker which goes exactly with “every frame is perfect”‘
However, though I understand and respect the decision that took place, personally, I don’t agree with this. Personally, I believe in users having the power to change the actual video mode or gamma ramp etc. on-the-fly, regardless of whether or not this will screw up the display.
So instead, what I’m thinking is for this to become “yet another backend” in libweston. So that users, if they wish, can use this backend in order to be able to change the video mode, gamma ramp, etc. I personally require something like this in order to switch to wayland, since I use a software that changes the gamma ramp in order to match kelvin temperatures (makes it easier on the eyes for me).
I understand the difference between protocol vs API specification. An API specification is an abstract API that can be implemented by any application. Useful for portability …. however, this isn’t the only goal of this software. The main reason behind making this is the whole changing-video-modes-and-gamma-ramps-on-the-fly thing. Applications need to talk to a server that handles the display, hence the need for a protocol.
OpenWF seems interesting .. however, I’m not sure that it allows applications to do video modesetting on-the-fly .. it seems to only be an API for compositors to use in order to manage the display.
I personally love the idea. We basically did something very similar for https://github.com/openxt and ran into basically the same problem that you did. Our goal is to provide a display backend for multiple VMs, so we were not attempting to create a window manager per se, but the basic idea was the same. Good luck and let us know if there is anything we can do to help.
Thank you very much for your offer!! I am checking out the source code now, I take it that ‘input’ is the repository in question?
Also, which problem are you talking about?
At the moment, surfman is where we have our LibDRM code is, but we have a brand new display technology based on DRM that has a lot of designs from wayland and Mir that should be open sourced in the next couple of months. If what you are describing here had been done, we likely would have used that instead of rolling our own. In our case, we didn’t need wayland (don’t apply to VMs at all), and thus could not use Weston either. There is however a lot of code in Weston that manages the displays that could have been shared. My guess is, there are a lot of projects out there that use DRM, but don’t use wayland / Weston, all doing the same thing. Seems like a lot of duplication to me.
Our case isn’t really the same problem your trying to solve, but something like your describing really has more then one benefit.
Pingback: Follow up on the non-windowing display server idea | lkubuntu
I wrote quite a bit of a reply here, because there are so many misunderstandings in flight here. Unfortunately I lost it due Google+ auth problems. It is a pain to try and have a discussion in the comments of a blog post.
Don’t confuse users with arbitrary programs, please. We are in the middle of a great paradigm shift with display servers, and Wayland is not only for desktops. The “no display configuration extension” discussion applies only to desktops. If you want to see a whole different Wayland world that is nothing like the desktop, check out ivi-shell from Weston. Or the fullscreen-shell, for that matter.