Window manager showing split view of home (left) and of turbine cam (right).

Architecture of Qt Embedded Systems: Single vs. Multiple GUI Applications

Our quest for a successful architecture of a harvester terminal continues. In this episode, we discuss when a system with a single application is good enough and when a system with multiple applications and a window manager is the better choice. Sometimes, a single-application system with a window manager is just the right choice.

Introduction

In the initial episode Gettting Started of the series on the Architecture of Qt Embedded Systems, I posed over 50 questions that help unearth the architecturally significant requirements (ASRs) for Qt embedded systems. In this post, I am going to answer some of the questions from the area Applications and Window Manager.

  • When is a system with a single GUI application and without a window manager good enough?
  • How does a system with a single GUI application benefit from a window manager?
  • When is a system with multiple GUI applications and with a window manager the best choice?
  • What are the requirements on the window manager?
Architecture of Qt Embedded Systems
High-level architecture of Qt Embedded Systems

My running example is a fictitious driver terminal for a harvester architected in 2021. We assume that the terminal runs on a custom Linux distro built with Yocto or Buildroot. We’ll discuss in a later episode why Linux is a good choice for the operating system of a harvester terminal.

Constraint. The system runs on a custom Linux distro built with Yocto or Buildroot.

Single GUI Application

The driver terminals of the KRONE and ROPA harvesters run a single GUI application full-screen. As only one application is rendering its GUI onto one window, we don’t need window managers like X11 or Wayland compositors (e.g. Weston). The QPA plugin EGLFS is all we need. For Linux, EGL converts OpenGL ES commands into the frame buffer content. FS in EGLFS stands for full-screen.

EGLFS supports QML applications displayed in a single top-level QWindow or QQuickView, OpenGL applications displayed in a QOpenGLWidget, and even QWidget-based applications displayed in a QWidget. QWidget-based applications are rendered in software using the CPU instead of the GPU. Other top-level widgets like dialogs, pop-up menus or combo boxes are composed into the single top-level window.

EGLFS cannot open a second window. It doesn’t allow the mixing of QML GUI parts and QWidget-based GUI parts, as this would require two top-level windows for rendering. Qt’s drag-and-drop and copy-and-paste features don’t work either, because they must also work between two applications rendered on two separate windows.

Despite its limitations, EGLFS is a good and common choice for Qt embedded systems. EGLFS is well-suited for QML applications benefiting from graphics acceleration with OpenGL ES 2.0 or later. Harvester terminals display many frequently changing parameters (e.g., vehicle speed, engine speed, harvested area) in digital or analogue gauges. Terminals also use animations to move between screens and dialogs.

OpenGL acceleration requires the SoC to have a GPU. The i.MX6 and i.MX8 SoCs (see overview of Toradex Colibri SoCs) have a GPU (called Vivante GPU). This gives rise to the following influential functional requirement.

Requirement. If the SoC has a GPU with support for OpenGL ES 2 or newer, terminals with a single GUI application should use EGLFS.

The i.MX6 SoCs, the i.MX8 SoCs and even the older i.MX53 SoC are powerful enough to run a simple window manager like a Wayland compositor. Although Wayland has been around since 2008, only the third generation of the KRONE terminals use it. The ROPA terminals don’t use it. A short trip down memory lane reveals different ASRs shaping the architecture decisions. Timing plays a central role, too.

When I started building the first KRONE terminal in 2012, the terminal ran on an i.MX53 with a single Cortex-A8 core and a GPU. The i.MX53 didn’t support Wayland and would never do. X11 was too resource-hungry for the i.MX53 and too hard to configure.

Wayland would have made the integration of a VNC server much easier and would have saved us about a month of work. The VNC server transfers the content of the frame buffer to any remote computer. EGLFS accesses the same frame buffer to render the GUI. Wayland is a protocol to control the access of multiple processes to the frame buffer.

All pieces would have been in place nicely, if Wayland had been available for the i.MX53. As it was not, we were forced to implement the sharing of the frame buffer ourselves (see the post Remote Support for Harvester Terminal via VNC). The next four ASRs describe the situation at KRONE in 2012.

Constraint. The i.MX53 does not support Wayland.
Constraint. X11 is too resource-hungry for the i.MX53.
Requirement. The terminal shall run a VNC server.
Requirement. The terminal shall use EGLFS as its “window manager”.

When KRONE moved to an i.MX6 with four Cortex-A9 cores and a more powerful GPU in 2015, Wayland still wasn’t an option. It wasn’t integrated well with Qt. So, the second terminal generation still used EGLFS.

Influence. The development costs for a Wayland compositor are too high.
Influence. Telling X11 to draw windows of a given size at given positions and configuring X11 for optimal use of the GPU are very difficult and time-consuming.
Requirement. The terminal shall use EGLFS as its “window manager”.

For the third generation (from 2018), KRONE switched to the ISOBUS terminal CCI 1200 with the same i.MX6 SoC. The ISOBUS consortium had invested significant time and money to implement a window manager based on a Wayland compositor to run multiple standard and custom applications (see The Standard: New ISOBUS Terminal CCI 1200). KRONE used the ISOBUS terminals for its pulled machines. So, it was a good business decision to use the same terminals on the harvesters to get a bigger price discount for a higher volume of terminals.

Influence. KRONE’s pulled machines use the ISOBUS terminal. Using the same terminal for the harvesters, too, reduces the per-terminal cost.
Influence. The ISOBUS consortium sponsors the development of the new terminal including a Wayland-based window manager. The development costs are split over the consortium members.
Constraint. The ISOBUS terminal CCI 1200 shall be used for future developments. The terminal uses a Wayland compositor as the window manager.

ROPA started its new terminal development in 2017. The ISOBUS terminal, which KRONE used in its third generation, was one of the contenders. ROPA would have minimised its development risk by choosing the ISOBUS terminal. But they had a good reason to take a more risky path. ROPA wouldn’t be able to differentiate their terminal from the thousands of ISOBUS terminals.

ROPA partnered with Christ Electronic Systems to build their own terminal with a quad-core i.MX6. Christ’s custom-built Linux image didn’t provide Wayland. It didn’t even use the GPU. One of my first tasks was to add EGLFS support to the Yocto build. The situation hadn’t changed when I left the project in 2019. ROPA saw the following three ASRs in 2017.

Influence. ROPA wants a unique design for their terminal to differentiate their terminals from the thousands of other terminals on agricultural machines.
Constraint. The custom Linux image built with Yocto doesn’t provide Wayland.
Requirement. The terminal shall use EGLFS as its “window manager”.

Back from memory lane, we look at SoCs without GPUs like the i.MX7, i.MX6ULL, VF61 and VF50 SoCs (see overview of Toradex Colibri SoCs). This makes them considerably cheaper than their brethren with GPU. We can run QML applications on SoCs without GPU as well (see post The Qt Quick Graphics Stack in Qt 5.8). When we configure the Qt build with -no-opengl, Qt will render QML applications in software using the CPU.

Without a GPU, the terminal should display mostly static content. More than a few content changes, analogue instruments or animations quickly overburden the CPU. The GUI will stutter or even freeze. This discussion yields two ASRs for SoCs without GPU.

Constraint. Window managers (e.g., X11, Wayland compositor) are too resource-hungry for SoCs without GPU.
Requirement. If the SoC has no GPU, terminals with a single GUI application shall use Qt’s software renderer.

Multiple GUI Applications

We saw in the previous section that a window manager may be the right choice even for a Qt embedded system with a single GUI application. A VNC server adds a second application that accesses the frame buffer. It adds a second GUI application. Adding a window manager is the quickest way to integrate the VNC server.

Using a window manager for a system with a single application may be the convenient or future-proof choice. The latest Yocto versions will often provide Weston, the standard Wayland compositor, as the window manager by default. Switching to EGLFS would cause additional efforts. As we may need a window manager in 2-3 years any way, we may want to keep Weston (see my post Showing Wayland Compositor Fullscreen in Portrait Orientation).

The KRONE and ROPA terminals are made up of components like the home screen, user settings, ECU parameter settings, customer and field manager, diagnosis, calibration, climate control, updater and machine communication. All components live in a single process with multiple threads. The driver uses the components with very different frequencies: from almost always to hardly ever.

If one component crashes, the whole application crashes and becomes unusable. We can mitigate this risk by splitting up the single application into multiple applications. We then need a window manager to make the multiple GUI applications look like a single one. This is the classic architecture of infotainment systems in cars: a system with a window manager coordinating multiple GUI applications.

Quality. The applications essential for the harvest must have a reliability of 99.9%.

In other words, the system must not be down for more than 1 hour in 1000 operating hours. There is a good chance that the system gets through a 6-week harvest (7 days per week, 16 hours per day) without any problems. Less essential or inessential applications have lower or no reliability requirements. We must decide for each application how essential it is for the system and set a proper reliability.

Partitioning the system into multiple application is not sufficient to reach the reliability goals. It must be complemented by TDD, automated tests on all levels (system, components, units), continuous integration, code reviews and more.

Of course, partitioning a system into multiple applications doesn’t come for free. The applications cannot communicate by function calls any more but must use inter-process communication (IPC). Qt Remote Objects are a good choice, as the applications emit signals, call slots or use properties as before. Qt Remote Objects turn signals, slots and properties into IPC calls behind the scenes. We should ensure that the application doesn’t know that Qt Remote Objects are used, as we may have to use another IPC mechanism in the future.

Reliability is one important driver for moving to multiple applications. Extensibility is another one. Third parties can develop applications for field navigation, field and customer management, accounting, ECU firmware updates or telematics. We’ll soon see Media, Radio, Phone and other applications on the terminal – as we know them from cars. Harvester makers don’t need to develop these applications on their own but can buy them from third parties. They save good money.

Quality. The system shall be extensible by third-party applications.

Web applications are often used to achieve such an extensibility goal. They are written with Electron, ReactNative or another JavaScript framework and are executed in the runtime environment provided by Qt WebEngine. As I explained in my talk Qt vs. Web – Total Cost of Ownership, the user experience (UX) is wanting compared to the UX on native applications. This UX may, however, be good enough for third-party applications.

We can run applications on our smartphones and beam their GUI to the terminal’s display with Apple CarPlay, Google Auto or MirrorLink. Conceptually, the smartphone runs a VNC server that mirrors the app’s GUI to the terminal. The terminal runs a VNC client to display the app’s GUI in a window. Apple CarPlay, Google Auto and MirrorLink require the terminal to run something akin to a VNC client. The VNC client is the only software that is needed on the terminal.

We can also provide a Qt SDK, on which third parties can base the development of their applications. The Qt SDK, which we use for developing the core applications of the terminal, is a good basis. We enrich the SDK by some high-level APIs to system components like the customer and field manager, the error manager and digital twins of the ECUs. These system components could be services running as non-GUI applications with a clearly defined interface.

Window Manager

By default, the window manager shows the home application and offers a tool bar for switching between the core applications. The home application provides all the instruments and controls so that the driver can perform his normal harvesting tasks. The applications in the tool bar offer features that the driver needs frequently but that are not as important as the Home application.

Window manager showing home app with all the relevant information for normal harvesting tasks.
Window manager showing home app for normal harvesting tasks

The most important information and controls of the Home application must always be visible. When the driver taps on the Turbine Cam button, the window manager splits the screen in two and shows the Home application on the left and the video stream of the turbine cam on the right. When the driver taps on, say, the Fields button, the window manager replaces the Turbine Cam application by the application for field and customer management. The window manager removes the split and shows the Home application full screen, when the driver taps the Home button.

Window manager showing split view of home (left) and of turbine cam (right).
Window manager with a split view of Home app (left) and Turbine Cam app (right)

The window manager could split the right half vertically to show two apps on top of each other. In the upper and lower quarter, the driver can cycle through multiple apps by swiping. No matter what the window manager does in the right half, it always shows the Home app in the left half. We have found another four influential functional requirements.

Requirement. By default, the window manager shows the Home application full-screen.
Requirement. The window manager can split the display in two half windows and show a second application in the right window. The driver can cycle through several applications in the right window by swiping.
Requirement. The window manager can split the right window vertically. The driver can cycle through several applications in the upper and lower quarter window by swiping.
Requirement. The window manager must always show the Home application in the left window.

The terminal has more applications than fit in the tool bar. When the driver long-taps the Home button, the window manager shows all available applications in a grid. When the driver taps on an application icon, the window manager starts the application and shows it in the right window. When the driver drags an icon onto one of the icons in the tool bar, the window manager swaps the application in the tool bar with the application from the grid.

Window manager showing the app grid.
Window manager showing the grid of all available apps

The above requirements make it clear that the window manager must be able to split the screens into zones of given sizes and positions. The half and quarter windows and the tool bar are placed into these zones. The window manager can change the zone arrangement dynamically.

X11 is a desktop window manager that places windows where there is some space. Telling X11 to place a window of a given size at a given position is close to impossible. Direct rendering is scattered over several places in the X11 server and may not be supported by embedded systems. Rendering happens on the X11 server. The rendered data (the frame buffer content) must be copied back to the X11 client, which is inefficient. These are some of the reasons why X11 should be avoided on embedded systems.

Wayland, which is under MIT license, solves the performance problems of X11. Wayland-based window managers or Wayland compositors support window arrangements according to our specifications. Many board-support packages (BSPs) build Weston, the standard Wayland compositor, by default these days (see my post Showing Wayland Compositor Fullscreen in Portrait Orientation). Weston’s IVI shell demonstrates how to configure different zone arrangements. Unfortunately, the documentation of the IVI shell seems to be dated and its source code seems to be scattered over multiple projects. This is offset by Wayland being an actively developed project with a supportive community.

As Qt developers, we have a second option. We write our own Wayland compositor in QML based on the module Qt Wayland compositor. We can even use QML animations as visual cues for switching between applications in a zone. Starting with Qt 5.14, the Wayland Compositor and the Application Manager are not available under LGPLv3 any more but only under GPLv2+ or Qt Commercial. If we want to use Qt LGPLv3 in our system, we must base our compositor on the code in Qt 5.13. Hence, we would have to keep the Qt Wayland Compositor module up-to-date ourselves and maintain it.

If we can go with Qt LGPLv3, we’ll use Weston as the window manager. Otherwise with Qt Commercial, we can use Weston or we can develop our own Wayland compositor. We have found the last two ASRs of this episode.

Influence. Whether we use Weston or a custom Qt Wayland compositor depends on the Qt license used.
Requirement. The terminal shall use Weston as its window manager. The fallback window manager is a custom Qt Wayland compositor.

So Far in the Series

So far, the series on the Architecture of Qt Embedded Systems comprises the following posts.

  • Getting Started. Lists 50+ questions about the architecture of Qt embedded systems, defines architecturally significant requirements (ASRs) and classifies architectures in good, right and successful.
  • Operating Conditions. Explores how operating conditions like light, temperature, water, dust and vibration influence the system architecture.
  • Single vs. Multiple GUI Applications (this post). Discusses when a single-application system is good enough, when a multi-application system with a window manager is the best choice and when even a single-application system needs a window manager.

2 Comments

  1. jussi kauppinen
    2021/06/02

    Nice article ! Using window manager builds the base also for mixing applications using different qt versions I believe. What are your thoughts on that?

    1. Burkhard Stubert
      2021/06/07

      A window manager (e.g., a Wayland compositor) would indeed allow applications to use different Qt versions or any other GUI toolkit (e.g., Clutter, EFL, GTK) that can render to Wayland surfaces. As a consequence, you must ensure that the application links against the correct (shared) Qt libraries. You can do that by setting the runtime path or LD_LIBRARY_PATH, or by packaging the application together with its libraries (using CPack, Conan, Docker containers).

      You could also exploit Qt’s binary compatibility between minor releases. If your application was originally built against Qt 5.12, it will also link against any newer minor Qt versions like 5.13, 5.14 and 5.15.

Comments are closed.

Scroll to top