Update: Take Two
Linux was successfully ported to Wii by the gc-linux team as early as Feb 2008, shortly after the release of the Twilight hack. In the following two years, huge progress and improvements were made to the port and some of the work has already made into mainline Linux kernel.
The fact that Wii resembles its predecessor, GameCube, to such a degree that Wii has often been labeled as ‘not next-gen’, played an important part in making the Wii port possible within such a short time frame, because the gc-linux people had already been working on the GC port of linux for many years. One of the many important similarities between GameCube and Wii is the use of YUY2 colour format in the framebuffer (XFB). There might have been valid reasons for Nintendo to choose such a format for their game consoles, but for the purpose of running Linux on these devices, this choice caused serious inconveniences.
In computer world, colour framebuffers on graphic cards have been predominantly using packed RGB formats for ages. Consequently, computer-oriented OS and software have grown to assume as much about the hardware and generally made no attempt of accommodating the possibility of ‘otherwise’. There certainly were many valid reasons for the choice of RGB for computer graphics, probably way more than there could have been for Nintendo’s choice of YUY2. Unfortunately, this ‘disagreement’ over formats between hardware and software meant more work for the porting guys, as they had to implement converting mechanisms so that the right colour could be displayed on the screen.
Analysis of the ‘Problem’
The way the issue is handled in the current Wii port of Linux is like this:
At the kernel level, the framebuffer driver, called ‘gcnfb’, advertises itself to everything else as only capable of displaying 16bit RGB565 format.This is because RGB565 matches the pixel data size of 2 bytes of YUY2, making it easier to handle resolution-related stuff. In other words, the driver fakes an RGB565 framebuffer and all unknowing programs will try to dump RGB565 data into it. The driver maps the physical framebuffer memory into kernel space and makes it available to other kernel modules such as fbcon (framebuffer console). The mapped framebuffer memory can in turn be mapped into user space for direct reading/writing. RGB565 -> YUY2 conversion is implemented in the driver, not only in a hackish way, but also specifically and exclusively for fbcon’s writes to framebuffer, which ensures correct display, most of the time, in text console mode. While using ‘Up’ arrow to access command line history or using ‘Backspace’ in nano editor, garbled colours are sometimes displayed, as a result of such an imperfect solution.
At the user space level, it’s up to individual programs themselves, that are directly accessing the framebuffer, to make sure that data written to the mapped memory are of the correct format and that data read from the mapped memory are properly converted before being manipulated by RGB-based processing. This means that all of such programs have to be patched in order to work. Programs accessing framebuffer through another program, for instance SDL applications running on top of SDL and X applications running on top of X, do not need to be patched if the underlying framebuffer-interfacing programs already incorporate proper YUY2-RGB565 conversions. In X, a special Xorg framebuffer driver called is available, which handles the format conversion issues, and since all X programs automatically use this driver if it is specified in xorg.conf, they will ‘just work’ if ‘cube’ is configured correctly. The issue with ‘cube’ is that it currently supports only one desktop resolution and makes no attempt to accommodate TV overscans, leaving parts of the display area invisible to some users.
In short, various programs at kernel and user space levels have to be patched directly or indirectly to be able to output correct display. This creates a system that is difficult to install/maintain, and leaves lots of console-mode programs not working because no patches have been developed for them and they are simply dumping RGB565 image data directly into the YUY2 framebuffer. For example, although you can get correct text display in a console mode shell, Debian Installer, which also runs in console mode, will give you screens like this one if run on top of Wii-Linux:
Wouldn’t it be nice if the Wii framebuffer driver could fake an RGB framebuffer more competently and handle all RGB-YUY2 conversions internally and transparently as far as everything else concerned? This way, no patching would be required and everything would ‘just work’ as on a ‘normal’ PC. Let me explain how this could be achieved with some simple hacks.