GSoC 2015: Android-based Remote Display – Final Report

This is my final report for my GSoC 2015 project with BeagleBoard. The idea of the project was to allow an Android device to be used as USB mouse, keyboard, speaker and display device i.e replace 4 different type of device with a single android phone/tablet. Though the project was focused mainly for BeagleBone Black, but my goal was to make it usable with any Linux distribution.

For this project I had to work on 4 different areas:

  1. USB Display Driver
  2. USB Input Driver (mouse/keyboard)
  3. Sound Card Driver (USB speaker)
  4. Android Application

I’ll try to give a brief description of my contribution on each area. I’ll also try to add some tutorial if I get some time.

  • USB Display Driver: This was a complete project for GSoC 2014. My responsibility was to improve it further focusing on some specific issues. The display driver was developed based on the codebase of DisplayLink‘s framebuffer device driver [udlfb]. This driver works on the principle of Page Fault. It registers for a callback function with the kernel. Each time there is a page fault, the kernel notifies our driver through the callback function. Then its our drivers responsibility to update the dirty pages on the android devices over a USB connection. Though the actual kernel notification doesn’t go as simple as I said, but that’s the gist. Interested reader can read more about it here.

As I said, on a page fault, we need to go through all the dirty pages and update those in our USB display which is an android device in our case. Take a look at the following pseudo code segment:

list_for_each_entry(modified-page-list) {


Here we’re just traversing a kernel linked list which contains the modified pages and rendering each page. What the render function does is simply pass it to our android device over USB connection.

Now the above approach is okay for a system with decent amount of cache memory. My laptop has a cache size of 4MB for which the above approach provided a decent frame rate on the android display. But in BeagleBone, the cache size is very limited, which means for same video, there will be huge number of page fault in BeagleBone compared to a high performance workstation.

As a comparison of the two system, I run a 7 seconds video in both system and counted the number of page faults. In my Ubuntu-14.04 Laptop with 4GB RAM and 4MB Cache, I had a total 5640 page faults, whereas for same video I had 21731 page faults in BeagleBone. So the above approach try to send huge amount of data over BeagleBone USB connection. Now my calculation shows the BeagleBone USB bandwidth is 3.5 MB/s when it is connected to the android display. Our page size is 4KB. With each page we have to send 4 bytes of header information. So each transfer consists of 4100 bytes of data.

Now do a simple calculation: (4100 * 21731) / (3.5 * 1024 * 1024) = 24.28 seconds. This simple calculation shows that the above approach needs 24.28 seconds to transfer a 7 second video data. This was one of the major area which needed improvements.

My major contribution on display driver was to fix the above problem as far as possible. To solve this I had to compromise video quality with USB bandwidth. I implemented Lazy Update method, which means I don’t transfer the dirty pages as soon as there is a page fault. Rather I wait for a few milliseconds with a hope that a dirty page will be updated soon in the near future and I’ll send the most recently updated page. Though this approach does reduce video quality, but it solves the extended video time issue. Also we can’t improve the USB bandwidth, so we are kind of out of options here. The mentioned approach is easy to implement. All I did is to keep an indexed array. Each time a page fault occurs, I mark the dirty pages index on the array. I implemented a kernel array which periodically traverse this indexed array and send the marked dirty pages over USB connection.

My test shows that with this approach we can only have 10 FPS video on our android display. This is okay for reading or writing using our display, but not so cool for playing videos.

  • USB Input Drvier: As part of the project, I developed a USB mouse and keyboard driver which allows us to use the android device’s touch pad and soft key pad as a USB mouse & keyboard. Almost all keys including function keys (F1-F12) and methakeys(SHIFT, PgDn, PgUp etc) are included in the android app and taken care of from the driver side. Also mouse movement, mouse left/right click, double click are taken care of from both application and driver side.
  • Sound card driver: Supporting audio over USB was another goal of the project. We wanted the Android device to be used as a USB speaker. Enthusiastic readers might already understand that for this the best solution would be to implement a sound card driver. So when we attach our android phone on the system, the kernel will recognize it as a sound card and register it for all audio streams.

Linux kernel comes with a software framework named ALSA which provides necessary API for sound card drivers. After successful registration with the ALSA subsystem, our driver only needs to take care of the audio buffer pointer and stream transfer over USB. During the registration process we have to define the audio parameters like sampling rate, number of channels etc. The higher the sampling rate, the better the sound quality is. But higher sampling rate also means more data on the audio stream. Due to the limited USB bandwidth, we didn’t have the luxury of the best quality audio. For our audio driver I used a sampling rate of 16 KHz as at this rate the sound quality was acceptable with a video quality degradation. For same reason I used MONO stream instead of STEREO.

As I already said we can get only 10 FPS video when there is no audio stream. Now audio stream adds more pressure on our limited USB bandwidth. So I had to further reduce video quality when there is some active audio stream. If there is some active audio stream, we get 3 FPS video acceptable audio. To be noted, by acceptable audio I mean perceptible audio. We don’t want interrupt in the audio stream as that would clearly make it useless. So during active audio, I put preference on the quality of audio stream.

  • Summary:

Input Driver Performance                    Satisfactory

Audio Driver Performance                   Satisfactory

Video Timing                                       Satisfactory

Video Driver Performance                   Not up to Expectation (10 FPS only)

Audio + Video Performance                Audio up to expectation, Video below expectation

  • Future Work: I’ve found a bug which I’ll try to solve. I would also love to contribute patches for new bugs. Anyone is more than welcome to open any new issue in driver tree or application tree.

Leave a Reply