GSoC 2015: Android-based Remote Display: Weekly Update

  • Week #8:
    • I updated the build script significantly which will reduce users setup step. Same script can be used for building & installing the binaries and running X on frame buffer device. Also it will copy the ALSA .asoundrc file to the appropriate directory during the installation of the driver. So in short this script will reduce a lot of initial setup steps for user.
    • Did some code cleanup on frame buffer driver, audio driver and input driver.
  • Week #7:
    • Finally I fixed the audio issue in bone. So now all the drivers and the android application are fully functional.
    • Doing code cleanup. Now my plan is to do code cleanup and documentation and any change mentioned by the mentors.
  • Week #6: 
    • I implemented the lazy update method on framebuffer driver. So now the framebuffer driver doesn’t send to much data over USB connection and as a result we have an accurate timing of the video play time. But as a side effect, frame rate is hampered a little bit which won’t matter while working in terminal or reading or writing codes or mouse movement.
    • The above update doesn’t help much if we also use audio stream. So I dynamically reduced the frame rate when there is audio data. If there is no audio data, the frame rate will be raised to normal level which is now 10 FPS.
    • Also fixed some bugs on android side.
    • ToDo: I’ll try some optimization to increase the frame rate a bit more.
  • Week #5:
    • This week I was more involved in doing some performance testing on multiple platform rather than coding. I tested the USB transfer rate on multiple architecture using virtual machine. The architecture I tested are on Ubuntu, Arch Linux and Debian. The test results lead us to believe the smaller cache size might be responsible for higher page fault in bone.  Since we can’t reduce the number of page faults in bone some optimization needs to be done to solve our problem. Right now I’m thinking of lazy update meaning a page won’t be transferred over USB immediately on the hope that it’ll be updated again in near future.  I’m not sure how effective this might be or how complex the solution will be. Also it’ll reduce the FPS rate. Next week I’ll put some time on this. I’m also open to better suggestions.
  • Week #4:
    • This week I mostly tried to understand the usage of ffmpeg library. For laptop’s USB, we can get decent speed for both audio and video, but not when we run both together. In bone, we get enough bandwidth for audio, but not for video. Playing both audio and video together in bone is very slow.
    • Also trying to fix some issues related to smaller resolution with the hope of little speed up in smaller resolution.
  • Week #3:
    • This week I worked on audio driver. I fixed the audio driver fast-forward problem. So the driver can play audio file correctly.
    • I found the reason of audio driver unload issue and practically it is not possible to unload the audio driver as long as the device is connected to USB port.
    • Issues:
      • Sometimes there is continuous popping noise with the original audio data. So even though the audio goes on perfectly, there are some background noise.
      • The audio driver still causes problem in bone when I try to play something for a 2nd time. It seems to be a tricky issue as I couldn’t solve it after putting enormous time on it.
  • Week #2:
    • This week I worked mainly on keyboard features. I added most of the key’s in both android and driver side.
    • Added a popup for special keys like function key, PgDn, PgUp, DELETE, INSERT etc.
    • With that most of the works on input driver is finished. A demo available here.
    • Issues: 
      • Android doesn’t fire any event for some of the important soft keys like “BackSpace” key. So for now I’ve added an extra key in the “Special Key Popup” for “BackSpace”. I’ll try to find an efficient solution for “BackSpace” key using “TextWatcher” class.
      • Audio driver unloading issue still exists. I couldn’t find any solution for that. So far it seems ALSA attach the driver with the device and as long as the device is connected ALSA considers the driver is being used by the device.
    • Plan for Next Week: Focus on Audio driver. Make the android device work as a fully functional USB speaker.
  • Week #1:
    • I’m done with integrating 3 modules together and tested all together in bone.
    • Added some configuration settings in both driver and app so the user can optionally drop frames to speed up video play time.
    • Did some experiments for comparing the transfer speed of bone and my laptop which shows we need at least 4x speed up in USB transfer for bone. Which means the compression times + transfer time should be 1/4th of the uncompressed transfer time.
    • Redesigned input driver functions and data structures to pass more than one input event at a time. It is necessary for mouse drag drop feature and meta key(SHIFT, CTRL etc) functions and took care of those events from application side. So we can now use the android app as a fully functional mouse. Mouse single/double/right click, drag-drop are all in place.
    • Issues:
      • Audio driver can’t be unloaded without disconnecting from USB port
      • Abrupt disconnection from USB causes framebuffer driver crash.
  • Community Bonding Period:
    • I experimented on several approach for sending framebuffer data to improve its performance. In previous implementation, the data transfer was a blocking data transfer which caused very low FPS rate in the application side. I’m going to give a very brief descriptions of some other approaches I tested on:
      • RingBuffer & Kernel Thread: I created a ring buffer which holds the new frames. So from framebuffer side whenever there is a new frame, we just put it in the ring buffer. I created a separate kernel thread which picks up a frame from the ring buffer and send it over USB connection(blocking without URB).
        • Problem: Due to the blocking nature of our data transfer, their is no improvement of FPS in the application side. A possible solution might be to use non-blocking transfer. I had some implementation of that also which I’ve described in the URB Transfer section.
      • URB Transfer: I used URB for sending data over USB. Since URB is non-blocking, I had better FPS rate in application side.
        • Problem: The USB core maintains all URB requests in a FIFO structure and dispatch them one by one to the endpoints. Because of the massive data rate and slow transfer speed, I noticed a significant lag of video frames in the application side.
      • RLE Implementation: I did a very straight-forward simple RLE encoding for the data stream. But didn’t see any significant reduction in the output buffer size. I talked with mentors and understood that the reason was due to the use byte level granularity during encoding scheme. We have two byte in each pixel and the frame consists of pixels. Which means during RLE encoding we have to compare in pixel granularity, not in byte granularity. I’ll do some work with pixel level granularity and check the results though I don’t think its also not going to be effective, because RLE reduces the stream significantly only if there are a lot of same consecutive bytes/pixels in the stream which will not be the case when we are going to play some video file.
  • Before Proposal:

I implemented a simple USB input driver and a USB Audio playback driver on top of ALSA with minimal functionality. Both the drivers were tested mainly on computer, no test on beaglebone. The result in the computer was satisfactory. The input driver didn’t show any problem whatsoever. The Audio driver had a fast-forward issue which means whenever I run any audio stream it fast forwards though the stream. No crash issue with none of the driver so far.

Leave a Reply