GSoC 2015 Proposal for BeagleBoard: Android-based Remote Display

About you


  • What is your name?

**************

  • What is your email address?

*************@gmail.com

  • What is your eLinux wiki username?

*********

  • What is your IRC nickname?

*********

  • What is the name of your School and in what country?

*****************

  • What is your primary language? (We have mentors who speak multiple languages and can match you with one of them if you’d prefer.)

****************

  • Where are you located, and what hours do you tend to work? (We also try to match mentors by general time zone if possible.)Have you participated in an open-source project before? If so, please send us URLs to your profile pages for those projects, or some other demonstration of the work that you have done in open-source. If not, why do you want to work on an open-source project this summer?

I’m located in *************************************.

About your project


  • What is the name of your project?

Android-based Remote Display

  • Describe your project in 10-20 sentences. What are you making? For whom are you making it, and why do they need it? What technologies (programming languages, etc.) will you be using?

The goal is to extend an existing project. The main focus of the previous project was to implement android based remote display: to create a USB framebuffer driver which will allow an android application to act as an USB monitor. My goal will be to extend that project with features of mouse, keyboard and audio playback giving the support of reduced number of cape/input device for BeagleBone developers.

I’ll use ‘C’ for kernel side USB device driver implementation and ‘Java’ for corresponding android application. The whole project has some subsection which needs to be addressed

Mouse/Keyboard: I’ll write an USB input device driver for mouse and keyboard functionality. For this part I’ll follow the existing implementation of input driver in linux kernel. I’ve implemented a start up of this feature[1][2]. In my implementation, I’m using data packet size of 8 bytes for input- 1 byte for mouse/keyboard/control identifier, 2 bytes for type of input event(MOUSE MOVE, MOUSE CLICK etc) and remaining bytes for actual payload. For keyboard, the payload is keycode index and for mouse it is direction and magnitude of movement.

Audio Playback: For BeagleBone, I’ll write an audio playback driver on top of ALSA. I’ve some very basic implementation[1][2] to achieve that and the remaining tasks will be tune it up. I’ll follow ALSA driver tutorial for creating a complete audio playback driver. For audio streaming in android app, the AudioTrack class from Android API will be used.

Multi-type Data Packet on One Link: Since we’ve only one interface to work with, we’ll need to send multiple types(audio/video/input) of data packet over a single link. So I’ll add some packet identification code in each packet as an identifier of the packet type. Right now my goal is to send same size packet for audio and video. So the packet header should be simple right now. One extra byte in each packet should be enough for identification. If we want to send variable size packet, we have to add that information in the packet header. In that case the packet header should be 5 bytes ( 1 byte identification, 4 bytes data length). I’ll start with packet size of 4100 bytes, where 1 byte will be packet type identifier, and 4096 bytes for actual payload. In each video packet, there’ll be two bytes of page index which is followed in previous project.

Minimize Setup Steps: Right now the setup of framebuffer requires a lot of steps. My goal will be reduce the number of setup steps by merging the ADK and UDLFB driver together. Also I’ll add support for more android devices by a generic header file containing VENDOR_ID list of most popular android devices. Overall goal of this step will be to let user use the tool by just connecting to USB and running the X-server.

Framebuffer Driver Code Cleanup: During my investigation of the UDLFB driver used in the existing project, I found some codes that are meant to be specifically for Displaylink Framebuffer Devices. I’ll give some time in cleaning up those part.

Error Check: Right now I’m not sure if there is any data loss in USB communication. For Audio, data loss is not significant. But for video header(containing page index), keyboard and mouse input device, data loss is significant as change in only one bit will cause scratchy display or incorrect input. I’ll discuss with the mentors about it more and if necessary I’ll implement error detection/correction in required area.

[1]  https://github.com/azizulhakim/beagleusb

[2]  https://github.com/azizulhakim/BeagleRemoteDisplay

  • What is the timeline for development of your project? The Summer of Code work period is about 11 weeks long; tell us what you will be working on each week.
    • Week #1 – #2:
      • ToDo: Integrate frame buffer driver with my input driver and audio driver project. Also integrate mouse, keyboard and audio streaming in android application. Main goal will be to integrate everything together and make sure all implemented feature works successfully after integration. I’ll also add a generic header file containing list of most popular android devices so that the driver will work on most of the android device.
    • Week #3:
      • ToDo: I’ll mostly work on the mouse driver and application part this week. I’ll take care of mouse double click, map the android app co-ordinates with BeagleBone co-ordinates and possible work on mouse drag-drop feature.
    • Week #4:
      • ToDo: This week I’ll work on keyboard driver and corresponding part of the android application. Right now I’ve implementation of a very simple input driver which takes care of ‘a-z0-9’ and some punctuation marks. I’ll try to extend that with meta-key(CAPS Lock, SHIFT etc) support. Also work on showing function keys(F1-F12) on the application side and corresponding features on driver side.
    • Week #5:
      • ToDo: Main goal of this week will be audio playback. I’ve already implemented an audio playback driver based on ALSA which I tested on Ubuntu. Right now it can play the audio stream but it has problem with proper syncrhonization. It moves very fast along the audio stream. I’ll work on finding out possible reasons of that and try to solve that.
    • Week #6 & #7:
      • ToDo: By this week I’ll expect to make the audio-video work together. I’m assuming there will be synchronization problem of audio and video stream. Right now this is the part of which I’m completely in dark. I don’t have any idea until I test both together. I’ll try to find out ways of synchronizing both streams together. If I get time I’ll also work on some code cleanup and test all features together.
    • Week #8:
      • ToDo: Modify framebuffer driver to use URB for bulk transfer. Also start work for merging ADK driver with framebuffer driver to reduce the setup steps of the driver.
    • Week #9:
      • ToDo: FrameBuffer code cleanup. Remove unnecessary codes written specifically for displaylink device. Also implement error correction/detection if required. I’ll also put some time for finding out possible reason of framebuffer crash in Ubuntu.
    • Week #10:
      • ToDo: This week will be specially for Integration test and bug fix.
    • Week #11:
      • ToDo: Work on documenting everything.
  • Convince us, in 5-15 sentences, that you will be able to successfully complete your project in the timeline you have described. This is usually where people describe their past experiences, credentials, prior projects, schoolwork, and that sort of thing, but be creative. Link to prior work or other resources as relevant. Provide references such as professors who know your work if you like. Please feel encouraged to visit our IRC channels, #beagle and #beagle-gsoc on irc.freenode.net, and ask for help.

I’m hard-working and passionate enough to complete this project. I’ve worked a lot for this project and made a good progress. Through my work, I’ve got a very good idea on how to achieve the remaining tasks. I’ve clear idea and necessary knowledge about the approaches I’ve to take to finish the project. I believe the most important thing is my passion toward this project. I started to prepare the project and wanted to make a simple demo for my proposal but end up with implementing a very good and some challenging parts of the project. I’m so keen to complete it that I’m still working on it and planned to finish it by myself even in the case that I don’t get to do the project. One important thing about myself is that, once I start something, I always dig deeper to finish it and will do the same for this project. In case I don’t get to do the project, I’ll be glad to help with the winning proposal and mentors and other developers in BeagleBoard community through IRC channel.

You and the community


  • If your project is successfully completed, what will its impact be on the BeagleBoard.org community? Consider who will use it and how it will save them effort. Give 3 answers, each 1-3 paragraphs in length. The first one should be yours. The other two should be answers received from feedback of members of the BeagleBoard.org community, at least one of whom should be a BeagleBoard.org GSoC mentor. Provide email contact information for non-GSoC mentors.
    • This project will be a great help to the BeagleBoard developer as it will significantly reduce the number of capes necessary for the development. Only an android device and an USB cable will reduce the necessity of LCD display, keyboard and mouse.
    • Praveen Kumar Pendyala(praveendath92): This would be a perfect complement to the existing remote display. Now that we figured a way to support all Android devices, this could be a perfect out-of-box working setup for beginners, especially those who are not much of a terminal aficionado. Lastly, the fact that this is a plug-and-play over ubiquitous Android devices makes it cool!
    • Vlad Victor Ungureanu (vvu): A lot of beaglebones black have the issue of not working out of the box with some HDMI displays. By including a usb display driver in the default debian distro and having it working with minimal effort helps the new users to have a working station only by using their android tablet/phone. Also the LCD issue can be leveraged like this. The main focus of this project I think it needs to be easier installation, better support for multiple devices and also good documentation.
  • What will you do if you get stuck on your project and your mentor isn’t around?

This project consists of multiple independent module. So if I ever get stuck on one module and my mentor isn’t around, I’ll always be able to switch to another module to make progress. In the unlikely case when I get stuck on every path, google and IRC channel will be my friend.

Miscellaneous


  • Please submit to the beagleboard-gsoc mailing list a statically-linked ARM Linux “hello world” style executable that prints out your name and the date. Please keep it under 1MB. Provide a link here to that executable as archived on the mailing list and provide any instructions required for invoking it. You are welcome to test it on an ARM QEMU environment. Please feel free to visit our IRC channels, #beagle and #beagle-gsoc on irc.freenode.net, and ask for help.

Posted on mailing list. Tested using qemu-arm-static and also tested on BeagleBone Black. The repository is available in github[1] with instructions.

[1] https://github.com/azizulhakim/gsoc-application

  • Is there anything else we should have asked you?

Yes. The question should be “Why do you think you can finish the project” and the answer is because of my work on this project during my preparation. I’ve already implemented an input driver and audio playback driver. I started work on the project and already overcame some of the most challenging parts(audio driver). The remaining tasks will be to integrate everything together and tune up for perfect operation. I’m still working on those issues and is sure to finish everything by the given timeline.

 

Leave a Reply