How to pre-populate Guava CacheBuilder cache

I haven't blogged for a while now and this is my attempt at a fresh start. In the meanwhile I have started a new job in San Francisco leading the Android apps team at Betfair and it has been hectic. I will try to be more regular in my posts here as there are a lot of things I want to share.

In my current job we are using Guava CacheBuilder, its part of Guava Collections open source library for implementing LRU cache for some data in our Android app. But there is one deficiency in this, it does not allow us by default to prepopulate the cache. When an object is requested, CacheBuilder looks for it in its cache, if it can't find it, CacheBuilder fetches it, caches it and then returns it to the user. This wasn't what I needed as I needed to pre-populate the cache. Here is how I did it. First initialize the cache.

 Cache<Integer, NavigationNode> lruCache =   
     CacheBuilder  
     .newBuilder()  
     .concurrencyLevel(4)  
     .maximumSize(500)  
     .expireAfterWrite(3, TimeUnit.MINUTES)  
     .build(  
         new CacheLoader() {  
           @Override  
           public NavigationNode load(Integer nodeId)  
               throws Exception {  
             return null;  
           }  
         }  
      );  

Then this is how you pre-populate the cache:

 lruCache.asMap().putIfAbsent(id, navNode);  

lruCache.asMap() returns a view of the entries stored in the cache as a thread-safe map. And modifications to this map are same as modifiying the cache. So putIfAbsent() just puts the navNode object in cache, if it is not already in it. Now if we use this line in a for loop that makes it trivial to pre-populate the cache.

Usefulness of Commit Emails from Version Control Systems

A few days back I found this link on Hacker News about a blog post that talked about best practices for using version control systems. I also thought about something which I think ties in very well with his blog post. I recommend setting up commit emails for every commit done, whatever the VCS may be, Git, SVN, Perforce etc. Sometimes the usefulness of this system is not completely understood, so I will talk about here. My notes on this are based on my experience with using commit emails from SVN (set up with mailer.py and svnspam), but I think it should be applicable to any version control system.

Code Review Tool: Ideally all code that is checked-in should be code reviewed beforehand. Sometimes it is not possible to do so, maybe the check-in is of two lines or there are time constraints. Then you can just say "check it in, I will review the commit email". And then based on what you see in that email you can recommend changes. Now this is not as good as code review before check-in but this still helps. This way a lead engineer can also recommend further changes in code if necessary.

Enforce Standards: Pre-commit hooks can be used to ensure agreed upon formatting, but sometimes coding conventions are harder to enforce. Having commit emails ensures that a lead engineer can request more changes in code to ensure code confirms to a standard and is consistent with existing conventions.

Watch Modules: With email filters and module names in commit email subjects, it is very easy to setup filters to track specific modules. In a large company with a number of modules in development this makes it easy watch specific modules, even ones that are being developed by other teams.

Increased Visibility: Anyone in the company can ask to be included in this mailing list. This is especially useful for managers who want to track progress on various modules and don't want to install version control tools.

Handy record of releases: Many times I have had to look up a release version I created months back. With the commit emails I can just search my inbox for the tag and thus get all the info I need. I don't have to use Tortoise Repo-browser to look up the tag.

Merge requests: Commit emails can also be used by the team to forward specific changes to one another, in order to be merged in another modules. This is very useful for small changes that need to be merged, and an email will be sufficient for that.

I am sure there are other uses of this kind of system, do you have this setup where you work? Have you found it useful in other ways? Let me know.

How to compile Android source on Snow Leopard (Mac OS X 10.6.7)

So after that relatively long hiatus I am back. I have been busy mainly working on Android, and to a limited extent, iPhone apps. I wanted to compile android from source, I had heard that it wasn't too difficult. I had recently bought a MacBook Pro laptop and dedicated a weekend to this project. Well, like many things, it took longer, and was more involved and had several missteps. I will document what worked for me here, in case anyone else wants to do the same.

Google's instructions for this are provided here. There are other excellent blog posts about this, here and here. Unfortunately they all were about Android install on Mac OS x 10.5 or older. They also didn't talk about what happens if you install Xcode 4.

My Mac environment for this exercise is Snow Leopard Mac OS X 10.6.7 and Java 6 (1.6.0_24-b07-334). I also had Xcode 4 installed. Also ensure that you have a Google account, as you will be prompted for it during installation.

(Incidentally if you want to remove a version of Java and install another in Snow Leopard, delete /System/Library/Frameworks/JavaVM.framework folder and run Software Update on your mac. This will install it again).

So as described in Google website, run Disk Utility app on your Mac to create a 12 GHz disk image, with the format being Mac OS Extended (Case-sensitive, Journaled). It gets mounted in /Volumes, you can call it any name.

Now you have to install Macports. Some of the Google group posts say that it isn't needed for Snow Leopard, but I had already installed it. Also, to install it you have to compile it from source (following website instructions). Apparently the .pkg installer needed root permission but didn't not prompt for password. But with source compile we can use sudo command to work around that. After installing Macports don't forget to synchronize with the MacPort's rsync server.

Now ensure that /opt/local/bin is added to path. Macport install automatically modifies your .bash_profile to do that. If it is not setup correctly run 

$ export PATH=/opt/local/bin:$PATH.

Next follow the instructions on Google website to install gmake libsdl etc. And then revert gmake to older 3.81 version as detailed there.

After this add ulimit -S -n 1024 to your .bash_profile file to increase simultaneous file descriptor cap.

Now download the source using the Repo tool, set it your path, by following Google website instructions. Note that when you run repo sync it will take 10 or so minutes to download all source code.

Now run the following commands to compile:

$ source build/envsetup.sh

$ lunch full-eng

$ make -j2

Now on my MacBook Pro with Snow Leopard it will error out saying MacOSX10.5sdk folder not found. So download Xcode 3.2.6 from apple developer website, install it NOT in /Developer folder but in /Developer326 folder (otherwise it will overwrite your existing Xcode 4 install). Now copy /Developer326/SDKs/MacOSX10.5.sdk folder into /Developer/SDKs folder. Run again make -j2 and the compile will run for 25 or so minutes and you are done. You can now run emulator command to launch the emulator.

Java OTA download error messages

In Java ME the success or failure of an OTA download attempt is reported by means of installation status reports. These status reports are posted to the server using the URL specified in the following JAD attributes: MIDlet-Install-Notify  for installations and MIDlet-Delete-Notify for deletions. Otherwise when the application installation fails it just displays the status code and a rather cryptic error message. I am going to document what I know about these errors.

900: Success: This means that the app has been successfully installed.

901: Insufficient Memory: If there is insufficient memory to store the app on the device, it returns this error.

902: User Canceled: It is returned when the user cancels the application download.

903: Loss of service: This status code is returned if the installation failed due to a network failure. Note that usually nothing is sent to server as the network connectivity itself is not available.

904: JAR size mismatch: If the downloaded application JAR file size does not match the size specified in the JAD file, this error is returned.

905: Attribute Mismatch: This error indicates that there is a mismatch between the JAD file attributes and those in the JAR file's manifest. The JAD file attributes (both key and value) that are in manifest file in JAR should match exactly.

906: Invalid Descriptor: The following attributes are mandatory and should be present in all JAD files.

1. MIDlet-Name
2. MIDlet-Version
3. MIDlet-Vendor
4. MIDlet-Jar-URL
5. MIDlet-Jar-Size

And if any of these attributes are also in JAR file's manifest, then they will have to match exactly or else you will get this error.

In some cases, we need to specify MIDlet-Data-Size attribute in JAD and manifest files. This attribute will determine the amount of RMS allocated for the app. If you specify too much then the installation may be denied. Usually for apps not using RMS this attribute is not defined in JAD or manifest files. And other times when this error crops up, setting this attribute (with a 1024 size) fixes it.

907: Invalid JAR:  This is a server error. It occurs when the one of the following HTTP errors happen: 400, 402, 404, 405, 406, 408 etc. Mime type setting on server may be one possible cause. One other cause is the JAR file specified in MIDlet-Jar-URL line is not available for download.

For BlackBerry handsets sometimes we get this error: "907 invalid COD. Unable to import ZIP file". This is because someone was trying to download the COD files that were still zipped up. The JAR file should be unzipped and then the app should be downloaded.

908: Incompatible configuration or profile: Sometimes the error string will be "Application has malformed attribute". There may be a few possible causes like, the required JAD attributes in JAD file may be wrong. There may be a typo in the JAD file.

909: Application authentication failure: This is mostly caused because the application was signed incorrectly, or the signature did not match the certificate on handset.

910: Application authorization failure: This error happens when we are trying to replace a signed app on the handset with an unsigned one. It is especially noticable for preload handset which have apps baked in with the firmware. Its a common developer mistake to try to download a unsigned app on the preload handset, which has a signed version on it already.

911: Push registration failure: Dont know much about this error, but I think it is not supported anymore.

912: Deletion Notification: This error is displayed when we try to delete the app on the handset and the delete attempt fails. It usually happens for apps that were preloaded and thus cant be deleted. However, if we have a later version of the same app with correct signing then the older version can be overwritten.

Android Error: Conversion to Dalvik format failed with error 1

I wanted to document something that I learnt about Android development, particularly, this error:

Conversion to Dalvik format failed with error 1

It can happen typically when someone starts their Android project in Eclipse. The project can be as simple as a "Hello World" project, this error may still come up. Apparently Eclipse does some internal caching for ADT (version doesn't seem to matter, I have seen it from 0.8.0 version to 0.9.6 version) and when we update Android Tools in Eclipse, it gets corrupted. Here is what I tried and worked for me. My installation is Eclipse 3.5.1 and ADT 0.9.6.

Option One:

In Eclipse, delete your project (back up the source before, of course). Restart Eclipse, recreate the project and the error will go away. This worked for some, but it didn't for me.

Option Two:
Next is to delete your project, start Eclipse on command line with clean option:

eclipse -clean

This cleans up any cached data that Eclipse stores. Now recreate your project and hopefully the error is gone. Well, sometimes it doesn't work. On to the next option.

Option Three:

Delete your project. Uninstall all Android plug-ins in Eclipse and restart it with clean option. Now install your required Android plug-ins.

Create your project, hopefully the error is gone, if not continue reading. In your project home, eclipse will create a .classpath file. Open it in an editor, and ensure that each of the following lines exist in it.

<?xml version="1.0" encoding="UTF-8"?>
<classpath>
    <classpathentry kind="src" path="src"/>
    <classpathentry kind="con" path="com.android.ide.eclipse.adt.ANDROID_FRAMEWORK"/>
    <classpathentry kind="src" path="gen"/>
    <classpathentry kind="output" path="bin"/>
</classpath>

Note that there will be other lines relevant to your project like

<classpathentry combineaccessrules="false" kind="src" path="/project-name"/>

that is okay, the key thing is this line is:

<classpathentry kind="con" path="com.android.ide.eclipse.adt.ANDROID_FRAMEWORK"/>

It needs to be in there. Make this change and save the file, and restart Eclipse. In main menu, do Project -> Clean. Now in project explorer, on the new project you just created, Right Click -> Properties -> Android -> Check Android 1.5 -> Click Apply Button.

Next project, Right Click -> Properties. In the menu on left, click on Java Build Path, on right panel, click on Libraries tab. Ensure that you see a folder called "Android 1.5". Now hit OK and your project should compile.

In the Library tab, if you see Android.jar then it must be deleted manually (highlight and click "Remove" button) and then this error will be gone.

Apart from this I have heard one other option to try to remove this error. I didn't try it but it is supposed to work. First delete Android.jar from right click on Project -> Properties ->Java Build Path -> Libraries tab. Then Right Click the Project -> Android Tools -> Fix Project Properties. This will add Android 1.5 folder and this error will be gone.

I haven't tested this with other versions of Andriod SDKs but I don't see why they wont work.

Best Practices for developing Portable Mobile Applications

Introduction
I will begin with an introduction about myself. I have been working in mobile software development field for a number of years now, developing applications for pretty much most of the mobile platforms (J2ME, BREW and Android). I have wanted to start blogging for a while now, but didn't find time to get started. My topics for blogging will be anything that interests me, but for a number of years now I have been interested in mobile software development, so at least initially I will be blogging about that.

For my first post I want to talk about best practices for developing mobile applications that are portable across multiple handsets.

The guidelines are written primarily written from J2ME application development perspective, but you will see that they make sense across different platforms; the implementation mechanism for these platforms will of course be different. It also doesn't make any difference if we are developing a mobile game or an application, the guidelines apply to both (there are some significant differences between these two, for example asset management is a lot more sophisticated in games, but for current topic they are irrelevant and thus ignored).

In a typical company, an engineer (or two) will be assigned to create the first mobile application. He either has prior mobile development experience or maybe he doesn't. In any case he will do whatever it takes to get the application ready. Now hopefully everyone in the company likes it and want it ported to 10+ different handsets ASAP so that they can do a commercial launch. There may be other scenarios about how this application came exist, they don’t matter much here. What matters is that a mobile application that was developed for one mobile handset is now being asked to be made portable to support 10+ (and eventually a lot more) handsets.

Herein lies the problem, that the application wasn't designed to be portable.

Whatever the reason, we now need to make the application portable. These guidelines I am going to talk about, if followed during the application development process, will facilitate porting. They can be also followed after initial development to make an application portable.

Keep the application design modular
This will come as a no brainer (with almost all software best practices recommending modular design) but the application logic, custom UI layer and platform specific API calls should be modular. This way, it would be very easy to create new applications just by changing the application logic. The custom UI layer will have custom UI controls and custom fonts, but some low-end handsets (the so called feature phones) don’t have enough handset memory to support them, so we can swap them out with native UI (Native J2ME controls etc) easily if the application has modular design. Now this doesn't mean that both native UI and custom UI layer need to be created during initial development, just that the design should be modular enough to support these kind of changes.

Most established mobile companies have reusable frameworks that they use to rapidly create mobile applications and games. Most of the time, these frameworks have evolved from these UI and platform specific modules. Thus this initial effort to keep the design modular pays off many times.

Classify handsets into categories
Most mobile development companies have a classification system for handsets, usually called something like high-end, medium-end and low-end categories. They use various parameters to decide which category a handset belongs. Most of the time, the handset screen size is an important parameter, primarily because it is harder to package image assets of multiple screen sizes into one build.

So, it is important to classify handsets into a number of categories and create builds for each category. Having said that, this exercise can be very time intensive (most companies have a set of applications that take are run on a given handset that help determine which category they belong to). However, it is very easy to create classes of handsets based on screen sizes and start development; they can be modified later without a lot of effort. So at the beginning, we can create say, large, medium and small screen size categories and create builds for just these three. This will allow us to design the application so that it facilitates this kind of classification and thus allow multiple handset support.

For example for J2ME applications, we can create builds for say, the following screen sizes.

A. 176x220

B. 240x320

C. 240x400

D. 320x480

E. 360x640

F. 480x800

Notice that we don’t have to stick to only 3 categories. Now having created builds for these screen sizes, it will be trivial to port to handset having these screen sizes (most of the time there are no code changes required).

Sometimes the annunciator bar at the top takes say 20 pixels (and cannot be removed by api calls like setfullscreenmode) and 240x320 screen size now becomes 240x300. To support this new screen size we should design the mobile application so that the 240x320 screen size takes a jad parameter that will allow us to offset height by say 20 pixels and draw the display. It has to be designed to support this as it will curtail proliferation of builds based on minor differences in screen sizes.

Note that with proliferation of touch screen handsets it is a good idea to create a touch screen build for each screen size greater that 240x320.

Design mobile applications with a housekeeping tasks module
It is a good idea to design mobile applications to perform certain housekeeping tasks when the application is first launched after download. It should be designed to allow this, with a module just for initial housekeeping tasks. These tasks can be varied and change for different types of mobile applications, but some of the more common ones are detailed below.

Nowadays most applications require server connectivity, so on first launch after download it should be able to establish server connection and identify itself. This identification step is very important, and, depending on requirements can be as simple as just creating and sending a handset ID (based on user agent string, or a prepackaged handset ID in app) or a user ID (based on phone number, or some other unique string creation mechanism) or a sophisticated authentication mechanism.

Note that it is quite feasible and in some cases recommended that the user ID is generated on server and sent to handset when the application is downloaded and first launched. It then saves this user ID on handset (in RMS or config file in file system or some other similar mechanism). This user ID is used on all subsequent network calls to identify user to server, thus greatly enhancing reporting, usage tracking and even in customer support issues.

Also, it may be helpful if we have the mobile application make a server call in background each time it is launched. This functionality can be very useful. It can allow us do any initial housekeeping tasks or put the application in debug mode based on a debug flag sent from server to troubleshoot customer support issues. It can also allow on the fly customization of menus and UI. In well designed applications it is even possible to change the look and feel of the app, with all UI assets being downloaded when the application is launched (a detailed look at server interaction for applications is worth an entire blog post in itself, which will be coming in future). In any case a housekeeping module will be greatly helpful for tasks like these.

Landscape mode support
Except for in handsets with small screen sizes like say 176x220, the mobile application should have landscape mode support built in. It should support portrait and landscape mode, with all assets required for that packaged in into the build (API calls like sizeChanged(int w, int h) in J2ME let the application know about portrait to landscape mode changes). Obviously this should also be done for touch screen handset builds.

There is an important aspect to this. Suppose a handset of screen size 240x320 requires landscape mode support. Now since we have designed our application to support landscape mode, it should work without any code changes. Now, there is another handset of screen size 320x240 that needs to be supported (requiring landscape support too). The 240x320 build should work on this handset too, the application internally figuring out what assets to use in portrait mode. To do that, we need to provide a mechanism in the application, to specify the screen size of portrait mode. In J2ME, this can be done as a jad parameter. The application can then read this value, and use it as default portrait mode (one way is to have all assets in width x height directories in jar, and the code accesses the assets as $jad-parameter/imagename.png). Using this mechanism we can use 240x320 build to work on a handset with screen size 320x240, without worrying about portrait/landscape mode confusion.

Another way to address this is to make a utility method that internally determines what screen size assets to use. This method is called to get the assets directory name in code to display the image.

We should also provide a mechanism to disable landscape mode with, a configuration parameter like a jad parameter, to be used in handsets that don’t need landscape mode support. It can be argued that this is not required, that the application should be able to handle this internally, but I prefer this as it provides me a mechanism in code, to disable landscape mode support explicitly.

QWERTY keypad, virtual keypad and 4x3 keypad support
If any mobile application requires text entry, then it should support text entry from a QWERTY keypad, virtual keypad (on touch screen handsets) and 4x3 keypad. If any special characters need to be supported, that application design should incorporate that. One another thing to keep in mind is to ensure that this design should be flexible to allow addition to this list of characters; they should not be hardcoded (details about this design will be in a future post).

Applications should support both native and custom UI controls
All mobile platforms provide native UI controls like text box, menu etc to use in mobile applications. And a lot of them are inadequate. For example the native UI controls in BREW and J2ME are notoriously hard to customize. For this reason it is a good idea to create a library of these controls, with the flexibility we need, to use in our apps.

The same rule applies to fonts, it is a good idea, even advisable to create and use our own fonts. Now most font files are in PNG format, make sure that you create them in indexed PNG format as they reduce the file size significantly.

Customer Service web page
Most applications save usage and reporting data on server and so, to help debug customer service issues, a web interface for this data should be created that will allow us to view and modify it. This is a relatively easy thing to do and this web page need not be on internet, but it pays of many times over.

Design the application to facilitate localization, even if we know that it will never support another language.
All user visible strings and image names should be in one central place, ideally loaded as a resource or a language file (like say US_en_copy-strings.txt for english language). If that is not possible (J2ME doesn't support resource bundles) then in a text file, or a java class. Thus it will facilitate support for localization as then we would just need to swap out these files. Also if we need to do copy changes or change an image name we don’t need to search for it code and do so.

Logging module
All mobile applications should have basic logging built in, that is disabled by default, but can be enabled from server to display debug messages or enabled in a debug build.

Nightly builds and Release Management System
All applications being actively developed should have a nightly build, which compiles all the configurations. The application framework, if it exists should also have its own nightly build. This pays off exponentially, most people dont see the value of this system until it exists and then they dont know how they did without it.

There is one more benefit of this kind of continuous integration system, it can be easily modified to create a Release Management System where all release builds are created from the release server, not from developer machines. This removes any inconsistencies that may be caused by developer environments and standardizes the release process.

These guidelines are by no means complete, being primarily based on what I learnt over the years, but are good enough for a start. Any feedback on this topic is very much appreciated, the goal of this post just to start this discussion.