Saturday, October 10, 2015

An app to make democracy better

I have an app idea to improve our rusty Indian democracy using crowd-sourcing, but I don't have enough time to prioritize it and actually build it. So I am posting this in public domain for someone to take it up and build it. Don't worry, I won't claim any credit.

The problem
We cast our vote in the elections assuming our elected representatives will work for us and make our daily lives better. For the taxes we pay, we expect better roads, good public transport, easily available drinking water, good quality govt. schools and hospitals, reasonably priced ration, helpful police force and all the other required social infrastructure that is government's duty to provide in lieu of all our taxes. This is an ideal democracy. We are still very far from it.

What can an app do to fix this?
An app can do wonders. Consider the following use cases:
1. The road in your colony is broken badly. You take your phone, launch an app, click the big red 'Report an issue' button with an exclamation mark. App starts the camera, you point the phone cam to the road and click the pic. App fetches your GPS co-ordinates (this is a key point). You optionally choose a category (e.g. Transportation) and add some description and submit it.
2. There is a walled area labelled park in your colony. But inside there is a thick growth of shrubs, and no jogging track or children play amenities. What kind of park is that? You use the app to report the issue with a pic and GPS co-ordinates.
3. You go to apply for a driving license. Lunch is from 1 to 1:30 PM but the guy at the counter left at 12:30 for lunch and you need to wait in the queue till 2:30PM or whenever he comes from his home. He is an arrogant a-hole and thinks that this job his his father's jageer. He needs to be taught a lesson. Pull the app, put a picture with missing guy's blank chair and wall clock with time in the background, report the issue.
4. You are in a market and see that the local police is extracting hafta from the shop-cart owners. When the policemen go away (yeah, stay safe), pull up your phone report the issue. Then tell the shop owners too to do the same with their phones.
5. You went to some govt. office and you anticipate you will be asked for bribe. Launch the app and it will record the audio, attach GPS co-ordinates and put it live for the world to see.

So what after I report an issue?
Imagine millions of Indians reporting their daily issues. We will have a huge database of issues now with proofs, all of which can be neatly seen on a map, panned and zoomed; seen with heatmaps, seen categorized by constituency and seen categorized by departments. Most of these might contain proofs in terms of pictures. We can browse and visualize this data in multiple innovative ways. We can compare MLAs across constituencies, we can compare states. We can figure out whether our MLA or MP is working or not and whether he/she should be given a second term. Vigilance department can act swiftly

What about the addressed issues?
Once a govt. servant marks an issue as addressed and puts a justification, it needs to be reviewed and approved by the complainer or by a fixed number of people in the vicinity of that area.

I have scratched the surface, now if you are really serious about it, spend sometime thinking about it and implementing it in the right way.

Tuesday, August 26, 2014

My problems with Facebook

1. On Facebook, I have 400+ friends. Out of which, I consider around 10 as real friends. Around 20 of them aren't friend-close to me but I had interactions of intellectual nature  with them for extended periods which makes them important enough for me to value their opinions. The rest? They are not my friends. They are just in my friends list. A few of them like each of my posts and post a comment with just the word 'superlike' or 'wow this is great' or something of that sort. Although they do this in good faith, but this annoys me. Also I don't think they are competent enough to post a random comment on a topic I started on my wall. Looking from a different viewpoint, I don't want to say this in their face and un-friend them right away. Because I may need their help some day when, for some reason space time continuum deterministically puts me in their city and I am on harm's way.
The bottomline? Facebook is not the app for me. I know you will say how can I create lists and do something with them like not showing my posts to specific lists. But so many lists with overlapping members aren't practical and worth my time to create and update regularly. Facebook's concept of friendship is fundamentally broken for me. It might not be that way for people who talk to each of their Facebook friends on a regular basis. Twitter's concept of followers suits me better. Facebook has that, but not the way it should be.
2. News feed's algorithm is out of my understanding. It hides many posts from me which I would like to see. If facebook thinks that a probabilistic set of algorithms is smart enough to determine what I would like to see on my newsfeed, they are wrong. Machine learning techniques are in fashion but the output is useless when it is used to forcefully solve a problem which isn't meant for it. I am sorry Facebook, but the hammer you are using for this nail is of wrong size.
3. I can not see all my wall posts. Many events and posts are missing from my timeline. I want my timeline to be a journal of everything I do on Facebook. Timeline literally isn't doing what it stands for in plain English.
4. I don't want to connect every other website with Facebook. I don't want Facebook to know what I do on that website. And I don't want that website to know what I do on Facebook.

Friday, July 18, 2014

I don't like Google Maps

The quality of Google Maps is deteriorating day by day. With their update on Android that killed the handy scale, to out of place venues, to UI irritations. Its all piling up.

Lets see the search quality. This screenshot will tell you everything. I searched for "KFC near me". It recommended me the one which is a 2 hour 25 minutes walk from my current location, despite showing up so many of them which are far closer.
 Did I search for "KFC near me" or "KFC far from me"?

Lets see some example screenshots of POIs which exist on Google Maps within 2KM radius of my home but do not exist physically on planet earth. I guess, Google has indexed POIs from one of the parallel universes.

Look at that "Grand Sweets". I followed Maps and reached there to find someone's residence. I had indications early on that Google is lying to me and just wants me to walk so as to burn some calories. Because, why will Grand Sweets open their outlet inside a residential area?
See that Domino's pizza. According to Google, it lies right there on the middle of an expressway!!!! If you can't employ human operators to fix it, then at least get it fixed using the crowd way. If n number of people, report a venue address as false/bogus please put someone to look at it and remove from maps.

One fine sunday, while returning back from a weekend getaway, I suggested my friend who was driving; to take a shortcut through a residential area to save a few kilometers. We used Google Maps to route us. It took us deep inside a colony showing us that we are inching towards our destination closer and closer. Suddenly we saw a wall in front of us. Google was telling us to take the car through that wall. Another time, I was with a friend on bike and needed to get the puncture repaired. I searched for bike repair shop. It showed us one at around 400 meters from our current location. We dragged the bike there only to find a chaiwala stall there. 

There are umpteen numbers of factual inaccuracies on Google Maps. And they are stacking up pretty quickly. Within 2KM radius of me, I showed you two of them. I don't know how many more are there. According to a random estimate, I assume there are 1000+ factual inaccuracies in Google Maps, just in Chennai.

I have decided not to believe Google Maps for address of places. The best way is the old way. Ask the local people. If more than one person who are spaced at least 300 meters apart, tell you an address; then believe it. It must be true.

Tuesday, May 28, 2013

Debug Hadoop source code using an IDE(Intellij Idea)

This is my 100th blog post ;-)

If you are someone who wants to dive into Hadoop source code and get a feel of the implementation details of all the abstracted out nitty-gritties of Hadoop's architectural overview, and want to get your hands dirty by modifying a thing or two; may be because you have just started your masters research on Hadoop or just for the sake of understanding the control flow; this post is for you.

For learning practical applications of Hadoop, I have two recommendations for you. Hadoop - The Definitive Guide and Hadoop in Action are amazing books to start with. I started with them to understand the practical aspects of Hadoop.

I use IntelliJ Idea Community Edition as my IDE(yes, because I don't like Eclipse), but this post should be fairly understandable to Eclipse fans too; although I won't be providing the steps for Eclipse. If you are not proficient with Eclipse then please download IntelliJ Idea from here and use it, instead of posting mundane comments like how to perform step number X in Eclipse(or Netbeans or JCreator or Java IDE #9510). Make sure, you scroll down and choose the Community Edition to download. If you are on *nix, better use your distro's package manager to get it.
Ok, lets start:
Step #1: Download Hadoop
Downloading the latest version of Hadoop along with source code is simple. Just type Download Hadoop in your browser's omni search bar and follow your instinct. For the lazy soles in the kingdom of Dark Room at 3AM, here is the link. There are two tarballs of interest. One is hadoop-<version>.tar.gz which is around 60MB in size and the other is hadoop-<version>-bin.tar.gz which is around 33MB in size. The one with a bin in the name doesn't have the source code, only the binary executable is there. So, obviously download the one without bin in the name.

Step #2: Unpack the tarball and import in IntelliJ Idea
After the download, unpack the tarball. With the following command(if you are on *nix):
tar -xzf hadoop-<version>.tar.gz

Now fire up IntelliJ Idea. If you have just installed it, you will need to accept the License agreement. You will then, get to a screen like this:
Tip #1: Full resolution images
Click on any screenshot thumbnail to view the large image.

Click Import Project and choose the directory named hadoop-<version> eg. hadoop-1.0.4 which got materialized when you unpacked the tarball. An Import Project dialog will open. Then, blindly keep clicking next. During this, Idea will first search for sources, then libraries, then modules and then move to selecting project SDK. I would recommend setting the SDK as Sun Java 6. If you don't have it in your machine and you just have OpenJDK then download it from Oracle's site here. Extract JDK to somewhere, for example /opt and make IntelliJ Idea point there in the Select Project SDK page of Import Project wizard. Afterwards, it will try searching for frameworks used and will find nothing. Here are the screenshots for all these steps, if you get stuck somewhere.

Click finish in the last step and you have successfully imported Hadoop in the IDE. You will then be greeted by a screen like this.

Step #3: Add the build.xml as Ant build file
Right click the file build.xml in the left pane(Project Structure) and click the last option that says Add as Ant build file
To test whether all is well, click the Ant Build button in the extreme right bar to reveal Ant Build dock. Then double click the clean target to execute it. Once it is successfully executed double click the compile target.

If all is well, both clean and compile targets should execute successfully. If the compile target gets stuck at Executing task: get, you probably need a non-proxied internet connection. You can still get it working over proxy, but that is beyond the scope of this post.

Tip #2: Change keymap to Eclipse
But before we get into the source code, I will recommend setting your keymap to Eclipse style. That can be done in File > Settings > Keymap as shown in below screenshot.
We did this because Eclipse is Ubiquitous and most of you are familiar with Eclipse shortcuts.

Step #4: Create a debug configuration
Now we have to setup a Run/Debug configuration. In the Run menu, click Edit Configurations. Click the + sign on the top left and click Application.

Now what to fill in the text fields in this dialog? Let's find out!!!
Open the file hadoop-1.0.4/bin/hadoop in a text editor. Scroll down to the end and modify the two lines with exec with echo; shown in the screenshots below.
 Modify exec to echo.

This will let us see the exact command line for running a MapReduce job. Now open a terminal and navigate to hadoop directory and type this command:
bin/hadoop jar hadoop-examples-1.0.4.jar wordcount conf output
You will get a huge output. The syntax is as follows:
javaExecutablePath VMOptions mainClassFile programArguments
The output on my machine looks like this:
/opt/java/bin/java -Dproc_jar -Xmx1000m -Dhadoop.log.dir=...
...jsp-api-2.1.jar org.apache.hadoop.util.RunJar hadoop-examples-1.0.4.jar WordCount conf output 

  • /opt/java/bin/java is my javaExecutablePath
  • org.apache.hadoop.util.RunJar is mainClassFile that will start hadoop.
  • hadoop-examples-1.0.4.jar WordCount conf output is the programArguments list.
  • the huge thing denoted with dots above is the VM options.
So, fill in the text fields in debug configurations dialog accordingly. In the Before Launch section add the ant targets clean and compile as shown in the screenshot. In the Use Classpath of Module field, select hadoop-1.0.4.  The below screenshot shows my configuration
Click Ok. Now lets test our configuration. Click the Debug Hadoop button from the toolbar as shown in the screenshot.
If all goes well, you will get expected output in Console tab of the bottom dock as shown in screenshot.

Next let us see how to put breakpoints and step through the code.

Step #5: Add breakpoints in source code
Press Ctrl+Shift+R and type RunJar. Select the from dropdown list and press enter. RunJar is the main class in Hadoop-1.0.4. 
The source for will open up. Press Ctrl+O and type main and press enter. You will jump to the main method. At the first line of the code in main method, click in the gutter to add a breakpoint in that line. See screenshot below. Click at the location where a red circle is shown in the screenshot. That's gutter area. For you the red circle will appear after clicking.
Now that you have added a breakpoint, you can click the Debug button in the toolbar and after the clean and compile targets are executed, the program execution will begin and it will stop at the line where you added the breakpoint. From there, you can step into, step over and step out in the code from the run menu or F5, F6 or F7 keys.
Now you are free to modify hadoop code and testing your changes.

Once you are done with this and spend some time on it, you will find out that you aren't able to follow the JobTracker or the TaskTracker's execution. This is because they are separate processes and run in different JVMs. In the next blog post I will cover how to debug JobTracker and TaskTracker.