Tuesday, August 20, 2013

Reading tweets and store them into MongoDB (Java)

Few days back, I was looking for searching tweets example in java using API v1.1 and found none but Stackoverflow threads and API helps me to come up with small example to cover both Twitter and  MongoDB to store these tweets and I decided to share with my circle and readers of this blog.

In Twitter API v1.1 developers are come up with the big change which is "every request to the API to be authenticated". It simply means using Twitter API developer must use OAuth to obtain access token behalf on user account. If you’re new to Twitter Developer API you can follow this URL to create Twitter App in 8 Easy Steps.

I am going to use Twitter4j API. It’s an unofficial java library for the Twitter API. I found it straight, easy to integrate your application with Twitter Services. Please use same link to download and read more about this library. By using this API I do search to get tweets in a form of JSON and save it to database. For database I used MongoDB as I found fastest growing with NoSQL database. Its JSON query style, easy installation make it more preferable. This post cover other advantages about using MongoDB along with code so this make more sense to you. lets dive

Pre-installation Tasks

Create your app in Twitter
Please follow this Twitter App in 8 Easy Steps to create.

Installation for MongoDB
if you are using windows OS I recommend to flow MKYONG Tutorial How To Install MongoDB On Windows and for other platform use Installation Guides on mongoDB official website.

At this point I assume you have already setup MongoDB and downloaded the library from twitter4j official website.

I am using a Netbeans IDE for write this task you can use any IDE you like.
Create Java Project and named it "TwitterMongoDBApp" follow the picture.

No add the libraries into the project Right-Click on the project this open the window where you'll libraries tab at left side. Click on Add Jar/Folder button to choose mentioned libraries into the project.

Before write code make sure you have done your OAuth Setting in twitter as we gonna use four values from this settings.

1) Consumer Key
2) Consumer Secret
3) Access Token
4) Access Token Secret

Please follow the image for this reference.

Twitter Configuration & MongoDB Connection

After all verified with all the prerequisite I created a class TwitterMongoDBApp I used the static block for configuration twitter account with ConfigurationBuilder class in twitter4j.which accept OAuth parameters we discussed earlier.

In constructor calling initMongoDB() method to initialized MongoDB to connect with running server and initialize the class level variable db with using name of "tweetDB". this is same like in SQL "CREATE SCHEMA IF NOT EXISTS tweetDB"

 import com.mongodb.BasicDBObject;  
 import com.mongodb.DB;  
 import com.mongodb.DBCollection;  
 import com.mongodb.DBCursor;  
 import com.mongodb.Mongo;  
 import com.mongodb.MongoException;  
 import java.net.UnknownHostException;  
 import java.util.List;  
 import java.util.Scanner;  
 import twitter4j.Query;  
 import twitter4j.QueryResult;  
 import twitter4j.Status;  
 import twitter4j.Twitter;  
 import twitter4j.TwitterException;  
 import twitter4j.TwitterFactory;  
 import twitter4j.UserMentionEntity;  
 import twitter4j.conf.ConfigurationBuilder;  
  * @author Muhammad.Saifuddin  
 public class TwitterMongoDBApp{  
   private static ConfigurationBuilder cb;  
   private DB db;  
   private DBCollection items;  
   public TwitterMongoDBApp() {  
     try {  
       // on constructor load initialize MongoDB and load collection  
       items = db.getCollection("tweetColl");  
     } catch (MongoException ex) {  
       System.out.println("MongoException :" + ex.getMessage());  
    * static block used to construct a connection with tweeter   
    * with twitter4j configuration with provided settings.   
    * This configuration builder will be used for next search   
    * action to fetch the tweets from twitter.com.  
   static {  
     cb = new ConfigurationBuilder();  
   public static void main(String[] args) {  
     TwitterMongoDBApp taskObj = new TwitterMongoDBApp();  
    * initMongoDB been called in constructor so every object creation this  
    * initialize MongoDB.  
   public void initMongoDB() throws MongoException {  
     try {  
       System.out.println("Connecting to Mongo DB..");  
       Mongo mongo;  
       mongo = new Mongo("");  
       db = mongo.getDB("tweetDB");  
     } catch (UnknownHostException ex) {  
       System.out.println("MongoDB Connection Errro :" + ex.getMessage());  

After done initialization for both Twitter and MongoDB. This program display menu on console and ask user to input their selection that what loadMenu() method do.

 public void loadMenu() {  
     System.out.println("================\n\tTwitter Task\n===============");  
     System.out.println("1 - Load 100 Tweets & save into Mongo DB");  
     System.out.println("2 - Load Top 5 Retweet");  
     System.out.println("3 - Load Top 5 mentioned");  
     System.out.println("4 - Load Top 5 followed");  
     System.out.println("5 - Exit");  
     System.out.print("Please enter your selection:\t");  
     Scanner scanner = new Scanner(System.in);  
     int selection = scanner.nextInt();  
     if (selection == 1) {  
     } else if (selection == 2) {  
     } else if (selection == 3) {  
     } else if (selection == 4) {  
     } else if (selection == 5) {  
     } else {  
       System.out.println("Wrong Selection Found..\n\n");  

Searching Tweets and Store in MongoDB

Behind the first selection getTweetByQuery(true) get called which use to creates a TwitterFactory with the given configuration and after create Query object to define the search criteria for tweets and passed into Twitter.search method initialized from TwitterFactory using getInstance method. I set “Java” keyword for query and set number of result to retrieve. Then queryResult Class from twitter4j library holds the result after executed search.

Now you done with the getting tweets from Twitter. Moving on two next step to store these records into MongoDB. Before we look into the code a common terminologies used in MongoDB are corresponding to SQL concept and terminology:

• Database: Database
• Table: Collection
• Row: Document

To read more Terminology and Concepts in MongoDB use this URL.

MongoDB is a document-oriented database where each document in a same collection may have a totally different structure and query the database has become easier and efficient. Can go with without Join

    * void getTweetByQuery method used to fetch records from twitter.com using  
    * Query class to define query for search param with record count.  
    * QueryResult persist result from twitter and provide into the list to  
    * iterate records 1 by one and later on item.insert is call to store this  
    * BasicDBObject into MongoDB items Collection.  
    * @param url an absolute URL giving the base location of the image  
    * @see BasicDBObject, DBCursor, TwitterFactory, Twitter  
   public void getTweetByQuery(boolean loadRecords) {  
     if (cb != null) {  
       TwitterFactory tf = new TwitterFactory(cb.build());  
       Twitter twitter = tf.getInstance();  
       try {  
         Query query = new Query("java");  
         QueryResult result;  
         result = twitter.search(query);  
         System.out.println("Getting Tweets...");  
         List<Status> tweets = result.getTweets();  
         for (Status tweet : tweets) {  
           BasicDBObject basicObj = new BasicDBObject();  
           basicObj.put("user_name", tweet.getUser().getScreenName());  
           basicObj.put("retweet_count", tweet.getRetweetCount());  
           UserMentionEntity[] mentioned = tweet.getUserMentionEntities();  
           basicObj.put("tweet_mentioned_count", mentioned.length);  
           basicObj.put("tweet_ID", tweet.getId());  
           basicObj.put("tweet_text", tweet.getText());  
           try {  
           } catch (Exception e) {  
             System.out.println("MongoDB Connection Error : "  
                 + e.getMessage());  
         // Printing fetched records from DB.  
         if (loadRecords) {  
       } catch (TwitterException te) {  
         System.out.println("te.getErrorCode() " + te.getErrorCode());  
         System.out.println("te.getExceptionCode() " + te.getExceptionCode());  
         System.out.println("te.getStatusCode() " + te.getStatusCode());  
         if (te.getStatusCode() == 401) {  
           System.out.println("Twitter Error : \nAuthentication "  
               + "credentials (https://dev.twitter.com/pages/auth) "  
               + "were missing or incorrect.\nEnsure that you have "  
               + "set valid consumer key/secret, access "  
               + "token/secret, and the system clock is in sync.");  
         } else {  
           System.out.println("Twitter Error : " + te.getMessage());  
     } else {  
       System.out.println("MongoDB is not Connected!"  
           + " Please check mongoDB intance running..");  

after set all the related field in BasicDBObject object finally store this prepared document into MongoDB by calling this line of code items.insert(basicObj); isn't it simple :).

Fetching Records from MongoDB

As you have seen previously in code that we are using getTweetsRecords method here is the implementation, which use the items Collections to find documents with BasicDBObject with the defined field to fetch similar like Select query statement and iterate over the results by DBCursor.

 * void method print fetched records from mongodb This method use the  
 * preloaded items (Collection) for fetching records and print them on  
 * console.  
   public void getTweetsRecords() {  
     BasicDBObject fields = new BasicDBObject("_id", true).append("user_name", true).append("tweet_text", true);  
     DBCursor cursor = items.find(new BasicDBObject(), fields);  
     while (cursor.hasNext()) {  

To get the top 10 Retweets from collection applied desc sort in MongoDB query to fetch them.

   * void method print fetched top retweet records from   
   * preloaded items collection with the help of BasicDBObject   
   * class defined sort with desc with fixed limit 10.  
   * @see BasicDBObject, DBCursor  
   public void getTopRetweet() {  
     if (items.count() <= 0) {  
     BasicDBObject query = new BasicDBObject();  
     query.put("retweet_count", -1);  
     DBCursor cursor = items.find().sort(query).limit(10);  
     System.out.println("items length " + items.count());  
     while (cursor.hasNext()) {  

Source Code:

Please use this url ( https://github.com/m-saifuddin/TwitterExample) to download the discussed example.

Tuesday, July 09, 2013

Calling Activity Method From JavaScript in Webview Android (vise-versa)

Hi everyone, Few days back I am looking for a solution to get access the html form submit data into native Android. Suppose when you have criteria to load the HTML form server or local into Webview and you want to invoke some REST request to submit this form data with additional info or to notify native regarding this form submission.

 After some Googling I found this addJavaScriptInterface method in Android. Facilitate developer to  call Activity method inside from javascript function and vise-versa.

Click here to read get the background and detail of this addJavaScriptInterface method of Webview.

Hands on Code:
I am going to use an ItelliJ IDEA IDE and I suppose that you have some experience in Android development using the ItelliJ IDE. if you it simple just download Community Edition FREE from here and move a head.

Create Simple Project.

Define Activity Name and project package.

Create the html file in assets folder name it sample.html. Copy and paste the below code into this file.

   <link rel="stylesheet" type="text/css" href="style.css"/>  
   <script type="text/javascript">  
     function bar(data){  
     var obj = eval ("(" + data + ")");  
     var html = "<p><b>Data received</b><br>"+  
       "<br>Name: <span>"+obj.cs_name+"</span><br>"+  
       "City: <span >"+obj.cs_city+"</span><br>"+  
       "Phone: <span>"+obj.cs_phone+"</span><br>"+  
       "Address: <span >"+obj.cs_address+"</span></p>";  
     var x=document.getElementById("te");  
       <!--x.innerHTML = "<p>"+ message+"</p>";-->  
     x.innerHTML = html;  
     function foo(){  
     var txt_cs_name = document.getElementById("txt_cs_name").value;  
     var txt_cs_phone = document.getElementById("txt_cs_phone").value;  
     var txt_cs_city = document.getElementById("txt_cs_city").value;  
     var txt_cs_address = document.getElementById("txt_cs_address").value;  
     var dataObj = new Object();  
     dataObj.cs_name = txt_cs_name;  
     dataObj.cs_phone = txt_cs_phone;  
     dataObj.cs_city = txt_cs_city;  
     dataObj.cs_address = txt_cs_address;  
     var jsonObj = JSON.stringify(dataObj);  
     /* NOTE: Calling Activity Method using the Activity defined 
        instance "android" in JavaScript */
 <div id="te" ></div>  
   <form id="form1">  
     <table width="100%" class="mytable">  
         <td><input type="text" id="txt_cs_name" ></td>  
         <td><input type="text" id="txt_cs_phone" ></td>  
         <td><input type="text" id="txt_cs_city" ></td>  
         <td><textarea id="txt_cs_address"> </textarea></td>  
 <button onClick="foo()"> Submit</button>  

This html file has a two javascript function
  1. foo() This function calls the activity method.
  2. bar(data) This function calls from the activity method with parameters.
Below is the foo() function line invokes the activity method.

Now create the Activity part as we already create through wizard, it time to setup TestMethod in activity for javascript invoking.

 package com.example.WebviewExample;  
 import android.app.Activity;  
 import android.os.Bundle;  
 import android.webkit.JavascriptInterface;  
 import android.webkit.WebView;  
 import android.widget.Toast;  
 import org.json.JSONException;  
 import org.json.JSONObject;  
 public class MyActivity extends Activity {  
   private WebView wv;  
   // the method to call from the html button click  
   public void TestMethod(String formData) {  
     System.out.println("TestMethod FormData " + formData);  
     try {  
       JSONObject jsonData = new JSONObject(formData);  
     } catch (JSONException e) {  
     wv.loadUrl("javascript:bar('" + formData + "');");  
     Toast.makeText(getApplicationContext(), formData, 5000).show();  
   public void onCreate(Bundle savedInstanceState) {  
     wv = (WebView) this.findViewById(R.id.web_view1);  
     // requires javascript  
     // make this activity accessible to javascript  
     wv.addJavascriptInterface(this, "android");
     // set the html view to load sample.html

The onCreate method simply initialize Webview with layout which has only Webview declared with id web_view1 inside After initialization we call Webview settings to enable the JavaScript.
and in the next line calling another addJavascriptInterface Webview method.
wv.addJavascriptInterface(this, "android"); 
it accept two parameters one is class instance to bind to JavaScript and the second one is used to define instance name to expose and use in JavaScript and finally we call/load our sample.html file in Webview.

The TestMethod method annotated with JavascriptInterface this means that this method can be accessed from JavaScript this method simply calling javascript function "bar" using Webview loadUrl method
 wv.loadUrl("javascript:bar('" + formData + "');");   
and also show the result in Toast as well.

Testing Time:

Friday, May 31, 2013

Use SQL Adapter in IBM Worklight 5.0.x

I prepare a simple demo app for team to knowledge sharing to achieve a related goal into worklight.
then I think about why not share on blog by following a great dialog stated "Human knowledge belongs to the world" in movie (I don't remember which one :).

In this blog entry I am not going in much about introduction of Worklight and some of its component but expecting a reader to do few reading using a Google or IBM info center. So

IBM Worklight provide multiple adapters to interact with different media and work as mediator between mobile application and enterprise system. please follow the link to read Overview of IBM Worklight  Adapters. This entry is cover the SQL Adapter.

What is SQL Adapter?

The name itself is expressing, it provides access to enterprise databases to execute the parametrized SQL queries and stored procedures that retrieve or update data in the database.
Below image is illustrate high-level process involve.

Its Time for Action

Before start assuming you have already setup Worklight environment and now follow the steps is to create a SQL Adapter in Worklight.I have running my MySQL database on my local system. Please follow the step 1 to use this script add table into your schema.
Step 1:
CREATE TABLE `product` (
  `productID` int(11) NOT NULL AUTO_INCREMENT,
  `productName` varchar(255) DEFAULT NULL,
  `Qty` int(10) DEFAULT '0',
  `Model` varchar(255) DEFAULT NULL,
  `price` int(10) DEFAULT '0',
  PRIMARY KEY (`productID`)

now next step is to create Worklight Adapter.

Step 2:

after hit the finish button this will create the adapter with name Product under adapters folder include 2 files Product.xml and Product-impl.js.

  • Product.xml (to define Data Source for DB and declare procedures you target to invoke)
  • Product-impl.js ( to defined queries to invoke against every procedure you defined in Product.xml)

Now open the Product.xml file and do define the Datasource like this.
<connectionpolicy xsi:type="sql:SQLConnectionPolicy">
  <loadconstraints maxconcurrentconnectionspernode="5">
Note: You have to include “mysql-connector-java-5.1.24-bin.jar” library. If you don’t have one, then download here Download.

if you see your Product-impl.js file you find all you procedures in Step 2 image are automatically generate structure for developer to just change the column name and redefined the query as per need like below. for short explanation I am include two procedure code snippet here one is getProducts fetch all the project and another one is to insert.

/* Select Statement */
var selectProductStatement = WL.Server.createSQLStatement("select "+
"productID, productName, Qty, Model, price from product "+
"ORDER BY productName");

function getProducts() {
 return WL.Server.invokeSQLStatement({
  preparedStatement : selectProductStatement,
  parameters : []
/* Insert Statement */
var insertProductStatment = WL.Server
  .createSQLStatement("insert into product "+
" (productName, Qty, Model, price) values (?, ?, ?, ?)");

function addProduct(productName, qty, model, price) {
 return WL.Server.invokeSQLStatement({
  preparedStatement : insertProductStatment,
  parameters : [ productName, qty, model, price ]
for testing adapter you can right click on adapter folder -> run as -> Deploy as Worklight adapter. once its deployed successfully you .
now to call this procedure in client side adding a example of Add product.

function addProductRec() {
   var validate = true;
   var query = {};
   if ($("#prod_name").val() != "") {
        query.productName = $("#prod_name").val();
   } else {
        validate = false; 

   if ($("#prod_model").val() != "") {
        query.Model = $("#prod_model").val();
   } else {
        validate = false; 
   if ($("#prod_qty").val() != "") {
        query.Qty = $("#prod_qty").val();
   } else {
        validate = false;

   if ($("#prod_price").val() != "") {
        query.price = $("#prod_price").val();
   } else {
        validate = false;

   if (validate) {
       var queryData = JSON.parse(JSON.stringify(query));

                adapter : "Product",
  procedure : "addProduct",
  parameters : [ queryData.productName, queryData.Qty,
                      queryData.Model, queryData.price ]
  }, {
                  onSuccess : function(result) {
                  $.mobile.changePage("#list", {
          transition : "slide",
   reverse : false,
   changeHash : false
    onFailure : failureCallback
 } else {
            alert("All fields required");
now we are done and its time for testing.
Worklight Console : Screen 1
Worklight Console : Application Screen

Please click here to download the source code.

Saturday, April 27, 2013

Integrate ChildBrowser into Worklight App (IOS Version)

Our team recently trying to get Worklight to open an external URL within a hybrid mobile app. Worklight has an OpenURL function that we first implemented, but the problem with this opens an external URL in the same window as the app,and this isn’t what we are looking for because when URL opens in the same window as the app, it’s difficult to exit the external website, and return to the app flow. So we went with the ChildBrowser plugin to implement child browser functionality within our current running Worklight app. 

During googling I found many people struggling with PhoneGap/Cordova to implement ChildBrowser up.So I come with the solution with easy step and I decided to write a post that hopefully shows up in google for people running and looking for solution.

The ChildBrowser plugin I’m using can downloaded from here.

Create a simple Worklight App name it ChildBrowserDemo. please follow the Figure 1.1

Figure 1.1
Define you app name in second screen or same ChildBrowserDemoApp see Figure 1.2

Figure 1.2

Copy the ChildBrowser.js from ChilBrowserPlugin folder and paste it under common/js folder in your project.
now add this code snipped under the project main html file.
a JavaScript code snippet I used to get it to work. I added a listener on button and call to the install method of the ChildBrowser object. Finally I calling a showWebPage function of childbrowser object that handles the URL opening.

 <h1> Child Browser Demo </h1>        
 <button id="open-child-browser">Open google.com</button>  
 .addEventListener("click", function() {   
  alert ("CB Error :"+e);  
 <!-- application UI goes here -->  
 <script src="js/initOptions.js"></script>  
 <script src="js/ChildBrowser.js"></script>  
 <script src="js/ChildBrowserDemoApp.js"></script>  
 <script src="js/messages.js"></script>  

Figure 1.3
Now add iphone Worklight environment now build & deploy application this is done for studio side.
After Build & Deploy project from studio right-click on iphone icon and run as xcode project. please follow the below Figure 1.4 for the reference.

Figure 1.4
after opening the project in Xcode drag all of the .m, .h, and .bundle file(s) from the ChilBrowserPlugin folder to the Plugins folder of your project. Make sure the copy option is selected. check Figure. 1.5 below.
Figure 1.5

and then drag the following files into Resources folder into your project.
  1. ChildBrowser.bundle
  2. ChildBrowserViewController.xib
In your project, under the Supporting Files folder, open the Cordova.plist file. You’ll see a bunch of key/value pairs and the one you want is under Plugins. You’ll need to add 2 by clicking the add button that shows when you highlight the Plugins section.
Key: ChildBrowser,        Type:String, Value: ChildBrowser.js
Key: ChildBrowserCommand, Type:String, Value:ChildBrowserCommand

Figure 1.6
Now final step to add these .h and .m files again into Compile Sources to make this plugin available for app in xcode. (otherwise it will prompt ChildBrowserCommand plugin not found error. ) Please follow Figure 1.7.

Figure 1.7
Its time for Testing on Emulator.