Comparison of Message broker (rabbitMq, ActiveMq,Kafka, ZeroMq)

Recently i got an opportunity to compare  Message brokers(rabbitMq, ActiveMq,Kafka, ZeroMq) for a specific business need. To do that i have selected few parameters which are  important for choosing message broker for important business functions. Overall i found that rabbitMq has features which are easy to configure, a very good admin console, very good support for development (almost in all languages), basic features of reliability available out of box(acknowledgement, confirmation, durability). We also did benchmarking and found publishing and consumption rate good. Detail comparison is as below:

Parameter	RabbitMq	ActiveMq	Zero Mq	Kafka
1. Clustering/Load Balancing mechanism.	Clustering Available, Queues clustering have to be handled separately.Clustering queue will be only for HA not for load balancing Feature	Available	Can be achieved by wriritng lots of customize code.	Available but producer has to know to which partition it is writing..
2. Replication among different nodes.	Available	Available	Not automatic as there is no broker but can be coded. But lot of customization.	Available
3. Fault tolerance feature. Turned around time in case of failure.	Durable Queue, Durable Message and Clustering support. Another cluster node will take over but in case of queue it is different(connection has to be established with new node again by client.)	Durable Queue, topic and durable consumer supports and availability through clustering is ensured.	Features available but not out of the box.	Zookeeper is required to manage it.
4. Supported libraries for go and other languages like dot net (CRM , ERP and CMS are on window stack).	Available in languages Java, Go, Python and .Net	Go client not available. Rest based http interface is available.	Go support available	Available support for Go.
5. security	Basic Level of Authentication like restricting users for read/write/configure(administration) exist.	Authentication support using different plugin.	One has to built on top of it.	Not available in current version.
6. Interopretability in case Message broker is to be changed. (No binding)	AMQP 0.9 complaint. So changing one AMQP complaint broker with another one should not need a change in client code. Rest based plugin available.	Same as rabbitMq.It is AQMP 1.0 compliant	Specific client has to be written.	Rest interface plugis are available.
7. Performance throughput (read/write).	Moderate as per benchmarking data available. (I read in pivotal blog that it can receive and deliver more than one million messages per second.)	Comparable to RabbitMq.	Very fast	Very fast
8. Administration interface	Available, Http based having basic functionality.	Basic Web console.	Not available has to be built in.	Very basic interface. Third party web console is available.Less features as compared to RabbitMq interface like User Management
9. Open Source	Yes	Yes	Yes	Yes
10. Support for Big Data	Publishing & Consumption rate comparison to kafka is less. So Can be a bottleneck in a situation like click straem where continuous publishing is required without a pause. One apache project "Flume" which can be used to tranfer data to Hadoop.	Same as for rabbit MQ. Flume can also be used in active mq as it works with AMQP.	Good in terms of fast writing and reading	Kafka Hadoop Consumer API
11. Push Notification	Libraries support both push and pull notification.	Libraries support both push and pull notification.	Libraries support both push and pull notification.	Libraries support both push and pull notification.
12. Other				Worker has to manage what it has consumed or not. Broker does not take care of it. Message remain in the storage until a specified time. Worker has to provide partition id broker details.

Security Tips for Struts based web application

Some Tips on Securing web based application (Struts -2, Mysql, Hibernate, JBoss 7.0.2)

1. AUTOComplete OFF

Add autocomplete=’’off” tag in form tag to prevent auto-fill user credentials in the form.

2. Session Time out implementation: Session time

Configure session time in web.xml. It takes time in minutes.

 

3. Brute force attack:

If your application has any login/password screen then there is a possibility for brute force attack if you have not implemented captcha or blocking of user after some unsuccessful login attempts. There could be many solution to solve that issue:

a. Introduced captcha in login/password screen

b. After two-three unsuccessful attempt, ask user to fill up a captcha screen.

c. Block the user after certain unsuccessful attempt and unblock him through administrative user or automatically after certain period of time.

Point to remember before implementing any of the above solution:

Solution (a) will be easy to implement but it will be inconvenient to user if application is regularly used by him/her and application is only for some restricted user set for example for an Organization.

Solution (b) would be a good fit, mainly popular site like gmail, yahoomail etc. use it but then one has to make sure that user should not be allowed to attempt again by changing the session either by opening application in new tab of browser or opening in different browser.

Solution (c) is more apt if application is quite sensitive and we really do not want user to try more then specified number of time. Again solution should take care of new sessions and different browsers.

We have implemented a database based solution where we kept failure information of a user in database and updated it based on successful login or reset it after certain period.

4. Insecure Direct Object References:

The functionalities/pages which need to be accessed through application flow are directly accessible by providing their URI in the address bar i.e. directly going to a particular page through change in its object id from URL. It is observed for most of the functions, the attackers can manipulate these references to access unauthorized data without any access control check or other protection. The Forward & Back button can also be used to access the function directly bypassing the application flow.

Include below code in .jsp page to disable back button.

 

Include this code in .java file to disable back button in default interceptor.

5. Absolute path disclosure/Path traversal is possible:

Application displays the absolute path for all the function/objects which may be misused for getting the access of even those functions which needs to be protected by application flow & access to be allowed only to authenticated & authorized users which are supposed to use these. The attacker can manipulate the parameter to access the pages directly.

There could be two solutions for it:

1. Masked actual URL with some fixed URL (I have seen this in many site but could not implement it.)
2. Implement role based access control on particular page access. In case of struts one can extract the role from session which can be populated at the time of login. In default interceptor check user’s role and corresponding allowed action. If called action is not allowed then throw user on login screen.

 

6. Failure to Restrict URL Access:

No check is implemented in the application to restrict the access of protected links as web application is not checking the URL access rights before rendering protected links and buttons. As observed the lower level user e.g. tester not having the access link to functions related to admin as add, modify or delete the user/role etc. can directly access & use these functionality through supplying the URL address of these pages in the address bar without any authorization check.

This point would be solved if one implements solution 2 of point 5.

7. Application Error message/ Exception handling:

Application is unable to handle the exceptions/ Runtime error as it displays the error message showing the details of platform info & database structure which may be misused by the attackers. Application error messages reveal the server details.

Different webserver handle this differently. In JBOSS we have handled it through web.xml of application. We have put checks for 500,404 and thrown a error page.

8. Cross-Site Request Forgery:

The same request was sent twice in different sessions and the same response was received. This shows that none of the parameters are dynamic (session identifiers are sent only in cookies) and therefore that the application is vulnerable to this issue.

If all points from 1-10 are rectified. This issue will be automatically addressed.

9. Client-Side (JavaScript) Cookie References:

Appscan found a reference to cookies in the JavaScript. The complete logic of cookie generation is available to client which can easily be retrieved & misused. Remove business and security logic from the client side

Please write below code in web.xml.

 

10. Insecure HTTP Methods Enabled:

It is possible to upload, modify or delete web pages, scripts and files on the web server. The Allow header revealed that hazardous HTTP Options are allowed, indicating that WebDAV is enabled on the server which means GET, HEAD, POST, PUT, DELETE, TRACE, OPTIONS are permitted. Disable WebDAV, or disallow unneeded HTTP methods

Make below changes in web.xml and above issue will be resolved. This is a specific solution for JBoss. Tomcat and other server provide some other mechanism to solve this issue.

Cloud Computing a potential game changer in Goverment IT Space

E-governance is the mantra for all the state and union government now days. They are now spending heavily on IT systems to provide citizens speedy, efficient and transparent public services. Every department of government has now a website and some citizen centric IT application. But almost all of them struggling to provide citizens secure,reliable, highly available and scalable services. Main reason behind it is the lack of a centralized approach to handle IT system development, implementation,maintenance and up gradation. Currently almost all government departments develop and maintain its IT systems. They are responsible for development,procurement of hardware & software, setting up of IT infrastructure and further maintenance and up gradation. This is a big bottleneck in providing effective E-governance as these departments lack IT system lifecycle management skills.
Cloud computing can be an effective way to solve above problem. It is a cost effective and centralized solution for government e-governance initiative. It provides essentially two types of services
1. Platform as service
2. Software as service
Platform as service will solve all the issue related to hardware, software, security of applications,availability, scalability, reliability. Platform will be managed by specialists so quality of service will be great. Government departments can now focus on improving their processes and operations to provide quality services to citizens instead of grappling with IT system management. Another good thing will be that government can procure cloud space and services on need basis and scale up as services grow in demand in no time. So no lock in of money on
Hardware, Software for government.
Software as service will solve another major issue of application development and maintenance. Right now same department of different state governments are developing same type of custom IT applications for providing same type of citizen services. Some of the examples are MNREGA Application, PDS, Scholarships distribution, pension distribution and many more. Cloud based centralized application ,which are highly componentized, flexible, configurable and based on open standards, will solve major issues of application development, maintenance and up gradation.Such cloud based application can be configured quickly for different state government departments and they can roll out application quickly with minimum investment. This will help state governments to use its IT budget efficiently and optimally with assurance of quality applications and services.
Security is a big concern on cloud. So governments can go for combination of private and public cloud.Applications which are informative and less critical can be put on public cloud that to shared between different state governments which are still quite secure. Critical applications can be put on private cloud. These solutions will be quite secure then the current piece mill approach of IT system.
By embracing cloud government can optimally use it IT budget and provide great service to citizens.

UID usage in credit /debit card industry

UID (Aadhaar) will be a unique number which will be issued to every resident of India. Currently as per available data 13.46 crores residents have been issued Aadhaar number and these numbers will swell rapidly in coming days . This unique identity will open new vistas for streamlining and improving efficiency of many existing services.

In this post I would like to touch upon its usage in credit/debit card transctions. Currently credit/debit card transactions on POS assume that card holder is real card owner. No authentication check is done before transaction. This makes usage of credit/debit card very unsafe for card holder. Credit card industry is criticized severely by RBI and various customer forums for this. Though on web most of the credit card companies and now using two factor authentications, first by verifying
your filled details and then by authenticating using PIN. This feels somewhat
more secure.
UID can be used to plug this security hole for credit card. UIDAI provide authentication services for a given UID. A person can be authenticated using below combination:
1. UID+biometric(Finger Print)
2. UID+OTP
3. UID+Demographic attribute(name, address,DOB etc..)
4. UID+bio+OTP+Demographic
First two options are quite
suitable for credit/debit card industry perspective. Below are indicative
process flows:
Using UID+biometric:


Using UID+OTP:

Cost of using Open Source Products

From past 4-5 years i am using open source and branded software products for large enterprise applications. All of these applications had very large number of users base and were very critical to organization. Mostly i have used mix of open source and branded softwares. Though it is quite evident that branded software has a license and maintenance cost but it appears that open source products are "Free". I want to share my views on open source product usage challenges and cost associated with them.
Using open source in enterprise applications have number of challenges. First and foremost is understanding license, licenese of type GPL,Apache are quite open and can be used in commercial application and distributed to customer but licenses like BSD,Artistic are somewhat restrictive. So it is very essential to consult your organizations general counsel before using open source in your product/application.
Second, open source product works great in POC but when used full blown in projects then they give you very hard times because online forums are the only help avenue for such products. So these products are always a major risk to your project plan. I faced a lot of such issues in my project. Open source usage ask for High skill manpower which may become a bottleneck for project and add certain cost of skilled manpower.
Third challenge is maintaining these products as most open source projects are community base so dedicate support is not available for most of them.Moreover most of the time one open source product usage prompts you to use many related open source products which works great in tandem. Soon you have a bunch of open source products in your Application and now to find a person to manage all those is a nightmare. Thankfully now enterprise versions of all popular open source products are coming and they will ease somewhat customer pain. Maintainability is the major concern.
There are many more issues in using open source in an enterprise application but i will not dwell upon all these now. May be in next post.

Having said above things , i am great admirer of open source concept and products. I use all open source for any personal application development and POCs. But i want to bring some of the issues/costs of using such products for enterprise application. Just to Highlight " Open Source usage has a cost".

Cheers!!!!

Junk Characters while reading Excel using Jxl API

In my project, i have to read some data from an Excel file to use that for showing some data on JSP. Its a multilingual application so data has words in various european langauges. I have used Jxl api to read data from excel file. We have an ant task in our build file which runs at build time and read excel file to make a java file which is further used by JSP to show the data. This build when we run on windows every data in excel file comes correctly in java file but when we run the same build file on unix few of the records come with junk characters.

I investigated this issue and found that Solaris OS locale was default "en " which does not support all european characters. So first i changed that locale to en_US.UTF-8 which supports all European languages.( we have to install separate package to get en_US.UTF-8 locale )

After this change also junk character issue remains the same. Then i checked for the encoding option in the WorkbookSetting() class of Jxl. If we do not specify any encoding for workbook then it takes default encoding for that OS. I tried specifying UTF-8 as encoding for workbook explicitly but it did not work even on windows. Then i tried to get the encoding used by the window by using WorkbookSetting().getEncoding(). Then i come to know that window uses Cp1252 encoding. I then specified this encoding in work book setting using WorkbookSetting.setEncoding("Cp1252"). This solved issue of junk characters for me.

Using Eclipse TPTP Tool for Application profiling

One of my web based application has some issues related to performance . So I thought of profiling my application.I searched for the tools available for this and found a lot of tools. After analysis i found that TPTP tools from eclipse is suitable for my use. I installed latest release of eclipse ie. Galelio and futher installed "Test & Performance" tools using "Install New Software " option of eclipse work bench.
After installation i followed the "Tutorials" for using TPTP which was simple. But i could not get the list of collector options in the profile window as mentioned in tutorial. Work Bench keep me showing message "Fetching children of...ector (33%)" and i could not get any collector option in my profile window for long time.

I searched a lot but could not find any solution which worked for me.Then i investigated further and came to know that there is a local agent with the name (ARCServer) which is responsible for showing this collector. I searched in the task manager but could not found any such task. Then i got suspicious that it may happen that it is getting blocked by my local firewall. I turned off the firewall and to my surprise profile window started showing collector option.

It worked one day but next day i again faced the same issue despite of turning firewall off, i could not see collector options. Then i again checked ARCServer process it was not running. Then i thought of checking used ports as i saw in the agent configuration that it runs on port 10002. (You can check it by going to window->preferences->Agent Controller->Hosts). I used netstat -anb command to check that. I found that on 10002 port some other service was running. Then i killed that process and started again profiling. This time it showed me the option and i could profile my application. I thought of sharing it to help people who are facing same issue.