Legolas in Minas Tirith

ws-eventing clustering problem

We got some troubles using ws-eventing jbossws implementation in our clustered/load balanced environment (for a deeper description read here). I’ll try to describe them in this post, with some proposed solutions:

Click here for a larger image.

The previous diagram describes the two main common problems we had with ws-eventing deployed in a clustered environment:

• Subscription on one of the clustered environment’s machine
  

  DESCRIPTION: when a client subscribes to our ws-eventing service the load balancer sends his requests to a real server that receives the subscription and store it locally (in current jbossws eventing implementation this is achieved using a Map owned by SubscriptionManagerMbean). An event source generates a notification and delegates its notification to jbossws; but in a cluster environment this event source could use a different real server knowing nothing about the subscriber. The notification would be lost. See messages from 1 to 5 in the above diagram. Of course a very similar situation may happen when renewing a subscription: a renew message cold be send to a server different from the one that received the first subscription, resulting in subscription renew failure. The client will receive an “Unable to renew” exception, and on the server side the subscription will expire for timeout! All notifications are lost until client re-subscribe to the right real server. See messages from 6 to 13 in the above diagram.

  PROPOSED SOLUTION: These two problems could be solved clustering the SubscriptionMap. The easiest solution would probably be arranged using HAJNDI to store subscriptions. Of course the use of a local Map instead of HAJNDI have to be configured at deploy time.

• Shutdown of one of the cluster’s machine
  

  DESCRIPTION: SubscriptionManagerMbean sends notifications to remove every subscriptions when it is stopped or the real server it runs on shutdowns. But in a cluster environment other servers continue to work and potentially to send notifications. Those notifications will be lost. See messages from 14 to 26 in the above diagram.

  PROPOSED SOLUTION: SubscriptionManagerMbeans have to be clustered and have to exchange their status before they can decide to send this kind of notifications. The solution could be implemented using jGroups (of course), but it needs further investigations and discussions, mainly to understand if SubscriptionManager is the only part of jbossws needing clustered solutions

I started a thread in JbossWS Design forum aboute this issue.