WS-Eventing: event notification failure

Still working on ws-eventing and JBossWS… We noticed that event notification failures are completely lost. If an event notification fails (for example because the event sink is down) the current jbossws engine (1.0.4.GA) just logs the error and does nothing else. Of course it could be useful at least to keep track of failed notification in order to send them again later or simply to warn the sink’s owner.

So we’ve just made a patch to jbossws, saving notification failures into a list stored in the SubscriptionManager mbean.

You can find the patch on jboss jira: http://jira.jboss.com/jira/browse/JBWS-1406

ws-eventing clustering problem

We got some troubles using ws-eventing jbossws implementation in our clustered/load balanced environment (for a deeper description read here). I’ll try to describe them in this post, with some proposed solutions:


Click here for a larger image.

The previous diagram describes the two main common problems we had with ws-eventing deployed in a clustered environment:

  • Subscription on one of the clustered environment’s machine

    DESCRIPTION: when a client subscribes to our ws-eventing service the load balancer sends his requests to a real server that receives the subscription and store it locally (in current jbossws eventing implementation this is achieved using a Map owned by SubscriptionManagerMbean). An event source generates a notification and delegates its notification to jbossws; but in a cluster environment this event source could use a different real server knowing nothing about the subscriber. The notification would be lost. See messages from 1 to 5 in the above diagram. Of course a very similar situation may happen when renewing a subscription: a renew message cold be send to a server different from the one that received the first subscription, resulting in subscription renew failure. The client will receive an “Unable to renew” exception, and on the server side the subscription will expire for timeout! All notifications are lost until client re-subscribe to the right real server. See messages from 6 to 13 in the above diagram.

    PROPOSED SOLUTION: These two problems could be solved clustering the SubscriptionMap. The easiest solution would probably be arranged using HAJNDI to store subscriptions. Of course the use of a local Map instead of HAJNDI have to be configured at deploy time.

  • Shutdown of one of the cluster’s machine

    DESCRIPTION: SubscriptionManagerMbean sends notifications to remove every subscriptions when it is stopped or the real server it runs on shutdowns. But in a cluster environment other servers continue to work and potentially to send notifications. Those notifications will be lost. See messages from 14 to 26 in the above diagram.

    PROPOSED SOLUTION: SubscriptionManagerMbeans have to be clustered and have to exchange their status before they can decide to send this kind of notifications. The solution could be implemented using jGroups (of course), but it needs further investigations and discussions, mainly to understand if SubscriptionManager is the only part of jbossws needing clustered solutions

I started a thread in JbossWS Design forum aboute this issue.

Document/Literal endpoint with MTOM+XOP attachment

During last months, we’ve been working to provide our customers with webservices with attachments. JBossWS currently (v.1.0.3) supports both SwA and MTOM+XOP although only the first one was available in earliest releases. For this reason, we started some month ago with SwA, then we added a new webservice endpoint with MTOM+XOP in RPC style. Finally, a couple of weeks ago some customers asked us to provide them with a Doc/Lit style endpoint in order to better support interoperability with Microsoft webservice clients. Considering that Document/Literal is the way to go for future webservices and that official JBossWS mtom samples are rpc-only, the process required to get your wonderfull doc/lit mtom endpoint may be interesting to be blogged ;-)
As long as JBossWS wstools.sh utility doesn’t generate correct artifacts when ran in java2wsdl mode on classes containing javax.activation.DataHandler instances, you have to do something like this:

  1. prepare a copy of your source file removing any reference to DataHandler from the enpoint interface and/or any data transfer object it contains
  2. prepare a wstools-config.xml: do not forget to set style=”document”. If your enpoint methods contain at most one parameter each, prefer setting parameter-style=”bare” since this will make you work easier on the following steps
  3. run wstools.sh on the modified sources: this will generate the wsdl, jaxrpc-mapping.xml, webservices.xml and eventually some classes
  4. manually modify the wsdl/xml artifacts to enable attachments: this is the most critical phase and the most boring one…
    • import xmime namespace in wsdl
      <import schemaLocation="http://www.w3.org/2005/05/xmlmime" namespace="http://www.w3.org/2005/05/xmlmime"/>
    • add/change elements for the attachments in the right point of your schema in the wsdl
      <element name='attachment' type='xmime:base64Binary' xmime:expectedContentTypes='*/*'/>
    • fix in/out method-param-parts-mappings and variable-mappings in jaxrpc-mapping.xml if required or whether you previously choose wrapped parameter style.
  5. restore your original source files (with DataHandler instances)
  6. eventually merge the webservices.xml obtained from wstools.sh with the one you already have for other endpoints: beware of “impl” namespaces references that of course are different for each endpoints
  7. build your project and look at the server.log on the console ;-)

Of course you can skip wstools.sh usage and simply manually write your artifacts considering you have to put your hands on them in any case… however I found this way very usefull since our schema is quite huge (wstools.sh saved me a lot of time… and bug hunt/fix).
Hope this post will help someone; anyway if you know a better, faster, etc way to achieve the same result before JBossWS 2.0 become available (http://jira.jboss.org/jira/browse/JBWS-1155)… feel free to post a comment below.

WS-Eventing and subscription expiration time

We just made a patch for jbossws implementing two features:

  1. The max expiration time and the default expiration time are hard coded in source code. Our patch made these values configurable in SubscriptionManager MBEAN
  2. We noticed jbossws throws an InvalideExpirationTime Excetion when we try to subscribe with an expiration time longer than maxLeaseTime. According to specification (5.4) this exception should be used only when the expiration time is in the past or the expiration duration is zero. It should answer with a valid registration forcing expiration time to maxLeaseTime (Specification 3.1).

Problems was discovered and reported by Alessio in this user’s forum post:
http://www.jboss.org/index.html?module=bb&op=viewtopic&t=90201

and briefly discussed by me in this design’s forum post:
http://www.jboss.com/index.html?module=bb&op=viewtopic&t=92340

You can find the patch (over jbossws-1.0.3.SP1 from jbossws svn) here or on jira:
http://jira.jboss.org/jira/browse/JBWS-1290

Multiple virtual host and soap:address problem and our patch to jbossws

Working on our ws we found a trouble with soap:address location in wsdl happening if the server answers to multiple addresses.
I’ll describe our environment to better explain the problem:

We have a cluster of about 10 jboss/linux servers with an active weighted load balancer (2 real machines in vrrp cluster active-active) in front of them (linux virtual server LVS). The virtual addresses configured in LVS are from 10 to 20 depending on test addresses configured for partners and similar situations. Some scripts monitor each real server’s healt and decide whether the cluster’s topology has to be changed. In order to have a simple and understandable description, I’m describing the production cluster only. We also have development and integration test clusters affected by the same issue I’m going to describe; in fact, these clusters are even more dynamic than the production one because of the frequent address changes, the high number of joined machines, the continuous restarts, and so on making it more concerned by the problem described here.

The WSDL, that is a required deployment artifact for an endpoint, has a soap:address element which points to the location of the endpoint. JBoss supports rewriting of that SOAP address.
The problem of soap:adress declared in wsdl in environment like our is the following: while a server answers to different virtual addresses, the location of the endpoint (soap:address) declared in the wsdl and obtained from the contextServlet could be either the address setted on the wsdl at deploy time or
the address setted in properties of the file jbossws.beans/META-INF/jboss-beans.xml (webServiceHost, webServicePort, alwaysModifySOAPAddress).

Both addresses are staticaly mapped (in the wsdl or on the server configuration file) and this is the problem in our environment (but would be a problem also on more trivial environment with one multiheaded machine!). Note that this issue concerns the wsdl generation phase only; of course webservice calls work perfectly with all virtual addresses. In fact the problem is present also on the ContextServlet (is the servlet called when you ask for the list of deployed services), but it’s a minor trouble.

For these reasons we decided to make a patch over jbossws. Our patch adds a configuration parameter in jbossws.sar/jbossws.beans/META-INF/jboss-beans.xml named rewriteSOAPAddressWithCalledHost (true/false). If this parameter is set to true soap:address is rewrited using the host and port used by the client to get the wsdl, and also the ContextServlet does the same. Of course it solves the problem because soap:address and url in ContextServlet depend on the Virtual host used while asking them.
Without this patch, the only solution for us was to provide our customers with personalized static wsdl having soap:address modified to match the correct virtual address. Of course it was time wasting and made change management very very hard.

Our patch is posted on jbossws jira at http://jira.jboss.com/jira/browse/JBWS-1178 hoping it could be useful for some other people. If you are interested in making it part of standard jbossws please vote for them!
I also attached it here You have to apply it on jbossws-1.0.3 source code (tag on jboss svn)