JBUG Milan: first meeting will focus on AS7

We (alessio and me) are going to have a speech on AS7 during the first JBUG Milan’s meeting next 20 Septmber.

If you are around Milan join us it will be a lot of fun. For your convenience I have pasted here the full agenda, but don’t forget to visit the official group and sign in for partecipation.

9.30: Welcome coffee
9.45: Welcome and first infos about JBUG
10.15: JBoss AS7
11.00: Coffee Break
11.15: JBoss AS7 and webservices
12.00: JBoss in cloud with OpenShift
12.45: Closing
13.00: luch
14:00: Hacking AS7

And don’t miss the hacking party we will have in the afternoon: Alessio and me will be in the public area of Red Hat office working on JBoss AS7, and would be cool to have you hacking with us. Don’t leave your laptop at home, join the community and send your first patch in that afternoon.

 

And now, for our italian reader, for this italian event…the post in italian language:

Alessio ed io faremo due presentazioni su AS7 durante il primo meeting del JBUG di Milano il prossimo 20 Settembre.

Se passte da Milano, non perdetevi l’evento, ci divertiremo. Per vostra comodità riporto qui l’agenda definitiva dell’incontro, ma non scordatevi di fare un giro sul gruppo ufficiale del JBUG ed iscrivetevi all’evento mandando la mail come indicato in uno dei messaggi.

9.30: Welcome coffee
9.45: Benvenuto e prime informazioni sul JBUG
10.15: JBoss AS7
11.00: Coffee Break
11.15: JBoss AS7 e i webservices
12.00: JBoss in cloud con OpenShift
12.45: Closing
13.00: Pranzo a buffet
14:00: Sviluppo di AS7

E mi raccomando, non perdetevi l’hacking su AS7 nel pomeriggio. Io ed Alessio ci fermeremo in area pubblica dell’ufficio Red Hat di Milano a lavorare su JBoss AS7. E sarebbe bello sviluppare insieme a voi. Non dimenticatevi il computer , partecipate alla community mandando il pomeriggio stesso la vostra prima patch ;)

 

See you there!

Ci vediamo li!

JBossWS wsconsume error “undefined element declaration ‘sch:schema’”

Today I was importing a .NET generated wsdl and getting this error using JBossWS wsconsume.sh tool (yes I’m not using Wise for this particular case!! But the problem would be the same in Wise since it use wsconsume under the hood) :

[WARNING] src-resolve.4.2: Error resolving component 'sch:schema'. It was detected that 'sch:schema' is in namespace 'http://www.w3.org/2001/XMLSchema', but components from this namespace are not referenceable from schema document 'file:/home/oracle/Desktop/security.asmx#types?schema1'. If this is the incorrect namespace, perhaps the prefix of 'sch:schema' needs to be changed. If this is the correct namespace, then an appropriate 'import' tag should be added to 'file:/home/oracle/Desktop/security.asmx#types?schema1'.
  line 85 of file:/home/oracle/Desktop/security.asmx#types?schema1
[ERROR] undefined element declaration 'sch:schema'
  line 85 of file:/home/oracle/Desktop/security.asmx
[ERROR] undefined element declaration 'sch:schema'
  line 178 of file:/home/oracle/Desktop/security.asmx
[ERROR] undefined element declaration 'sch:schema'
  line 217 of file:/home/oracle/Desktop/security.asmx
[ERROR] undefined element declaration 'sch:schema'
  line 231 of file:/home/oracle/Desktop/security.asmx
[ERROR] undefined element declaration 'sch:schema'
  line 245 of file:/home/oracle/Desktop/security.asmx
[ERROR] undefined element declaration 'sch:schema'
  line 259 of file:/home/oracle/Desktop/security.asmx
[ERROR] undefined element declaration 'sch:schema'
  line 273 of file:/home/oracle/Desktop/security.asmx

Failed to invoke WsImport
java.lang.NullPointerException
        at com.sun.tools.xjc.reader.internalizer.SCDBasedBindingSet.apply(SCDBasedBindingSet.java:237)
        at com.sun.tools.xjc.ModelLoader.createXSOM(ModelLoader.java:518)
        at com.sun.tools.xjc.api.impl.s2j.SchemaCompilerImpl.bind(SchemaCompilerImpl.java:232)
        at com.sun.tools.xjc.api.impl.s2j.SchemaCompilerImpl.bind(SchemaCompilerImpl.java:85)
        at com.sun.tools.ws.processor.modeler.wsdl.JAXBModelBuilder.bind(JAXBModelBuilder.java:134)
        at com.sun.tools.ws.processor.modeler.wsdl.WSDLModeler.buildJAXBModel(WSDLModeler.java:2244)
        at com.sun.tools.ws.processor.modeler.wsdl.WSDLModeler.internalBuildModel(WSDLModeler.java:187)
        at com.sun.tools.ws.processor.modeler.wsdl.WSDLModeler.buildModel(WSDLModeler.java:133)
        at com.sun.tools.ws.wscompile.WsimportTool.run(WsimportTool.java:182)
        at org.jboss.ws.tools.jaxws.impl.SunRIConsumerImpl.consume(SunRIConsumerImpl.java:217)
        at org.jboss.wsf.spi.tools.cmd.WSConsume.importServices(WSConsume.java:223)
        at org.jboss.wsf.spi.tools.cmd.WSConsume.main(WSConsume.java:81)

After some search Alessio pointed me to this excellent post where the problem is well explained and solution provided. The “problem” is not specific to JBossWS and solution explained there is fine for JBossWS too.

Hoping this cross post could help someone looking for the error in relation to JBossWS in google ;)

Reasons for a code donation

As my reader have learn by my last post I (and Alessio) have just donated Wise (and LMS) to JBoss.org.

But what does it mean? And more important why?

What does it mean?

Wise has been since the beginning released on LGPL license. Of course donating our code to JBoss.org we haven’t changed nothing about license. Wise still be an open source project and neither is changed, you still have code, you still have a business friendly open source license.

Code donation just change the Intellectual Property owners, switching from us to JBoss, divison of Red Hat. So it seems we are leaving all what we have on this project, but we are doing that for some good reasons.

And what does it mean for the project? More or less nothing. We are not donating and leaving project, but we will be active members (more precisely I’m going to lead the project). The good news are two new contributors from JBoss world are lending a hand with the code.

Why?

The real question is why have I decided to donate? Well, I’ve used “we” until now because Alessio and me have generally agreed on this donation. I’m switching to first singular person because Alessio kindly give me last word on the donation both because his position could condition him and because all last code written to transform wise from a general web client to zero code web service client easily integrable in a lot of server side application comes by me.

Reasons are essentially 4:

  • Wise is essentially an hacking against jbossws and jax-ws tools. Quite frankly I believe a pure hacking could be good for a personal use project, which more or less have been for years Wise. But when Wise have gotten a lot of interest integrating with the (cool) jbossesb project my point of view has changed. The question is: is possible to maintain wise and add features without having a strict join with jbossws and/or jbossesb with which is strictly dependent/integrated? And are there any better way to get it joined than make Wise a JBoss.org project? More users have been using Wise, and I have gotten some good feedback, but I think they need a serious approach of the project ensuring them Wise will not break (or at lest it will have great chance to don’t) it compatibility and interoperability which are the main reasons a lot of users is coming to us.
  • Becoming a JBoss.org project Wise would probably have a larger and greater community. It should mean more and better feedbacks and contribution (already verified since we have gotten 2 new contributors). More feedbacks and contribution mean better software and it’s all what I care.
  • I believe JBoss open source model is good. I think what they call professional open source is great. Maybe not all will agree with me, but it’s my own thoughts. It’s not planned at the moment, but I wouldn’t have any problem if one day Wise would be included in a JBoss product and someone will pay for support. Moreover I think it would be great, and for sure I wouldn’t have sufficient time to provide a professional support to them. I think open source is great, but home made product haven’t a real future, they need some kind of support, some company behind. I hope JBoss could be the company behind Wise.
  • Last but not least because I’m honoured of JBoss’s interest on my project. A lot of the strength spurring on an open source developer is narcissism ;) And I can ensure you sometimes it’s really needed with a little daughter and a lovely wife (with a great sense of humour) saying “ok, you don’t smoke, but you write code”

Have I taken a mistake? I don’t believe so. Anyway Wise is not going to an end, but to a bright new beginning: we have a lot of ideas, both increasing wise current capability and adding totally new features making it much more useful both in its traditional world (web services client/test) and in some new way I’m currently exploring and I’ll post about very soon. Stay tuned, and please join us and our building community contributing on our public forums, reporting bugs or writing some code. BTW Wise now have its own blog even if I’ll continue to cross post here for a while for wise’s older fans ;) .

Moreover we are not discontinuing this blog where we will continue to post our experiences and point of views not necesseraly related to Wise (BTW the two most read posts aren’t about Wise, but subversion branching and a new approach to unit tests).

Stay tuned.

JBossESB and Wise to implement ETL phase for a big DataWareHouse

As I wrote in some previous posts me and my fine team are working from a while to a project using JBossESB Wise action in a real world enterprise application. We are using it for the ETL (Extract Transfor Load) phase for a big DWH (Data Ware House) with an incremental loading of data.

In a nutshell we trace logical changes on an OLTP database (it’s a financial DB where all changes can be associated logically to a single company or at least to a network of company related for various reasons). Then we use JBossESB (and in particular SQLGateway) to periodically treat modified companies and extracting and enriching information to be loaded on the DWH instance. Where wise have its place? Well a lot of information and business rule to extract or enrich data have been implemented as webservices in last 3/4 years. So it’s pretty natural to reuse them to implement this last application.

Ok, it’s the bird eye view of the problem and the solution. On the rest of the post I’ll go in more formal details, starting with requirement and environment description

Requirement and environment description

The main requirement have been to collect a set of data regarding a large set of company (about 5 million) in a DWH for a marketing analysis. This data comes from different systems: 3 different OLTP relational database, and legacy host based system, an external provider. The good news is that both host system and external provider are accessible using webservices. Moreover OLTP databases have some webservices extracting data applying complex business rules; they doesn’t cover all requirements, but these DBs are completely under control of our development team, and dedicated jdbc and/or EJB3 access could be developed for new goals.

The final users would update it’s DWH with daily frequency. The large amount of data made impossible to extract transform and load the whole data every night. We have decided to keep track of changes on the main OLTP DB, and reload completely companies changed (some thousands a day).

Of course this approach isn’t totally new, incremental ETL are pretty common in DWH world, and all vendors have its own proprietary solution. While these proprietary system have its place and its plus, isn’t IMHO sufficient flexible to support an heterogeneous environment as one described. I thought it’s better to track with proprietary triggers logical significative changes (not a lot in fact) and adopt a SOA solution for ETL. It would be better in terms of flexibility and would permit us to reuse much more easily a lot of already written services containig complex business rules.

So the solution adopted have been based on JBossESB ant its composed by these macro steps:

  1. A set of triggers on 2 of 3 named OLTP DB collect changes and write a unique identifier of the company in a dedicated table
  2. A SQLGateway consume this table (the frequency of wake up and filters of the query are designed to avoid excessive and and not useful double treatment of companies due to double linked changes)
  3. Any company is processed by a set of action chains. This actions could be locally defined actions reading relational database or Wise based web services invocations. A content based router policy route messages from an action chain to the next one.
  4. Finally data extracted and transformed are written on the DWH.

Point 3 is of course the core of the system. The SQLGateway create a message containing a pojo object called Company and any successive action trasform or enrich this object with data collected and business rules applied. Wise’s based action calls webservices and use smooks to transform and enrich input object with ws returned values. Using CBR and continuous enrichment of the same object we get at last action (writeOnDWH) an object with all data needed t be written on the DWH.

Focus on Wise

A lot of actions are simply webservices calls implemented with a zero-code approach using Wise. We had just to write jboss-esb.xml fragment for webservice call and smooks config files to get a lot of business rules reused. It have been really GREAT!

I need to add some patch to current integration in ESB to obtain the max response from wise, but results have been really impressive: we had something like 90K company processed in an hour. What does it mean in finer details? Well from wise point of view about 300K web services calls in an hour!
Well also performance and numbers of ESB have been impressive: we are running on a single Linux64 machine (AMD64 double dual core) with 10 jms-listener processing 10 different chains  (200 concurrent 3ad for any jms-listener) for a total of 1.7M (wise and not) of actions called in an hour.

Isn’t it impressive numbers?

There is a list of patches I applied to wise/esb integration to support my requirement. All the code are committed on my workspace (maeste) in ESB svn:

Feature Request JBESB-2019 wise should pass to smooks response mapper also input data to permit continuos enrichement of message Major
Bug JBESB-2020 wise have a bug for which it may download too many wsdls and store them in a temporary dir Major
Feature Request JBESB-2021 add configurability for location where wise store smooks reports for its transformation Major
Bug JBESB-2022 wise doesn’t clean its internal smooks cache Major
Bug JBESB-2023 Wise is failed to consume a wsdl which contains two schema element with same name and different namespace . Major
Bug JBESB-2036 wise’s sample have problem because targetPackage not specified in properties files Major
Feature Request JBESB-2037 Avoid excessive reflective inspection of wise classes for better performance Major

I can’t go in more detail of the implementation or put here configs files because I cna’t reveal any business details of the application. I’ll try in next future to arrange an example totally equivalent in technology content, but without any link to real business content. If you are interested let me know, but be patients…it’s not a joke and I’m very very busy these days.

Thanks to my team (special thanks to Paolo and Luca)  and all contributors of Wise and ESB to make it possible :)

PS: what about huge split and route qs included in ESB 4.4. Well they cover different problems, even if not far each other. The main difference is that here we haven’t a huge message to split and route, but a lot of little message to enrich and then route (content based) to next enrichment phases.

JBossESB 4.4 have a new zero-code webservice invoker

We are proud to announce that recently released JBossESB 4.4 contain a wise based implementation of webservice client invoker.

In a nutshell it is a zero-code webservice caller supporting smooks based mapping, and pluggable JAX-WS handler. Here is an abstract of the message with which I presented it to ESB community (here you find original message and related discussion):

It uses wsconsume API to dynamically generate client object and invoke web service, delagating to JBossWS JAX-WS implementation the dirty job.
It use smooks under the hood to transform user defined object into JAX-WS generated ones.

It support also standard JAX-WS handler and a generic smooks transformation handler to apply transformation to generated soap messages.

You can find it in my workspace under product/services/soap/src/main/java/org/jboss/soa/esb/actions/soap/wise/
I also wrote javadoc for the action class explaining how to use it and e example demonstrating 3 common use case:

* Direct call of a simple service without any mapping is needed
* Call of a service using a smooks mapper java-to-java
* Call a simple webservices without mapping, but with an handler
modifying header with smooks and an handler logging on System.out
request and response
In this 3 examples don’t forget to have a look to wise-core.properties for some important configs. Of course they could be integrated in action’s config in jboss-esb.xml in next future, but this first implementation leave them there.

On wise roadmap I have the implementation of webservices’ call receiving different resources (CSV, XML and so on) using smooks to map it on JAX-WS generated client objects, giving another interesting opportunity in ESB environment.

It is an initial implementation, and I need to integrate wise objects generation with new smooks configgenerator ( http://milyn.codehaus.org/Smooks+User+Guide#SmooksUserGuide-GeneratingtheSmooksBindingConfiguration ) to make user experience easier.

Moreover we are working on wise-core to improve it and make it more configurable an pluggable and support much more stuffs. I’ll post a roadmap soon.

Stay tuned!