JBoss DNA chat

I had an interesting chat with Randall Hauch about JBossDNA and JBossOverlord. Here is the log of our discussion. I put it online (with Randall’s authorization of course) because I think it could be an interesting reading and a good introduction to JBossDNA as it have been for me. I have also summarized here some interesting point also for the report between JBossDNA and JBossOverlord. you will find in this chat.

Apr 25 16:08:27 rhauch: Well, 0.1 is really about sequencers. We’re using Maven 2 for builds, and we have a Maven 2 classloader that we’ve chosen to put on hold.

Apr 25 16:08:35 rhauch: Re the classloader (real quick)
Apr 25 16:08:45 maeste: oki
Apr 25 16:09:11 rhauch: Basically, since DNA is to be very extensible (i.e., sequencers, analyses, connectors, etc.)
Apr 25 16:09:49 rhauch: we thought it be good to have a runtime repository of JARs
Apr 25 16:09:59 rhauch: structured the same was as Maven2
Apr 25 16:10:15 maeste: ok I understand
Apr 25 16:10:57 rhauch: Works like a charm, but writing the tooling was lower priority.
Apr 25 16:11:10 rhauch: tooling for loading an “extension”, for example. Or upgrading one.
Apr 25 16:11:34 rhauch: So, the first release really has centered around sequencers.
Apr 25 16:12:14 rhauch: Basically, DNA monitors 1+ JCR repositories, and when a client uploads some content
Apr 25 16:12:37 rhauch: the system figures out whether any of the registered sequencers would be interested
Apr 25 16:12:46 rhauch: if there are any, it runs them.
Apr 25 16:13:03 maeste: oki
Apr 25 16:13:13 rhauch: A sequencer processes the content, extracts any relevant information, and then saves the information back into the repository.
Apr 25 16:13:41 rhauch: The idea is that you can just upload various kinds of files, and DNA automatically extracts meaningful structured information.
Apr 25 16:14:16 maeste: oki as metadata I suppose
Apr 25 16:14:23 rhauch: The simplest example of a sequencer now is an one that processes image files, extracting metadata (size, date, etc.)
Apr 25 16:14:28 rhauch: Right.
Apr 25 16:15:01 rhauch: There are a lot of really interesting ones, tho.
Apr 25 16:15:07 maeste: Great….the killer apps in a soa governace point of view
Apr 25 16:15:13 maeste: and not only IMHO
Apr 25 16:15:16 rhauch: WSDL, XSD, …
Apr 25 16:15:23 rhauch: Exactly.
Apr 25 16:15:25 maeste: exactly
Apr 25 16:15:27 maeste: lol
Apr 25 16:15:39 maeste: policy ws-security and so on
Apr 25 16:15:42 rhauch: ha … great minds …
Apr 25 16:16:09 rhauch: MetaMatrix is interested in this because they have a lot of models of data sources
Apr 25 16:16:37 rhauch: and they’re tooling converts those models into info they use at runtime to integrate data.
Apr 25 16:16:52 maeste: ok
Apr 25 16:16:52 rhauch: But people want to know what models they have, and what’s in them.
Apr 25 16:17:06 rhauch: Same story for SOA – what services do I have, and what are the details?
Apr 25 16:17:19 maeste: About SOA
Apr 25 16:17:21 rhauch: What are the policies? Where are the services used in an orchestration, etc.
Apr 25 16:17:32 maeste: that is my main interest
Apr 25 16:17:37 rhauch: This is your area of interest?
Apr 25 16:17:40 rhauch: Ha!
Apr 25 16:18:10 maeste: yep I arrived to dna starting from soa governance project overlord
Apr 25 16:18:29 maeste: and discussig these points with Mark Little
Apr 25 16:18:38 rhauch: So, there are a lot of possibilities for different sequencers.
Apr 25 16:18:55 maeste: about policies: in few words sla definitions
Apr 25 16:19:03 rhauch: Right. He’s one of our main “users”.
Apr 25 16:19:28 rhauch: Sure, but also security policies… whatever is needed.
Apr 25 16:19:41 maeste: well for sure there are a lot of possibilities for sequencers
Apr 25 16:19:54 rhauch: The idea is that we can automatically extract information into JCR, and then use analyses and other tools to tie all that information together.
Apr 25 16:20:33 rhauch: Okay … the 0.1 release.
Apr 25 16:20:44 maeste: oki and how do you extract infos?
Apr 25 16:20:50 rhauch: Most of the coding is complete, and we just need to put together some documentation.
Apr 25 16:20:51 maeste: jcr api?
Apr 25 16:21:04 maeste: about 0.1 ok
Apr 25 16:21:14 rhauch: Do you mean, how do the sequencers extract the information?
Apr 25 16:21:31 rhauch: (No, we can discuss whatever you’re interested in…)
Apr 25 16:22:05 maeste: no, how clients can extract infos? is them put back in original jcr?
Apr 25 16:22:44 maeste: and how dna helps in query them?
Apr 25 16:22:53 rhauch: Yeah, the information that sequencers extract is put back into the JCR repository (perhaps a different one).
Apr 25 16:23:13 rhauch: At that point, clients can just use JCR … search, navigation, editing, etc.
Apr 25 16:23:19 rhauch: That’s where we are now.
Apr 25 16:23:34 rhauch: “Analyses” are another component of DNA …
Apr 25 16:23:38 maeste: oki, any plan for a query language different than xpath?
Apr 25 16:24:01 rhauch: Well, I see JCR 2 is moving away from XQuery and XPath, and more towards SQL.
Apr 25 16:24:07 rhauch: I’m not sure how I feel about that.
Apr 25 16:24:19 maeste: me too :)
Apr 25 16:24:31 rhauch: They’re really working toward a more generic abstract query language,
Apr 25 16:24:44 rhauch: and I think they’d like to have multiple query languages on top.
Apr 25 16:24:57 rhauch: But that different JCR implementations (products) would offer these as features.
Apr 25 16:24:57 maeste: oki like sparql have been for rdf
Apr 25 16:25:04 rhauch: Sure.
Apr 25 16:25:20 rhauch: In fact, you could probably implement SPARQL.
Apr 25 16:25:36 maeste: I agree
Apr 25 16:25:37 rhauch: As long as you can generate a query in their abstract query language (AQL)
Apr 25 16:26:34 maeste: sorry for a lot of interrupt and question, please go on with your explaination of the future of the project…you are saying about analysis
Apr 25 16:26:49 rhauch: No, don’t apologize.
Apr 25 16:27:01 rhauch: One more thing …
Apr 25 16:27:08 rhauch: Someone else on the forums has started working on a Java sequencer
Apr 25 16:27:26 maeste: I dind’t see
Apr 25 16:27:30 maeste: user forum?
Apr 25 16:27:31 rhauch: and another (from MetaMatrix) has just started working on a MetaMatrix model sequencer.
Apr 25 16:28:23 rhauch: Yeah, its listed on our project page (http://www.jboss.org/dna/)
Apr 25 16:28:40 rhauch: Here’s the forum: http://www.jboss.com/index.html?module=bb&op=viewforum&f=272
Apr 25 16:29:06 rhauch: Nobody has uploaded any code yet.
Apr 25 16:29:31 rhauch: So, maybe this is a good segway back to the 0.1 release.
Apr 25 16:29:43 maeste: I known forum, just I didn’t see the message discussing these stuffs :)
Apr 25 16:30:31 rhauch: See this thread: http://www.jboss.com/index.html?module=bb&op=viewtopic&t=132494
Apr 25 16:31:24 rhauch: Yeah, i know. we don’t have much documentation.
Apr 25 16:31:31 rhauch: :-(
Apr 25 16:31:55 rhauch: Anyway, I hope to release 0.1 next week.
Apr 25 16:32:10 rhauch: Most of the code is done, so we’re just working on some documentation.
Apr 25 16:32:30 rhauch: We’re going for frequent releases,
Apr 25 16:32:56 rhauch: and this initial release will get something stable that people can play with … primarily the sequencing framework.
Apr 25 16:33:36 rhauch: Then, we’ll start with the federation engine. I think that’ll be getting to some of the fun stuff.
Apr 25 16:34:14 maeste: As said in my mail to ML I’d like the idea to contribute in federation engine
Apr 25 16:37:30 rhauch: The federation engine is really going to be the interesting part of DNA.
Apr 25 16:37:41 rhauch: With it, we’ll be able to do a lot of very interesting things.
Apr 25 16:37:49 maeste: like?
Apr 25 16:38:00 rhauch: Well, remember the Maven 2 classloader?
Apr 25 16:38:04 maeste: yep
Apr 25 16:38:15 rhauch: The problem was getting stuff uploaded, right?
Apr 25 16:38:31 maeste: yep
Apr 25 16:38:42 rhauch: Well, we can have a connector that connects to an existing Maven 2 repository (via http), and presents it as if it’s already in a JCR repository.
Apr 25 16:39:07 rhauch: Then, the tooling can just copy from one JCR into another.
Apr 25 16:39:11 rhauch: That’s an example.
Apr 25 16:39:20 rhauch: Here’s another …
Apr 25 16:39:46 rhauch: A connector can talk to a relational database, and present the database schema information as if it’s already in JCR.
Apr 25 16:40:15 rhauch: Now I have a “federated JCR repository” that shows my database schemas live…. no copying, and it’s always up to date.
Apr 25 16:40:18 rhauch: Here’s another ….
Apr 25 16:40:35 rhauch: A connector can talk to a UDDI repository, and present it as if its in JCR.
Apr 25 16:40:58 rhauch: So now I can have a federated repository that lets me access my UDDI registry, but through JCR.
Apr 25 16:41:13 rhauch: And my policy information is also in the same federated JCR repository.
Apr 25 16:41:14 maeste: present a lot of information “as if it’s in JCR” this is the key
Apr 25 16:41:25 rhauch: That’s what the federation engine does …
Apr 25 16:41:31 maeste: exactly
Apr 25 16:41:51 rhauch: 1) connectors talk to some external system and build up subgraphs (as they’re queried/navigated)
Apr 25 16:42:06 rhauch: 2) integrate those different subgraphs into a unified graph, merging where required.
Apr 25 16:42:35 maeste: and then sequencer can do their job on this unified graph
Apr 25 16:42:41 rhauch: Bingo!
Apr 25 16:42:57 maeste: Great!
Apr 25 16:43:07 rhauch: Connector should be able to generate events to the federation engine (for example, new node at A/B/C/D)
Apr 25 16:43:27 rhauch: Now, here’s another connector that Mark’s team is interested in…
Apr 25 16:44:06 rhauch: A connector that sits on top of an SCM system (e.g, SVN, CVS, etc.), which is where the developers store their rules, policies and other artifacts.
Apr 25 16:44:36 rhauch: That connector should make those available to any tooling (e.g., Guvn’r/BRMS) that sits on top of JCR.
Apr 25 16:44:54 maeste: yes of course
Apr 25 16:45:14 rhauch: Data should be stored (as master data) only in one spot, and the federation engine lets this happen.
Apr 25 16:45:20 rhauch: Leave the data in SVN if that’s the master data.
Apr 25 16:45:28 rhauch: Or in the database (for the schema).
Apr 25 16:45:29 maeste: and maybe also a connector collecting runtime infos of soa environments
Apr 25 16:45:40 rhauch: Yes, exactly.
Apr 25 16:45:56 maeste: to collect status and how slas are respected or not
Apr 25 16:46:12 rhauch: Right. That’s a perfect example.
Apr 25 16:46:43 maeste: well you were telling me about analyzer
Apr 25 16:46:49 rhauch: Right ….
Apr 25 16:47:14 rhauch: analyzer are similar to sequencers in the sense that they’re automatically run based upon some criteria being met.
Apr 25 16:47:26 maeste: oki
Apr 25 16:47:35 rhauch: In the case of an analyzer, this criteria might be a scheduler,
Apr 25 16:47:45 rhauch: or it might watch for changes under certain paths.
Apr 25 16:48:07 rhauch: Unlike sequencers which really target a node’s content,
Apr 25 16:48:25 maeste: oki
Apr 25 16:48:37 rhauch: analyzers really process subgraphs as input, and produce output in the form of a subgraph
Apr 25 16:48:44 maeste: so it’s supposed to make some kind of massive job
Apr 25 16:48:46 rhauch: Essentially, they’re report generators.
Apr 25 16:49:17 maeste: oki and they don’t change jcr content, right?
Apr 25 16:49:26 rhauch: Yeah, the idea is that they could be set up to find statistics… about adhering to sla’s for example.
Apr 25 16:49:38 maeste: oki
Apr 25 16:49:46 rhauch: and then write those statistics to an area where they can be found.
Apr 25 16:50:13 maeste: I have a clear ides on how they can be useful in overlord
Apr 25 16:50:23 rhauch: So, a web tool (lets say) can look at that report in JCR. It’s always there, and the analyzers make sure it’s kept up to date.
Apr 25 16:50:25 maeste: exactly for sla reports
Apr 25 16:50:56 maeste: lol I’m thinking the same thing a web tools called overlord :D
Apr 25 16:51:10 rhauch: Basically, analyzers are big queries (or sets of queries) that take a while to run.
Apr 25 16:51:24 rhauch: So can you explain what overlord is?
Apr 25 16:51:39 rhauch: I’ve heard bits and pieces.
Apr 25 16:51:41 maeste: It’s not totally clear
Apr 25 16:51:59 maeste: it’s the project created to make a soa governace project
Apr 25 16:52:14 maeste: and it would include 2 kinds of tools
Apr 25 16:52:19 maeste: design time tools
Apr 25 16:52:39 rhauch: like the service modeler that Mark pointed to in the forums?
Apr 25 16:52:46 maeste: service modeler for instance based on Thomas Erl’s donation
Apr 25 16:52:53 maeste: lol
Apr 25 16:53:01 rhauch: … great minds … :-)
Apr 25 16:53:14 maeste: and the other kinds it’s runtime governance tools
Apr 25 16:53:36 maeste: that would be some kind of (web) console
Apr 25 16:54:08 maeste: to collect infos and statistic of the soa environments
Apr 25 16:54:17 rhauch: ah!
Apr 25 16:54:23 maeste: slas, error, number of class and so on
Apr 25 16:54:40 maeste: and it would be able to change also
Apr 25 16:54:55 maeste: some paramiter runtime
Apr 25 16:55:06 maeste: to have policies enforcements and so on
Apr 25 16:55:30 maeste: in fact a lot of these things need also a developments under the hood
Apr 25 16:56:07 maeste: since for example there is no standard to define slas in ws-* or oasis panorama
Apr 25 16:56:15 maeste: but of course we need one
Apr 25 16:57:23 maeste: and it would contain also a BAM or as mark said better a SAM (service Activity Manager)
Apr 25 16:57:38 maeste: and of course a ws-cdl runtime tool too
Apr 25 16:57:58 maeste: and of course a lot of other stuffs we didn’t thought (yet)
Apr 25 16:58:28 rhauch: Right, and you’re thinking a repository for the artifacts, plus a place (repository?) to store the collected data?
Apr 25 16:58:32 maeste: As you can figure out there are a lot of point where we need a good repository
Apr 25 16:58:49 maeste: lol and rotfl
Apr 25 16:58:56 rhauch: :-)
Apr 25 16:59:03 maeste: yep we are thinking to dna in fact
Apr 25 16:59:39 maeste: because a jcr repository isn’t sufficient, we need some sequencer and analyzer for sure
Apr 25 17:00:02 rhauch: Yeah, the JCR API is fairly low level — it’s basically just a graph API.
Apr 25 17:00:40 maeste: As said overlord is just started, and we (me and Mark) think the best starting point for my contribution is to be involved in dna
Apr 25 17:00:43 rhauch: The functionality is where the good stuff is.
Apr 25 17:00:52 maeste: to have a full understand of it
Apr 25 17:00:59 maeste: at least
Apr 25 17:01:10 maeste: BTW dna it’s big fun
Apr 25 17:01:23 rhauch: i hope that’s not sarcasm. :-)
Apr 25 17:01:35 maeste: I had already a copy of trunk on my laptop before Mark told me about ;)
Apr 25 17:01:44 maeste: Absolutely not
Apr 25 17:02:06 rhauch: Well, we’re glad your interested!
Apr 25 17:02:11 maeste: are there sarcasm in my phrase? If yes, I’m sorry for my english
Apr 25 17:02:52 rhauch: No, just my bad sense of humor not coming through…
Apr 25 17:02:57 rhauch: You’re english is great.
Apr 25 17:03:48 rhauch: There’s another area of DNA that is a bit further out on the roadmap … RESTful server.
Apr 25 17:04:14 rhauch: I thought I’d mention it in case that was much more up your alley.
Apr 25 17:04:46 rhauch: But if you’re interested in the federation engine, then there’s certainly a lot of work there!
Apr 25 17:05:40 maeste: Yep I’m interested in federation engine for sure
Apr 25 17:16:57 rhauch: The federation engine will have a couple of major components.
Apr 25 17:17:51 rhauch: At its core is a graph, stored in a cache and populated from the connectors.
Apr 25 17:18:05 rhauch: On top of that the fed engine will implement JCR API.
Apr 25 17:18:11 maeste: cache…JBoss cache? Pojo cache?
Apr 25 17:18:32 rhauch: Well, I’d like to not tie it to any particular cache, but I think we’d use JBoss cache.
Apr 25 17:18:48 rhauch: Pojo cache seems a bit heavy for this purpose.
Apr 25 17:19:12 maeste: yes of course not tie to a particular implementation is always good
Apr 25 17:19:14 rhauch: It is more efficient in a distributed sense, but it requires a lot of AOP weaving.
Apr 25 17:20:23 rhauch: anyway, so there are a couple of different areas.
Apr 25 17:20:57 maeste: yep
Apr 25 17:21:02 rhauch: As far as the JCR API implementation, we’ll want to set up the JCR TCK relatively early.
Apr 25 17:21:37 rhauch: I think we’ll have the mid-level graph API done pretty quickly, so we’ll definitely want your input there.
Apr 25 17:22:07 rhauch: BTW, my thought on that graph API is that the federation engine & cache should basically be a simple graph with versioning.
Apr 25 17:22:43 rhauch: If we do that, then we can put really almost any API on top of that, including UDDI (should we ever want to do that).
Apr 25 17:23:11 rhauch: If you look at the JCR API, there’s a lot of stuff that should be implemented on top of a more simple graph…
Apr 25 17:23:38 rhauch: node types, namespaces, version history, etc.
Apr 25 17:23:56 rhauch: They all need a simple graph that supports versioning.
Apr 25 17:24:12 rhauch: Anyway, that’s the middle of the federation engine.
Apr 25 17:24:16 rhauch: Then there are the connectors.
Apr 25 17:24:59 rhauch: I think we have JIRA feature requests for several different kinds of connectors, with a few initially slotted earlier than the others.
Apr 25 17:25:15 rhauch: I hope I’m making at least some sense.
Apr 25 17:25:21 maeste: ok, I understand

The future of wise (And Lms)

It’s time to discuss with the community the future of Wise and Lms. First of all a clarification: Alessio and me have being very busy in last months, but we are not planning to stop development of Wise and Lms.

We have 2 different approach possible, not mutually exclusive each other:

  1. integrate wise and Lms adding some functionality like saving request/response and code generation, making wise+lms a suite to test webservice, keeping QA people in mind and their relation with development people. We are thinking in this area about functionality to save, review and resend request and a common dashboard to manage wsdls, servers and so on.
  2. Separate wise-core (dynamic generation of jaxws client and object exchange) from wise-gui (or wise-seam as you like to call it). This step would be important to use wise-core as generic API to call webservice

Ok, first point is well known for our blog’s readers, but why we are starting a discussion now about the second point? Well, it comes from the idea to use wise to make an action for jbossesb for call web service instead of the current implementation. I took a look to what jbossesb is doing at the moment in this area, and I thought wise can do a lot of these things, using standard JAX-WS wsconsume APIC generated classes.

I don’t want to discuss here the specif needs of jbossesb, but in general I believe it could be a good approach when you need a total and effective decoupling of the client and server using ws to inter operate. It’s matter of fact that wsconsume tools is great for java developer, generating needed stub class, but it introduce a new (or renewed :) ) level of coupling very similar to corba IDL. Generating statically webservice stub you are in fact coupling client and server.

So what is the alternative? Writing dynamic client using Dynamic dispatch JAX-WS API? Maybe, but I would prefer a solution using dynamic mapping on generated stub, exactly what wise 1.x does. In fact Wise did a mapping between a generuic user interface and generated stub objects. My idea is to extend this concept and permit to call a webservice mapping a generic Object Model to jaxws generated one. Are we reinventing the wheel? I don’t think so because IMHO JAX-WS dynamic dispatch isn’t a perfect turning wheel, and I hope Wise-core does the job better ;)

How wise-core does perform this generic task? In a nutshell it does the same wise 1.x did, generating on the fly classes using wsconsume runtime API, loading them in current class loader and use them with Java Reflection API. What we add is generic mapping API to transform an arbitrary object model in wsconsume generated ones, make the call, and mapping the answer back again to custom model.

The implementation can be viewed in our svn (trunk), and it’s based on a refactoring of our original code and an extensive use of Smooks. Smooks is an amazing tool! Have a look! I will post about shortly, stay tuned!

Here is some ideas done or in plan in the wise-core

  • Much more unit tests. The core was really poor in this area. [DONE]
  • Calling WS using a custom object model, and delegating to wise-core the responsability of required mapping action [DONE]
  • Adding jaxws client handler(custom one and some ones implemented in the core, at least one smooks’ based [DONE]
  • smooks based javabeans mapping from user object to wsconsume generated object [DONE]
  • write examples of use [partially DONE]
  • write good javadoc and user manual [TODO]
  • write integration and example of integration with jbossesb [DONE, not in repository]
  • generating Smooks config skeleton for generated object. Without them who write the smooks config file for mappers have to know the classes hierarchy generated by wsconsume (IOW generate it statically first time to have a look :) ) [TODO]
  • Use wise-core in wise-gui, using smooks mapper against our proprietary TreeElement Object to map to generated classes [TODO, it needs next point]
  • Dynamic generation of smooks config knowing target Object hirearchy (our TreeElement) and wsconsume’s generated objects. [TODO]
  • Evaluate different mapping tools (to be added to smooks, not to substitute it for sure!). Maybe Dozer, even if I suspect some problem with it for our goals. [TODO]

In our svn you can find a test demostrating the use in a stand alone application.

any opinion?