I had an interesting chat with Randall Hauch about JBossDNA and JBossOverlord. Here is the log of our discussion. I put it online (with Randall’s authorization of course) because I think it could be an interesting reading and a good introduction to JBossDNA as it have been for me. I have also summarized here some interesting point also for the report between JBossDNA and JBossOverlord. you will find in this chat.

Apr 25 16:08:27 rhauch: Well, 0.1 is really about sequencers. We’re using Maven 2 for builds, and we have a Maven 2 classloader that we’ve chosen to put on hold.

Apr 25 16:08:35 rhauch: Re the classloader (real quick)
Apr 25 16:08:45 maeste: oki
Apr 25 16:09:11 rhauch: Basically, since DNA is to be very extensible (i.e., sequencers, analyses, connectors, etc.)
Apr 25 16:09:49 rhauch: we thought it be good to have a runtime repository of JARs
Apr 25 16:09:59 rhauch: structured the same was as Maven2
Apr 25 16:10:15 maeste: ok I understand
Apr 25 16:10:57 rhauch: Works like a charm, but writing the tooling was lower priority.
Apr 25 16:11:10 rhauch: tooling for loading an “extension”, for example. Or upgrading one.
Apr 25 16:11:34 rhauch: So, the first release really has centered around sequencers.
Apr 25 16:12:14 rhauch: Basically, DNA monitors 1+ JCR repositories, and when a client uploads some content
Apr 25 16:12:37 rhauch: the system figures out whether any of the registered sequencers would be interested
Apr 25 16:12:46 rhauch: if there are any, it runs them.
Apr 25 16:13:03 maeste: oki
Apr 25 16:13:13 rhauch: A sequencer processes the content, extracts any relevant information, and then saves the information back into the repository.
Apr 25 16:13:41 rhauch: The idea is that you can just upload various kinds of files, and DNA automatically extracts meaningful structured information.
Apr 25 16:14:16 maeste: oki as metadata I suppose
Apr 25 16:14:23 rhauch: The simplest example of a sequencer now is an one that processes image files, extracting metadata (size, date, etc.)
Apr 25 16:14:28 rhauch: Right.
Apr 25 16:15:01 rhauch: There are a lot of really interesting ones, tho.
Apr 25 16:15:07 maeste: Great….the killer apps in a soa governace point of view
Apr 25 16:15:13 maeste: and not only IMHO
Apr 25 16:15:16 rhauch: WSDL, XSD, …
Apr 25 16:15:23 rhauch: Exactly.
Apr 25 16:15:25 maeste: exactly
Apr 25 16:15:27 maeste: lol
Apr 25 16:15:39 maeste: policy ws-security and so on
Apr 25 16:15:42 rhauch: ha … great minds …
Apr 25 16:16:09 rhauch: MetaMatrix is interested in this because they have a lot of models of data sources
Apr 25 16:16:37 rhauch: and they’re tooling converts those models into info they use at runtime to integrate data.
Apr 25 16:16:52 maeste: ok
Apr 25 16:16:52 rhauch: But people want to know what models they have, and what’s in them.
Apr 25 16:17:06 rhauch: Same story for SOA – what services do I have, and what are the details?
Apr 25 16:17:19 maeste: About SOA
Apr 25 16:17:21 rhauch: What are the policies? Where are the services used in an orchestration, etc.
Apr 25 16:17:32 maeste: that is my main interest
Apr 25 16:17:37 rhauch: This is your area of interest?
Apr 25 16:17:40 rhauch: Ha!
Apr 25 16:18:10 maeste: yep I arrived to dna starting from soa governance project overlord
Apr 25 16:18:29 maeste: and discussig these points with Mark Little
Apr 25 16:18:38 rhauch: So, there are a lot of possibilities for different sequencers.
Apr 25 16:18:55 maeste: about policies: in few words sla definitions
Apr 25 16:19:03 rhauch: Right. He’s one of our main “users”.
Apr 25 16:19:28 rhauch: Sure, but also security policies… whatever is needed.
Apr 25 16:19:41 maeste: well for sure there are a lot of possibilities for sequencers
Apr 25 16:19:54 rhauch: The idea is that we can automatically extract information into JCR, and then use analyses and other tools to tie all that information together.
Apr 25 16:20:33 rhauch: Okay … the 0.1 release.
Apr 25 16:20:44 maeste: oki and how do you extract infos?
Apr 25 16:20:50 rhauch: Most of the coding is complete, and we just need to put together some documentation.
Apr 25 16:20:51 maeste: jcr api?
Apr 25 16:21:04 maeste: about 0.1 ok
Apr 25 16:21:14 rhauch: Do you mean, how do the sequencers extract the information?
Apr 25 16:21:31 rhauch: (No, we can discuss whatever you’re interested in…)
Apr 25 16:22:05 maeste: no, how clients can extract infos? is them put back in original jcr?
Apr 25 16:22:44 maeste: and how dna helps in query them?
Apr 25 16:22:53 rhauch: Yeah, the information that sequencers extract is put back into the JCR repository (perhaps a different one).
Apr 25 16:23:13 rhauch: At that point, clients can just use JCR … search, navigation, editing, etc.
Apr 25 16:23:19 rhauch: That’s where we are now.
Apr 25 16:23:34 rhauch: “Analyses” are another component of DNA …
Apr 25 16:23:38 maeste: oki, any plan for a query language different than xpath?
Apr 25 16:24:01 rhauch: Well, I see JCR 2 is moving away from XQuery and XPath, and more towards SQL.
Apr 25 16:24:07 rhauch: I’m not sure how I feel about that.
Apr 25 16:24:19 maeste: me too
Apr 25 16:24:31 rhauch: They’re really working toward a more generic abstract query language,
Apr 25 16:24:44 rhauch: and I think they’d like to have multiple query languages on top.
Apr 25 16:24:57 rhauch: But that different JCR implementations (products) would offer these as features.
Apr 25 16:24:57 maeste: oki like sparql have been for rdf
Apr 25 16:25:04 rhauch: Sure.
Apr 25 16:25:20 rhauch: In fact, you could probably implement SPARQL.
Apr 25 16:25:36 maeste: I agree
Apr 25 16:25:37 rhauch: As long as you can generate a query in their abstract query language (AQL)
Apr 25 16:26:34 maeste: sorry for a lot of interrupt and question, please go on with your explaination of the future of the project…you are saying about analysis
Apr 25 16:26:49 rhauch: No, don’t apologize.
Apr 25 16:27:01 rhauch: One more thing …
Apr 25 16:27:08 rhauch: Someone else on the forums has started working on a Java sequencer
Apr 25 16:27:26 maeste: I dind’t see
Apr 25 16:27:30 maeste: user forum?
Apr 25 16:27:31 rhauch: and another (from MetaMatrix) has just started working on a MetaMatrix model sequencer.
Apr 25 16:28:23 rhauch: Yeah, its listed on our project page (http://www.jboss.org/dna/)
Apr 25 16:28:40 rhauch: Here’s the forum: http://www.jboss.com/index.html?module=bb&op=viewforum&f=272
Apr 25 16:29:06 rhauch: Nobody has uploaded any code yet.
Apr 25 16:29:31 rhauch: So, maybe this is a good segway back to the 0.1 release.
Apr 25 16:29:43 maeste: I known forum, just I didn’t see the message discussing these stuffs
Apr 25 16:30:31 rhauch: See this thread: http://www.jboss.com/index.html?module=bb&op=viewtopic&t=132494
Apr 25 16:31:24 rhauch: Yeah, i know. we don’t have much documentation.
Apr 25 16:31:31 rhauch:
Apr 25 16:31:55 rhauch: Anyway, I hope to release 0.1 next week.
Apr 25 16:32:10 rhauch: Most of the code is done, so we’re just working on some documentation.
Apr 25 16:32:30 rhauch: We’re going for frequent releases,
Apr 25 16:32:56 rhauch: and this initial release will get something stable that people can play with … primarily the sequencing framework.
Apr 25 16:33:36 rhauch: Then, we’ll start with the federation engine. I think that’ll be getting to some of the fun stuff.
Apr 25 16:34:14 maeste: As said in my mail to ML I’d like the idea to contribute in federation engine
Apr 25 16:37:30 rhauch: The federation engine is really going to be the interesting part of DNA.
Apr 25 16:37:41 rhauch: With it, we’ll be able to do a lot of very interesting things.
Apr 25 16:37:49 maeste: like?
Apr 25 16:38:00 rhauch: Well, remember the Maven 2 classloader?
Apr 25 16:38:04 maeste: yep
Apr 25 16:38:15 rhauch: The problem was getting stuff uploaded, right?
Apr 25 16:38:31 maeste: yep
Apr 25 16:38:42 rhauch: Well, we can have a connector that connects to an existing Maven 2 repository (via http), and presents it as if it’s already in a JCR repository.
Apr 25 16:39:07 rhauch: Then, the tooling can just copy from one JCR into another.
Apr 25 16:39:11 rhauch: That’s an example.
Apr 25 16:39:20 rhauch: Here’s another …
Apr 25 16:39:46 rhauch: A connector can talk to a relational database, and present the database schema information as if it’s already in JCR.
Apr 25 16:40:15 rhauch: Now I have a “federated JCR repository” that shows my database schemas live…. no copying, and it’s always up to date.
Apr 25 16:40:18 rhauch: Here’s another ….
Apr 25 16:40:35 rhauch: A connector can talk to a UDDI repository, and present it as if its in JCR.
Apr 25 16:40:58 rhauch: So now I can have a federated repository that lets me access my UDDI registry, but through JCR.
Apr 25 16:41:13 rhauch: And my policy information is also in the same federated JCR repository.
Apr 25 16:41:14 maeste: present a lot of information “as if it’s in JCR” this is the key
Apr 25 16:41:25 rhauch: That’s what the federation engine does …
Apr 25 16:41:31 maeste: exactly
Apr 25 16:41:51 rhauch: 1) connectors talk to some external system and build up subgraphs (as they’re queried/navigated)
Apr 25 16:42:06 rhauch: 2) integrate those different subgraphs into a unified graph, merging where required.
Apr 25 16:42:35 maeste: and then sequencer can do their job on this unified graph
Apr 25 16:42:41 rhauch: Bingo!
Apr 25 16:42:57 maeste: Great!
Apr 25 16:43:07 rhauch: Connector should be able to generate events to the federation engine (for example, new node at A/B/C/D)
Apr 25 16:43:27 rhauch: Now, here’s another connector that Mark’s team is interested in…
Apr 25 16:44:06 rhauch: A connector that sits on top of an SCM system (e.g, SVN, CVS, etc.), which is where the developers store their rules, policies and other artifacts.
Apr 25 16:44:36 rhauch: That connector should make those available to any tooling (e.g., Guvn’r/BRMS) that sits on top of JCR.
Apr 25 16:44:54 maeste: yes of course
Apr 25 16:45:14 rhauch: Data should be stored (as master data) only in one spot, and the federation engine lets this happen.
Apr 25 16:45:20 rhauch: Leave the data in SVN if that’s the master data.
Apr 25 16:45:28 rhauch: Or in the database (for the schema).
Apr 25 16:45:29 maeste: and maybe also a connector collecting runtime infos of soa environments
Apr 25 16:45:40 rhauch: Yes, exactly.
Apr 25 16:45:56 maeste: to collect status and how slas are respected or not
Apr 25 16:46:12 rhauch: Right. That’s a perfect example.
Apr 25 16:46:43 maeste: well you were telling me about analyzer
Apr 25 16:46:49 rhauch: Right ….
Apr 25 16:47:14 rhauch: analyzer are similar to sequencers in the sense that they’re automatically run based upon some criteria being met.
Apr 25 16:47:26 maeste: oki
Apr 25 16:47:35 rhauch: In the case of an analyzer, this criteria might be a scheduler,
Apr 25 16:47:45 rhauch: or it might watch for changes under certain paths.
Apr 25 16:48:07 rhauch: Unlike sequencers which really target a node’s content,
Apr 25 16:48:25 maeste: oki
Apr 25 16:48:37 rhauch: analyzers really process subgraphs as input, and produce output in the form of a subgraph
Apr 25 16:48:44 maeste: so it’s supposed to make some kind of massive job
Apr 25 16:48:46 rhauch: Essentially, they’re report generators.
Apr 25 16:49:17 maeste: oki and they don’t change jcr content, right?
Apr 25 16:49:26 rhauch: Yeah, the idea is that they could be set up to find statistics… about adhering to sla’s for example.
Apr 25 16:49:38 maeste: oki
Apr 25 16:49:46 rhauch: and then write those statistics to an area where they can be found.
Apr 25 16:50:13 maeste: I have a clear ides on how they can be useful in overlord
Apr 25 16:50:23 rhauch: So, a web tool (lets say) can look at that report in JCR. It’s always there, and the analyzers make sure it’s kept up to date.
Apr 25 16:50:25 maeste: exactly for sla reports
Apr 25 16:50:56 maeste: lol I’m thinking the same thing a web tools called overlord
Apr 25 16:51:10 rhauch: Basically, analyzers are big queries (or sets of queries) that take a while to run.
Apr 25 16:51:24 rhauch: So can you explain what overlord is?
Apr 25 16:51:39 rhauch: I’ve heard bits and pieces.
Apr 25 16:51:41 maeste: It’s not totally clear
Apr 25 16:51:59 maeste: it’s the project created to make a soa governace project
Apr 25 16:52:14 maeste: and it would include 2 kinds of tools
Apr 25 16:52:19 maeste: design time tools
Apr 25 16:52:39 rhauch: like the service modeler that Mark pointed to in the forums?
Apr 25 16:52:46 maeste: service modeler for instance based on Thomas Erl’s donation
Apr 25 16:52:53 maeste: lol
Apr 25 16:53:01 rhauch: … great minds …
Apr 25 16:53:14 maeste: and the other kinds it’s runtime governance tools
Apr 25 16:53:36 maeste: that would be some kind of (web) console
Apr 25 16:54:08 maeste: to collect infos and statistic of the soa environments
Apr 25 16:54:17 rhauch: ah!
Apr 25 16:54:23 maeste: slas, error, number of class and so on
Apr 25 16:54:40 maeste: and it would be able to change also
Apr 25 16:54:55 maeste: some paramiter runtime
Apr 25 16:55:06 maeste: to have policies enforcements and so on
Apr 25 16:55:30 maeste: in fact a lot of these things need also a developments under the hood
Apr 25 16:56:07 maeste: since for example there is no standard to define slas in ws-* or oasis panorama
Apr 25 16:56:15 maeste: but of course we need one
Apr 25 16:57:23 maeste: and it would contain also a BAM or as mark said better a SAM (service Activity Manager)
Apr 25 16:57:38 maeste: and of course a ws-cdl runtime tool too
Apr 25 16:57:58 maeste: and of course a lot of other stuffs we didn’t thought (yet)
Apr 25 16:58:28 rhauch: Right, and you’re thinking a repository for the artifacts, plus a place (repository?) to store the collected data?
Apr 25 16:58:32 maeste: As you can figure out there are a lot of point where we need a good repository
Apr 25 16:58:49 maeste: lol and rotfl
Apr 25 16:58:56 rhauch:
Apr 25 16:59:03 maeste: yep we are thinking to dna in fact
Apr 25 16:59:39 maeste: because a jcr repository isn’t sufficient, we need some sequencer and analyzer for sure
Apr 25 17:00:02 rhauch: Yeah, the JCR API is fairly low level — it’s basically just a graph API.
Apr 25 17:00:40 maeste: As said overlord is just started, and we (me and Mark) think the best starting point for my contribution is to be involved in dna
Apr 25 17:00:43 rhauch: The functionality is where the good stuff is.
Apr 25 17:00:52 maeste: to have a full understand of it
Apr 25 17:00:59 maeste: at least
Apr 25 17:01:10 maeste: BTW dna it’s big fun
Apr 25 17:01:23 rhauch: i hope that’s not sarcasm.
Apr 25 17:01:35 maeste: I had already a copy of trunk on my laptop before Mark told me about
Apr 25 17:01:44 maeste: Absolutely not
Apr 25 17:02:06 rhauch: Well, we’re glad your interested!
Apr 25 17:02:11 maeste: are there sarcasm in my phrase? If yes, I’m sorry for my english
Apr 25 17:02:52 rhauch: No, just my bad sense of humor not coming through…
Apr 25 17:02:57 rhauch: You’re english is great.
Apr 25 17:03:48 rhauch: There’s another area of DNA that is a bit further out on the roadmap … RESTful server.
Apr 25 17:04:14 rhauch: I thought I’d mention it in case that was much more up your alley.
Apr 25 17:04:46 rhauch: But if you’re interested in the federation engine, then there’s certainly a lot of work there!
Apr 25 17:05:40 maeste: Yep I’m interested in federation engine for sure
Apr 25 17:16:57 rhauch: The federation engine will have a couple of major components.
Apr 25 17:17:51 rhauch: At its core is a graph, stored in a cache and populated from the connectors.
Apr 25 17:18:05 rhauch: On top of that the fed engine will implement JCR API.
Apr 25 17:18:11 maeste: cache…JBoss cache? Pojo cache?
Apr 25 17:18:32 rhauch: Well, I’d like to not tie it to any particular cache, but I think we’d use JBoss cache.
Apr 25 17:18:48 rhauch: Pojo cache seems a bit heavy for this purpose.
Apr 25 17:19:12 maeste: yes of course not tie to a particular implementation is always good
Apr 25 17:19:14 rhauch: It is more efficient in a distributed sense, but it requires a lot of AOP weaving.
Apr 25 17:20:23 rhauch: anyway, so there are a couple of different areas.
Apr 25 17:20:57 maeste: yep
Apr 25 17:21:02 rhauch: As far as the JCR API implementation, we’ll want to set up the JCR TCK relatively early.
Apr 25 17:21:37 rhauch: I think we’ll have the mid-level graph API done pretty quickly, so we’ll definitely want your input there.
Apr 25 17:22:07 rhauch: BTW, my thought on that graph API is that the federation engine & cache should basically be a simple graph with versioning.
Apr 25 17:22:43 rhauch: If we do that, then we can put really almost any API on top of that, including UDDI (should we ever want to do that).
Apr 25 17:23:11 rhauch: If you look at the JCR API, there’s a lot of stuff that should be implemented on top of a more simple graph…
Apr 25 17:23:38 rhauch: node types, namespaces, version history, etc.
Apr 25 17:23:56 rhauch: They all need a simple graph that supports versioning.
Apr 25 17:24:12 rhauch: Anyway, that’s the middle of the federation engine.
Apr 25 17:24:16 rhauch: Then there are the connectors.
Apr 25 17:24:59 rhauch: I think we have JIRA feature requests for several different kinds of connectors, with a few initially slotted earlier than the others.
Apr 25 17:25:15 rhauch: I hope I’m making at least some sense.
Apr 25 17:25:21 maeste: ok, I understand