I had another interesting chat with Randall Hauch about JBossDNA next steps. Here is the log of our discussion. I put it here in agreement with Randall, because IRC chat sometime contain interesting ideas for all comunity memeber of JBossDNA, and it’s good it become public and with a longer time to leave than irc channel logs.

I haven’t time to extract a synthesis document, but in a nutshell we discussed about next steps of the project: federation engine and its connectors, graph SPI, new sequencers, views system.

Hoping it would be useful for someone, here you have it unabridged:

rhauch: I’ll get the graph API into SVN soon.
rhauch: Is there an area you want to work?
rhauch: Other sequencers? Federation?
maeste: Federation would be fun
maeste: and I’m thinking about sequencers useful for ESB/Overlord
rhauch: any experience with JBoss Cache?
maeste: in particulòar I’m looking at ESB code which write on JCR messages
maeste: just a little. Just ine project, not big
maeste: but I have an idea about it, not completely ignorant
maeste: concluding about ESB messages, I think a sequencer on it can be very useful, in particular if used togheter an UDDI sequencer
maeste: but as said Federation and connectors would be big fun for me
rhauch: i’m not sure i understand what would be sequenced
maeste: ESB track messages exchanged
maeste: and would be good to sequence message header
maeste: containing EPRs
maeste: of request and response
rhauch: those headers would be stored in JCR?
maeste: I think so. Am I wrong?
rhauch: oh, well they certainly could be stored in JCR.
rhauch: I just didn’t know how ESB would use JCR for that.
maeste: I think it would be used to query jcr to understand how many times messages pass on EPRs
rhauch: EPR = ?
maeste: and eventually we can sequence the route of message
maeste: EndPoint……I can’t recall the ‘R’
maeste: a moment…
rhauch: router?
maeste: no, it’s the ws-addressing EPR, the unique uidentifier of source/destination of a message
rhauch: ah.
maeste: R=Reference
rhauch: would it be useful to have a connector that publishes the message headers?
maeste: well ESB already have a JCR repository
rhauch: I guess I’m wondering how long those need to be stored? forever would mean a lot of data, right?
maeste: Yep
maeste: it’s the reason because ESB leave to the user the decision on how store on the repository
maeste: there is an action
maeste: that register a message on the repository
maeste: just body and attributes at the moment
maeste: not header
maeste: but also body (and mainly body) may be a lot of data
rhauch: would you rather I create JIRA tasks, and you assign them to yourself?
rhauch: or would you rather discuss what needs to be done and then you create the tasks for what you’d work on?
maeste: well it’s better to have a little discussion before
rhauch: where did you want to start, maeste?
maeste: well we can start from the roadmap
maeste: IOW what you have in mind for next release?
rhauch: Two areas: more sequencers and federation (at least read, probably not write)
maeste: oki
maeste: about sequencers i’d like to write some SOA related
rhauch: okay, that’d be great.
maeste: WSDLs, ESB Message….maybe UDDI
rhauch: johnny verhaeg works with me in STL, and he’ll be looking into a MetaMatrix sequencer (and maybe XSD)
maeste: I don’t think policy one yet, because we (in overlord) have to discuss about what’s kind of policy want to support
rhauch: Someone else on the discussion forums started on a Java sequencer, too. Not sure where he is.
rhauch: (where he is with the sequencer; I think he’s in germany)
rhauch: John is also looking into the view service, but really just some rough prototyping and investigation of some technologies.
rhauch: He won’t be spending too much time on that, tho.
maeste: about java seq…code or class?
rhauch: I believe source, although I’d love to have a bytecode sequencer!
maeste: about view service…another interesting area
rhauch: Both should really generate the same output structure.
maeste: I’d like to give my thought when it will be time
maeste: since I think some of these view could be used on overlord
rhauch: very cool.
rhauch: Oh, absolutely.
rhauch: I anticipate views for all kinds of data
maeste: and also because I’m used to works with web applications
maeste: already some technology in mind?
maeste: I suppose seam
rhauch: yes, seam
maeste: and….GWT? or JavaFX?
rhauch: not sure about view technology: jsf or Flex
maeste: ah…the two I not mentioned…;)
rhauch: well, GWT and JavaFX are options too.
maeste: Flex is indeed good technology too
rhauch: I’m concerned about the ability in GWT to dynamically generate views.
maeste: I have some more doubt about jsf
maeste: and I speak as leader of some big jsf projects
rhauch: We already tried that, and it worked, but it was a little painful.
rhauch: great feedback, then.
rhauch: i’ve never been a big fan of jsf.
maeste: yep right about dynamic generation of GWT
rhauch: i don’t know much about JavaFX
rhauch: Flex has an interesting XML-based view definition.
maeste: I’m studying it right now
rhauch: Hmm… data-driving views.
maeste: it looks very very promising
maeste: Yep I know about this Flex feature…I think something like that could be managed in JAvaFX, with a reflection like approach
rhauch: i need to read up on JavaFX
maeste: anyway it’s another story…we can discuss about views in next future on ML
rhauch: well, I’ll have to introduce you to John, maybe on the dna-dev@ list.
maeste: I need to understand better which kind of dynamic generation you have in mind
maeste: to give my thought
rhauch: well, think about this
rhauch: …
rhauch: in JCR there’s a structure representing a Java class
rhauch: so a node type for the class, with child nodes for the different members.
rhauch: and all these nodes have their own properties.
rhauch: The whole point of the view system is to build a domain-specific view using a subgraph.
rhauch: So the “Java class” view would know it needs to get the properties on the class node AND get the member child nodes and their properties.
rhauch: The resulting “message” (wrong term, really) would contain all the information necessary to build a view using a specific UI technology.
rhauch: The big thing is when navigating the repository
rhauch: and you come across a node that matches the Java class, that the “Java class” view is selected and used to generate the view.
maeste: ok, so I have to see in views a class as a class and not as row xml
maeste: and wsdls as wsdls and so on
maeste: great
rhauch: Right, and not a node with properties and links to child nodes.
rhauch: exactly
rhauch: Ideally, the views could be composed
maeste: of course
rhauch: so that if a node “looks like” a Java class AND a WSDL, then a single view would include both facets.
maeste: well it’s not a joke…doesn’t matter what technology we will choose…it still to don’t be a joke!
rhauch: But in the end, the user should understand what they’re looking at, because it’s expressed in their terms and in ways they understand.
maeste: ok, a grreat idea indeed
rhauch: Oh, and we should have views that show the views.
rhauch:
rhauch: Basically, we want to manage and configure DNA using DNA.
maeste: I can’t immagine an application implemented in jsf like this one which stay under control… jsf tend to don’t scale when you have an abstract approach
rhauch: Now can you see why I like the FLEX approach?
maeste: the easiest and natural way would be XML+XSL or XML driven Flex indeed…but let me investigate a little more about JAvaFX
rhauch: architecturally, it fits very nicely.
rhauch: that’d be great. again, I’ll introduce you to John over email.
maeste: oki
rhauch: so we talked about view system
rhauch: and a bit about sequencers.
maeste: coming back to Federation…
maeste: (BTW view system isn’t expected for next release, right?)
rhauch: no, views is not this next release.
rhauch: again, just investigating an approach so we know how to plan and how it might fit with anything we’re doing now.
maeste: yep of course…and it’s fun
rhauch: we talked a bit about federation a week or so ago
rhauch: and there’s some detail in the Getting Started doc.
maeste: yep I read and I recall our chat
rhauch: so any questions? areas that were vague?
maeste: the general idea is clear
maeste: my question is about
maeste: how do you think to tier this component
maeste: where stay cache
maeste: where the tree api
maeste: and so on
rhauch: ok
maeste: and another one
maeste: is
maeste: what do you think as container for DNA
maeste: JBoss Microcontainer?
rhauch: yes on MC
maeste: A standalone server or a deployable app?
rhauch: hmm…
maeste: And what about configuartion system?
maeste: And management system (MBean?)
rhauch: Ideally (from a user’s view) a standalone server, but that’s more work.
maeste: sure about that?
maeste: A deployable application is easier to understand
maeste: and easier to go live in user’s server farms
rhauch: well, easier for a developer. a bit more difficult from admins perspective.
rhauch: good points.
maeste: Not totally agree
maeste: I know some big farms with a lot of JBoss instance
rhauch: I’m still not convinced.
maeste: and for example some admins
maeste: prefers ESB as deployable
maeste: than standalone server
maeste: because JBossAS is already known by them
rhauch: no doubt. people have different preferences.
rhauch: I came from the MetaMatrix group
maeste: Anyway ideal approach
rhauch: and MetaMatrix is really like a database.
maeste: would be both
rhauch: and deploying a “database-like thing” into an app server really confused people.
maeste: as ESB group did for their product
rhauch: yes, ideal would be both.
maeste: yep u are right on this point
rhauch: in fact, the deployable would really be just a bundled JBoss server, right?
maeste: right
maeste: it’s just matter of packaging and let me say marketing
rhauch: sorry, I mean to say “stand alone server would really be just a deployable and a JBoss server”
maeste: I understood
rhauch: Right, and productization. (with my JBoss hat on)
maeste: LOL
rhauch: I think for the near term, we’ll actually be used inside other apps/projects/products.
maeste: or better with your RED HAT on
rhauch: my red fedora
maeste: lol
rhauch: we all have them, ya know
maeste: oki, so we start with deployable one..
maeste: my other questions:
maeste: config and management system
rhauch: oh yeah.
maeste: (not really coupled only with federation)
rhauch: and monitoring.
maeste: Let me say 0.1 is more an API than a server
maeste: I think next releases have to be PRODUCTIZED
maeste:
maeste: adding these features
rhauch: I think the best approach is to support MBeans, to allow for tools like JBoss ON to monitor and control the “system”
maeste: yes of course
rhauch: (although that’s strictly required for JBoss ON)
rhauch: I think there are two ways we can proceed with config/mgmt/monitoring
rhauch: One is the “traditional” way, with MBeans, Microcontainer config, and such.
rhauch: Another is to use the repository.
maeste: hmmm….I’m not sure to understand
rhauch: Let the repository define the configuration, and be the place we store monitoring results.
rhauch: To change the configuration, change the data in the “/dna:system” branch of the repository, and the system responds.
rhauch: For example, to configure a sequencer, add (or change) the data in “/dna:system/dna:sequencers/MySequencer”
maeste: The question here is….would the user be happy to have to learn a new approach?
rhauch: well, that’s where we’d have to adapt it to what they’re familiar with.
maeste: One more time the ideal approach is to use both and “gateway” the two
rhauch: Yes. Federation, anyone?
rhauch:
maeste: gotcha
rhauch: Seriously. I think that some of the “/dna:system” repository data could actually be federated from microcontainer config or from MBeans.
maeste: Federation read/write traditional config
rhauch: Yup.
maeste: And here we get back to views to admin DNA
rhauch: Exactly!
maeste: Great…I’m really impressed
rhauch: It’s just taking the “data-centric” approach to an extreme, I think.
maeste: And a lot of ideas crowd my mind about what we can do with this approach also for soa governance
rhauch: And, it’s possible that the projects like the Microcontainer could use DNA for managing their configuration.
rhauch: (once we advance enough)
maeste: yep
rhauch: I don’t want to push them this way, but if they see the benefits …
rhauch: like versioning of a configuration
maeste: I see the benefits for soa governance world for sure
maeste: and also for microcontainer
rhauch: Right now, I think we can basically use DI and IOC in our design
rhauch: and not have to pick a path.
maeste: yep
rhauch: I think we will want to use the Microcontainer inside the “DNA Service”
maeste: of course
rhauch: The MC is extensible enough that I think we can integrate the repository below it (sort of via a bootstrapping repository)
rhauch: in their configuration adapters.
maeste: my first question was about tiering of federation….MC is the base and then…?
rhauch: well, i think all the services and components will be managed by the MC.
rhauch: (and wired together)
maeste: (cool idea bootstrapping repository)
rhauch: I hope you could see that flavor in the example application code.
rhauch: I’ve been designing everything so that it can be configured in the MC.
rhauch: okay, so some of the other services/components.
maeste: yep I see
rhauch: One, we’ll have different connectors.
rhauch: the federation engine will delegate to the connectors to load a node,
rhauch: and the engine will merge the results for the same node from different connectors, placing all this in the cache
rhauch: Then, we’ll have a graph API that allows working with the stuff in the cache.
maeste: oki
rhauch: We can have a JCR implementation that uses this generic graph API.
maeste: merging done by JBossCache right?
rhauch: or a UDDI implementation.
rhauch: Merging the node data from different connectors is probably not something JBoss Cache can do, because we really need to keep the different “node fragments” separate
maeste: oki, JCR Implementation and UDDI implementation is better of course
rhauch: so we can manage them effectively.
maeste: oki understood
rhauch: But we’ll use JBoss Cache (should be pluggable, I think) to manage our cache of all the fragments.
maeste: any palns of certify these implementations (JCR and UDDI)?
rhauch: Basically, the graph implementation would work with the different fragments and do the merging.
rhauch: Yes, we’ll need to run the JCR TCK.
rhauch: Don’t know if there is a UDDI TCK. Do you?
maeste: No in fact
maeste: I think there some interoperability meeting
maeste: like ws-* ones
maeste: not totally sure
rhauch: the graph API also is much simpler than JCR
rhauch: basically, it includes versioning.
rhauch: All other JCR functionality (e.g., namespaces, node types, activities, etc.) would all be done by working with graph nodes.
rhauch: JCR 2.0 is closer to this model …
rhauch: moving closer to everything being accessible (and even editable) via JCR nodes.
maeste: oki… 2.0 is much better on these aspects
maeste: What about querying? Metamatrix engine?
rhauch: So I think if our graph API does versioning, then everything else can be done on top.
maeste: I agree
rhauch: (I think versioning will be important for any persistence layer, which is why I think it has to be in the graph API.)
rhauch: I’ve talked generically about the graph API being “above the cache”
rhauch: but really parts if it would also be used by the connectors.
rhauch: things like property values, names, etc. are the same at the connector level too.
rhauch: I have a strawman SPI that I’m about ready to check in.
rhauch: (We chose “SPI” because it fits for use at the bottom, but we also thought the “API”s would be JCR or UDDI or something else.)
rhauch: I was working on it late last night, and came up with something I think is pretty cool.
maeste: right
rhauch: basically, the connectors execute commands, where commands are things like “get node children” or “get node properties” or “create node” …
maeste: I’m anxious to have a look
rhauch: I’ll let you know when it’s in, because I’d love to get your feedback.
maeste: command…because command pattern is used?
rhauch: yes
maeste: so all command are rollbackable?
maeste: IOW transaction compensation?
rhauch: Well, that’s certainly my hope.
rhauch: I haven’t taken it that far yet, but theoretically using commands should make rollbacks (and compensated transactions) easier.
maeste: yep it was what I was thinking
rhauch: I’ve done that in other applications (rollbacks, not compensated txns)
rhauch: oh, regarding the JCR API
rhauch: depending upon how the JCR 2.0 spec shakes out
rhauch: (they’re trying to make it backward compatible)
rhauch: if it is not perfectly compatible, we should be able to have separate JCR 1.0 and JCR 2.0 implementations.
rhauch: I’d hate for us to have to do this, but it’s good to know it’s possible.
rhauch: i hope this all makes sense
maeste: all is very very interesting
maeste: so next steps?
maeste: you are checking the SPI in
rhauch: Well, the first is getting some SPI
rhauch: right.
rhauch: Then, we can proceed in a couple of directions.
maeste: and then you are going to create issues?
maeste: Or can we discuss now directions?
rhauch: 1) implementing the graph API on top of caching and on top of connectors.
rhauch: 2) implement some connectors
rhauch: 3) implement JCR API on top of graph SPI.
rhauch: (or at least part of the JCR API)
rhauch: I’d like to separate the caching from the federating logic.
maeste: I agree 100%
rhauch: We should be able to run without a cache (not good for performance, but caching shouldn’t really affect whether it works or not)
maeste: yes of course (not only for performance, also for cluster)
rhauch: so 1) above will probably break up into several bigger JIRA tasks.
rhauch: yup
maeste: yep
maeste: about 2) what you have in mind?
rhauch: well, probably a connector to another JCR repository. That should be straightforward.
maeste: yep
rhauch: Not sure we need others to get 1,2 & 3 working together.
rhauch: We’ll want others for 0.2 release, like file system and SCM (for ESB & Guvnor)
rhauch: file system being exposing the files as nt:files and nt:folder
maeste: yep
rhauch: That last one might be useful for testing.
maeste: well one also for UDDI would be fine
rhauch: I’d also like 0.2 to have a JBDC metadata connector, for exposing database schemas through the repository.
rhauch: That’s great demo stuff.
maeste: yep u r right
rhauch: (plus it would get us a long way in the MetaMatrix user community)
maeste: (I’m sure you are right…even if I don’t know anything about metamatrix )
rhauch: No problem.
rhauch: A few of the threads on the discussion forums had good use cases for federation,
rhauch: and interestingly they were all read-only.
maeste: yep very intersting
rhauch: So I think we can get buy with no solving the updates until 0.3.
maeste: yep, it simplify a lot of things
rhauch: Sure will.
rhauch: Anything I didn’t cover?
rhauch: Looking back at your questions from earlier …
maeste: I don’t think so. Point 3 is auto explaining
rhauch: Oh, we didn’t talk about caching policy and parameters.
maeste: right
rhauch: The doc covered this a bit, so not sure if you get the idea.
rhauch: I think the approach DNS uses will work really well.
rhauch: each fragment has a cache policy (with TTL, expiry time, etc.)
rhauch: and the federation engine respects those parameters.
rhauch: pretty simple.
rhauch: There was a dna-dev@ topic about that yesterday.
maeste: yep I saw
maeste: it makes sense for me
maeste: I know DNS protocol (not in deeper detail, but I have a general idea)
maeste: BTW I just have got an idea about a demo
rhauch: I think it will also work well if we federate multiple DNA federated repositories.
maeste: we can implement also a connector on apache access_log
rhauch: yeah? What’s the idea?
maeste: and sequence it to extract infos
rhauch: Oh, that’s good!
maeste: and do some statistic about
maeste: I’m sure it would be good
maeste: probably with a little work also for an application concurrent to aw-stat
rhauch: 0.2 should also have the event system, so the connector could publish events as the log is updated.
maeste: yep
maeste: and when we will have views………..
rhauch: ah, and great example of an analysis
maeste: aw-stats concurrent will be here
maeste: yep
rhauch: oh, that reminds me of one other thing
maeste: what about analyzer…03?
rhauch: about the graph API and sequencers/analysers
rhauch: I don’t know if you noticed this, but there is a StreamSequencer interface in the SPI.
rhauch: and a more generic Sequencer interface in the ‘dna-repository’ project.
maeste: yep I noticed
rhauch: Sequencer is much more powerful, but it uses the JCR node in its interface.
maeste: and adaptors to works together
rhauch: Right.
rhauch: So right now, StreamSequencer doesn’t depend on JCR, making it easier to implement and test.
maeste: yep I saw
rhauch: But other sequencers may want to do more advanced things.
rhauch: Okay, so here’s where the graph SPI comes in.
maeste: of course
rhauch: If the Sequencing framework is changed to work against the graph SPI
maeste: are yop thinking to Graèh API dependent sequencer?
rhauch: yes … your faster typist than me.
maeste: lol
rhauch: And analyses, too.
maeste: I made more typo
rhauch: LOL
rhauch: Anyway, they’d work regardless of what connectors are used and what API implementation is used.
maeste: they = StreamSequencer?
rhauch: Plus, since it’s more generic, analyzers and sequencers would work against nodes that represent built-in JCR things (like node types, namespaces, activities, etc.) without doing anything special.
rhauch: all sequencers.
maeste: oki
rhauch: StreamSequencers would still do what they do.
rhauch: but the StreamSequencerAdapter implementation would change.
maeste: yep it is what I have understood reading code
rhauch: At that point, the SPI for sequencers would become more powerful.
rhauch: It’s all about the graph.
maeste: and it is what I have in mind when I wrote you my complimets for very elegant code
rhauch: (ooh, some of my semantic days coming back!)
maeste: lol
rhauch:
rhauch: anything else to touch on, or should we call it for today?
maeste: oki, my last queston is about analysers: when do you plan to implement first?
rhauch: I dunno.
rhauch: I’ll be flexible
rhauch: They’re not too different than sequencers.
maeste: oki
rhauch: In my mind, federation is more important, so if we don’t have the bandwidth to do both, then federation should come first.
rhauch: however, if we have enough people to spread out, then we certainly can tackle them.
maeste: ok, that’s all…I’ll post the transcript of this chat on dna-dev
rhauch: My preference, tho, is to do shorter releases with more focused functionality.
maeste: and I will wait for spi checked in
rhauch: I’ll get that in today.
rhauch: Question:
maeste: and I’ll take a look to jira issues to take some one
maeste: shoot
rhauch: would you rather apply a patch, or do you think it is fine to check in?
maeste: Well I’m not sure to have understood you question about patch…
rhauch: Okay, this commit wouldn’t be invasive and wouldn’t break anything, so it may not matter in this case.
rhauch: But I’ve seen projects post a patch (to a JIRA) that the author wants feedback on
rhauch: people download and apply the patch, give their feedback, and the author changes then commits.
rhauch: Alternatively, I can just commit it and you can look at it that way.
maeste: oki… here comes my question about your svn policy….
maeste: all on trunk?
maeste: we can use branches for that
rhauch: well, that’s a good point.
maeste: I have a great experience of svn in quite large team….
maeste: svn could do a lot for us
rhauch: yeah, the larger the team, the more branches are the way to go.
maeste: hmmm..not totally agree
rhauch: so you’d prefer to do all major development on branches?
maeste: no absolutely not
maeste: not major development
maeste: but a specific development which need feedback yes
rhauch: ah, gotcha
maeste: well…may I post my ideas about svn on dna-dev
rhauch: That’d be a great discussion topic.
maeste: I can resume how I use svn in a lot of projects with plus and minus
rhauch: That’d be great.
maeste: I think with a good use of svn solve a lot of problems
rhauch: okay, so action items:
maeste: 2 post by me (svn and this chat)
rhauch: 1) I’ll check on commit privileges
maeste: yep
rhauch: 2) I’ll introduce you to John.
maeste: good
rhauch: 3) commit SPI (on trunk for now, or other?)
maeste: well if it isn’t invasive it’s good on trunk
maeste: or if you prefer create a branch with a good name
rhauch: 4) I’ll start putting in some JIRA tasks.
maeste: yep, and I’ll look at them
rhauch: I may do that, simply so that we can maybe try something else.
rhauch: something else if the SPI gets bad reviews.
maeste: