Tuesday, January 3, 2012

Xtext Connected scoping - CDO and Xtext 2.0

Just last year I ventured into integrating an expression language which can be used to perform operations on parts of an EMF model. The building blocks were decided to be Xtext and CDO, both based on EMF but not interworking out of the box.

There are likely several ways in which Xtext and CDO could integrate. The serialization of an Xtext model into a CDO resource could be thought of. What I needed was different. I wanted to be able to execute expressions on objects stored in a CDO repository. The solution: We would use a regular Xtext resource, but would need to reference other objects stored in CDO. The serialized version of our Xtext resource would need to be stored somewhere as well. For this I decided to simply store it as a String in one of the features of CDO model.

In the diagram here, it shows the idea.

Model A and B are Ecore models which have been adapted to work with CDO. In order for EMF models to benefit from the CDO capabilities, CDO can convert an EMF .genmodel so it can generate CDO optimized java.

The Script.xtext is an Xtext grammar, which implements an expression DSL,  It does arithmetic operations like + - * /.

The Script.xtext grammar includes import statements for model A and B, so the grammar rules reference these external models in the defined grammar expressions. Xtext terminology for these references is a "cross-link".

In Xtext referencing external model objects is a well known pattern which is documented in the great  Xtext Documentation. I won't go into how this is done exactly, but it boils down to telling the workflow where to find the A and B model.

The cross-links pattern, is also explained in the documentation. This is the interesting part. The formal grammar for cross links is:

CrossReference :
  '[' type=ReferencedEClass ('|' terminal=CrossReferenceTerminal)? ']'

In our Script.xtext we use cross-links linking to the external models A and B in similar fashion.
Behind the scene Xtext will resolve the links in the so-called linking process. By default Xtext generates a linker named LazyLinker, which works just fine with CDO based resources. the second step in linking is the definition of the linking semantics. The Xtext linker is lazy, it links the object when needed. The proxy URI which is created and set. An Xtext proxy uri looks like this.

"xtext resource.extension"#xtextLink_::

See the class LazyURIEncoder [src] for details on the coding technique. Next the Xtext Scoping API comes in to play and has been the focus of most of my efforts.

In Xtext scoping is either local or global. The local scope is from the own grammar definition, and global is scoping from external models. We are especially interested in the IGlobalScopeProvider. It is the scope provider's responsibility to return an IScope for a given EReference, in our case a cross-link to either model  A or B.

Xtext ships with two GlobalScopeProvider implementations, one is based on explicitly referencing resources by a grammar naming convention. This is the ImportUriGlobalScopeProvider. The other is the DefaultGlobalScopeProvider, which leverages the Java class path to find resources.

What I needed was a GlobalScopeProvider which was none of these two, but would rather use a fixed set of CDO resources as the base to determine the applicable IScope. The CDO resources holding objects from package A and B, were well known to me and resolvable by a URI, so I looked for a way to use  the CDO resource URI as a base for building the IScope. The set of CDO resource URI's in my case is fixed, so I could simply hardcode the URI's in the IGlobalScopeProvider.

With the highly customizable nature of Xtext, it was rather straighforward to replace/enhance the following components.


Here I simply enhanced the DefaultResourceServiceProvider to override the method canHande(URI uri)
This method needs to support CDO URI's which look like this:

cdo://"repo name"/folder/resource

In my implementation, I look for a URI scheme starting with "cdo".
The CDO  implementation of the ServiceProvider can be installed in the xxxRuntimeModule Guice class.

public Class<? extends IResourceServiceProvider> bindIResourceServiceProvider() {

   return CDOResourceServiceProvider.class;


So this service provider can now handle CDO URI's, next the actual GlobalScopeProvider is needed.


In the CDOGlobalScopeScopeProvider we need to return an IScope based on the requested context.
Our implementation uses the EClass of the referenced type in package A or B, to load the CDO resource.

When initializing the CDOGlobalScopeProvider we build a map of EClass to CDO resource URI(s), which is queried when an IScope is requested. The target EClass is derived from the EReference which is part of the API call for the IGlobalScopeProvider as seen in this method signature.

IScope getScope(Resource context, EReference reference, Predicate<IEObjectDescription> filter);

when the CDO URI is retrieved from the Map, we then invoke (from our customCDO IResourceDescriptions) :

IResourceDescription description = descriptions.getResourceDescription(uri);


As we don't want to load the CDO resource each time an IScope is requested, we use an index for the IResourceDescription produced by the IResourceDescription.Manager, in our IResourceDescriptions implementation.

Xtext comes with a SimpleCache implementation which can be used for that. Additionally we want the index to be updated when the CDO models change, so I borrowed some of the Dawn CDO listeners, boiling dawn to an implementation of the following method:

public void handleViewInvalidationEvent(CDOViewInvalidationEvent event);

from this, I could tell the CDO URI / IResourceDescription index to clean the URI entry for the CDO event dirty objects or any other objects invalidated by the event. The next time an IScope would be requested for such a URI, the cache would simply rebuild the IResourceDescription

Conclusions so far. 

I am aware this blog doesn't provide the details and actual implementation for people to re-use. The reason is that the code is pretty specific for my CDO models, hence my implementation of the CDOGlobalScopeProvider map which tells me in which CDO URI, an EClass resides is really build for my CDO model and not re-usable in a generic way.

private Map<EClass, List<URI>> eClassToURIMap;

Additionally, I have some custom code to open a CDO session and view, to actually retrieve the CDO resource, this won't be re-usable directly in any other app.

I still wanted to share this experience, and simply state CDO and Xtext work great together! I am well inclined to share more details to anyone interested to learn from this experience.

Next steps

It's pretty cool to commit an object in CDO and seconds later see it appear at code completion in an Xtext editor! however.

  1. For larger resources, building the IResourceDescription is time consuming, and it will take some time to build. It becomes important how CDO Resources are stuffed with objects of certain types. In my experience, at some point CDO resources have to be chopped in CDO folders etc... to make them smaller units which can be worked with. 
  2. related to 1), the index should actually be build in application idle time, and ready for use whenever I want to link or get a proposal for a CDO cross-refed object from Xtext.  A solution could be a background job doing this, similar to the Xtext builders which do this in background. 
  3. I find it's needed to sometimes to limit the scope of exposed CDO objects by a certain context. I haven't figured out exactly if this should be done in the ProposalProvider, by the generated DeclarativeScoping or in the CDO global scope provider. I would appreciate some advise on this. 
  4. Perhaps with some help of the Xtext and CDO guys, we could turn this into an off-the-shelf Xtext fragment, I am personally a bit disappointed by the lack of focus of Xtext for runtime RCP apps, although the way it's build up with Guice, makes it super easy to replace or leave out capabilities.