Coach repeating table pagination toolkit

My first contribution to IBM BPM Wiki

A toolkit for paginating repeating tables in coaches. Not the only one available but the only one that does it just the way I like it.

Using Teamworks code to write JavaScript

Here’s a little trick I learned recently and might be of use to someone else out there.

The basic requirement is to dynamically change the options in a combo box, based on data held in teamworks variables. More specifically, I want the options in ComboBox0 to change depending on what I select in ComboBox1. And I don’t want to wait for an AJAX round trip every time I change the selection.

So, suppose we have two local variables, both of which are lists of NameValuePair, and suppose they are initialised with data.

Here’s a pic of the variables:

And here’s the code that initialises them (lazy, I know, but it’ll do):

tw.local.itemList1 = new tw.object.listOf.NameValuePair();
tw.local.itemList1.insertIntoList(tw.local.itemList1.listLength, new tw.object.NameValuePair());
tw.local.itemList1.insertIntoList(tw.local.itemList1.listLength, new tw.object.NameValuePair());
tw.local.itemList1[0].name = "name1";
tw.local.itemList1[0].value = "value1";
tw.local.itemList1[1].name = "name2";
tw.local.itemList1[1].value = "value2";
tw.local.itemList2 = new tw.object.listOf.NameValuePair();
tw.local.itemList2.insertIntoList(tw.local.itemList2.listLength, new tw.object.NameValuePair());
tw.local.itemList2.insertIntoList(tw.local.itemList2.listLength, new tw.object.NameValuePair());
tw.local.itemList2[0].name = "name3";
tw.local.itemList2[0].value = "value3";
tw.local.itemList2[1].name = "name4";
tw.local.itemList2[1].value = "value4";

Then I create a coach with two Combo Box controls.

One control (ComboBox1) with manual data:

The other control (ComboBox0) bound to itemList1. I’m only binding it so it’s initially populated.

An finally, the interesting part, a block of custom html with the code to do the business:

<script type="text/javascript" src=<#=tw.system.model.findManagedFileByPath("jquery-1.4.2.min.js" , TWManagedFile.Types.Web).url #> > </script>
 var optionList1 = [];
 var optionList2 = [];
 for (var i = 0; i < tw.local.itemList1.listLength; i++){
 var optionItem = [];
 optionItem.push("{\"text\":\"" + tw.local.itemList1[i].name + "\"", "\"value\":\"" + tw.local.itemList1[i].value + "\"}") ;
 for (var i = 0; i < tw.local.itemList2.listLength; i++){
 var optionItem = [];
 optionItem.push("{\"text\":\"" + tw.local.itemList2[i].name + "\"", "\"value\":\"" + tw.local.itemList2[i].value + "\"}") ;
$('#ComboBox1').bind('change', function(e){
 var items = [];
 if($('#ComboBox1 option:selected').val() == "one"){
 items = [<#=optionList1#>];
 else {
 items = [<#=optionList2#>];
 $.each(items, function(index, opt){

So, what exactly are we doing here?

First I’m importing JQuery, because I’m lazy and I find it helpful. But don’t get caught up with the JQuery stuff. It’s not the point of this post.

The code in question is divided into two parts:

First javascript that runs on the server, this is the chunk enclosed by pointy hashes (<# … #>).

And second, javascript that runs on the browser (the rest).

This is the conceptual leap I had to make, in Teamworks, not all javascript is equal, there’s the TW stuff and the plain old JS stuff. One executes on the server, before the coach is served, the other one is JS as we know it, for the browser.

With this in mind we can do all sorts of funky stuff.

And the best way to grok what’s happening here is to look at the source as served to the browser. So, I’m running the coach, I’m asking IE to show me the page source, and here’s the relevant bit of code:

<DIV class="customHTML" id="CustomHTML0"><script type="text/javascript" src= > </script>

$('#ComboBox1').bind('change', function(e){
 var items = [];
 if($('#ComboBox1 option:selected').val() == "one"){
 items = [{"text":"name1","value":"value1"},{"text":"name2","value":"value2"}];
 else {
 items = [{"text":"name3","value":"value3"},{"text":"name4","value":"value4"}];
 $.each(items, function(index, opt){

The key thing to notice is how the items assignments work. The browser shows you how, within the conditional statement, the ‘items’ array is populated with JSON objects, but these JSON objects are actually created on the server, based on tw variables!

So basically what we are doing within the pointy hashes is generating the javascript that ultimately runs on the browser.

And once you crack that concept, well,  that’s where the fun starts, really :-)


Custom Coach Overrides and Single-Sign-On

At my client, we have defined and built a business process using IBM BPM 7.5.1, which includes coaches, and we display these coaches within our existing IBM Portal based solution.

There were a few challenges along the way in order to integrate BPM and Portal, including configuration of Single Sign On (SSO), and none of which actually caused real pain, apart from one little, almost trivial issue, which took us quite some time and effort to sort out.

We want the BPM coaches to look and feel just like all other Portal content, so we lifted the relevant CSS classes from the Portal solution and use them to override the default coach CSS. Of course that was fun to do, there was a lot of Firebug involved and eventually things were looking rather good, to the point where our coaches are now indistinguishable from any other portal content in the system.

But once we successfully completed the SSO configuration we lost the custom look and feel, which was puzzling, because we could see the customised coaches when testing the BPD from Process Designer, we could see them when we were running them on the playback server, but they reverted to the default coach appearance when going through Portal.

So eventually I had to roll up my sleeves, pump some Bassdrive into my skull and really dig in.

I fired up Fiddler, started monitoring HTTP traffic from my browser and discovered that my requests for the custom_coach.css file resulted in a 302 redirecting to the teamworks login page.

Interesting, SSO worked fine for everything but the CSS file. Teamworks did not see that particular request as coming from a trusted source.

And then I saw it. All other requests were to, but the CSS one was to http://bpmhost:appserverport/teamworks/csspath

When Teamworks serves the Coach, it sends to the browser links to a bunch of things, including the CSS file in question, and this link was being composed using the machine hostname, without the fully qualified domain, and with its app server port, rather than the web server port.

There are two problems with this. One is the port as it was forcing the browser to talk directly to a particular app server in the cluster, when we really want it to go through IHS, though this wasn’t breaking things. The other one is the non qualified hostname. This did break stuff.

The way SSO seems to work, when the browser receives the LTPA cookie, it is allowed to propagate it to servers in the same domain. So, after logging in to it was successfully requesting coach content to but failing to retrieve the CSS file from bpmhost.

It turns out that there is a configuration file in the system folder for the bpm installation called 99Local.xml that controls such things. It defines URL prefixes for quite a number of moving parts, including Teamworks web resources, so a global replace of all hostnames to include the domain, and port to point at the web server, followed by a server restart did the trick.



BPM 7.5.1 – Waging war against cyclic dependencies

I’m wearing the hat of toolkit producer these days and I’m responsible for delivering a number of outbound JDBC Advanced Integration Services.

Another team member is working on another toolkit, which also implements JDBC AISs.

Both toolkits need the JDBC connector module to compile, and both toolkits will bundle the JDBC RAR (embedded adapter).

This arrangement creates two interesting problems. One at ID workspace compile time, the other at runtime deployment time.

When I create the outbound JDBC import on my toolkit implementation module, the JDBC connector project with the adapter RAR is created, and in order to compile the toolkit, this project has to be associated with it or its dependencies.

So I associated the connector with my toolkit. Splendid.

Now, when I bring into my workspace the other team member’s toolkit, which also has the connector associated with it, things get hairy. Process Center tells me the connector already exists in my workspace, and I say “fine, makes sense, don’t bother loading it to my workspace a second time then” and I deselected it from the projects to bring in.

But then, what happens is, the connector is associated with only one of the toolkits, and my ID workspace can’t build. Joy.

To make matters worse, the runtime starts moaning about duplicate contributions, along the lines of “CWYBC_JDBC is already installed”.

Now, we can install the rar standalone on the server and that would resolve the deployment issue, but we still need a way to build the toolkits.

So what we did is factored out the connector project into a third toolkit and made both my toolkit and the other team member’s depend on it.

This approach worked fine, the connector module is associated with a single toolkit and that solves the build problems, and, somewhat counter intuitively, even though both our toolkit implementation modules dependencies ask for the connector project to be embedded, the deployment works OK as well.

Business Objects and Lombardi Coaches

We’re working with IBM BPM 7.5 these days, or rather making it work, and a little idiosyncrasy of the product had us head-scratching for a few hours.

You may or may not know that the Lombardi part of the product uses variables to represent business process data. You would usually define variables local to the business process definition (BPD) and use data mappings to select what data will be used as inputs and outputs to the various activities within the process.

You can define your own data types (business objects) for these variables. So as way of example we defined an Individual, with string attributes for first and last names. Nothing fancy.

Then added a private variable to the BPD named individual of type Individual. Note how we did not specify a default value for the variable.

Suppose you now create a very simple BPD to use a UI Coach to populate the individual name and a simple javascript service to write the names to the log:

The log activity has a simple system script service to write the individual’s attributes to the system log:

When you run this, the first activity will let you populate the individual variable values:

What happens next is not that predictable. One would assume that the UI form took the user supplied values for fname and lname and used them to initialise and populate the individual.

Not quite.

Once the Coach task completed the process moved on to the log activity and failed. Crashed and burned.

The BPD instance has failed and the error is in the Log activity:

The details of the error are self explanatory, the individual variable is undefined:

It turns out that complex type variables need to be explicitly initialised, particularly when Coaches are involved!

So, the IBM documented approach is to use a script activity to intitialise them:

But we also found out in our tests that giiving the variable default values also solves the problem, even if the values are empty strings:

So, hope this tip comes useful to someone else and you can look fwd to more posts on BPM 7.5

I’m sure we’re not going to be short of material for quite some time…


WebSphere JDBC adapter and hierarchical objects with Oracle

At my client’s we have used the JDBC adapter to integrate with Oracle tables extensively in the past. In fact, the solution in place is pretty much hinging on Oracle tables for marketing campaigns and product configuration details.

So far we have always been conservative in our approach, creating integration flows one table at a time or working against Oracle views. Trying to not ask too much from the adapter.

New business requirements for a future release demand a bit more from our JDBC connectivity. We have to retrieve, update and create from multiple related tables, which is mostly fine, but we ran into a little rough patch doing the inserts.

It is worth mentioning than none of this would have been an issue with DB2, which supports Identity columns. The JDBC adapter understands these Identity fields and generates the Business Objects accordingly. But with Oracle things are a little different.

Oracle implements auto generated primary keys with a combination of Sequences and Triggers. The sequence supplies the ID value and the trigger fires before the insert populating the primary key with it.

Let’s take a simple example of two related tables, AUTHOR and BOOK.
AUTHOR.ID will be the primary key and BOOK.AUTHID will be the foreign key relating book rows to their author. I’m using Oracle XE 10g and created the tables using the web ui.

You can start creating a new WID integration project and drop an outbound JDBC  adapter in the assembly. Go through the wizard as usual but make sure you click the ‘Edit Query…’ button and check the ‘Prompt for additional configuration settings when adding business objects’ check box, as shown in the image below:

After you run the query you can start adding tables. Add the AUTHOR table first accepting all defaults.

Next add the BOOK table and build the child/parent relationship as in the following image:

Complete the wizard (I don’t generate business graphs, clear the checkbox)

Now is when the manual changes happen. Open the generated BO for the AUTHOR table, I’m using the system squema and the BO is called SystemAuthor.

You have to manually add the UID annotation to the ‘id’ attribute and supply the sequence name as shown below. This is done by a right-click on the metadata element.

You can repeat the process for the BOOK object for good measure, though for this case it isn’t strictly required.

Next we have to modify the Oracle trigger:

  before insert on "AUTHOR"
  for each row
    select "AUTHOR_SEQ".nextval into :NEW.ID from dual;

The manual change highlighted in red is required because the JDBC adapter, once is told that the ID field is generated, will query the sequence and populate the attribute with the retrieved value. It also synchronises any foreign keys on child objects. But just before the insert operation, without this manual change, the trigger will fire, replace the value of the auto-gen field with the next value in the sequence and cause the parent/child relationship to be out of sync.

You can also modify the trigger for BOOK inserts. We don’t have child objects of BOOK in this example to worry about but it makes sense to do it for completeness and to be ready for them.

You can now deploy and test. One thing to watch out for is the fact that the integration test client will populate the foreign key on the child object with a default value unless you manually unset it. So make sure you explicitly set any authid book attribute to unset as below:

The response should look like this:

You can also check your tables and verify that the rows have been inserted as expected.

Best regards


SQL Exception: Unable to insert null into process_template

This is an old foe that has reared its ugly head again at my client’s.

One of the coding standards we have in place states that in a BPEL process, invoke activities must catch their corresponding interface faults, and if necessary throw a process specific fault to be caught at the top level of the BPEL.

A developer was very diligent following this standard, which is good news, but he forgot to populate the namespace of the fault he was throwing.

Version 7 does this for you, which is neat, but in v6 if you don’t specify the namespace as in the image below, it will be left empty.

If you leave the fault namespace blank, the process might not behave as expected, but it will deploy and start on your integrated test environment.

However, when deployed to a real environment using Oracle as the persistence mechanism this process will not start.

Enabling detailed trace on the server (*=all) revealed a rather terse ORACLE SQL EXCEPTION: UNABLE TO INSERT NULL into PROCESS TEMPLATE(null,tempalteId,).

Only ‘thanks’ to past experience and recollection I was able to diagnose and correct the problem.

SCA Recursion

A service I’m designing for a client will retrieve mortgage details to be displayed by the web tier.

It is common for mortgages to consist of a main account which itself is composed of a number of sub-accounts, and it is not inconceivable for the sub-accounts themselves to have further sub-accounts.

To retrieve the entire tree my service has to call the mortgage account back-end multiple times while drilling down into accounts for sub-accounts to retrieve.

All those main and sub accounts are the same data type, so recursion seemed a good fit: I receive a request with a mortgage account Id, call the back end service who responds with account details, including a list of sub-account Ids, and for each sub-account I call myself, appending responses to the list of mortgage details in the final response.

The end result, is a response structure with nested mortgage details.

Let’s start by taking a look at the assembly diagram:

The MortgageAccount untyped component represents the back-end. I will be manually emulating this, so an untyped component will suffice.

RetrieveMortgageDetails is a BPEL microflow which references the back-end mock and also references itself.

I’ve implemented two interfaces, one for the microflow and another one for the mock back-end.

Both interfaces have a message input with a mortgage account Id but the response objects are slightly different.

The microflow returns a MortgageDetails object, which has as an attribute a list of MortgageDetails, as follows:

The mock back-end returns a MortgageAccount object, which has as an attribute a list of sub-account Ids:

The microflow is implemented as follows:

The first invoke calls the back-end, then for each sub-account Id in the response I call myself, appending the response to the subAccount attribute in my response to the calling client. Of course the ‘calling client’ will be myself when traversing the nested sub-accounts, and the real client once all sub-accounts are retrieved.

Here’s the result of a test run:

You can see how the response has a main mortgage account (id 1) which has two sub-accounts (1.1 and 1.2) You can also see that sub-account 1.1 has two nested sub-accounts (1.1.1 and 1.1.2)

It’s Store-and-Forward, Jim, but not as we know it.

You might have heard of v7 formal support for store-and-fwd and wondering what this post is about. Well, it’s not about v7’s support for it.

The product feature relates to its ability to recognise when a target system is unavailable, work upstream until it finds the first asynchronous interaction point, and start storing requests until human intervention via a dedicated Business Space widget to reopen the information flow.

The key points here are 1) only asynchronous interactions are supported 2) human intervention is necessary to restore the flow.

I’m working against a different scenario here. What I want to do is to automatically switch from synchronous to asynchronous processing of selected interactions when a target system is not available. I want synchronous consumers of my service to always get a synchronous response, either complete or partial, depending on whether my respective target systems are reachable or not.

I also want a retry mechanism that will continue accepting requests and process them all ‘offline’ once the failed external system is restored.

And I want the business process to pick up where it left off and carry on with its activity sequence.

So, imagine you apply for a credit card online, and there are 3 key steps to process your application. First we score your risk, then we validate your application and last we fulfill your order.

You can write a short running BPEL process to orchestrate those three services and give the web front end a synchronous response.

Now, suppose risk scoring is a third party service that’s notorious for being down for housekeeping a few hours every day.

Clearly we can’t fulfill your application without having scored your risk, but neither we want to just tell you to come back later, much later, and that we are sorry but you just wasted your time filling up a form.

What we want is to tell you that your application has been received, it is being processed, and you can look forward to your new credit card arriving in the post real soon (or if you haven’t qualified, a communication to that effect).

So, lets look at a simple prototype of the short running process, without any store and forward capabilities.

Not a lot going on here. We receive a credit card application request, we prepare requests to a number of external systems, we invoke them in sequence and we reply to the client.

The external systems are implemented as mediation modules and stubbed with Java SCA components. I log the message and create a response from these components.

For the scoring service I did a bit more work. I configured a jndi string binding that I can manipulate through the admin console and depending on its value I throw a modeled fault. This is so I can emulate the system being unavailable.

I assume you can complete these tasks without assistance.

You can then run some basic tests and confirm that all your modules are hanging together and everything behaves as it should.

So now we can start thinking about how to approach the case when the scoring service is offline.

The first thing you’re going to need is a new module with a long running process implementing a new ScoringService interface with a one way operation taking the same input parameter type as the actual scoring service mediation.

You can think about this asynchronous LRP as a ‘wrapper’ to the synchronous scoring service.

So, this LRP is called asynchronously (there is no reply) and is instantiated once and only once. You will have to work on your correlation properties/set, so requests are routed to the running instance.

On initial request, an instance is created, the request is placed in a list and an attempt is made to call the scoring service. This call is likely to fail (we wouldn’t be here at all otherwise), so the fault handler executes, which puts the process in receive mode. Every additional request is appended to the list and every time we end up putting the process in receive mode again.

I we haven’t received a request for some time, we timeout the fault handler so we can probe the Scoring Service.

At some point the Scoring Service will be up and running again and for each pending request we will invoke it, get its response, remove the current pending request from the list and invoke the credit card application short running process, letting it know the scoring activity is now complete (we pass in the score result).

Note that ‘resuming’ the credit card application process does not technically resume anything. It simply creates a new instance but with the scoring data already present.

Next you have to modify the short running process so it can detect that the Scoring Service is down, call the async ‘wrapper’ and reply a partial response to the client.

When this short running process is called from the UI and the Scoring Service is up, it behaves exactly as before, and the UI receives a complete response.

When the Scoring Service is down, the fault handler runs, the long running process is called, and the UI receives a partial response.

When this short running process is called by the long running one, the scoring invoke is not attempted, the process proceeds with validating and fulfilling the credit card application, and the reply goes back to the long running process, which you can use for generating customer communications.

This approach keeps the business logic in a single place (the short running process), and effectively deals with offline treatment of requests when a given system is down.

It also addresses resource management, by creating a single long running process, rather than one for each pending request.

And because a long running process state is persisted, all those pending requests survive a server restart, so nothing is ever lost.

ttfn – gabz

WID Heap Status

This is a back to basics tip, but I’m surprised by the number of questions about this from experienced developers.
I like to keep an eye of how much java heap WID is using, particularly while building big workspaces.

So I check ‘Show heap status’ in General Preferences:

Then you’ll see it at the bottom right corner:

WID’s default max heap is 512M. I normally change it to 1024 by editing eclipse.ini and changing -Xms512m to -Xms1024m

Short but sweet