Using BLOBs in ACMS applications - an approach for dealing with the 64K workspace limit

1. Introduction

Since it was originally released in 1984, ACMS has imposed a limit of 64K on the maximum amount of data that can be exchanged in a single task call. Not only can individual workspaces not exceed 64K in size, but the sum of the sizes of all workspaces for a particular task call cannot exceed 64K. The reasons for this limitation are manifold, but they essentially stem from the fact that data is passed around within ACMS by descriptor, and on the 32-bit VAX systems for which ACMS was originally developed, the length field of a descriptor structure is specified as an unsigned 16-bit value. When ACMS was originally developed, this 64K limit no doubt seemed very reasonable, but as the scale and complexity of software solutions has evolved, and as the use of data types such as images and audio and video media files has become commonplace, the 64K restriction presents a very significant limiting factor to the evolution of many ACMS-based applications.

Unfortunately the design of ACMS is such that eliminating the 64K workspace limit would be extremely problematical, and would effectively require the creation of a 64-bit version of the product that could therefore use 64-bit descriptors. While this in itself is arguably achievable, the flow-on effects are likely to be considerable. For example, ACMS applications would need to be converted to 64-bit, which may be problematical in many cases, and modifications to other products such as DECforms and Oracle CDD would be required in order for them to function and interact correctly with a 64-bit version of the ACMS product.

Given that ACMS is inherently a TP system, it is generally the case that most transactions (task calls) within a given ACMS application do not need to exchange more than 64K of data, and there will typically be only a small number of business functions in any application that are impacted by the 64K limit. However, considerable time and cost can be incurred devising workarounds to the 64K limit for these few functions. Typical workarounds might involve the allowing the client application to interact directly with the underlying database to retrieve more than 64K of data in a single operation, or using ACMS Systems Interface API Stream Services to transfer the required data in multiple chunks. These solutions are not ideal and they are generally very specific to the application in question. 

The following text (originally written June 2012) describes a simple yet powerful and totally generic solution that may be used to overcome the ACMS 64K workspace restriction in many situations. The solution provides a set of simple-to-use API calls and utilises a transient data store to effectively provide ACMS applications with a “pass by reference” mechanism. Data objects of greater than 64K in size are passed between clients and the backend ACMS applications via the transient data store, and (unique) references to the data objects are passed as standard ACMS workspace fields. The solution has been designed to allow developers to implement their own transient data store; however the default data store provided with the prototype solution presented here should be more than adequate for most purposes. Minimal change to existing application code is required in order to utilise the solution; other layered products are not impacted; and data transferred via the pass-by-reference mechanism may be of arbitrary size and content.

2. Basic approach

As described above, the devised solution essentially provides a pass-by-reference mechanism that allows opaque data types of arbitrary size (subsequently referred to as blobs) to be exchanged between client programs and backend ACMS applications. 

The following diagram illustrates how the solution might be used to exchange blobs between ACMS client and server processes via a user-written ACMS agent program. The key components of the solution are the transient data store and a small set of API calls that can be used by agent programs and the ACMS server to interact with the data store. If the agent program wants to pass a blob (of size greater than 64K) to the backend ACMS application, it issues an API call to insert the blob into the transient data store, and upon successful completion the API call will return a UUID that acts as a key to uniquely identify the stored blob. The agent program then performs an ACMS task call that passes the UUID along with any other relevant data to the backend ACMS application. The backend ACMS application may then use the UUID to retrieve the blob, via another simple API call. A similar sequence of events would occur if the backend ACMS application was required for a particular task call to return a blob to the agent program. 

Figure 1. A basic illustration of how the solution might be used to exchange blobs between ACMS client and server processes via a user-written ACMS agent program. Blobs are placed into a transient key-value data store, and the key (typically a UUID string) is passed between client front-end and ACMS backend processes as an ACMS workspace field. The key may then be used by the receiving process to retrieve the associated blob. This simple “pass-by-reference” mechanism allows arbitrary data items of any size to be passed in a generic manner between the client and back-end ACMS server processes.

While the basic approach is simple to understand and simple to use, there are a number of subtle points that need to be carefully considered with respect to the design of the transient data store, and indeed it is important to appreciate that different applications may impose different requirements (typically of a non-functional nature) on the design of the transient data store, which is why the solution has been implemented in such a way as to allow application developers to implement their own transient data store (see Section 6). For example, for some applications it might be essential that the transient data store can participate in distributed transactions or that data is replicated or persisted to disk, while other applications may simply require an efficient scalable in-memory cache with no particular transactional, guaranteed delivery, or durability requirements. 

Most ACMS applications with desktop clients or web server-based front-ends do not require support for and will not currently implement end-to-end distributed transactions, and in most cases there will be no requirement for a transient data store to persist data across system failures, so long as the consistency of the underlying application database is appropriately managed by the backend ACMS application code and clients are properly informed of error conditions. Assuming this to be the case, Open Source key-value stores such as Redis (http://redis.io/) and MemcacheD (http://memcached.org/) represent ideal solutions for the basis of a transient data store. While Redis is arguably the more functionally rich technology, supporting features such as replication and persistence of data (to some degree), a detailed discussion of the relative merits of Redis and MemcacheD is beyond the scope of this post, and it suffices to say that either technology may be used to implement the transient data store within the scope of the solution described here. Both technologies provide C-based APIs that can be readily adapted to work with ACMS application code, and both products and their C APIs have been ported to HP OpenVMS, although it should be noted that communication with both Redis and MemcacheD is TCP/IP-based[1] and there is therefore no particular requirement that the transient data store reside on an OpenVMS system or on the same server as the ACMS application. 

Another advantage of Redis and MemcacheD is that they support message expiration. This means that it is not necessary to guarantee that entries are always removed from the cache when they are no longer relevant (for whatever reason), as items can be set to expire, and stale items will be automatically removed if required in order to make room for new items. The only consideration is to ensure that the size of the Redis or MemcacheD cache is sufficient to accommodate the peak steady-state load. Redis and MemcacheD are also multi-threaded to allow coordinated simultaneous access by multiple processes, and they can be built on OpenVMS to use 64-bit pointers, providing access to P2 address space to facilitate cache sizes greater than ~1GB if required. 

For simplicity of implementation, MemcacheD has been used to develop the prototype solution described in this document, and for ease of deployment the solution is entirely OpenVMS-based. The transient data store is simply a port of MemcacheD to OpenVMS listening on a specified TCP/IP port,  and a set of six API functions have been implemented on top of the MemcacheD C API to provide the necessary interface between ACMS application components and the transient data store. The API functions are basically just wrappers around MemcacheD C API calls that can be easily used by OpenVMS application code written in any language; however application developers could use the underlying MemcacheD API calls directly if so desired. The API routines implemented for the solution are described in detail in the next section; however they basically comprise routines to establish communication with the cache, “put” and “get” blobs, disconnect from the cache, free memory associated with items retrieved from the cache, and an auxiliary routine is provided to translate error codes returned by the cache into meaningful textual messages. Upon successful completion, the “put” routine returns a 36-byte UUID string that may be used by a subsequent “get” call to retrieve the blob data item in question. Any number of blobs can be associated with a single task call, so long as there is sufficient space in the task workspace(s) to accommodate the UUID strings.

As described, the prototype solution is designed to address the situation illustrated in Figure 1; however given that APIs for communication with MemcacheD are available for a variety of programming languages across a range of platforms, it would theoretically be possible to implement a variant of the solution similar to that illustrated in Figure 2 below, where Microsoft Windows-based clients send and retrieve blobs directly, and communicate with ACMS via a standard gateway such as the ACMS Desktop Connector or the ACMS Web Connector. Hypothetically the solution could also be used in conjunction with DECforms clients; however this would be a less common use-case and ACMS SI API Stream Services would arguably be a more appropriate solution in this context.

Figure 2. The solution described in this post may be modified for use with Windows-based clients that communicate with the ACMS backend via standard gateways such as the ACMS Desktop Connector and the ACMS Web Connector. Under this model, the client applications would interact directly with the transient data store to put and retrieve blobs. Such an approach assumes that there is an implementation of the MemcacheD API available that can be used by the client application.

Before moving on to discuss the API functions, it should be noted that it would also be possible to implement transient data stores using a variety of features provided by the OpenVMS operating system. For example, global sections could be used to implement an efficient in-memory cache, or RMS could be used to implement a non-volatile caching solution (or indeed a combination of these facilities could be used). Technologies such as Redis and MemcacheD provide considerable flexibility and are well-proven; however for a multitude of technical and non-technical reasons these technologies may not be appropriate for all situations, and for this reason the solution described here has been designed in such a way that developers can implement their own transient data store, so long as it conforms to the necessary API interface (see Section 6).

3. API calls

This section describes the API functions that can be used by ACMS agent programs to interact with the transient data store. The API is deliberately simple, comprising only six functions, and can be easily used by programs written in most OpenVMS 3GLs, such as C, COBOL, or FORTRAN. It should be noted that the API is really little more than an interface to the functionality provided by the shareable image defined by the logical name ACMS$OBJECT_STORE (see Section 4). It is the functions in this shareable image that do the real work, with the API acting as a generic interface such that different transient data store implementations may be interchanged without requiring any changes to application code[2]. 

1.       ACMS$BLOB_INIT

This function must be called before calling any of the other API functions to initialise the API and to establish a connection with the transient data store. The function attempts to load the shareable image specified by the logical name ACMS$OBJECT_STORE (see Section 4 below) and if successful it then calls the corresponding API routine in the shareable image. 

The prototype for this function is shown below. The function takes as input a single parameter, which is the address of an unsigned longword that upon successful completion of the function call will specify a unique handle that can be used by other API functions to communicate with the data store.

unsigned long ACMS$BLOB_INIT(unsigned long *handle);

The return value of the function is an unsigned longword OpenVMS status code that should be checked to determine whether the call completed successfully or not. 

Note that with the exception of the ACMS$BLOB_INIT call, the function ACMS$BLOB_PUTMSG (see below) may be used to display the text associated with any error codes returned by the API functions; however if ACMS$BLOB_INIT returns an error status, it is likely that handle is not valid, and consequently any calls to other API routines, including ACMS$BLOB_PUTMSG, will fail. 

2.       ACMS$BLOB_DONE

This function can be called to disconnect from the transient data store and to free up any resources that were associated with the connection. Upon completion of this call (successful or otherwise), the connection handle will no longer be valid.

The function prototype for this function is given below. The function takes as input a single unsigned longword value that contains the value of the connection handle associated with the connection that you wish to disconnect. 

unsigned long ACMS$BLOB_DONE(unsigned long handle);

The return value of the function is an unsigned longword OpenVMS status code that should be checked to determine whether the call completed successfully or not. If an error status (severity error or fatal) was encountered, the function ACMS$BLOB_PUTMSG may be used to display the error text associated with the returned status code.

3.       ACMS$BLOB_PUT

This function is used to insert blobs into the transient data store. Upon successful completion, the function returns a 36-byte UUID string that can be passed in an ACMS workspace field between ACMS agent and backend processes and used by ACMS$BLOB_GET to retrieve the bob. The prototype for this function is shown below.

unsigned long ACMS$BLOB_PUT(
                  unsigned long handle,
                  struct dsc$descriptor_s *uuid,
                  char *data,
                  unsigned int size,
                  int flags);

The function arguments are the unsigned longword connection handle, a string descriptor to hold the returned UUID string, a pointer to the blob data, an unsigned integer specifying the size of the blob, and an integer flags parameter. The size of the string descriptor must be at least 36 bytes. The flags parameter is currently unused and should be set to 0. The return value of the function is an unsigned longword OpenVMS status code that should be checked to determine whether the call completed successfully or not.

4.       ACMS$BLOB_GET

This function is used to retrieve items from the transient data store based on the UUID key. Upon successful completion the function returns a pointer to the retrieved blob item and the size of the blob. The prototype for this function is shown below.

unsigned long ACMS$BLOB_GET(
                  unsigned long handle,
                  struct dsc$descriptor_s *uuid,
                  char **data,
                  unsigned int *size,
                  int flags);

The function arguments are the unsigned longword connection handle, a string descriptor containing the UUID string that identifies the blob of interest, the address of a pointer that will be used to store the retrieved blob data, the address of an unsigned integer that will be used by the function to store the size of the retrieved blob, and an integer flags parameter. The flags parameter is currently unused and should be set to 0. The return value of the function is an unsigned longword OpenVMS status code that should be checked to determine whether the call completed successfully or not.

It is important to note that this function dynamically allocates memory to hold the returned blob, and it is the responsibility of the calling program to free this memory when the blob data is no longer required. Memory must be freed using ACMS$BLOB_FREE (see below).

When using this function with languages other than C, it may be necessary to copy retrieved blobs into local storage in order to more easily perform operations on them, as working with pointers in languages other than C/C++ is generally problematical. Blobs may be copied into local storage using functions such as OTS$MOVE3 or DECC$MEMCPY (the C RTL memcpy() function).

5.       ACMS$BLOB_FREE

As commented above, the function ACMS$BLOB_GET dynamically allocates memory for blobs that are retrieved from the transient data store. The function ACMS$BLOB_FREE should be used to free this memory when it is no longer required. The prototype for this function is shown below. The function accepts two arguments, specifying the value of the relevant connection handle and the address of the blob to be freed, respectively.

unsigned long ACMS$BLOB_FREE(unsigned long handle, void *addr);

The function returns an unsigned longword OpenVMS status code that can be checked to determine whether the call was successful or not. An error status will be returned if an invalid argument is supplied (such as an invalid connection handle, or if a NULL pointer is specified for the address of the blob). 

It should be noted that for the MemcacheD-based transient data store implementation, memory for blobs is simply allocated using the standard C RTL malloc() function, and ACMS$BLOB_FREE is essentially little more than a wrapper around the free() C RTL function. However, other transient data store implementations may implement their own memory management routines, and it is therefore important for applications to use ACMS$BLOB_FREE as opposed to calling free() directly.

6.       ACMS$BLOB_PUTMSG

This utility routine can be used to display the message text associated with status codes returned by other API functions. The prototype for this function is shown below. The function takes as input two unsigned longword values, specifying the transient data store connection handle and the status code, respectively. Output is written to SYS$OUTPUT.

unsigned long ACMS$BLOB_PUTMSG(
                  unsigned long handle, unsigned long status);

The return value of the function is an unsigned longword OpenVMS status code; however it should be noted that the function always returns a successful status, and there is therefore no point in checking the return value.

4. Logical names

The MemcacheD-based transient data store has been implemented in such a way that no specific configuration settings are required in order to get the data store running or to use the API. This is achieved by keeping the number of required run-time parameters to a minimum and by having the server and API assume sensible default values for all required run-time parameters. For example, by default the API code assumes that the MemcacheD server process runs on the same node as the application using the API and that the server uses the default MemcacheD port number (see below). However, there will be situations where the various default values will not be appropriate, and logical names can be used to specify alternative values for these runtime parameters. 

It should be noted that there is also one mandatory logical name, namely ACMS$OBJECT_STORE. This logical name determines the shareable image that will be used by the API for communication with the transient data store. As discussed elsewhere in this document, for the MemcacheD-based transient data store, the value of this logical name must be MEMCACHE$ACMS_SHR. 

The following table lists the logical names that can be defined in order to specify values for various run-time parameters. It should be noted that these logical names may be defined at the system, group, or process level. If the logical names are concurrently defined at multiple levels, the process-level definition will take precedence over the group-level definition, which in turn will take precedence over the system-level definition. 

Logical name	Description
MEMCACHE$EXPIRES	This logical name can be used to specify an expiry time to be used by the ACMS$BLOB_PUT API call when putting blobs into MemcacheD-based the transient data store. If this logical name is not defined or is incorrectly defined then a value of 10 will be used, meaning that items will expire (and will be logically deleted) 10 seconds after being added to the data store. The defined value must be a positive integer specifying the expiry interval in seconds. It should be noted that in the event that the cache fills up (the maximum size of the cache is specified at start-up), the oldest items in the cache will be deleted in an LRU manner in order to make room for new items. Consequently, inadvertently specifying too large a value for the expiry interval will generally have few (if any) consequences.
MEMCACHE$HOST	As commented previously, the MemcacheD server need not reside on the same host as the ACMS server application (indeed, the MemcacheD server need not even reside on an OpenVMS system). The logical name MEMCACHE$HOST may be used to specify the name or TCP/IP address of the server on which MemcacheD resides. If no value is defined then the API will attempt to connect to a MemcacheD instance on the current host.
MEMCACHE$PORT	By default the MemcacheD server uses port 11211. The logical name MEMCACHE$PORT may be used to instruct the API to use an alternative port to communicate with MemcacheD. If this logical name is not defined then the API will attempt to connect to MemcacheD using the default port number (11211).

It should be emphasised that the logical names described above are specific to the MemcacheD-based transient data store implementation. Other API and transient data store implementations may use other logical names to specify and control various aspects of data store and API operation. In addition to these data store-specific logical names, the following logical names are used to define the shareable image that implements that implements the API described in Section 3, and the shareable image that will be used by the API for communication with the transient data store. These two logical names would typically be defined at the system level as part of the start-up procedure for the transient data store in question.

Logical name	Description
ACMS$OBJECT_STORE	As noted above, this logical name is used to specify the shareable image that will be used by the API for communication with the transient data store. When the ACMS$BLOB_INIT function is called to initialize the API, it attempts to dynamically load this shareable image. If the logical name ACMS$OBJECT_STORE is not defined, ACMS$BOB_INIT will return an error status of NOLOGNAM.
ACMS$SUPPLEMENTARY	This logical name defines the shareable image that implements the API functions described in Section 3. This logical name is typically defined at the system level as part of the prototype solution start-up procedure. The associated shareable image is ACMS$SUPPLEMENTARY.EXE (which resides in SYS$SHARE), and applications that use the API routines described in Section 3 must link with this image.

5. Example

This section includes some simple example code fragments that illustrate how the API described in Section 3 may be used to pass blobs between ACMS agent and backend ACMS server programs.  The client (agent) code loads a JPEG image from disk and posts it into the transient data store using the ACMS$PUT_BLOB function. The UUID returned by the ACMS$PUT_BLOB call is then passed through to the backend ACMS server application as a standard ACMS workspace argument. The server code uses this UUID to retrieve the image from the data store using ACMS$GET_BLOB, and writes the image to disk. The ACMS server application attaches to the transient data store on start-up by calling ACMS$BLOB_INIT in the ACMS server initialization routine and detaches from the data store by calling ACMS$BLOB_DONE from the server termination routine. Note that only the main routine for the client code is provided; details of the functions that interact with ACMS Systems Interface API to sign in and out of ACMS and issue ACMS task calls are not provided.

Client code

/* Header file containing magic function definitions */

#include "acms$blob.h"

#ifdef vaxc

#dictionary "wksp"

#else

#pragma dictionary "wksp"

#endif

int main()

{

       const char *               image = "IMG_2490.JPG"; 

       struct wksp                wksp;

       char *                     tmp = NULL; 

       struct dsc$descriptor_s    dsc;

       unsigned long              ah;

       unsigned long              ch; 

       unsigned long              status;

       FILE *                     fp;

       struct stat                sb;

       if ((fp = fopen(image, "rb")) == NULL)

       {

          perror("fopen()");

          exit(0);

       }

       fstat(fileno(fp), &sb);

       fprintf(stderr, "Total file size = %ld\n", sb.st_size);

       if ((tmp = (char *) malloc(sb.st_size * sizeof(char))) == NULL)

       {

          perror("malloc()");

          exit(0);

       }

  

       fprintf(stderr, "%d\n", fread(tmp, sb.st_size, 1, fp));

       fclose(fp);

       ADC$ACMS_INIT();

       status = ADC$ACMS_SIGNIN(NULL, &ah);

       if (! OKAY(status))

       {

          lib$stop(status);

       }

       /* Get a handle to the data store */ 

       status = ACMS$BLOB_INIT(&ch);

       if (! OKAY(status))

       {

         ACMS$BLOB_PUTMSG(ch, status);

         exit(0);

       }

  

       /* Put blob into data store and get UUID */

       dsc.dsc$w_length  = sizeof(wksp.uuid.val);

       dsc.dsc$b_dtype   = DSC$K_DTYPE_T;

       dsc.dsc$b_class   = DSC$K_CLASS_S;

       dsc.dsc$a_pointer = wksp.uuid.val;

        status = ACMS$BLOB_PUT(ch, &dsc, tmp, sb.st_size, 0);

        if (! OKAY(status))

        {

          ACMS$BLOB_PUTMSG(ch, status);

          exit(0);

        }

       free(tmp);

  

       /* Call the ACMS task */

       strcpy(wksp.file, image);

       status = ADC$ACMS_CALL(ah, "HANDLE_BLOBS", "BRCPOC", &wksp, sizeof(struct wksp));

       if (! OKAY(status))

       {

          lib$stop(status);

       }

       status = ADC$ACMS_SIGNOUT(ah);

       if (! OKAY(status))

       {

          lib$stop(status);

       }

  

       return (0);

}

  

ACMS server code

#include <stdio.h>

#include <stdlib.h>

#include <descrip.h>

#include <lib$routines.h>

#include <ssdef.h>

#include "acms$blob.h"

#ifndef OKAY

#define OKAY(STATUS) (((STATUS) & 1) != 0)

#endif

#ifdef vaxc

#dictionary "wksp"

#else

#pragma dictionary "wksp"

#endif

/* Data store connection handle */

static unsigned long poc$l_cache_handle = 0;

/* ACMS application initialization routine - attaches to the data store*/ 

unsigned long poc$init()

{

       return (ACMS$BLOB_INIT(&poc$l_cache_handle));

}

  

/* ACMS server termination routine - closes connection to the data store */

unsigned long poc$term()

{

       unsigned long                     status;

  

       if (poc$l_cache_handle != 0)

       {

          status = ACMS$BLOB_DONE(poc$l_cache_handle);

          if (! OKAY(status))

          {

             ACMS$BLOB_PUTMSG(poc$l_cache_handle, status);

          }

          return (status);

       }

       else

       {

          return (SS$_NORMAL);

       }

}

  

/* Task cancel procedure (does nothing here) */

unsigned long poc$cancel()

{

       return (SS$_NORMAL);

}

  

/* Simple task that is "passed" a blob and saves it to disk */

void poc$handle_blobs(struct wksp *wksp)

{ 

       unsigned long              status;

       struct dsc$descriptor_s    dsc;

       char *                     tmp; 

       FILE *                     fp;

       unsigned int               len;

       /* Extract item from data stire using supplied UUID */

       dsc.dsc$w_length  = sizeof(wksp->uuid.val);

       dsc.dsc$b_dtype   = DSC$K_DTYPE_T;

       dsc.dsc$b_class   = DSC$K_CLASS_S;

       dsc.dsc$a_pointer = wksp->uuid.val;

       status = ACMS$BLOB_GET(poc$l_cache_handle, &dsc, (char **) &tmp, &len, 0);

       if (! OKAY(status))

       {

         ACMS$BLOB_PUTMSG(poc$l_cache_handle, status);

         abort();

       }

       /* Save data to disk */

       if ((fp = fopen(wksp->file, "wb")) == NULL)

       {

          perror("fopen()"); 

          abort();

       }

       fwrite(tmp, len, 1, fp);

       fclose(fp);

       /* Free up memory allocated by the blob "get" call */ 

       status = ACMS$BLOB_FREE(poc$l_cache_handle, tmp);

       if (! OKAY(status))

       {

         ACMS$BLOB_PUTMSG(poc$l_cache_handle, status); 

         abort();

       } 

}

  

6. Using an alternative transient data store

As commented previously, the prototype solution has been implemented in such a way that alternative transient data stores can easily be used in place of MemcacheD. In order to use an alternative data store, it is necessary to implement a shareable image that exports the following list of functions. These functions have the same prototypes as their namesakes described in Section 3, and must implement whatever functionality is required in order to communicate with the data store in question.

• ACMS$$BLOB_INIT
• ACMS$$BLOB_DONE
• ACMS$$BLOB_PUT
• ACMS$$BLOB_GET
• ACMS$$BLOB_FREE
• ACMS$$BLOB_PUTMSG

The logical name ACMS$OBJECT_STORE should then be defined to point to the new shareable image, whereupon the shareable image will be loaded by ACMS$BLOB_INIT (see Section 3) and used by the API to interact with the desired transient data store.

7. Limitations

Tests have shown that the transient data store solution described here can provide an efficient and easy to use method for transferring blobs of arbitrary size (within given constraints) and content between ACMS client and server applications. However, it is important to appreciate that the solution does have limitations, and it may not be suitable for all situations. From a general perspective, it is important to understand that the transient data store should not be viewed as some sort of replacement for ACMS and used to transfer all data objects (other than UUID strings) between client and server. The transient data store is not a TP system, nor is it middleware, and it should not be used as such; it should be used only as necessary to transfer large objects between client and server that cannot be handled as ACMS workspace arguments. Attempting to use the transient data store pass by reference mechanism to transfer all data objects could at best be described as “courageous”.

Looking more specifically at the MemcacheD-based transient data store implementation, there are potentially a number of limiting factors that it is important to be aware of. In particular, it is important to be aware that MemcacheD does not provide any mechanism to authenticate or restrict access, which may be an issue in some environments, particularly given that the cache typically uses a well-known network port. Other considerations such as distributed transaction support and ability to persist data across system failures have been discussed elsewhere, and the lack of such capabilities is generally unlikely to be a limiting factor with regard to use of the MemcacheD-based solution or other transient data store solutions. However, the capabilities and limitations of a particular transient data store implementation must ultimately be assessed against the specific requirements of the particular ACMS application.

 8. Conclusions

This post describes a simple and generic solution that can be used in a variety of situations to overcome the ACMS 64K workspace limit and allow ACMS agent programs to exchange messages of arbitrary type and size with backend ACMS server applications. A transient data store to move blobs between agent clients and backend ACMS applications and unique 36-byte UUID-based references to the blobs are passed between the cooperating processes as standard ACMS workspace fields. In this way the solution effectively provides ACMS with a simple pass by reference mechanism. The prototype solution has been implemented using MemcacheD, a powerful Open Source distributed object caching system; however the API used by agent and ACMS backend programs to interact with the transient data store has been implemented in such a way that alternative data stores may readily be utilised. The solution is totally generic, simple to use, and requires minimal changes to application code while providing considerable flexibility in terms of deployment options and “plug-and-play” support for alternative data stores. Future plans for further development of the solution include incorporating support for the solution into HP WSIT (Web Services Integration Toolkit) and into our gSAOP/ACMS integration facility.

---

[1] It is noted that MemcacheD also support a UDP-based interface, which while somewhat faster than the TCP/IP interface, is inherently less reliable. 

[2] Arguably this flexibility could have been implemented directly at the API level; however if the API functions were eventually incorporated into the ACMS SI API, such an approach would be inappropriate. Having an additional level of abstraction is of little or no consequence from either a development or an operational perspective.