1. Introduction
Since it was
originally released in 1984, ACMS has imposed a limit of 64K on the maximum amount
of data that can be exchanged in a single task call. Not only can individual
workspaces not exceed 64K in size, but the sum of the sizes of all workspaces
for a particular task call cannot exceed 64K. The reasons for this limitation
are manifold, but they essentially stem from the fact that data is passed
around within ACMS by descriptor, and on the 32-bit VAX systems for which ACMS
was originally developed, the length field of a descriptor structure is
specified as an unsigned 16-bit value. When ACMS was originally developed, this
64K limit no doubt seemed very reasonable, but as the scale and complexity of
software solutions has evolved, and as the use of data types such as images and
audio and video media files has become commonplace, the 64K restriction
presents a very significant limiting factor to the evolution of many ACMS-based
applications.
Unfortunately
the design of ACMS is such that eliminating the 64K workspace limit would be
extremely problematical, and would effectively require the creation of a 64-bit
version of the product that could therefore use 64-bit descriptors. While this
in itself is arguably achievable, the flow-on effects are likely to be
considerable. For example, ACMS applications would need to be converted to
64-bit, which may be problematical in many cases, and modifications to other
products such as DECforms and Oracle CDD would be required in order for them to
function and interact correctly with a 64-bit version of the ACMS product.
Given that
ACMS is inherently a TP system, it is generally the case that most transactions
(task calls) within a given ACMS application do not need to exchange more than
64K of data, and there will typically be only a small number of business functions
in any application that are impacted by the 64K limit. However, considerable
time and cost can be incurred devising workarounds to the 64K limit for these
few functions. Typical workarounds might involve the allowing the client
application to interact directly with the underlying database to retrieve more
than 64K of data in a single operation, or using ACMS Systems Interface API Stream
Services to transfer the required data in multiple chunks. These solutions are
not ideal and they are generally very specific to the application in question.
The following
text (originally written June 2012) describes a simple yet powerful and totally generic solution that may be
used to overcome the ACMS 64K workspace restriction in many situations. The
solution provides a set of simple-to-use API calls and utilises a transient
data store to effectively provide ACMS applications with a “pass by reference”
mechanism. Data objects of greater than 64K in size are passed between clients
and the backend ACMS applications via the transient data store, and (unique)
references to the data objects are passed as standard ACMS workspace fields.
The solution has been designed to allow developers to implement their own
transient data store; however the default data store provided with the prototype
solution presented here should be more than adequate for most purposes. Minimal
change to existing application code is required in order to utilise the
solution; other layered products are not impacted; and data transferred via the
pass-by-reference mechanism may be of arbitrary size and content.
2. Basic approach
As described
above, the devised solution essentially provides a pass-by-reference mechanism
that allows opaque data types of arbitrary size (subsequently referred to as
blobs) to be exchanged between client programs and backend ACMS applications.
The following
diagram illustrates how the solution might be used to exchange blobs between
ACMS client and server processes via a user-written ACMS agent program. The key
components of the solution are the transient data store and a small set of API
calls that can be used by agent programs and the ACMS server to interact with
the data store. If the agent program wants to pass a blob (of size greater than
64K) to the backend ACMS application, it issues an API call to insert the blob
into the transient data store, and upon successful completion the API call will
return a UUID that acts as a key to uniquely identify the stored blob. The
agent program then performs an ACMS task call that passes the UUID along with
any other relevant data to the backend ACMS application. The backend ACMS
application may then use the UUID to retrieve the blob, via another simple API
call. A similar sequence of events would occur if the backend ACMS application
was required for a particular task call to return a blob to the agent program.
 |
Figure 1. A basic illustration of how the solution
might be used to exchange blobs between ACMS client and server processes via a
user-written ACMS agent program. Blobs are placed into a transient key-value
data store, and the key (typically a UUID string) is passed between client
front-end and ACMS backend processes as an ACMS workspace field. The key may
then be used by the receiving process to retrieve the associated blob. This
simple “pass-by-reference” mechanism allows arbitrary data items of any size to
be passed in a generic manner between the client and back-end ACMS server
processes.
|
While the
basic approach is simple to understand and simple to use, there are a number of
subtle points that need to be carefully considered with respect to the design
of the transient data store, and indeed it is important to appreciate that
different applications may impose different requirements (typically of a
non-functional nature) on the design of the transient data store, which is why
the solution has been implemented in such a way as to allow application
developers to implement their own transient data store (see Section 6). For example, for some applications it
might be essential that the transient data store can participate in distributed
transactions or that data is replicated or persisted to disk, while other
applications may simply require an efficient scalable in-memory cache with no
particular transactional, guaranteed delivery, or durability requirements.
Most ACMS
applications with desktop clients or web server-based front-ends do not require
support for and will not currently implement end-to-end distributed transactions,
and in most cases there will be no requirement for a transient data store to
persist data across system failures, so long as the consistency of the
underlying application database is appropriately managed by the backend ACMS
application code and clients are properly informed of error conditions. Assuming
this to be the case, Open Source key-value stores such as Redis (http://redis.io/) and MemcacheD (http://memcached.org/) represent ideal solutions for the basis of
a transient data store. While Redis is arguably the more functionally rich
technology, supporting features such as replication and persistence of data (to
some degree), a detailed discussion of the relative merits of Redis and
MemcacheD is beyond the scope of this post, and it suffices to say that either
technology may be used to implement the transient data store within the scope
of the solution described here. Both technologies provide C-based APIs that can
be readily adapted to work with ACMS application code, and both products and
their C APIs have been ported to HP OpenVMS, although it should
be noted that communication with both Redis and MemcacheD is TCP/IP-based[1]
and there is therefore no particular requirement that the transient data store
reside on an OpenVMS system or on the same server as the ACMS application.
Another
advantage of Redis and MemcacheD is that they support message expiration. This
means that it is not necessary to guarantee that entries are always removed
from the cache when they are no longer relevant (for whatever reason), as items
can be set to expire, and stale items will be automatically removed if required
in order to make room for new items. The only consideration is to ensure that
the size of the Redis or MemcacheD cache is sufficient to accommodate the peak
steady-state load. Redis and MemcacheD are also multi-threaded to allow
coordinated simultaneous access by multiple processes, and they can be built on
OpenVMS to use 64-bit pointers, providing access to P2 address space to
facilitate cache sizes greater than ~1GB if required.
For
simplicity of implementation, MemcacheD has been used to develop the prototype
solution described in this document, and for ease of deployment the solution is
entirely OpenVMS-based. The transient data store is simply a port of MemcacheD
to OpenVMS listening on a specified TCP/IP port, and a set of six API functions have been
implemented on top of the MemcacheD C API to provide the necessary interface
between ACMS application components and the transient data store. The API functions
are basically just wrappers around MemcacheD C API calls that can be easily
used by OpenVMS application code written in any language; however application
developers could use the underlying MemcacheD API calls directly if so desired.
The API routines implemented for the solution are described in detail in the
next section; however they basically comprise routines to establish
communication with the cache, “put” and “get” blobs, disconnect from the cache,
free memory associated with items retrieved from the cache, and an auxiliary
routine is provided to translate error codes returned by the cache into
meaningful textual messages. Upon successful completion, the “put” routine
returns a 36-byte UUID string that may be used by a subsequent “get” call to
retrieve the blob data item in question. Any number of blobs can be associated
with a single task call, so long as there is sufficient space in the task
workspace(s) to accommodate the UUID strings.
As
described, the prototype solution is designed to address the situation
illustrated in Figure 1; however
given that APIs for communication with MemcacheD are available for a variety of
programming languages across a range of platforms, it would theoretically be
possible to implement a variant of the solution similar to that illustrated in Figure 2 below,
where Microsoft Windows-based clients send and retrieve blobs directly, and
communicate with ACMS via a standard gateway such as the ACMS Desktop Connector
or the ACMS Web Connector. Hypothetically the solution could also be used in
conjunction with DECforms clients; however this would be a less common use-case
and ACMS SI API Stream Services would arguably be a more appropriate solution
in this context.
 |
Figure 2. The
solution described in this post may be modified for use with Windows-based
clients that communicate with the ACMS backend via standard gateways such as
the ACMS Desktop Connector and the ACMS Web Connector. Under this model, the
client applications would interact directly with the transient data store to
put and retrieve blobs. Such an approach assumes that there is an
implementation of the MemcacheD API available that can be used by the client
application.
|
Before
moving on to discuss the API functions, it should be noted that it would also
be possible to implement transient data stores using a variety of features
provided by the OpenVMS operating system. For example, global sections could be
used to implement an efficient in-memory cache, or RMS could be used to
implement a non-volatile caching solution (or indeed a combination of these
facilities could be used). Technologies such as Redis and MemcacheD provide
considerable flexibility and are well-proven; however for a multitude of
technical and non-technical reasons these technologies may not be appropriate
for all situations, and for this reason the solution described here has been
designed in such a way that developers can implement their own transient data
store, so long as it conforms to the necessary API interface (see Section 6).
3. API calls
This section
describes the API functions that can be used by ACMS agent programs to interact
with the transient data store. The API is deliberately simple, comprising only
six functions, and can be easily used by programs written in most OpenVMS 3GLs,
such as C, COBOL, or FORTRAN. It should be noted that the API is really little
more than an interface to the functionality provided by the shareable image
defined by the logical name ACMS$OBJECT_STORE (see Section 4). It is the functions in this shareable
image that do the real work, with the API acting as a generic interface such
that different transient data store implementations may be interchanged without
requiring any changes to application code[2].
1. ACMS$BLOB_INIT
This function must be called before calling
any of the other API functions to initialise the API and to establish a
connection with the transient data store. The function attempts to load the
shareable image specified by the logical name ACMS$OBJECT_STORE (see Section 4 below) and if successful it then calls the
corresponding API routine in the shareable image.
The prototype for this function is shown
below. The function takes as input a single parameter, which is the address of
an unsigned longword that upon successful completion of the function call will specify
a unique handle that can be used by other API functions to communicate with the
data store.
unsigned long ACMS$BLOB_INIT(unsigned long *handle);
The return value of the function is an
unsigned longword OpenVMS status code that should be checked to determine
whether the call completed successfully or not.
Note that with the exception of the ACMS$BLOB_INIT call, the function ACMS$BLOB_PUTMSG (see below) may be used to display the text
associated with any error codes returned by the API functions; however if ACMS$BLOB_INIT returns an error status, it is likely that handle is not valid, and consequently any calls to
other API routines, including ACMS$BLOB_PUTMSG, will fail.
2. ACMS$BLOB_DONE
This function can be called to disconnect
from the transient data store and to free up any resources that were associated
with the connection. Upon completion of this call (successful or otherwise),
the connection handle will no longer be valid.
The function prototype for this function is
given below. The function takes as input a single unsigned longword value that
contains the value of the connection handle associated with the connection that
you wish to disconnect.
unsigned long ACMS$BLOB_DONE(unsigned long
handle);
The return value of the function is an
unsigned longword OpenVMS status code that should be checked to determine
whether the call completed successfully or not. If an error status (severity error or fatal) was encountered, the function ACMS$BLOB_PUTMSG may be used to display the error text
associated with the returned status code.
3. ACMS$BLOB_PUT
This function is used to insert blobs into
the transient data store. Upon successful completion, the function returns a
36-byte UUID string that can be passed in an ACMS workspace field between ACMS
agent and backend processes and used by ACMS$BLOB_GET to retrieve the bob. The prototype for this
function is shown below.
unsigned long ACMS$BLOB_PUT(
unsigned long handle,
struct dsc$descriptor_s *uuid,
char *data,
unsigned int size,
int flags);
unsigned long handle,
struct dsc$descriptor_s *uuid,
char *data,
unsigned int size,
int flags);
The function arguments are the unsigned
longword connection handle, a string descriptor to hold the returned UUID
string, a pointer to the blob data, an unsigned integer specifying the size of
the blob, and an integer flags parameter. The size of the string descriptor
must be at least 36 bytes. The flags parameter is currently unused and should be
set to 0. The return value of the function is an unsigned longword OpenVMS
status code that should be checked to determine whether the call completed
successfully or not.
4. ACMS$BLOB_GET
This function is used to retrieve items from
the transient data store based on the UUID key. Upon successful completion the function
returns a pointer to the retrieved blob item and the size of the blob. The
prototype for this function is shown below.
unsigned long ACMS$BLOB_GET(
unsigned long handle,
struct dsc$descriptor_s *uuid,
char **data,
unsigned int *size,
int flags);
unsigned long handle,
struct dsc$descriptor_s *uuid,
char **data,
unsigned int *size,
int flags);
The function arguments are the unsigned
longword connection handle, a string descriptor containing the UUID string that
identifies the blob of interest, the address of a pointer that will be used to
store the retrieved blob data, the address of an unsigned integer that will be
used by the function to store the size of the retrieved blob, and an integer
flags parameter. The flags parameter is currently unused and should be
set to 0. The return value of the function is an unsigned longword OpenVMS
status code that should be checked to determine whether the call completed
successfully or not.
It is important to note that this function
dynamically allocates memory to hold the returned blob, and it is the
responsibility of the calling program to free this memory when the blob data is
no longer required. Memory must be freed using ACMS$BLOB_FREE (see below).
When using this function with languages other
than C, it may be necessary to copy retrieved blobs into local storage in order
to more easily perform operations on them, as working with pointers in
languages other than C/C++ is generally problematical. Blobs may be copied into
local storage using functions such as OTS$MOVE3 or DECC$MEMCPY (the C RTL memcpy() function).
5. ACMS$BLOB_FREE
As commented above, the function ACMS$BLOB_GET dynamically allocates memory for blobs that
are retrieved from the transient data store. The function ACMS$BLOB_FREE should be used to free this memory when it
is no longer required. The prototype for this function is shown below. The
function accepts two arguments, specifying the value of the relevant connection
handle and the address of the blob to be freed, respectively.
unsigned long ACMS$BLOB_FREE(unsigned long
handle, void *addr);
The function returns an unsigned longword
OpenVMS status code that can be checked to determine whether the call was
successful or not. An error status will be returned if an invalid argument is
supplied (such as an invalid connection handle, or if a NULL pointer is specified for the address of the
blob).
It should be noted that for the
MemcacheD-based transient data store implementation, memory for blobs is simply
allocated using the standard C RTL malloc() function,
and ACMS$BLOB_FREE is essentially little more than a wrapper around
the free() C RTL function. However, other transient
data store implementations may implement their own memory management routines,
and it is therefore important for applications to use ACMS$BLOB_FREE as opposed to calling free() directly.
6. ACMS$BLOB_PUTMSG
This utility routine can be used to display
the message text associated with status codes returned by other API functions.
The prototype for this function is shown below. The function takes as input two
unsigned longword values, specifying the transient data store connection handle
and the status code, respectively. Output is written to SYS$OUTPUT.
unsigned long ACMS$BLOB_PUTMSG(
unsigned long handle, unsigned long status);
unsigned long handle, unsigned long status);
The return value of the function is an
unsigned longword OpenVMS status code; however it should be noted that the
function always returns a successful status, and there is therefore no point in
checking the return value.
4. Logical names
The
MemcacheD-based transient data store has been implemented in such a way that no
specific configuration settings are required in order to get the data store
running or to use the API. This is achieved by keeping the number of required
run-time parameters to a minimum and by having the server and API assume
sensible default values for all required run-time parameters. For example, by
default the API code assumes that the MemcacheD server process runs on the same
node as the application using the API and that the server uses the default MemcacheD
port number (see below). However, there will be situations where the various default
values will not be appropriate, and logical names can be used to specify
alternative values for these runtime parameters.
It should be
noted that there is also one mandatory logical name, namely ACMS$OBJECT_STORE. This logical name determines the shareable
image that will be used by the API for communication with the transient data
store. As discussed elsewhere in this document, for the MemcacheD-based
transient data store, the value of this logical name must be MEMCACHE$ACMS_SHR.
The
following table lists the logical names that can be defined in order to specify
values for various run-time parameters. It should be noted that these logical
names may be defined at the system, group, or process level. If the logical
names are concurrently defined at multiple levels, the process-level definition
will take precedence over the group-level definition, which in turn will take
precedence over the system-level definition.
|
Logical
name
|
Description
|
|
MEMCACHE$EXPIRES
|
This
logical name can be used to specify an expiry time to be used by the ACMS$BLOB_PUT API call when putting blobs
into MemcacheD-based the transient data store. If this logical name is not
defined or is incorrectly defined then a value of 10 will be used, meaning
that items will expire (and will be logically deleted) 10 seconds after being
added to the data store. The defined value must be a positive integer specifying
the expiry interval in seconds. It should be noted that in the event that the
cache fills up (the maximum size of the cache is specified at start-up), the
oldest items in the cache will be deleted in an LRU manner in order to make
room for new items. Consequently, inadvertently specifying too large a value
for the expiry interval will generally have few (if any) consequences.
|
|
MEMCACHE$HOST
|
As
commented previously, the MemcacheD server need not reside on the same host
as the ACMS server application (indeed, the MemcacheD server need not even
reside on an OpenVMS system). The logical name MEMCACHE$HOST may be used to specify the name or TCP/IP
address of the server on which MemcacheD resides. If no value is defined then
the API will attempt to connect to a MemcacheD instance on the current host.
|
|
MEMCACHE$PORT
|
By default
the MemcacheD server uses port 11211. The logical name MEMCACHE$PORT may be used to instruct the
API to use an alternative port to communicate with MemcacheD. If this logical
name is not defined then the API will attempt to connect to MemcacheD using
the default port number (11211).
|
It should be
emphasised that the logical names described above are specific to the
MemcacheD-based transient data store implementation. Other API and transient
data store implementations may use other logical names to specify and control
various aspects of data store and API operation. In addition to these data
store-specific logical names, the following logical names are used to define
the shareable image that implements that implements the API described in
Section 3, and the shareable image that will be used
by the API for communication with the transient data store. These two logical
names would typically be defined at the system level as part of the start-up
procedure for the transient data store in question.
|
Logical
name
|
Description
|
|
ACMS$OBJECT_STORE
|
As noted
above, this logical name is used to specify the shareable image that will be
used by the API for communication with the transient data store. When the ACMS$BLOB_INIT function is called to
initialize the API, it attempts to dynamically load this shareable image. If
the logical name ACMS$OBJECT_STORE is not defined, ACMS$BOB_INIT will return an error status of
NOLOGNAM.
|
|
ACMS$SUPPLEMENTARY
|
This
logical name defines the shareable image that implements the API functions
described in Section 3. This logical name is
typically defined at the system level as part of the prototype solution
start-up procedure. The associated shareable image is ACMS$SUPPLEMENTARY.EXE (which resides in SYS$SHARE), and applications that use
the API routines described in Section 3 must link with this image.
|
5. Example
This section
includes some simple example code fragments that illustrate how the API
described in Section 3 may be used to pass blobs between ACMS agent
and backend ACMS server programs. The
client (agent) code loads a JPEG image from disk and posts it into the
transient data store using the ACMS$PUT_BLOB function. The UUID returned by the ACMS$PUT_BLOB call is then passed through to the backend
ACMS server application as a standard ACMS workspace argument. The server code
uses this UUID to retrieve the image from the data store using ACMS$GET_BLOB, and writes the image to disk. The ACMS
server application attaches to the transient data store on start-up by calling ACMS$BLOB_INIT in the ACMS server initialization routine
and detaches from the data store by calling ACMS$BLOB_DONE from the server termination routine. Note
that only the main routine for the client code is provided; details of the
functions that interact with ACMS Systems Interface API to sign in and out of
ACMS and issue ACMS task calls are not provided.
Client code
/*
Header file containing magic function definitions */
#include
"acms$blob.h"
#ifdef
vaxc
#dictionary
"wksp"
#else
#pragma
dictionary "wksp"
#endif
int
main()
{
const char * image = "IMG_2490.JPG";
struct wksp wksp;
char * tmp
= NULL;
struct dsc$descriptor_s dsc;
unsigned long ah;
unsigned long ch;
unsigned long status;
FILE * fp;
struct stat sb;
if ((fp = fopen(image, "rb"))
== NULL)
{
perror("fopen()");
exit(0);
}
fstat(fileno(fp), &sb);
fprintf(stderr,
"Total file size = %ld\n", sb.st_size);
if ((tmp = (char *) malloc(sb.st_size *
sizeof(char))) == NULL)
{
perror("malloc()");
exit(0);
}
fprintf(stderr, "%d\n",
fread(tmp, sb.st_size, 1, fp));
fclose(fp);
ADC$ACMS_INIT();
status = ADC$ACMS_SIGNIN(NULL, &ah);
if (! OKAY(status))
{
lib$stop(status);
}
/* Get a handle to the data store */
status = ACMS$BLOB_INIT(&ch);
if (! OKAY(status))
{
ACMS$BLOB_PUTMSG(ch, status);
exit(0);
}
/* Put blob into data store and get UUID
*/
dsc.dsc$w_length = sizeof(wksp.uuid.val);
dsc.dsc$b_dtype = DSC$K_DTYPE_T;
dsc.dsc$b_class = DSC$K_CLASS_S;
dsc.dsc$a_pointer = wksp.uuid.val;
status = ACMS$BLOB_PUT(ch, &dsc,
tmp, sb.st_size, 0);
if (! OKAY(status))
{
ACMS$BLOB_PUTMSG(ch, status);
exit(0);
}
free(tmp);
/* Call the ACMS task */
strcpy(wksp.file, image);
status = ADC$ACMS_CALL(ah,
"HANDLE_BLOBS", "BRCPOC", &wksp, sizeof(struct wksp));
if (! OKAY(status))
{
lib$stop(status);
}
status = ADC$ACMS_SIGNOUT(ah);
if (! OKAY(status))
{
lib$stop(status);
}
return (0);
}
ACMS server code
#include
<stdio.h>
#include
<stdlib.h>
#include
<descrip.h>
#include
<lib$routines.h>
#include
<ssdef.h>
#include
"acms$blob.h"
#ifndef
OKAY
#define
OKAY(STATUS) (((STATUS) & 1) != 0)
#endif
#ifdef
vaxc
#dictionary
"wksp"
#else
#pragma
dictionary "wksp"
#endif
/*
Data store connection handle */
static
unsigned long poc$l_cache_handle = 0;
/*
ACMS application initialization routine - attaches to the data store*/
unsigned
long poc$init()
{
return
(ACMS$BLOB_INIT(&poc$l_cache_handle));
}
/*
ACMS server termination routine - closes connection to the data store */
unsigned
long poc$term()
{
unsigned long status;
if (poc$l_cache_handle != 0)
{
status = ACMS$BLOB_DONE(poc$l_cache_handle);
if (! OKAY(status))
{
ACMS$BLOB_PUTMSG(poc$l_cache_handle, status);
}
return (status);
}
else
{
return (SS$_NORMAL);
}
}
/*
Task cancel procedure (does nothing here) */
unsigned
long poc$cancel()
{
return (SS$_NORMAL);
}
/*
Simple task that is "passed" a blob and saves it to disk */
void
poc$handle_blobs(struct wksp *wksp)
{
unsigned long status;
struct dsc$descriptor_s dsc;
char * tmp;
FILE * fp;
unsigned int len;
/* Extract item from data stire using
supplied UUID */
dsc.dsc$w_length = sizeof(wksp->uuid.val);
dsc.dsc$b_dtype = DSC$K_DTYPE_T;
dsc.dsc$b_class = DSC$K_CLASS_S;
dsc.dsc$a_pointer = wksp->uuid.val;
status =
ACMS$BLOB_GET(poc$l_cache_handle, &dsc, (char **) &tmp, &len, 0);
if (! OKAY(status))
{
ACMS$BLOB_PUTMSG(poc$l_cache_handle, status);
abort();
}
/* Save data to disk */
if ((fp = fopen(wksp->file,
"wb")) == NULL)
{
perror("fopen()");
abort();
}
fwrite(tmp, len, 1, fp);
fclose(fp);
/* Free up memory allocated by the blob
"get" call */
status = ACMS$BLOB_FREE(poc$l_cache_handle,
tmp);
if (! OKAY(status))
{
ACMS$BLOB_PUTMSG(poc$l_cache_handle, status);
abort();
}
}
6. Using an alternative transient data store
As commented
previously, the prototype solution has been implemented in such a way that
alternative transient data stores can easily be used in place of MemcacheD. In
order to use an alternative data store, it is necessary to implement a
shareable image that exports the following list of functions. These functions
have the same prototypes as their namesakes described in Section 3, and must
implement whatever functionality is required in order to communicate with the
data store in question.
- ACMS$$BLOB_INIT
- ACMS$$BLOB_DONE
- ACMS$$BLOB_PUT
- ACMS$$BLOB_GET
- ACMS$$BLOB_FREE
- ACMS$$BLOB_PUTMSG
The logical
name ACMS$OBJECT_STORE
should then be defined to point
to the new shareable image, whereupon the shareable image will be loaded by ACMS$BLOB_INIT (see Section 3) and used
by the API to interact with the desired transient data store.
7. Limitations
Tests have
shown that the transient data store solution described here can provide an
efficient and easy to use method for transferring blobs of arbitrary size
(within given constraints) and content between ACMS client and server applications.
However, it is important to appreciate that the solution does have limitations,
and it may not be suitable for all situations. From a general perspective, it
is important to understand that the transient data store should not be viewed
as some sort of replacement for ACMS and used to transfer all data objects (other
than UUID strings) between client and server. The transient data store is not a
TP system, nor is it middleware, and it should not be used as such; it should
be used only as necessary to transfer large objects between client and server
that cannot be handled as ACMS workspace arguments. Attempting to use the
transient data store pass by reference mechanism to transfer all data objects
could at best be described as “courageous”.
Looking more
specifically at the MemcacheD-based transient data store implementation, there
are potentially a number of limiting factors that it is important to be aware
of. In particular, it is important to be aware that MemcacheD does not provide
any mechanism to authenticate or restrict access, which may be an issue in some
environments, particularly given that the cache typically uses a well-known
network port. Other considerations such as distributed transaction support and
ability to persist data across system failures have been discussed elsewhere,
and the lack of such capabilities is generally unlikely to be a limiting factor
with regard to use of the MemcacheD-based solution or other transient data
store solutions. However, the capabilities and limitations of a particular
transient data store implementation must ultimately be assessed against the
specific requirements of the particular ACMS application.
8. Conclusions
This post describes a simple and generic solution that can be used in a variety of
situations to overcome the ACMS 64K workspace limit and allow ACMS agent
programs to exchange messages of arbitrary type and size with backend ACMS
server applications. A transient data store to move blobs between agent clients
and backend ACMS applications and unique 36-byte UUID-based references to the
blobs are passed between the cooperating processes as standard ACMS workspace
fields. In this way the solution effectively provides ACMS with a simple pass
by reference mechanism. The prototype solution has been implemented using
MemcacheD, a powerful Open Source distributed object caching system; however
the API used by agent and ACMS backend programs to interact with the transient
data store has been implemented in such a way that alternative data stores may
readily be utilised. The solution is totally generic, simple to use, and
requires minimal changes to application code while providing considerable
flexibility in terms of deployment options and “plug-and-play” support for
alternative data stores. Future plans for further development of the solution
include incorporating support for the solution into HP WSIT (Web Services
Integration Toolkit) and into our gSAOP/ACMS integration facility.
[1] It is noted
that MemcacheD also support a UDP-based interface, which while somewhat faster
than the TCP/IP interface, is inherently less reliable.
[2] Arguably
this flexibility could have been implemented directly at the API level; however
if the API functions were eventually incorporated into the ACMS SI API, such an
approach would be inappropriate. Having an additional level of abstraction is
of little or no consequence from either a development or an operational
perspective.
No comments:
Post a Comment