For those charged with "merger and migration marching orders', OpenLink's Virtuoso Universal Server Provides an Ideal Data Integration Workbench and Web Services Infrastructure.
Alan Wilensky, Principal
Encompass Consulting, LLC.
"What goes around comes around. There is a major SOAP[1] opera playing out in the data access market place."
Kingsley Idehen, CEO, OpenLink Software, Inc.
Abstract
The swath of time straddling the late 20th to early
21st centuries might be coined, 'the era of
mergers'. Not only did AOL swallow Time-Warner whole, but
hundreds of regional and neighborhood banks were devoured by
behemoths like Bank of America. It is no longer a question of
whether this is good for the consumer âÄì it's
the default situation.
A dynamic of consolidation is the impact on technology
procurement and human capital utilization. It is a major
undertaking to integrate systems between merged parties. The
Herculean task of wrangling the brittle business processes
interposed between the merged entities is, to say the least, a
vexing challenge âÄì but one not without
opportunities.
The savvy CIO being crushed between the stones of an impending or
ongoing merger might ask, 'How do new technologies, like Web
Services, SOA[2], and ESB[3], make these merger gymnastics easier, more
robust, and maintainable'?
In this monograph, the author will describe a new 'superplatform'
for enabling the merger dynamic with the minimum of tool and
platform splaying, and the ruminations of a great thinker in the
enterprise data integration sector. This new superplatform is
Virtuoso Universal Server, from OpenLink Software, and the
profound thinking comes straight from OpenLink's CEO, Mr.
Kingsley Idehen, a true innovator in the database and enterprise
middleware market for over 20 years.
Cultural issues aside, every merger implies the melding and
reengineering of business processes. Behind these processes lies
a vast sea of custom and commercial
line-ofâÄìbusiness software and supporting
systems. Yet deeper we find a set of common data storage platform
products, generally of the relational database ilk, that no
modern enterprise platform can operate without.
This current state of affairs applies whether we are examining
common ERP and CRM applications, or vertical applications for
primary Line-of Business, such as banking. The core functions of
these systems lie primarily within the underlying database stored
procedures[4], and we can posit therefore that merging
organizations and factoring business processes comes down to
three key issues:
1) The ability to access data stored in the various critical Line-of-Business application databases,
2) The creation of application-to-application communication, decoupled from actual process overlap and dependencies, and
3) Process to Process communications that enable higher level, structured transactions with the current LOB applications within the enterprise, and with external partner systems outside the enterprise âÄì some of which may be long running and demand intelligent failover and routing.
Later in this monograph, we will discuss the actual process of
gaining access to and integrating these data sources, demystify
the composition of inter-application communication using web
services, and finally, create the composite business processes
for creating new web-based applications from all of the
foregoing, all while working with and extending the functionality
of the current LOB applications.
There is tremendous post merger pressure to get systems working together, eliminate redundancies, and insure operational continuity; the first impulse, for even the most disciplined executives, are to let slip the dogs of IT, and watch the madness.
I asked Kingsley Idehen, OpenLink's CEO, if there was a overarching, top-down philosophy, that could steer a merger right, technology aside' Kingsley was a top tier Unisys database integration expert, and single-handedly jump started the performance ODBC product sector âÄì again, chief, the question, what's the guiding principle?
Kingsley answers, 'Each business has a set of processes that enable a number of critical success factors, and these are further decomposed when you drill down into the actual applications that compose the processes'.
So, a process might be the conjunction of a human in a seat, accessing several separate applications in order to complete one or more related processes, in order to accomplish a business task?
'You are correct', says the chief, who is also one of the top SQL industry experts, 'customer account management, inventory, POS - and then we hit the diverse data problem, the number one failure causing artifact in the IT merger game. This is caused by one or more systems, inherited from one side of the family or another, not clearly separating application logic from data storage, or not knowing precisely where the division occurs and can be exploited. But of course OpenLink has an answer in Virtuoso Universal Server'.
How can management drive the integration and insure a return
on investment, and more importantly, a return on effort?
'Good question', exclaims Idehen with a slap on the desk, 'There
are three points that must be tattooed on the foreheads of the
merger team, both executive and IT management, as 'merger
marching orders', to wit:
1) Consciously identify the market leadership discipline âÄì innovation of cutting edge features and technology, operational excellence, or customer intimacy
2) Devise an IT plan, spell it out, as to how your leadership goal will leverage IT in the merger
3) Identify products that align with your chosen IT driven leadership roadmap âÄì eliminate post merger systems that interfere, but most important, keep the underlying data and integrate!
'OpenLink has made this holistic process the critical foundation upon which the Virtuoso Universal Server is established, and it just happens to be the ideal environment in which to accomplish an IT driven merger'
'Think of Virtuoso not as a server infrastructure, or middleware platform, but as an encompassing environment that has all the tools needed to accomplish the complete spectrum of the integration process', says Kingsley, continuing, 'but at the bottom of it all, is an enterprise class object-relational database engine and application server that leverages OpenLink's twelve years in the universal data access market'.
OpenLink has a palette of solutions that satisfy the above lofty yet eminently achievable merger goals; The OpenLink Universal Data Access (UDA) product family is among the best performing and most secure ODBC connectors for the widest array of database management systems âÄì the selfsame systems that house the aforementioned line-of-business data. Although the UDA family only allows a 'one-to-one' relationship between a customized application and LOB data âÄì it is a valid portrayal of the separation of application logic from data storage.
In the Virtuoso product family, many data sources may be attached, and data from all LOB, legacy, and post merger sources, can be driven throughout the integrated environment.
Virtuoso performs with distinction in the realm of all
competent enterprise databases, with the additional muscular
ability to provide these integration services, perform
bi-directional replications between all tables and indexes, and
normalize SQL syntax across dissimilar data management
systems.
Virtuoso has the additional benefit of providing an application
services platform for all of this (now connected and integrated)
post merger data. Data can traverse the Virtuoso server
environment as XML generated from SQL data sources, and these
dynamic XML documents can be made accessible to the world, or
internally, via the ubiquitous URI [5]. Now that is short course in creating Web
Services! Virtuoso may be the only data access and application
server environment that allows even semi-skilled database
administrative personnel to perform such integration and web
services gymnastics in a no-code environment.
However, even after data integration issues are solved, the
high-level merger team must face the challenge of domain and
business unit process integration. On one side of the merger
marriage, there may be 'monolithic' or 'old-line' business
applications, where the processes are either not clearly
documented, or must be retrofitted. Of course, in the case of
recently created enterprise applications, such as the J2EE or the
MS .NET family, the division of application logic and data may be
a great deal clearer. We turn again to the OpenLink's Chief, in
order to get a handle on how to proceed:
'The problem here is that such an effort requires this
re-factoring of application functions. We now have the data where
we want it âÄì we now have to re-design by
isolating functions, and this is the heart of Web Services. We
will take the pre-merger entities, and decompose each LOB
function on both sides into component services. Then, we can
eliminate redundant function, and expose the keepers via SOAP
(Simple Object Access Protocol), and WSDL (Web Services
Description Language), all with security and reliability based on
W3C standards', says Kingsley in pleasant, British
phonemes.
Virtuoso, as it happens, does this very well, as it is a
native run-time environment for not only connected and internal
SQL stored procedures, but for the whole panoply of application
languages, i.e., Java, .NET, C++, Python, and PHP.
One of the reasons that OpenLink can call Virtuoso a 'Universal
Server', with a completely straight face, is that Virtuoso can
run a Microsoft .NET language run-time on a Linux Server;
making this is an encompassing data integration and
application serving workbench worthy of its name.
The merger team rounds third base by creating the composite business services, or more properly, processes that are the end-product of our services decomposition.
Some process must run for a relatively long time' certainly longer than a single user session, as in a loan applications process. This also occurs in retail and manufacturing, where trade exchanges for inventory replenishment and cost optimization is a relatively new phenomenon. Note that we say relatively âÄì older EDI[6] (Electronic Data Interchange), POS, and banking are still carried over a dwindling yet still costly handful of value added networks (VANS).
Kingsley expounds further on this topic, 'The advent of the
XML era and Web Services âÄì the Service
Oriented Architecture so touted, has rendered most EDI and VAN
applications obsolete. Why?' In the old days, while I was at
Unisys, all that trading-partner data transit came from
relational data sources. Where in the world was all of this XML
going to come from' This is why so many small and innovative pure
XML database vendors failed or were bought cheap.'
Continuing, 'Now, with twelve years under our belt, in the UDA business, and Virtuoso entering its 4.0 version, we cover the full-cycle of relational data to XML to Web Services, back to relational or object data. With Virtuoso's Business Process Execution Language engine, we cover macro business processes, reliable transactions, and external trading partner transactions'.
I wanted a bit more from Kingsley, so I asked where these
layered transactional services would be applicable. 'For
instance, OpenLink will be working with Rosettanet.org to enable
XML based trading between companies in electronic component
distribution and semiconductor manufacturing. Similarly, Virtuoso
will work with other XML schemas, such as the ACORD[7] schema commonly employed in the banking and
underwriting industries'.
The merger dynamic implies a gargantuan task spanning
organizational cultures and technology. The choices today are
more than just numerous âÄì they are
confounding. Amplifying the noise within the Web Applications
solution space is an IT media juggernaut that cedes editorial
coverage to major industry players and the
professional[8] analyst trade.
It is refreshing to examine a product that is the brainchild of a
single, bright and focused mind âÄì one of
imagination and bedrock practicality. Such a product is Virtuoso
Universal Server; a complete environment for gathering,
integrating, and deploying new services across the deep reaches
of an organization's entrenched systems, and spanning the gulf
that exists between new partnerships.
Not only is Virtuoso a workbench for the merger dynamic, but it
is truly a Superplatform for re-enabling and invigorating
completely new services and organizational processes when paired
with contemporary line-of-business applications, from whatever
industry.
Virtuoso stacks up well as a unified data access system,
application hosting environment, and business process
orchestration environment. If your team has received 'IT Systems
Merger Marching Orders', it's good to remember not only the
virtues of OpenLink's Virtuoso, but the pedigree of the company,
it's founder, and the incredible team they have assembled.
In the few months that I have been working with Kingsley, I have learned a great deal about application integration and deploying merged services across corporate boundaries âÄì I have also had many myths dispelled.
Q: When you deploy any of these enterprise solutions - it's understood that there is a big three database product sitting underneath before anything will work, i.e., if it's a new install, here comes a SQL Server or Oracle, or DB2 license, comparable in cost to the ERP package - or if you have been running some flavor of database as an anchor application (as many do), then you are in for an upgrade that's going to hurt. Can any of these fat LOB apps be retrofitted to use an Open Source or Alternative Database Engine, such as Virtuoso's core engine?
A: Spot On. LOBs (JDE, SAP, PeopleSoft etc) are
typically built atop Oracle, DB2, and SQL Server. Rarely are they
built atop ODBC or JDBC, and never specifically atop MySQL or
PostgreSQL[9].
Web based applications on the other hand are typically built atop
MySQL (native interface), PostgreSQL (native interface), and SQL
Server (via ODBC under Windows). Note that native interfaces are
called Call Level Interfaces (CLIs) in DBMS parlance. These CLIs
are very ODBC-like (OpenLink has a kit for converting MySQL CLIs
based apps into their ODBC equivalents).
LOB vendors are DBMS specific as a result of development myopia. They typically claim that DBMS independence isn't attainable, hence the DBMS specificity of their products. They aren't deliberately excluding MySQL or any other DBMS, the issue is DBMS specificity at the data access level. SAP also had their own SQL DBMS Engine which they sold to MySQL sometime in 2003.
The data access level is how data enters a DBMS and includes the routines for managing the various data manipulations. A LOB that isn't ODBC or JDBC based cannot prevent data access via ODBC or JDBC for instance. Thus, you can access data created by LOBs using ODBC, JDBC, OLE DB, ADO.NET etc. You can interrogate the schemas of these DBSM engines and unravel the data structures and persistent Stored Procedures using ODBC, JDBC, OLE DB, ADO.NET etc..
SAP or any other LOB can leverage VUS to be genuinely DBMS independent for sure. Most are unaware of the existence of such technology.
Q: Is there a way for common LOB/ERP packages to
natively provide data integration services - in the case of
products preceding Virtuoso Universal Server - common
database middleware - brute force API work, mass batch data
migration? We are discussing enterprises that have large datasets
and historical or recent transaction data; I fail to see how SAP
et al could have ignored that requirement?
A: They have ignored this requirement because they
strategically seek to become "one stop shops". At best, they may
have ODBC and/or JDBC access via OEM or reseller relationships
with middleware vendors. You may be aware that PeopleSoft (now
owned by Oracle) made a $60M acquisition of Cohera from Dr. Mike
Stonebraker. (Founder and CTO of Cohera). They never put this
data integration to any use.
Q: We know that many manufacturing, logistics, and telecom leviathans ran on VAX VMS, IBM VM and System OS390 - AS400 - as well as older ISAM and VSAM libraries, and older Oracle, MS SQL, etc., - are these all historically being managed by industry specific System's Integration specialists for data integration?
A: Horizontal and vertically focused SIs (some of these are VARs that have morphed into SIs).
Q: What about the usual suspects of TIBCO, BEA, Web Methods âÄì it is a crowded market with newcomers such as IONA, Cape Clear.
A: They all need data integration and data access.
Today, they may go to a variety of data access tools vendors, but
they are perhaps missing an important opportunity regarding
heterogeneous data integration and data access of the form
delivered by Virtuoso (or the security sensitive data access of
our UDA product family).
Hopefully, this paper and OpenLink's renewed outreach will better
broadcast that message'.how much am I paying you'
Q, Does a company such as SAP or Seibel need to have
some type of internal data integration suite?
For Example: Say that I own a sheet metal fabrication company,
and that I run a legacy vertical LOB application, and a
combination of AS400 for my inventory system, and a PC server for
invoicing and a plug-in for some accounting system such as Great
Plains. Now, I want to go SAP; where do I start?
A: This is where Virtuoso would come into play; how do you get your data into SAP' This is a Classic data access challenge. SAP may not have the complete repertoire of tools for easy migration. The bigger opportunity lies in the accessibility to SAP via its Netweaver APIs which are SOAP based. BPEL is about constructing composite processes from a myriad of discrete Web Services.
Q: OpenLink claims that application functions can be
hosted and run cross-platform- - can you explain the data access
methods for these LOB applications? Are there accessible
frameworks in which to break out functions for merger driven
integrations?
A: These LOB applications implement their application
logic using SQL Stored Procedures. That's how you write SQL based
LOB applications. In rare cases this may be locked via 4GL
style code, but this is rare (in certain cases 4GL code generates
libraries that we can exploit). However, a majority of the code
in question is implemented as SQL Stored Procedures.
Q: Most custom applications are written in Java, C++, or more recently, C#, and complied as specific binaries, or a CLR, and then linked into executable packages. Are you saying this is not the case for SAP or Peoplesoft?
A: The functional core of these popular LOB applications is SQL Stored procedures. Architecturally, there are many tiers to these applications; presentation, application / business logic, and data storage. The application is typically Stored Procedure based. In the case of C/C++ style libraries, you can integrate these with Virtuoso once one understands the interfaces that are implemented. These bindings are at the API level. We are talking about interfaces and implementations which may take various forms such as Java assemblies, C/C++ modules (functions and objects respectively), accessible via dynamic / shared libraries, and .NET assemblies.
The constant is the data repository. In most cases we can take
the data into the Virtuoso space (in a variety of ways) and
perform the decomposition there.
Q: In a Bank merger, using a costly integrated banking
application, or a big SAP deployment (which I have to assume is
somewhat more open), how do you decompose functions for
deployment via web services?
A: We locate the SQL Stored Procedures, Java Classes, .NET
assemblies, or C/C++ modules hosting the application logic on a
functionality realm basis. In the worst case we use Virtuoso to
create wrapper stored procedures that create equivalent
functionality abstractions pre exposure for Web Services protocol
invocation.
Note, this process may require hand holding in many cases, but
this is a small price to pay in relation to the benefits of the
ultimate solution. Each vendor worth its salt provides APIs
(yes these take the form of DLLs, Shared Libraries, Java Classes,
.NET Assemblies). Beneath these APIs lie Stored Procedures and we
can build connectors into Virtuoso that aren't bound to
J2EE.[10]
Q: Kingsley, I hear you are quite the database historian, and have been around for some of the key innovations; can you give us a peek at your history?
A: I worked on SequeLink before I founded OpenLink; I
assume you see the connection to the company name. In the
beginning there was Pioneer (what is today Data Direct),
TechGnosis (SequeLink), VisionWare, and a consulting firm called
PAL Consulting (now OpenLink Software).
Today, VisionWare is out of the market (part of Tarantella Inc.
nee the real Santa Cruz Operation (SCO as opposed to SCO Group
nee Caldera)), TechGnosis and Pioneer (after becoming Intersolv,
and Merant) are now part of the Progress Software unit called
DataDirect.
When we were PAL consulting, I worked on a product called SequeLink for Progress. In short I was one of the first people to have Progress exposed to data access middleware. What goes around comes around!
Indeed!
[1] This is the author's play on words and nomenclature âÄì Mr. Idehen meant soap opera in the colloquial sense, while the author re-cast the quote as a play on the Simple Object Access Protocol, a foundational Web services technology
[2] Service Oriented Architecture
[3] Enterprise Service Bus âÄì a figment of the industry's imagination
[4] Stored procedures are declarative programs that run in response to a database operation âÄì usually written in SQL or a vendor specific language
[5] Uniform Resource Identifier - A new name for URL
[6] EDI is a mature text encoding and format standard for purchase orders, and wire bound files transmitted over private networks âÄì the sun is rapidly setting on EDI and transitioning the EDI vocabulary to XML data.
[7] Association for Cooperative Operations Research and Development)
[8] Professional = Paid to cover a company or product and never objective
[9] The two top Open Source relational databases
[10] http://expertanswercenter.techtarget.com/eac/knowledgebaseAnswer/0,295199,sid63_gci983379,00.html see here for an SAP example