The Open Source Edition doesn't include the Virtual Database Engine and Data Replication Functionality of the Commercial Edition.
Virtuoso Universal Server supports the key SOA concepts including creating composite applications from loosely coupled data, construction of web services from heterogeneous datasources, as well as integration, management and synchronization of business processes.
Virtuoso is a Universal Server, but can be considered an ESB or (Enterprise Service Bus). Virtuoso through support of a number of web service protocols including WS-RM can act as a shared messaging layer for connecting application logic and other web services or business processes.
Virtual database in the simplest form provides the ability to search across several databases with a single query. Virtuoso extends this query functionality and data transformation through hosted logic, SQL, SQL stored procedures and XML, and the Virtuoso PL.
Virtuoso supports a number of databases platforms and can simultaneously connect your ODBC, JDBC, UDBC, OLE-DB client applications and services to data. These databases include Oracle, Microsoft SQL Server, DB/2, Informix, Progress, Ingres and as sell as other ODBC compliant database engines.
Under Windows, Virtuoso can act as an MS DTC (Distributed Transaction Coordinator) resource manager and client. This means that for a client aware of distributed transactions, Virtuoso can be enlisted as a participant in a 2PC transaction via ODBC, OLE/DB and .net client interfaces. If Virtuoso acts as a participant in a distributed transaction and involves data from remote databases, it will propagate the enlistment if the remote database in vfaq_question supports this. A virtuoso stored procedure can also initiate a distributed transaction into which it then enlists multiple other resource managers as the need may arise. The enlistment is transparently handled by the VDB logic once the Virtuoso transaction has been declared as distributed.
Any transactions involving updates to exactly one remote database plus Virtuoso can be handled as single phase commit without danger of corruption.
Under non-Windows platforms, Virtuoso can act as an XA compatible resource manager. In these environments Virtuoso cannot at present enlist remote databases but can be enlisted itself. The most convenient client API for this is JDBC with its distributed transaction support.
At the time of defining a remote data source, Virtuoso makes queries concerning its capabilities. It will use these results in generating SQL statements to pass to this data source. For example, if a remote database does not support a specific SQL function or construct, e.g. outer join, it will not present it with SQL statements containing these and will perform the unsupported operation locally instead. This is entirely transparent to the Virtuoso client.
Select a data source from the available ODBC data sources on your system, view the tables, views and stored procedures available on each, select which ones you want to access via Virtuoso and you are set. From now on, you may manipulate these remote objects from Virtuoso SQL and applications. A web based UI takes you through this process in a few clicks.
The Security Manager inside the Virtual Database Engine is responsible for protecting data and data transmission through encryption. The security manager is also responsible for managing Application, User, Group, Role and Domain privileges as they relate to the creation, manipulation and destruction of Virtual Database data and metadata.
Certainly, Virtuoso provides the end-user or application developer with the ability to make single concurrent connections to multiple back end databases and data sources (XML and XML Web Services). It is important to note that Virtuoso's data access interfaces are all industry standards compliant (XML, Web Services, ODBC, JDBC, NET, and OLE DB) thereby broadening the number of applications, tools, and application development environments capable of exploiting this Virtual Database functionality.
A simple heterogeneous join scenario could involve joining the "Customers" table in an Oracle database with the "Orders" tables in a Microsoft SQL Server database, and then making this available for ODBC and/or XML application consumption.
A more complex scenario could involve joining data across Oracle, SQL Server, and an XML URI (such as an RDF or RSS data feed from a web log, content syndicate, or web service), and then making this available to ODBC, XML, Web Services, JDBC, OLE DB and NET client applications.
Virtuoso implements a traditional relational database engine's functionality. This functionality includes SQL (Structured Query Language) for relational data, a Query processor, View Support, Standard Datatype support including character, number and dates, stored procedures, concurrency support, transactions, distributed transactions, scrollable cursor support just to name a few.
ODBC, JDBC, .net data provider, OLE/DB. The Java and .net clients are pure Java and .Net respectively. ODBC is Virtuoso's native low level CLI.
Virtuoso is fully SQL-92 compliant and supports several of the SQL 200n features, including:
Virtuoso supports standard SQL role based security and full table and column level granting of privileges. Additionally, Virtuoso offers row level security through a system of policy functions. Event hooks can be defined for performing extra validation at login time, such as consulting an LDAP directory for the user's credentials.
Virtuoso's Transaction Manager component ensures that transactions are Atomic, Consistent, Isolated and Durable (ACID). The Transaction Manager ensures VDB Engines are capable of supporting Online Transaction Processing (OLTP) and Distributed Transaction oriented applications and services. VirtuosoA~?A^?s Transaction Manager component ensures that transactions are Atomic, Consistent, Isolated and Durable (ACID). Virtuoso provides the 4 standard levels of isolation, dirty read, read committed, repeatable read and serializable. Repeatable read is the default setting. See Virtual Database Questions for a discussion of distributed transactions.
Virtuoso preserve transaction atomicity, consistency, integrity, and durability across it own database servers as well as heterogeneous servers through the support of transaction commits and rollbacks using a 2-phase commit protocol.
Virtuoso has a cost-based SQL optimizer. It uses table row count, data size and value distribution statistics for evaluating the cost of diverse execution plans. For each plan, loop invariants are extracted, loop and hash join types are evaluated, different indices are compared for access performance and predicates are evaluated as early as possible, most restrictive first, for any join order being contemplated. Additionally, the programmer can explicitly specify the join order and join type (loop/hash) for each table.
Virtuoso database can be administered through the Virtuoso Admin Interface or through the interactive SQL Tool ISQL.
Backups can be handled in a variety of ways. Virtuoso has both off-line and on-line incremental backups.
Virtuoso's transaction mechanism is based on keeping a read-only checkpoint plus a transaction log for any committed but non-checkpointed transactions. Virtuoso has an incremental, on-line backup function, which can backup a running database in its checkpoint state without restricting concurrent update activity. Virtuoso records at each checkpoint the pages, which were changed since the last backup. Hence the next backup will only cover pages , which are changed in the checkpoint state since the last backup. Each checkpoint can optionally start a new transaction log, leaving a full audit trail of transaction logs. Having a series of full plus optionally incremental backups up to a certain checkpoint and then the transaction logs consecutive to the last backed up checkpoint will guarantee a full recovery up to the latest transaction committed at the time of failure. The backup function automatically compresses the backup and chunks it into fixed size chunks for convenience in handling large databases.
On the other hand, as long as no checkpoint intervenes, it is safe to copy the database file(s) plus the log for an up-to-data clean image of the database.
Virtuoso has a very extensive procedure language called Virtuoso PL. . The syntax of the Virtuoso PL resembles C with in-line SQL 99 and PSM 96 features such as exception handlers. Stored procedures provide a significant performance increase over client applications on the same machine for any application involving a number of short SQL operations, as in the case of OLTP.
Virtuoso can extend SQL by import Java and .NET classes and functions as well as build-in SQL functions written in C. In addition , you can persist instances of imported classes into Virtuoso tables. For all intents and purposes, a hosted class instance is indistinguishable from a native SQL user defined type instance.
Virtuoso supports full text indexing which provides the ability to choose complex, multi-part document id's for application specific sorting of hits, efficient storage of secondary, non-free text data in the free text index for best retrieval performance, options for restarting searches at a specific hit as well as ascending and descending orders of the document id's.
Virtuoso supports Unicode (NCHAR/LONG NVARCHAR) columns as well as supporting a national character set, which define how strings will get converted from narrow to wide characters. There is a number of pre-defined character sets included in a system table called SYS_CHARSETS. This list can be updated by defining new characters using a build-in function called charset_define().
Virtuoso supports Unicode, ASCII and UTF.
Virtuoso comes with a library of online tutorials which demonstrate all salient aspects of software development in Virtuoso/PL. Examples include XML processing, full text features, dynamic web pages, hosting Java and .Net and various scripting languages etc.
Virtuoso has a rich set of debugging and profiling tools. The interactive SQL utility has a debug mode where it can set breakpoints and single step stored procedures with functionality similar to what gdb or dbx offer for C.
On Windows, there is a MS Visual Studio extension package which allows defining Virtuoso SQL projects, provides IntelliSense syntax highlighting and completion and many other features. The Virtuoso .net data provider has design time interfaces to Visual Studio for drag and drop use in C# projects.
For performance profiling, Virtuoso offers a call graph profiler and test coverage utility. These show elapsed times and line-by-line execution counts for stored procedure code.
For regular database statistics, Virtuoso provides index by index cache hit rates, wait times, deadlock counts and other information for assisting database tuning.
Yes, Virtuoso enables you to write stored procedures that reference tables hosted in different database engines. This has two major benefits:
Yes, you can create logical VIEWS that include joins across tables within different database engines and data sources (XML and Web Services).
Virtuoso supports the basic SQL OLAP extensions for GROUP BY, i.e. CUBE, ROLLUP, GROUPING etc. Virtuoso does not offer specialized storage for OLAP cubes.
Virtuoso SQL supports SQL2000-style objects as standalone data in procedures and as column data types. The Virtuoso Object System supports single inheritance, late binding, polymorphism and persistence of objects as column values in SQL tables. An object's implementation may be native SQL, with methods in/PL, or the objects may be implemented in Java, or any Microsoft .NET or Mono ECMA CLI bound language. Hosted and native objects are indistinguishable from the perspective of a Virtuoso SQL application. The native methods and data members are automatically made accessible from SQL when the class or class hierarchy is imported into the SQL schema (data dictionary). Virtuoso supports single inheritance between tables through the SQL-200n UNDER clause. A subtable will inherit the supertable's columns, primary key and indices and can for itself add new columns and indices. Selects on supertables will also include rows that belong to subtables.
Virtuoso supports SQL 2K User Defined Types (UDTs) and User-Defined Functions (UDFs) which can be defined in SQL, Java and .NET-bound languages. Virtuoso also offers Procedure Views which are similar to table-valued functions that allows you to define a view as a stored procedure.
Runtime Hosting Languages such as Perl, Python, Ruby loaded in process. These are callable via special functions from SQL. Web pages in these languages can be hosted without any special programming.
Privileges are roles based at the table, column, procedure and type levels. Row Level security is based on table by table policy functions.
Virtuoso supports all combinations of narrow, wide and binary strings.
Virtuoso allows you to create UDTs which can reference any other UDTs. You can also declare columns to contain sort and long UDT instances regardless of implementation be it SQL, Java or .NET.
User-Defined Functions' arguments and return-values are passed by value and by reference. OID-referenced objects have no persistent identity. Virtuoso functions and composite types support dot-operators for data-members and member functions like other formal languages. In addition, since the objects are types at runtime arrays and other complex data structures can also contain UDT instances as well as other data.
Single inheritance for tables is provided via the under table and under for UDT functionality. Function-overloading is provided via dynamic binding of methods based on runtime type of self when there are multiple eligible methods, discriminating on compile-time types of arguments.
Virtuoso supports SQL99 row-level triggers that can call outside code for signalling to clients although no special client-notification mechanism is supported by the Virtuoso client-server protocol.
Virtuoso offers the XMLType datatype. This data-type can be made from a text representation and can again be converted into text. XPATH, XSLT, XQuery and SQLX operations can be applied to this type. It can be stored as either text or in a pre-parsed binary format. XMLType defined as a column value, a XML tree can be specifically full text indexed so that XML elements are taken into account. Besides normal text index operations, this indexing also allows resolving occurrence of elements, inclusion of text inside elements etc.
Virtuoso has several mechanisms for transforming SQL data into XML. One option is the "FOR XML" clause, which is a SQL option that can convert the output of a SELECT statement into a tree.
The SQL-based XML-generation features do not differentiate between local and remote tables, so any relational data will be accessible via these means as XML.
Virtuoso SQL offers special predicates for testing stored XML against XPATH and text conditions as well as extracting fragments from stored XML trees. New XML can be made from these with the SQLX functions or XSLT stylesheets, for example. A result column of a query may fully well consist of XML data. Inside a stored procedure, further XML operations can be applied to the data, in the case of a client the XML is seen as text via the client API.
Another approach is to write queries in XQuery and access SQL data from there. Any XML mapping schema appears as a document accessible with the XMLView XPATH extension function. Joins of tables can thus be accessed and filtered as if they were XML data to begin with. Note that this does not entail constructing XML for data, which is actually not needed by the Xquery statement.
XSLT and XQuery are most convenient for extensive processing of XML. For quickly generating simple XML trees from SQL data, SQLX functions are most convenient. For complex mapping of relational and XML data, mapping schemas are most convenient. For a one-to-one mapping between a foreign-key relation in SQL and tree-hierarchy in XML, the `FOR XML' SQL extension is most convenient.
Virtuoso currently supports XPATH 2.0, XSLT 1.1, and XQuery 1.0.
XPath, Xquery, Freetext queries, and, or not proximity and wildcards. A free-text expression may be embedded inside an XPath predicate using a special XPath expression.
Virtuoso supports offers full support of SQLX which is SQL with a collection of functions added for creating XML entities from standard relational queries. The SQL/XML is an emerging standard driven by the H2.3 Task Group (formerly the SQLX Group). Current support for these functions includes XMLELEMENT, XMLATTRIBUTES, XMLFOREST, XMLCONCAT, and XMLAGG when combined with SQL result in XML that is returned in a column in a result set.
This is possible in either SQL or XQuery. In SQL, for example, one can join a table of XML documents with author information by writing a SQL join between a publications table and an authors table. The author reference would just be extracted with XPATH after application of a free-text-based content-filtering.
In XQuery, the documents table and authors tables would be represented as documents via mapping schemas and the one could write flwr statements for joining the two.
You can load N3, Turtle, and RDF/ XML files into a Virtuoso-hosted "named graph" using Virtuoso SQL functions. The same functionality is also available to single triple statements.
Virtuoso also has the ability to automatically extract metadata from DAV resources via metadata extractors. Virtuoso includes a number of metadata-extractors for a range of known data-formats (typically microformats and some popular binary file types). These metadata extractors enable automatic triple-generation, graph-association, and storage in Virtuoso's RDF Triple Store. It is also important to note that Virtuoso actually converts WebDAV metadata into RDF providing richer query capability against WEbDAV resources.
SPARQL statements can be written inside SQL statements or presented as top-level SQL queries. This means that any ODBC, JDBC, .NET or OLE/DB application can simply make SPARQL queries just as if they were SQL queries.
Virtuoso also supports the SPARQL transport protocol, allowing SPARQL queries to be executed over HTTP. It also supports the SPARQL XML results serialization format.
If an RDF type corresponds to a SQL data type, the data is stored as a native instance of the SQL type in vfaq_question. For strings with language tags and other RDF data that has no direct SQL counterpart, a special representation preserving the RDF semantic is used.
Unlimited.
SPARQL protocol over HTTP; any SQL client library can be used for issuing SPARQL queries just as well as SQL queries.
Virtuoso implements the complete stack of Web Services foundation protocols; SOAP, WSDL, and UDDI. It enables SQL Stored Procedures, Microsoft .NET, Mono, and Java-based application logic to be invoked using SOAP. These SOAP-compliant services are automatically described using WSDL, and advertised for binding via UDDI. This entire process is achievable without writing a single line of new code.
Exposure of existing time-tested application-logic for invocation using Web Services protocols without any code re-writes. Code format support includes SQL Stored Procedures, .NET assemblies, Java Classes, C/C++ modules, etc.
Service Invocation endpoint are HTTP/WebDAV based virtual directory and multi-homing functionality that provides endpoints for SOAP-, WS-Security-, WSDL-, and UDDI-compliant interactions with composite services.
OpenLink Virtuoso can act as a repository through DAV, or WebDAV, a protocol for Web-based Distributed Authoring and Versioning. Repository content elements are called documents, corresponding to files, and folders/collections, corresponding to directories. Collectively these documents and folders (collections) are known as resources.
OpenLink Virtuoso implements the DAV protocol, allowing you to create and manage resources either directly through repository manipulations or indirectly, through a variety of WebDAV services.
Yes, as an HTTP/WebDAV-compliant server all of the components within its WebDAV repository are accessible by URL and WebDAV protocol using HTTP/WebDAV-compliant clients (user agents such as web browsers). The same applies to local operating systems files (subject to security controls of course).
Examples of WebDAV clients include the Windows (via Web Folders feature), Mac OS X, Nautilus Desktop (Linux and Solaris). An ever increasing pool web development tools and content-management systems include in-built support for WebDAV.
Migrating your existing Web Site to one hosted by OpenLink Virtuoso is as simple as using the OpenLink Virtuoso HTML-based UI to import the site.
Virtuoso has provided a built-in RFC-959/RFC-2389 Protocol Server for FTP as well as an FTP client since V3.2. OpenLink virtuoso FTP Client allows through OpenLink virtuoso PL code the ability to retrieve, submit and list files from any FTP server and stored the results on a local file system or in the database. The FTP Server provides FTP access to the OpenLink virtuoso WebDAV repository using the same authentication and permissions system as WebDAV.
Yes, the Web application development language is called OpenLink Virtuoso Server Pages (VSP; file extension .vsp) which can be used in conjunction with XML-based server side control called OpenLink Virtuoso Server Pages for XML (VSPX, file extension .vspx). VSPX offers a suite of data-bound controls for browsing and updating SQL data, input-validation, session-management and any other common web application development environment features.
XSLT can be used for pre- and post-processing VSPX pages. On the preprocessing side, it can serve to divide logic from layout by expanding simple markup into complex controls and scripting. On the output side it can be used to process the HTML generated by a dynamic web page to add HTML layout and graphic elements to bare-bones data produced by the business logic.
Yes, like any search engine it possesses Free Text search capabilities that leverage its ability to produce Free Text indexes on all text data (SQL or otherwise). OpenLink Virtuoso free text search includes word-proximity searches and the ability to combine XPATH, Free Text and Regular Expressions, if required, in the same query. Ultimately you can harvest and then index any form of text-based web data (HTML, XHTML, XML etc.) stored in Virtuoso.
It addresses the fundamental vfaq_question: from where is all the XML data going to come? You can't exploit the benefits of XML without XML data; likewise you can't manually recreate XML data in an attempt to address this reality. OpenLink Virtuoso enables you to create XML data from existing data sources such as your SQL databases. It also enables the creation of XML data from data external URL accessible data. An in-built validating XML parser and an XSL-T engine lie at the root of Virtuoso's XML Services offerings.
Yes, XML documents are stored in an XML repository. These documents may be parsed or unparsed at time of storage; in either case indexes are built which provide rapid access to these documents.
A major benefit of XML is its ability to provide an open format for data representation, exchange, protocol and application modeling. By using XML as a uniform data interface to disparate data sources, it becomes much easier to cost-effectively develop and deploy next generation web applications; increasingly these applications will depend on data hosted in a variety of databases and data sources.
Yes, Virtuoso implements SQL extensions that enable the results of standard SQL queries to be transformed into XML documents, which are openly accessible to user agents such as Web Browsers via HTTP and/or WebDAV. It is important to note that these SQL-XML documents are accessible by URL and sensitive to the underlying changes that occur in the underlying database tables from which they have been derived.
Yes, the SQL to XML functionality sits above Virtuoso's virtual database (VDB) functionality, which enables a unified logical and physical representation of database tables and views that reside in disparate database engines from different database vendors. The only requirement is that these databases have to be ODBC- or JDBC-accessible - implying the existence of data-access drivers for these databases.
It supports any database that supports ODBC and/or JDBC, so pretty much every database.
A lot of data that you would typically like to use as the foundation of your web application initiatives more than likely resides in application databases that are predominantly SQL-based.
SOAP 1.1, 1.2, WS-Security, WS-Trust and WS-Policy.
Virtuoso support of Web Services Security includes enabling the use of symmetric and asymmetric encryption, digital signing, and identity authentication as defined by the WS-Security specification.
Published SOAP Services can be tested through VSMX which is a Virtuoso-generated test page allowing you to test SOAP Services. The SOAP Services can be referenced by the Server instances URL and the virtual directory containing the logical path of the services.
Business logic defined in stored procedures and functions can be exposed as SOAP services whether they are native within Virtuoso or from remote data sources. These stored procedure can be published by linking the selected stored procedure through the Remote Procedure Interface, then creating a new or selecting an existing Virtual Directory and finally publishing to the Virtual Directory using the publish function in the Virtual Director User Interface.
Virtuoso participates in interoperability tests rounds for different implementations of the SOAP and WSDL specifications defined by the SOAP Builders and WS-RM Interop communities. The Virtuoso SOAP server and client implements SOAP protocol versions 1.1 and 1.2. The SOAP services hosted in Virtuoso can be described with an automatically generated WSDL document or with a user-supplied WSDL document. The Virtuoso SOAP Server and client with protocol support version 1.1 and automatically generated WSDL documents have been tested. Interop tests are organized into rounds and each round consists of the following groups: (List of service endpoints and client results is available in html form or via SOAP).
Virtuoso's SOAP server and client with protocol support for version 1.2 have been tested with test cases as per W3C SOAP Version 1.2 Specification Assertions and Test Collection. Rounds 1-4 tests are located on Virtuoso Interop and through our SOAP on-line tutorials. The WS-RM interop tests can be experienced through our on-line WS-RM tutorials.
BPEL4WS (BPEL for short) is an XML vocabulary for orchestrating SOAP and WSDL-compliant Web Services. It is the critical standard for creating composite processes from a collection of Web Services using the principles of Service Oriented Architecture (SOA).
Virtuoso BPEL Process Manager (VBPM) engine is a run-time and administration environment for executing processes based on BPEL4WS 1.1 (BPEL for Web Services), the latest version of a specification designed by Microsoft, IBM, BEA Systems and Siebel Systems. The software vendors are shepherding the spec through e-business standards body OASIS.
Virtuoso provides a number of WS protocols on the WS-I standards including Security (WS-Security) and Reliability (WS-Reliable Messaging). Virtuoso BPEL engine also includes a number of Web Services protocols that add security, reliability and Enterprise scalability.
Processes in can be deployed for any business-critical transactions over the Internet using the WS-Reliable Messaging (WS-RM) specification through guaranteed (at-most-once, at-least-once, and exactly once) messaging for any partner via BPEL Process Manager.
Yes. Virtuoso BPEL Process manager fully implements the BPEL specification and can deploy any BPEL document created using any BPEL modeling tool that supports this specification.
The BPEL Process Manager includes a web-based user interface for testing, debugging, and monitoring deployed business process.
The BPEL Process manager includes support for compensation, fault-handling and event handling.
The BPEL Process Manager product includes a debugger to test business processes in the Debug Message Queue, which includes the state of a given process, actions for the process and list of instances for the process.
The BPEL Process Manager provides real-time status for all running processes and transactions on any BPEL processes deployed on the server including Information on bpel source, wsdl and partner links.
The BPEL Process manager provides statistics and reporting details on process and instances which can be so that processes can be analyzed and later optimized for deployment.
The BPEL Process manager has extensive database-, XML- and Web Service integration and provides for intelligent transformation for XML and non-XML applications and data. Virtuoso allows integration of databases into Service-Oriented Architectures (SOAs), by automating the creation of Web Services from multiple tables in any ODBC-, JDBC- or ADO.Net-compliant database.
Through the BPEL process manager any BPEL processes can directly invoke local hosted Java or .NET logic as well as access SQL data resident in local and remote tables.
Virtuoso BPEL has gone through an extensive interoperability testing against Microsoft, Oracle, and other vendors' implementations. Virtuoso also offers an interoperability site for testing and sharing results, which validate the Virtuoso BPEL engine, and ensures the rapid orchestration of existing Web services.
The BPEL engine requires at least Virtuoso Universal Server 4.5 and the BPEL Process Manager, which is compatible with any browser on any platform.
Virtuoso includes an SMTP sink/drive that enables any SMTP-compliant Mail Transfer Agent (MTA, such as sendmail, exim) to write its mail into a Virtuoso database. Email data stored in Virtuoso is retrievable via the POP3 protocol with IMAP4 support planned for a later release. SMTP and POP3 are also available as Virtuoso PL functions enabling the creation of sophisticated mail-oriented applications.
Yes, Virtuoso provides e-mail storage drivers for popular SMTP servers such as sendmail and Windows SMTP. This means that you can configure your mail server such that it stores e-mails inside Virtuoso rather than in the OS.
There are two ways of doing this: the first is use Virtuoso as your POP3 server, and then let your POP3-based mail client interact directly with Virtuoso. Secondly, you can move mail from within the Virtuoso SQL repository into the WebDAV repository for direct access.
Virtuoso enables you to combine the functionality of a mail system and database so that you can custom develop spam filters or customize Virtuoso (via its server extensions kit) to work with 3rd party Spam filter tools. Virtuoso can enable spam-filtering to occur immediately after mails are deposited in your mailbox, and prior to POP3 mail retrieval.
Yes, Virtuoso implements NNTP (Network News Transport Protocol) allowing a very simple interface for creating your own newsgroups.
Yes, you can use Virtuoso to attach to newsgroups hosted on any NNTP Server.
For ODBC/JDBC Access:
Database:
Web Services:
Approximately 300MB. If you exclude tutorial materials, sample applications, demos, and documentation the installation is approximately 100MB.
Nothing special, it can run within an 64MB environment if this is all you have, but it can also take advantage of large system memory providing you with the ability configure Virtuoso as an in-memory database if you so desire.
Virtuoso currently supports Windows (XP/2K/2003), Linux (Redhat,Suse) Mac OS X, FreeBSD, Solaris, and other UNIX (32- & 64-bit platforms).
Virtuoso comes in two distinct formats: Standard Edition and Enterprise Edition. The Enterprise Edition includes out of the box bindings for third-party distributed transaction-management environments such as MTS/COM+ and J2EE.
Virtuoso clients connect to Virtuoso via standard APIs and protocols. The APIs includes ODBC, JDBC, a .NET data provider and OLE/DB. Client libraries are provided for all these. Protocols include SOAP, most WS* protocols, DAV, XML for Analysis (XML/A) and others. Common operating systems and development environments offer clients for these protocols.
For adding application logic server side, one may use Virtuoso/PL, Java, .NET bound languages or C. Examples of all these are included in the package.
A single query runs on single thread (plus I/O in separate thread) in Virtuoso. Without waiting for disk, it's faster to run it in a single thread than to parallelize and get synchronization overheads . While waiting for disk access, CPU times does not matter as the idle CPUs are waiting for disk reads.
The key requirement to an RDBMS database is its ability to serve hundreds clients in parallel. Having 4 or more queries from 4 or more clients, all cores of the machine will have something to run.
When we complete the Virtuoso cluster version you will be able to parallelize a query between boxes, in that case many boxes may simultaneously run many threads per query; but any given box of the cluster will run no more than one thread per query at time. The cluster wins in speed because the total RAM is bigger, not because multi-threading per query is so useful.
See detailed information here.
This depends on the index scheme. If indexed 2 ways, assuming that the graph will always be stated in queries, this is 31 bytes.
With 4 indices, supporting queries where the graph can be left unspecified (i.e., triples from any graph will be considered in query evaluation), this is 39 bytes. The numbers are measured with the LUBM validation data set of 121K triples, with no full-text index on literals.
With 4 indices and a full text index on all literals, the Billion Triples Challenge data set, 1115M triples, is about 120 GB of database pages. The database file size is larger due to space in reserve and other factors. 120 GB is the number to use when assessing RAM-to-disk ratio, i.e., how much RAM the system ought to have in order to provide good response. This data set is a heterogeneous collection including social network data, conversations harvested from the Web, DBpedia, Freebase, etc., with relatively numerous and long text literals.
The numbers do not involve any database page stream compression such as gzip. Using such compression does not save in terms of RAM because cached pages must be kept uncompressed but will cut the disk usage to about half.
The more triples are inserted at a time, the faster this goes. Also, the more concurrent triple insertions are going on, the better the throughput. When loading data such as the US Census, a cluster of 2 commodity servers can insert up to 100,000 triples per second.
A single 4-core machine can load 1 billion triples of LUBM data at an average rate of 36K triples per second. This is limited by disk.
The delete cost is similar to insert cost.
If we are looking for equality matches, a single 2GHz core can do about 250,000 single triple random lookups per second as long as disk reads are not involved. If each triple requires a disk seek the number is naturally lower.
Parallelism depends on the query. With a query like counting all x and y such that x knows y and y knows x, we get up to 3.4 million single-triple lookups-per-second on a cluster of 2 8-core Xeon servers. With complex nested sub-queries the parallelism may be less.
Lookups involving ranges of values, such as ranges of geographical coordinates or dates use an index, since quads are indexed in a manner that collates in the natural order of the data type.
Virtuoso supports all RDF data types, including language-tagged and XML schema typed strings as native data types. Thus there is no overhead converting between RDF data types and types supported by the underlying DBMS.
Subclass, subproperty, identity by inverse-functional
properties, and owl:sameAs are processed at
run time if an inference context option is specified in the
query.
There is a general-purpose transitivity feature that can be used for a wide variety of graph algorithms. For example:
SELECT ?friend
WHERE
{
<alice> foaf:knows ?friend option
(transitive)
}
would return all the people directly or indirectly known by <alice>.
The mentioned types of inferencing are enabled by a switch in the query and are done at run-time, with no step for materialization of entailed triples needed. The pattern:
{<thing> a ?class}
will, if the match of ?class has
superclasses, also return the superclasses even though the
superclass membership is not physically stored for each
superclass.
Of course, one can always materialize entailed triples by running SPARQL/SPARUL statements to explicitly add any implied information.
If two subjects have the same inverse functional property with the same value, they will be considered the same. For example, if two people have the same email address, they will be considered the same.
If two subjects are declared to be owl:sameAs, either
directly or through a chain of x owl:sameAs y, y
owl:sameAs z, and so on, they will be considered the
same.
These features can be individually enabled and disabled. They all have some run time cost, hence they are optional. The advantage is that no preprocessing of the data itself is needed before querying, and the data does not get bigger. This is important, especially if the database is very large and queries touch only small parts of it. In such cases, materializing implied triples can be very costly. See discussion at E Pluribus Unum ...
Virtuoso has an optional full-text index on RDF
literals. Searching for text matches using the SPARQL regex
feature is very inefficient in the best of cases. This is
why Virtuoso offers a special bif:contains
predicate similar to the SQL contains
predicate of many relational databases. This supports a
full-text query language with proximity, and/or/and-not,
wildcards, etc.
While the full-text index is a general-purpose SQL feature in Virtuoso, there is extra RDF-specific intelligence built into it. One can, for example, specify which properties are indexed, and within which graphs this applies.
Virtuoso supports the standard SPARQL protocol.
Virtuoso offers drivers for the Jena, Sesame, and Redland frameworks. These allow using Virtuoso's store and SPARQL implementation as the back end of Jena, Sesame, or Redland applications. Virtuoso will then do the query optimization and execution. Jena and Sesame drivers come standard; contact us about Redland.
Virtuoso SPARQL can be used through any SQL call level interface (CLI) supported by Virtuoso (i.e., ODBC, JDBC, OLE-DB, ADO.NET, XMLA). All have suitable extensions for RDF specific data types such as IRIs and typed literals. In this way, one can write, for example, PHP web pages with SPARQL queries embedded, just using the SQL tools. Prefixing a SQL query with the keyword "sparql" will invoke SPARQL instead of SQL, through any SQL client API.
Virtuoso Cluster partitions each index of all tables containing RDF data separately. The partitioning is by hash. The result is that the data is evenly distributed over the selected number of servers. Immediately consecutive triples are generally in the same partition, since the low bits of IDs do not enter in into the partition hash. This means that key compression works well.
Since RDF tables are in the end just SQL tables, SQL can be used for specifying a non-standard partitioning scheme. For example, one could dedicate one set of servers for one index, and another set for another index. Special cases might justify doing this.
With very large deployments, using a degree of application-specific data structures may be advisable. See "Does Virtuoso support property tables" below.
With free-form data and text indexing enabled, 500M triples per 16G RAM can be a ballpark guideline. If the triples are very short and repetitive, like the LUBM test data, then 16G per one billion triples is a possibility. Much depends on the expected query load. If queries are simple lookups, then less memory per billion triples is needed. If queries will be complex (analytics, join sequences, and aggregations all over the data set), then relatively more RAM is necessary for good performance.
The count of quads has little impact on performance as long as the working set fits in memory. If the working set is in memory, there may be 15-20% difference between a million and a billion triples. If the database must frequently go to disk, this degrades performance since one can easily do 2000 random accesses in memory in the time it takes to do one random access from disk. But working-set characteristics depend entirely on the application.
Whether the quads in a store all belong to one graph or any number of graphs makes no difference. There are Virtuoso instances in regular online use with hundreds of millions of triples, such as DBpedia database for example.
Performance dynamics change when going from a single server to a cluster. If each partition is around a billion triples in size, then the single triple lookup takes the same time, but there is cluster interconnect latency added to the mix.
On the other hand, queries that touch multiple partitions or multiple triples in a partition will do this in parallel and usually with a single message per partition. Thus throughput is higher.
In general terms, operations on a single triple at a time from a single thread are penalized and operations on hundreds or more triples at a time win. Multiuser throughput is generally better due to more cores and more memory, and latency is absorbed by having large numbers of concurrent requests.
See Post about Virtuoso & SPARQL scalability.
Since quads (triple plus graph) are stored in a regular SQL table with special data types, changing the table layout to add a column is possible. This column would not however be visible to SPARQL without some extra tuning. For coarse grain provenance and security information, we recommend doing this at the graph level, where triples that belong together are tagged with the same provenance or security are in the same graph. The graph can then have the relevant metadata as its properties.
If tagging at the single triple level is needed, this will most often not be needed for all triples. Hence altering the table for all triples may not be the best choice. Making a special table that has the graph, subject, predicate and object of the tagged triple as a key and the tag data as a dependent part may be more efficient. Also, this table could be more easily accessed from SPARQL.
Using the RDF reification vocabulary is not recommended as a first choice but is possible without any alterations.
Alterations of this nature are possible but we recommend contacting us for specifics. We can provide consultancy on the best way to do this for each application. Altering the storage layout without some extra support from us is not recommended.
If the application has high heterogeneity of schema and frequent need for adaptation, then RDF is recommended. The alternative is making a relational database.
Making a custom non-RDF object-attribute-value representation on Virtuoso or some other RDBMS is possible but not recommended.
The reason for this is that this would miss many of the optimizations made specifically for RDF, use of the SPARQL language, inference, compatibility with diverse browsers and front end tools, etc. Not to mention interoperability and joinability with the body of linked data. Even if the application is strictly private, using entity names and ontologies from the open world can still have advantages.
If some customization to the quad (triple plus graph layout) is needed, we can provide consultancy on how to do this while staying within the general RDF framework and retaining all the interoperability benefits.
All server and client components are multithreaded, using pthreads on Unix/Linux, Windows native on Windows. Multithread/multicore scalability is good; see BSBM
In the case of Virtuoso Cluster, in order to have the maximum number of threads on a single query, we recommend that each server on the cluster be running one Virtuoso process per 1.2 cores.
Yes. Unlike Oracle RAC, for example, Virtuoso Cluster does not require a SAN. Each server has its own database files and is solely responsible for these. In this way, having shared disk among all servers is not required. Running on a SAN may still be desirable for administration reasons. If using a SAN, the connection to the SAN should be high performance, such as Infiniband.
Partitioning is entirely transparent to the application. Virtuoso has a highly optimized message-flow between cluster nodes that combines operations into large batches and evaluates conditions close to the data. See Post about Virtuoso & Web Scale RDF.
This is a planned extension. The logic for optimizing message flow between multiple end-points on a wide-area network is similar to the logic for message-optimization on a cluster. This will allow submitting a query with a list of end-points. The query will then consider triples from each of the end points, as if the content of all the end points were in a single store.
End-point meta information, such as voiD descriptions of the graphs in the end-points, may be used to avoid sending queries to end points that are known not to have a certain type of data.
There is no fixed limit. If you have a large cluster installed, you can try Virtuoso there. Having an even point-to-point latency is desirable.
We are working on a system whereby servers can be added and removed from a cluster during operation and no repartitioning of the data is needed.
In the first release, the number of server processes that make up the cluster is set when creating the database. These processes with their database files can then be moved between machines but this requires stopping the cluster and updating configuration files.
Performance of a cluster depends on the latency and bandwidth of the interconnect. At least dual 1Gbit ethernet is recommended for each node. Thus a cluster should be on a single local or system area network.
If regional copies are needed, we would replicate between clusters by asynchronous log shipping. This requires some custom engineering.
When a transaction is committed at one site, it is logged and sent to the subscribing sites if they are online. If there is no connection, the subscribing sites will get the data from the log. This scheme now works between single Virtuoso servers, and needs some custom development to be adapted to clusters.
If replicating all the data of one site to another site is not possible, then application logic should be involved. Also, if consolidated queries should be made against large, geographically-separated clusters, then it is best to query them separately and merge the results in the application. All depends on the application level rules on where data resides.
Virtuoso SPARQL and SQL offer an "anytime" option that will return partial results after a configurable timeout.
In this way, queries will return in a predictable time and indicate whether the results are complete or not, as well as give a summary of resource utilization.
This is especially useful for publishing a SPARQL endpoint where a single long running query could impact the performance of the whole system. This timeout significantly reduces the risk of denial of service.
This is also more user-friendly than simply timing-out a
query after a set period and returning an error. With the
anytime option, the user gets a feel for what
data may exist, including whether any data exists at all.
This feature works with arbitrarily complex queries,
including aggregation, GROUP BY, ORDER BY,
transitivity, etc.
Since the Virtuoso SPARQL endpoint supports open authentication (OAuth), the authentication can be used for setting timeouts, so as to give different service to different users.
It is also possible to set a timeout that will simply abort a query or an update transaction if it fails to terminate in a set time.
Disconnecting the client from the server will also terminate any processing on behalf of that client, regardless of timeout settings.
The SQL client call-level interfaces (ODBC, JDBC, OLE-DB, ADO.NET, XMLA) each support a cancel call that can terminate a long running query from the application, without needing to disconnect.
There is no off-the-shelf API for this but making an adaptation of the SPARQL endpoint that could proceed after the client disconnected and, for example, could send results by email is trivial. Since SOAP and REST Web services can be programmed directly in Virtuoso's stored procedure language, implementing and exposing this type of application logic is easy.
There is no set limit. As with any DBMS, response times get longer if there is severe congestion.
For example, having 2 or 3 concurrent queries per core is a good performance point which will keep all parts of the system busy. Having more than this is possible but will not increase overall throughput.
With a cluster, each server has both HTTP and SQL listeners, so clients can be evenly spread across all nodes. In a heavy traffic Web application, it is best to have a load balancer in front of the HTTP endpoints to divide the connections among the servers and to keep some cap on the number of concurrently running requests, enforcing a maximum request-rate per client IP address, etc.
This is application dependent. In Virtuoso, SPARQL and SQL share the same query execution engine, query optimizer, and cost model. If data is highly regular (i.e., a good fit for relational representation), and if queries typically access most of the row, then SQL will be more efficient. If queries are unpredictable, data is ragged, schema changes frequent, or inference is needed, then RDF will do relatively better.
The recent Berlin SPARQL Benchmark shows some figures comparing Virtuoso SQL and SPARQL and SPARQL in front of relational representation. However, the test workload is heavily biased in favor of relational. See also BSBM: MySQL vs Virtuoso.
With the TPC-H workload, relationally stored data, and SPARQL mapped to SQL, we find that with about half the queries there is no significant cost to SPARQL. With some queries there is additional overhead because the mapping does not produce a SQL query identical to that specified in the benchmark.
For large applications, we would recommend RDF whenever there is significant variability of schema, but would still use an application-specific, relational style representation for those parts of the data that are regular in format. This is possible without loss of flexibility for the variable-schema part. However, this will introduce relational-style restrictions on the regular data; for example, a person could only have a single date-of-birth by design. In many cases, such restrictions are quite acceptable. Querying will still take place in SPARQL, and the representation will be transparent.
A relational table where the primary key is the RDF subject and where columns represent single-valued properties is usually called a property table. These can be defined in a manner similar to defining RDF mappings of relational tables.
There is an extensive array of performance metrics. This includes:
explain functionAll client interfaces and server-side processes are multithreaded. As usual, each thread of an application should use a different connection to the database.
Virtuoso supports all 4 isolation levels from dirty read to serializable, for both relational and RDF data.
The recommended default isolation is read-committed, which offers a clean historical read of data that has uncommitted updates. This mode is similar to the Oracle default isolation and guarantees that no uncommitted data is seen, and that no read will block waiting for a lock held by another client.
There is transaction logging and roll forward recovery, with two phase commit used in Virtuoso Cluster if an update transaction modifies more than one server.
For RDF workloads which typically are not transactional and have large bulk loads, we recommend running in a "row autocommit" mode without transaction logging. This virtually eliminates log contention but still guarantees consistent results of multithreaded bulk loads.
Setting this up requires some consultancy and custom development but is well worthwhile for large projects.
Yes. Doing this with millions of triples per transaction may run out of rollback space. Also, there is risk of deadlock if multiple such inserts run at the same time. For good concurrency, the inserts should be of moderate size. As usual, deadlocks are resolved by aborting one of the conflicting transactions.
Yes. The reading client should specify serializable isolation and the inserting client should perform the insert as a transaction, no row autocommit mode.
Starting a Virtuoso server process takes a few seconds. Making a new graph takes no time beyond the time to insert the triples into it. Once the server process(es) are running, all the data is online.
With high-traffic applications, reaching cruising speed may sometimes take a long time, specially if the load is random-access intensive. Filling gigabytes of RAM with cached disk pages takes a long time if done a page at a time. To alleviate this, Virtuoso pre-reads 2MB-sized extents instead of single pages if there is repeated access to the same extent within a short time. Thus cache warm-up times are shortened.
For SQL, we have the standard role-based security and an Oracle-style row-level security (policy) feature.
For SPARQL, users may have read or update roles at the level of the quad store.
With RDF, a graph may be owned by a user. The user may specify read and write privileges on the graph. These are then enforced for SPARUL (the SPARQL update language) and SPARQL.
When an RDF graph is based on relationally stored data in Virtuoso or another RDBMS through Virtuoso's SQL federation feature (i.e., if the graph is an RDF View of underlying SQL data), then all relational security controls apply.
Further, due to the dual-nature of Virtuoso, sophisticated ontology-based security models are feasible. Such models are not currently used by default, but they are achievable with our consultancy.