http://virtuoso.openlinksw.com/blog/vdb/blog/
OpenLink Virtuoso (Product Blog)
A great place to track Virtuoso's rapid evolution.
Virtuso Data Space Bot
kidehen@openlinksw.com
2008-10-11T00:22:14Z
Virtuoso Universal Server 05.08.3034
OpenLink Software 1998-2006
http://virtuoso.openlinksw.com/weblog/public/images/vbloglogo.gif
Virtuoso Cluster Paper Update
http://virtuoso.openlinksw.com/blog/vdb/blog/?id=1451
2008-10-02T10:02:33Z
<p>An updated version of the paper about <a href="http://virtuoso.openlinksw.com" id="link-id0xc0abc50">Virtuoso</a> Cluster is available at <a href="http://www.openlinksw.com/weblog/oerling/2008webscale_rdf.pdf" id="link-id16459248">2008webscale_rdf.pdf</a> </p>
Virtuso Data Space Bot
kidehen@openlinksw.com
2008-10-03T04:38:06.000-04:00
Virtuoso Update, Billion Triples and Outlook
http://virtuoso.openlinksw.com/blog/vdb/blog/?id=1450
2008-10-02T10:02:32Z
<div> <div style="display:none;">Virtuoso Update, Billion Triples and Outlook</div> <p>I will say a few things about what we have been doing and where we can go.</p> <p>Firstly, we have a fairly scalable platform with <a href="http://virtuoso.openlinksw.com" id="link-id0x1aa82dc0">Virtuoso</a> 6 Cluster. It was most recently tested with the workload discussed in the previous <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1445" id="link-id1638a5b8">Billion Triples post</a>.</p> <p>There is an updated version of <a href="http://www.openlinksw.com/weblog/oerling/2008webscale_rdf.pdf" id="link-id16280a68">the paper about this</a>. This will be presented at the web scale workshop of ISWC 2008 in Karlsruhe.</p> <p>Right now, we are polishing some things in Virtuoso 6 -- some optimizations for smarter balancing of interconnect traffic over multiple network interfaces, and some more <a href="http://dbpedia.org/resource/SQL" id="link-id0x1abd3f38">SQL</a> optimizations specific to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1adbe410">RDF</a>. The must-have basics, like parallel running of sub-queries and aggregates, and all-around unrolling of loops of every kind into large partitioned batches, is all there and proven to work.</p> <p>We spent a lot of time around the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1aaa0e78">Berlin SPARQL Benchmark</a> story, so we got to the more advanced stuff like the <a href="http://challenge.semanticweb.org/" id="link-id0x1a860a50">Billion Triples Challenge</a> rather late. We did along the way also run <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1a27f2a8">BSBM</a> with an <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0x1ad5c918">Oracle</a> back-end, with Virtuoso mapping <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1cf0e4a0">SPARQL</a> to SQL. This merits its own analysis in the near future. This will be the basic how-to of mapping OLTP systems to RDF. Depending on the case, one can use this for lookups in real-time or ETL.</p> <p>RDF will deliver value in complex situations. An example of a complex relational mapping use case came from Ordnance Survey, presented at the <a href="http://www.w3.org/2005/Incubator/rdb2rdf/" id="link-id0x1ab96bb0">RDB2RDF XG</a>. Examples of complex warehouses include the <a href="http://neurocommons.org/page/Main_Page" id="link-id0x1adb2db0">Neurocommons</a> database, the Billion Triples Challenge, and the <a href="http://www.garlik.com/" id="link-id0x1925c7b0">Garlik DataPatrol</a>.</p> <p>In comparison, the Berlin workload is really simple and one where RDF is not at its best, as amply discussed on the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1c6d1480">Linked Data</a> forum. BSBM's primary value is as a demonstrator for the basic mapping tasks that will be repeated over and over for pretty much any online system when presence on the <a href="http://dbpedia.org/resource/Data" id="link-id0x1a937400">data</a> web becomes as indispensable as presence on the HTML web.</p> <p>I will now talk about the complex warehouse/web-harvesting side. I will come to the mapping in another post.</p> <p>Now, all the things shown in the <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1445" id="link-id14de1d18">Billion Triples post</a> can be done with a relational system specially built for each purpose. Since we are a general purpose <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x1a457c70">RDBMS</a>, we use this capability where it makes sense. For example, storing statistics about which tags or interests occur with which other tags or interests as RDF blank nodes makes no sense. We do not even make the experiment; we know ahead of time that the result is at least an order of magnitude in favor of the relational row-oriented solution in both space and time.</p> <p>Whenever there is a data structure specially made for answering one specific question, like joint occurrence of tags, RDB and mapping is the way to go. With Virtuoso, this can fully-well coexist with physical triples, and can still be accessed in SPARQL and mixed with triples. This is territory that we have not extensively covered yet, but we will be giving some examples about this later.</p> <p>The real value of RDF is in agility. When there is no time to design and load a new warehouse for every new question, RDF is unparalleled. Also SPARQL, once it has the necessary extensions of aggregating and sub-queries, is nicer than SQL, especially when we have sub-classes and sub-properties, transitivity, and "same as" enabled. These things have some run time cost and if there is a report one is hitting absolutely all the time, then chances are that resolving terms and identity at load-time and using materialized views in SQL is the reasonable thing. If one is inventing a new report every time, then RDF has a lot more convenience and flexibility.</p> <p>We are just beginning to explore what we can do with data sets such as the online conversation space, linked data, and the open ontologies of <a href="http://umbel.org/about/" id="link-id0x1aa5ea18">UMBEL</a> and <a href="http://dbpedia.org/resource/Cyc" id="link-id0x1a631a20">OpenCyc</a>. It is safe to say that we can run with real world scale without loss of query expressivity. There is an incremental cost for performance but this is not prohibitive. Serving the whole billion triples set from memory would cost about $32K in hardware. $8K will do if one can wait for disk part of the time. One can use these numbers as a basis for costing larger systems. For online search applications, one will note that running the indexes pretty much from memory is necessary for flat response time. For back office analytics this is not necessarily as critical. It all depends on the use case.</p> <p>We expect to be able to combine geography, social proximity, subject matter, and <a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-id0x1aebdcc8">named entities</a>, with hierarchical taxonomies and traditional full text, and to present this through a simple user interface.</p> <p>We expect to do this with online response times if we have a limited set of starting points and do not navigate more than 2 or 3 steps from each starting point. An example would be to have a full text pattern and news group, and get the cloud of interests from the authors of matching posts. Another would be to make a faceted view of the properties of the 1000 people most closely connected to one person.</p> <p>Queries like finding the fastest online responders to questions about romance across the global board-scape, or finding the person who initiates the most long running conversations about crime, take a bit longer but are entirely possible.</p> <p>The genius of RDF is to be able to do these things within a general purpose database, ad hoc, in a single query language, mostly without materializing intermediate results. Any of these things could be done with arbitrary efficiency in a custom built system. But what is special now is that the cost of access to this type of <a href="http://dbpedia.org/resource/Information" id="link-id0x1ab88490">information</a> and far beyond drops dramatically as we can do these things in a far less labor intensive way, with a general purpose system, with no redesigning and reloading of warehouses at every turn. The query becomes a commodity.</p> <p>Still, one must know what to ask. In this respect, the self-describing nature of RDF is unmatched. A query like <i>list the top 10 attributes with the most distinct values for all persons</i> cannot be done in SQL. SQL simply does not allow the columns to be variable.</p> <p>Further, we can accept queries as text, the way people are used to supplying them, and use structure for drill-down or result-relevance, and also recognize named entities and subject matter concepts in query text. Very simple NLP will go a long way towards keeping SPARQL out of the user experience.</p> <p>The other way of keeping query complexity hidden is to publish hand-written SPARQL as parameter-fed canned reports.</p> <p>Between now and ISWC 2008, the last week of October, we will put out demos showing some of these things. Stay tuned.</p> </div>
Virtuso Data Space Bot
kidehen@openlinksw.com
2008-10-02T12:47:07.4000-04:00
OpenLink Software's Virtuoso Submission to the Billion Triples Challenge
http://virtuoso.openlinksw.com/blog/vdb/blog/?id=1446
2008-09-30T16:24:34Z
<div> <h2>Introduction</h2> <p>We use <a href="http://virtuoso.openlinksw.com" id="link-id0xb03e418">Virtuoso</a> 6 Cluster Edition to demonstrate the following:</p> <ul> <li>Text and structured <a href="http://dbpedia.org/resource/Information" id="link-id0xbd9dae8">information</a> based lookups</li> <li>Analytics queries</li> <li>Analysis of co-occurrence of features like interests and tags.</li> <li>Dealing with identity of multiple IRI's (<a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0xb383dd8">owl</a>:sameAs)</li> </ul> <p>The demo is based on a set of canned <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xbda6298">SPARQL</a> queries that can be invoked using the <a href="http://ode.openlinksw.com/" id="link-id0xbb292f0">OpenLink Data Explorer</a> (<a href="http://ode.openlinksw.com/" id="link-id0xc263528">ODE</a>) Firefox extension.</p> <p>The demo queries can also be run directly against the SPARQL end point.</p> <p>The demo is being worked on at the time of submission and may be shown online by appointment.</p> <p>Automatic annotation of the <a href="http://dbpedia.org/resource/Data" id="link-id0xa173378">data</a> based on <a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-id0xbdda558">named entity extraction</a> is being worked on at the time of this submission. By the time of ISWC 2008 the set of sample queries will be enhanced with queries based on extracted <a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-id0xa66fbe0">named entities</a> and their relationships in the <a href="http://umbel.org/about/" id="link-id0xa06e2c8">UMBEL</a> and Open CYC ontologies. </p> <p>Also examples involving owl:sameAs are being added, likewise with similarity metrics and search hit scores.</p> <h2>The Data</h2> <p>The database consists of the billion triples data sets and some additions like Umbel. Also the Freebase extract is newer than the challenge original.</p> <p>The triple count is 1115 million.</p> <p>In the case of web harvested resources, the data is loaded in one graph per resource.</p> <p>In the case of larger data sets like <a href="http://dbpedia.org/resource/DBpedia" id="link-id0xc2bf770">Dbpedia</a> or the US census, all triples of the provenance share a data set specific graph.</p> <p>All string literals are additionally indexed in a full text index. No stop words are used.</p> <p>Most queries do not specify a graph. Thus they are evaluated against the union of all the graphs in the database. The indexing scheme is SPOG, GPOS, POGS, OPGS. All indices ending in S are bitmap indices. </p> <h2>The Queries </h2> <p>The demo uses Virtuoso SPARQL extensions in most queries. These extensions consist on one hand of well known <a href="http://dbpedia.org/resource/SQL" id="link-id0xaf8cb40">SQL</a> features like aggregation with grouping and existence and value subqueries and on the other of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xafdceb8">RDF</a> specific features. The latter include run time RDFS and OWL inferencing support and backward chaining subclasses and transitivity. </p> <h3>Simple Lookups</h3> <pre>sparql select ?s ?p (bif:search_excerpt (bif:vector ('<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0xbb64dd0">semantic', 'web</a>'), ?o)) where { ?s ?p ?o . filter (bif:contains (?o, "'semantic web'")) } limit 10 ; </pre> <p>This looks up triples with semantic web in the object and makes a search hit summary of the literal, highlighting the search terms. </p> <pre>sparql select ?tp count(*) where { ?s ?p2 ?o2 . ?o2 a ?tp . ?s foaf:nick ?o . filter (bif:contains (?o, "plaid_skirt")) } group by ?tp order by desc 2 limit 40 ; </pre> <p>This looks at what sorts of things are referenced by the properties of the foaf handle plaid_skirt.</p> <p>What are these things called?</p> <pre>sparql select ?lbl count(*) where { ?s ?p2 ?o2 . ?o2 rdfs:label ?lbl . ?s foaf:nick ?o . filter (bif:contains (?o, "plaid_skirt")) } group by ?lbl order by desc 2 ; </pre> <p>Many of these things do not have a rdfs:label. Let us use a more general concept of lable which groups dc:title, foaf:name and other name-like properties together. The subproperties are resolved at run time, there is no materialization. </p> <pre>sparql define input:inference 'b3s' select ?lbl count(*) where { ?s ?p2 ?o2 . ?o2 b3s:label ?lbl . ?s foaf:nick ?o . filter (bif:contains (?o, "plaid_skirt")) } group by ?lbl order by desc 2 ; </pre> <p>We can list sources by the topics they contain. Below we look for graphs that mention terrorist bombing. </p> <pre>sparql select ?g count(*) where { graph ?g { ?s ?p ?o . filter (bif:contains (?o, "'terrorist bombing'")) } } group by ?g order by desc 2 ; </pre> <p>Now some web 2.0 tagging of search results. The <a href="http://dbpedia.org/resource/Tag" id="link-id0xa8b89f8">tag</a> cloud of "computer"</p> <pre>sparql select ?lbl count (*) where { ?s ?p ?o . ?o bif:contains "computer" . ?s sioc:topic ?tg . optional { ?tg rdfs:label ?lbl } } group by ?lbl order by desc 2 limit 40 ; </pre> <p>This query will find the posters who talk the most about sex.</p> <pre>sparql select ?auth count (*) where { ?d dc:creator ?auth . ?d ?p ?o filter (bif:contains (?o, "sex")) } group by ?auth order by desc 2 ; </pre> <h3>Analytics </h3> <p>We look for people who are joined by having relatively uncommon interests but do not know each other.</p> <pre>sparql select ?i ?cnt ?n1 ?n2 ?p1 ?p2 where { { select ?i count (*) as ?cnt where { ?p foaf:interest ?i } group by ?i } filter ( ?cnt > 1 && ?cnt < 10) . ?p1 foaf:interest ?i . ?p2 foaf:interest ?i . filter (?p1 != ?p2 && !bif:exists ((select (1) where {?p1 foaf:knows ?p2 })) && !bif:exists ((select (1) where {?p2 foaf:knows ?p1 }))) . ?p1 foaf:nick ?n1 . ?p2 foaf:nick ?n2 . } order by ?cnt limit 50 ; </pre> <p>The query takes a fairly long time, mostly spent counting the interested in 25M interest triples. It then takes people that share the interest and checks that neither claims to know the other. It then sorts the results rarest interest first. The query can be written more efficently but is here just to show that database-wide scans of the population are possible ad hoc. </p> <p>Now we go to SQL to make a tag co-occurrence matrix. This can be used for showing a Technorati-style related tags line at the bottom of a search result page. This showcases the use of SQL together with SPARQL. The half-matrix of tags t1, t2 with the co-occurrence count at the intersection is much more efficiently done in SQL, specially since it gets updated as the data changes. This is an example of materialized intermediate results based on warehoused RDF. </p> <pre>create table tag_count (tcn_tag iri_id_8, tcn_count int, primary key (tcn_tag)); alter index tag_count on tag_count partition (tcn_tag int (0hexffff00)); create table tag_coincidence (tc_t1 iri_id_8, tc_t2 iri_id_8, tc_count int, tc_t1_count int, tc_t2_count int, primary key (tc_t1, tc_t2)) alter index tag_coincidence on tag_coincidence partition (tc_t1 int (0hexffff00)); create index tc2 on tag_coincidence (tc_t2, tc_t1) partition (tc_t2 int (0hexffff00)); </pre> <p>How many times each topic is mentioned?</p> <pre> insert into tag_count select * from (sparql define output:valmode "LONG" select ?t count (*) as ?cnt where { ?s sioc:topic ?t } group by ?t) xx option (quietcast); </pre> <p>Take all t1, t2 where t1 and t2 are tags of the same subject, store only the permutation where the internal id of t1 < that of t2.</p> <pre>insert into tag_coincidence (tc_t1, tc_t2, tc_count) select "t1", "t2", cnt from (select "t1", "t2", count (*) as cnt from (sparql define output:valmode "LONG" select ?t1 ?t2 where { ?s sioc:topic ?t1 . ?s sioc:topic ?t2 }) tags where "t1" < "t2" group by "t1", "t2") xx where isiri_id ("t1") and isiri_id ("t2") option (quietcast); </pre> <p>Now put the individual occurrence counts into the same table with the co-occurrence. This denormalization makes the related tags lookup faster. </p> <pre>update tag_coincidence set tc_t1_count = (select tcn_count from tag_count where tcn_tag = tc_t1), tc_t2_count = (select tcn_count from tag_count where tcn_tag = tc_t2); </pre> <p>Now each tag_coincidence row has the joint occurrence count and individual occurrence counts. A single select will return a Technorati-style related tags listing. </p> <p>To show the <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x9d4bc60">URI</a>'s of the tags: </p> <pre>select top 10 id_to_iri (tc_T1), id_to_iri (tc_t2), tc_count from tag_coincidence order by tc_count desc; </pre> <h3>Social Networks </h3> <p>We look at what interests people have </p> <pre>sparql select ?o ?cnt where { { select ?o count (*) as ?cnt where { ?s foaf:interest ?o } group by ?o } filter (?cnt > 100) } order by desc 2 limit 100 ; </pre> <p>Now the same for the Harry Potter fans </p> <pre>sparql select ?i2 count (*) where { ?p foaf:interest <<a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0xba0b390">http</a>://www.livejournal.com/interests.bml?int=harry+potter> . ?p foaf:interest ?i2 } group by ?i2 order by desc 2 limit 20 ; </pre> <p>We see whether knows relations are symmmetrical. We return the top n people that others claim to know without being reciprocally known.</p> <pre>sparql select ?celeb, count (*) where { ?claimant foaf:knows ?celeb . filter (!bif:exists ((select (1) where { ?celeb foaf:knows ?claimant }))) } group by ?celeb order by desc 2 limit 10 ; </pre> <p>We look for a well connected person to start from.</p> <pre>sparql select ?p count (*) where { ?p foaf:knows ?k } group by ?p order by desc 2 limit 50 ; </pre> <p>We look for the most connected of the many online identities of Stefan Decker.</p> <pre>sparql select ?sd count (distinct ?xx) where { ?sd a foaf:Person . ?sd ?name ?ns . filter (bif:contains (?ns, "'Stefan Decker'")) . ?sd foaf:knows ?xx } group by ?sd order by desc 2 ; </pre> <p>We count the transitive closure of Stefan Decker's connections </p> <pre>sparql select count (*) where { { select * where { ?s foaf:knows ?o } } option (transitive, t_distinct, t_in(?s), t_out(?o)) . filter (?s = <mailto:stefan.decker@deri.org>) } ; </pre> <p>Now we do the same while following owl:sameAs links.</p> <pre>sparql define input:same-as "yes" select count (*) where { { select * where { ?s foaf:knows ?o } } option (transitive, t_distinct, t_in(?s), t_out(?o)) . filter (?s = <mailto:stefan.decker@deri.org>) } ; </pre> <h2>Demo System</h2> <p>The system runs on Virtuoso 6 Cluster Edition. The database is partitioned into 12 partitions, each served by a distinct server process. The system demonstrated hosts these 12 servers on 2 machines, each with 2 xXeon 5345 and 16GB memory and 4 SATA disks. For scaling, the processes and corresponding partitions can be spread over a larger number of machines. If each ran on its own server with 16GB RAM, the whole data set could be served from memory. This is desirable for search engine or fast analytics applications. Most of the demonstrated queries run in memory on second invocation. The timing difference between first and second run is easily an order of magnitude. </p> </div>
Virtuso Data Space Bot
kidehen@openlinksw.com
2008-10-03T06:20:48.94000-04:00
Requirements for Relational-to-RDF Mapping
http://virtuoso.openlinksw.com/blog/vdb/blog/?id=1436
2008-09-08T09:41:25Z
<div> <div style="display:none;">Requirements for Relational-to-RDF Mapping</div> <p>Many of you will know about the W3C relational-to-<a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1e1be0a8">RDF</a> mapping incubator activity. The group is planning to suggest forming a working group for drawing up a specification for relational-to-RDF mapping.</p> <p>To this effect, I recently summarized the group discussions and some of our own experiences around the topic at <<a href="http://esw.w3.org/topic/Rdb2RdfXG/ReqForMappingByOErling" id="link-id146030e8">http://esw.w3.org/topic/Rdb2RdfXG/ReqForMappingByOErling</a>>.</p> <p>I will here discuss this less formally and more in the light of our own experience. A working group goal statement must be neutral vis à vis the following points, even if any working group will unavoidably encounter these issues on the way. A <a href="http://dbpedia.org/resource/Blog" id="link-id0x1e6b3950">blog</a> post on the other hand can be more specific.</p> <p>I gave a talk to the <a href="http://www.w3.org/2005/Incubator/rdb2rdf/" id="link-id0xa0932c68">RDB2RDF XG</a> this spring, with these <a href="http://virtuoso.openlinksw.com/wiki/main/Main/VirtPresentations/Relational2RDF.ppt" id="link-id14572540">slides</a>.</p> <p>The main point is that people would really like to map on-the-fly, if they only could. Making an RDF warehouse is not of value in itself, but it is true that in some cases this cannot be avoided.</p> <p>At first sight, one would think that a mapping specification could be neutral as regards whether one stores the mapped triples as triples or makes them on demand. There is almost no comparison between the complexity of doing non-trivial mappings on-the-fly versus mapping as ETL. Some of this complexity spills over into the requirements for a mapping language. </p> <h2>Eliminating JOINs</h2> <p>We expect to have a situation where one virtual triple can have many possible sources. The mapping is a union of mapped databases. Any integration scenario will have this feature. In such a situation, if we are <code>JOIN</code>ing using such triples, we end up with <code>UNION</code>s of all databases that could produce the triples in question. This is generally not desired. Therefore, in the on-demand mapping case, there must be a lot of type inference logic that is not relevant in the ETL scenario.</p> <p>To make the point clearer, suppose a query like "list the organizations whose representatives have published about <i>xx</i>." Suppose that there are three databases mapped, all of which have a table of organizations, a table of persons with affiliation to organizations, a table of publications by these persons, and finally a table of tags for the publications. Now, we want the laboratories that have published with articles with <a href="http://dbpedia.org/resource/Tag" id="link-id0xa0977bf0">tag</a> <i>XX</i>. It is a matter of common sense in this scenario that a publication will have the author and the author's affiliation in the same database. However, the RDB-to-RDF mapping does not necessarily know this, if all that it is told is that a table makes IRIs of publications by applying a certain pattern to the primary key of the publications table. To infer what needs to be inferred, the system must realize that IRIs from one mapping are disjoint from IRIs from another: A paper in database <i>X</i> will usually not have an author in database <i>Y</i>. The IDs in database <i>Y</i>, even if perchance equal to the IDs in <i>X</i>, do not mean the same thing, and there is no point joining across databases by them.</p> <p>This entire question is a non-issue in the ETL scenario, but is absolutely vital in the real-time mapping. This is also something that must be stated, at least implicitly, in any mapping. If a mapping translates keys of one place to IRIs with one pattern, and keys from another using another pattern, it must be inferable from the patterns whether the sets of IRIs will be disjoint.</p> <p>This is critical. Otherwise we will be joining everything to everything else, and there will be orders of magnitude of penalty compared to hand-crafted <a href="http://dbpedia.org/resource/SQL" id="link-id0xa09490f8">SQL</a> over the same <a href="http://dbpedia.org/resource/Data" id="link-id0xa095efd0">data</a> sources.</p> <h2>Expectations and Limitations on Queries</h2> <p> <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1e360230">SPARQL</a> queries translate quite well to SQL when there is only one table that can produce a triple with a subject of a given class, when there are few columns that can map to a given predicate, and when classes and predicates are literals in the query.</p> <p> <a href="http://virtuoso.openlinksw.com" id="link-id0x1f5edb30">Virtuoso</a> has some SQL extensions for dealing with breaking a wide table into a row per column. This facilitates dealing with predicates that are not known at query compile time. If the table in question is not managed by Virtuoso, Virtuoso's SQL virtualization/federation takes care of the matter. If a mapping system goes directly to third-party SQL, no such tricks can be used.</p> <p>The above example suggests that for supporting on-the-fly mapping without relying on owning the SQL underneath, some subsets of SPARQL may have to be defined. For example, one will probably have to require that all predicates be literals. The alternative is prohibitive run-time cost and complexity.</p> <p>But we must not lose the baby with the bath-water. Aside from offering global identifiers, RDF's attractions include subclasses and sub-predicates. In relational terms, these translate to <code>UNION</code>s and do involve some added cost. A mapping system just has to have means of dealing with this cost, and of recognizing cases where this cost is prohibitive. Some further work is likely to be required for defining well-behaved subsets of SPARQL and mappings.</p> <h2>ETL Ou Ne Pas ETL?</h2> <p>Whether to warehouse or not? If one has hundreds of sources, of which some are not even relational, some ETL would seem necessary. Kashiup Vipul gave a position paper at last year's RDB-to-RDF mapping workshop in Cambridge, Massachusetts, about a system of relational mapping and on-demand RDF-izers of diverse semi-structured biomedical data, e.g., spreadsheets. The issue certainly exists, and any mapping work will likely encounter integration scenarios where one part is fairly neatly mapped from relational stores, and another part comes from a less structured repository of ETLed physical triples.</p> <p>Our take is that if something is a large or very large relational store, then map; else, ETL. With Virtuoso, we can mix mapped and local triples, but this is not a generally available feature of triple stores and standardization will likely have to wait until there are more implementations.</p> <h2>Conclusions</h2> <ul> <li>If you map on demand, watch out for an explosion of <code>UNION</code>s when integrating sources that talk of similar things.</li> <li>If you integrate lots of sources, some ETL is likely unavoidable. Look for ways of dealing with part ETL, part mapping. ETLing everything is not always best or even possible.</li> <li>If you map a single fairly-clean RDB to RDF, mapping will work well, potentially much faster than triple storage. Higher storage density and more data per index lookup on the relational side.</li> <li>If you map on demand, some restrictions to SPARQL may be practically necessary. These have to do with variables in predicate position, variables in class position, etc. Individual implementations may support these, but standardization will likely have to put limits on them.</li> </ul> <p>This was a quick summary, by no means comprehensive, on what an eventual RDB2RDF working group would come across. This is a sort of addendum to the requirements I outlined on the ESW wiki.</p> </div>
Virtuso Data Space Bot
kidehen@openlinksw.com
2008-09-08T15:03:09.000-04:00
Transitivity and Graphs for SQL
http://virtuoso.openlinksw.com/blog/vdb/blog/?id=1435
2008-09-08T09:41:24Z
<div> <div style="display:none;">Transitivity and Graphs for SQL</div> <h2>Background</h2> <p>I have mentioned on a couple of prior occasions that basic graph operations ought to be integrated into the <a href="http://dbpedia.org/resource/SQL" id="link-id0xa1a18c58">SQL</a> query language.</p> <p>The history of databases is by and large about moving from specialized applications toward a generic platform. The introduction of the DBMS itself is the archetypal example. It is all about extracting the common features of applications and making these the features of a platform instead.</p> <p>It is now time to apply this principle to graph traversal.</p> <p>The rationale is that graph operations are somewhat tedious to write in a parallelize-able, latency-tolerant manner. Writing them as one would for memory-based <a href="http://dbpedia.org/resource/Data" id="link-id0xaf8c730">data</a> structures is easier but totally unscalable as soon as there is any latency involved, i.e., disk reads or messages between cluster peers.</p> <p>The ad-hoc nature and very large volume of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xae41ef0">RDF</a> data makes this a timely question. Up until now, the answer to this question has been to materialize any implied facts in RDF stores. If <i>a</i> was part of <i>b</i>, and <i>b</i> part of <i><a href="http://dbpedia.org/resource/C_(programming_language)" id="link-id0xac9d8790">c</a></i>, the implied fact that <i>a</i> is part of <i>c</i> would be inserted explicitly into the database as a pre-query step.</p> <p>This is simple and often efficient, but tends to have the downside that one makes a specialized warehouse for each new type of query. The activity becomes less ad-hoc.</p> <p>Also, this becomes next to impossible when the scale approaches web scale, or if some of the data is liable to be on-and-off included-into or excluded-from the set being analyzed. This is why with <a href="http://virtuoso.openlinksw.com" id="link-id0xb68f9d0">Virtuoso</a> we have tended to favor inference on demand ("backward chaining") and mapping of relational data into RDF without copying.</p> <p>The SQL world has taken steps towards dealing with recursion with the <code>WITH - UNION</code> construct which allows definition of recursive views. The idea there is to define, for example, a tree walk as a <code>UNION</code> of the data of the starting node plus the recursive walk of the starting node's immediate children.</p> <p>The main problem with this is that I do not very well see how a SQL optimizer could effectively rearrange queries involving <code>JOIN</code>s between such recursive views. This model of recursion seems to lose SQL's non-procedural nature. One can no longer easily rearrange <code>JOIN</code>s based on what data is given and what is to be retrieved. If the recursion is written from root to leaf, it is not obvious how to do this from leaf to root. At any rate, queries written in this way are so complex to write, let alone optimize, that I decided to take another approach.</p> <p>Take a question like "list the parts of products of category <i>C</i> which have materials that are classified as toxic." Suppose that the product categories are a tree, the product parts are a tree, and the materials classification is a tree taxonomy where "toxic" has a multilevel substructure.</p> <p>Depending on the count of products and materials, the query can be evaluated as either going from products to parts to materials and then climbing up the materials tree to see if the material is toxic. Or one could do it in reverse, starting with the different toxic materials, looking up the parts containing these, going to the part tree to the product, and up the product hierarchy to see if the product is in the right category. One should be able to evaluate the identical query either way depending on what indices exist, what the cardinalities of the relations are, and so forth — regular cost based optimization.</p> <p>Especially with RDF, there are many problems of this type. In regular SQL, it is a long-standing cultural practice to flatten hierarchies, but this is not the case with RDF.</p> <p>In Virtuoso, we see <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xb3bdcc0">SPARQL</a> as reducing to SQL. Any RDF-oriented database-engine or query-optimization feature is accessed via SQL. Thus, if we address run-time-recursion in the Virtuoso query engine, this becomes, <i>ipso facto</i>, an SQL feature. Besides, we remember that SQL is a much more mature and expressive language than the current SPARQL recommendation.</p> <h2> SQL and Transitivity </h2> <p>We will here look at some simple social network queries. A later article will show how to do more general graph operations. We extend the SQL derived table construct, i.e., <code>SELECT</code> in another <code>SELECT</code>'s <code>FROM</code> clause, with a <code>TRANSITIVE</code> clause.</p> <p>Consider the data:</p> <blockquote> <pre><code>CREATE TABLE "knows" ("p1" INT, "p2" INT, PRIMARY KEY ("p1", "p2") ); ALTER INDEX "knows" ON "knows" PARTITION ("p1" INT); CREATE INDEX "knows2" ON "knows" ("p2", "p1") PARTITION ("p2" INT); </code> </pre></blockquote> <p>We represent a social network with the many-to-many relation "knows". The persons are identified by integers.</p> <blockquote> <pre><code>INSERT INTO "knows" VALUES (1, 2); INSERT INTO "knows" VALUES (1, 3); INSERT INTO "knows" VALUES (2, 4);</code> </pre> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2" FROM "knows" ) "k" WHERE "k"."p1" = 1;</code></pre></blockquote> <p>We obtain the result:</p> <blockquote> <table width="100"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> </tr> <tr> <td align="center">1</td> <td align="center">3</td> </tr> <tr> <td align="center">1</td> <td align="center">2</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> </tr> </table> </blockquote> <p>The operation is reversible:</p> <blockquote> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2" FROM "knows" ) "k" WHERE "k"."p2" = 4; </code> </pre> <table width="100"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> </tr> <tr> <td align="center">2</td> <td align="center">4</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> </tr> </table> </blockquote> <p>Since now we give <i>p2</i>, we traverse from <i>p2</i> towards <i>p1</i>. The result set states that 4 is known by 2 and 2 is known by 1.</p> <p>To see what would happen if <i>x</i> knowing <i>y</i> also meant <i>y</i> knowing <i>x</i>, one could write:</p> <blockquote> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2" FROM (SELECT "p1", "p2" FROM "knows" UNION ALL SELECT "p2", "p1" FROM "knows" ) "k2" ) "k" WHERE "k"."p2" = 4;</code> </pre> <table width="100"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> </tr> <tr> <td align="center">2</td> <td align="center">4</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> </tr> <tr> <td align="center">3</td> <td align="center">4</td> </tr> </table> </blockquote> <p>Now, since we know that 1 and 4 are related, we can ask how they are related.</p> <blockquote> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2", T_STEP (1) AS "via", T_STEP ('step_no') AS "step", T_STEP ('path_id') AS "path" FROM "knows" ) "k" WHERE "p1" = 1 AND "p2" = 4;</code> </pre> <table width="250"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> <th align="center" width="50">via</th> <th align="center" width="50">step</th> <th align="center" width="50">path</th> </tr> <tr> <td align="center">1</td> <td align="center">4</td> <td align="center">1</td> <td align="center">0</td> <td align="center">0</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> <td align="center">2</td> <td align="center">1</td> <td align="center">0</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> <td align="center">4</td> <td align="center">2</td> <td align="center">0</td> </tr> </table> </blockquote> <p>The two first columns are the ends of the path. The next column is the person that is a step on the path. The next one is the number of the step, counting from 0, so that the end of the path that corresponds to the end condition on the column designated as input, i.e., <i>p1</i>, has number 0. Since there can be multiple solutions, the last column is a sequence number allowing distinguishing multiple alternative paths from each other.</p> <p>For LinkedIn users, the friends ordered by distance and descending friend count query, which is at the basis of most LinkedIn search result views can be written as: </p> <blockquote> <pre><code>SELECT p2, dist, (SELECT COUNT (*) FROM "knows" "c" WHERE "c"."p1" = "k"."p2" ) FROM (SELECT TRANSITIVE t_in (1) t_out (2) t_distinct "p1", "p2", t_step ('step_no') AS "dist" FROM "knows" ) "k" WHERE "p1" = 1 ORDER BY "dist", 3 DESC;</code> </pre> <table width="150"> <tr> <th align="center" width="50">p2</th> <th align="center" width="50">dist</th> <th align="center" width="50">aggregate</th> </tr> <tr> <td align="center">2</td> <td align="center">1</td> <td align="center">1</td> </tr> <tr> <td align="center">3</td> <td align="center">1</td> <td align="center">0</td> </tr> <tr> <td align="center">4</td> <td align="center">2</td> <td align="center">0</td> </tr> </table> </blockquote> <h2>How?</h2> <p>The queries shown above work on Virtuoso v6. When running in cluster mode, several thousand graph traversal steps may be proceeding at the same time, meaning that all database access is parallelized and that the algorithm is internally latency-tolerant. By default, all results are produced in a deterministic order, permitting predictable slicing of result sets.</p> <p>Furthermore, for queries where both ends of a path are given, the optimizer may decide to attack the path from both ends simultaneously. So, supposing that every member of a social network has an average of 30 contacts, and we need to find a path between two users that are no more than 6 steps apart, we begin at both ends, expanding each up to 3 levels, and we stop when we find the first intersection. Thus, we reach 2 * 30^3 = 54,000 nodes, and not 30^6 = 729,000,000 nodes.</p> <p>Writing a generic database driven graph traversal framework on the application side, say in Java over <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0xa8a9ef8">JDBC</a>, would easily be over a thousand lines. This is much more work than can be justified just for a one-off, ad-hoc query. Besides, the traversal order in such a case could not be optimized by the DBMS.</p> <h2>Next</h2> <p>In a future <a href="http://dbpedia.org/resource/Blog" id="link-id0xb526a40">blog</a> post I will show how this feature can be used for common graph tasks like critical path, itinerary planning, traveling salesman, the 8 queens chess problem, etc. There are lots of switches for controlling different parameters of the traversal. This is just the beginning. I will also give examples of the use of this in SPARQL.</p> </div>
Virtuso Data Space Bot
kidehen@openlinksw.com
2008-09-08T15:43:07.000-04:00
Epistemology of the Sponger, or How Virtuoso Drives a Web Query
http://virtuoso.openlinksw.com/blog/vdb/blog/?id=1432
2008-09-05T09:20:56Z
<div> <div style="display:none;">Epistemology of the Sponger, or How Virtuoso Drives a Web Query</div> <p> <a href="http://virtuoso.openlinksw.com" id="link-id0x1ed6cf28">Virtuoso</a> has an extensive collection of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1f8d1f78">RDF</a>-izers called Sponger Cartridges. These take a web resource in one of 30+ formats (so far) and extract RDF from it. The Virtuoso <a href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id0x1edc90e8">Sponger</a> is a device which evaluates a query and along the way, finds dereferenceable links, dereferences them, and iteratively re-evaluates the query, until either nothing new is found or some limit is reached.</p> <p>We could call this <i>query-driven crawling</i>. The idea is intuitive — what one looks for, determines what one finds.</p> <p>This does however raise certain questions pertaining to the nature and ultimate possibility of <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x1f836b68">knowledge</a>, i.e., epistemology.</p> <p>The process of querying could be said to go from the few to the many, just like the process of harvesting <a href="http://dbpedia.org/resource/Data" id="link-id0x1edb1648">data</a> from the web, the way any search engine does. One follows links or makes joins and thereby increases one's reach.</p> <p>The difference is that a query has no <i>a priori</i> direction. If I ask for the phone numbers of my friends and there are no phone numbers in the database, then it is valid to give an empty result without looking at my friends at all. <a href="http://dbpedia.org/resource/Closed_world_assumption" id="link-id0x1edf1f30">Closed world</a>, as it is said. Never mind that the friends would have had a "see also" link to a retrievable document that did have a phone number.</p> <p>The problem is that a query execution plan determines what possible dereferenceable material the query will encounter during its execution. What is worse, a query plan tends toward the minimal, i.e., toward minimizing the chances of encountering something dereferenceable along the way. Where query and crawl appeared to have a similarity, in fact they have two opposite goals.</p> <p>The user generally has no idea of the execution plan. In the general case, the user <i>cannot</i> have an idea of this plan. There are valid, over 40 year old reasons for leaving the query planning to the database. In exceptional situations the user can read or direct these, but this is really quite tedious and requires understanding that is basically never present.</p> <p>So, given a query, how do we find data that will match it, short of having a pre-loaded database of absolutely everything? This is certainly a desirable goal, and all in the <a href="http://dbpedia.org/resource/Open_world_assumption" id="link-id0x1eb46548">open world</a>, distributed spirit of the web.</p> <p>Let us limit ourselves to queries that have some literals in the object or subject positions. A <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1ed293f8">SPARQL</a> query is basically a graph. Its vertices are variables and literals, and its edges are triple patterns. An edge is labeled by a predicate. For now, we will consider the predicate to always be a literal. From each literal, we can draw a tree, following each edge starting at this literal and descending until we find another literal. Each tree is not always a spanning tree of the graph, but all the trees collectively span the graph.</p> <p>Consider the query </p> <blockquote> <code>{ <john> knows ?x . <mary> knows ?x . ?x label ?l }.</code> </blockquote> The starting points are the literals <code>john</code> and <code>mary</code>. The <code>john</code> tree has one child, <code>?x</code>, which has the children <code>mary</code> and <code>?l</code>. One could notate it as <blockquote> <code>{ <john> knows ?x . {{ <mary> knows ?x} UNION {?x label ?l}}}</code> </blockquote> That is, the head first, and if it has more than one child, a union listing them, recursively. <p>If one composed such queries for each literal in the original pattern and evaluated each as a breadth first walk of the tree, no query optimization tricks, and for each binding of each variable, recorded whether there was something to dereference, one would in a finite time have reached all the directly reachable data. Then one could evaluate the original query, using whatever plan was preferred.</p> <p>The check for dereferenceable data applied to each IRI-valued binding formed in the above evaluation, would consist of looking for "see also", "same as", and other such properties of the IRI. It could also consult text based search engines. Since the evaluation is breadth first, it generates a large number of parallel tasks and is fairly latency tolerant, i.e., it will not die if it must retrieve a few pages from remote sources. We will leave the exact rewrite rules for unions, optionals, aggregates, subqueries, and so on, as an exercise; the general idea should be clear enough.</p> <p>We have here shown a way of transforming SPARQL queries in such a way as to guarantee dereferencing of findable links, without requiring the end user to either explicitly specify or understand query plans.</p> <p>The present Sponger does not work exactly in this manner but it will be developed in this direction. Fortunately, the algorithms outlined above are nothing complicated.</p> </div>
Virtuso Data Space Bot
kidehen@openlinksw.com
2008-09-05T16:04:28.000-04:00
A quick look at SP2B, the SPARQL Performance Benchmark
http://virtuoso.openlinksw.com/blog/vdb/blog/?id=1423
2008-08-27T16:03:40Z
<div> <div style="display:none;">A quick look at SP2B, the SPARQL Performance Benchmark</div> <p>I finally got around to running the <a href="http://dbis.informatik.uni-freiburg.de/index.php?project=SP2B" id="link-id17bac628">SP<sup>2</sup>B SPARQL Performance Benchmark</a> on the current <a href="http://virtuoso.openlinksw.com" id="link-id0x1dcaaa48">Virtuoso</a> Open Source Edition, v5.0.8.</p> <p>I ran it with the 5M triples scale, which is the highest scale for which the authors give numbers.</p> <p>I got a run time of 25 minutes for the 12 queries, giving an arithmetic mean of the query time of 125 seconds. This is better than the 800 or so seconds that the authors had measured. Also, Q6 of the set had failed for the authors, but we have since fixed this; the fix is in the v5.0.8 cut.</p> <p>I also tried it with a scale of 25M, but this became I/O bound and took a bit longer. I will try this with v6 and v7 cluster later, which are vastly better at anything I/O bound.</p> <p>The machine was a 2GHz Xeon with 8G RAM. The query text was the one from the authors, with an explicit <code>FROM</code> clause added; the client was the command line Interactive <a href="http://dbpedia.org/resource/SQL" id="link-id0x1be2c808">SQL</a> (iSQL).</p> <p>If one does the test with the default index layout without specifying a graph, things will not work very well. Also, returning the million-row results of these queries over the <a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id0x1d7ac018">SPARQL protocol</a> is not practical.</p> <p>I will say something more about SP<sup>2</sup>B when I get to have a closer look.</p> </div>
Virtuso Data Space Bot
kidehen@openlinksw.com
2008-09-02T09:49:57.000-04:00
Configuring Virtuoso for Benchmarking
http://virtuoso.openlinksw.com/blog/vdb/blog/?id=1419
2008-08-25T14:06:11Z
<div> <div style="display:none;">Configuring Virtuoso for Benchmarking</div> <p>I will here summarize what should be known about running benchmarks with <a href="http://virtuoso.openlinksw.com" id="link-id0xc152cf0">Virtuoso</a>.</p> <h2>Physical Memory</h2> <p>For 8G RAM, in the <code>[Parameters]</code> stanza of <code>virtuoso.ini</code>, set —</p> <blockquote> <code> [Parameters]<br /> ...<br /> NumberOfBuffers = 550000 </code> </blockquote> <p>For 16G RAM, double this—</p> <blockquote> <code> [Parameters]<br /> ...<br /> NumberOfBuffers = 1100000 </code> </blockquote> <h2>Transaction Isolation</h2> <p>For most cases, certainly all <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xb7ba270">RDF</a> cases, <i>Read Committed</i> should be the default transaction isolation. In the <code>[Parameters]</code> stanza of <code>virtuoso.ini</code>, set —</p> <blockquote> <code> [Parameters]<br /> ...<br /> DefaultIsolation = 2 </code> </blockquote> <h2>Multiuser Workload</h2> <p>If <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x1a40f308">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x1e003cf8">JDBC</a>, or similarly connected client applications are used, there must be more <code>ServerThreads</code> available than there will be client connections. In the <code>[Parameters]</code> stanza of <code>virtuoso.ini</code>, set —</p> <blockquote> <code> [Parameters]<br /> ...<br /> ServerThreads = 100 </code> </blockquote> <p>With web clients (unlike ODBC, JDBC, or similar clients), it may be justified to have fewer <code>ServerThreads</code> than there are concurrent clients. The <code>MaxKeepAlives</code> should be the maximum number of expected web clients. This can be more than the <code>ServerThreads</code> count. In the <code>[HTTPServer]</code> stanza of <code>virtuoso.ini</code>, set —</p> <blockquote> <code> [HTTPServer]<br /> ...<br /> ServerThreads = 100 <br /> MaxKeepAlives = 1000 <br /> KeepAliveTimeout = 10 </code> </blockquote> <p> <i><b>Note</b> — The <code>[HTTPServer] ServerThreads</code> are taken from the total pool made available by the <code>[Parameters] ServerThreads</code>. Thus, the <code>[Parameters] ServerThreads</code> should always be at least as large as (and is best set greater than) the <code>[HTTPServer] ServerThreads</code>, and if using the closed-source Commercial Version, should not exceed the licensed thread count.</i> </p> <h2>Disk Use</h2> <p>The basic rule is to use one stripe (file) per distinct physical device (not per file system), using no RAID. For example, one might stripe a database over 6 files (6 physical disks), with an initial size of 60000 pages (the files will grow as needed). </p> <p>For the above described example, in the <code>[Database]</code> stanza of <code>virtuoso.ini</code>, set —</p> <blockquote> <code> [Database]<br /> ...<br /> Striping = 1<br /> MaxCheckpointRemap = 2000000 </code> </blockquote> <p>— and in the <code>[Striping]</code> stanza, on one line per <code>SegmentName</code>, set —</p> <blockquote> <code> [Striping]<br /> ...<br /> Segment1 = 60000 , /virtdev/db/virt-seg1.db = q1 , /data1/db/virt-seg1-str2.db = q2 , /data2/db/virt-seg1-str3.db = q3 , /data3/db/virt-seg1-str4.db = q4 , /data4/db/virt-seg1-str5.db = q5 , /data5/db/virt-seg1-str6.db = q6</code> </blockquote> <p>As can be seen here, each file gets a background IO thread (the <code>= q<i>xxx</i></code> clause). It should be noted that all files on the same physical device should have the same <code>q<i>xxx</i></code> value. This is not directly relevant to the benchmarking scenario above, because we have only one file per device, and thus only one file per IO queue.</p> <h2> <a href="http://dbpedia.org/resource/SQL" id="link-id0xc8b97c0">SQL</a> Optimization</h2> <p>If queries have lots of joins but access little <a href="http://dbpedia.org/resource/Data" id="link-id0x193b2fa8">data</a>, as with the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1b283ca0">Berlin SPARQL Benchmark</a>, the SQL compiler must be told not to look for better plans if the best plan so far is quicker than the compilation time expended so far. Thus, in the <code>[Parameters]</code> stanza of <code>virtuoso.ini</code>, set —</p> <blockquote> <code> [Parameters]<br /> ...<br /> StopCompilerWhenXOverRunTime = 1 </code> </blockquote> </div>
Virtuso Data Space Bot
kidehen@openlinksw.com
2008-08-25T15:29:06.36000-04:00
BSBM With Triples and Mapped Relational Data
http://virtuoso.openlinksw.com/blog/vdb/blog/?id=1410
2008-08-06T19:41:50Z
<div> <div style="display:none;">BSBM With Triples and Mapped Relational Data</div> <p>The special contribution of the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id10039db0">Berlin SPARQL Benchmark</a> (<a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id106b2538">BSBM</a>) to the <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id101a75f8">RDF</a> world is to raise the question of doing OLTP with <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xae54170">RDF</a>.</p> <p>Of course, here we immediately hit the question of comparisons with relational databases. To this effect, <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1e847b08">BSBM</a> also specifies a relational schema and can generate the <a href="http://dbpedia.org/resource/Data" id="link-id1206c378">data</a> as either triples or <a href="http://dbpedia.org/resource/SQL" id="link-id1667f040">SQL</a> inserts.</p> <p>The benchmark effectively simulates the case of exposing an existing <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id10a93518">RDBMS</a> as RDF. <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id13e46d80">OpenLink Software</a> calls this <i>RDF Views</i>. <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id12027578">Oracle</a> is beginning to call this <i>semantic covers</i>. The <a href="http://www.w3.org/2005/Incubator/rdb2rdf/" id="link-id161dc678">RDB2RDF XG</a>, a W3C incubator group, has been active in this area since Spring, 2008.</p> <h3>But why an OLTP workload with RDF to begin with?</h3> <p>We believe this is relevant because RDF promises to be the interoperability factor between potentially all of traditional IS. If <a href="http://dbpedia.org/resource/Data" id="link-id0x1e7119d8">data</a> is online for human consumption, it may be online via a <a href="http://dbpedia.org/resource/SPARQL" id="link-id106a8908">SPARQL</a> end-point as well. The economic justification will come from discoverability and from applications integrating multi-source structured data. Online shopping is a fine use case.</p> <p>Warehousing all the world's publishable data as RDF is not our first preference, nor would it be the publisher's. Considerations of duplicate infrastructure and maintenance are reason enough. Consequently, we need to show that mapping can outperform an RDF warehouse, which is what we'll do here.</p> <h3>What We Got </h3> <p>First, we found that <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1400" id="link-id150ea748">making the query plan took much too long</a> in proportion to the run time. With BSBM this is an issue because the queries have lots of joins but access relatively little data. So we made a faster compiler and along the way retouched the cost model a bit.</p> <p>But the really interesting part with BSBM is mapping relational data to RDF. For us, BSBM is a great way of showing that mapping can outperform even the best triple store. A relational row store is as good as unbeatable with the query mix. And when there is a clear mapping, there is no reason the <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xae5aff0">SPARQL</a> could not be directly translated.</p> <p>If Chris Bizer et al launched the mapping ship, we will be the ones to pilot it to harbor!</p> <p>We filled two <a href="http://virtuoso.openlinksw.com" id="link-id12dbdc70">Virtuoso</a> instances with a BSBM200000 data set, for 100M triples. One was filled with physical triples; the other was filled with the equivalent relational data plus mapping to triples. Performance figures are given in "query mixes per hour". (An update or follow-on to this post will provide elapsed times for each test run.)</p> <p>With the unmodified benchmark we got:</p> <blockquote> <table> <tr> <td><i>Physical Triples:</i> </td> <td> </td> <td>1297 qmph</td> </tr> <tr> <td><i>Mapped Triples:</i> </td> <td> </td> <td><b>3144 qmph</b> </td> </tr> </table> </blockquote> <p>In both cases, most of the time was spent on Q6, which looks for products with one of three words in the label. We altered Q6 to use text index for the mapping, and altered the databases accordingly. (There is no such thing as an e-commerce site without a text index, so we are amply justified in making this change.)</p> <p>The following were measured on the second run of a 100 query mix series, single test driver, warm cache.</p> <blockquote> <table> <tr> <td><i>Physical Triples:</i> </td> <td> </td> <td> 5746 qmph</td> </tr> <tr> <td><i>Mapped Triples:</i> </td> <td> </td> <td> <b>7525 qmph</b> </td> </tr> </table> </blockquote> <p>We then ran the same with 4 concurrent instances of the test driver. The qmph here is 400 / the longest run time.</p> <blockquote> <table> <tr> <td><i>Physical Triples:</i> </td> <td> </td> <td> 19459 qmph</td> </tr> <tr> <td><i>Mapped Triples:</i> </td> <td> </td> <td> <b>24531 qmph</b> </td> </tr> </table> </blockquote> <p>The system used was 64-bit Linux, 2GHz dual-Xeon 5130 (8 cores) with 8G RAM. The concurrent throughputs are a little under 4 times the single thread throughput, which is normal for SMP due to memory contention. The numbers do not evidence significant overhead from thread synchronization.</p> <p>The query compilation represents about 1/3 of total server side CPU. In an actual online application of this type, queries would be parameterized, so the throughputs would be accordingly higher. We used the <code>StopCompilerWhenXOverRunTime = 1</code> option here to cut needless compiler overhead, the queries being straightforward enough.</p> <p>We also see that the advantage of mapping can be further increased by more compiler optimizations, so we expect in the end mapping will lead RDF warehousing by a factor of 4 or so.</p> <h3>Suggestions for BSBM</h3> <ul> <li> <p> <b>Reporting Rules.</b> The benchmark spec should specify a form for disclosure of test run data, TPC style. This includes things like configuration parameters and exact text of queries. There should be accepted variants of query text, as with the TPC.</p> </li> <li> <p> <b>Multiuser operation.</b> The test driver should get a stream number as parameter, so that each client makes a different query sequence. Also, disk performance in this type of benchmark can only be reasonably assessed with a naturally parallel multiuser workload.</p> </li> <li> <p> <b>Add business intelligence.</b> SPARQL has aggregates now, at least with <a href="http://jena.sourceforge.net/" id="link-id11a25ac0">Jena</a> and <a href="http://virtuoso.openlinksw.com" id="link-id0xb003180">Virtuoso</a>, so let's use these. The BSBM business intelligence metric should be a separate metric off the same data. Adding synthetic sales figures would make more interesting queries possible. For example, producing recommendations like "customers who bought this also bought xxx."</p> </li> <li> <p> <b>For the SPARQL community</b>, BSBM sends the message that one ought to support parameterized queries and stored procedures. This would be a <a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id109e2448">SPARQL protocol</a> extension; the SPARUL syntax should also have a way of calling a procedure. Something like <code>select proc (??, ??)</code> would be enough, where <code>??</code> is a parameter marker, like <code>?</code> in <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id13febf48">ODBC</a>/<a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id120416a8">JDBC</a>.</p> </li> <li> <p> <b>Add transactions.</b>Especially if we are contrasting mapping vs. storing triples, having an update flow is relevant. In practice, this could be done by having the test driver send web service requests for order entry and the SUT could implement these as updates to the triples or a mapped relational store. This could use stored procedures or logic in an app server.</p> </li> </ul> <h3>Comments on Query Mix</h3> <p>The time of most queries is less than linear to the scale factor. Q6 is an exception if it is not implemented using a text index. Without the text index, Q6 will inevitably come to dominate query time as the scale is increased, and thus will make the benchmark less relevant at larger scales.</p> <h2>Next</h2> <p>We include the sources of our RDF view definitions and other material for running BSBM with our forthcoming Virtuoso Open Source 5.0.8 release. This also includes all the query optimization work done for BSBM. This will be available in the coming days.</p> </div>
Virtuso Data Space Bot
kidehen@openlinksw.com
2008-08-06T16:29:44.3000-04:00
Exploiting the RDF-based Linked Data Web using .NET via LINQ
http://virtuoso.openlinksw.com/blog/vdb/blog/?id=1403
2008-08-01T17:58:19Z
<div> <div style="display:none;">Exploiting the RDF-based Linked Data Web using .NET via LINQ</div> Recently OpenLink has been investigating <a href="http://code.google.com/p/linqtordf/" id="link-id0x20d8a248">LinqToRdf</a>, an exciting project from <a href="http://aabs.wordpress.com" id="link-id0x21f48218">Andrew Matthews</a> which aims to bring the Semantic Web to .NET. Because of their language bindings and heritage, existing RDF APIs such as Sesame, Jena and Redland predominantly favour non-Windows clients. Conversely Microsoft's ADO.NET Data Services provides a Redmond vision of exposing data on the Web but has no support for RDF. LinqToRdf is, as far as we're aware, the first serious effort to fill this gap and provide a bridge between Windows applications and the Semantic Web.<br /> <br />OpenLink has produced a whitepaper <a href="http://virtuoso.openlinksw.com/Whitepapers/html/linqtordf/linqtordf1.htm" id="link-id0x21f47348">Exploiting the RDF-based Linked Data Web using .NET via LINQ</a> which provides a brief overview of LinqToRdf and an example of its use to retrieve data from the <a href="http://musicbrainz.org" id="link-id0x21f49a88">MusicBrainz</a> music metadatabase via an OpenLink Virtuoso Quad Store. The document also illustrates the use of the <a href="http://virtuoso.openlinksw.com/Whitepapers/pdf/sponger_whitepaper_10102007.pdf" id="link-id0x21f92758">Virtuoso Sponger</a>, an "RDFizer" forming part of the RDF toolset provided with OpenLink Virtuoso Universal Server, to convert the raw MusicBrainz data to RDF on-the-fly. A further aim of the whitepaper is to draw attention to Andrew's excellent effort and hopefully tempt members of the Semantic Web community to contribute.<br /> <br />Andrew was kind enough to incorporate some changes into LinqToRdf in response to OpenLink's testing. These have been included with major improvements of his own in a new release - <a href="http://aabs.wordpress.com/2008/08/01/announcing-linqtordf-v08/" id="link-id0x21f465b8">LinqToRdf v0.8</a>.<br /> <br />Carl Blakeley<br /> </div>
Virtuso Data Space Bot
kidehen@openlinksw.com
2008-08-01T13:58:19.4000-04:00
Virtuoso Optimizations for the Berlin SPARQL Benchmark
http://virtuoso.openlinksw.com/blog/vdb/blog/?id=1401
2008-07-30T18:52:11Z
<div> <div style="display:none;">Virtuoso Optimizations for the Berlin SPARQL Benchmark </div> <p>We had a look at Chris Bizer's initial results with the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id105c9f78">Berlin SPARQL Benchmark</a> (<a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id102d62b0">BSBM</a>) on <a href="http://virtuoso.openlinksw.com" id="link-id13eb9780">Virtuoso</a>. The first results were rather bad, as nearly all of the run time was spent optimizing the <a href="http://dbpedia.org/resource/SPARQL" id="link-id14a51258">SPARQL</a> statements and under 10% actually running them.</p> <p>So I spent a couple of days on the <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xaad28d0">SPARQL</a>/<a href="http://dbpedia.org/resource/SQL" id="link-id108745b0">SQL</a> compiler, to the effect of making it do a better guess of initial execution plan and streamlining some operations. In fact, many of the queries in <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0xaa230b8">BSBM</a> are not particularly sensitive to execution plan, as they access a very small portion of the database. So to close the matter, I put in a flag that makes the <a href="http://dbpedia.org/resource/SQL" id="link-id0x1e9e8e28">SQL</a> compiler give up on devising new plans if the time of the best plan so far is less than the time spent compiling so far.</p> <p>With these changes, available now as a diff on top of 5.0.7, we run quite well, several times better than initially. With the compiler time cut-off in place (ini parameter <code>StopCompilerWhenXOverRunTime = 1</code>), we get the following times, output from the BSBM test driver:</p> <blockquote> <pre> Starting test... 0: 1031.22 ms, total: 1151 ms 1: 982.89 ms, total: 1040 ms 2: 923.27 ms, total: 968 ms 3: 898.37 ms, total: 932 ms 4: 855.70 ms, total: 865 ms Scale factor: 10000 Number of query mix runs: 5 times min/max Query mix runtime: 0.8557 s / 1.0312 s Total runtime: 4.691 seconds QMpH: 3836.77 query mixes per hour CQET: 0.93829 seconds average runtime of query mix CQET (geom.): 0.93625 seconds geometric mean runtime of query mix Metrics for Query 1: Count: 5 times executed in whole run AQET: 0.012212 seconds (arithmetic mean) AQET(geom.): 0.009934 seconds (geometric mean) QPS: 81.89 Queries per second minQET/maxQET: 0.00684000s / 0.03115700s Average result count: 7.0 min/max result count: 3 / 10 Metrics for Query 2: Count: 35 times executed in whole run AQET: 0.030490 seconds (arithmetic mean) AQET(geom.): 0.029776 seconds (geometric mean) QPS: 32.80 Queries per second minQET/maxQET: 0.02467300s / 0.06753000s Average result count: 22.5 min/max result count: 15 / 30 Metrics for Query 3: Count: 5 times executed in whole run AQET: 0.006947 seconds (arithmetic mean) AQET(geom.): 0.006905 seconds (geometric mean) QPS: 143.95 Queries per second minQET/maxQET: 0.00580000s / 0.00795100s Average result count: 4.0 min/max result count: 0 / 10 Metrics for Query 4: Count: 5 times executed in whole run AQET: 0.008858 seconds (arithmetic mean) AQET(geom.): 0.008829 seconds (geometric mean) QPS: 112.89 Queries per second minQET/maxQET: 0.00804400s / 0.01019500s Average result count: 3.4 min/max result count: 0 / 10 Metrics for Query 5: Count: 5 times executed in whole run AQET: 0.087542 seconds (arithmetic mean) AQET(geom.): 0.087327 seconds (geometric mean) QPS: 11.42 Queries per second minQET/maxQET: 0.08165600s / 0.09889200s Average result count: 5.0 min/max result count: 5 / 5 Metrics for Query 6: Count: 5 times executed in whole run AQET: 0.131222 seconds (arithmetic mean) AQET(geom.): 0.131216 seconds (geometric mean) QPS: 7.62 Queries per second minQET/maxQET: 0.12924200s / 0.13298200s Average result count: 3.6 min/max result count: 3 / 5 Metrics for Query 7: Count: 20 times executed in whole run AQET: 0.043601 seconds (arithmetic mean) AQET(geom.): 0.040890 seconds (geometric mean) QPS: 22.94 Queries per second minQET/maxQET: 0.01984400s / 0.06012600s Average result count: 26.4 min/max result count: 5 / 96 Metrics for Query 8: Count: 10 times executed in whole run AQET: 0.018168 seconds (arithmetic mean) AQET(geom.): 0.016205 seconds (geometric mean) QPS: 55.04 Queries per second minQET/maxQET: 0.01097600s / 0.05066900s Average result count: 12.8 min/max result count: 6 / 20 Metrics for Query 9: Count: 20 times executed in whole run AQET: 0.043813 seconds (arithmetic mean) AQET(geom.): 0.043807 seconds (geometric mean) QPS: 22.82 Queries per second minQET/maxQET: 0.04274900s / 0.04504100s Average result count: 0.0 min/max result count: 0 / 0 Metrics for Query 10: Count: 15 times executed in whole run AQET: 0.030697 seconds (arithmetic mean) AQET(geom.): 0.029651 seconds (geometric mean) QPS: 32.58 Queries per second minQET/maxQET: 0.02072000s / 0.03975700s Average result count: 1.1 min/max result count: 0 / 4 real 0 m 5.485 s user 0 m 2.233 s sys 0 m 0.170 s </pre></blockquote> <p>Of the approximately 5.5 seconds of running five query mixes, the test driver spends 2.2 s. The server side processing time is 3.1 s, of which SQL compilation is 1.35 s. The rest is miscellaneous system time. The measurement is on 64-bit Linux, 2GHz dual-Xeon 5130 (8 cores) with 8G RAM. </p> <p>We note that this type of workload would be done with stored procedures or prepared, parameterized queries in the SQL world.</p> <p>There will be some further tuning still but this addresses the bulk of the matter. There will be a separate message about the patch containing these improvements.</p> </div>
Virtuso Data Space Bot
kidehen@openlinksw.com
2008-08-06T16:29:42.000-04:00
Virtuoso 5.0.7 Release, Now With Jena and Sesame APIs
http://virtuoso.openlinksw.com/blog/vdb/blog/?id=1393
2008-07-17T17:18:09Z
<div> <div style="display:none;">Virtuoso 5.0.7 Release, Now With Jena and Sesame APIs</div> <h2>Improvements</h2> <ul> <li> <a href="http://docs.openlinksw.com:80/virtuoso/rdfnativestorageproviders.html" id="link-id13e54d98">Full operation</a> with <a href="http://jena.sourceforge.net/" id="link-id0x11a3d360">Jena</a> and <a href="http://sourceforge.net/projects/sesame/" id="link-id0x1108d428">Sesame</a> <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1288aa00">RDF</a> Frameworks. This fully replaces any previous attempts at interop, and introduces samples and test suites.</li> <li>Better support for alternate RDF indexing schemes</li> <li>Parallel operation of the RDF Sponger, importing multiple sources concurrently.</li> <li>New <a href="http://dbpedia.org/resource/Data" id="link-id0x128a9810">data</a> formats supported for on-demand RDF-ization in the Sponger</li> <li>More efficient support for inference of subclass and sub-property; now capable of efficiently handling taxonomies of tens of thousands of classes</li> <li> <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x6af0678">OWL</a> <a href="http://docs.openlinksw.com:80/virtuoso/rdfsparqlrule.html#rdfsparqlruleintro" id="link-id104d58d8">equivalentClass and equivalentProperty</a> support.</li> <li> <a href="http://docs.openlinksw.com:80/virtuoso/rdfdatarepresentation.html#rdfdynamiclocal" id="link-id109606a8">Dynamic IRI host part</a> support for mapped data and for metadata of local resources. Renaming the host or using multiple virtual hosts will accept URIs with the right host part and refer to the same thing, no duplicate storage required.</li> <li> <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x12e0cc38">SPARQL</a> optimizations for <code>LIMIT</code> and <code>OFFSET</code> </li> </ul> <h2>Documentation</h2> <ul> <li> <a href="http://docs.openlinksw.com:80/virtuoso/perfdiag.html#perfdiagqueryplans" id="link-id10a56dd0">How to read query plans and how to use the key performance meters</a> </li> <li> <a href="http://docs.openlinksw.com:80/virtuoso/rdfperformancetuning.html#rdfperfcost" id="link-id106cb5c0">How to diagnose SPARQL queries and how to decide what indexing scheme is right for each RDF use case</a> </li> <li>How to debug RDF views</li> <ul> <li> <a href="http://docs.openlinksw.com:80/virtuoso/sparqldebug.html" id="link-id133b4420">Better documentation of SPARQL extensions and options</a> </li> <li> <a href="http://docs.openlinksw.com:80/virtuoso/rdfviews.html#rdfviewnorthwindexample1" id="link-id1060fdd8">A sample of correct RDF view usage with the Northwind demo data</a> </li> </ul> </ul> <h2>Bug Fixes</h2> <ul> <li>Generally improved safety of built-in functions, better argument checking.</li> <li>Verified UTF8 international character support in all RDF use cases, <a href="http://dbpedia.org/resource/SQL" id="link-id0x12839fd0">SQL</a> client/<a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id0x1288f350">SPARQL protocol</a>/all data formats.</li> </ul> </div>
Virtuso Data Space Bot
kidehen@openlinksw.com
2008-07-17T15:28:22.2000-04:00
De Paradigmata and The Foundational Issues
http://virtuoso.openlinksw.com/blog/vdb/blog/?id=1383
2008-06-09T14:02:21Z
<div> <div style="display:none;">De Paradigmata and The Foundational Issues</div> <p>I thought that we had talked ourselves to exhaustion and beyond over the issue of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x1dd07c68">semantic web</a> layer cake. Apparently not. There was a paper called <i>Functional Architecture for the Semantic Web</i> by <a href="http://gerberaj.googlepages.com/" id="link-id106b8130">Aurona Gerber</a> et al at <a href="http://www.eswc2008.org/" id="link-id0x17137300">ESWC2008</a>.</p> <p>The thrust of the matter was that for newcomers the layer cake was confusing and did not clearly indicate the architecture. Why, sure. My point is that no rearranging of the boxes will cut it for the general case.</p> <p>Any diagram containing the boxes of the layer cake (i.e., <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x1a9138c0">URI</a>, <a href="http://dbpedia.org/resource/XML" id="link-id0x1cc4a8d8">XML</a>, <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xa21c1308">SPARQL</a>, <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x1aa28050">OWL</a>, <a href="http://dbpedia.org/resource/Rule_Interchange_Format" id="link-id0x137268d0">RIF</a>, Crypto, etc., etc.) in whatever order or arrangement can at best be a sort of overview of how these standards reference each other.</p> <p>Such diagrams are a little like saying that a car combines the combustion properties of fuel/air mixes with the tension and compression resistance properties of metals and composites for producing motion and secondly links to Newton's laws of motion and to aerodynamics.</p> <p>Not false. But it does not say that a car is good for economical commute or showing off at the strip or any number of niches that a mature industry has grown to serve.</p> <p>Now, talking of software engineering, modules and interfaces are good and even necessary. The trick is to know where to put the interface.</p> <p>Such a thing cannot possibly be inferred from the standards' inter-reference picture. APIs, especially if these are Web service APIs, should go where there is low <a href="http://dbpedia.org/resource/Data" id="link-id0x196fcba0">data</a> volume and tolerance for latency. For example, either inference is a preprocessing step or it is embedded right inside a SPARQL engine. Such a thing cannot be seen from the picture. Same for trust. Trust is not an after-thought at the top of the picture, except maybe in the sense of referring to the other parts.</p> <p>We hear it over and over. Scale and speed are critical. Arrange the blocks of any real system as makes sense for data flow; do not confuse literature references with control or data structure.</p> <p>The even-more foundational issue is the promotion of the general concept of a Web of Data.</p> <p>The core idea that the Web would be a query-able collection of data with meaningful reference between data of different provenance cannot be inferred from the picture, even though this should be its primary message. Or it is better to say that the first picture shown should stress this idea and then one could leave the layer cake, in whatever version, for explaining the standards' order of evolution or inter-reference.</p> <p>So, the value proposition:</p> <p>Why? Explosion of data volume, increased need of keeping up-to-date, increasing opportunity cost of not keeping in real time.</p> <p>What? An architecture that is designed for unanticipated joining and evolution of data across heterogeneous sources, either at Web or enterprise scale.</p> <p>How? URI everything and everything is cool, or, give things global names. Use <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x13700d00">RDF</a>. Reuse names or ontologies where can. (An ontology is a set of classes and property names plus some more.) Map relational data on the fly or store as RDF, whichever works. Query with SPARQL, easier than <a href="http://dbpedia.org/resource/SQL" id="link-id0x17865208">SQL</a>.</p> <p>So, my challenge for the graphics people would be to make an illustration of the above. Forget the alphabet soup. Show the layer cake as a historical reference or literature guide. Do not imply that this proliferation of boxes equates to an equal proliferation of Web services, for example.</p> </div>
Virtuso Data Space Bot
kidehen@openlinksw.com
2008-06-11T15:54:49.000-04:00
voiD, or Will the LOD Cloud Bring Rain?
http://virtuoso.openlinksw.com/blog/vdb/blog/?id=1382
2008-06-09T14:02:20Z
<div> <div style="display:none;">voiD, or Will the LOD Cloud Bring Rain?</div> <p>At <a href="http://www.eswc2008.org/" id="link-id0x1c3bec48">ESWC2008</a>, we saw the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x1f0db270">Linked Open Data</a> Cloud condense its first drops of precipitation.</p> <p> <a href="http://community.linkeddata.org/MediaWiki/index.php?MetaLOD#Kick-off_meeting_at_ESWC08" id="link-id106ee858">voiD, Vocabulary of Interlinked Datasets</a>, is an idea whose time has clearly come. By the end of the conference, many speakers had already adopted the <a href="http://dbpedia.org/resource/Meme" id="link-id0x16c99ad0">meme</a>.</p> <p>The point is to describe what is inside the <a href="http://dbpedia.org/resource/Data" id="link-id0x1c540958">data</a> sets. People may know this from having worked with the sets or from putting them together but to an outsider this is not evident.</p> <p>The Semantic Sitemap says where there are files or end points for access. But it does not say what is inside these. Also for federation, it is important to be able to determine whether it makes sense to send a particular query to a particular end point.</p> <p>If we play this right, this is what voiD will provide. I have to think of Dan Simmons' flamboyant Hyperion sci-fi series where the "void which binds" was a sort of hyperspace containing the thoughts of entities, past and present and even provided teleportation.</p> <p>So what does the voiD hold, aside infinite potentialities?</p> <p>The obvious part is DC-like provenance, version, authorship, license and such data set wide <a href="http://dbpedia.org/resource/Information" id="link-id0x16c05280">information</a>. Also the subject matter could be classified by reference to <a href="http://umbel.org/about/" id="link-id0x1abf1558">UMBEL</a> or the <a href="http://www.mpi-inf.mpg.de/~suchanek/downloads/yago/" id="link-id0x1b49ee78">Yago</a> classification of <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x184dea28">DBpedia</a>.</p> <p>More is needed, though. The simple part is listing the ontologies, if any. Also a set of namespaces would be an idea but this could be very large.</p> <p>So let us look at what we'd like to be able to answer with the voiD set.</p> <p>The below could be a sample of voiD questions?</p> <ul> <li> <p> <i>What subjects are in the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x1bbac318">LOD</a> cloud?</i> </p> </li> <li> <p> <i>Given this <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x1f74c7e8">URI</a>, what set in the LOD cloud can tell me more?</i> This is divided into asking a text index like <a href="http://sindice.org/" id="link-id0x1d57a8f8">Sindice</a> for the location, getting the namespace or data set and then querying voiD.</p> </li> <li> <p> <i>What need I federate/load in order to combine all that is reachable from a given vocabulary?</i> There could be for example a graph showing the data sets and edges between them, edges being qualified by a set of same as assertions, itself a voiD described set, if translations were needed.</p> </li> <li> <p> <i>What sets are from the same or equally trusted publisher as this one?</i> </p> </li> </ul> <p>These things are roughly divided into description of the set and then some details on how it is stored on a given end point.</p> <ul> <li> <p> <i>Given this set, in which other sets will I find use of the same URIs?</i> For example, if I have language version x, I wish to know that language version y will have the same URIs insofar the things meant are the same.</p> </li> <li> <p> <i>Given this set, which sets of same as assertions will I have for mapping to which other sets?</i> For example, if I have <a href="http://www.geonames.org/" id="link-id0x1b372140">Geonames</a>, I wish to know that set x will map at least some of the URIs in Geonames to DBpedia URIs.</p> </li> </ul> <p>Let me further point out that it is increasingly clear to the community that universal sameAs is dubious, hence sameAs assertions ought to be kept separate and included or excluded depending on the usage context.</p> <ul> <li> <p> <i>Given this set, what are the interesting queries I can do?</i> This is a sort of advertisement for human consumption. This is not a list of queries for crashing the end point. Denial of service can be done in <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1b25dea8">SPARQL</a> without knowing the end point content anyhow, so this is not an added risk exposer.</p> </li> <li> <p> <i>Vocabularies used.</i> This is a reference to the OWL or RDFS resources giving the applicable ontologies, if present. Also, a complete list of classes whose direct instances actually occur in the set is useful.</p> </li> <li> <p> <i>Ballpark cardinality.</i> Something like a <a href="http://darq.sourceforge.net/" id="link-id0x1ed8f580">DARQ</a> optimization profile would be a good idea. I would say that there should be a possibility of just including a DARQ description file as is. This is a sort of baseline and since it already exist, we are spared the committee trouble of figuring out what it ought to contain and what not. If we start defining this from scratch, it will take long. Further, let this be optional. Quite Independently of this, query processors may make optimization related queries to remote end points insofar the specific end point supports these. This will come in time. For now, just the basics.</p> </li> </ul> <p>Along with this, LOD SPARQL end points could adopt a couple of basic conventions. The simplest would be to agree that each would host a graph with a given URI that would contain the voiD descriptions of the data sets contained, along with the graph URI used for each set, if different from the publisher's URI for the graph. There is a point to this since an end point may load multiple data sets into one graph.</p> <p>We hope to have a good idea of the matter in a couple of weeks, certainly a general statement of direction to be published at <a href="http://www.linkeddataplanet.com/" id="link-id0x1b049830">Linked Data Planet</a> in a couple of weeks.</p> </div>
Virtuso Data Space Bot
kidehen@openlinksw.com
2008-06-11T15:15:21.000-04:00
The DARQ Matter of Federation
http://virtuoso.openlinksw.com/blog/vdb/blog/?id=1381
2008-06-09T14:02:19Z
<div> <div style="display:none;">The DARQ Matter of Federation</div> <p>Astronomers propose that the universe is held together, so to speak, by the gravity of invisible "dark matter" spread in interstellar and intergalactic space.</p> <p>For the <a href="http://dbpedia.org/resource/Data" id="link-id0x19dbf410">data</a> web, it will be held together by federation, also an invisible factor. As in Minkowski space, so in <a href="http://dbpedia.org/resource/Cyberspace" id="link-id0x9fc13ff8">cyberspace</a>.</p> <p>To take the astronomical analogy further, putting too much visible stuff in one place makes a black hole, whose chief properties are that it is very heavy, can only get heavier and that nothing comes out.</p> <p> <a href="http://darq.sourceforge.net/" id="link-id0x1d06bd88">DARQ</a> is Bastian Quilitz's federated extension of the <a href="http://jena.sourceforge.net/" id="link-id0x1cf28f70">Jena</a> <a href="http://jena.sourceforge.net/ARQ/" id="link-id0x1cba22c8">ARQ</a> <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x171c7dc8">SPARQL</a> processor. It has existed for a while and was also presented at <a href="http://www.eswc2008.org/" id="link-id0x1ed53cd0">ESWC2008</a>. There is also SPARQL FED from Andy Seaborne, an explicit means of specifying which end point will process which fragment of a distributed SPARQL query. Still, for federation to deliver in an open, decentralized world, it must be transparent. For a specific application, with a predictable workload, it is of course OK to partition queries explicitly.</p> <p>Bastian had split <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x1ce846c0">DBpedia</a> among five <a href="http://virtuoso.openlinksw.com" id="link-id0x1cad0640">Virtuoso</a> servers and was querying this set with DARQ. The end result was that there was a rather frightful cost of federation as opposed to all the data residing in a single Virtuoso. The other result was that if selectivity of predicates was not correctly guessed by the federation engine, the proposition was a non-starter. With correct join order it worked, though.</p> <p>Yet, we really want federation. Looking further down the road, we simply must make federation work. This is just as necessary as running on a server cluster for mid-size workloads.</p> <p>Since we are convinced of the cause, let's talk about the means.</p> <p>For DARQ as it now stands, there's probably an order of magnitude or even more to gain from a couple of simple tricks. If going to a SPARQL end point that is not the outermost in the loop join sequence, batch the requests together in one <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x19a48280">HTTP</a>/1.1 message. So, if the query is "get me my friends living in cities of over a million people," there will be the fragment "get city where x lives" and later "ask if population of x greater than 1000000". If I have 100 friends, I send the 100 requests in a batch to each eligible server.</p> <p>Further, if running against a server of known brand, use a client-server connection and prepared statements with array parameters. This can well improve the processing speed at the remote end point by another order of magnitude. This gain may however not be as great as the latency savings from message batching. We will provide a sample of how to do this with Virtuoso over <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x1cf18278">JDBC</a> so Bastian can try this if interested.</p> <p>These simple things will give a lot of mileage and may even decide whether federation is an option in specific applications. For the open web however, these measures will not yet win the day.</p> <p>When federating <a href="http://dbpedia.org/resource/SQL" id="link-id0x1cf7d0e8">SQL</a>, colocation of data is sort of explicit. If two tables are joined and they are in the same source, then the join can go to the source. For SPARQL this is also so but with a twist:</p> <p>If a foaf:Person is found on a given server, this does not mean that the Person's geek code or email hash will be on the same server. Thus <code>{?p name "Johnny" . ?p geekCode ?g . ?p emailHash ?h }</code> does not necessarily denote a colocated join if many servers serve items of the vocabulary.</p> <p>However, in most practical cases, for obtaining a rapid answer, treating this as a colocated fragment will be appropriate. Thus, it may be necessary to be able to declare that geek codes will be assumed colocated with names. This will save a lot of message passing and offer decent, if not theoretically total recall. For search style applications, starting with such assumptions will make sense. If nothing is found, then we can partition each join step separately for the unlikely case that there were a server that gave geek codes but not names.</p> <p>For Virtuoso, we find that a federated query's asynchronous, parallel evaluation model is not so different from that on a local cluster. So the cluster version could have the option of federated query. The difference is that a cluster is local and tightly coupled and predictably partitioned but a federated setting is none of these.</p> <p>For description, we would take DARQ's description model and maybe extend it a little where needed. Also we would enhance the protocol to allow just asking for the query cost estimate given a query with literals specified. We will do this eventually.</p> <p>We would like to talk to Bastian about large improvements to DARQ, specially when working with Virtuoso. We'll see.</p> <p>Of course, one mode of federating is the crawl-as-you-go approach of the Virtuoso <a href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id0x1e163140">Sponger</a>. This will bring in fragments following seeAlso or sameAs declarations or other references. This will however not have the recall of a warehouse or federation over well described SPARQL end-points. But up to a certain volume it has the speed of local storage.</p> <p>The emergence of voiD (Vocabulary of Interlinked Data) is a step in the direction of making federation a reality. There is <a href="http://www.openlinksw.com/weblog/oerling/?id=1377" id="link-id1109a4c8">a separate post</a> about this.</p> </div>
Virtuso Data Space Bot
kidehen@openlinksw.com
2008-06-11T15:15:14.000-04:00