The JSONX Importer

Introduction

Exhibit loads its data using importers: small software modules tied to a given MIME type that read <link> elements in an Exhibit’s HTML, load the referenced data, and translate it into an array of JavaScript item objects which populate the Exhibit database. Exhibit ships with a standard JSON importer (JavaScript Object Notation), mapped to “application/json”, which can be used by including a link element like this:

<link rel="exhibit/data" type="application/json" href="educational_thinkers.js" />

During its work developing Exhibit based prototypes to study the semantic web and case based learning, the Ensemble Project found itself having to load various live data source, such as data fetched using SPAQL queries from the Mulgara triplestore. These data sources, while in legitimate JSON format, do not contain the field names expected by Exhibit.

The standard Exhibit JSON importer provides no opportunity to manipulate the data programmatically after it is loaded and translated to a structure, but before Exhibit creates a database from it. Such manipulation would permit an Exhibit author (with JavaScript knowledge) to alter the incoming data, check its completeness and validity, create extra fields, etc.

Solution

A new importer was written by the Ensemble Project, tied to the dummy MIME type “application/jsonx”, which works the same as the standard JSON except it provides a callback function to allow code to be injected between the data loading and database creation part of the initialisation process.

<link rel="exhibit/data" type="application/jsonx" href="educational_thinkers.js" ex:converter="converter"/>

In the above example code the type of the link is “application/jsonx”, triggering the customised Ensemble JSON importer rather than the standard Simile importer. The parameter ex:converter defines a JavaScript function, converter, which will run with access to the item data structure before it is translated into the Exhibit database. The custom importer does not automatically get included with Exhibit, so it is necessary to include its code manually, after Exhibit’s main API is loaded:

<link rel="exhibit/data" type="application/jsonx" href="educational_thinkers.js" ex:converter="converter"/>

[…]

<link rel='stylesheet' href='/simile/styles/common.css' type='text/css' />
<script type="text/javascript" src="/simile/exhibit/api/exhibit-api.js"></script>
<script type="text/javascript" src="/simile/extensions/api/scripts/data/importers/jsonx-importer.js"></script>

[…]

<script type="text/javascript">
function converter(json) {
  var items = json.items;
  for(var i=0;i&lt;items.length;i++) {
    var it = items[i];
    […manipulation here…]
  }
  return json;
}
</script>

Second generation

Ensemble’s first version of the JSONX importer was useful if the page author had JavaScript skills, but not so useful if they only knew HTML. In many scenarios the author just wants to strip away unwanted items from a large dataset received from a web service (like a triplestore), or remove invalid items from a JSON file (bad URLs, missing data, etc). To cope with this, a second version of the JSONX importer was written by Ensemble, supporting filters definable from within the HTML using a syntax familiar to Exhibit authors. These filters retain certain items in the data and drop any items that do not match.

To use the filters, include elements with a ex:role of preFilter or postFilter (case is important) in the HTML. Pre filters are called (in the order they appear in the HTML) before the ex:converter callback function is called. Post filters, unsurprisingly, are called after the converter callback function.

<div ex:role="preFilter" ex:itemType="person" ex:property="birth_date" ex:value="18.."></div>
<div ex:role="postFilter" ex:itemType="person" ex:property="death_date" ex:value="19.."></div>

To clarify, the execution sequence is:

Filters are applied using HTML elements with ex:role=”preFilter”, in the order they appear in the document.
The converter callback is called, if defined in the <link> element.
Filters are applied using HTML elements with ex:role=”postFilter”, in the order they appear in the document.

Filter attribute

Each filter has several settings. They are:

ex:itemType : if present, specifies the item type the data must be. Any items not of this type are discarded.
ex:property: if present, specifies that a given property must be present on the item. Any items without said property are discarded.
ex:value : can be optionally specified along with the ex:property attribute to insist on a given value for a property. The value string is actually a regular expression, if the expression fails to match the item’s value, the item is discarded.

Filter examples

In the below, the first filter runs before any callback code and retains only those items with a birth_date in the 19th century, while the second filter runs after the callback and further restricts the items to only those with a death_date in the 20th century. The net result is a list of person items whose life straddled the turn of the 20th century.

<div ex:role="preFilter" ex:itemType="person" ex:property="birth_date" ex:value="18.."></div>
<div ex:role="postFilter" ex:itemType="person" ex:property="death_date" ex:value="19.."></div>

Attributes may be omitted.

<!--Type must be "person", item.birth_date matches "18.." -->
<div ex:role="preFilter" ex:itemType="person" ex:property="birth_date" ex:value="18.."></div>

<!-- Any type, item.birth_date matches "18.." -->
<div ex:role="preFilter"  ex:property="birth_date" ex:value="18.."></div>

<!-- Any type, item.birth_date must be present (value unimportant) -->
<div ex:role="preFilter" ex:property="birth_date"></div>

<!-- Type must be "person", everything else unimportant -->
<div ex:role="preFilter"  ex:itemType="person"></div>

<!-- The following lines should have an identical effect -->
<div ex:role="preFilter" ex:itemType="person"></div>
<div ex:role="preFilter" ex:property="type" ex:value="person"></div>

Ensemble