Archive for open data

Radically Open Cultural Heritage Data on the Web

I was extremely fortunate to be part of a panel session at SXSW Interactive 2012, held this month as always in the amazing City of Austin, Texas.  The panel, Radically Open Cultural Heritage Data on the Web, was put together by Jon Voss, Strategic Partnerships Director at Historypin, leader of Linked Open Data for Library Archives and Museums (LODLAM) activity and a fantastic voice in the LODLAM space. Joining Jon on the panel was Adrian Stevenson, Senior Technical Innovations Coordinator at Mimas and Project Manger of the excellent LOCAH project and Rachel Frick, Director of the Digital Library Federation and heavily involved in the Digital Public Library of America (DPLA).

Details (and audio) of the panel session can be found on the SXSW site, and my slides are available on slideshare. Adrian has also made his slides available.

Between the four of us, we gave an overview of what linked data is, why it matters to libraries, archives and museums, how people have put data out there already, how others are beginning to consume that data, and how people might get involved. My presentation focussed on York’s OpenART project where we have put almost 40,000 linked data documents on the web. This represents a huge success for our project, which was run on a shoestring budget and timescale. Our approach to exposing data is certainly not perfect, but it’s a significant step for us towards opening our data up for others to work with.

The linked and open data area is certainly growing and beginning to be recognised within the semantic web community. I attended Libraries, Media & The Semantic Web hosted by the BBC on the 28th March 2012 where Jon Voss and Adrian spoke on the same platform as speakers from the New York Times, the BBC and Google. It was particularly encouraging to hear that the BBC has invested a huge amount (20% of it’s Digital budget) into linked data for the Olympics coverage, and also from Dan Brickley (Google) who confirmed the forthcoming support for RDF/A in schema.org. Video from that event will be made available online on the BBC Academy YouTube Channel, and it’s worth watching all of the speakers. JISC have also recently published an interesting article on Linked Data in the recent issue of JISC Inform.

I’m encouraged by the signs of open relationships and willingness to work together to make standards and approaches work, and by the increasing efforts to open up data. I feel that York Digital Library (and perhaps the University more widely) should continue to invest effort into linked open data.

Leave a Comment

OpenART Final Report

Made by OpenART:

  • The OpenART ontology, an event-driven ontology produced to describe the ‘artworld’ dataset. The ontology is split into a number of parts to allow greater re-usability. It should be considered ‘work in progress’, although the version published is complete for the OpenART project: dlib.york.ac.uk/ontologies/
  • An ontology browser version of the ontologies can be found at dlib.york.ac.uk/ontologies/openart – for easier reading!
  • Sample data for each ‘primary’ entity in the form of a rdf/xml document and turtle document are available: dlib.york.ac.uk/data
  • Void description for the dataset: dlib.york.ac.uk/data
  • A script for creating all of the documents and ingesting them into the Digital Library Fedora repository.
  • RDF/A embedded in the pages of artworld.york.ac.uk (forthcoming)

Next steps for OpenART and the University of York:

For the University of York, there is some work to complete in order to get the full (current) dataset into our Fedora repository, mainly in setting up the url rewriting and content negotiation rules.

After that, we would ideally like to apply the same linked data principles to other Digital Library content, particularly some of the rich image content that we have. This would involve mapping and modelling work, for example VRA image metadata to linked data, and automating the generation of RDF.

Some thoughts

The approach taken in OpenART is somewhat twofold, with prototyping using a variety of tools (summarised in the technical approaches post) which could be further explored in future work.

The dataset which drives OpenART was released as a web application in October at http://artworld.york.ac.uk, to meet the requirements of the separate AHRC-funded project which it was created out of. This site has been developed as a database-driven application, an approach chosen as a best fit for the time available. The site, always envisaged as a human-user end point, was not initially designed for linked data. Indeed, one might argue that databases and ontologies do not make happy bedfellows. However, what we have found in the project is that is was relatively straightforward to (1) create a script to extract open data documents from the database and ingest that into our Fedora Digital Library, and (2) add RDF/A tagging to the web site itself.

One of the benefits of this approach is that it provides us with a non-proprietory back-up and preservation routine for the database, playing to one of Fedora’s strengths. It also demonstrates how Fedora can be used in place of a simple file structure to serve up linked data documents, bringing with it the advantages of data management, indexing and version control.

What this rather document-centric approach does not provide is a fully indexed RDF store with a SPARQL end point. Although Fedora has these as part of its stack they are internal tools for Fedora, not designed for indexing anything other than the core Fedora datastreams. Future work to enable Fedora’s ‘semantic’ capacity for external content would be extremely useful. The European Interactive Knowledge Stack (IKS) Project is doing interesting work in this area (http://www.iks-project.eu/).

Opportunities

OpenART was always focussed on a narrow rich seam of data, rather than a broad simpler dataset. There is an opportunity here to see how these two approaches can co-exist. Good ontology modelling will allow rich drilled-down terms to be mapped back to broader concepts for greater findability of content, whilst allowing much finer-grained analysis of the detail captured by the ontology. Where there may be a gap is in the tools which query, visualise and analyse the data sources.

Extending existing applications to better support open data is another opportunity, allowing standard repository platforms such as EPrints, DSpace and Fedora Commons to offer standard linked data endpoints, with options for configuring the data exposed.

Google Refine has come out strongly in OpenART as an extremely useful tool for manipulating datasets. It is particularly well suited to people who do not have in-depth programming skills but want to get RDF out of semi-structured documents. There is an opportunity to demystify some of the myth around creating open data and RDF, which can be quite simple to do.

Evidence of reuse

Our data has not been re-used, although we have had interesting discussions with a range of stakeholders from Tate, to be summarised in a blog post on how others could follow in OpenART’s footsteps. Skills What skills were used in your project? Did you already have these skills in your team or did you need to develop them or bring in external experts? Are the processes you have developed embedded in your institutional practice now or are there plans to embed them? Do you plan to develop these skills further? OpenART did have a range of skills in the team, in java programming, databases, Fedora Commons and metadata. External experts brought ontology modelling and RDF expertise, along with extra Fedora Commons expertise. These were essential for the project. OpenART has seen members of the project team gain much deeper knowledge of RDF and ontologies which we would hope to embed in York with further projects around linked open data.

Lessons

Lesson 1

Ontology modelling is complex so allow plenty of time for it. Take time to consider the data model and consider the best approach, a simpler ‘mix and match’ of schema terms might be suitable. For OpenART, where the data is very specific, an ontology was considered the best approach.

Lesson 2

Use Turtle during development phases and get familiar with validation and inspection tools. Turtle is a simple notation for RDF and is very easy to write and to understand. It can be exported directly out of Google Refine and validated by common tools (eg. http://www.rdfabout.com/). Any23 (http://any23.org/) can be used to generate other formats, such as RDF/XML or RDF/A. Sindice’s Inspector tool (http://inspector.sindice.com/) is a useful tool for viewing the relationships in RDF and checking that documents are not just valid, but also correct. Google Refine can be used as a relatively rapid application for generating RDF samples.

Lesson 3

Come up with use cases. What questions do users want to answer with your data? What links do they want to follow? Understanding the uses and potential uses of the data can help both with modelling but also with making the case for doing linked data in the first place.

Leave a Comment

OpenART Technology Choices

During the course of the OpenART project we’ve come across a number of different technologies and standards that we could use. In this post we aim to go into the ones we’ve used and found useful.

We aim to answer the following series of questions:

What technologies, frameworks, standards are you using in your project? What are your impressions of them? What difficulties have you encountered? What approaches and techniques have worked well? What advice would you give to others engaging with them?

“What technologies, frameworks, standards are you using in your project?”

Data preparation and mapping/triplification/lifting

The source information was supplied as a set of Excel spreadsheets containing semi-structured data. These spreadsheets were provided by the researcher and were the “live” data capture mechanism for the transcribed source data. As such they were subject to structural changes during the lifetime of the project. Various tools and approaches for manipulation of this source data were explored.

  • Relational Database (RDBMS)
    • Early in the project an approach of migrating the data to an RDBMS was explored
    • This resulted in a cleaner version of the source information, but the approach was abandoned as the spreadsheets were a live tool
    • The RDBMS ontology mapping plugin for Protege was explored for transfer of RDBMS data to RDF/OWL
  • Excel Spreadsheet manipulation
    • Google Refine was used for very quickly visualising and manipulating source information
    • RDF Extension for Google Refine was used for exporting cleaned up data held in Refine to test versions of RDF linked data, and for for matching nodes to validate and connect parts of the data (via the reconciliation feature)
    • GNU SED (Stream Editor) is a powerful general purpose text manipulation tool. This hits the cleaning spots the developers can’t reach (quickly) in the visual Refine environment.

Ontology development

  • Protege version 4.1 was used for ontology development. Protege is a powerful and fully featured ontology IDE (integrated development environment)

Linked Data manipulation and production

  • Rapper (part of the Raptor RDF Parser toolkit) was used for translating between various RDF serialisations. Rapper is a parsing and serialising utility for RDF.
  • SPARQL (A standard query language for RDF) for querying and checking dtata
    • Rasqal via the online service at Triplr. Rasqal provides RDF querying capabilities, including SPARQL queries
    • ARQ, a SPARQL query engine for Jena

Hosting and base systems

  • openSUSE Linux, a major Linux distribution, was used as a base operating system
  • SuseStudio was used to build and deploy bespoke virtual machines to Amazon EC2. SuseStudio allows you to choose packages to build the virtual machine and will directly deploy to an Amazon web service account all within a web-based interface
  • Amazon Web Services (AWS) was used for hosting, particularly the Elastic Compute Cloud (EC2) service for virtualisation
  • The Apache web server with mod_rewrite was used for web hosting of ontologies and data, with content negotiation
  • OntologyBrowser was used for for storing, viewing, manipulating and accessing the ontology, using a web front end and REST API

“What difficulties have you encountered?”

Source data manipulation and conversion

  • The source information was being actively worked upon during the lifetime of the project. This presented difficulties as working on the ontology and triplification to RDF concurrently with changes in the source information made decision-making and coordination difficult
  • The source information was provided as Excel spreadsheets. These were used as an organisational environment for capturing data by the researcher, and provided free text search, categorisation and a basic namiing scheme for entities. Although a degree of rigour was applied, the environment could not be described as “structured data” such as that provided by a database, and this provided challenges in interpreting and modelling the information.
  • Originally the project assumed that hosting of the information as part of the Court, Country, City project would provide a structured source for the information; however the requirements from this project were sufficiently divergent from the OpenART project’s requirements for this not to be the case. As a result the project did not consider alternative automated processes to assist in source data manipulation and triplification until late in the project.

Ontology development tooling

  • Bulk changes in the ontology using Protege was difficult to do efficiently
  • Synchronisation between environments was a challenge with new versions of ontologies

Ontology development informational issues

  • Difficulties in extending the combined ontologies – particularly when trying to move from earlier versions expressing “simplifications” to a more complex framework later in the project. It would have been easier to start with a more complex framework initially rather than trying to extend the earlier version.
  • Naming conventions became very cumbersome, for example when trying to create inverse properties that sometimes had no natural or simple English language fit

“What are your impressions of them?”

  • Protege and RDF tools used locally were found to be very mature. Protege is now mature enough to be usable by newcomers; and also provides a route to further development as it is based on an OWL-API. However it was difficult trying to reason with anything other than a small set of data points.
  • Google Refine was surprisingly efficient and workable. Additional functionality such the as reuse of RDF mapping templates was useful.
  • Cloud services (Amazon EC2) were relatively easy to use; and the combination of openSUSE and SuseStudio provided an efficient and repeatable mechanism for deploying virtual machines
  • OntologyBrowser presents a nice iinterfacae for ontology browsing in HTML. Its user-friendly syntax for ontology axioms means that you don’t need to be an OWL or logic expert. However it is very “data-centric” and not an end-user environment.

“What approaches and techniques have worked well?”

  • Google Refine is good for rapid development when a non-trivial ontology requires that mappings and ontology need to be developed in tandem
  • Use of Protege (see above)
  • Use of cloud services (see above)

“What advice would you give to others engaging with them?”

  • Investigate using a collaborative ontology environment, such as knoodl
  • Invest a small amount of time in collaborative environments, this can lead to a big payoff; cloud services are relatively easy to use
  • Start with an ontology framework complex-enough to represent the modelling required; it can be difficult moving from a simple to a more complex version
Author: Martin Dow

Leave a Comment

OpenART – Some Wins and Fails

Win! Researcher on hand to explain the data and answer questions.

Win! Openness to being open with the data.

Win! Increased understanding of open data and ontologies for the domain.

Win! Ready-made expertise on the team.

Win! Google Refine as a quick way of experimenting with spreadsheet data and getting RDF out of spreadsheets.

Win! Indexing in SINDICE should be a quick win.

Fail! The ontology took longer to create than we anticipated.

Fail! The data is complex and still a work in progress, the spreadsheets memory-hungry and in need of some cleanup and post-processing.

Fail! Lots of re-visiting and round-tripping slows things down.

Fail! There is a gap between the precision needed for an ontology and the working spreadsheets of a researcher.

Leave a Comment

OpenART – Costs and Benefits

OpenART – Costs and Benefits

One of the blog posts required for OpenART project is around ‘costs and benefits’: “This should be a very rough estimation of how much it has cost you in terms of time and resources to make your data openly available. What do you expect the benefits to be? And how do these 2 assessments balance out against each other?”.

Ontology Development

It’s taken around 10 days to develop the ontology in it’s current state. Bearing in mind this has been created by someone with a high level of knowledge and expertise in ontologies, related technologies and tooling, someone who already knows of existing relevant ontologies and could rapidly prototype an approach. A quicker and simpler approach would have been to ‘mix and match’ by selecting properties from a range of existing ontologies, without going so far as to develop a dedicated ontology. A longer and more complex approach would have been to experiment with different ontology approaches to establish the best and richest solution, eg. descriptions and situations. Given the time available we opted for the middle ground, but this has to some extent delayed other work and has made for a more complex process of ‘understanding’.

Data Analysis and Manipulation

Analysis of the spreadsheets, importing and exporting the data, building web views of the data and data cleanup, I’d put at 25 days.

Generating the Data

Generating RDF instance data from the spreadsheets, post-implemention would take around 10 days to set-up and produce a sub-set of data. Longer if we want to process all of the data, which we aren’t doing right now.

Technical Implementation

Prototyping and implementing a simple approach to resolving identifiers and content negotiation will come in around 10 days.

Building Understanding

Developing an understanding of the ontology, of linked data principles and of how to construct RDF, I’d estimate at taking up around 25 days, including background reading and research, digging into the ontology, working through examples and selecting the right tools. This also includes time spent on understanding the data itself, working through the spreadsheets, talking with the content creator.

Resources

In terms of people involved, there are a number, including: consultant partners (expert in the semantic aspects and technologies), developer (for implementation and data processing), researcher (for understanding the data), Tate web manager and curator (for exploring the Tate use case and validation), Digital Library core staff (for future sustainability).

Summary

This gives us a grand total of around 80 days, or 16 weeks, around 4 months of work involving a range of people. This is starting out with complex data, gaps in understanding of the data and in how to ‘do’ open data, along with a myriad of potential mechanisms and methods.

What this doesn’t include, though, is the ongoing cost, in maintaining and building on the project outcomes and rolling out the generation of data to the full dataset and working with additions and changes to the dataset. Extending the work to do more ‘linking’, assessing the scalability of this approach, experimenting with different approaches and promoting the content.

Benefits

Given that it is still early days for ‘linked data’ and for our content, benefits are harder to quantify. The immediate benefits are engagement and seeing our researcher and Tate partners really begin to think about what open and linked data might offer them. Another immediate benefit is a new class of content for the Digital Library and the potential for extending our reach into storing and serving ‘data’ from York Digital Library and enabling us make progress in opening our data.

Looking forward, what OpenART and open data promise are greater visibility and usability of information, making research richer and easier and reducing the amount of information that needs to be stored locally and constructed manually. For cultural institutions like Tate, open and linked data offer opportunities for increasing web traffic and usage, by offering new ways of exposing and consuming data, greater possibilities for dynamic links between artworks, artists, places and events and richer visualisations of data. For researchers, and other consumers, open data offers mechanisms to follow different paths through information, tracing art history from creation, through the art market and into a galleries, telling interlinked and often unexpected stories along the way. Whilst OpenART deals with only a fragment of the art market, it offers a  possible model that, if extended, could open up art history to much more detailed analysis.

How do these balance out?

Personally, I think that the cost is worth it, but only if we start to see real use of the data, and more data being opened up. Given that this latter can’t happen without investment in the former, we need to make a persuasive case to continue the work, engage in the community and build tools to aggregate disparate content for the non-technical user. One lesson learnt though is that not all data is made equal, and ours is still ‘under construction’ which adds a layer of complexity when working with a moving target.

Leave a Comment