The Semantic Annotation Tool was created by the Media Ecology Project to facilitate time-based annotations of video on the web. It is one link in a larger ecosystem of software that supports machine and human annotation, analysis, and publication of scholarship connected to video files. SAT’s role in the ecosystem is to provide an annotation toolset that can be integrated into existing web applications and publishing tools.

The Semantic Annotation Tool consists of two parts: Waldorf.js and Statler. Waldorf.js is a drop-in jQuery module that facilitates the creation and sharing of time-based media annotations on the Web. Waldorf.js wraps HTML5 video tags on a page with an intuitive authoring and presentation environment for time-based video annotations. Annotations are loaded from and saved to Statler, a linked data-compliant server that communicates with the plugin to collect and disseminate user-generated comments and tags compliant with the W3C Open Annotation specification. This system creates an end-to-end open source video annotation workflow that can be used as either an off the shelf or customizable solution for a wide variety of applications.

In this workshop we will focus on Waldorf.js. However, to understand where Waldorf fits into the larger ecosystem, here is an example of a full workflow that uses Waldorf:

Step Role Uses Creates Language/Platform
Archive Source Provides primary materials to be studies Researcher selection criteria Video file corpus n/a
Distant Vision Toolkit Machine annotation for face and cut detection Video file corpus Cut list (csv) Python (OpenCV)
Onomy.org Controlled vocabulary builder Term list and definitions Taxonomy of terms (json, xml) PHP web site, SQL
Waldorf.js Human annotation, validation of machine annotations Video corpus, machine generated annotations Annotations, tags, playback interface Javascript (jQuery) web plugin
Statler Annotation server W3C-formatted annotations, API calls W3C-formatted annotations Ruby on Rails, SQL
R Studio Analysis of annotation data W3C formatted annotations Statistical analyses and visualizations of annotations R, desktop application
Scalar Web publication Video corpus, didactic text, annotations, Waldorf.js Integrated text/video essays on the web PHP web application

Each step of this workflow uses software that requires a different environment. Waldorf.js is expected to be integrated into a web site and therefore needs a web server to run. For the purposes of the workshop, we will use AMPPS to start a local web server on your computer.

Required Downloads

This workshop will require a local web server, a code editor, and sample files. If you are in the workshop at DH2019, these will be provided on a USB drive. Otherwise, choose whichever of the three options below is appropriate for your operating system:

  • If you already have a local web server and code editor, you can skip Installing AMPPS and SublimeText and only need the sample files: dh_demo.zip
  • If you need a local web server or code editor for Windows: windows.zip
  • If you need a local web server or code editor for MacOS: mac.zip

Installing AMPPS

AMPPS is free software available for Windows, MacOS, and Linux. The appropriate version for your computer can be found on the AMPPS site or on the USB drive supplied for this workshop.

AMPPS will install a development version of the Apache web server required for Waldorf.js along with related software that is often used with it like MySQL, PHP, and Python. Complete installation instructions for Mac and Windows are available on the AMPPS wiki.

Once AMPPS is installed and running you will be given a control panel where you can turn individual servers on and off. Make sure that Apache is turned on, then click the home icon to open the control panel in your browser. Click on ‘manage domains’ and note the path listed next to 127.0.0.1–this path is your web root. This is where you will put the Waldorf files.

Installing SublimeText

This step is not necessary if you already have a code editor on your computer (note: a code editor is slightly different than word processors like Word or Pages).

SublimeText is a lightweight code editor that we will use to edit HTML and Javascript for Waldorf.js. You can download an evaluation copy from SublimeText’s site. An installer for MacOS and Windows is also on the USB drive for the workshop.

SublimeText for MacOS can be installed by double-clicking the install package. SublimeText for Windows is available as a portable executable that can be run directly rather than being installed.

Copying Waldorf.js Sample Files

Sample files used for the workshop are on the USB drive in the dh_demo directory. Copy these files from the USB drive to your computer and put them in the web root directory you noted after installing AMPPS.

Files in the dh_demo directory include:

  • index.html – the main HTML file we’ll use in this workshop
  • Sleepfor1950.mp4 – the video we will be annotating
  • Sleepfor1950.mp4.csv – a spreadsheet of cuts generated by the Distant Viewing Toolkit
  • Sleepfor1950.mp4.json – the cut list, reformatted so it can be imported into SAT
  • waldorf.html – the final version of index.html we will end up with at the end of the workshop
  • js/main.js – the code that configures and invokes Waldorf.js, including supporting functions added for the workshop

Once the files are copied, open http://127.0.0.1/dh_demo/index.html. You should see a page with a sample video file.

Changing the Video File Name

For the purposes of the workshop, it is very important to change the name of your video file! Waldorf indexes annotations based on the URL of video files it finds embedded in a web page. Since everyone is using the same file and URL, and all annotations are being stored on the same Statler server, everyone’s test annotations will show up when you load them from the server.

Find the Sleepfor1950.mp4 file in your web root. Rename the file by adding another number to the end of the name (before the extension, e.g. Sleepfor1950-001.mp4).

If you reload http://127.0.0.1/dh_demo/index.html you will find the video no longer appears because you renamed the file it was loading. Navigate to your web root and open index.html in SublimeText. The HTML that loads the video file appears on line 48:

<source src="Sleepfor1950.mp4" type="video/mp4">

Change the src name here to match the new file name you chose and reload your browser window. The video should reappear.

Adding the Waldorf Interface

Line 18-28 of index.html are HTML comments that list the steps necessary to add Waldorf.js to a page. Copy the following HTML to the matching empty lines in index.html:

19: <script src="js/vendor/jquery-3.1.1.min.js"></script>
21: <script src="js/vendor/jquery-ui.min.js"></script>
22: <link rel="stylesheet" href="css/jquery-ui.css">
24: <link href="css/select2.min.css" rel="stylesheet"/>
25: <script src="js/vendor/select2.min.js"></script>
27: <script type="text/javascript" src="js/annotator-frontend.js"></script>
28: <link rel="stylesheet" href="css/annotator-frontend.css">

These lines load the javascript and css core of Waldorf. To run Waldorf, add this line on the next-to-last line of index.html, just before the </html> tag:

<script type="text/javascript" src="js/main.js"></script>

Save the index.html file and reload it in your browser. The video should now have the waldorf.js annotation interface loaded on top of it.

Adding an Annotation

Hit the ‘+’ icon in the lower-right hand corner of the video to open the annotation interface. Enter a start time of 0:00, an end time of 0:10, no tags, and a text comment of “hello world!” Now hit the green checkmark near the top of the screen. The annotation will be saved to the Statler server that has been configured for use in this workshop and should appear below the video.

Now, click on the header for the annotation you just created (it should be called ‘Annotation 1’ and will be just below the video). This will reopen that annotation for editing. Click the bomb icon to the left of the red X to delete your test annotation.

Adding Tags

Create a new annotation, again running from 0 to 10 seconds. This time, in addition to adding a text comment, click into the ‘tags’ field. When you do so you will see a list of suggested tags appear; begin typing to filter the list. If the tag you want to add appears in the list you can click on it to add it to the annotation. If you don’t see the tag you want to use, type the entire word. That word will appear as the only suggested tag in the list; click on it to add it to the annotation.

Adjusting the Start and End Time

With the ‘create annotation’ interface open, scrub the video to roughly 0:20. (To get the playhead closer to exactly 0:20, you can use the nudge forward and back buttons next to the play button.) Hit the marker button to the right of the ‘start time’ field. The starting time for your annotation should adjust to match the current location of the playhead. Now scrub to roughly 0:30 and hit the marker button next to the ‘end time’ field. The ending time for your annotation will match the current location of the playhead.

Adding a Geometric Target

With the ‘create annotation’ interface open, click on the pencil icon. This will hide most of the annotation interface to show the full video frame. If you want to indicate an annotation only applies to a portion of the screen you can draw a polygon around that area. Click in the video to add one point of a polygon. Once you have added three points you will be able to see the polygon appear in the frame. Hit the green checkmark to store the polygon as a target of the annotation.

Uploading Annotations From a File

Hit the ‘upload’ button near the lower-right corner of the video player, just to the left of the ‘+’ button that creates a new annotation. This button lets you import a set of annotations that have been saved as a json file. Navigate to the web root and find the ‘Sleepfor1950.mp4.json’ file. When you choose the file, the annotations within it will be automatically uploaded to the Statler annotation server and appear in the annotation interface. (Note: It does not matter that you previously renamed your video file because any annotations in the json file will be applied to the currently loaded video, regardless of the original URL they were associated with.)

Playing Annotations

To show annotations in sync with the video, you only need to play the video. Annotations will appear and disappear at the appropriate time just below the video frame.

Customizing Annotation Playback

If you want to change where annotations are displayed you can create an HTML div tag with a special ID and Waldorf will use that div rather than creating a new one below the video player. The waldorf-info class can be styled and positioned using standard CSS, though care should be taken to avoid breaking compatibility with screen readers.
<div class="waldorf-info" aria-live="polite" aria-atomic="true"></div>

Adding an Annotation List

The version of main.js that we are using for this workshop contains extra Javascript functions for working with annotation data. To enable the annotation list, copy this code into the blank space in your index.html file at line 50:
<div id="index-container">
<h4>Annotation Index</h4>
<div>
Filter by tag: <select id="tag-filter"></select><br>
Filter by author: <select id="author-filter"></select>
<div class="waldorf-index"></div>
</div>

Exporting Annotations

Annotations are transmitted from the Statler server in standards-compliant json. To view or download the json for the current video, click the “Open Annotation Manifest in New Window” button below the video. The window that opens will be the complete json file; simply save it to your hard drive to export annotations from Statler. json is a common format that can be parsed by many programming languages for analysis.

Advanced Topics

You have seen how to add Waldorf.js to a web page with an existing HTML5 video, add, edit, export, and delete annotations, and add new functionality to the basic Waldorf.js annotator. At this point we can go into greater depth on specific questions or ideas you have for using Waldorf.js. A few ideas include:

  • Exploring Waldorf.js’s configuration options
  • Importing custom controlled vocabularies
  • Integrating Waldorf.js with an authentication system
  • Discussing workflows involving machine-generated annotations
  • Using Waldorf.js as part of a larger content management system