Master and Derivative File Locations

Owned by Michael B. Klein
Last updated: Mar 08, 2017 by ChrisC
2 min read

In release 3, Avalon introduced a mechanism for tracking the absolute locations of master/mezzanine and derivative files. The URLs it generates are not necessarily actionable, but are designed to give an administrator the ability to locate a particular file unambiguously. Avalon takes its best guess at how to build sensible URLs, but can be configured for better specificity and other modes of access.

Obtaining File Location URLs from Fedora

For Avalon 6.0:

Master file and Derivative file locations can be found as an ebucore:filename property on the Fedora object by visiting: http://fedoraserver.example.edu/rest/{treeified-id}

 

For Avalon 5.0 and lower:

Master file locations can be found on the Fedora object's masterFile datastream by visiting: http://fedoraserver.example.edu/objects/{pid}/datastreams/masterFile/content

Derivative file locations can be found on the Fedora object's derivativeFile datastream by visiting: http://fedoraserver.example.edu/objects/{pid}/datastreams/derivativeFile/content

Configuring the Creation of File Location URLs

Master Files

Master files (the original, pre-transcoding, uploaded files) are stored in the path specified by the matterhorn.media_path key in config/avalon.yml, and (optionally) deleted or moved elsewhere after processing. The location of the master file is turned into a URL via the following steps:

  1. Avalon grabs a list of mount points from the system and parses it, discarding everything that's not tagged as either smbfs, cifs, or nfs. (These are the only three remote file systems Avalon recognizes at the moment; we can add more if necessary.)
  2. It then creates a hash of mount points to absolute prefixes (e.g., { '/var/avalon/masterfiles/' => 'nfs://some.server.northwestern.edu/masterfiles/' })
  3. If the Avalon file location starts with any of the known mount points, the local mount point is stripped off the beginning and the rest is appended to the respective mount location. Otherwise, it just has file:// added to the beginning.

  4. There's also an optional config file, config/mounts.yml, that can be used to override/add to the mount hash. So if the master files directory were actually accessible via HTTP somehow, you could create the following config/mounts.yml:

    ---
/var/avalon/masterfiles/: http://masterfiles.northwestern.edu/

    That would cause Avalon to link to the master file at /var/avalon/masterfiles/dropbox/my_collection/my_video.mp4 as http://masterfiles.northwestern.edu/dropbox/my_collection/my_video.mp4.

Derivatives

Derivatives (transcoded streaming files) are handled differently, since, unlike master files, there's no guarantee that the Avalon server has access to or knowledge of the locations where streaming content is stored. Instead of having to mirror Matterhorn's streaming configuration within Avalon, Avalon simply requests an avalon.stream_base property from Matterhorn. This property can be found in Matterhorn's etc/load/org.opencastproject.organization-mh_default_org.cfg file. By default, the property is a file:// URL prefix pointing to the directory where derivatives are stored:

prop.avalon.stream_base=file://${org.opencastproject.streaming.directory}

You can, however, set the property to any other well-formed URL prefix if there is a better way to locate and retrieve files from that location:

prop.avalon.stream_base=http://streamfiles.media.northwestern.edu/avalon-streams