Definitions

• Batch Ingest: the process of creating multiple media items by uploading an Ingest Package to an Avalon dropbox
• Ingest Package: the combination of content and metadata, consisting of one Manifest File and one or more Content Files
• Manifest File: a spreadsheet containing the metadata for the items to be created as well as the names of the Content Files that make up each item
• Content File: a single media file that is part of an item; one or more Content Files can make up an item in the Avalon Media System
• Item: a single media object accessible through one Avalon page; consists of one or more media files and metadata describing the media file(s)
• Collection: a grouping of items in Avalon for administrative and discovery purposes. Items belong to one and only one collection

Preliminaries

When a new collection is created, Avalon creates a sub-directory with the name of that collection (substituting underscores for any blanks) within the Avalon dropbox directory. A Batch Ingest is initiated by uploading an Ingest Package to the collection sub-directory in the dropbox. To connect to the Avalon dropbox, see Uploading Content to an Avalon Dropbox.

The Manifest File is a spreadsheet (xls, xlsx, csv, or ods) containing the metadata for the items to be created, as well as the file paths to the content files that make up each item; download manifest_example.xlsx for a blank manifest template. The Manifest File lists many metadata fields, but only a few of these fields are required: "Title," "Date Issued," and "File." However, "Bibliographic ID" may be used in place of "Title" and "Date Issues." For a description of the other fields, as well instructions for adding structure or captions, see Batch Ingest Package Format. 

Creating a Basic Manifest File

1. Open the Manifest File that will contain the metadata for the Content Files.
2. Column A Row 1 contains a reference name for the batch; this field is only for reference and should be renamed to something convenient or descriptive.
3. Column B Row 1 contains the email address or username of the submitter, to be used for notifications and exceptions. The submitter's username or email address must already be assigned as manager, editor, or depositor for the collection to which the batch is submitted.
4. Each of the columns on Row 2 specifies the potential metadata fields for each Item. If a field for an item is multi-valued (e.g. has more than one topical subjects or languages, or the Item is made up of more than one Content File), the column can be duplicated with the same header.
5. Row 3 and and all subsequent rows each represent a single Item to be created in Avalon. The minimum required fields for each item include:
   1. Title:
      1. Not repeatable: each item may only have one title.
      2. The title field represents the name of a single item, which may consist of one or more Content Files.
      3. If title is not available or missing, create a title that describes something about the content of the item.  This is necessary for identifying items in search results.
   2. Date Issued:
      1. Not repeatable: each item may only have one date issued.
      2. Date should be the main publication date associated with the item.
      3. Simple format should follow "yyyy-mm-dd", see EDTF specifications for other options, including a date range or an unknown date.
   3. File:
      1. The name of a Content File, which consists of two parts:
         1. The path to the file, relative to the Manifest File
         2. The name of the file itself, including the file extension (e.g. video1.mp4, songB.aac)
         3. Both path name and file name are case-sensitive; incorrect cases will result in a processing error.
         4. Example: A Manifest File and a folder titled "video_content" (with .mp4 Content Files) are uploaded to the dropbox collection "MyVideos". The Manifest File and the folder are now located in the same directory; therefore, each file name would follow the format of video_content/videoname.mp4. 
      2. Repeatable: if an item consists of several Content Files, the column "File" should be duplicated for each individual Content File constituting the item.
         1. Each "File" column should be immediately followed by a "Label" column.
         2. "Label" represents the name or short description of each Content File (e.g. Part 1, Introduction, essay topic, musical movement, etc.)

Example Manifest File showing reference name, username, Title, and Date fields (please note there are some intentional omissions in the Title and Date fields in this example):

Continued (file name and path name, file labels):

Uploading an Ingest Package to the Dropbox

A Batch Ingest will begin automatically once the Manifest File and the Content Files are uploaded to the appropriate collection in the Avalon dropbox. If you are not yet connected to the Avalon dropbox, follow the instructions at Uploading Content to an Avalon Dropbox. Once you are connected, uploading the files is a simple drag-and-drop process. The image below shows the files to be transferred in the left pane, and the files after they have been transferred in the right pane.

Image Modified

Troubleshooting a Failed Ingest

Invalid Spreadsheet

Avalon will check for new spreadsheets once every minute. Upon detection, the system will open the spreadsheet and attempt to validate it for correct information. Spreadsheets may fail validation for the following reasons:

• File is corrupt
• No username listed
• The listed username account does not have permission to upload to the collection

If the user account exists in the Avalon system, but the spreadsheet is invalid for some reason, the user will be emailed a failure message with explaining why the spreadsheet was not accepted; multiple reasons may be listed.

Example error message:

User USER_KEY does not have permission to add items to collection: COLLECTION_NAME

If the user account does not exist in the Avalon system (i.e. username was left blank, entered incorrectly, or no email address is associated with the account), the failure message will be sent to the notification email address for Avalon. The default notification email address is "avalon-notification@example.edu," and can be located in the Settings.yml file:

email:
  comments: 'avalon-comments@example.edu'
  notifications: 'avalon-notifications@example.edu'
  support: 'avalon-support@example.edu'

Valid Spreadsheet

If the spreadsheet passes validation, the user will receive an email stating that their If the spreadsheet passes validation, an email notification will be sent to the user listed on the Manifest File stating that the spreadsheet has been accepted:

Your metadata package was validated and no errors were found. Your batch is now queued.
Manifest file: example_manifest_file.xlsx

This message does not mean that the ingest was successfully completed, just that the sheet was accepted and the rows have been queued for ingest.

At this point, no further action is necessary. The content files will automatically be created as items in Avalon with the metadata from the Manifest file.

However, an unsuccessful ingest will generate a message to the user with the details of the attempted process. Please note that even if only a single record contains an error, the entire batch ingest will fail and no items will be created. 

Example failure message:

Image Removed

This example, which was generated from the failed batch ingest of the Ingest Package shown in the above images, demonstrates some of the most common issues preventing a successful ingest. A few of the files were missing fields (either Date Issued, Row 2, or Title, Row 6), but the rest of the files were indicated as "No files listed." This can either indicate that a Content File was not included in the transferred files, or that there was an error in the "File" field. In this particular case, the folder was named "Other_Short_Films" while the path names were written as "other_short_films". Path and file names are case-sensitive, and therefore the batch ingest failed. 

Other issues may involve incorrect formatting in other metadata fields; refer to the Batch Ingest Package Format for specific details about each fieldImage Added

After all of the objects listed on the Manifest File have been successfully uploaded, an email notification will be sent to the user regarding the completion of the Batch Ingest:

Image Added

This concludes a successful Batch Ingest process; all the items should be uploaded into the Avalon Media System and either ready for further editing or available for viewing. However, if there are any errors with the spreadsheet or with any of the media objects on the spreadsheet, the user will receive an error message. To resolve any issues with a Batch Ingest, see Troubleshooting a Batch Ingest.