Data Migration from DSpace to Hydra

DSpace AIP Packager export: Importing and Exporting Content via Packages

Writing custom import rake job to process AIP Package

Storing Community & Collection naming in metadata fields for faceting. Currently coverage and sponsorship respectively, but that is likely to change.

Major: The size of data and frequency of calls with this approach appear to tax Fedora leading to timeouts after approx. 150 items or so. Will likely need to address the system resources on the fedora server during migration.

Major: Ambiguity of bitstream naming. Bitstream file names come out as bitstream_[some number].[extension]. The file name is in the metadata as "dc.title" with the dspaceType of "BITSTREAM". But this will likely be very difficult to address for multiple bitstreams on one item.

SOLUTION: Complete

In the AIP Package mets.xml

• • • Grab the MD5 hash and filename from the premis block
    • Query the exported filename from the fileGrp based on the MD5 checksum key

Minor: Duplication of dublin core fields for "system" data vs. item metadata
i.e. <dc.creator>Collier, Aaron</dc.creator> is the author,
while <dc.creator>acollier@calstate.edu</dc.creator> is the submitter. (solution: check for email address format) 

Minor: Embargoes work somewhat differently in Sufia. During the embargo period for an item, it is completely private. This might be expected behavior in some cases, in most cases for us DSpace allows viewing of the metadata. This functionality will need to be adapted into Sufia, but is not a super high priority yet.

https://github.com/scholarworks/dspace_packager/blob/master/lib/tasks/packager.rake

Very beta code for now and could use a thorough refactoring to be more "ruby-like". So far, it's working ok.

We are doing this more to stress test Hyrax. So we are not concerning ourselves presently with permissions of the items or bitstreams; where to store the bitstream descriptions; or the mapping of the items to their respective collections. There's also other issues which we are tabling for the time.

The one issue we did encounter is that the dspace exports have lots of numeric character references in the dublin_core.xml file. In order to convert them back to utf-8 characters we are using htmlentities. Here is a link that helped me out with this: https://makandracards.com/makandra/898-encode-or-decode-html-entities

These files:

https://github.com/mlibrary/deep-blue/blob/master/lib/append_content_service.rb

https://github.com/mlibrary/deep-blue/blob/master/lib/build_content_service.rb

https://github.com/mlibrary/deep-blue/blob/master/lib/tasks/populate_dev_app.rake

DSpace Replicate plugin to export Collection and Item BAGS. dspace-replicate

Stand alone ruby application, with configurations, to map metadata from each BAG into properly formed data to publish works into a Hyrax server "through the front door".

We are focused on striking a balance on how to operate the application, some of the concepts in mind are;

• Operate on BAG files that are exported from Dspace at first, with the idea in mind to extend capability to include bags generated by something/someone else
• Require explicit metadata mapping configuration so that nothing can slip through the cracks, we want to be confident that we handle all metadata and files purposefully (even if it means we explicitly ignore something)
• Use the commandline, drive the application making the most of things like shell for-loop a list of ITEMs to migrate
• Make a backup of the transformed data that was posted to the Hyrax server for troubleshooting/replay/sanity check
• Adhere to the API and shape of the data that the Hyrax server expects. The application cleans/maps/transforms and then posts data, the server takes care of the proper logic and functionality for persisting the works, files, etc.