Batches
If the batch cron job stops working
First, check the log file in log/whenever_cron.log to see if there are any error messages. If there aren't the lock file might be stuck. To ensure the batch job doesn't run more than one job at a time there is a lock file tmp/pids/batch_ingest.*. Delete the matching files and wait for the next run of the cron job.
Matterhorn
Ingest service reports as WARNING on Avalon's about/health page
When an ingest fails, it may leave the Matterhorn service in a bad state until subsequent ingests succeed. If you know that was not caused by a Matterhorn issue, you can bring its status back to OK by sanitizing the service:
1. Goto: http://MATTERHORN_URL:PORT
2. Login using credentials (IU: Test+server+documentation)
3. Goto: http://MATTERHORN_URL:PORT/docs.html?path=/services
4. Submit the sanitize form with serviceType=org.opencastproject.ingest&host=http://MATTERHORN_URL:PORT
Tomcat
Another Tomcat from a different repo was installed and replaced the old one
In Avalon 5.x and priors, we use a specific version of Tomcat from nul-repo. If a different version (for example one from epel) was installed, it will cause problems. To revert:
- Move old logs: /usr/local/tomcat/logs /usr/local/tomcat/logs.old
Downgrade tomcat to 7.0.32-1.el6 from nul_public: sudo yum downgrade tomcat --disablerepo=epel
Restore server.xml, tomcat-users.xml and web.xml in fedora/WEB-INF with a working versions from github or another server
Restart Tomcat
Fedora
Corrupted thumbnail
Symptom
Item cannot be opened. Fetching the MasterFile in Rails console returns:
Ldp::HttpError: STATUS: 500 org.modeshape.persistence.relational.RelationalProviderException: java.nio.charset.MalformedInputException: Input length = 1
Remedy
The following steps attempt to reverse engineer and modify DB blobs. They should be tested thoroughly on a non-production replica first.
Find the Modeshape uuid for the MasterFile using its noid (eg 4q77fs77w
)
select * from MODESHAPE_REPOSITORY where content like '%4q77fs77w%';
Look in the result and find the last uuid, fcf96b47505d64ecc9ea96-31cf-49c4-b4e3-f4b193723378
in this case
| fcf96b47505d644a95ff85-b6ab-4bdb-a150-6d5f21c9cda0 | 2017-06-08 01:17:35 | h metadata @ id 3 fcf96b47505d644a95ff85-b6ab-4bdb-a150-6d5f21c9cda0 content key 3 fcf96b47505d644a95ff85-b6ab-4bdb-a150-6d5f21c9cda0 parent 3 fcf96b47505d64454d94d5-2fb7-48d9-8847-90a8b0d79f07 properties � http://www.jcp.org/jcr/1.0 � createdBy bypassAdmin primaryType $name nt:folder created . $date 2017-06-08T01:17:35.518-04:00 mixinTypes P 0 H $name 8 {http://fedora.info/definitions/v4/repository#}Pairtree children ] 0 U key 3 fcf96b47505d64ecc9ea96-31cf-49c4-b4e3-f4b193723378 name 4q77fs77w childrenInfo count |
Dump the MasterFile row using this uuid
mysqldump -u mysql_user -h mysql_host -p fedora_db --skip-add-drop-table --no-create-info --where="id = 'fcf96b47505d64ecc9ea96-31cf-49c4-b4e3-f4b193723378'" MODESHAPE_REPOSITORY > masterfile.sql
Edit masterfile.sql
using vim
and delete everything after poster
through thumbnail
\0\0\0poster
\0\0\0^CchildrenInfo\0^T\\0\0^C1\0U\0\0\0^Bkey\03\0\0\0fcf96b47505d643edfc8a7-a84d-46a8-ae6f-38767eed38d3\0^Bname\0\n\0\0\0thumbnail
Drop the existing DB row
delete from MODESHAPE_REPOSITORY where id='fcf96b47505d64ecc9ea96-31cf-49c4-b4e3-f4b193723378';
Insert the updated row back
mysql -u mysql_user -h mysql_host -p fedora_db < masterfile.sql
Open the item in a web browser and recreate the thumbnail using the player's built-in function.