/
Upgrading Avalon 7.8 to Avalon 8.0

Upgrading Avalon 7.8 to Avalon 8.0

Requirements

Ruby 3.3+ and Node.js 20 are recommended.

Redis 6+ is required.

Config changes

  • Solr 6 configuration files removed
    Ensure solr has been upgraded to solr 9 (see instructions included in Avalon 7.6 → 7.7 upgrade notes)

  • dropbox.sharepoint (options) - settings to enable sharepoint integration in dropbox modal
    See Sharepoint Dropbox for details about setting it up.

  • ffprobe (optional) - path to ffprobe executable on Avalon server (defaults to ffprobe)

Troubleshooting

Restarting export or migrations in case of errors

If you run into issues when running the Fedora 4 export, it can be resumed by re-running the export with a couple different flags: repositoryRoot and resourcesFile instead of resource. You will have to find the remaining_*.log file that matches that last failing run of the exporter.

jdk-21/bin/java -jar fcrepo-import-export-1.2.0.jar -b --dir fcrepo4.7.5_export --user fedoraAdmin:fedoraAdmin --mode export --repositoryRoot http://fedora:8080/fedora/rest --resourcesFile remaining_TIMESTAMP.log --binaries --membership --auditLog > importexport_`date +%Y%m%dT%H%M%S`.log 2>&1

The same approach can be used for the migrations by passing in --resource-info-file and the matching remaining_*.log file.

Checking that export and migration are complete

If the export ran to completion (log file should include (Exporter) Export complete) and there isn’t a remaining_*.log file then the export should be complete.

Upgrade Steps

For Manual Installations

Pre-upgrade steps

  1. Ensure section_list migration has been run (See Upgrading Avalon 7.7 to Avalon 7.8 | MediaObject Section List Migration)

  2. Export Fedora 4 repository and migrate it to Fedora 6 OCFL

    cd /tmp mkdir fc4_to_fc6 cd fc4_to_fc6 # Setup Java 21+ (if needed) curl -O -L https://download.java.net/java/GA/jdk21/fd2272bbf8e04c3dbaee13770090416c/35/GPL/openjdk-21_linux-x64_bin.tar.gz tar xvzf openjdk-21_linux-x64_bin.tar.gz # Download specific version of import/export tool curl -O -L https://github.com/fcrepo-exts/fcrepo-import-export/releases/download/fcrepo-import-export-1.2.0/fcrepo-import-export-1.2.0.jar # Export from fedora 4 jdk-21/bin/java -jar fcrepo-import-export-1.2.0.jar -b --dir fcrepo4.7.5_export --user fedoraAdmin:fedoraAdmin --mode export --resource http://fedora:8080/fedora/rest --binaries --membership --auditLog > importexport_`date +%Y%m%dT%H%M%S`.log 2>&1 # Download specific build of upgrade-utils curl -O -L https://github.com/avalonmediasystem/fcrepo-upgrade-utils/releases/download/6.3.0-AVALON/fcrepo-upgrade-utils-6.3.0-AVALON.jar # Run migration Fedora 4 -> Fedora 5 (inside container again) jdk-21/bin/java -jar fcrepo-upgrade-utils-6.3.0-AVALON.jar --input-dir fcrepo4.7.5_export --output-dir fcrepo5_export --source-version 4.7.5 --target-version 5+ > upgrade_5_`date +%Y%m%dT%H%M%S`.log 2>&1 # Run migration Fedora 5 -> Fedora 6 jdk-21/bin/java --add-opens java.base/java.util.concurrent=ALL-UNNAMED -jar fcrepo-upgrade-utils-6.3.0-AVALON.jar --input-dir fcrepo5_export --output-dir fcrepo6_export --source-version 5+ --target-version 6+ --base-uri http://fedora:8080/fedora/rest > upgrade_6_`date +%Y%m%dT%H%M%S`.log 2>&1

  3. Setup Fedora 6 (See https://wiki.lyrasis.org/display/FEDORA6x/Deployment )

  4. Copy migrated Fedora 6 OCFL directory to fcrepo.home and restart Fedora

Upgrade steps

  1. Install ruby 3.3.x (if needed)

  2. Update gems, and migrate DB

    bundle install --with production --without development test RAILS_ENV=production bundle exec rake db:migrate

  3. Install node.js 20.x (if needed)

  4. Update JS dependencies

    yarn install # in Avalon root dir

  5. Update configuration, see Config changes above

  6. Generate credentials (if needed)

    bundle exec rails credentials:edit

  7. Restart passenger (if using)

    passenger-config restart-app

 

For Docker Installations

Before upgrading an AWS docker install based on avalon-terraform or the aws_min branch of avalon-docker make a backup of your supplemental files prior to destroying the avalon container!

docker cp avalon-docker-aws_min_avalon_1:/home/app/avalon/storage ./active_storage_backup

Then load them into an S3 bucket:

aws s3 cp active_storage_backup/* s3://${SETTINGS__ACTIVE_STORAGE__BUCKET}/ --recursive

See Investigate Supplemental File Storage · Issue #77 · avalonmediasystem/avalon-docker for more discussion on this. If you run into any problems, please reach out on slack or the email list.

Pre-upgrade steps

  1. Ensure section_list migration has been run (See Upgrading Avalon 7.7 to Avalon 7.8 | MediaObject Section List Migration)

  2. Export Fedora 4 repository and migrate it to Fedora 6 OCFL (inside avalon container)

    cd /tmp mkdir fc4_to_fc6 cd fc4_to_fc6 # Setup Java curl -O -L https://download.java.net/java/GA/jdk21/fd2272bbf8e04c3dbaee13770090416c/35/GPL/openjdk-21_linux-x64_bin.tar.gz tar xvzf openjdk-21_linux-x64_bin.tar.gz # Download specific version of import/export tool curl -O -L https://github.com/fcrepo-exts/fcrepo-import-export/releases/download/fcrepo-import-export-1.2.0/fcrepo-import-export-1.2.0.jar # Export from fedora 4 jdk-21/bin/java -jar fcrepo-import-export-1.2.0.jar -b --dir fcrepo4.7.5_export --user fedoraAdmin:fedoraAdmin --mode export --resource http://fedora:8080/fedora/rest --binaries --membership --auditLog > importexport_`date +%Y%m%dT%H%M%S`.log 2>&1 # Download specific build of upgrade-utils curl -O -L https://github.com/avalonmediasystem/fcrepo-upgrade-utils/releases/download/6.3.0-AVALON/fcrepo-upgrade-utils-6.3.0-AVALON.jar # Run migration Fedora 4 -> Fedora 5 (inside container again) jdk-21/bin/java -jar fcrepo-upgrade-utils-6.3.0-AVALON.jar --input-dir fcrepo4.7.5_export --output-dir fcrepo5_export --source-version 4.7.5 --target-version 5+ > upgrade_5_`date +%Y%m%dT%H%M%S`.log 2>&1 # Run migration Fedora 5 -> Fedora 6 jdk-21/bin/java --add-opens java.base/java.util.concurrent=ALL-UNNAMED -jar fcrepo-upgrade-utils-6.3.0-AVALON.jar --input-dir fcrepo5_export --output-dir fcrepo6_export --source-version 5+ --target-version 6+ --base-uri http://fedora:8080/fedora/rest > upgrade_6_`date +%Y%m%dT%H%M%S`.log 2>&1

  3. Copy migrated data out into new fedora_data directory (outside avalon container)

    docker cp avalon-docker-avalon-1:/tmp/fc4_to_fc6/fcrepo6_export/data fedora_data

Upgrade steps

  1. Stop Avalon

    docker-compose down

  2. Update new code from avalon-docker

    git checkout main git pull

  3. Pull new images

    docker-compose pull

  4. Setup fedora 6 cache database

    docker-compose up db -d docker-compose exec db /bin/bash /docker-entrypoint-initdb.d/init-fedora-cache-db.sh docker-compose down

  5. Check config changes then restart the Avalon containers

    docker-compose up

You can also build your own image, Docker Buildkit is recommended

docker-compose build avalon

 

For Terraform installations

These instructions are a work in progress.

Backup all data including the EC2 filesystem and s3 buckets (if possible) prior to applying the new terraform as it will destroy and rebuild most of the infrastructure including the EC2 instance.

Before upgrading an AWS docker install based on avalon-terraform or the aws_min branch of avalon-docker make a backup of your supplemental files prior to destroying the avalon container!

docker cp avalon-docker-aws_min_avalon_1:/home/app/avalon/storage ./active_storage_backup

Then load them into an S3 bucket:

aws s3 cp active_storage_backup/* s3://${SETTINGS__ACTIVE_STORAGE__BUCKET}/ --recursive

See Investigate Supplemental File Storage · Issue #77 · avalonmediasystem/avalon-docker for more discussion on this. If you run into any problems, please reach out on slack or the email list.

Pre-upgrade steps

  1. Ensure section_list migration has been run (See Upgrading Avalon 7.7 to Avalon 7.8 | MediaObject Section List Migration)

  2. Export Fedora 4 repository and migrate it to Fedora 6 OCFL (inside avalon container)

    cd /tmp mkdir fc4_to_fc6 cd fc4_to_fc6 # Setup Java curl -O -L https://download.java.net/java/GA/jdk21/fd2272bbf8e04c3dbaee13770090416c/35/GPL/openjdk-21_linux-x64_bin.tar.gz tar xvzf openjdk-21_linux-x64_bin.tar.gz # Download specific version of import/export tool curl -O -L https://github.com/fcrepo-exts/fcrepo-import-export/releases/download/fcrepo-import-export-1.2.0/fcrepo-import-export-1.2.0.jar # Export from fedora 4 jdk-21/bin/java -jar fcrepo-import-export-1.2.0.jar -b --dir fcrepo4.7.5_export --user fedoraAdmin:fedoraAdmin --mode export --resource http://fedora:8080/rest --binaries --membership --auditLog > importexport_`date +%Y%m%dT%H%M%S`.log 2>&1 # Download specific build of upgrade-utils curl -O -L https://github.com/avalonmediasystem/fcrepo-upgrade-utils/releases/download/6.3.0-AVALON/fcrepo-upgrade-utils-6.3.0-AVALON.jar # Run migration Fedora 4 -> Fedora 5 (inside container again) jdk-21/bin/java -jar fcrepo-upgrade-utils-6.3.0-AVALON.jar --input-dir fcrepo4.7.5_export --output-dir fcrepo5_export --source-version 4.7.5 --target-version 5+ > upgrade_5_`date +%Y%m%dT%H%M%S`.log 2>&1 # Run migration Fedora 5 -> Fedora 6 jdk-21/bin/java --add-opens java.base/java.util.concurrent=ALL-UNNAMED -jar fcrepo-upgrade-utils-6.3.0-AVALON.jar --input-dir fcrepo5_export --output-dir fcrepo6_export --source-version 5+ --target-version 6+ --base-uri http://fedora:8080/rest > upgrade_6_`date +%Y%m%dT%H%M%S`.log 2>&1

  3. (optional) Tar up export and migrations for backup (still in avalon container):

    tar cvzf fcrepo4_export.tgz fcrepo4.7.5_export tar cvzf fcrepo5_export.tgz fcrepo5_export tar cvzf fcrepo6_export.tgz fcrepo6_export

  4. Update avalon-terraform on your local machine to the latest in main and run the following to create the s3 bucket for the Fedora 6 OCFL data store:

    terraform apply --target=aws_s3_bucket.fcrepo_ocfl_bucket

  5. Outside of the avalon container copy migrated Fedora data to the new s3 bucket:

    docker cp avalon-docker-avalon-1:/tmp/fc4_to_fc6/fcrepo6_export/data fedora_data aws s3 sync fedora_data/ocfl-root s3://avalon-dev-fedora-ocfl/

  6. (optional) Put export/migration backups into s3:

    mkdir fedora_data/backups docker cp avalon-docker-avalon-1:/tmp/fc4_to_fc6/fcrepo4_export.tgz fedora_data/backups docker cp avalon-docker-avalon-1:/tmp/fc4_to_fc6/fcrepo5_export.tgz fedora_data/backups docker cp avalon-docker-avalon-1:/tmp/fc4_to_fc6/fcrepo6_export.tgz fedora_data/backups aws s3 sync fedora_data/backups s3://avalon-dev-fedora-ocfl/backups

Upgrade steps

  1. Set avalon_docker_code_branch and avalon_branch in terraform.tfvars if necessary

  2. Perform the upgrade:

    terraform apply

    This might error on destroying fedora 4 s3 buckets if they aren't empty. You can safely ignore these errors and decide later if you want to backup and retain these buckets or delete them.

    You may also see an error when attempting to create an iam user which already exists. If this happens rerunning the terraform apply should resolve the issue and successfully run to completion