...

caption_default:
  # Language should be 3 letter ISO 639-2 code
  language: 'eng'
  name: 'English'

Troubleshooting

Before starting the upgrade there are some steps to keep a close eye on when running.

Reindex

When running the reindex watch for any errors in the log output. If errors occur the reindex can be rerun by either clearing out the reindexing DB table or resetting the specific failed items and trying them again.

# Clearing out the reindexing DB table
RAILS_ENV=production bundle exec rails r script/reindex.rb -v --drop-table

# Resetting failed items (in rails console)
DB = Sequel.connect(ENV["DATABASE_URL"])
DB[:reindexing_nodes].where(state: "errored").update(state: "waiting reindex")

The order of indexing is important so watch to see if any Hydra::AccessControl or Hydra::AccessControls::Permission objects failed to index. If any did this might impact the access control of items indexed later. If you’re continuing to run into problems the reindex script can be rerun on specific models and without parallelization. This should be the safest way to run the reindex.

# Note the --skip-identification flag avoids having to re-walk the Fedora tree to gather nodes needing to be reindexed
RAILS_ENV=production nohup bundle exec rails r script/reindex.rb -v --parallel-indexing --parallel-threads 1 --only-models "Hydra::AccessControl,Hydra::AccessControls::Permission" --skip-identification &
RAILS_ENV=production nohup bundle exec rails r script/reindex.rb -v --parallel-indexing --parallel-threads 1 --only-models "Admin::Collection" --skip-identification &
RAILS_ENV=production nohup bundle exec rails r script/reindex.rb -v --parallel-indexing --parallel-threads 1 --only-models "MediaObject" --skip-identification &
RAILS_ENV=production nohup bundle exec rails r script/reindex.rb -v --parallel-indexing --parallel-threads 1 --only-models "MasterFile" --skip-identification &
RAILS_ENV=production nohup bundle exec rails r script/reindex.rb -v --parallel-indexing --parallel-threads 1 --only-models "Derivative" --skip-identification &
RAILS_ENV=production nohup bundle exec rails r script/reindex.rb -v --parallel-indexing --parallel-threads 1 --skip-identification &

When reindexing is done be sure to check that the Manage Content page loads for an admin. If it returns a 500 error then rerun the indexing of Admin::Collection objects:

RAILS_ENV=production bundle exec rails r "Admin::Collection.all.to_a.map(&:update_index)"

Caption Migration

Sometimes the caption migration fails to migrate all captions. Before running the migration first take a count of captions:

RAILS_ENV=production bundle exec rails r "puts IndexedFile.where('id: */captions'').count"

Compare this number to the output of the caption migration rake task. If they do not match, try running the migration again until the total “successfully migrated” between all of the runs matches the count from the output before running the migration.

Upgrade Steps

For Manual Installations

1. Install Ruby 3.2.x 

2. Update gems, and migrate DB

   Code Block
   bundle install --with production --without development test RAILS_ENV=production bundle exec rake db:migrate

3. Install node.js 20.x (if needed)

4. Update JS dependencies

   Code Block
   yarn install # in Avalon root dir

5. Update configuration, see Config changes above

6. Restart passenger (if using)

   Code Block
   passenger-config restart-app

7. Install Solr 9 (optional)

   1. Make backup of existing Solr using instructions on https://samvera.atlassian.net/wiki/spaces/AVALON/pages/2303328775/Backing+up+and+restoring+supporting+services#Solr

   2. Setup Solr 9 in a new location and update config files (solr.yml and blacklight.yml) (safest approach)
      Or the same location as previous Solr and clear it out

8. Run reindex
   First download the latest version of the reindex script to script/reindex.rb then run:

   Code Block
   RAILS_ENV=production nohup bundle exec rails r script/reindex.rb -v --parallel-indexing &

9. Run rake migrations

   Code Block
   RAILS_ENV=production bundle exec rake avalon:migrate:collection_managers RAILS_ENV=production bundle exec rake avalon:migrate:caption_files

...

1. Stop Avalon

   Code Block
   docker-compose down

2. Backup solr data using instructions on https://samvera.atlassian.net/wiki/spaces/AVALON/pages/2303328775/Backing+up+and+restoring+supporting+services#Solr

3. Update new code from avalon-docker

   Code Block
   git checkout main git pull sudo chmod a+w solr_data

4. Pull new images

   Code Block
   docker-compose pull avalon solr

5. Check config changes then restart the Avalon containers

   Code Block
   docker-compose up

6. Run reindex (inside avalon container)
   First download the latest version of the reindex script to script/reindex.rb then run:

   Code Block
   RAILS_ENV=production nohup bundle exec rails r script/reindex.rb -v --parallel-indexing &

7. Run rake migrations (inside avalon container)

   Code Block
   RAILS_ENV=production bundle exec rake avalon:migrate:collection_managers RAILS_ENV=production bundle exec rake avalon:migrate:caption_files

...