These instructions provide a recipe for building your own all-in-one Avalon system from scratch on CentOS or Red Hat Enterprise Linux version 7.x.  Please note that while an all-in-one installation as outlined here is certainly suitable for testing and demos, a single, all-in-one, server may not be suitable for production environments.

Ready the Installation Environment

Storage requirement

Avalon and components need about 20GB of disk space to install.

Open ports requirement

The Avalon Media System requires several ports to be open to client browsers.

Here are the port settings that will need to be configured:

...

The preferred method is to create a shell script that will do the work for you. Here is an example script that you should look through and customize as needed: avalon-iptables-config.sh

Save your script to /etc/sysconfig/avalon-iptables-config.sh, make it executable and run it.

chmod +x /etc/sysconfig/avalon-iptables-config.sh
/etc/sysconfig/avalon-iptables-config.sh

If you run into connection issues you can disable the iptables, by running "service iptables stop". This will completely drop your firewall. When finished troubleshooting run "service iptables start".

Disable SELinux

vim /etc/selinux/config #change the value of `SELINUX` from `enforcing` to `permissive`

You may have to disable SELinux completely if there's Passenger installation problem

vim /etc/selinux/config #change the value of `SELINUX` to `disabled`

Reboot to apply change

shutdown -r now

Install EPEL

This package has libyaml-devel which is required by ruby and not provided by Redhat.

yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

Install development libraries and packages for building Ruby

yum groupinstall "Development Tools"
yum install readline-devel zlib-devel libyaml-devel libffi-devel openssl-devel libxml2-devel libxslt-devel cmake

Install Java 8

yum install java-1.8.0-openjdk

Main Components

MariaDB

Avalon uses MariaDB for storing search queries, user data and roles, and as a back end our encoding dashboard. 

Install MariaDB server 

yum install mariadb-server
systemctl start mariadb

Fedora Commons Repository

Tomcat

Fedora runs as a webapp in Tomcat

Install Apache Tomcat 

yum install tomcat
vim /etc/tomcat/server.xml #line 71, change the Tomcat connector port from 8080 to 8984

Add Tomcat manager user

By default, no user has access to the Tomcat Manager App. Define a user in /etc/tomcat/tomcat-users.xml with access to the manager-gui role. Below is a very basic example.

<tomcat-users>
  <role rolename="manager-gui"/>
  <user username="admin" password="<insert strong password here>" roles="manager-gui"/>
</tomcat-users>

Create Fedora user and database

Enter the mariadb client

...

These instructions provide a recipe for building your own all-in-one Avalon system from scratch on CentOS or Red Hat Enterprise Linux version 7.x.  Please note that while an all-in-one, single machine installation as outlined here is certainly suitable for testing and demos, production environments are typically implemented across multiple servers running the main components of the application.

Ready the Installation Environment

Storage requirement

Avalon and components need about 20GB of disk space to install.

Open ports requirement

The Avalon Media System requires several ports to be open to client browsers.

Here are the port settings that will need to be configured:

The preferred method is to create a shell script that will do the work for you. Here is an example script that you should look through and customize as needed: avalon-iptables-config.sh

Save your script to /etc/sysconfig/avalon-iptables-config.sh, make it executable and run it.

chmod +x /etc/sysconfig/avalon-iptables-config.sh
/etc/sysconfig/avalon-iptables-config.sh

If you run into connection issues you can disable the iptables, by running "service iptables stop". This will completely drop your firewall. When finished troubleshooting run "service iptables start".

Disable SELinux

vim /etc/selinux/config #change the value of `SELINUX` from `enforcing` to `permissive`

You may have to disable SELinux completely if there's Passenger installation problem

vim /etc/selinux/config #change the value of `SELINUX` to `disabled`

Reboot to apply change

shutdown -r now

Install EPEL

This package has libyaml-devel which is required by ruby and not provided by Redhat.

yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

Install development libraries and packages for building Ruby

yum groupinstall "Development Tools"
yum install readline-devel zlib-devel libyaml-devel libffi-devel openssl-devel libxml2-devel libxslt-devel cmake

Install Java 8

yum install java-1.8.0-openjdk

Main Components

MariaDB

Avalon uses MariaDB for storing search queries, user data and roles, and as a back end our encoding dashboard. 

Install MariaDB server 

yum install mariadb-server
systemctl start mariadb

Fedora Commons Repository

Tomcat

Fedora runs as a webapp in Tomcat

Install Apache Tomcat 

yum install tomcat
vim /etc/tomcat/server.xml #line 71, change the Tomcat connector port from 8080 to 8984

Add Tomcat manager user

By default, no user has access to the Tomcat Manager App. Define a user in /etc/tomcat/tomcat-users.xml with access to the manager-gui role. Below is a very basic example.

<tomcat-users>
  <role rolename="manager-gui"/>
  <user username="admin" password="<insert strong password here>" roles="manager-gui"/>
</tomcat-users>

Create Fedora user and database

Enter the mariadb client

mysql
mariadb> create database fcrepo CHARACTER SET utf8 COLLATE utf8_general_ci;
mariadb> create user 'fcrepo'@'localhost' identified by '<fcrepo_password>';
mariadb> grant all privileges on fcrepo.* to 'fcrepo'@'localhost';

mariadb> create database rails CHARACTER SET utf8 COLLATE utf8_general_ci;
mariadb> create user 'rails'@'localhost' identified by '<rails_pasword>';
mariadb> grant all privileges on rails.* to 'rails'@'localhost';

mariadb> flush privileges;

Check your work and exit

...

Try it out on your local machine and on another machine. If you can't reach the app from another machine, your iptables1957955315 might need to be changed to allow access. If Fedora is not up, check the tomcat logs in /var/log/tomcat/. Catalina.out and localhost.<date>.log usually provide the best information.

...

mkdir -p /tmp/avalon_solr/
wget https://raw.githubusercontent.com/avalonmediasystem/avalon/mastermain/solr/config/solrconfig.xml -O /tmp/avalon_solr/solrconfig.xml
wget https://raw.githubusercontent.com/avalonmediasystem/avalon/mastermain/solr/config/schema.xml -O /tmp/avalon_solr/schema.xml
su solr # Needs to run as solr user
/opt/solr/bin/solr create_core -c avalon -d /tmp/avalon_solr
exit

...

FFmpeg & Mediainfo

Download and install ffmpeg

...

(for transcoding & thumbnails)

mkdir -p /tmp/ffmpeg && cd /tmp/ffmpeg
curl https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz | tar xJ
cp `find . -type f -executable` /usr/bin/

Install Mediainfo (for technical metadata)

yum install mediainfo

HTTPD

Install and start the httpd service.

...

yum install ruby sqlite-devel # Needed to build Ruby using RVM.
useradd avalon
su - avalon
curl -L https://get.rvm.io | bash -s stable --ruby=2.5.75

Source the RVM shell (as avalon user) or close the terminal and open it back up.

source /home/avalon/.rvm/scripts/rvm
rvm use 2.5.75
exit

Install Passenger apache module requirements (as root)

su - root
yum install -y pygpgme curl
yum install -y mod_passenger || yum-config-manager --enable cr && yum install -ycurl --fail -sSLo /etc/yum.repos.d/passenger.repo https://oss-binaries.phusionpassenger.com/yum/definitions/el-passenger.repo

yum install mod_passenger || yum-config-manager --enable cr && yum install mod_passenger

Create a virtual host for avalon

wget --no-check-certificate https://raw.github.com/avalonmediasystem/config-files/mastermain/apache/20-avalon.conf -P /etc/httpd/conf.d/
vim /etc/httpd/conf.d/20-avalon.conf

...

SSLProtocol all -SSLv2 -SSLv3
SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:!RC4:+HIGH:+MEDIUM:-LOW

Restart apache.  With apache running, check passenger-status

...

Modify /etc/httpd/conf.d/passenger.conf

PassengerRuby /home/avalon/.rvm/rubies/ruby-2.5.5/bin/ruby

Validate passenger install and restart apache

passenger-config validate-install
systemctl start httpd
su - avalon
passenger-status

Avalon

Grab Avalon code from github

su - root
git clone https://github.com/avalonmediasystem/avalon.git /var/www/avalon
chown -R avalon:avalon /var/www/avalon/public/

...

Set rails environment to production, if it has not defaulted to this. On the first line of /var/www/avalon/config/environment.rb make sure it says 'production'

ENV['RAILS_ENV'] ||= 'production'

Configure database settings

Anchor
mysql2 mysql2

cd /var/www/avalon/config
vim database.yml

Replace database.yml with the correct values for your production environment

...

. Note that the pool setting should be equal or exceed the number of concurrent jobs in Sidekiq.

production:
  adapter: mysql2
  host: localhost
  database: rails
  username: rails
  password: rails
  pool: 520
  timeout: 5000

Install the mysql2 adapter 

...

# as root
yum install nodejs # Javascript runtime 

# as avalon
su - avalon
cd /var/www/avalon
gem install bundler
bundle install --with mysql production --without development test

...

exit

Finish configuring Avalon

Edit /var/www/avalon/config/solr.yml and /var/www/avalon/config/blacklight.yml

#  asproduction:
root curl --silent --location httpsurl: http://dl.yarnpkg.com/rpm/yarn.repo | sudo tee /etc/yum.repos.d/yarn.repo
yum install yarn

# as avalon
yarn install

Finish configuring Avalon

...

localhost:8983/solr/avalon

Edit /var/www/avalon/config/solrfedora.ymland /var/www/avalon/config/blacklight.yml

  production:
    url: http://localhost:8983/solr/avalon

Edit /var/www/avalon/config/fedora.yml

  production:
    useruser: fedoraAdmin
    password: fedoraPasswordfedoraAdmin
    url: http://127.0.0.1:8984/fedora4/rest
    base_path: ""

Create streaming directory

# as root
mkdir -p /var/

avalon/derivatives
chown avalon:avalon /var/avalon/derivatives

...

streaming:
  server: :nginx
  http_base: 'http://localhost:8980/avalon'
  content_path: '/var/avalon/derivatives'

cd /var/www/avalon
export RAILS_ENV=production
rake secret

grab the output of rake secret and add it to secrets.yml where instructedinstruSTDOUTSTDOUTSTDOUTcted.

More information: Configuration Files#config/secrets.yml

Create controlled_vocabulary.yml

cp config/controlled_vocabulary.yml.example config/controlled_vocabulary.yml

Create the database using rake

# as avalon user
rake db:create

If you get an error message saying that you can't connect to the database, take a look at this post and follow some of the troubleshooting steps.

...

rake db:migrate

Set rails environment to production, if it has not defaulted to this. On the first line of /var/www/avalon/config/environment.rb make sure it says 'production'

ENV['RAILS_ENV'] ||= 'production'

Sidekiq

Avalon uses Sidekiq for background processing, which relies on Redis as its key-value store.

Install Redis

yum install redis

Install Sidekiq

# as root
wget https://raw.githubusercontent.com/mperham/sidekiq/master/examples/systemd/sidekiq.service -O /lib/systemd/system/sidekiq.service

Edit the following lines in sidekiq.service 

...

Install yarn and node modules

# as root
curl --silent --location https://dl.yarnpkg.com/rpm/yarn.repo | sudo tee /etc/yum.repos.d/yarn.repo
yum install yarn

# as avalon
su - avalon
cd /var/www/avalon
yarn install

Precompile assets

# as avalon
RAILS_ENV=production bundle exec rake assets:precompile

Restart Apache

# as root
systemctl startrestart sidekiq

Sidekiq logs STDOUT. 

Additional Configurations

...

httpd

Install ImageMagick

groupadd# -ras dropboxroot
useraddyum -r avalondrop
usermod -G dropbox avalon
mkdir -p /srv/avalon/dropbox
chown avalondrop:dropbox /srv/avalon/dropbox
chmod 2775 /srv/avalon/dropbox

Edit /etc/ssh/sshd_config

# override default of no subsystems
Subsystem sftp internal-sftp
 
# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# ForceCommand cvs server
Match Group dropbox
ChrootDirectory /srv/avalon
X11Forwarding no
AllowTcpForwarding no
ForceCommand internal-sftp

Restart SSH

service sshd restart

Batch ingest

To manually start a batch ingest job, run as avalon user

rake avalon:batch:ingest

To make batch ingest run automatically whenever a manifest is present, you need to add a cron job. This cron job can be created by the whenever gem from reading config/schedule.rb. To preview, run

whenever

this will translate content in schedule.rb to cron job syntax. Once verified, run the following to write job to crontab

whenever --update-crontab

...

install imagemagick

Sidekiq

Avalon uses Sidekiq for background processing, which relies on Redis as its key-value store.

Install Redis

# as root
yum install redis
systemctl start redis

Install Sidekiq

# as root
wget https://raw.githubusercontent.com/mperham/sidekiq/main/examples/systemd/sidekiq.service -O /lib/systemd/system/sidekiq.service

Edit the following lines in sidekiq.service 

WorkingDirectory=/var/www/avalon
ExecStart=/bin/bash -lc '/home/avalon/.rvm/gems/ruby-2.5.5/bin/bundle exec sidekiq -e production'
User=avalon
Group=avalon

# as root
systemctl start sidekiq

Sidekiq logs to STDOUT. 

Additional Configurations

Dropbox

groupadd -r dropbox
useradd -r avalondrop
usermod -G dropbox avalon
mkdir -p /srv/avalon/dropbox
chown avalondrop:dropbox /srv/avalon/dropbox
chmod 2775 /srv/avalon/dropbox

Edit /etc/ssh/sshd_config

# override default of no subsystems
Subsystem sftp internal-sftp
 
# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# ForceCommand cvs server
Match Group dropbox
ChrootDirectory /srv/avalon
X11Forwarding no
AllowTcpForwarding no
ForceCommand internal-sftp

Restart SSH

service sshd restart

Batch ingest

To manually start a batch ingest job, run as avalon user

rake avalon:batch:ingest

To make batch ingest run automatically whenever a manifest is present, you need to add a cron job. This cron job can be created by the whenever gem from reading config/schedule.rb. To preview, run

whenever

this will translate content in schedule.rb to cron job syntax. Once verified, run the following to write job to crontab

whenever --update-crontab

You should get the cron job automatically if you were deploying from Capistrano.

Authentication Strategy

Avalon comes with Persona by default but it can be configured to work with other authentication strategies by using the appropriate omniauth gems. The following example is applicable to Indiana University CAS, it may need some adjustments in order to work with other CAS implementation.

Add to Gemfile

gem 'net-ldap'
gem 'omniauth-cas', :git => "https://github.com/cjcolvar/omniauth-cas.git"

Install new gems

bundle install

Add to config/initializers/my-ldap.rb

module Avalon
      MY_GUEST_LDAP = Net::LDAP.new
      MY_GUEST_LDAP.host = "eads.myuni.edu"
      MY_GUEST_LDAP.authenticate 'cn=******,ou=Accounts,dc=eads,dc=myuni,dc=edu', '******'

      GROUP_LDAP = Net::LDAP.new
      GROUP_LDAP.host = "ads.myuni.edu"
      GROUP_LDAP.authenticate 'cn=******,ou=Accounts,dc=ads,dc=myuni,dc=edu', '******'
      GROUP_LDAP_TREE = "dc=ads,dc=myuni,dc=edu"
end

Add config/initializers/user_auth_cas.rb

require 'net/ldap'

User.instance_eval do
  def self.find_for_cas(access_token, signed_in_resource=nil)
    logger.debug "#{access_token.inspect}"
    #data = access_token.info
    username = access_token.uid
    email = nil

    user = User.where(:username => username).first

    unless user
      if email.nil?
        tree = "dc=ads,dc=myuni,dc=edu"
        filter = Net::LDAP::Filter.eq("cn", "#{username}")
        email = Avalon::GROUP_LDAP.search(:base => tree, :filter => filter, :attributes=> ["mail"]).first.mail.first
      end
      user = User.find_or_create_by_username_or_email(username, email)
      raise "Finding user (#{ user }) failed: #{ user.errors.full_messages }" unless user.persisted?
    end
    user
  end
end

Add to config/settings/production.local.yml

auth:
  configuration:
    - :name: My University
      :logo: my_logo.png
      :provider: :cas
      :params:
        :host: cas.myuni.edu
        :login_url: /cas/login
        :service_validate_url: /cas/validate
        :logout_url: /cas/logout
        :ssl: true

Using the System

You should be able to visit the webpage with just the hostname (ie http://localhost)

Create an admin account

Click on "Sign in" in the upper right corner of the website main page. Set up a default identity with administrative privileges using the following properties.

archivist1@example.com
<some password>

...

You can  create an account from the command line in the root of your avalon install:

...

You can find specific information about using the system in the Collection Manager's Guide.  /wiki/spaces/AVALON/pages/1957954522 is available for your convenience.  Upload new items individually or by batch directly via SFTP using the avalondrop account you created above.

Configure additional feataures

Known Issues - a list of bugs, workarounds, and cautions.

...

Code Block
chkconfig --level 345 tomcat on chkconfig --level 345 mariadb on chkconfig --level 345 nginx on chkconfig --level 345 sshd on chkconfig --level 345 redis on chkconfig --level 345 sidekiq on chkconfig --level 345 httpd on

...