Note |
---|
This documentation is for Release 6.x. For documentation on previous releases, please select from the options below. |
These instructions provide a recipe for building your own all-in-one Avalon system from scratch on CentOS or Red Hat Enterprise Linux, version 6.x is supported, 7.x will be supported soon. Please note that while an all-in-one installation as outlined here is certainly suitable for testing and demos, a single, all-in-one, server may not be suitable for production environments.
Table of Contents | ||
---|---|---|
|
Ready the Installation Environment
Make sure a valid hostname is resolvable
The default hostname is “avalon.dev”, so name the machine this and enter it into /etc/hosts
Code Block |
---|
# hostname
avalon.dev
# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 avalon.dev |
...
The Avalon Media System requires several ports to be open to client browsers.
Here are the port settings that will need to be configured:
Port | Purpose | External? |
---|---|---|
80 | HTTP (Avalon) | Yes |
1935 | RTMP (red5) | Yes |
5080 | HTTP (red5) | No |
8983 | HTTP (Solr) | No |
8984 | HTTP (Fedora) | No |
8080 | HTTP (Matterhorn) | Yes |
The preferred method is to create a shell script that will do the work for you. Here is an example script that you should look through and customize as needed: avalon-iptables-config.sh
Warning |
---|
If you're connected over ssh, it might kick you off. |
Save your script to /etc/sysconfig/avalon-iptables-config.sh, make it executable and run it.
Code Block |
---|
chmod +x /etc/sysconfig/avalon-iptables-config.sh
/etc/sysconfig/avalon-iptables-config.sh |
If you run into connection issues you can disable the iptables, by running "service iptables stop". This will completely drop your firewall. When finished troubleshooting run "service iptables start".
Disable SELinux
Code Block |
---|
echo 0 > /selinux/enforce
vim /etc/selinux/config #change the value of `SELINUX` from `enforcing` to `permissive` |
Info | ||
---|---|---|
| ||
vim /etc/selinux/config #change the value of `SELINUX` to `disabled` |
Install EPEL
This package has libyaml-devel which is required by ruby and not provided by Redhat.
Code Block |
---|
rpm -ivh http://linux.mirrors.es.net/fedora-epel/6/i386/epel-release-6-8.noarch.rpm |
...
Create the Avalon repository config file:
Code Block |
---|
vim /etc/yum.repos.d/avalon-public.repo |
Append the following code:
Code Block |
---|
[avalon_public]
name=Avalon Public RHEL repository
baseurl=http://repo.avalonmediasystem.org/x86_64
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-avalon
cost=150 |
Install and place the Avalon GPG key in the proper location:
Code Block |
---|
curl http://repo.avalonmediasystem.org/RPM-GPG-KEY-avalon -o /etc/pki/rpm-gpg/RPM-GPG-KEY-avalon |
Install development libraries and packages for building Ruby
Code Block |
---|
yum groupinstall "Development Tools"
yum install readline-devel zlib-devel libyaml-devel libffi-devel openssl-devel libxml2-devel libxslt-devel cmake |
Install Java 8
Code Block |
---|
yum install java-1.8.0-openjdk |
Main Components
Fedora Commons Repository
Tomcat
Fedora runs as a webapp in Tomcat
Install Apache Tomcat
Code Block | ||
---|---|---|
| ||
yum install tomcat
vim /etc/tomcat/server.xml #line 71, change the Tomcat connector port from 8080 to 8984
|
Add Tomcat manager user
By default, no user has access to the Tomcat Manager App. Define a user in /etc/tomcat/tomcat-users.xml
with access to the manager-gui role. Below is a very basic example.
Code Block |
---|
<tomcat-users>
<role rolename="manager-gui"/>
<user username="admin" password="<insert strong password here>" roles="manager-gui"/>
</tomcat-users> |
Configure Tomcat for Fedora
Append the following to /etc/sysconfig/tomcat
...
These instructions provide a recipe for building your own all-in-one Avalon system from scratch on CentOS or Red Hat Enterprise Linux version 7.x. Please note that while an all-in-one, single machine installation as outlined here is certainly suitable for testing and demos, production environments are typically implemented across multiple servers running the main components of the application.
Table of Contents | ||
---|---|---|
|
Ready the Installation Environment
Info |
---|
The instructions below require being run as root unless specifically noted otherwise. |
Storage requirement
Avalon and components need about 20GB of disk space to install.
Open ports requirement
The Avalon Media System requires several ports to be open to client browsers.
Here are the port settings that will need to be configured:
Port | Purpose | External? |
---|---|---|
80 | HTTP (Avalon) | Yes |
8983 | HTTP (Solr) | No |
8984 | HTTP (Fedora) | No |
8980 | HTTP (Nginx) | Yes |
The preferred method is to create a shell script that will do the work for you. Here is an example script that you should look through and customize as needed: avalon-iptables-config.sh
Warning |
---|
If you're connected over ssh, it might kick you off. |
Save your script to /etc/sysconfig/avalon-iptables-config.sh, make it executable and run it.
Code Block |
---|
chmod +x /etc/sysconfig/avalon-iptables-config.sh
/etc/sysconfig/avalon-iptables-config.sh |
If you run into connection issues you can disable the iptables, by running "service iptables stop". This will completely drop your firewall. When finished troubleshooting run "service iptables start".
Disable SELinux
Code Block |
---|
vim /etc/selinux/config #change the value of `SELINUX` from `enforcing` to `permissive` |
You may have to disable SELinux completely if there's Passenger installation problem
Info |
---|
vim /etc/selinux/config #change the value of `SELINUX` to `disabled` |
Reboot to apply change
Code Block |
---|
shutdown -r now |
Install EPEL
This package has libyaml-devel which is required by ruby and not provided by Redhat.
Code Block |
---|
yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm |
Install development libraries and packages for building Ruby
Code Block |
---|
yum groupinstall "Development Tools"
yum install readline-devel zlib-devel libyaml-devel libffi-devel openssl-devel libxml2-devel libxslt-devel cmake |
Install Java 8
Code Block |
---|
yum install java-1.8.0-openjdk |
Main Components
MariaDB
Info | ||
---|---|---|
| ||
MariaDB is now the default database system for CentOS/RHEL7 and can be used interchangeably with MySql. MySql or PostgreSQL can be substituted if desired. |
Avalon uses MariaDB for storing search queries, user data and roles, and as a back end our encoding dashboard.
Install MariaDB server
Code Block |
---|
yum install mariadb-server
systemctl start mariadb |
Fedora Commons Repository
Tomcat
Fedora runs as a webapp in Tomcat
Install Apache Tomcat
Code Block | ||
---|---|---|
| ||
yum install tomcat
vim /etc/tomcat/server.xml #line 71, change the Tomcat connector port from 8080 to 8984
|
Add Tomcat manager user
By default, no user has access to the Tomcat Manager App. Define a user in /etc/tomcat/tomcat-users.xml
with access to the manager-gui role. Below is a very basic example.
Code Block |
---|
<tomcat-users>
<role rolename="manager-gui"/>
<user username="admin" password="<insert strong password here>" roles="manager-gui"/>
</tomcat-users> |
Create Fedora user and database
Enter the mariadb client
Code Block |
---|
mysql
mariadb> create database fcrepo CHARACTER SET utf8 COLLATE utf8_general_ci;
mariadb> create user 'fcrepo'@'localhost' identified by '<fcrepo_password>';
mariadb> grant all privileges on fcrepo.* to 'fcrepo'@'localhost';
mariadb> create database rails CHARACTER SET utf8 COLLATE utf8_general_ci;
mariadb> create user 'rails'@'localhost' identified by '<rails_pasword>';
mariadb> grant all privileges on rails.* to 'rails'@'localhost';
mariadb> flush privileges; |
Check your work and exit
Code Block |
---|
mariadb> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| fcrepo |
| performance_schema |
| test |
+--------------------+
5 rows in set (0.00 sec)
mariadb> exit;
Bye |
Configure Tomcat for Fedora
Append the following to /etc/tomcat/tomcat.conf
Code Block |
---|
JAVA_OPTS="-Dfcrepo.modeshape.configuration=classpath:/config/filejdbc-simplemysql/repository.json -Dfcrepo.mysql.home=/username=fcrepo -Dfcrepo.mysql.password=<fcrepo_password> -Dfcrepo.mysql.host=localhost -Dfcrepo.mysql.port=3306 -Dfcrepo.home=/var/avalon/fedora/" |
Restart Tomcat
Code Block |
---|
service tomcat restart |
Download and run the fcrepo installer
Code Block |
---|
mkdir -p /var/avalon/fedora chown tomcat:tomcat /var/avalon/fedora wget https://github.com/fcrepo4/fcrepo4/releases/download/fcrepo-4.7.35/fcrepo-webapp-4.7.35.war -O /usr/share/tomcat/webapps/fedora4.war |
Restart Tomcat
Code Block |
---|
systemctl restart tomcat |
See if you can access Fedora's REST interface at http://<server host name>:8984/fedora4/rest
Try it out on your local machine and on another machine. If you can't reach the app from another machine, your iptables1957955315 might need to be changed to allow access. If Fedora is not up, check the tomcat logs in /var/log/tomcat/. Catalina.out and localhost.<date>.log usually provide the best information.
...
Code Block |
---|
wget http://archive.apache.org/dist/lucene/solr/6.46.26/solr-6.46.26.tgz tar xzf solr-6.46.26.tgz solr-6.46.26/bin/install_solr_service.sh --strip-components=2 bash ./install_solr_service.sh solr-6.46.26.tgz |
By default, the script extracts the distribution archive into /opt
, configures Solr to write files into /var/solr
, and runs Solr as the solr
user. Follow the linked guide if you wish to change these defaults.
...
Code Block |
---|
mkdir -p /tmp/avalon_solr/ wget https://raw.githubusercontent.com/avalonmediasystem/avalon/mastermain/solr/config/solrconfig.xml -O /tmp/avalon_solr/solrconfig.xml wget https://raw.githubusercontent.com/avalonmediasystem/avalon/mastermain/solr/config/schema.xml -O /tmp/avalon_solr/schema.xml su solr # Needs to run as solr user /opt/solr/bin/solr create_core -c avalon -d /tmp/avalon_solr exit |
...
Instructions on how to manually start/stop Solr: https://cwiki.apache.org/confluence/display/solr/Running+Solr
MySQL
Info | ||
---|---|---|
| ||
MariaDB is now the default database system for CentOS/RHEL7. Feel free to change Mysql below to MariaDB |
Avalon uses MySQL for storing search queries, user data and roles, and as a back end for asynchronously sending requests to Matterhorn.
Install MySQL server
Code Block |
---|
yum install mysql-server
service mysqld start |
Create databases and users
Enter the mysql monitor
Code Block |
---|
#mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
...etc...
mysql> |
Create a database for the Avalon web application and add a user to it
Code Block |
---|
create database rails;
create user 'rails'@'localhost' identified by 'rails';
grant all privileges on rails.* to 'rails'@'localhost';
flush privileges; |
Check your work and exit
...
Media Streaming Server
An HLS-enabled server like Wowza, Adobe Media Server (commercial) or Nginx + the HLS module (open-source) can take an mp4 created by Avalon and stream it on the fly.
Nginx instructions
Code Block | ||||
---|---|---|---|---|
| ||||
rpm -ihv http://installrepo.kaltura.org/releases/kaltura-release.noarch.rpm
yum install kaltura-nginx
|
Add /etc/nginx/nginx.conf
Code Block |
---|
user nginx; worker_processes 4; events { worker_connections 1024; } http { server { listen 8980; vod_mode local; vod_last_modified 'Sun, 19 Nov 2000 08:52:00 GMT'; vod_last_modified_types *; vod_metadata_cache metadata_cache 512m; vod_response_cache response_cache 128m; gzip on; gzip_types application/vnd.apple.mpegurl; open_file_cache max=1000 inactive=5m; open_file_cache_valid | +--------------------+ 5 rows2m; in set (0.00 sec) mysql> select user, host from mysql.user; +--------+--------------+ | user | host open_file_cache_min_uses 1; open_file_cache_errors on; location ~ ^/avalon/(?<stream>.+)/(?<resource>.+\.(?:m3u8|ts)) { alias /var/avalon/derivatives/$stream; | +--------+--------------+ | root vod hls; | 127.0.0.1 set | |$token "$arg_token"; | 129.79.32.87 | | rootadd_header X-Stream-Auth-Token "$token"; | 129.79.32.87 | | sub_filter_types application/vnd.apple.mpegurl; | localhost | | railssub_filter_once off; | localhost | | rootsub_filter '.ts' ".ts?token=$token"; | localhost | +--------+--------------+ 7 rows in set (0.00 sec) mysql> exit; Bye |
See documentation for your version of MySQL Server for detailed syntax (http://dev.mysql.com/doc/refman/5.1/en/create-database.html )
Red5 Media Server
Red5 is an open source alternative to Adobe Media Server. If using the Adobe Media Server you can skip to the next step.
Create a red5 user.
Code Block |
---|
useradd red5 |
Download and install Red5.
Code Block |
---|
wget http://repo.avalonmediasystem.org/red5-1.0.1.tar.gz
tar xvf red5-1.0.1.tar.gz
mv red5-server-1.0 /usr/local/red5 |
Download the init script and add it to the init.d directory.
Code Block |
---|
wget https://raw.github.com/avalonmediasystem/avalon-installer/master/modules/red5/templates/red5_init_script.erb -O red5_init_script.sh
mv red5_init_script.sh /etc/rc.d/init.d/red5 |
Give Red5 permissions to the red5 directory and the init script.
Code Block |
---|
chown -R red5:red5 /usr/local/red5
chmod +x /etc/rc.d/init.d/red5 |
FFmpeg
Installation prerequisites
Install prerequisite packages using yum and the Avalon repository (note: mediainfo is pinned to 0.7.61-1 because of a bug with time fragment formatting in most recent version 0.7.87-1 in epel):
Code Block |
---|
yum install SDL-devel a52dec-devel bzip2-devel faad2-devel freetype-devel frei0r-plugins-devel \
gsm-devel imlib2-devel lame-devel libdc1394-devel libraw1394-devel librtmp-devel libtheora-devel \
libva-devel libfaac-devel libvdpau-devel libstdc++-devel libvorbis-devel libvpx-devel \
mediainfo-0.7.61-1 opencore-amr-devel opencv-devel openjpeg-devel openssl-devel schroedinger-devel \
speex-devel texi2html vo-aacenc-devel x264-devel xvidcore-devel yasm zlib-devel |
Install rpmdev-setuptree
Code Block |
---|
yum install rpmdevtools |
Install ffmpeg srpm
The following commands need to run under a user other than root. Change to the red5 user to continue.
Code Block |
---|
su - red5 |
Run the ffmpeg installer
Code Block |
---|
rpmdev-setuptree
rpm -ivh https://github.com/avalonmediasystem/avalon-installer/blob/master/files/ffmpeg/ffmpeg-2.4.2-1.el6.src.rpm?raw=true
# Retrieving https://github.com/avalonmediasystem/avalon-installer/blob/master/files/ffmpeg/ffmpeg-2.4.2-1.el6.src.rpm
# 1:ffmpeg warning: user makerpm does not exist - using root
# warning: group makerpm does not exist - using root
########################################### [100%]
# warning: user makerpm does not exist - using root
# warning: group makerpm does not exist - using root
# Build ffmpeg binary as non-root and install as root
rpmbuild -bb rpmbuild/SPECS/ffmpeg24.spec |
Log back in as root and finish the install.
Code Block |
---|
su - root
rpm -ivh /home/red5/rpmbuild/RPMS/x86_64/ffmpeg-*.rpm |
You can also build a more modern ffmpeg from source, but not all versions work. Version 3.1 is known to work - check out the release/3.1 branch and build from there. If your modern ffmpeg uses fdk_aac instead of libfaac, adjust the Matterhorn etc/encoding/avalon.properties accordingly.
HTTPD
Install and start the httpd service.
Code Block |
---|
yum install httpd
service httpd start |
With newer httpd you may need to in /etc/httpd/conf.d/10-mod_rewrite.conf, replace RewriteLock line with Mutex sem
Matterhorn
Install Matterhorn
Create a user for Matterhorn and then install Matterhorn
Code Block |
---|
useradd matterhorn wgetauth_request /auth; add_header Access-Control-Allow-Headers '*'; add_header Access-Control-Expose-Headers 'Server,range,Content-Length,Content-Range'; add_header Access-Control-Allow-Methods 'GET, HEAD, OPTIONS'; add_header Access-Control-Allow-Origin '*'; expires 100d; } location = /auth { # resolver 127.0.0.1; proxy_pass http://127.0.0.1/authorize.txt?token=$token&name=$stream; proxy_pass_request_body off; proxy_set_header Content-Length ""; proxy_set_header X-Original-URI $request_uri; } } } |
Info |
---|
listen should use a public open port.
|
Add /lib/systemd/system/nginx.service
Code Block |
---|
[Unit]
Description=The NGINX HTTP and reverse proxy server
After=syslog.target network.target remote-fs.target nss-lookup.target
[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/usr/sbin/nginx -s reload
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true
[Install]
WantedBy=multi-user.target |
Add nginx
user and let it own nginx stuff
Code Block |
---|
useradd -M -s /bin/nologin nginx
chown -R nginx:nginx /etc/nginx /var/log/nginx |
Make 8980 bindable and start nginx
Code Block |
---|
semanage port -a -t http_port_t -p tcp 8980
systemctl start nginx |
Later: Avalon config should be updated to be compatible with Nginx:
Code Block |
---|
streaming:
server: :nginx
http_base: 'http://localhost:8980/avalon'
content_path: '/var/avalon/derivatives' |
Warning |
---|
If you enable SSL on Avalon server, you should also enable SSL on the streaming server to avoid Mixed content warning. |
FFmpeg & Mediainfo
Info | ||
---|---|---|
| ||
The following prebuilt binaries are provided by a third party. Proceed with caution. |
Download and install ffmpeg (for transcoding & thumbnails)
Code Block |
---|
mkdir -p /tmp/ffmpeg && cd /tmp/ffmpeg
curl https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz | tar xJ
cp `find . -type f -executable` /usr/bin/ |
Install Mediainfo (for technical metadata)
Code Block |
---|
yum install mediainfo |
HTTPD
Install and start the httpd service.
Code Block |
---|
yum install httpd
systemctl start httpd |
Apache Passenger and Ruby
Change current user to avalon then install RVM and ruby 2.4.1
Code Block |
---|
yum install ruby sqlite-devel # Needed to build Ruby using RVM.
useradd avalon
su - avalon
curl -L https://get.rvm.io | bash -s stable --ruby=2.5.5 |
Source the RVM shell (as avalon user) or close the terminal and open it back up.
Code Block |
---|
source /home/avalon/.rvm/scripts/rvm
rvm use 2.5.5
exit |
Install Passenger apache module requirements (as root)
Code Block |
---|
yum install pygpgme curl
curl --fail -sSLo /etc/yum.repos.d/passenger.repo https://oss-binaries.phusionpassenger.com/yum/definitions/el-passenger.repo
yum install mod_passenger || yum-config-manager --enable cr && yum install mod_passenger |
Create a virtual host for avalon
Code Block |
---|
wget --no-check-certificate https://raw.github.com/avalonmediasystem/config-files/main/apache/20-avalon.conf -P /etc/httpd/conf.d/
vim /etc/httpd/conf.d/20-avalon.conf |
In 20-avalon.conf add this line inside the VirtualHost tag:
Code Block |
---|
RailsEnv production |
If using SSL, the following fix should be added to address BEAST, POODLE, RC4 issues (after the SSLEngine on
)
Code Block |
---|
SSLProtocol all -SSLv2 -SSLv3
SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:!RC4:+HIGH:+MEDIUM:-LOW |
Modify /etc/httpd/conf.d/passenger.conf
Code Block |
---|
PassengerRuby /home/avalon/.rvm/rubies/ruby-2.5.5/bin/ruby |
Validate passenger install and restart apache
Code Block |
---|
passenger-config validate-install
systemctl start httpd |
Avalon
Grab Avalon code from github
Code Block |
---|
git clone https://github.com/avalonmediasystem/avalon-felix/archive/1.4.x.tar.gz tar xvf 1.4.x.tar.gz mv avalon-felix-1.4.x /usr/local/matterhorn wget --no-check-certificate https://raw.github.com/avalonmediasystem/config-files/master/matterhorn/matterhorn_init.sh mv matterhorn_init.sh /etc/init.d/matterhorn chmod +x /etc/init.d/matterhorn chown -R matterhorn:matterhorn /usr/local/matterhorn |
Add avalon user and create avalon directory.
Code Block |
---|
useradd avalon
mkdir /var/www/avalon
chown -R avalon:avalon /var/www/avalon |
Create and configure the media_path (upload) and streaming directories.
Code Block |
---|
mkdir -p /usr/local/masterfiles
chown avalon:avalon /usr/local/masterfiles
mkdir -p /usr/local/red5/webapps/avalon/streams
mkdir /var/avalon
mkdir -p /var/www/avalon/public/streams
chown red5:avalon /usr/local/red5/webapps/avalon/streams
chmod 0775 /usr/local/red5/webapps/avalon/streams
ln -s /usr/local/red5/webapps/avalon/streams /var/avalon/rtmp_streams
mkdir /var/avalon/hls_streams
chown matterhorn:matterhorn /var/avalon/hls_streams/
ln -s /var/avalon/hls_streams/ /var/www/avalon/public/streams
chmod 0775 /var/avalon/hls_streams/ |
Configure Matterhorn
Download Matterhorn config and verify property values.
Code Block |
---|
wget --no-check-certificate https://raw.github.com/avalonmediasystem/config-files/master/matterhorn/config.properties
vim config.properties |
And verify the configuration of the streaming directories
Code Block |
---|
org.opencastproject.streaming.directory=/var/avalon/rtmp_streams
org.opencastproject.hls.directory=/var/avalon/hls_streams |
Also check in /usr/local/matterhorn/etc/load/org.opencastproject.organization-mh_default_org.cfg
Code Block |
---|
prop.avalon.stream_base=file:///var/avalon/rtmp_streams |
Move the config to the appropriate spot
Code Block |
---|
mv config.properties /usr/local/matterhorn/etc/ |
Add matterhorn user to the avalon group.
Code Block |
---|
usermod -G avalon matterhorn |
Optional, but recommended to avoid problems with batch ingest: /wiki/spaces/AVALON/pages/1957954226
Apache Passenger and Ruby
Change current user to avalon then install RVM and ruby 2.2.5
Code Block |
---|
su - root
yum install sqlite-devel
su - avalon
curl -L https://get.rvm.io | bash -s stable --ruby=2.2.5 |
Source the RVM shell (as avalon user) or close the terminal and open it back up.
Code Block |
---|
source /home/avalon/.rvm/scripts/rvm
rvm use 2.2.5 |
Install Passenger via Gem (as avalon user)
Code Block |
---|
gem install passenger |
Check to make sure passenger installed in the expected location (as avalon user)
Code Block |
---|
$ passenger-config --root
/home/avalon/.rvm/gems/ruby-2.2.5/gems/passenger-5.0.17 |
Install Passenger apache module requirements (as root)
Code Block |
---|
su - root
yum install curl-devel httpd httpd-devel apr-devel apr-util-devel |
Build passenger for your version of Apache and Ruby (as avalon user)
Code Block |
---|
su - avalon
passenger-install-apache2-module #copy the suggested Apache configuration file settings for later |
Create an apache configuration file (as root)
Code Block |
---|
su - root
vim /etc/httpd/conf.d/passenger.conf |
Example contents of /etc/httpd/conf.d/passenger.conf, which may need to be changed based off of the current version of ruby and passenger:
Code Block |
---|
LoadModule passenger_module /home/avalon/.rvm/gems/ruby-2.2.5/gems/passenger-5.0.17/buildout/apache2/mod_passenger.so
<IfModule passenger_module>
PassengerRoot /home/avalon/.rvm/gems/ruby-2.2.5/gems/passenger-5.0.17
PassengerDefaultRuby /home/avalon/.rvm/wrappers/ruby-2.2.5/ruby
PassengerMaxPoolSize 30
PassengerPoolIdleTime 300
PassengerMaxInstancesPerApp 0
PassengerMinInstances 3
PassengerSpawnMethod smart-lv2
</IfModule> |
Apache security configuration
Code Block |
---|
wget --no-check-certificate https://raw.github.com/avalonmediasystem/config-files/master/sbin/avalon_auth -O /usr/local/sbin/avalon_auth
chmod +x /usr/local/sbin/avalon_auth
wget --no-check-certificate https://raw.github.com/avalonmediasystem/config-files/master/apache/10-mod_rewrite.conf -P /etc/httpd/conf.d/ |
Create a virtual host for avalon in /etc/httpd/conf.d/avalon.conf
Code Block |
---|
wget --no-check-certificate https://raw.github.com/avalonmediasystem/config-files/master/apache/20-avalon.conf -P /etc/httpd/conf.d/
vim /etc/httpd/conf.d/20-avalon.conf |
In 20-avalon.conf add this line inside the VirtualHost tag:
Code Block |
---|
RailsEnv development |
If using SSL, the following fix should be added to address BEAST, POODLE, RC4 issues (after the SSLEngine on
)
Code Block |
---|
SSLProtocol all -SSLv2 -SSLv3
SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:!RC4:+HIGH:+MEDIUM:-LOW |
Restart apache. With apache running, check passenger-status
Code Block |
---|
service httpd restart
su - avalon
which passenger-status
#> ~/.rvm/gems/ruby-2.2.5/bin/passenger-status |
Avalon
Grab Avalon code from github
Code Block |
---|
cd ~
git clone git://github.com/avalonmediasystem/avalon.git
cd avalon
git checkout master #make sure you are in the master branch (should be by default)
su - root
chown avalon:avalon /var/www/avalon/public/
su - avalon
mv public/* /var/www/avalon/public/
rmdir public
mv * /var/www/avalon/ |
Configure Avalon
Warning |
---|
If using vim with default settings and pasting the the code below, it will automatically comment out the last line. To prevent that, enable paste using the command :set paste and then use just ctrl+shift+v instead of going into insert mode. |
Create /var/www/avalon/config/setup_load_paths.rb and add:
Code Block |
---|
if ENV['MY_RUBY_HOME'] && ENV['MY_RUBY_HOME'].include?('rvm')
begin
gems_path = ENV['MY_RUBY_HOME'].split(/@/)[0].sub(/rubies/,'gems')
ENV['GEM_PATH'] = "#{gems_path}:#{gems_path}@global"
require 'rvm'
RVM.use_from_path! File.dirname(File.dirname(__FILE__))
rescue LoadError
raise "RVM gem is currently unavailable."
end
end
# If you're not using Bundler at all, remove lines bellow
ENV['BUNDLE_GEMFILE'] = File.expand_path('../Gemfile', File.dirname(__FILE__))
require 'bundler/setup' |
Code Block |
---|
cd /var/www/avalon/config
cp authentication.yml.example authentication.yml
|
...
Code Block |
---|
vim database.yml |
Replace database.yml with the correct values for your production environment
Code Block |
---|
development:
adapter: mysql2
host: localhost
database: rails
username: rails
password: rails
pool: 5
timeout: 5000 |
Install the mysql2 adapter
Code Block |
---|
yum install cmake #<--will be required for rugged gem
yum install mariadb-devel
su - avalon
gem install activerecord-mysql2-adapter
gem install mysql2
vim /var/www/avalon/Gemfile |
Add this line to the Gemfile
Code Block |
---|
gem 'mysql2', '~>0.3.20' |
If you are using mysql instead of sqllite (as is done throughout this document), then comment out the following lines in the Gemfile
Code Block |
---|
# gem 'activerecord-jdbcsqlite3-adapter'
# gem 'jdbc-sqlite3'
# gem 'sqlite3' |
Run the bundle install
Code Block |
---|
# as root
yum install nodejs # Javascript runtime
# as avalon
cd /var/www/avalon
gem update debugger-ruby_core_source
gem install bundler
bundle install |
Finish configuring Avalon
Edit /var/www/avalon/config/solr.yml and /var/www/avalon/config/blacklight.yml
Code Block |
---|
development:
url: http://localhost:8983/solr/avalon |
Edit /var/www/avalon/config/fedora.yml
Code Block |
---|
development:
user: fedoraAdmin
password: fedoraPassword
url: http://127.0.0.1:8984/fedora4/rest
base_path: /dev |
Create /var/www/avalon/config/matterhorn.yml
Code Block |
---|
development:
url: http://matterhorn_system_account:CHANGE_ME@localhost:8080/ |
Create /var/www/avalon/config/avalon.yml and base it off of /var/www/avalon/config/avalon.yml.example. Consult the documentation to customize this file for your installation.
Code Block |
---|
development:
dropbox:
path: '/var/avalon/dropbox/'
upload_uri: 'sftp://localhost/var/avalon/dropbox'
username: 'test'
password: 'test'
notification_email_address: ''
domain:
port: 80 |
...
Code Block |
---|
cd /var/www/avalon/config
rake secret |
grab the output of rake secret and add it to secrets.yml where instructed.
More information: Configuration Files#config/secrets.yml
Create controlled_vocabulary.yml
Code Block |
---|
cp controlled_vocabulary.yml.example controlled_vocabulary.yml |
Create the database using rake
Code Block |
---|
# as avalon user
cd /var/www/avalon
rake db:create |
If you get an error message saying that you can't connect to the database, take a look at this post and follow some of the troubleshooting steps.
Run the database migrations
Code Block |
---|
rake db:migrate |
Set rails environment to development, if it has not defaulted to this. On the first line of /var/www/avalon/config/environment.rb make sure it says 'development'
Code Block |
---|
ENV['RAILS_ENV'] ||= 'development' |
Visit your new Avalon site!
You should be able to visit the webpage with just the hostname (ie http://localhost)
Click on "Sign in" in the upper right corner of the website main page. Set up a default identity with the following properties.
Code Block |
---|
archivist1@example.com
<some password> |
This is a known identity with administrative privileges.
Redis & Resque
Avalon uses Resque for background processing, which relies Redis as its key-value store.
Install Redis
Code Block |
---|
yum install redis |
Start Resque
Code Block |
---|
# as avalon (replace production with development if necessary)
cd /var/www/avalon/
RAILS_ENV=production BACKGROUND=yes bundle exec rake resque:scheduler
RAILS_ENV=production BACKGROUND=yes QUEUE=* bundle exec rake resque:work |
Resque logs to log/resque.log in the avalon directory.
To restart rescue, simple kill its two processes (`ps aux | grep resque`) and run the above commands again.
Red5 Avalon Security Webapp
Code Block |
---|
cd /usr/local/red5/webapps
wget https://github.com/avalonmediasystem/avalon-installer/raw/master/modules/avalon/files/red5/red5-avalon.tar.gz
tar xvzf red5-avalon.tar.gz
chown -R red5:red5 avalon/ |
Edit /usr/local/red5/webapps/avalon/WEB-INF/red5-web.properties
Code Block |
---|
avalon.serverUrl=http://localhost/ |
Restart Red5
Code Block |
---|
service red5 restart |
If Red5 is installed and running you should be able to access http://avalon.dev:5080/
Additional Configurations
Dropbox
Code Block |
---|
groupadd -r dropbox
useradd -r avalondrop
usermod -G dropbox avalon
mkdir -p /var/avalon/dropbox
chown avalondrop:dropbox /var/avalon/dropbox
chmod 2775 /var/avalon/dropbox |
Edit /etc/ssh/sshd_config
Code Block |
---|
# override default of no subsystems
Subsystem sftp internal-sftp
# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# ForceCommand cvs server
Match Group dropbox
ChrootDirectory /var/avalon
X11Forwarding no
AllowTcpForwarding no
ForceCommand internal-sftp |
Restart SSH
Code Block |
---|
service sshd restart |
Batch ingest
To manually start a batch ingest job, run as avalon user
Code Block |
---|
rake avalon:batch:ingest |
To make batch ingest run automatically whenever a manifest is present, you need to add a cron job. This cron job can be created by the whenever gem from reading config/schedule.rb
. To preview, run
Code Block |
---|
whenever |
this will translate content in schedule.rb
to cron job syntax. Once verified, run the following to write job to crontab
Code Block |
---|
whenever --update-crontab |
You should get the cron job automatically if you were deploying from Capistrano.
...
.git /var/www/avalon
chown -R avalon:avalon /var/www/avalon |
Set rails environment to production, if it has not defaulted to this. On the first line of /var/www/avalon/config/environment.rb make sure it says 'production'
Code Block |
---|
ENV['RAILS_ENV'] ||= 'production' |
Configure database settings
Anchor | ||||
---|---|---|---|---|
|
Code Block |
---|
cd /var/www/avalon/config
vim database.yml |
Replace database.yml with the correct values for your production environment. Note that the pool setting should be equal or exceed the number of concurrent jobs in Sidekiq.
Code Block |
---|
production:
adapter: mysql2
host: localhost
database: rails
username: rails
password: rails
pool: 20
timeout: 5000 |
Install the mysql2 adapter
Code Block |
---|
yum install cmake #<--will be required for rugged gem
yum install mariadb-devel |
Install gems
Run the bundle install
Code Block |
---|
# as root
yum install nodejs # Javascript runtime
# as avalon
su - avalon
cd /var/www/avalon
gem install bundler
bundle install --with mysql production --without development test
exit |
Finish configuring Avalon
Edit /var/www/avalon/config/solr.yml and /var/www/avalon/config/blacklight.yml
Code Block |
---|
production:
url: http://localhost:8983/solr/avalon |
Edit /var/www/avalon/config/fedora.yml
Code Block |
---|
production:
user: fedoraAdmin
password: fedoraAdmin
url: http://127.0.0.1:8984/fedora4/rest
base_path: "" |
Create streaming directory
Code Block |
---|
# as root
mkdir -p /var/avalon/derivatives
chown avalon:avalon /var/avalon/derivatives |
Info | ||
---|---|---|
| ||
Avalon settings now live in /var/www/avalon/config/settings.yml. The default values should be sufficient to start with. They can be selectively overwritten by creating a settings/<environment>.yml, or by using environment variables. Consult the config gem doc to understand how it works, or Avalon's documentation to customize this file for your installation. |
Let Avalon know where your HLS streams are
Code Block | ||
---|---|---|
| ||
streaming:
server: :nginx
http_base: 'http://localhost:8980/avalon'
content_path: '/var/avalon/derivatives' |
Anchor | ||||
---|---|---|---|---|
|
Code Block |
---|
export RAILS_ENV=production
rake secret |
grab the output of rake secret and add it to secrets.yml where instruSTDOUTSTDOUTSTDOUTcted.
More information: Configuration Files#config/secrets.yml
Create controlled_vocabulary.yml
Code Block |
---|
cp config/controlled_vocabulary.yml.example config/controlled_vocabulary.yml |
If you get an error message saying that you can't connect to the database, take a look at this post and follow some of the troubleshooting steps.
Run the database migrations
Code Block |
---|
rake db:migrate |
Install yarn and node modules
Code Block |
---|
# as root
curl --silent --location https://dl.yarnpkg.com/rpm/yarn.repo | sudo tee /etc/yum.repos.d/yarn.repo
yum install yarn
# as avalon
su - avalon
cd /var/www/avalon
yarn install |
Precompile assets
Code Block |
---|
# as avalon
RAILS_ENV=production bundle exec rake assets:precompile |
Restart Apache
Code Block |
---|
# as root
systemctl restart httpd |
Install ImageMagick
Code Block |
---|
# as root
yum install imagemagick |
Sidekiq
Avalon uses Sidekiq for background processing, which relies on Redis as its key-value store.
Install Redis
Code Block |
---|
# as root
yum install redis
systemctl start redis |
Install Sidekiq
Code Block |
---|
# as root
wget https://raw.githubusercontent.com/mperham/sidekiq/main/examples/systemd/sidekiq.service -O /lib/systemd/system/sidekiq.service |
Edit the following lines in sidekiq.service
Code Block |
---|
WorkingDirectory=/var/www/avalon
ExecStart=/bin/bash -lc '/home/avalon/.rvm/gems/ruby-2.5.5/bin/bundle exec sidekiq -e production'
User=avalon
Group=avalon |
Code Block |
---|
# as root
systemctl start sidekiq |
Sidekiq logs to STDOUT.
Warning | ||
---|---|---|
| ||
When ingesting a media file, you may encounter an error message saying that file:///tmp/filename can’t be accessed or located. This may result from the protected temp file settings that are defaults in CentOS 7. Fix by changing “true” to “false” for PrivateTmp in these files in /usr/lib/systemd/system: sidekiq.service PrivateTmp=false |
Additional Configurations
Dropbox
Code Block |
---|
groupadd -r dropbox
useradd -r avalondrop
usermod -G dropbox avalon
mkdir -p /srv/avalon/dropbox
chown avalondrop:dropbox /srv/avalon/dropbox
chmod 2775 /srv/avalon/dropbox |
Edit /etc/ssh/sshd_config
Code Block |
---|
# override default of no subsystems
Subsystem sftp internal-sftp
# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# ForceCommand cvs server
Match Group dropbox
ChrootDirectory /srv/avalon
X11Forwarding no
AllowTcpForwarding no
ForceCommand internal-sftp |
Restart SSH
Code Block |
---|
service sshd restart |
Batch ingest
To manually start a batch ingest job, run as avalon user
Code Block |
---|
rake avalon:batch:ingest |
To make batch ingest run automatically whenever a manifest is present, you need to add a cron job. This cron job can be created by the whenever gem from reading config/schedule.rb
. To preview, run
Code Block |
---|
whenever |
this will translate content in schedule.rb
to cron job syntax. Once verified, run the following to write job to crontab
Code Block |
---|
whenever --update-crontab |
You should get the cron job automatically if you were deploying from Capistrano.
Authentication Strategy
Avalon comes with Persona by default but it can be configured to work with other authentication strategies by using the appropriate omniauth gems. The following example is applicable to Indiana University CAS, it may need some adjustments in order to work with other CAS implementation.
Add to Gemfile
Code Block |
---|
gem 'net-ldap'
gem 'omniauth-cas', :git => "https://github.com/cjcolvar/omniauth-cas.git" |
Install new gems
Code Block |
---|
bundle install |
Add to config/initializers/my-ldap.rb
Code Block |
---|
module Avalon
MY_GUEST_LDAP = Net::LDAP.new
MY_GUEST_LDAP.host = "eads.myuni.edu"
MY_GUEST_LDAP.authenticate 'cn=******,ou=Accounts,dc=eads,dc=myuni,dc=edu', '******'
GROUP_LDAP = Net::LDAP.new
GROUP_LDAP.host = "ads.myuni.edu"
GROUP_LDAP.authenticate 'cn=******,ou=Accounts,dc=ads,dc=myuni,dc=edu', '******'
GROUP_LDAP_TREE = "dc=ads,dc=myuni,dc=edu"
end |
Add config/initializers/user_auth_cas.rb
Code Block |
---|
require 'net/ldap'
User.instance_eval do
def self.find_for_cas(access_token, signed_in_resource=nil)
logger.debug "#{access_token.inspect}"
#data = access_token.info
username = access_token.uid
email = nil
user = User.where(:username => username).first
unless user
if email.nil?
tree = "dc=ads,dc=myuni,dc=edu"
filter = Net::LDAP::Filter.eq("cn", "#{username}")
email = Avalon::GROUP_LDAP.search(:base => tree, :filter => filter, :attributes=> ["mail"]).first.mail.first
end
user = User.find_or_create_by_username_or_email(username, email)
raise "Finding user (#{ user }) failed: #{ user.errors.full_messages }" unless user.persisted?
end
user
end
end |
Add to config/settings/production.local.yml
Code Block |
---|
auth:
configuration:
- :name: My University
:logo: my_logo.png
:provider: :cas
:params:
:host: cas.myuni.edu
:login_url: /cas/login
:service_validate_url: /cas/validate
:logout_url: /cas/logout
:ssl: true |
Using the System
You should be able to visit the webpage with just the hostname (ie http://localhost)
Create an admin account
You can create an account from the command line in the root of your avalon install:
Code Block |
---|
bundle exec rake avalon:user:create avalon_username=user@example.com avalon_password=password avalon_groups=administrator |
Additional information
You can find specific information about using the system in the Collection Manager's Guide. /wiki/spaces/AVALON/pages/1957954522 is available for your convenience. Upload new items individually or by batch directly via SFTP using the avalondrop account you created above.
Configure additional feataures
Known Issues - a list of bugs, workarounds, and cautions.
...
Code Block |
---|
chkconfig --level 345 tomcat on chkconfig --level 345 mysqldmariadb on chkconfig --level 345 nginx on chkconfig --level 345 sshd on chkconfig --level 345 red5redis on chkconfig --level 345 httpdsidekiq on chkconfig --level 345 matterhornhttpd on |