Deploying Hydra unconference notes

Notes from Deployment break-out session - Thursday October 2, 2014

What do people want to talk about/ think we are talking about? Puppet docker Capistrano Packer Maker vmware bash scripts Chef

This list displays a good breadth of ideas/interests/tools

Hydra did not have any installation documentation at all a year ago - we are still behind in documentation

It would be great to develop/document war stories about deployment

Most attendees manage their own infrastructure – maybe 40% have other teams to manage it for them

Most attendees came looking for solutions rather than offering them

Using vm’s mostly as an end run around differences in development environments – hardware/OS's – for staff/developers

Can also use CI server to check code against a particular environment that matches production - one example is Travis triggered by check-ins on Github. Travis does require you to mock up http responses and other things  - Travis can be slow – and it can mask production issues because it runs against hydra-jetty not production fedora/solr

Best practice: run full suite locally before pushing, smoke test on check-in, overnight full test suite that ran the longer stuff

Can also break up your test suite for optimization – pull out longer/slower stuff

Puppet is most useful in preparing for big changes (like Fedora4) – easy testing of the new environment. Also helpful for processes you don’t do very often, like building a whole new server once every six months. Puppet scripts preserve your present-day knowledge for reliable future use.

Why did people choose a particular solution (puppet, chef, ansible)? Mostly because sysops people chose it (or another institution started the bandwagon). Ansible made deployment “less spooky”.  Allows you to revert to an older version if the latest deployment doesn’t work out.

Can use a single tool to bring up production servers and development vm’s.  Chef uses .yml config files, puppet config was easier to handle.

Avalon installer is part of the plan to encourage outside adoption – uses Capistrano for web application, Puppet for everything else, also includes Vagrant so you can spin up a vm quite easily – this approach works, but it’s complicated and requires the right yaml files in the right places.

Cleveland split their production systems: Web node with code/apache/passenger and ffmpeg, data node with tomcat/fedora/solr.

Aaron (institution?) has split production systems even further – solr and fedora separate. Not using puppet there. Chose this setup  for performance improvements – with lots of ingests going on, fedora hogs memory on tomcat – if solr is running on the same container the web service slows down because solr has fewer resources available.

Ansible has the concept of roles so you can manage a distributed environment.

Performance monitoring is also very important. Have things in place so you know about problems before they get bad.

Resources – look at these repos for examples/ideas:

https://github.com/dpla/automation

https://github.com/jcftang/ansible-hydra

https://github.com/uclibs/scholar-puppet

Note-taker: Alicia Cozine, DCE