Deployment and DevOps

@Critchlow, Matthew (UCSD)

Tim Marconi (UCSD)

Ron Stanonik (UCSD)

@bess (Stanford)

@David Chandek-Stark (old account) (Duke)

@Jim Coble (Duke)

@Steven Ng (Chinese Historical Society of Southern California)

@aliciac (DCE)

Ian Lessing (UCSB)

Dermot Frost (Trinity College, Dublin)

@Jimmy Tang (Trinity College, Dublin)

@Glen Horton (Deactivated) (University of Cincinnati)

Challenges of going from development to production?

Fedora?

Hardware specs relative to collection size and users?

How long will the migration/ingest process take?

For folks in production: what were easy, challenges, what does prod environment look like?

Streaming video?

How are people dealing with migrating fedora instances?

New Hydra Head: separate repository or new?

Chef or Puppet or Ansible? Community using all three, can we organize or rally around this?

• consider Avalon as a starting point

• Campus IT or external hosts are nervous about granting access to run config, or have already made the decision of Chef/Puppet/Ansible.

• Ideally Sufia/Hydramata might have Chef and Puppet scripts for deployment so adopters can choose

Single Machine or multi-machine deployment?

• Stanford: load balanced with F5. Searchworks, generally shared Solr cluster and single Fedora instance that apps are connecting to. Trying to load balance as resources are available to do so.

• UCSB: haproxy system in front of any sites. does load balancing + security wall

Slashdotted?

• Stanford: Yes, crashed. spun up new instances of server and was able to keep up with traffic. Puppet helped significantly in this

• UCSD: got on Reddited.

Monitoring Systems?

• Nagios (Stanford, UCSD, others)

• Rails apps? -- “Is it working” gem https://github.com/tribune/is_it_working 

•  

  • nagios hits the “is it working” URL and reports out

OS’s? CentOS, Scientific (Trinity)

• CentOS repos are often out of date

• Stanford: RHEL

• UCSD: moving away from RHEL due to new Redhat university licensing

Server Configuration management?:

• 50% of room already using. 100% moving towards/considering

Trinity College: A few months from going to production. distributed file systems

• Using http://buildbot.net/ 

UCSB: Deploying in next few months. worried about deploying stable system that can keep up with dev. system that is performance tuned. Want staging to resemble production.

Stanford: How do we anticipate demand and sizing? GIS/Spatial data infrastrure brings new questions.

• migrating huge amounts of content from SDR1

• created positions for devops, could possibly share the job descriptions for others to reference

• still struggling with what operations procedures should be

•  

  • gotten much better w/ puppet

  • 20-ish instances of hydra, using different ruby gems

  •  

    • puppet helped locate/patch during a ruby vulnerability

• devs have a burndown box - VM assigned to them. when they want to deploy an app, they write a puppet recipe for it. devs use a base script, make minor manifest tweaks for their app. once working, ticket put in to devops person who reviews it, once passed it is deployed to server

• considering going to vagrant for laptops (so devs can get up to speed quicker)

• all puppet scripts in github, but currently private. only sysadmins can merge PR’s to master, devs work on scripts in branches

• The group all agreed that a HydraCamp covering operations/devops would be a great step forward

•  

  • DCE may have some sample scripts from different engagements that could be a good starting point

• Case Studies / White Papers

•  

  • starting point (recognized as largely out of date): https://wiki.duraspace.org/display/hydra/Deployment+Hardware+Information 

  • the group requests white papers from the community

  • case studies for people already in production to share w/ community.

  •  

    • https://wiki.duraspace.org/display/hydra/Deployment+Hardware+Information 

    • trinity college volunteered

    • stanford volunteered

  • requested case studies

  •  

    • ops, dev, or devops - what does that look like?

    •  

      • less than half on hardware they don’t control

    • migrating fedora instances

    • how to care/feed for production fedora (daily/weekly/monthly tasks, maintanence)

    • communication strategies

    •  

      • press releases from collection migration

    • cloud repository instance

    • campus IT or other external environment hosted

• University of Cincinnati has an excel spreadsheet on production hardware configuration and deployment  from all hardware partners they will share

•  

  • will add to existing wiki page

• A RailsCast style deployment video - @Steven Ng will take the lead on this concept