Prototype Binary Software Distribution Methods¶

SQuaRE is investigating a number of methods for distributing binary builds of LSST DM’s science pipeline or “application” software. Initial target platforms are CentOS 6 & 7, with support for OS X planned for as soon as build resources become available.

We envision producing binary distribution products for official releases, weekly development releases, and eventually on a nightly basis. Construction of a binary product on demand is also a possibility. Work is underway to fully automate this process.

Methods being evaluated include:

• Conda packages for Anaconda/Miniconda
• OpenStack images on the NCSA Nebula system
• AWS images (AMIs)
• docker containers
• CernVM-FS (see Fabio Hernandez’s lsst-cvmfs)
• tarballs of an installed build tree
• “all-in-one” RPMs
• QEMU/KVM images

Conda Packages¶

A repo of “demo” conda packages (a “channel” in conda-speak) has been made available. Presently, only packages built on RHEL5 are available that are expected to have broad compatibility across current distributions. OSX packages are planned to be added in the future.

The URL of the conda repo is: https://conda-test.lsst.codes/dev/

curl -sSL https://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh > miniconda.sh && bash miniconda.sh -b -p ~/miniconda && rm -f miniconda.sh
export PATH="$PWD/miniconda/bin:$PATH"
conda config --add channels https://conda-test.lsst.codes/dev/
conda install -y lsst-apps

# lives at ./miniconda/bin/eups-setups.sh
source eups-setups.sh

curl -L https://github.com/lsst/lsst_dm_stack_demo/archive/master.tar.gz | tar xvzf -
cd lsst_dm_stack_demo-master
setup obs_sdss
./bin/demo.sh --small

Vagrant based demonstration¶

Vagrant is a tool to “Create and configure lightweight, reproducible, and portable development environments.” – in other words, it is essentially a user friendly interface for spawning and accessing virtual machines.

A demonstration of launching VM instances on the NCSA Nebula OpenStack system with Vagrant has been created called vagrant-nebula. The two brief walk throughs below use this repository.

This simple vagrant cheat sheet may be useful.

git clone https://github.com/lsst-sqre/vagrant-nebula.git
cd vagrant-nebula
vagrant up el7-docker
vagrant ssh el7-docker

docker run -ti lsstsqre/centos:7-stack-lsst_distrib-w_2016_08

git clone https://github.com/lsst-sqre/vagrant-nebula.git
cd vagrant-nebula
vagrant up el7
vagrant ssh el7

source /opt/lsst/software/stack/loadLSST.bash
curl -L https://github.com/lsst/lsst_dm_stack_demo/archive/master.tar.gz | tar xvzf -
cd lsst_dm_stack_demo-master
setup obs_sdss
./bin/demo.sh --small

vagrant destroy -f el7

vagrant halt el7

OpenStack/Nebula images¶

Two images have been published under the LSST project on the NCSA Nebula system.

Docker Containers¶

Demo docker images have being published via Docker Hub under the lsstsqre/centos namespace. Instructions for configuring Docker on CentOS 7 are provided in Installing Docker.

Example of pulling and running a docker container

docker run -ti lsstsqre/centos:7-stack-lsst_distrib-w_2016_08

Amazon Web Services (AWS) AMIs¶

Public AMIs have be posted in the two most popular [Continental US] AWS regions. We are assuming that AWS/EC2 users are familiar with how to launch an instance. AWS also provides copious documentation. See Launching an Instance for a reasonable documentation entry point.

Note that cloud-init-growpart is not working in this images. The instance will have a fixed size of 40GiB.

See Also¶

• vagrant-nebula
• Installing Vagrant
• Installing Docker
• vagrant cheat sheet
• aws.amazon.com