Running pentaho on mesos - part 1

May 21, 2016

Introduction

I have been amazed by the ease of getting software through the software development cycle once you start doing it the Docker way. I picked it up soon after it came to light, and hammered it hard. At that point I got all developers aligned to use Docker as well.

As we started using Docker more extensively, combining different Frontends services and Backends services through Docker compositions, running multiple hosts, intranet, things started to get a bit more complicated.

Adrian Mouat acknowledged my next step in a piece he wrote for Oreilly Swarm v. Fleet v. Kubernetes v. Mesos Comparing different orchestration tools my prefered orchestration framework for our containers would be Mesos.

Define

This is an overview of a mesos cluster in its simplest form. Mesos architectural overview Source

There is so much more to be told about mesos, but I suggest you just Run, Forrest, Run! to Oreilly, Manning or some other it-drug delivery service and read up on this topic.

Create

I am going to assume a working docker / docker-compose / docker-machine setup, some recent hardware.

For now I will focus on a virtualbox based cluster running locally. I have created a crude but usefull bash-script to get you up and running, it cleans, creates masters, slave and a loadbalancer by utilising docker-machine. I have 4cpu, 16G of RAM at my disposal, just tinker with virtualbox-cpu-count, virtualbox-memory and virtualbox-disk-size to make this baby fit your needs.

#!/bin/sh
clear

echo "creating my epic local cluster"

clean() {
  echo "---> rm old machines, no mercy baby!"
  docker-machine rm -f mesos-lb1 mesos-m1 mesos-m2 mesos-m3 mesos-s1 mesos-s2 > /dev/null
}

create_masters() {
  echo "---> create masters"
  echo "---> create mesos-m1"
  docker-machine create -d virtualbox \
    --virtualbox-cpu-count "1" \
    --virtualbox-memory "1024" \
    --virtualbox-disk-size "50000" \
    mesos-m1

  echo "---> create mesos-m2"
  docker-machine create -d virtualbox \
    --virtualbox-cpu-count "1" \
    --virtualbox-memory "1024" \
    --virtualbox-disk-size "50000" \
    mesos-m2

  echo "---> create mesos-m3"
  docker-machine create -d virtualbox \
    --virtualbox-cpu-count "1" \
    --virtualbox-memory "1024" \
    --virtualbox-disk-size "50000" \
    mesos-m3
}

create_slaves() {
  echo "---> create slaves"
  echo "---> create mesos-s1"
  docker-machine create -d virtualbox \
    --virtualbox-cpu-count "2" \
    --virtualbox-memory "4092" \
    --virtualbox-disk-size "50000" \
    mesos-s1

  echo "---> create mesos-s2"
  docker-machine create -d virtualbox \
      --virtualbox-cpu-count "2" \
      --virtualbox-memory "4092" \
      --virtualbox-disk-size "50000" \
      mesos-s2
}

create_loadbalancer() {
  echo "---> create loadbalancer"
  echo "---> create mesos-lb1"
  docker-machine create -d virtualbox \
    --virtualbox-cpu-count "1" \
    --virtualbox-memory "1024" \
    --virtualbox-disk-size "50000" \
    mesos-lb1
}
main() {
  clean
  create_masters
  create_slaves
  create_loadbalancer
  echo "done"
}

main

After running this script, check the results:

docker-machine ls | grep virtualbox
mesos-lb1                    -        virtualbox   Running   tcp://192.168.99.105:2376           v1.11.1   
mesos-m1                     -        virtualbox   Running   tcp://192.168.99.100:2376           v1.11.1   
mesos-m2                     -        virtualbox   Running   tcp://192.168.99.101:2376           v1.11.1   
mesos-m3                     -        virtualbox   Running   tcp://192.168.99.102:2376           v1.11.1   
mesos-s1                     -        virtualbox   Running   tcp://192.168.99.103:2376           v1.11.1   
mesos-s2                     -        virtualbox   Running   tcp://192.168.99.104:2376           v1.11.1

That was easy!

Run