9

We are currently in the process of designing the architecture of our new Apache Mesos cloud setup. The goal is to unify our systems by moving different stacks onto the same architecture. The main workloads are big data analytics using Apache Spark and our corporate infrastructure including web servers, mail servers, etc.

The idea is to run our web services in Docker containers running on top of one of the available schedulers for Mesos (Marathon/Chronos, Aurora or Singularity). This would thus be the first Mesos framework group. Next to it, we would have the Apache Spark framework and several database frameworks for data storage. This would be the second group of Mesos frameworks. We will choose the specifics after running them all in parallel for testing.

We have trouble deciding, however, on which basis to run Mesos itself. Ideally, we want to run it as close to the metal as possible. We also want to use an orchestration solution to make sure that the Mesos & framework daemons are always running/restarted on failure. The options we are considering are as follows:

1) Running Mesos & the frameworks as docker containers in a minimal OS. In this respect, we are currently leaning towards CoreOS and Fleet.

2) Running Mesos & the frameworks directly on Ubuntu/Debian servers. For this option, we are leaning towards Foreman and Puppet.

As for the question, we are looking to identify the solution which, in order of importance:

  • is the least complex to configure
  • is the easiest to maintain & keep updated
  • has the least overhead

We have not worked with CoreOS before, but it is the option that we seem to be heading towards. One big (subjective) issue I have with this is that we run Mesos on Docker containers and then we run Docker containers on Mesos. This seems "unclean" and wrong to me. Is this consideration without merit?

A similar thought concerns the redundancy between layers. To explain where I'm coming from, I would prefer if Mesos was an actual OS that just runs right on top of the metal. It seems that no matter what basis you use, you end up with the same intended functionality on more than one layer of the architecture (i.e. CoreOS&Fleet&SystemD == Mesos&Marathon&Chronos). Is this unavoidable?

Are there other good options to run the layer below Mesos that we failed to consider, keeping in mind our criteria?

awfm9
  • 91
  • 4
  • This sounds complex. What is the appeal of Mesos in this context? – ewwhite Jun 18 '15 at 16:16
  • Mesos complements big data/HPC stuff well, like Spark or even Hadoop, but I don't see the value in putting _everything_ under it, particularly web or mail services. – Michael Hampton Jun 18 '15 at 16:30
  • @ewwhite The appeal in this context is that we can distribute the available hardware resources between all applications without having to segregate our cluster. If we run two configurations, we will have to manually divide resources between them. – awfm9 Jun 19 '15 at 16:37
  • @MichaelHampton We believe that Mesos architectures are the way forward for cloud computing, but I don't claim by any means that it's the only answer. It should be obvious, though, that it's easier to manage on cluster than managing the same cluster plus some extra stuff on the side. – awfm9 Jun 19 '15 at 16:39
  • 1
    This might be worth a look: https://mesosphere.com/product/ - the enterprise edition is apparently going to be viable to provision on bare metal. – Mary Jul 26 '15 at 14:51
  • Very interesting question and takes a long discussion.... But Have you consider to build an Openstack infrastructure solution to do that? – Francesco P Nov 18 '15 at 09:30
  • You may want to check out Paasta https://github.com/Yelp/paasta This framework uses a lot of very solid tools like Mesos and Docker, and allows for a lot of customization depending on what you want to do with it. – JBiggs Jan 12 '17 at 15:17

1 Answers1

2

Configuring and running services under Mesos can be a complex operation or an easy one, and you should first define a schema of services you want to run under it, depending on your needs and goals, to get the soultion you want.

I run a setup of >70 machines and a variety of different services under HAProxy for dynamic load balancing with Mesos-DNS and Marathon, Api gateway, Chronos, Jenkins, Docker, Collectd and Graphite,...

Now to answer your direct questions:

  • Mesos is least complex to configure using your "favorite" Linux distribution that you are most familiar with.
  • Easiest to maintain is again, the distro you are most familiar with.
  • Regarding overhead, Mesos is a software system that uses underlying OS libraries and other software functionality aside it's own and having Mesos as an OS (working both with hardware and software...) is kind of completely wrong picture.

So my best answer for you is to use your favorite Linux distro and put Mesos on, or if you would like to learn something new and possibly quick and effortlessly, use (open-sourced) DCOS and CoreOS.

Breign
  • 106
  • 8