Eucalyptus problem

Question

I have some problems regarding to eucalyptus private cloud building. And two versions of this problem here, a simple one, and a long one.

Simple:

Anyone familiar with private cloud could draw me a picture to illustrate the 'IP SCHEME' of he cloud? I'm having trouble to decide where should be each component to. If I want to install CLC, Walrus, CC, SC, NC all on different physical machines, what should be the network topology?

More information: I have 12 physical machines running the cloud now, but I can't attach volumes to instances. I wondered where should I put storage controller to, public network? under CLC? under CC? same level with CC? No picture, no explanation found at all. And I tried everything I can, instances are running, ok to access them, just the damn volume thing bothered me for a whole week.

Long version:

1. I have successfully configured eucalyptus on 4 physical servers and it runs great.
2. The four servers are: CLC, SC, CC, NC
3. The problem is I'm not very sure where to put SC, Since the topology of my cloud is like this:

4. Now that CC are connected to two switches : ENTSWitch and PrivateSwitch. And has two IPs: public ip->10.11.25.115, private IP-> 10.11.20.1. Node controller is connected to privateswitch only and has private IP 10.11.20.2
5. This works fine by now, using managed-novlan mode, node controller can deploy instances and get public ip addresses for them. Everything is sweet.
6. Here comes the problem, I can create volume on storage controller. But the volume is failed to be attached to instances.
7. Wondering if it's because node controller is under a different network with storage controller.

Tried to solve the problem by doing following:

• A. maybe i shouldn't connect storage controller to ENTSwitch?. Changed SC from public network t o private network
• B. Try to register storage controller again but failed. Reason: Cloud can't reach private network.
• C. Try to register sc on CC, can't, it must be registered on CLC
• D. Add route to CLC, now CLC and SC can connect to each other.
• E. Register successfully, still can't attach volume.
• F: Error log:

    [root@HeroNodeServer1 images]# cat /var/log/eucalyptus/nc.log | grep Volume
    [Fri Aug 20 16:36:12 2010][015662][EUCAINFO ] doAttachVolume() invoked (id=i-33BD0649 vol=vol-59C90629 remote=/dev/etherd/e0.1 local=/dev/sdb)
    [Fri Aug 20 16:37:53 2010][015662][EUCAERROR ] AttachVolume() failed (err=-1) XML=
    [Fri Aug 20 16:37:53 2010][015662][EUCAERROR ] ERROR: doAttachVolume() failed error=1
    [Fri Aug 20 16:37:53 2010][015662][EUCAINFO ] doAttachVolume() invoked (id=i-33BD0649 vol=vol-59C90629 remote=/dev/etherd/e0.1 local=/dev/sdb)
    [Fri Aug 20 16:37:53 2010][015662][EUCAERROR ] AttachVolume() failed (err=-1) XML=
    [Fri Aug 20 16:37:53 2010][015662][EUCAERROR ] ERROR: doAttachVolume() failed error=1
    [Fri Aug 20 16:37:53 2010][015662][EUCAINFO ] doAttachVolume() invoked (id=i-33BD0649 vol=vol-59C90629 remote=/dev/etherd/e0.1 local=/dev/sda5)
    [Fri Aug 20 16:37:53 2010][015662][EUCAERROR ] AttachVolume() failed (err=-1) XML=
    [Fri Aug 20 16:37:53 2010][015662][EUCAERROR ] ERROR: doAttachVolume() failed error=1
    [Fri Aug 20 16:37:58 2010][015662][EUCAINFO ] doAttachVolume() invoked (id=i-33BD0649 vol=vol-59C90629 remote=/dev/etherd/e0.1 local=/dev/sda4)
    [Fri Aug 20 16:39:38 2010][015662][EUCAERROR ] AttachVolume() failed (err=-1) XML=
    [Fri Aug 20 16:39:38 2010][015662][EUCAERROR ] ERROR: doAttachVolume() failed error=1
    [Fri Aug 20 16:39:38 2010][015662][EUCAINFO ] doAttachVolume() invoked (id=i-33BD0649 vol=vol-59C90629 remote=/dev/etherd/e0.1 local=/ev/sdp)
    [Fri Aug 20 16:39:38 2010][015662][EUCAERROR ] ERROR: doAttachVolume() failed error=1
    [Fri Aug 20 16:39:38 2010][015662][EUCAINFO ] doAttachVolume() invoked (id=i-33BD0649 vol=vol-59C90629 remote=/dev/etherd/e0.1 local=/dev/sdp)
    [Fri Aug 20 16:39:38 2010][015662][EUCAERROR ] AttachVolume() failed (err=-1) XML=
    [Fri Aug 20 16:39:38 2010][015662][EUCAERROR ] ERROR: doAttachVolume() failed error=1
    [Fri Aug 20 16:41:15 2010][015662][EUCAINFO ] doAttachVolume() invoked (id=i-33BD0649 vol=vol-5A030630 remote=/dev/etherd/e0.2 local=/dev/sdp)
    [Fri Aug 20 16:42:55 2010][015662][EUCAERROR ] AttachVolume() failed (err=-1) XML=
    [Fri Aug 20 16:42:55 2010][015662][EUCAERROR ] ERROR: doAttachVolume() failed error=1
    [Fri Aug 20 16:43:12 2010][015662][EUCAINFO ] doAttachVolume() invoked (id=i-33BD0649 vol=vol-5A030630 remote=/dev/etherd/e0.2 local=/dev/sdc)
    [Fri Aug 20 16:44:52 2010][015662][EUCAERROR ] AttachVolume() failed (err=-1) XML=
    [Fri Aug 20 16:44:52 2010][015662][EUCAERROR ] ERROR: doAttachVolume() failed error=1
    [Fri Aug 20 16:50:02 2010][015662][EUCAINFO ] doAttachVolume() invoked (id=i-33BD0649 vol=vol-59C90629 remote=/dev/etherd/e0.1 local=/dev/sdd)
    [Fri Aug 20 16:51:42 2010][015662][EUCAERROR ] AttachVolume() failed (err=-1) XML=
    [Fri Aug 20 16:51:42 2010][015662][EUCAERROR ] ERROR: doAttachVolume() failed error=1

Questions:

1. Where on earth should I attach SC to? public network or private network?
2. If it should be attached to private network, how to register SC if CLC can't reach private network
3. Why two solutions all failed to attach volume to instance?

================================================================

Here comes second one:

1. I'm currently trying to do some more test on it.
2. I want to expand 4-physical architecture to 16-physical architecture.
3. No matter how many machines I want to use, there's a question I had for a long time without being solved by tons of documents read.
4. The new architecture I designed is like following picture
5. In the new topology, I assumed that storage controller should be attached to private network
6. In this model, instances running on node controller should have two IP addresses assigned, one private, one "public"
7. The public IP address should be in the network same as CC right?
8. That means instances can be accessed by the network which CC located in.
9. And CLC should have a gateway service running to let instances accessing outer networks

Questions:

1. Is there any problem about the design?
2. If instances only can get "Public IP" which is under same network with Cluster Controller, how are clients suppose to access them from outer networks?
3. Means, how clients from out of the cloud access Instances through CLC?
4. Does CLC has the same mechanism like CC to assign a "public IP" for instances so that it can be accesses?

================================

That'd all I need to ask.

Thank you very much for reading this messy post, and any kind of reply is highly appreciated!

Answer 1

The following answers all assumes that the network mode is MANAGED-NOVLAN (this probably all apply for MANAGED mode also).

Section 1:

-1) Unless you have specific needs preventing it, I would recommend coupling the SC with the CC.

-2) In Managed mode, the CC sits between the public network and the private network of (one of) the cloud cluster(s). If the SC is on the CC machine it can than communicate with both networks. If you decide to have the SC on a separate server on the private network, you'll need to create a NAT rule (with iptables) on the CC to enable communications between the SC and the CLC through the CC Firewall. The CLC will need that rule to communicate with the SC.

-3) In solution 1; your SC was not connected to the private switch (this is a must). In solutions 2; you probably did not created that iptables rule on the CC and the SC was then isolated behind the CC.

Section 2:

-1) One correction: merge the switches CTI and switch1, It should be the same switch that is the connection to the public network. I would also suggest (unless absolutely needed) to merge the CLC and WC into one machine and to merge the SCs on their respective CCs. This diagram (http://cssoss.files.wordpress.com/2010/05/eucalyptus_cloud.png?w=600&h=467) of a UEC cloud can help you.

-2) When you create a new instance (euca-run-instance) Eucalyptus will create a new iptable rule on the CC server managing the node on which the instance is running. That rule permit communications through the CC. This NAT traversal route the communications between publics IPs and their corresponding private IPs.

-3) The clients accesses the instances through the corresponding CC. The CLC is only there to manage the cloud but, once an instance is started, the CC handles the communications.

-4) In short: No.

Answer 2

Okay let's try to start responding to this

• SC question

The SC needs to always be accesible from the instances, so it needs to live in the same network as the node controller, the instances can't see the storage because it lives on a different network.

• Network topology

It's good to have separate networks, but I think that it would be better to base your security on firewalling and traffic control rather to get things too complicated, Eucalyptus likes it when the network is simple, start with one network and don't divide it in two until you got a very good sense of security. Concentrate your efforts in having a secure CLC and CC since the security on the other components will be provided by the private network and the protocols that eucalyptus provide. Also the NC in no case needs external IPs.

• Network design

That design looks solid to me, all the public addresses will be provided by the CC so the CC will in effect be your gateway, think about it as being the "availability zones" in Amazon EC2, by having more than one CC you're creating your zones.

In resume, keep a close eye on the CLC and the CC and Walrus, make sure you know the in&outs of security and communication in them, the rest of components should be fine and living happily in internal networks.