I have three networks on Rackspace:
- Public network
- Service network
10.181.XXX.XXX
(they call it a private network sometimes but it is not private per-se, it is datacenter-wide private so their tenants may share it) - A real private network of
192.168.3.0/24
created via the networking UI
My plan is to launch a secure CoreOS cluster whose etcd
peers and clients are limited to the real private networking so it is not available for the outside world.
So, I launched a CoreOS server and it had three interfaces:
eth0
has the IP address of the public network - goodeth1
has the IP address of the service network - goodeth2
has the IP address of the private network - great!
All looked well but etcd
is bound to the service network instead of the private one I created manually. Here's a piece of cloud-config
:
#cloud-config
coreos:
etcd2:
discovery: https://discovery.etcd.io/XXX
advertise-client-urls: http://$private_ipv4:2379,http://$private_ipv4:4001
initial-advertise-peer-urls: http://$private_ipv4:2380
listen-client-urls: http://$private_ipv4:2379,http://$private_ipv4:4001
listen-peer-urls: http://$private_ipv4:2380
Which means that $private_ipv4
variable gets expanded to eth1
IP address, here's the contents of /etc/environment
:
COREOS_PUBLIC_IPV4=166.78.XXX.XXX
COREOS_PRIVATE_IPV4=10.181.XXX.XXX
Ok, looks like Rackspace injects its own networks, it is explainable. But does this mean that etcd is locked only to the first two networks and there's no way to configure it to use the real private one?
I've tried a bunch of hacks such as this:
listen-peer-urls: http://`/usr/bin/ifconfig eth2 | /usr/bin/grep --word-regexp inet | /usr/bin/awk '{print $2}'`:2380
and others but neither got properly executed nor substituted in the cloud-config
.
My questions are:
- Am I doing something weird in terms on CoreOS cluster setup field, so that I can not achieve the target easily and naturally? If so, how would you lay out a secure cluster whose
etcd
operates within a real private network? - Are there ways to "inject" dynamic values into
cloud-init
file so it gets interpolated at runtime? The issue is that the IP address is not known beforehand so it needs to get gotten and injected somehow I think.