Contrail-ansible-deployer - Remote Compute Deployment [Collector down]

Question

I try to deploy environment with remote compute feature. Here is my instances file :

remote_locations:
    pop2:
      BGP_ASN: 12345
      SUBCLUSTER: pop2
      XMPP_SERVER_PORT: 5269
      BGP_PORT: 179
      CONTROL_INTROSPECT_LISTEN_PORT: 8083
provider_config:
    bms:
  instances:
    bms_contr:
      ip: 10.10.0.100
      provider: bms
      roles:
        config_database:
        config:
        control:
        analytics_database:
        analytics:
        analytics_alarm:
        webui:
        openstack_control:
        openstack_network:
    remoteCp_node:
      ip: 10.10.0.150
      provider: bms
      roles:
        control_only:
          location: pop2
          PHYSICAL_INTERFACE: em2
          DEFAULT_LOCAL_IP: 10.10.0.150
          DEFAULT_IFACE: em2
    bms_compute:
      ip: 10.40.0.100
      provider: bms
      roles:
        vrouter:
           CONTROL_NODES: 10.10.0.150
           VROUTER_GATEWAY: 10.40.0.1
           location: pop2
           PHYSICAL_INTERFACE: "em1"
        openstack_compute:
           network_interface: "em1"
        openstack_storage:
global_configuration:
    CONTAINER_REGISTRY: opencontrailnightly
contrail_configuration:
    CONTRAIL_VERSION: latest
    CLOUD_ORCHESTRATOR: openstack
    RABBITMQ_NODE_PORT: 5673
    AUTH_MODE: keystone
    KEYSTONE_AUTH_URL_VERSION: /v3
    KEYSTONE_AUTH_ADMIN_PASSWORD: contrail123
    UPGRADE_KERNEL: false
    AAA_MODE: rbac
    METADATA_PROXY_SECRET: contrail123
    CONTROLLER_NODES: 10.10.0.100
    CONTROL_DATA_NET_LIST: 10.10.0.0/24,10.40.0.0/24
    CONFIGDB_NODES: 10.10.0.100
kolla_config:
  kolla_globals:
    network_interface: "em2"
    kolla_external_vip_interface: "em2"
    kolla_external_vip_address: "10.10.0.100"
    kolla_internal_vip_address: "10.10.0.100"
    kolla_internal_vip_interface: "em2"
    enable_haproxy: "no"
    enable_ironic: "no"
    enable_swift: "yes"
    enable_cinder: "yes"
    enable_cinder_backend_lvm: "yes"
    enable_ceph: "no"
    cinder_backup_driver: "swift"
    horizon_keystone_multidomain: true

After ansible deployment in almost every log file i get such error:

SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ 10.10.0.100:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector connection down >>, ] >>

Collector log file:

SANDESH: Send FAILED: 1566470335049259 [SYS_NOTICE]: NodeStatusUVE: data= [ name = process_status= [ [ [ module_id = contrail-collector instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (_iter6) = 10.10.0.100:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = Cassandra server_addrs= [ [ (_iter6) = 10.10.0.100:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (_iter6) = 10.10.0.100:5673, ] ] status = Up description = RabbitMQ connection established ], [ type = Database name = :Global server_addrs= [ [ (_iter6) = 10.10.0.100:9042, ] ] status = Up description = ], [ type = Redis-UVE name = From server_addrs= [ [ (_iter6) = 127.0.0.1:6379, ] ] status = Up description = ], [ type = Redis-UVE name = To server_addrs= [ [ (_iter6) = 127.0.0.1:6379, ] ] status = Up description = ], [ type = KafkaPub name = 10.10.0.100:9092 server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = ], ] ] description = Collector, KafkaPub:10.10.0.100:9092 connection down ], ] ] ]

Contrail-status on main control node:

== Contrail control ==
control: active
nodemgr: active
named: active
dns: active

== Contrail analytics-alarm ==
nodemgr: active
kafka: active
alarm-gen: active

== Contrail database ==
nodemgr: active
query-engine: active
cassandra: active

== Contrail analytics ==
nodemgr: active
api: active
collector: active

== Contrail config-database ==
nodemgr: active
zookeeper: active
rabbitmq: active
cassandra: active

== Contrail webui ==
web: active
job: active

== Contrail device-manager ==

== Contrail config ==
svc-monitor: active
nodemgr: active
device-manager: active
api: active
schema: active

I simulate provider network on gns3 so its not connectivity between hosts issue. Openstack control node sees remote compute node.

When I deploy environment without analytics_alarm (and thus Kafka) I get such log in collector log file (other in other log files error is the same, Collector connection down):

[

Thread 140371426252544, Pid 1]: SANDESH: Send FAILED: 1566481747362411 [SYS_NOTICE]: NodeStatusUVE: data= [ name = process_status= [ [ [ module_id = contrail-collector instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (_iter6) = 10.10.0.100:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = Cassandra server_addrs= [ [ (_iter6) = 10.10.0.100:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (_iter6) = 10.10.0.100:5673, ] ] status = Up description = RabbitMQ connection established ], [ type = Database name = :Global server_addrs= [ [ (_iter6) = 10.10.0.100:9042, ] ] status = Up description = ], [ type = Redis-UVE name = From server_addrs= [ [ (_iter6) = 127.0.0.1:6379, ] ] status = Up description = Redis(From) handling the auth callback ], [ type = Redis-UVE name = To server_addrs= [ [ (_iter6) = 127.0.0.1:6379, ] ] status = Up description = Redis(To) connecting to CallbackProcess ], ] ] description = Collector connection down ], ] ] ]