Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [kafka]

Kafka is a distributed messaging queue application written in java.

Kafka website: http://kafka.apache.org/

When asking a question about provides relevant information like the kafka version and the version. The design of your cluster should be mentioned with the number of brokers, the topic configuration.

When you have a bug on a particular topic, perform the same test on another topic with different parameters and observe if you can reproduce.

94 questions
1
vote
0 answers

What is the attribute you need to specify for kafka Log start offset metric?

I have an environment where logstash (v6.2.3) is connecting kafka_2.11 cluster (v.1.1.0) using kafka input. What I want to achieve is fetch consumer offset information using kafka-consumer-group.sh . However, I end up with below…
Yu Watanabe
  • 606
  • 2
  • 8
  • 18
1
vote
1 answer

kafka + how to revert topic deletion

just in case we delete the wrong topic as /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper hdpmaster01:2181 --delete --topic gtom.poli.pri.proc Topic gtom.poli.pri.procis already marked for deletion and now we want to revert it what…
jango
  • 59
  • 2
  • 2
  • 12
1
vote
0 answers

Operation APPEND failed with HTTP500?

package org.apache.spark.examples.kafkaToflink; import java.io.ByteArrayOutputStream; import java.io.IOException; import java.io.OutputStream; import java.io.PrintStream; import java.nio.charset.StandardCharsets; import java.util.Properties; import…
Anubhav
  • 11
  • 2
1
vote
1 answer

Cannot use supervisor with Apache Kafka

I have an Ubuntu 16.04 machine where Apache Kafka is installed. Currently, I can make it work flawlessly by using a start_kafka.sh script with the following contents: JMX_PORT=17264 KAFKA_HEAP_OPTS="-Xms1024M -Xmx3072M"…
Matheus Portela
  • 171
  • 1
  • 9
1
vote
0 answers

Kafka module missing on syslog-ng.

I'm trying to install syslog-ng and its kafka module. I'm using docker debian:latest for this if that matters. After running 'apt-get install syslog-ng' the syslog-ng can't reach the kafka module even though they are both installed as I can see in…
kub1x
  • 111
  • 5
1
vote
2 answers

Rsyslog. Disk-assisted queue for omkafka

I have following pipeline: nginx -> unix_socket -> rsyslog -> omkafka module -> kafka For omkafka I am using following config: module( load="impstats" interval="10" # how often to generate stats resetCounters="on" # to get…
Samriang
  • 163
  • 1
  • 11
1
vote
1 answer

Securing Kafka REST API endpoints

We plan to use Kafka for publishing updates to our customers. The plan is to use the dedicated topic for each client. To makes things simpler for them we want to utilize REST API. So that user1 will be able to get updates from /api/topic1 and won't…
the_joric
  • 113
  • 4
1
vote
1 answer

Kafka acl without kerberos

I'm trying to set up a single kafka server with certificate authentication and access controls based on those certificates without the bother of setting up a kerberos service. Setting upp the certificate authentication is easy. But I enable the acl…
Elvar
  • 325
  • 2
  • 7
1
vote
1 answer

Page cache in Kafka

We are receiving alerts for kafka memory utilization in our kubernetes cluster and analysing it seems that most of the memory is page cache. I am aware that kafka uses loads of page cache for IO performance and reads/writes are cached before…
Putharekulu
  • 21
  • 1
1
vote
1 answer

No connection to Kafka from Faust client

I am having a hard time connection to a machine running Kafka from a client running a Faust script.The script looks like this: import faust import logging from asyncio import sleep class Test(faust.Record): msg: str app = faust.App('myapp',…
ElToro1966
  • 167
  • 2
  • 7
1
vote
1 answer

zookeeper ssl alert "alert bad certificate"

I am using Kafka (version 2.3.0) and Zookeeper (version 3.5.5-3) - the stable version is 3.6.3. When I test the SSL of my Zookeeper using this command: openssl s_client -showcerts -connect 127.0.0.1:2280 -CAfile /certs/ca-chain.cert.pem and I am…
John Mark
  • 13
  • 2
1
vote
0 answers

Kafka installation with Confluent vs Apache Kafka

for now we have HDP 2.6.5 version , with Kafka cluster with Kafka version 0.1 all Kafka machines are installed on RHEL version 7.6 all machines with RHEL 7.6 are on DELL physical machines we want to move to Apache Kafka or Confluent Platform The…
King David
  • 433
  • 4
  • 17
1
vote
2 answers

The Debezium Kafka based KAFKA_ADVERTISED_LISTENERS property setting is ignored

I have a Docker Compose based Swarm with a Zookeeper service that starts fine, but the Kafka service cannot start. It keeps using the default KAFKA_ADVERTISED_LISTENERS property: debezium_kafka.1.5hhrqfp5kqts@stephane-pc | Using…
Stephane
  • 227
  • 1
  • 12
1
vote
1 answer

PUB/SUB Message Queue with server side filtering?

I am looking for a message queue to realize a chat application. The most important requirement is: when a subscriber specifies a filter criteria, all the messages need to be filtered on the server side. Unfortunately I don't find definite answers to…
Purple Alien
  • 11
  • 1
1
vote
0 answers

How can I further analyze ram usage in a possible leak? [debian@HyperV + Kafka + Zookeeper]

We have a 3-box cluster Kafka+Zookeeper, in a virtualized enviroment (in Windows 2019 HyperV). In the machines' config dynamic memory is disabled, and they have 4G RAM each (I know it's a very few amount for this kind of cluster, but the usage is…
Mise
  • 11
  • 2
1 2
3
4 5 6 7