1
I have a problem that I am trying to solve which feels like it is a common problem and thus probably has a common solution in the form of a network protocol. My knowledge of network protocols is somewhat limited and I am unable to find an existing protocol to solve this problem using google search if using a different protocol is the solution I actually need.
My Configuration
Using MQTT, I have a single producer publishing messages to a topic and potentially several consumers subscribing to that topic. The producer can produce messages at a certain maximum rate. Each consumer wants to process messages at a rate that is unique to that individual consumer. If a consumer receives messages at a rate faster than they can process, they can simply throw away messages.
My Concern
For example, I might have a producer that can push images at say 100 images per second but my 2 consumers can only consume them at a rate of 8 per second and 5 per second respectively. To conserve network resources, the optimal rate of publishing here would be 8 images per second.
My Thought
One way to make this work, is to have the producers and consumers connect to a common “rate negotiating” service whose purpose would be to alter the rate at which the publisher is emitting messages to the shared topic.
Question
Should I consider looking at using another protocol for this purpose or would there be anything else common I should consider to help me address this concern and solve my problem?
Alex Ryan
Posted 2018-03-04T23:03:29.433
Reputation: 111
@PimpJuiceIT: I'm trying to conserve resources (especially network bandwidth but also memory and cpu) by using only what is necessary and am trying to figure out if there is an existing protocol which will help me to do that or if I have to invent my own. A key constraint is that the clients need the latest information possible and will throw away any messages received before the last one. I want to avoid waste. If the publisher is publishing too fast I need a method for the clients to tell it to slow down to a specific rate. – Alex Ryan – 2018-03-06T01:58:39.430
@PimpJuiceIT: Our need is very specific. The consumers must have the very latest image sent by the producer. The clients take much more time to process the image than the speed at which the images can be produced but each consumers processing time varies. We absolutely must have the ability for the consumers to tell the producer the rate at which they would like to receive images and for the producer to decide what the optimal rate is. Without this we are wasting resources which then cannot be used for other things we need to do. I don't see how the MQTT protocol "just handles this for me". – Alex Ryan – 2018-03-06T23:08:39.240
@PimpJuiceIT It's not a client limitation. It's a hard and fast and unalterable requirement. This is the need. If the technology does not meet the need, then a work-around must be found. MQTT is a low level messaging protocol. My intuition is that if a rate negotiating protocol exists, it is a more generic higher level protocol than MQTT which is just one of many ways of doing pubsub. The question was rewritten by someone who assumed the solution to the problem lay within the MQTT protocol but this in unlikely IMHO. – Alex Ryan – 2018-03-07T02:19:44.637
Okay, so you want to use another protocol and not MQTT since it's not sufficient in your configuration—have you already done a google search for "M2M or IOT data protocols" and done your fair share of homework on the subject? What are your thoughts after your assessment while you wait on others to give you an answer otherwise? I look forward to hearing what you find that works best in your environment and you confirm it is adequate and sufficient for the need. – Pimp Juice IT – 2018-03-07T02:55:46.667