-1

Several servers are connected via RPC. There are times when apps in OS have too many RPC calls, which causes the network to become closed. What is the best way to debug or configure my OS?

"msg":"sending ping message: write tcp 127.0.0.1:36802->127.0.0.1:1234: use of closed network connection"
"msg":"handle me:write tcp4 127.0.0.1:1234->127.0.0.1:56244: write: broken pipe"
Shervin Ivari
  • 49
  • 4
  • You ask for a precise and detailed answer, yet provide very little detail for the question. gRPC is generally used in application development so the question likely should be moved to Stack Overflow. – Doug Oct 01 '21 at 06:28
  • @Doug RPC request is something general, but perhaps some limitations in TCP protocol are the cause. The program gives me errors. When I monitor the resources, everything seems fine, and I believe that it's an issue with the operating system. If I can find useful logs on OS side,Please point me to it – Shervin Ivari Oct 02 '21 at 07:06
  • Yes it's something general, which is why you need to be specific. You tagged RPC and gRPC, so which one is the program using? Is this program something common for which source code is available, or developed in house? Which resources have you looked at, for example what does netstat show at the time of an error? Without some information to go on I can only offer my assumption that this is a poorly written custom gRPC application that needs to be modified to use channels appropriately. I can think of multiple other solutions, but without information I can't pick which one to give you. – Doug Oct 03 '21 at 02:58

1 Answers1

0

You don't mention what troubleshooting you have done to reach your conclusion that this is due to too many RPC calls, or any details on the state of the network connections at the point of failure. I assume this error is coming about from port exhaustion due to a lack of connection pooling.

To check for port exhaustion use netstat to get the state of ports on the server. If there are an excessive number of ports listed you likely have a port exhaustion problem.

gRPC pools connections automatically, however poorly written code can stop this from functioning properly by excessively creating new gRPC channels instead of reusing existing ones. I've referenced Microsoft's documentation as it has a description of how creating new channels results in creation of a new HTTP/2 connections.

To correct this you will need to assess your code and modify it to reuse channels more appropriately.

Performance best practices with gRPC

A gRPC channel should be reused when making gRPC calls. Reusing a channel allows calls to be multiplexed through an existing HTTP/2 connection.

If a new channel is created for each gRPC call then the amount of time it takes to complete can increase significantly. Each call will require multiple network round-trips between the client and the server to create a new HTTP/2 connection:

Performance Best Practices

Always re-use stubs and channels when possible.

While doing so you might consider Unix domain sockets rather than TCP sockets. If these applications will eventual operate distributed across multiple machines you should stick to TCP sockets. If they will always run on the same machine you should consder Unix domain sockets.

How to create a GRPC service over a local socket rather then inet in scala/java

gRPC server in Python with Unix domain socket

Doug
  • 842
  • 4
  • 7