gRPC via CloudFlare results in HTTP/2 internal error code 2

Question

My setup:

.Net gRPC Server <-> Nginx <-> CloudFlare <-> gRPC client (C#/Python)

My .Net gRPC Server configured to support insecured http2, listen at port 50052:

webBuilder.UseStartup<StartupGrpc>().UseUrls($"http://*:50052");
webBuilder.ConfigureKestrel(serverOptions => { serverOptions.ConfigureEndpointDefaults(listenOptions => { listenOptions.Protocols = HttpProtocols.Http2; }); });

Nginx is set to grpc_pass as follow:

server {
    server_name grpc.mydomain.com;
    listen      443 ssl http2;
    ssl_certificate    /etc/nginx/cf_origin_ssl/mydomain.pem;
    ssl_certificate_key /etc/nginx/cf_origin_ssl/mydomain.key;

    proxy_cache off;
 
    location / {
        grpc_pass grpc://localhost:50052;   
    }
}

server {
    server_name mydomain.com;
    listen      443 ssl http2;
    ssl_certificate    /etc/nginx/cf_origin_ssl/mydomain.pem;
    ssl_certificate_key /etc/nginx/cf_origin_ssl/mydomain.key;

    proxy_cache off;
 
    location / {
        proxy_pass localhost:50051;   
    }
}

CloudFlare: Network/gRPC -> On, SSL/TLS -> Full(strict) (with Origin Certificates generated by CloudFlare). I tested, and my web server at mydomain.com worked fine. However, gRPC calls from .Net/C# gRPC Client returns:

Unhandled exception. Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error starting gRPC call. IOException: The request was aborted. Http2StreamException: The HTTP/2 server reset the stream. HTTP/2 error code 'INTERNAL_ERROR' (0x2).", DebugException="System.IO.IOException: The request was aborted.
 ---> System.Net.Http.Http2StreamException: The HTTP/2 server reset the stream. HTTP/2 error code 'INTERNAL_ERROR' (0x2).
    --- End of inner exception stack trace ---
    at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
    at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
    at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer, Boolean partOfSyncRead)
    at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)
    at Grpc.Net.Client.StreamExtensions.ReadMessageAsync[TResponse](Stream responseStream, GrpcCall call, Func`2 deserializer, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken)
    at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)")

I also tried to make gRPC calls from Python, and got a similar error:

Traceback (most recent call last):
    ...
    File "/home/user/miniconda/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
    File "/home/user/miniconda/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.UNAVAILABLE
    details = "failed to connect to all addresses"
    debug_error_string = "{"created":"@1634609018.116476058","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3158,"referenced_errors":[{"created":"@1634609018.116472621","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":147,"grpc_status":14}]}"
    >

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
    ...
    File "/home/user/miniconda/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
    File "/home/user/miniconda/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.INTERNAL
    details = "Received RST_STREAM with error code 2"
    debug_error_string = "{"created":"@1634609018.553728473","description":"Error received from peer ipv4:172.67.179.119:443","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"Received RST_STREAM with error code 2","grpc_status":13}"
    >

In both cases, the gRPC requests did get through CloudFlare, Nginx and reach my gRPC server (The remote procedures got executed). Nginx logs also reported with 200 success code:

116.110.42.123 - - [19/Oct/2021:01:25:22 +0000] "POST /greet.Greeter/CsharpSayHello HTTP/2.0" 200 64 "-" "grpc-dotnet/2.40.0.0" "116.110.42.123" "grpc.mydomain.com" sn="grpc.mydomain.com" rt=0.002 ua="127.0.0.1:50052" us="200" ut="0.000" ul="71" cs=-
116.110.42.123 - - [19/Oct/2021:01:27:57 +0000] "POST /greet.Greeter/CsharpSayHello HTTP/2.0" 200 68 "-" "grpc-python/1.41.0 grpc-c/19.0.0 (linux; chttp2)" "116.110.42.123" "grpc.mydomain.com" sn="grpc.mydomain.com" rt=0.001 ua="127.0.0.1:50052" us="200" ut="0.000" ul="75" cs=-

I googled a lot about CloudFlare gGRPC and Nginx, but couldn't figure out what is wrong.

gRPC via CloudFlare results in HTTP/2 internal error code 2

0 Answers0