Describe the bug
If the gRPC server produces data faster than e.g. the network link can handle, instead of data being consumed slower, it piles up in a process inbox until the app runs out of memory.
This happens regardless of whether you use GRPC.Stream.run_with/3 or the older GRPC.Server.send_reply/2 (which run_with wraps anyway).
Based on what I see in the source code, the call chain is:
GRPC.Stream.run_with/3GRPC.Stream.send_response/3 (private method)GRPC.Server.send_reply/2GRPC.Server.Stream.send_reply/3[adapter].send_reply/3 (always GRPC.Server.Adapters.Cowboy)GRPC.Server.Adapters.Cowboy.Handler.stream_body/3send/2
The send call at the bottom is non-blocking and sends the message to what I assume is cowboys inbox.
To Reproduce
- Create an infinite stream by for instance sending the same message over and over.
- Open something that lets you view the top processes by memory, for instance
htop.
- Make a gRPC call that consumes slowly. This can be done by using
grpcurl and piping the response into less
- Watch the memory consumption of the beam process grow indefinitely.
- Alternatively inspect the message queue length directly:
:erlang.process_info(stream.payload.pid, :message_queue_len)) and watch it grow.
Expected behavior
GRPC.Stream.run_with/3 slows down consumption of data when downstream consumers can't keep up, and the memory consumption doesn't grow indefinitely.
Versions:
- OS: MacOS
- Erlang: 28
- Elixir: 19.5
- mix.lock(grpc, gun, cowboy, cowlib):
Additional context
A workaround is to periodically inspect the queue length with :erlang.process_info(stream.payload.pid, :message_queue_len)) and if its above some threshold just sleep until its below the threshold.
Describe the bug
If the gRPC server produces data faster than e.g. the network link can handle, instead of data being consumed slower, it piles up in a process inbox until the app runs out of memory.
This happens regardless of whether you use
GRPC.Stream.run_with/3or the olderGRPC.Server.send_reply/2(whichrun_withwraps anyway).Based on what I see in the source code, the call chain is:
GRPC.Stream.run_with/3GRPC.Stream.send_response/3(private method)GRPC.Server.send_reply/2GRPC.Server.Stream.send_reply/3[adapter].send_reply/3(alwaysGRPC.Server.Adapters.Cowboy)GRPC.Server.Adapters.Cowboy.Handler.stream_body/3send/2The send call at the bottom is non-blocking and sends the message to what I assume is cowboys inbox.
To Reproduce
htop.grpcurland piping the response intoless:erlang.process_info(stream.payload.pid, :message_queue_len))and watch it grow.Expected behavior
GRPC.Stream.run_with/3slows down consumption of data when downstream consumers can't keep up, and the memory consumption doesn't grow indefinitely.Versions:
Additional context
A workaround is to periodically inspect the queue length with
:erlang.process_info(stream.payload.pid, :message_queue_len))and if its above some threshold just sleep until its below the threshold.