Spring AMQP/RabbitMQ – java.net.SocketException: Connection reset by peer (Write failed)

by

I have a RabbitMQ server in a Kubernetes cluster that I am connecting to using Spring AMQP with default settings and RabbitTemplate class.

I have a scheduled method in a sender Spring Boot Microservice that sends a message to RabbitMQ queue using RabbitTemplate.convertAndSend(). The scheduled method is called every 15 seconds and the message is consumed by a receiver Spring Boot microservice and processed soon after it is received by the RabbitMQ server.

Generally, the connection between the RabbitMQ server and the sender Spring Boot Microservice is okay, but rarely (2 times in the last 6 months), the following exception is received and the microservice does not reconnect.

org.springframework.amqp.AmqpIOException: java.net.SocketException: Connection reset by peer (Write failed)
    at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:70)
    at org.springframework.amqp.rabbit.connection.RabbitAccessor.convertRabbitAccessException(RabbitAccessor.java:113)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:2192)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:2138)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:1065)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.convertAndSend(RabbitTemplate.java:1130)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.convertAndSend(RabbitTemplate.java:1123)

After this exception, the sender microservice gets the following broken pipe exception when trying to send messages to the RabbitMQ server.

org.springframework.amqp.AmqpIOException: java.net.SocketException: Broken pipe (Write failed)
    at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:70)
    at org.springframework.amqp.rabbit.connection.RabbitAccessor.convertRabbitAccessException(RabbitAccessor.java:113)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:2192)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:2138)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:1065)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.convertAndSend(RabbitTemplate.java:1130)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.convertAndSend(RabbitTemplate.java:1123)

The sender microservice has the following configured, aside from defaults:

# Don't requeue rejected deliveries
# Messages can get requeued due to exceptions thrown on the consumer thread
spring.rabbitmq.listener.simple.default-requeue-rejected = false

# Change Spring RMQ listener mode to NONE from default AUTO
# https://docs.spring.io/spring-amqp/reference/html/#containerAttributes
spring.rabbitmq.listener.simple.acknowledge-mode = NONE

The sender microservice needs to be restarted to re-connect to the RabbitMQ server and to send messages successfully.

I haven’t been able to replicate the issue. Is this just a network glitch that is not preventable?

Is there a way to force Spring AMQP to reconnect to the RabbitMQ server?

To fix the issue, I restarted the sender microservice and that reconnected to the RabbitMQ server.

I am using all default settings for the Spring AMQP connection, so I am not sure what could be causing the network issue.

I checked the connections between other microservices and the RabbitMQ server and they were not affected by this, even if they are subscribed to the same queue.

On the consumer side, the listener container will automatically keep trying to reconnect.

On the producer side, to retry, you need to add a RetryTemplate (with appropriate retry and back off policies) to the RabbitTemplate.

However, I don’t understand why you would get broken pipe exceptions and retrying might not help with that. Spring delegates all I/O operations to the amqp-client. You might need to look in the server logs for clues.

I suggest you reach out to the RabbitMQ engineers on the rabbitmq-users Google group. They don’t monitor StackOverflow.

Leave a Comment