Skip to content
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Common Issues

ActiveMQ logs shows Memory usage has reached or exceeded configured limits

Logs

 WARN | Transport Connection to: tcp://11.128.88.20:51976 failed: java.io.EOFException
 WARN | Transport Connection to: tcp://11.128.34.1:51176 failed: java.io.EOFException
 WARN | Transport Connection to: tcp://11.128.16.1:62767 failed: java.io.EOFException
 WARN | Transport Connection to: tcp://11.128.26.1:62148 failed: java.io.EOFException
 WARN | Transport Connection to: tcp://11.128.16.1:40315 failed: java.io.EOFException
 WARN | Transport Connection to: tcp://11.128.34.1:36189 failed: java.io.EOFException
 WARN | Transport Connection to: tcp://11.128.26.1:16568 failed: java.io.EOFException
 INFO | Usage(default:memory:queue://Spring.Cloud.Stream.dlq:memory) percentUsage=69%, usage=467812303, limit=668309914, percentUsageMinDelta=1%;Parent:Usage(default:memory) percentUsage=100%, usage=668329690, limit=668309914, percentUsageMinDelta=1%: Usage Manager Memory Limit reached. Producer (ID:transaction-manager-945bf56d59-xv2tq-34973-1736379783901-1:49:3:1) stopped to prevent flooding queue://Spring.Cloud.Stream.dlq. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 964s)

How ActiveMQ Works in This Context

ActiveMQ is a message broker designed to mediate communication between producers (applications sending messages) and consumers (applications receiving messages). Here’s a simplified flow relevant to your situation:

  1. Producer Sends Messages:

    • A producer sends messages to a queue or topic.

    • Messages are stored in the broker's memory or on disk if persistence is enabled.

  2. Consumer Processes Messages:

    • A consumer fetches messages from the queue and processes them.

    • If no consumer is available, messages accumulate in the queue.

  3. Flow Control Mechanism:

    • ActiveMQ enforces flow control to manage broker resources like memory.

    • If memory usage exceeds configured limits, producers are blocked to prevent overloading the broker.

  4. Dead Letter Queue (DLQ):

    • When a message cannot be delivered or processed (e.g., retries are exhausted), it is sent to a DLQ for further analysis.

Understanding the Current Situation

  1. Memory Limits Reached:

    • ActiveMQ is running out of memory, as indicated by the logs:

    • Memory usage details:

    • When memory is full, producers are stopped, resulting in delays or blocking (blocking for: 964s).

    Root Causes:

    • Large message backlog (e.g., unprocessed messages in Spring.Cloud.Stream.dlq).

    • Insufficient memory configuration for the broker.

  2. Connection Failures:

    • Multiple warnings like:

    • These indicate that some clients (producers/consumers) are losing their connections to the broker.

    Root Causes:

    • Network instability or client misconfiguration.

    • Resource exhaustion at the broker (e.g., threads, memory).

  3. DLQ Overload:

    • The queue Spring.Cloud.Stream.dlq is being flooded. This queue is typically used for undeliverable or failed messages.

    Root Causes:

    • Application issues causing messages to be redirected to the DLQ.

    • Lack of consumers processing messages from the DLQ.

How ActiveMQ Handles These Situations

  1. Memory Usage Management:

    • ActiveMQ allocates a memory pool to hold messages temporarily.

    • If this pool exceeds the configured limit, flow control stops producers to avoid memory overflow.

  2. Dead Letter Queue:

    • Messages are redirected to the DLQ when:

      • Delivery fails (e.g., no consumer or consumer errors).

      • Retry limits are exceeded.

  3. Transport Failures:

    • Connections between clients and the broker are maintained via TCP.

    • Failures (e.g., EOFException) occur due to abrupt disconnections or resource limitations.

Troubleshooting Steps

1. Memory Configuration:

  • Increase the memory limits for the broker in the activemq.xml configuration:

  • Monitor memory usage to ensure it's sufficient for your workload.

2. Dead Letter Queue (DLQ) Management:

  • The queue Spring.Cloud.Stream.dlq seems to be accumulating messages. Check why messages are being redirected to the DLQ.

  • Mitigate DLQ growth:

    • Address application issues causing messages to fail.

    • Enable DLQ message cleanup or archiving.

    • Consider applying a TTL (time-to-live) for messages.

3. Connection Stability:

  • Review network health to reduce EOFException occurrences.

    • Ensure low-latency, stable connectivity between clients and the broker.

  • Configure transportConnector with keep-alive options in activemq.xml:

4. Broker Monitoring and Metrics:

  • Monitor ActiveMQ metrics using tools like JMX, Prometheus, or built-in web consoles.

  • Key metrics:

    • Memory usage.

    • Queue sizes.

    • Consumer counts.

5. Thread and Resource Limits:

  • Ensure broker threads and resources are sufficient:

6. Producer Backoff Strategy:

  • Configure producers to handle flow control gracefully. For instance, retry with backoff when blocked.

7. Message Prioritization:

  • Use message priorities to ensure critical messages are not delayed or blocked.

Last updated