Troubleshooting "Cannot write to IPC socket: Broken pipe" in Zabbix
If you are seeing the error cannot write to IPC socket: Broken pipe in your Zabbix Server or Zabbix Proxy logs, it typically means one internal Zabbix process is trying to send data to another process that has already closed its end of the communication channel. This often leads to unstable performance or even a full stop of services like the preprocessing manager. Common Causes
System File Limits (ulimit): The most frequent cause is the Zabbix user hitting the Linux "Open Files" limit. When Zabbix cannot open new file descriptors for internal communication, it drops connections, resulting in a "Broken pipe."
Preprocessing Service Crashes: If the preprocessing service stops (due to memory leaks or bugs in specific versions like 6.0 or 7.0), other processes trying to send data to it will fail with this error.
Permission Issues: Lack of permissions for the Zabbix user to write to its own PID or log files can disrupt process communication.
TLS/Network Interference: In some containerized or cloud environments, NAT or TLS overhead can cause sudden connection drops between the server and its internal listeners. How to Fix It 1. Increase the Open Files Limit
In most Linux distributions, the default limit is 1024, which is often too low for busy Zabbix instances.
Manual Check: Run su - zabbix -c 'ulimit -aHS' -s '/bin/bash' | grep open to see current limits.
Systemd Configuration: Edit the Zabbix service file (e.g., /lib/systemd/system/zabbix-server.service) and add or update the following line: LimitNOFILE=4096 Use code with caution. Copied to clipboard
Apply Changes: Run systemctl daemon-reload and restart the service. 2. Verify Service Connectivity Check if the internal services are actually listening.
Log Check: Run tail -f /var/log/zabbix/zabbix_server.log to identify which specific process (e.g., "preprocessing service") is refusing connections.
Process Status: Check if all Zabbix daemons are running with ps ax | grep zabbix. 3. Adjust Preprocessing Settings
If you are on an affected version (like early 6.0.x or 7.0.x), consider these steps mentioned in Zabbix Community Forums:
Increase the StartPreprocessors value in your configuration to handle higher loads.
Check for recent updates; some "Broken pipe" issues are known bugs resolved in later sub-versions (e.g., upgrading to 7.0.3+). 4. Disable TLS for Testing
If the error occurs during communication between a Proxy and Server, temporarily disable TLS in the configuration to see if a certificate or encryption overhead is causing the timeout. zabbix cannot write to ipc socket broken pipe upd
Are you seeing this error specifically after a version upgrade, or did it start happening as your monitored host count increased?
Zabbix Server Unstable After Platform Migration/Upgrade to 6.0
The error "cannot write to IPC socket: Broken pipe" in Zabbix typically occurs when the Zabbix server or proxy process hits the operating system's limit for open file descriptors. This prevents internal processes (like the preprocessing worker) from communicating, causing the "Broken pipe" signal when one end of the communication channel closes unexpectedly. Step 1: Increase File Descriptor Limits
The most common cause is the default Linux ulimit (usually 1024), which is often insufficient for Zabbix 6.0+ or high-load environments.
Modify the Systemd Service File:The most reliable way to apply this is through the service unit file. Create or edit an override for the Zabbix server: sudo systemctl edit zabbix-server Use code with caution. Copied to clipboard
Add the Limit:In the editor, add the following lines to increase the limit to at least 4096 (or higher for large environments): [Service] LimitNOFILE=4096 Use code with caution. Copied to clipboard Reload and Restart:
sudo systemctl daemon-reload sudo systemctl restart zabbix-server Use code with caution. Copied to clipboard Step 2: Update System-Wide Limits
If you aren't using systemd or want to ensure the zabbix user has higher limits globally, update the security configuration: Open /etc/security/limits.conf and add these lines: zabbix soft nofile 4096 zabbix hard nofile 10240 Use code with caution. Copied to clipboard
Verify the change by switching to the zabbix user:su - zabbix -c 'ulimit -n' Step 3: Check for Service Crashes
A "Broken pipe" can also mean the preprocessing service has crashed, leaving other processes unable to write to its socket.
Inspect logs: Run tail -n 100 /var/log/zabbix/zabbix_server.log to check for "connection refused" or "cannot connect to preprocessing service" errors.
Check process status: Ensure the main service is running with ps ax | grep zabbix_server.
Database Health: Ensure your database isn't hitting connection limits, as this can cause Zabbix child processes to exit prematurely. Step 4: Verify Socket Permissions
Ensure the Zabbix user has the necessary rights to create and write to sockets in its temporary directory (usually /tmp or /var/run/zabbix/). If you recently moved these directories or changed permissions, the IPC communication may fail.
Did you notice this error immediately after a version upgrade or a surge in monitored items? Troubleshooting "Cannot write to IPC socket: Broken pipe"
Zabbix Server Unstable After Platform Migration/Upgrade to 6.0
The "cannot write to IPC socket: Broken pipe" error in Zabbix
usually indicates that a Zabbix process (like the server or proxy) tried to send data to another internal service (like the preprocessing manager
), but that service had already closed the connection or crashed Common Causes System File Limits (ulimit):
The Zabbix user may be hitting the maximum number of open files
. When Zabbix cannot open new file descriptors, internal communication fails. Resource Exhaustion: High memory usage or a full History Cache can cause internal services to hang or restart unexpectedly Service Crashes: preprocessing manager availability manager
crashes, subsequent attempts to communicate with them result in a broken pipe Upgrade Issues:
This error is frequently reported after migrating or upgrading to Zabbix 6.0 Troubleshooting & Fixes 1. Increase Open File Limits (ulimit)
If your logs also show "Too many open files," you must increase the limit for the Zabbix user Check current limits: su - zabbix -c 'ulimit -n' Increase the limit in /etc/security/limits.conf zabbix soft nofile 10000 zabbix hard nofile 10000 Use code with caution. Copied to clipboard
, add the following to your Zabbix server service file (usually /lib/systemd/system/zabbix-server.service [Service] LimitNOFILE=10000 Use code with caution. Copied to clipboard 2. Monitor and Expand History Cache If the IPC error is linked to the preprocessing service , it may be due to a full history cache Check logs: Look for "History cache is full" Increase the HistoryCacheSize zabbix_server.conf file (e.g., from or higher). 3. Check for Service Crashes Look for the preprocessing manager
PID in your logs. If you see it restarting frequently, it may be crashing due to a specific malformed item or excessive load Increase the for the specific service to get more detail: zabbix_server -R log_level_increase="preprocessing manager" 4. Verification & Clean Restart
After making changes, perform a clean restart of the Zabbix server. In some cases, orphaned processes can hold onto sockets even after the main service is stopped systemctl stop zabbix-server Verify no zabbix processes remain: ps aux | grep zabbix systemctl start zabbix-server Key Log Indicators Log Message Likely Root Cause cannot write to IPC socket: Broken pipe Peer service closed the connection unexpectedly [24] Too many open files for the Zabbix user is too low cannot send data to preprocessing service Preprocessing manager has crashed or is overwhelmed Connection refused [111] The target service is not running at all Are you seeing these errors on a Zabbix Server
? Knowing the component will help pinpoint the specific internal service that is failing.
Zabbix Server Unstable After Platform Migration/Upgrade to 6.0
Here’s a ready-to-use post for a technical forum (e.g., Reddit, Zabbix community, Stack Exchange, or LinkedIn). It clearly describes the issue, possible causes, and solutions for the Zabbix “cannot write to IPC socket: broken pipe” error, specifically related to UDP items. Raise log level for that process (not recommended long-term)
If you just need to silence the flood of errors while investigating:
But better to fix root cause.
If the script runs, prints nothing, and exits before Zabbix can read the output, the pipe breaks.
Fix: Ensure the script writes a valid value to stdout, even if it’s just an error message:
#!/bin/bash
# Example: always output something
value=$(some_command 2>/dev/null)
if [ -n "$value" ]; then
echo "$value"
else
echo "ZBX_NOTSUPPORTED"
fi
If this error appeared immediately after a yum update, apt upgrade, or manual binary replacement:
Full Stop: Ensure the Zabbix process is completely dead before starting.
systemctl stop zabbix-server
ps aux | grep zabbix # Ensure no stray processes exist
Stray processes from the old binary version often hold onto IPC sockets, causing "Broken pipe" errors for the new binary.
Clean Start: Start the service.
systemctl start zabbix-server
Search logs:
# On Zabbix server
grep -i "broken pipe" /var/log/zabbix/zabbix_server.log
Solution 6: Upgrade Zabbix (If on Version < 5.0)
Older Zabbix versions (3.x, 4.x) had known bugs in IPC handling under load, especially with the upd (update) process. Zabbix 6.2+ and 7.0+ have substantially improved IPC resilience.
Check your version: zabbix_server -V. If older than 5.0 LTS, plan an upgrade.
Debugging Steps
-
Enable debug mode on the agent:
sudo systemctl stop zabbix-agent
sudo zabbix_agentd -f -c /etc/zabbix/zabbix_agentd.conf -l /tmp/zabbix_debug.log
Then trigger the failing item via zabbix_get.
-
Check agent log for context around the broken pipe:
grep "broken pipe" /var/log/zabbix/zabbix_agentd.log -A 2 -B 2
-
Isolate the UserParameter – temporarily comment out others, test one by one.