HUGE amount of MySQL TIME_WAIT connections building up

All installation and configuration problems and questions

Moderators: gerski, enjay, williamconley, Op3r, Staydog, gardo, mflorell, MJCoate, mcargile, Kumba, Michael_N

HUGE amount of MySQL TIME_WAIT connections building up

Postby SPAMSAM » Tue Nov 22, 2022 5:13 am

Hi,

I have a cluster of 2 servers (DB/WEB/ASTERISK + WEB/ASTERISK). The CC runs an autodial camp for about 80 agents and everything is running fine without any noticeable issues currently.

However, running 'netstat -plant' yields the following:
Code: Select all
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:5038            0.0.0.0:*               LISTEN      12764/asterisk
tcp        0      0 0.0.0.0:2929            0.0.0.0:*               LISTEN      12775/ip_relay
tcp        0      0 0.0.0.0:18421           0.0.0.0:*               LISTEN      1595/sshd
tcp        0      0 0.0.0.0:40569           0.0.0.0:*               LISTEN      12773/ip_relay
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      2376/master
tcp        0      0 0.0.0.0:4577            0.0.0.0:*               LISTEN      1622/perl
tcp        0      0 0.0.0.0:10050           0.0.0.0:*               LISTEN      20339/zabbix_agentd
tcp        0      0 0.0.0.0:17543           0.0.0.0:*               LISTEN      12774/ip_relay
tcp        0      0 127.0.0.1:5038          127.0.0.1:35598         TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:26338     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:23950     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:25418     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:29180     1x9.xxx.xxx.8:3306      ESTABLISHED 16805/httpd-prefork
tcp        0      0 1x9.xxx.xxx.8:26240     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:26076     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:25538     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:29716     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:26352     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:29532     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:28686     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:26332     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp       18      0 127.0.0.1:5038          127.0.0.1:35226         ESTABLISHED 12764/asterisk
tcp        0      0 127.0.0.1:5038          127.0.0.1:36672         TIME_WAIT   -
tcp        0      0 127.0.0.1:5038          127.0.0.1:32760         TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:24164     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:27668     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:25960     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:28198     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:25710     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:24128     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:24474     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:28448     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:24592     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:24522     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:25602     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:28436     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:28820     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:26254     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:28930     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:27926     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:25358     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:28896     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:29280     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:27428     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:27788     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:27602     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:26508     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:28750     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:24324     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:24854     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:26176     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:29512     1x9.xxx.xxx.8:3306      ESTABLISHED 6838/perl
tcp        0      0 1x9.xxx.xxx.8:29186     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 127.0.0.1:5038          127.0.0.1:33526         TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:29436     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 127.0.0.1:5038          127.0.0.1:30802         TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:29686     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:28514     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:26598     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:25416     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:25406     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:24920     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:25766     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:24062     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:28774     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:29268     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:26370     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:23824     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:24136     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:26160     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:29650     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 127.0.0.1:4577          127.0.0.1:43910         ESTABLISHED 2282/perl
tcp        0      0 127.0.0.1:5038          127.0.0.1:34888         TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:26602     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:24174     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp        0      0 1x9.xxx.xxx.8:24168     1x9.xxx.xxx.8:3306      TIME_WAIT   -
tcp6       0      0 1x9.xxx.xxx.8:3306      1x9.xxx.xxx.20:48924    TIME_WAIT   -
tcp6       0      0 1x9.xxx.xxx.8:3306      1x9.xxx.xxx.20:49118    TIME_WAIT   -
tcp6       0      0 1x9.xxx.xxx.8:3306      1x9.xxx.xxx.20:48818    TIME_WAIT   -
tcp6       0      0 1x9.xxx.xxx.8:3306      1x9.xxx.xxx.20:47554    TIME_WAIT   -
tcp6       0      0 1x9.xxx.xxx.8:3306      1x9.xxx.xxx.20:49026    TIME_WAIT   -
tcp6       0      0 1x9.xxx.xxx.8:3306      1x9.xxx.xxx.20:48682    TIME_WAIT   -
tcp6       0      0 1x9.xxx.xxx.8:3306      1x9.xxx.xxx.20:48072    ESTABLISHED 1590/mysqld
tcp6       0      0 1x9.xxx.xxx.8:3306      1x9.xxx.xxx.20:46190    ESTABLISHED 1590/mysqld
tcp6       0      0 1x9.xxx.xxx.8:3306      1x9.xxx.xxx.20:49326    TIME_WAIT   -
tcp6       0      0 1x9.xxx.xxx.8:3306      1x9.xxx.xxx.20:47250    TIME_WAIT   -
tcp6       0      0 1x9.xxx.xxx.8:3306      1x9.xxx.xxx.20:47272    TIME_WAIT   -


This totals in more than 3000 connections to port 3306 that are in the TIME_WAIT state.

/var/log/mysql/mysqld.log shows the following entries:
Code: Select all
2022-11-22 11:32:25 140376940181248 [Warning] Aborted connection 115749182 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:32:33 140376945350400 [Warning] Aborted connection 115749816 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:32:45 140376920262400 [Warning] Aborted connection 115750653 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.20' (Got an error reading communication packets)
2022-11-22 11:32:48 140376923641600 [Warning] Aborted connection 115750833 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:32:48 140376512866048 [Warning] Aborted connection 115750881 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:33:00 140375882516224 [Warning] Aborted connection 115751626 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:33:22 140375905179392 [Warning] Aborted connection 115753233 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.20' (Got an error reading communication packets)
2022-11-22 11:33:22 140377384380160 [Warning] Aborted connection 115752922 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.20' (Got an error reading communication packets)
2022-11-22 11:33:26 140375881901824 [Warning] Aborted connection 115753151 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:33:26 140375843440384 [Warning] Aborted connection 115753176 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:33:27 140376937260800 [Warning] Aborted connection 115753258 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:33:28 140376928933632 [Warning] Aborted connection 115753266 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:33:30 140375863817984 [Warning] Aborted connection 115753407 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:33:33 140376923948800 [Warning] Aborted connection 115753590 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.20' (Got an error reading communication packets)
2022-11-22 11:33:33 140376926783232 [Warning] Aborted connection 115753611 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:33:36 140375902414592 [Warning] Aborted connection 115753749 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:39:18 140375881594624 [Warning] Aborted connection 115778154 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)
2022-11-22 11:39:18 140376922105600 [Warning] Aborted connection 115778187 to db: 'asterisk' user: 'cron' host: '1x9.xxx.xxx.8' (Got an error reading communication packets)


Is this normal behavior or something to be concerned about that can cause issues? if so how should i go about mitigating the issue?


VICI DETAILS:
VERSION: 2.14-869a
BUILD: 221021-1038
SVN: 3651
SPAMSAM
 
Posts: 70
Joined: Tue Jan 17, 2017 4:00 am

Re: HUGE amount of MySQL TIME_WAIT connections building up

Postby mflorell » Tue Nov 22, 2022 1:49 pm

That does appear to be pretty normal, I just checked on a dialer that has 23 agents and there were about 400 TIME_WAIT entries there.

Can you see what script specifically is causing the "error reading communication packets" errors? My guess is that it would be the send-child processes, which cannot drop off right away or Asterisk might not execute the commands.
mflorell
Site Admin
 
Posts: 18335
Joined: Wed Jun 07, 2006 2:45 pm
Location: Florida


Return to Support

Who is online

Users browsing this forum: Bing [Bot], Majestic-12 [Bot] and 108 guests