ViciBox v.7.0.4 Frequent Time Sync issue

All installation and configuration problems and questions

Moderators: gerski, enjay, williamconley, Op3r, Staydog, gardo, mflorell, MJCoate, mcargile, Kumba, Michael_N

ViciBox v.7.0.4 Frequent Time Sync issue

Postby arunvsadnikov » Tue Jul 10, 2018 2:30 am

Hi Team,

Current Setup.

ViciBox v.7.0.4-170113
Asterisk 11.25.1-vici
Vicidial Version: 2.14b0.5
SVN Version: 2659
DB Schema Version: 1485
32 G RAM, 2 TB HDD, 8 Core Xeon Processor

We currently experiencing "Time Synchronization" issue with vicidial. We got ~22-25 agents dialing on an outbound campaign with ratio 6 with answering machine detection and call recording enabled. We got ~100k leads in the vicidial_list table as well, system load will be varying from 2-4. We are seeing this "Time sync" issue now very frequently and I had to stop and start vicidial to recover. Im pretty much sure its not related to Time instead something broken with Asterisk. I see the following logs in asterisk messages. Anyone experienced this issue before? IS this a capacity problem? or a bug ?

[Jul 10 00:14:23] WARNING[28709][C-000229d4] channel.c: Unable to write to alert pipe on Local/58600085@default-000217e5;1 (qlen = 39): Resource temporarily unavailable!
[Jul 10 00:14:23] WARNING[29282][C-00022a43] channel.c: Unable to write to alert pipe on Local/58600075@default-00021846;1 (qlen = 33): Resource temporarily unavailable!
[Jul 10 00:14:23] WARNING[19966][C-000224ad] channel.c: Unable to write to alert pipe on Local/58600058@default-000212f2;1 (qlen = 35): Resource temporarily unavailable!
[Jul 10 00:14:23] WARNING[29271][C-00022a41] channel.c: Unable to write to alert pipe on Local/58600057@default-00021845;1 (qlen = 51): Resource temporarily unavailable!
[Jul 10 00:14:23] WARNING[29289][C-00022a44] channel.c: Unable to write to alert pipe on Local/58600084@default-00021847;1 (qlen = 81): Resource temporarily unavailable!

FYI, access to the server is very limited and only specific IPs are allowed to connect to SIP(udp 5050) and RTP(udp 10k-20k). I dont see any dos attacks in dmesg as well.


Thanks
~Arun
arunvsadnikov
 
Posts: 25
Joined: Mon Jan 23, 2017 1:50 am

Re: ViciBox v.7.0.4 Frequent Time Sync issue

Postby arunvsadnikov » Thu Jul 12, 2018 2:27 am

Can someone please help us here? This is impacting our production :(
arunvsadnikov
 
Posts: 25
Joined: Mon Jan 23, 2017 1:50 am

Re: ViciBox v.7.0.4 Frequent Time Sync issue

Postby mflorell » Thu Jul 12, 2018 6:45 am

I would suggest upgrading to a more recent svn/trunk revision of VICIdial. Also, look at the Modify Server page and confirm the settings are correct, like the Asterisk Version.
mflorell
Site Admin
 
Posts: 18335
Joined: Wed Jun 07, 2006 2:45 pm
Location: Florida

Re: ViciBox v.7.0.4 Frequent Time Sync issue

Postby arunvsadnikov » Thu Jul 12, 2018 6:59 am

Hi, thanks for the suggestion. We have some custom code changes in vicidial and Im afraid to upgrade it. Is this is something with vicidial or asterisk? Can we try upgrade only asterisk in vicibox?
arunvsadnikov
 
Posts: 25
Joined: Mon Jan 23, 2017 1:50 am

Re: ViciBox v.7.0.4 Frequent Time Sync issue

Postby mflorell » Thu Jul 12, 2018 7:53 am

The version of VICIdial code you are using will not work with the newer version of Asterisk. There were several VICIdial bug fixes since the version you are using, I would strongly suggest upgrading.
mflorell
Site Admin
 
Posts: 18335
Joined: Wed Jun 07, 2006 2:45 pm
Location: Florida

Re: ViciBox v.7.0.4 Frequent Time Sync issue

Postby arunvsadnikov » Thu Jul 12, 2018 1:39 pm

mflorell wrote:The version of VICIdial code you are using will not work with the newer version of Asterisk. .

There are many modifications, I will get in touch with the dev guy and plan for an upgrade.

I have attached the server performance report for one day, Im just wondering is this some kind of capacity issue with asterisk, like software limits. Any feedback on this will be appreciated.

Image
arunvsadnikov
 
Posts: 25
Joined: Mon Jan 23, 2017 1:50 am

Re: ViciBox v.7.0.4 Frequent Time Sync issue

Postby williamconley » Sun Jul 15, 2018 1:29 pm

Server load is visible on each server with "uptime" and/or "htop". If the server is a web server, it's also available at the bottom of the agent list in the real time screen. Server load should never exceed the actual number of cores (safest cruising average server load is half the number of cores).

If you're not exceeding the number of cores for your average server load, it's probably a networking issue causing packets to drop between your agent's web browsers and your server. Especially since it sounds like you have a single-server system, time sync errors can NOT be actual "sync" errors since there's only one server. So the odds are that you have a networking issue. Network packets dropping can cause the system to think there's a sync error because those packets update a time field (missed updates = time is out of sync error, just like actually being out of sync on a multiple server cluster).

Additionally, the asterisk error you reported should also be chased down, but you did not respond to matt's suggestion to verify the admin-> server values with any direct information (like ... the settings in admin->servers including, but not limited to, the actual entry for Asterisk Version). It may not be a bad idea to also list the asterisk version from /etc/astguiclient.conf (and if either of those has changed, you have a bit of an issue since changing those values often requires updating some configuration files which is best done by running the Vicidial install package which might overwrite any custom files).
Vicidial Installation and Repair, plus Hosting and Colocation
Newest Product: Vicidial Agent Only Beep - Beta
http://www.PoundTeam.com # 352-269-0000 # +44(203) 769-2294
williamconley
 
Posts: 20018
Joined: Wed Oct 31, 2007 4:17 pm
Location: Davenport, FL (By Disney!)

Re: ViciBox v.7.0.4 Frequent Time Sync issue

Postby arunvsadnikov » Tue Jul 17, 2018 7:14 am

williamconley wrote:Server load is visible on each server with "uptime" and/or "htop". If the server is a web server, it's also available at the bottom of the agent list in the real time screen. Server load should never exceed the actual number of cores (safest cruising average server load is half the number of cores).

Server load varies in between 2-4 depending on the number of agents online, we got a 8 core cpu, so yes it may go up to load 8 until fully utilized.

williamconley wrote:If you're not exceeding the number of cores for your average server load, it's probably a networking issue causing packets to drop between your agent's web browsers and your server. Especially since it sounds like you have a single-server system, time sync errors can NOT be actual "sync" errors since there's only one server. So the odds are that you have a networking issue. Network packets dropping can cause the system to think there's a sync error because those packets update a time field (missed updates = time is out of sync error, just like actually being out of sync on a multiple server cluster).

I got a chance to take a look at our PRTG graph and couldn't find any packet errors or bandwidth issues. Im working with my service provider for more details on this.

williamconley/Matt wrote:Additionally, the asterisk error you reported should also be chased down, but you did not respond to matt's suggestion to verify the admin-> server values with any direct information (like ... the settings in admin->servers including, but not limited to, the actual entry for Asterisk Version). It may not be a bad idea to also list the asterisk version from /etc/astguiclient.conf (and if either of those has changed, you have a bit of an issue since changing those values often requires updating some configuration files which is best done by running the Vicidial install package which might overwrite any custom files).

Sorry about that Matt, I have verified asterisk version in GUI, its 11.25.1 and in /etc/astguiclient.conf it is 11.

My current plan is to upgrade vicidial code to the latest trunk and work on the custom modifications we have done with the dev guy, btw this may take some time in testing, bug fixing etc. So I decided to add one more telephony server to current setup.

I got a couple of question for you William:-
1. The current installation is vicibox-express, can we convert a vicibox-express installation to cluster without reinstalling the entire stuff? I was trying to add a new telephony server to the existing express installation in my local environment and the installer failed back to Legacy mode. Do you recommend Legacy installation of new Telephony server?

2. Is it possible to checkout a specific revision of vicidial svn during the vicibox-install instead of trunk?

Thanks for all your support. Appreciate that!
arunvsadnikov
 
Posts: 25
Joined: Mon Jan 23, 2017 1:50 am


Return to Support

Who is online

Users browsing this forum: Google [Bot] and 81 guests