Recordings on 1 server not moving / getting lost

All installation and configuration problems and questions

Moderators: gerski, enjay, williamconley, Op3r, Staydog, gardo, mflorell, MJCoate, mcargile, Kumba, Michael_N

Recordings on 1 server not moving / getting lost

Postby perci100 » Mon Feb 19, 2018 12:42 pm

I have an odd issue with one of my asterisk servers .

I first noticed this issue when the drives were filling up on just this one server. I run small drives on the asterisk servers (72g/raid10) . I found out it was filling up with recordings. They are accumulating in var/spool/asterisk/monitorDONE/. Cron is exactly the same on this server as the others , running each audio script separately with --debugX I dont see any errors but the files still remain in the monitorDONE directory and I am always missing a few recordings from this server. What is odd is its not all of them.

Some differences in this server , it is the first telephone server I ever had. Seems to run a little slow via SSH . its an older hp g5 . It is the first sever hit by inbound calls. It is the voicemail server (no its not filling with voicemails) .

Could you point me in the direction of some information on how the whole recording storage/ ftp process works? Particularly when they are stored in monitorDONE and what moves them from there.
8 server cluster web/tel x 5/db/archive(rec only) in production inbound/outbound/AMD/full recording |Vicibox 5.0.3 Standard ISO | VERSION: 2.12-549a BUILD: 160404-0940 | Asterisk 1.8.25.0-vici | No added software all servers in RAID 10
perci100
 
Posts: 74
Joined: Thu Feb 09, 2012 1:47 pm

Re: Recordings on 1 server not moving / getting lost

Postby perci100 » Mon Feb 19, 2018 12:49 pm

to add to this , it seems like the recordings that get lost end up having a different link in the system afterwards. Its the public IP address of the recording server instead of the fqdn set in astguiclient.conf
8 server cluster web/tel x 5/db/archive(rec only) in production inbound/outbound/AMD/full recording |Vicibox 5.0.3 Standard ISO | VERSION: 2.12-549a BUILD: 160404-0940 | Asterisk 1.8.25.0-vici | No added software all servers in RAID 10
perci100
 
Posts: 74
Joined: Thu Feb 09, 2012 1:47 pm

Re: Recordings on 1 server not moving / getting lost

Postby williamconley » Mon Feb 19, 2018 4:04 pm

1) you should avoid adding a reply to yur own post that has never had a resonse by someone. It looks (in the system) like someone answered the post and disappears from the "unanswered" posts list. could cause it to get ignored completely (assuming someone else already took care of you). I clicked on it by accident, or I wouldn't be here. ;)

2) My only guess based on your description would be a DNS issue. If the server randomly loses DNS, it may lose connectivity to your target FTP location and not send files as a result. Oddly enough, lack of DNS will also cause slow login via SSH and make the server appear sluggish when in fact it's merely awaiting a DNS resonse before allowing SSH login (because engineers who program know more than us mere mortals and believe that this should be a thing, and DNS should never fail, of course).

3) this description confuses me:

Code: Select all
I dont see any errors but the files still remain in the monitorDONE directory and I am always missing a few recordings from this server.

Either they are still in monitorDONE or they are missing. They really can't be both.
Vicidial Installation and Repair, plus Hosting and Colocation
Newest Product: Vicidial Agent Only Beep - Beta
http://www.PoundTeam.com # 352-269-0000 # +44(203) 769-2294
williamconley
 
Posts: 20018
Joined: Wed Oct 31, 2007 4:17 pm
Location: Davenport, FL (By Disney!)

Re: Recordings on 1 server not moving / getting lost

Postby perci100 » Mon Feb 19, 2018 9:30 pm

1) I was unaware of the way the system works. Just wanted to include as much info as possible. I guess an edit would have accomplished that. Noted for the future.

2) Would that not also have an effect on Dialing?

3) I should have been more clear, "missing" meaning they are not retrievable from the link that they generate. All of those recordings remain in monitorDONE and never move. Nothing is technically missing , just not in the right place. If the server looses DNS would it not just pick up the files it never transferred on the next run of the script? I was trying to understand a bit better how that system works. I am non stop programming 70+ hours a week so i havent had the time. Just need it to work. Which seems to be a common phrase in production from everyone lol.

I made some adjustments DNS wise lets see how it performs tomorrow.

Thank you for the help. I appreciate it.
8 server cluster web/tel x 5/db/archive(rec only) in production inbound/outbound/AMD/full recording |Vicibox 5.0.3 Standard ISO | VERSION: 2.12-549a BUILD: 160404-0940 | Asterisk 1.8.25.0-vici | No added software all servers in RAID 10
perci100
 
Posts: 74
Joined: Thu Feb 09, 2012 1:47 pm

Re: Recordings on 1 server not moving / getting lost

Postby williamconley » Mon Feb 19, 2018 10:48 pm

perci100 wrote:2) Would that not also have an effect on Dialing?

Depends on your settings. If none of your carriers are DNS based (IP addresses only), then ... no.

perci100 wrote:3) I should have been more clear, "missing" meaning they are not retrievable from the link that they generate. All of those recordings remain in monitorDONE and never move. Nothing is technically missing , just not in the right place.

It's time to discuss where in monitorDONE they are. Each folder has a meaning.

monitor = call in progress
monitorDONE = call completed
ORIG & MP3 (two copies, same file one compressed and the other Not) = mix and compress completed (could be another compression folder, but MP3 is most common)
FTP = pushed to ftp server
FTP2 = pushed to backup ftp server

In theory, the link should ONLY be changed when the file has been confirmed on the FTP location, just before the file is moved to the FTP folder.

perci100 wrote:If the server looses DNS would it not just pick up the files it never transferred on the next run of the script?

That's a good theory. I'd say yes. Read the scripts you have activated to be sure the order of events. Verify your FTP storage isn't full and your HD isn't full on the Dialer.

Note that in a properly working system, ORIG files are disposable. Some point that to /dev/null or run an hourly rm to keep it clean. Only useful if a server dies in the middle of a compression and the MP3 is corrupted.

Same *should* apply to FTP since FTP means its been copied to the FTP server ... but it's best to have the FTP2 system set up so you have two copies of everything and then only delete from the FTP2 folder.
Vicidial Installation and Repair, plus Hosting and Colocation
Newest Product: Vicidial Agent Only Beep - Beta
http://www.PoundTeam.com # 352-269-0000 # +44(203) 769-2294
williamconley
 
Posts: 20018
Joined: Wed Oct 31, 2007 4:17 pm
Location: Davenport, FL (By Disney!)

Re: Recordings on 1 server not moving / getting lost

Postby perci100 » Tue Feb 20, 2018 2:12 am

Im talking specifically about the root of monitorDONE/
none of the other servers store recordings there at least not for any amount of time. for some reason my tel1 server does this and they never move the other 4 dont , same exact cron settings

These are the scripts being run in cron i just cant figure out why the initial recordings are stored in monitorDONE and not moved. ( I checked permissions as well, same as the other servers)

AST_CRON_audio_1_move_mix.pl --MIX
AST_CRON_audio_1_move_VDonly.pl
AST_CRON_audio_2_compress.pl --MP3
AST_CRON_audio_3_ftp.pl --MP3

The archive is an on premise storage array with about 37 Terabytes free the other tel servers have no problem sending recordings to it.

I did just notice there was no ORIG directory inside monitorDONE on that server that is acting up , maybe that was it? Does it need to store them In ORIG before converting them?

I appreciate the help.
8 server cluster web/tel x 5/db/archive(rec only) in production inbound/outbound/AMD/full recording |Vicibox 5.0.3 Standard ISO | VERSION: 2.12-549a BUILD: 160404-0940 | Asterisk 1.8.25.0-vici | No added software all servers in RAID 10
perci100
 
Posts: 74
Joined: Thu Feb 09, 2012 1:47 pm

Re: Recordings on 1 server not moving / getting lost

Postby perci100 » Tue Feb 20, 2018 4:13 am

Wow , Figured out the ORIG directory was missing on that server , Must have been deleted somehow. It was probably me trying to move recordings while WinSCP was freezing/crashing

Anyhow , as soon as I mkdir ORIG , all the recordings moved from monitorDONE.
8 server cluster web/tel x 5/db/archive(rec only) in production inbound/outbound/AMD/full recording |Vicibox 5.0.3 Standard ISO | VERSION: 2.12-549a BUILD: 160404-0940 | Asterisk 1.8.25.0-vici | No added software all servers in RAID 10
perci100
 
Posts: 74
Joined: Thu Feb 09, 2012 1:47 pm

Re: Recordings on 1 server not moving / getting lost

Postby williamconley » Tue Feb 20, 2018 8:33 pm

Excellent postback. 8-)
Vicidial Installation and Repair, plus Hosting and Colocation
Newest Product: Vicidial Agent Only Beep - Beta
http://www.PoundTeam.com # 352-269-0000 # +44(203) 769-2294
williamconley
 
Posts: 20018
Joined: Wed Oct 31, 2007 4:17 pm
Location: Davenport, FL (By Disney!)


Return to Support

Who is online

Users browsing this forum: No registered users and 95 guests