Of course, you did not mention whether these agents with on-hook phones are external or SIP or IAX.
Try using a call path to the agent that does not include the "agi" line. DO NOT use this same path for outbound calls to prospects, only to the Agents.
And my earlier remark of "same problem" applies. It's tempting to "summarize" your problem because it's "same as" and not show all your configuration information. Then we can't help you because we have to assume you just have a setting incorrect (which is normal, actually) and suggest you check the manual for configuration.
So ... list all the pertinent configuration information *for your system and all settings related to this particular issue specifically) and perhaps we can help.

1) Welcome to the Party!

2) As you are obviously new here, I have some suggestions to help us all help you:
When you post, please post your entire configuration including (but not limited to) your installation method and vicidial version with build.
This IS a requirement for posting along with reading the stickies (at the top of each forum) and the manager's manual (available on EFLO.net, both free and paid versions)
You should also post: Asterisk version, telephony hardware (model number is helpful here), cluster information if you have one, and whether any other software is installed in the box. If your installation method is "from scratch" you must post your operating system and should also post the .iso version from which you installed your original operating system. If your installation is "Hosted" list the site name of the host.
If this is a "Cloud" or "Virtual" server, please note the technology involved along with the version of that techology (ie: VMware Server Version 2.0.2). If it is not, merely stating the Motherboard model # and CPU would be helpful.
Similar to This:
Vicibox X.X from .iso | Vicidial X.X.X-XXX Build XXXXXX-XXXX | Asterisk X.X.X | Single Server | No Digium/Sangoma Hardware | No Extra Software After Installation | Intel DG35EC | Core2Quad Q6600