You may well be on track with the Router issue, its quite unlikely to be Apache - at least as I see it. Apache itself does not care one bit about the source of a request UNLESS you have made rules (directives or .htaccess) that force it to filter requests.
For the purpose of 'fixing' this, if possible I would get the most 'plain jane' router possible that still handles Fast Ethernet (100Mbit) on all ports and put in inline. You could be dealing with a purely router issue.
But routers don't exist in a vacuum either, there is also how the ISP's traffic management (shaping) is set up. That's something you rarely get a glimpse into. You might contact their help desk and describe the issue - they may know the source.
You also might want to use the traceroute utility from outside your lan, I use that often to see where things have gone awry- sometimes its defective routing tables at the ISP. You should also determine if this delay happens on ALL external networks or just ones you have tested. http://en.wikipedia.org/wiki/Traceroutehttp://www.yougetsignal.com/tools/visual-tracert/
These are just my observations as a network engineer.