By clicking exit, do you mean 'X' out of the window or right-click on the taskbar and click Close?
If so, that does not actually close the control panel (and is an intended feature). It hides itself and minimizes itself onto the taskbar as an icon (down by where your network, sound, and other small icons reside). You can restore it by clicking that icon as well.
If you actually want to close the control panel, you have to hit the 'Quit' button on the interface.
If you are using 1.7.7, there is also a newer 'beta' control panel that is much better if you are not already using it (it is called "xampp-control-3-beta.exe" in the "/xampp" folder).