, I will have to start downloading the headers for the larger group all over.
Not if you just tell it to download headers normally. Once you "download all headers" one time, it'll continue to download them from there it last left off.
I doubt restarting will fix anything though. Since you're downloading from 4 groups at once. It might just mean that the headers for the particular group you're looking at haven't been imported yet. What's your "Storage Age" set to? If it's set to a shorter age than the age of the newly downloaded headers, the new headers are just deleted out of hand until you get headers newer than "Storage Age".