I haven't worked with CE, but I do extensive work with BOE, so I have some thoughts:
Check the server where CE is running for available disk space. You may be running out of space due to the temp files that Crystal was not good about cleaning up. To do this, go to the Local Settings\Temp folder for the user ID that runs the CE services and clear out any files that are there. When Crystal processes reports, it does a lot of swapping to disk. If your disk is getting full or the available space is fragmented, this will slow down report processing.
Beyond that, you're correct that the summaries are being recalculated on drill-down, so there's probably nothing much that you can do about that.
-Dell