OK, first of all, you are likely to get frustrated. Any time I have to do custom sorting in a chart, I end up really frustrated at least once.
You are going to have to put the date into the chart somehow, somewhere. Otherwise, you just can't sort on it. However, it may be possible to hide it.
Here's the "easy" solution. Create your bar chart on both the Date and Build ID fields, with Date first. This will do your sorting properly. Then, go into Chart Options > Legend. At the bottom, you should see a tick box for "Swap Series/Groups." Go ahead and select this. You probably want to turn off the legend at this point, as long as you're in here. This will now show your data by ID in order of Date. (Make sure you're using a stacked bar chart, or you will get a lot of blank space.) This method tends to fail, however, if you have more than one ID for a given date (well, not fail really, but you'll get multiple bars that are the same color), or a single ID appearing on multiple dates (in which case you will see stacked bars, with the area for each date). Depending on your data, this may or may not be an issue for you.
Here's another possible solution. It requires some manual intervention on your part, generally. However, I use it frequently for simulating groups where none actually exist. Create a formula, that would look like:
{MyReport.BuildDate} & " " & {MyReport.BuildID}
Now you can simply create your chart on this. Then, you can go in an manually edit each axis label to delete the date portion. (I have seen a tip in Brian's book that you could use HTML tags to modify the string, and set the color of the Date portion to be color white, and thus be invisible. I have never tried this myself, however.) This will do what you want perfectly, but it is a manual, tedious process each time you run the report.