Is it possible to sort a grouping in a graph by a value but display a string?

Specifically time - I've got a graph that groups by an HOUR value that's formatted like this:

ToText( MyTime, "h:mm tt","am","pm")

The problem is the sorting shows up like this, sorted alphabetically:

10:00 am
10:00 pm    
11:00 am
11:00 pm
etc...

One solution I have is to format to military time (HH:mm) - but this isn't a military application.