The easiest way I can think of to do this would be to set up a sub-report that has exactly the same data and parameters as your main report with the group by State (but not by student type). Then create a couple of formulas like this:
New Students
If {student.category} = 'New' then 1 else 0
Returning Students
If {student.category} = 'Returning' then 1 else 0
You'll then place three summaries (Insert menu, select "Summary") in the State group footer in the subreport:
Students from <state>: Count {student.studentID} for group 1.
New Students: Sum
{@New Students} for group 1.
Returning Students: Sum
{@Returning Students} for group 1.
This sub report would go in the Report Footer section of the main report.
-Dell