are you daisy chaining the joins?
if so make sure all the joins daisychained with the color table in that chain are also outer joined.
if that is not a solution I would next test if the where conditions are teh source of the issue. I would remove the where clauses, not use the color table, run the report, drop the color table and see if that again enforces the join and the record count changes.