How do I take mildew out of colored clothes?

You are right to avoid chlorine bleach, but how about oxygen bleach, like OxiClean? Try soaking the clothes in a very hot water solution with 2x or great amount OxiClean than recommended. I've used 4x for some real stained items without problems. Leave the clothes in the soak for at least 12 hours. Initially swish them around while the water is hot to make certain all parts get treated.