@Stahesh there is a negative and positive prompt area.

If you go to the negative section and put

White hair

Silver hair

Gray hair

That should stop it.