Hello Amanda.
The water from the dispenser is cold. If the water is not cold, the cold water in the refrigerator's reservoir has been used up.
The reservoir holds about a quart of water, or approximately 2 - 4 glasses of water. So if the water is warm, it will take time to cool down again. ^Gaby