No, this is normal. Using the microphone uses significantly more power. Here are the battery specs right off Apple.com
AirPods with Charging Case: More than 24 hours listening time, up to 11 hours talk time
AirPods (single charge): Up to 5 hours listening time, Up to 2 hours talk time
15 minutes in the case equals 3 hours listening time or over an hour of talk time