This happens when you have pending games, which count towards your active games. The pending games are the challenges you sent to other players to which they haven't responded yet as well as the challenges you have received but haven't responded to yet.

You can delete the game requests sent to your friends and opponents by tapping on them and then choosing 'Cancel Invite'.

Game requests sent to random players cannot be deleted manually. However, they will expire after 24 hours if the players don't respond.