The first two swaps every round are free. The following ones cost a number of Gems. Always check the number of Gems in the SWAP button so you know how much it will cost.