To stop players inviting you to a game, you can go to Settings, and toggle "Allow matchmaker challenges" Off. You can also toggle it back On from this menu.

When you get invited to a game, you can click the x in the top right of the game invite and select "Decline; Block Player" to never get invites from that player again

The suggested players you get that for example say "Similar Average Word Score" are not challenges so you can safely ignore these if you don't want to play them.