You've got to pop the switch out of the panel first. There's a metal retainer in the panel that the switch will come out of. Use that same small screwdriver between the switch escutcheon and trim panel (on the outside of panel) and pry the switch out of it's holder. You can see the little clips that hold the switch in, in my picture of the back of the switch. (at the ends of the switch.)
In your pictures, you are seeing the backside of the metal retainer that the switch is in and needs to be popped out of. Hope this helps.
You could also push real hard on the back (white plastic) of the switch (you'd actually be pushing on the plug-in that is on the switch.) and get the switch out of the retainer that way.