The key to its solution, though, is just to distinguish between knowledge and experience, both of which may be physical states, but different physical states. Mary may know everything physical about color vision but still not have experienced color vision, and this wouldn't tell you anything about whether or not the experience itself was a physical state. With this understanding, we can attack the argument in a number of ways:
- If the phrase "know what it's like" to feel something is just used as a synonym for feeling that thing -- so that, for example, pre-verbal infants, or cats, or bats could also be said to "know what it's like" to experience such-and-so even though they lack concepts -- then the word "know" is just being used to mean two different things. There may be no facts, physical or otherwise, left over from Mary's knowledge in the first sense of "know", even though she lacked experience in the second sense of "know".
- On the other hand, if we're really talking about things like the concept of "red", e.g., then we would just say that, according to physicalist interpretations, the premises of the thought experiment are self-contradictory -- it cannot be the case both that Mary has complete physical knowledge of color vision and that she lacks the experience of color vision (which is either a form of, or an essential ingredient of, physical knowledge).
The last case might seem a little more arguable, in that people might want to say that physical knowledge, as such, is objective knowledge, and is independent of phenomenal experience, which is inherently subjective. But this then gives the game away -- if you define "physical knowledge" as knowledge devoid of phenomenal content, then you hardly need a thought experiment to show that some knowledge is not physical (I'd say, in fact, that all knowledge is not physical in that sense), but you win the point only by definitionally begging the question.
In general, it seems to me that this debate, like many in this area, has been sustained by a failure to recognize how important is the distinction between basic phenomenal consciousness (i.e., experience) and the distinctive features of language-based consciousness (e.g., knowledge).