Steve Pinker mentions this first
It is certainly a virtue to be open-minded, but trouble arises in how we define “open-minded”. It is often used by true-believers as an equivalent of faith, meaning that any odd belief must be accepted regardless of the logic and evidence against it.
Fine, but the same can be said for some skeptics who won't even consider any so called woo woo as materialists call it regardless of overwhelming supportable evidence.
He then mentions
The scientific method is dependent upon methodological naturalism - meaning that we cannot invoke the equivalent of “magic” as an explanation. This is because all ideas in science must be testable - there must be a way to falsify any scientific hypothesis with evidence. A supernatural hypothesis by definition cannot be tested because it is not contained within the laws of nature.
Therefore supernatural notions are not scientific hypotheses because they cannot be tested, and they therefore do not belong on a list of alternate hypotheses. These are the rules of science - if you don’t play by these rules, you are not doing science.
Wait a second science is a method not a position. This is where Steven Pinker pulls in naturalism as the only possible way to gather scientific evidence about reality. That isn't so their are many different views of reality, such as dualism, idealism and neutral monism. If their is an supernatural interaction with our reality and another we can test for that interaction. To say that we can't because it isn't contained within the laws of nature is a cop-out.
He then mentions
But science is actually agnostic toward the question of whether or not there are supernatural forces at work in the universe. Again - definitions get tricky here, because one could argue that any force at work in the universe is by definition natural. But let’s say that a supernatural notion would include the claim that there is an undetectable agency at work in the universe that could arbitrarily suspend the laws of nature. Such a claim is untestable. At best science could detect enduring anomalies - observations that forever defy scientific explanation. But science could never confirm that a supernatural explanation were correct.
Nor could science ever confirm an natural explanation. I agree with you Novella, science is not about comfirmation or proof it is about evidence and probability.
Here is an excellent quote from Sherlock Holmes
“Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth.” - Sherlock Holmes