Here are what I think is the problem with that experiment.
- The author D.R. knew which bottles had the "treated" water in.
- D.R. was the person who received the photos from Tokyo and could have shown bias in his selection of physical photograph samples for evaluation
- The D.R author wrote the website which collected ratings and could have been biased in which photographs were presented for evaluation
- The "treated" bottles were stored in a secure location with limited access, but the control bottles were stored in an easily accessible location, a cardboard box on a desk. So the control bottles were easily accessible and could therefore be tampered with by the very person who knew which bottles were designated for treatment and control. In fact, if you wanted to fake this test then it is the control bottles you'd want access to in order to contaminate so they do not form crystals as readily as the unaltered water.
- If the ratings of the photos on the website (selected by D.R.) had been truly random & unbiased then we should see an equal number of observations for the treated and control samples, but what we actually see is that the control samples were only evaluated half as much as the treated samples.
This shows that the author D.R. not only had opportunity to physically influence the samples but also had knowledge of which samples would need to be influenced in order to produce specific results.
The only assurance we have that this was not the case is a statement from the testers themselves stating they had no bias in the results of the experiment. This is not sufficient, the purpose of a double-blind test is to eliminate the possibility of deliberate manipulation of an experiment. This test did not meet that criteria as D.R. had too much knowledge of the selection of the samples, combined with suitable access to the samples to influence the outcome of the experiment.