P-Value Misinterpretation

The wrong statement is saying P(no real effect) The correct statement is saying P(saw these results | no real effect) Consider two extremes, for the same 5% threshold:1) All of their ideas for experiments are idiotic. Every single experiment is for something that simply would never work in real life. 5% of those experiments pass the threshold and 0% of them are valid ideas.2) All of their ideas are brilliant. Every single experiment is for something that is a perfect wa

Sounds like the old p=0.05 problem!

You're making the same mistake in reverse. The p-value is not a marker of correctness, but of confidence. If something doesn't reach the p-value cutoff, that doesn't mean that the effect doesn't exist, just that it didn't meet the confidence level. A paper could be entirely correct in its hypothesis, yet not meet the confidence level in its experimentation.

Plaster this: https://en.wikipedia.org/wiki/P-value

You may be referring to p-values, and the arbitrary 0.05 threshold but in that case, the connection is risky at best and wrong in general (common misinterpretation of p-value as probability of alternative being false)

Yes isn't p usually supposed to be less than 0.05 for the it to usefull?

What’s wrong with statistical significance?

Statistical significance is not a bright-line test. The p value reported is the probability that the observed effects occur if there are actually no effects. Notably, p is always greater than zero.In general, when less data is available, larger p values are considered more interesting, while when more data is available, p must be smaller to be interesting. In particle physics, p p

That's a whole thing in statistics:https://en.wikipedia.org/wiki/Power_of_a_test

Related: "The ASA Statement on p-Values: Context, Process, and Purpose"- https://www.tandfonline.com/doi/full/10.1080/00031305.2016.1...- https://news.ycombinator.com/item?id=30324223