Checking hypotheses and the problem of p-values

Vast majority of scientific research begins with an idea how the world works according to the proposer. The proposer formulates his hypothesis and tries to prove it using scientific method, usually checking his experiments or observations using varying statistical tools. These tools are used to process the collected data and either confirm his initial hypothesis or to reject it in comparison to the alternatives.

One of the methods is the so-called critical value approach (e.g., see on Penn State Eberly College of Science website). This method relies on the researcher to set a precision standard to the statistical test and accept or reject hypothesis based on it. Usually different branches of science have their own set of rules how small the error could be tolerated. For example in life sciences it is common to see that most of published papers report statistical significance of \( p<0.05 \) (meaning that probability of error is less than \( 5\% \)), while in physics it is rather frequent to hear about the precision of \( 5 \sigma \) (probability of error is less than \( 5.7 \cdot 10^{-5} \% \)).

From the first glance it appears that the methods lacks drawbacks. But in the context of current science publishing tradition - mostly positive results being published - the drawbacks are evident. All statistical methods rely on numerous samples being made - so in order for these kind of test to work numerous independent groups should repeat the same experiment and obtain similar conclusion. Otherwise there is a significant possibility of a positive result being just a successful fluke. Having in mind pressure to publish more pressure there is also a risk that the same research group would repeat the same experiment until getting the desired statistical significance (waiting for a fluke to happen).

I did my best to enlighten you to this problem, but there is a rather significant chance that Hank Green will do better in this SciShow video I invite you to see.

For the ones who are more interested in technical detail I would like suggest reading a draft by Nicholas Nassim Taleb (see on Fooled by Randomness website: https://fooledbyrandomness.com/pvalues.pdf).

Extra Credits: Hyperinflation MMOs

In computer games players control characters and earn money by doing quests and killing monsters. They spend these money for upgrades and other neat stuff (such as own house, animal and etc.). But this mechanic has inherent problem - earned money are created literally from nothing. The money, which goblin chieftain had, were never used in games economy. They are just a reward for player. The same, though less obviously, story is with NPC - they give the same task they were programmed to give and the reward is also preset. Game economics has nothing to do with reward given. If player is just one, then there is no problem, but if you, as a game designer, have thousands of players... you will soon have to face a problem of hyperinflation. More on what it is and how game designers combat it in the following Extra Credits video.

Extra Credits: The history of paper money

Have you ever considered a simple question - what do the numbers you see in your banking account mean? Have you ever questioned yourself why some plastic card is accepted at stores? Everything seems simple - people just believe in money and furthermore electronic transactions. But humanity has walked a long way until we have reached modern understanding of money. Actually it took quite a long time to accept money as they are accepted now. We invite you to watch an excellent tale by Extra Credits on the history of paper money. Below you should see a link to their first video on this topic.

Numberphile: The Seven Bridges of Königsberg

In the first half of the 18th century citizens of Königsberg were very proud of their city and the seven bridges located in it. It is said that city had great views near the river, so the citizens liked to take walks by the river. While on a walk they, obviously, had to traverse the bridges. Naturally the question emerged - would it be possible to choose path so that it would allow to traverse each bridge only once. The answer and the analysis of the problem maybe found in this video by Numberphile.