Question 2. True or False: Hypothesis testing draws conclusions about the statistical sample that was studied.
There are two important definitions here:
Population: the entire collection of objects, materials, or people that is under investigation, or whose properties need to be determined.
Sample: a part of the population that is removed for testing/evaluation.
It should be intuitive that the sample should be "representative" of the population. There are entire books written just on sampling procedures. And in clinical studies, the number and type of patients is a critical component to success of the study (and remember that the success of the study eventually means regulatory approval for the device or treatment). So, for example, it would seem to make sense that if we are developing something to be used on/with the elderly, testing it on young patients doesn't seem to make much sense.
There are many ways to remove a sample from a population. The word "random" always is used, but it is more than just a random grab of subjects from the pool. We need to ensure that the number of, say, males and females, matches the percentages in the population that we are considering. There may be other considerations for age, weight, ethnicity, marital status, or any other factor that might be important to study. But in every case, we want the sample to match the population as closely as possible.
When we analyze the data, we will generate statistical results for the sample. But we don't really care about the sample; we care about the population. The assumption in our sampling plan is that our specified sample is going to be representative of the entire population, so that any conclusions that we reach about the sample can be applice to the population.
So, the answer to Question Two is: FALSE. The purpose of the testing is to draw conclusions about the population, not the sample. (Some might accuse me of "splitting hairs," or playing word games, or putting in a "trick" question. And they are correct, to some extent.) We must always remember what the larger goal is for the study (our hypothesis), and focus on getting the best possible data, so that we can generate the best possible conclusions. Generally, I think all of the responses had the right idea, even if they said true.
Now, let's relate this to the original issues regarding the safety of pharmaceuticals and medical devices. In a world of billions of potential users, we hope to draw conclusions from a few thousand, or in some cases, a few hundred members of the population. Is it any wonder that sometimes we miss a particular sub-group of patients, who might be acutely sensitive to the treatment? I am actually amazed that we don't have more problems than we do! It is a credit to the people who design the studies, and those who approve them, that they are able to get it right almost all of the time. But there is always the (5 %) probability that we will get it wrong - and say it did not have serious problems, when in fact it did (remember our Question One discussion!)
One group of advocates wants our pharmaceuticals to get to market sooner, and at less cost. Easy to do. We cut down the number of patients (size of the sample) and reduce the length of testing. Mission Accomplished! But wait. In doing so, we increase the probability of missing a problem, which means a higher probability of finding them once the treatment reaches the market (which is really just a larger sample).
Another advocacy group (including and/or supported by the legal community) wants "safer" treatments on the market. Fine. Larger test groups and longer studies should help that situation. Of course, the cost will go up along with the length of time to market.
The ultimate answer? We can't win. We can only manage the "probabilities."
Question Three: Hypothesis testing can answer the question"Is it safe?"
Post your answers.