vpFREE2 Forums

how to tell if your machine is fair?

#1

Frank started up a discussion on this topic a while ago. Originally it was going to be moved off of vpfree and then decided to remain on vpfree. I haven't seen the final product yet.

For those of you who wonder about such things, if you were to state your hypothesis and say what measurement you would accept to either prove or disprove your hypothesis, I'm sure the throngs of stats people on this board would tell you if your experiment is valid or not. ( Frank's program may do all of this already).

Here's a quickie outline of what the process should be:

1) figure out what you want to prove. This can be due to previous play, rumors, second hand stories or whatever. Note that the previous play cannot be used to confirm or refute the hypothesis.
    You need to make this very specific

2) Set up your experiment.

3) decide on your acceptance criteria

4) Post the info and let the stats guys go at it.

For example, Miss Craps recently posted about going 0 for 35 on dealt trips on 10 play ( converting to quads). This is step 1. Miss Craps believes that the machines she was playing have a quad distibution outside the normal range. So, her hypothesis is that On machine xxx, the distribution of quads is much less can be expected.

For step 2, you need to choose sample size and parameters. I will play 2000 dealt hands of triple play and record the number of times I am dealt trips and the number of times I convert them. I will also keep track of the total number of quads in the sample as an additional measure

For step 3, I can pick 90% sure, 95% sure, 99% sure or whatever number I want. I will need to adjust the number of samples based on the frequency of the event and the confidence level

Post the information and see what the group thinks.

Is this a major pain? Sure it is. But is the only way to really know if there is an issue or not.

This is a very simplified outline. For a good experiment, you would need to look at sources of error, repeats, blocking ( if you suspect certain quads are absent) and data recording errors. It is all doable but it is much easier to say " I played 4 hours and only got 1 quad, the machines must be cheating me"

[Non-text portions of this message have been removed]

#2

greeklandjohnny wrote:

Here's a quickie outline of what the process should be:

1) figure out what you want to prove. This can be due to previous play, rumors, second hand stories or whatever. Note that the previous play cannot be used to confirm or refute the hypothesis.
   You need to make this very specific

2) Set up your experiment.

3) decide on your acceptance criteria

4) Post the info and let the stats guys go at it.

For example, Miss Craps recently posted about going 0 for 35 on dealt trips on 10 play ( converting to quads). This is step 1. Miss Craps believes that the machines she was playing have a quad distibution outside the normal range. So, her hypothesis is that On machine xxx, the distribution of quads is much less can be expected.

For step 2, you need to choose sample size and parameters. I will play 2000 dealt hands of triple play and record the number of times I am dealt trips and the number of times I convert them. I will also keep track of the total number of quads in the sample as an additional measure

For step 3, I can pick 90% sure, 95% sure, 99% sure or whatever number I want. I will need to adjust the number of samples based on the frequency of the event and the confidence level

Post the information and see what the group thinks.

Is this a major pain? Sure it is. But is the only way to really know if there is an issue or not.

I don't see how this is adequate. What number are you trying to
determine? If the sample is x standard deviations from expected, what
chance of a gaffed machine does that translate to? A variable is
missing, but I'm not sure how to describe it. I've gone through the
kind of process you described and made conclusions based on it, but I
never ended up with a number which enabled me to say "I'm x% sure
what's being tested is gaffed." I still had to guess. At what point
is the probability that the machine is gaffed, say, 50%? Besides not
liking the idea of there being a significant possibility of going
years without winning even if the machines are fair, that I despair of
empirically proving they are fair is another reason I don't like
playing with a small theoretical advantage. Ultimately, it involves
trusting government.

#3

If the sample is x standard deviations from expected, what
chance of a gaffed machine does that translate to?

This will depend on

  (1) before any data, how likely do you believe the machine is biased
    e.g. Before the test, you thought there was a 1% change of a gaffed machine
    The test tells you afterwards that there is a 30-40% chance of a gaffed machine.

    Your question is very difficult because it relies on your "a priori" estimate of how likely a bad machine is...
      (a) For Las Vegas, I would set this very low.
      (b) For a machine built by a non-standard company in a foreign country or cruise ship, maybe higher.
          Especially if other people also report low numbers of quads.

  (2) how badly-biased you believe the machine to be
     e.g. a 99% machine (Fair = "Null Hypothesis") vs. 96% machine (96% machine = Hypothesis A)

    A machine that has too many non-quads, might be modeled by a "Zero-Inflated Poisson" distribution;
    e.g. 75% of no quads, 25% of a legitimate VP machine.
      http://data.princeton.edu/wws509/stata/overdispersion.html

  (3) Use t-tests or Bayesian reasoning or etc... to test hypothesis and evaluate beliefs.

  http://en.wikipedia.org/wiki/Statistical_hypothesis_testing
  http://www.stats.gla.ac.uk/steps/glossary/hypothesis_testing.html

In reality, since I don't know if a gaffed machine pays 85%, 90%, 95%, etc...
  (1) I would be really lazy and see if I was +/- 2-3 SDs out.
      In Vegas I would not worry unless I was >3 SDs below normal.
      In an unreliable area, I might even start worrying if I was >1 SD below normal (e.g. maybe play a cheaper denomination as a test?)
  
  In probability langauge,
  (A) Test just the "Null Hypothesis" (Machine is Fair) with a 5% or 1% significance.
  (B) If your result is outside the 5%/1% significance area (e.g. 2-3 SDs), then maybe the machine is not fair?
    +/- 3 SDs for a normal distribution is 0.3% likely.

    ...however (my statistics is rusty, apologies if I'm messing this up), for getting Quads, a Bernoulli or Poisson approximation is more relevant
    http://en.wikipedia.org/wiki/Poisson_distribution
    I forget what 3 SDs means for a Poisson distribution.

Mitchell

#4

And government is likely trusting the manufacturers. Or does gaming test every machine and reject ones that are more than say six sigma out of spec? Do they ever reject a machine for any reason? Have they put procedures in place to ensure an American Coin doesn't happen again?

http://en.wikipedia.org/wiki/Six_Sigma
http://www.google.com/search?q=american+coin+slot+machine+scandal

···

--- In vpFREE@yahoogroups.com, Tom Robertson <007@...> wrote:

Besides not
liking the idea of there being a significant possibility of going
years without winning even if the machines are fair, that I despair of
empirically proving they are fair is another reason I don't like
playing with a small theoretical advantage. Ultimately, it involves
trusting government.

#5

I'm not sure a person's opinion of the game ( what percent chance machine is non-fair) is all that important. If I think a machine has a 50% chance of being non-fair vs having a 30% chance of being non fair, how does that affect my experiment?

The whole idea of my post is to show that making an accurate conclusion based on your play is a lot more work than is commonly believed.

" I had no quads in 3 hours of play" is an almost worthless statement. Once a person has a hypothesis to test, this group can look at sample size, appropriate statistic, noise factors,e tc and see if the planned experiment will actually give you the answer you need.

Let's say I think quads are being shorted on a single line machine. My hypothesis is "Quads appear less frequently than expected". I will play 5000 hands of JOB and record my results. If my results are less than 10% likely to occur, I will repeat. If they are less than 10% likely again, I will conclude this machine is non-fair. We can go to chi squared type testing or confidence interval type testing and see what is the chance that I run 2 trials and both trials show a quad frequency in the bottom 10% of possibilities.

The main point is to be very, very detail oriented on both what you are trying to prove (or disprove) and rigorous in the data collection. Otherwise, you can't really make an intelligent conclusion.

Did Frank ever publish his program?

···

--- In vpFREE@yahoogroups.com, Mitchell Tsai <tsai@...> wrote:

> If the sample is x standard deviations from expected, what
> chance of a gaffed machine does that translate to?
>

This will depend on

  (1) before any data, how likely do you believe the machine is biased e.g. Before the test, you thought there was a 1% change of a gaffed machine The test tells you afterwards that there is a 30-40% chance of a gaffed machine.

#6

johnnyzee48127 wrote:

Let's say I think quads are being shorted on a single line machine. My hypothesis is "Quads appear less frequently than expected". I will play 5000 hands of JOB and record my results. If my results are less than 10% likely to occur, I will repeat. If they are less than 10% likely again, I will conclude this machine is non-fair. We can go to chi squared type testing or confidence interval type testing and see what is the chance that I run 2 trials and both trials show a quad frequency in the bottom 10% of possibilities.

I'd still like to see more numbers. Does "conclude" mean that you
believe it has more than a 50% chance of being unfair? I don't see
how the chance of 2 trials being in the bottom 10%, combined with
those being the results, translates to any particular chance of the
machine being gaffed.

I'm not sure a person's opinion of the game ( what percent chance machine is non-fair) is all that important. If I think a machine has a 50% chance of being non-fair vs having a 30% chance of being non fair, how does that affect my experiment?

I once concluded that a machine I saw was gaffed based on no results,
but entirely on the context. I see no way of eliminating the factor
of an a priori guess about the chance of the machine being gaffed.
The experiment can't be divorced from the theory so much that it
dictates results. If one completely trusts the Gaming Commission to
eliminate all gaffed machines, it's going to take a lot more
experiments to conclude that a machine in Las Vegas is gaffed than if
one finds a machine with a theoretical payback of 120% in Kiev.

#7

Your best guess before hand effects the Bayesian inference:

http://en.wikipedia.org/wiki/Bayesian_inference

Basically, you're using a test which always has a chance of false negative and false positive. Bayesian inference is a technique to adjust your previous assumption to the results of your data knowing that there is a certain false report rate. For example, you might assume that the ER of a certain machine is 100.76% with perfect play. After taking data, you can make a Bayesian inference on what the actual ER is, which will include your play error rate.

···

--- In vpFREE@yahoogroups.com, "johnnyzee48127" <greeklandjohnny@...> wrote:

--- In vpFREE@yahoogroups.com, Mitchell Tsai <tsai@> wrote:
>
> > If the sample is x standard deviations from expected, what
> > chance of a gaffed machine does that translate to?
> >
>
> This will depend on
>
> (1) before any data, how likely do you believe the machine is biased e.g. Before the test, you thought there was a 1% change of a gaffed machine The test tells you afterwards that there is a 30-40% chance of a gaffed machine.

I'm not sure a person's opinion of the game ( what percent chance machine is non-fair) is all that important. If I think a machine has a 50% chance of being non-fair vs having a 30% chance of being non fair, how does that affect my experiment?

#8

--- In vpFREE@yahoogroups.com, "nightoftheiguana2000" Have they put procedures in place to ensure an American Coin doesn't happen again?

http://en.wikipedia.org/wiki/Six_Sigma
http://www.google.com/search?q=american+coin+slot+machine+scandal

I believe the answer is NO. Despite repeated requests to Gaming, casino managers, gurus, message boards, etc. I have never gotten an answer to my question of "How many times a year does Gaming bust open video poker machines and make sure they are legit"
The only answer I have ever gotten is effectively "they won't cheat cause they'll lose their license"

#9

Casinos keep detailed records on the win/loss of each machine which is required in most gaming jurisdictions. Depending upon the jurisdiction these records are either reviewed and/or sent to the appropriate gaming jurisdiction. Any machine that has a significant deviation from the expected win/loss is replaced.
> To: vpFREE@yahoogroups.com

From: melbedewy1226@hotmail.com
Date: Thu, 12 Apr 2012 16:27:02 +0000
Subject: [vpFREE] Re: how to tell if your machine is fair?

--- In vpFREE@yahoogroups.com, "nightoftheiguana2000" Have they put procedures in place to ensure an American Coin doesn't happen again?
>
> http://en.wikipedia.org/wiki/Six_Sigma
> http://www.google.com/search?q=american+coin+slot+machine+scandal
>
I believe the answer is NO. Despite repeated requests to Gaming, casino managers, gurus, message boards, etc. I have never gotten an answer to my question of "How many times a year does Gaming bust open video poker machines and make sure they are legit"
The only answer I have ever gotten is effectively "they won't cheat cause they'll lose their license"

------------------------------------

vpFREE Links: http://www.west-point.org/users/usma1955/20228/V/Links.htm

Yahoo! Groups Links

[Non-text portions of this message have been removed]

#10

They definitely pull any machine that pays out more than expected, but I question if any casino has ever pulled a machine that has held more than expected. In addition, video poker and other slots that involve skill have another complication that plain old luck only slots don't have, that being that the return is also a function of how the machines are played.

···

--- In vpFREE@yahoogroups.com, kelso 1600 <kelso1600@...> wrote:

Any machine that has a significant deviation from the expected win/loss is replaced.

#11

Yes. I know of one particular instance where a machine with significant play had never hit a royal and that machine was replaced. American Coin got caught because there hold was substantially higher than norm and that started the investigation.
> To: vpFREE@yahoogroups.com

From: nightoftheiguana2000@yahoo.com
Date: Thu, 12 Apr 2012 17:48:55 +0000
Subject: [vpFREE] Re: how to tell if your machine is fair?

> Any machine that has a significant deviation from the expected win/loss is replaced.

They definitely pull any machine that pays out more than expected, but I question if any casino has ever pulled a machine that has held more than expected. In addition, video poker and other slots that involve skill have another complication that plain old luck only slots don't have, that being that the return is also a function of how the machines are played.

------------------------------------

vpFREE Links: http://www.west-point.org/users/usma1955/20228/V/Links.htm

Yahoo! Groups Links

[Non-text portions of this message have been removed]

···

--- In vpFREE@yahoogroups.com, kelso 1600 <kelso1600@...> wrote:

#12

Yes. I know of one particular instance where a machine with significant play had never hit a royal and that machine was replaced. American Coin got caught because there hold was substantially higher than norm and that started the investigation.

Wrong.

"They" (Gaming Enforcement) do not "pull" machines due to much or how little they pay, and never have. Casino management does do this, and frequently.

GE investigates a gaming device based on either 1) a complaint, or 2) their random chip inspection program. The latter occurs approximately monthly at each gaming site. I got this information years ago directly from the head of the Gaming Control Lab. When asked how random these inspections were, he told me they tried not to do it during busy times.

The chip inspection program was instituted in response to the American Coin scandal. There was no such program prior to it.

TC

[Non-text portions of this message have been removed]

···

Sent from my iPad
On Apr 12, 2012, at 11:13 AM, kelso 1600 <kelso1600@hotmail.com> wrote:

#13

Two observations that bolster your point:

1.) I watched, at about 3AM one night, a casino employee open each VP/slot muliti-game, multi-denom machine along an entire bank (except the one I was playing) and manually record the data on each option on the machine. I could only see the actual numbers on the machines adjacent to me, but the VP payout % was pretty close to, but consistently below, the machine's ER for the various games, as would be expected. The slots were all in the 80-something % range.

2.) A particular machine (out of a bank of 8) that I hit for 5 W2G payouts last year in fairly limited play was changed out as of my first trip in this year. The other machines, which happen to be consecutively numbered, remained. I'm not saying that it was my 5 W2G's that caused them to swap out the machine. But if others had similar success, and that caused a significant deviation, I have no doubt that is why only one machine was changed, as opposed the a wholesale change to the whole bank.

Sniff, sniff, good-bye # 010175, my old friend.

Certainly the game is rigged. Don’t let that stop you; if you don’t bet, you can’t win. -Lazarus Long
In theory, there is no difference between theory and practice. But, in practice, there is. -Yogi Berra
There is no such thing as luck. There is only adequate or inadequate preparation to cope with a statistical universe. -Robert Heinlein

________________________________

Casinos keep detailed records on the win/loss of each machine which is required in most gaming jurisdictions. Depending upon the jurisdiction these records are either reviewed and/or sent to the appropriate gaming jurisdiction. Any machine that has a significant deviation from the expected win/loss is replaced.

[Non-text portions of this message have been removed]

#14

1.) I watched, at about 3AM one night, a casino employee open each
VP/slot muliti-game, multi-denom machine along an entire bank
(except the one I was playing) and manually record the data on each
option on the machine. I could only see the actual numbers on the
machines adjacent to me, but the VP payout % was pretty close to,
but consistently below, the machine's ER for the various games, as
would be expected. The slots were all in the 80-something % range.

It's fun to look at actual machine data. At one casino, the Aruze
slots are recorded every 2nd week - usually Monday morning.

The screens include number of lifetime payouts between $500-1,000,
$1,000-2,000, etc...,
so you can see which machines are paying out more large jackpots.

On one slot, which claims better than 95% payout at max bet, the
payouts are set to 95.04% at max bet.
At lower bets, the payouts are 86-89%...and the ones in the "high
paying slot zone" are set no differently than the ones outside.

Some of my favorite slots are set low; e.g. 92.5% for a Double Diamond
Mine $1.

Casinos keep detailed records on the win/loss of each machine which
is required in most gaming jurisdictions. Depending upon the
jurisdiction these records are either reviewed and/or sent to the
appropriate gaming jurisdiction. Any machine that has a significant
deviation from the expected win/loss is replaced.

...only if the casino management is doing their job.

Last year, the head casino director was fired at one casino.
The Hee Haw slot bank had reduced the max bet from $6 to $3, but left
all three progressives the same (so they were paying double).
I "heard" that the casino lost ~$300,000 during the 6+ months that the
machines were mis-set.

One of my friends won three $6,000 jackpots which should have started
at $3,000.

···

On Apr 12, 2012, at 12:37 PM, GURU PERF wrote:
On Apr 12, 2012, at 10:35 AM, kelso 1600 wrote:

-----
At the Palazzo and Red Rock in Vegas, I saw Video Poker machines shut
down the day after big wins.
The next day Palazzo replaced the $2-$5 Jacks 9/6 with Jacks 9/5.
Red Rock shut down a whole bank of ten 10-play machines. I forget
whether they were downgraded.

On Apr 12, 2012, at 9:07 AM, nightoftheiguana2000 wrote:

Your best guess before hand effects the Bayesian inference:

http://en.wikipedia.org/wiki/Bayesian_inference

Basically, you're using a test which always has a chance of false
negative and false positive. Bayesian inference is a technique to
adjust your previous assumption to the results of your data knowing
that there is a certain false report rate. For example, you might
assume that the ER of a certain machine is 100.76% with perfect
play. After taking data, you can make a Bayesian inference on what
the actual ER is, which will include your play error rate.

"A priori" beliefs affect the interpretation of data.

For example, with an AIDS test which has 99% accuracy, if you get a
positive result, and you believe you are not infected (e.g. chance
<1/10,000),
then the positive result probably is wrong.

If I walk into a casino where I have been told the games are rigged
(e.g. chance >50% that the games are set too low or the video-poker-
plays-like-a-slot),
I may decide with only minimal data (< 1 SD loss) that the games are
indeed rigged.

-----
Another factor is that "computer chips are NOT random". They use
random number generators which are designed to have certain behaviors
(e.g. average payout = 98.3%, certain percentages of each hand, etc...)

They can exhibit behaviors very unlikely for truly random cards (and
real cards are not random either...);
e.g. I hold 4 to a flush on a 10-play, and get 6 flushes, 5 with the
"6 of hearts".

Video poker (in my observation) is very streaky.

In statistics (for a normal distribution), I might describe this as
high "peaked-ness" (or the third moment being rather high).
  Mean = 1st moment
  Variance (Standard Deviation) = 2nd moment
  Peakedness = 3rd moment
  Skewness = 4th moment
  etc...

Designers often check for a mean & standard deviation of a pseudo-
random algorithm to operate within certain parameters.
However, they might not pay attention to the 3rd, 4th, and higher
moments.

This has caused MANY problems for computer simulations, which
calculate faulty results (based on an expectation of randomness).

Thus, it's entirely possible to design a video poker chip which gives
98.6% payout, but has a lower-than-expected number of four-of-a-kinds,
...and a higher percentage of some other hands.

Mitchell

[Non-text portions of this message have been removed]

#15

I just found this thread. It was my B-Day this week and I've been taking time off from everything.

I like your description of the testing process and will include a lot of what you said in the manual.

I'm a month or more from completing the utility.

One thing, you can't use this utility on previous results. Most will understand why, some may not. So in your example of the gal that posted about running bad on 4K's, she'd have to go back and run another test after she suspected something was amiss. She could not go home plug in her preexisting results and learn anything. Statistical test cannot be used on anything that's already happened, or else one opens the door for selective recruitment and confirmation bias.

~FK

···

--- In vpFREE@yahoogroups.com, greeklandjohnny@... wrote:

Frank started up a discussion on this topic a while ago. Originally it was going to be moved off of vpfree and then decided to remain on vpfree. I haven't seen the final product yet.

For those of you who wonder about such things, if you were to state your hypothesis and say what measurement you would accept to either prove or disprove your hypothesis, I'm sure the throngs of stats people on this board would tell you if your experiment is valid or not. ( Frank's program may do all of this already).

Here's a quickie outline of what the process should be:

1) figure out what you want to prove. This can be due to previous play, rumors, second hand stories or whatever. Note that the previous play cannot be used to confirm or refute the hypothesis.
    You need to make this very specific

2) Set up your experiment.

3) decide on your acceptance criteria

4) Post the info and let the stats guys go at it.

For example, Miss Craps recently posted about going 0 for 35 on dealt trips on 10 play ( converting to quads). This is step 1. Miss Craps believes that the machines she was playing have a quad distibution outside the normal range. So, her hypothesis is that On machine xxx, the distribution of quads is much less can be expected.

For step 2, you need to choose sample size and parameters. I will play 2000 dealt hands of triple play and record the number of times I am dealt trips and the number of times I convert them. I will also keep track of the total number of quads in the sample as an additional measure

For step 3, I can pick 90% sure, 95% sure, 99% sure or whatever number I want. I will need to adjust the number of samples based on the frequency of the event and the confidence level

Post the information and see what the group thinks.

Is this a major pain? Sure it is. But is the only way to really know if there is an issue or not.

This is a very simplified outline. For a good experiment, you would need to look at sources of error, repeats, blocking ( if you suspect certain quads are absent) and data recording errors. It is all doable but it is much easier to say " I played 4 hours and only got 1 quad, the machines must be cheating me"

[Non-text portions of this message have been removed]

#16

Tom, I think you are getting bogged down on the "never sure" aspect of probability. While it's true one can never be absolutely sure, one can be a lot more sure than mere guessing. There are currently people out there with no knowledge of probability math that have convinced themselves of whatever they wanted to believe with no systematic process behind that belief whatsoever.

Anything, no matter how flawed has to be better than that.

Additionally, I'm primarily creating this utility as an extra FUN thing people can add to their play.

It doesn't need to prove or disprove anything to be fun.

Some people really like tracking their play and now they'll have a better tool to do that.

~FK

···

--- In vpFREE@yahoogroups.com, Tom Robertson <007@...> wrote:

greeklandjohnny wrote:

>Here's a quickie outline of what the process should be:
>
>1) figure out what you want to prove. This can be due to previous play, rumors, second hand stories or whatever. Note that the previous play cannot be used to confirm or refute the hypothesis.
> You need to make this very specific
>
>2) Set up your experiment.
>
>3) decide on your acceptance criteria
>
>4) Post the info and let the stats guys go at it.
>
>For example, Miss Craps recently posted about going 0 for 35 on dealt trips on 10 play ( converting to quads). This is step 1. Miss Craps believes that the machines she was playing have a quad distibution outside the normal range. So, her hypothesis is that On machine xxx, the distribution of quads is much less can be expected.
>
>For step 2, you need to choose sample size and parameters. I will play 2000 dealt hands of triple play and record the number of times I am dealt trips and the number of times I convert them. I will also keep track of the total number of quads in the sample as an additional measure
>
>For step 3, I can pick 90% sure, 95% sure, 99% sure or whatever number I want. I will need to adjust the number of samples based on the frequency of the event and the confidence level
>
>Post the information and see what the group thinks.
>
>Is this a major pain? Sure it is. But is the only way to really know if there is an issue or not.

I don't see how this is adequate. What number are you trying to
determine? If the sample is x standard deviations from expected, what
chance of a gaffed machine does that translate to? A variable is
missing, but I'm not sure how to describe it. I've gone through the
kind of process you described and made conclusions based on it, but I
never ended up with a number which enabled me to say "I'm x% sure
what's being tested is gaffed." I still had to guess. At what point
is the probability that the machine is gaffed, say, 50%? Besides not
liking the idea of there being a significant possibility of going
years without winning even if the machines are fair, that I despair of
empirically proving they are fair is another reason I don't like
playing with a small theoretical advantage. Ultimately, it involves
trusting government.

#17

Frank, that was a very quick, seat of the pants list. I am willing to fine tune the reporting requirements and am also very interested in your project.

Email me at greeklandjohnny at aol dot com, if you want some help with the project.

···

--- In vpFREE@yahoogroups.com, "Frank" <frank@...> wrote:

I just found this thread. It was my B-Day this week and I've been taking time off from everything.

I like your description of the testing process and will include a lot of what you said in the manual.

I'm a month or more from completing the utility.

#18

I know I am taking this quote out of context (sorry FK), but your statement:

···

--- In vpFREE@yahoogroups.com, "Frank" <frank@...> wrote:

Statistical test cannot be used on anything that's already happened, or else one opens the door for selective recruitment and confirmation bias.

~FK

is absolutely not true. In fact, statistical tests can only be used on "data"-- that is on stuff that already has been observed, computed, recorded, etc. In fact, statistical tests are used in determining (in the sense of ascribing a probability to) if there is or was bias, selective recruitment, etc. of events (and associated data) that has already occured.

Take a look at: http://en.wikipedia.org/wiki/Statistical_hypothesis_testing

#19

OK. You completely misunderstood what I was saying. It will completely invalidate the testing utility I'm making if people use their currently existing data. Why? Imagine this.

You post in the newspaper that you'd like to do a study into how likely it is to be hit by lighting. Not surprisingly, the people that answer your add are those most concerned about this issue (AKA people that have been hit). After looking at all your volunteer test subjects you conclude that the chances of being hit by lighting are 1 in 1.

Problem: All the people that weren't hit by lighting, didn't volunteer.

Solution: Take the volunteers, but toss out all that has happened to them in their lives before they signed up for your study. Dismiss their preexisting data, and collect new data from this point on.

The rule of thumb with statistical tests is never to use the data that made you want to do the test. Test forward from the point in time you decide to do the test and dismiss what's gone before.

All data by definition is past data. The past I'm talking about here, that should be ignored, is what's happened before you decided to do the test.

~FK

···

--- In vpFREE@yahoogroups.com, "cdfsrule" <cdfsrule@...> wrote:

I know I am taking this quote out of context (sorry FK), but your statement:

--- In vpFREE@yahoogroups.com, "Frank" <frank@> wrote:
>
>Statistical test cannot be used on anything that's already happened, or else one opens the door for selective recruitment and confirmation bias.
>
> ~FK
>

is absolutely not true. In fact, statistical tests can only be used on "data"-- that is on stuff that already has been observed, computed, recorded, etc. In fact, statistical tests are used in determining (in the sense of ascribing a probability to) if there is or was bias, selective recruitment, etc. of events (and associated data) that has already occured.

Take a look at: http://en.wikipedia.org/wiki/Statistical_hypothesis_testing

#20

Frank,

This is where Bayesian theory (and more accurate "a priori" beliefs) allow more accurate probability calculations.

If you use past data, and assume P(all events) = equal, then you often run into pre-selection bias; e.g. I picked a weird set of data.
So Bayesian analysis will use P(my data set is unusual) = whatever you set.

Another example, say I'm considering video poker games in Las Vegas at
  1) major casino in Las Vegas - P(prior belief in gaffed machine) < 0.01
  2) non-name casino at Indian reservation where other people are reporting suspicious result - P(prior belief in gaffed machines) = 0.25

Then P(belief machine is gaffed after test | prior belief) = function of test result and P(prior belief in gaffed machines).

If you use a non-random set of data (e.g. data you have gathered before), then
  P(belief machine is gaffed) = function of test result and P(prior belief in gaffed machines) and selection-bias-in-original test)

Mitchell

A similar example of selection-bias is one about weather.
My friend tells me that last week it rained 6 out of 7 days, and they ask how unusual that is...

Most people will just calculate how unlikely it is to have rain 6 of 7 days.
A better calculation will take into account that my friend is only telling me this because it is "somehow weird" (e.g. no royals in 120,000 hands)
and factor in the "selection-bias".

···

On Apr 16, 2012, at 3:09 PM, Frank wrote:

OK. You completely misunderstood what I was saying. It will completely invalidate the testing utility I'm making if people use their currently existing data. Why? Imagine this.

You post in the newspaper that you'd like to do a study into how likely it is to be hit by lighting. Not surprisingly, the people that answer your add are those most concerned about this issue (AKA people that have been hit). After looking at all your volunteer test subjects you conclude that the chances of being hit by lighting are 1 in 1.