sas statistics by example
LECTURE 1:
1.1. Data science process and components
learn history of statistics, science of statistics, application of statistics data analytics and their connections with data science.
1.2. Data Collection and Preparation
Types of Studies
dependent variable Vs. independent variable
Populations and Parameters
Samples and Statistics
Data collection
Types of Bias
1.3. Descriptive Statistics
Descriptive statistics vs. inferential statistics
Break
Parametric Vs. Nonparametric test
Introduction to EDA
LECTURE 1:
History of Data Science
In 2001, William S. Cleveland introduced data science as an independent discipline, extending the field of statistics to incorporate "advances in computing with data" in his article "Data Science: An Action Plan for Expanding the Technical Areas of the Field of Statistics," which was published in Volume 69, No. 1, of the April 2001 edition of the International Statistical Review / Revue Internationale de Statistique.
In April 2002, the International Council for Science: Committee on Data for Science and Technology (CODATA) started the Data Science Journal,
a publication focused on issues such as the description of data systems, their publication on the internet, applications and legal issues.
In the 2012 Harvard Business Review article "Data Scientist: The Sexiest Job of the 21st Century” and asserts that a data scientist is "a new breed", and that a "shortage of data scientists is becoming a serious constraint in some sectors“.
Statistics is the science of learning from data and is a crucial components of data science. Statistics research focuses on data collection and modelling, and there is little work on developing good questions, thinking about the shape of data, communicating results or building data products.
Data science is also known as data-driven science, is an interdisciplinary field about scientific process and system to extract knowledge or insights from data in various forms, either structured or unstructured, which is a continuation of some of the data analysis fields, such as statistics, machine learning, predictive analytics, etc
Why Study Statistics?
To evaluate printed numerical facts.
To interpret the results of sampling or to perform statistical analysis in your work
To make inferences about the population using information collected from the sample
To test some hypotheses about the population based on sample dataset
What Do Statisticians Do?
Gather data
Summarize data
Analyze data
Draw conclusions and report the results of their analysis
Observational
A study where a researcher records or observes the observations or measurements without manipulating any variables. These studies show that there may be a relationship but not necessarily a cause-and-effect relationship
Experimental
A study that involves some random assignment of a treatment; researchers can draw cause and effect (or causal) conclusions. An experimental study may also be called a scientific study or an experiment.
Example: We want to decide whether Advil or Tylenol is more effective in reducing fever.
Method 1
Ask the subjects which one they use and ask them to rate the effectiveness. Is this an observational study or experimental study?
This is an observational study since we just observe the data and have no control on which subject to use what type of treatment
Observational
A study where a researcher records or observes the observations or measurements without manipulating any variables. These studies show that there may be a relationship but not necessarily a cause-and-effect relationship
Experimental
A study that involves some random assignment of a treatment; researchers can draw cause and effect (or causal) conclusions. An experimental study may also be called a scientific study or an experiment.
Example: We want to decide whether Advil or Tylenol is more effective in reducing fever.
Method 1
Ask the subjects which one they use and ask them to rate the effectiveness. Is this an observational study or experimental study?
This is an observational study since we just observe the data and have no control on which subject to use what type of treatment
Method 2
Randomly assign half of the subjects to take Tylenol and the other half to take Advil. Ask the subjects to rate the effectiveness. Is this an observational study or experimental study?
This is an experimental study since we can decide which subject to use what type of treatment. Thus, the self selection bias will be eliminated.
Causation
It is very important to distinguish between observational and experimental studies since one has to be very skeptical about drawing cause and effect conclusions using observational studies. The use of random assignment of treatments (i.e. what distinguishes an experimental study from an observational study) allows one to employ cause and effect conclusions.
Ethics is an important aspect of experimental design to keep in mind. For example, the original relationship between smoking and lung cancer was based on an observational study and not an assignment of smoking behavior.
A variable is any characteristic, number, or quantity that can be measured, counted, or observed for record.
There may be many variables in a study. The variables may play different roles in the study. Variables can be classified as either explanatory or response variables
Response Variable
Variable that about which the researcher is posing the question. May also be called the outcome or the dependent variable or target variable.
Explanatory Variable
Variable that serve to explain changes in the response. They may also be called the predictor or feature or independent variables.
Note! A variable can serve as an explanatory variable in one study but response in another.
It depends on Business Problem, for example assume you have a dataset of customers of one bank that got loans,
If you want to predict loan amount loan amount is target( response /dependent variable/output/outcome/label) and other variables are independent variables (features/predictors)
If you want to predict loan status( If each customer paid his/her loan) loan status is target( dependent variable/response/outcome/output/lable) and other variables are independent variables (features/predictors/input variables)
If you want to predict purpose of loan purpose of loan is target( dependent variable / response/ outcome/ output/ lable) and other variables are independent variables (features/predictors/input variables)
The terms "response", "dependent variable", "output", "outcome", and "label" are often used interchangeably in the context of data analysis and machine learning, depending on the specific domain or context. They all refer to the variable you are trying to predict or explain.
The terms "independent variables", "features", and "predictors" are often used interchangeably in the context of data analysis and machine learning. They all refer to the variables used to predict or explain the outcome or target variable
The independent variable is the factor that you purposely change or control in order to see what effect it has.
The variable that responds to the change in the independent variable is called the dependent variable. It depends on the independent variable.
A scientist is testing the effect of light and dark on the behavior of moths by turning a light on and off. The independent variable is the amount of light and the moth's reaction is the dependent variable.
In a study to determine the effect of temperature on plant pigmentation, the independent variable (cause) is the temperature, while the amount of pigment or color is the dependent variable (the effect).
Populations and Parameters
Population
any large collection of objects or individuals, such as Americans, students, or trees about which information is desired
Parameter
any summary number, like an average or percentage, that describes the entire population
The population mean μ (the Greek letter "mu") and the population proportion p are two different population parameters. For example:
We might be interested in learning about the average weight of all middle-aged female Americans. The population consists of all middle-aged female Americans, and the parameter is μ
(population mean )
Or we might be interested in learning about the proportion of likely American voters approving of the president's job performance. The population comprises all likely American voters, and the parameter is p (population proportion).
The problem is that almost all of the time, we don't — or can't — know the real value of a population parameter. The best we can do is estimate the parameter! This is where samples and statistics come in to play
Samples and Statistics
Sample
is a representative group drawn from the population
Statistic
any summary number, like an average or percentage, that describes the sample.
in other words a descriptive measure of a sample is called a statistics
The sample mean, 𝑥 ̅, and the sample proportion, 𝑃 ̂, are two different sample statistics. For example:
We might use 𝑥 ̅, the average weight of a random sample of 100 middle-aged female Americans, to estimate μ, the average weight of all middle-aged female Americans.
Or we might use 𝑃 ̂, the proportion in a random sample of 1000 likely American voters who approve of the president's job performance, to estimate p, the proportion of all likely American voters who approve of the president's job performance.
Assume we want to find the average height of all grade 12 students in Canada:
Population= All grade 12 students in Canada
Parameter=𝜇= the average height of all grade 12 students in Canada
Since it is time consuming and needs a lot investment, we decide to randomly select 200 grade 12 students in Canada, and we measure height of them then calculate average height of them (descriptive statistics)
Sample= 200 grade 12 students in Canada
Statistic=𝑥 ̅ =the average height of 200 grade 12 students in Canada that we selected
Then based on statistic we try to predict parameter(inferential statistics)
Because samples are manageable in size, we can determine the actual value of any statistic. We use the known value of the sample statistic to learn about the unknown value of the population parameter.
Techniques of describing data in ways to capture the essence of the information in the data are called descriptive statistics. For example, the sample mean is a descriptive statistic. To draw conclusions from data about the population is called inferential statistics. Here is a picture that demonstrates inferential statistics.
Descriptive statistics describe what is going on
in a data set by summarizing existing data and
prediction based on them to a larger population
called "Inferential Statistics".
How can we get data? How do we select observations or measurements for a study?
There are two types of methods for collecting data, non-probability methods and probability methods.
Non-probability Methods
These might include:
Convenience sampling (haphazard): Collecting data from subjects who are conveniently obtained.
Example: surveying students as they pass by in the university's student union building
Gathering volunteers: Collecting data from subjects who volunteer to provide data
Example: using an advertisement in a magazine or on a website inviting people to complete a form or participate in a study.
Probability Methods
Simple random sample: making selections from a population where each subject in the population has an equal chance of being selected.
Stratified random sample: where you have first identified the population of interest, you then divide this population into strata or groups based on some characteristic (e.g. sex, geographic region), then perform simple random sample from each strata.
Cluster sample: where a random cluster of subjects is taken from the population of interest. For instance, if we were to estimate the average salary for faculty members at Penn State - University Park Campus, we could take a simple random sample of departments and find the salary of each faculty member within the sampled department. This would be our cluster sample.
Random sampling: When every unit of a population has a known chance of being included in a sample, such a sample is called a probability sample. Let us discuss each type of sampling under this category:
Simple random sampling: In this method, sample units are selected randomly. This is similar to drawing names from a hat.
Example: Suppose you want to find the average internet usage per person in India. You can simply write the names of all the Indians on small pieces of paper and put them in a hat. You can then draw, say, 100 names and calculate the average for these 100 Indians.
This sampling method is applicable only when your population is finite, such that you can list all its units, and you understand it completely. This process is thus quite cumbersome when working with large populations.
Stratified random sampling: Here, people are divided into subgroups and then selected randomly. Nevertheless, this is done in such a manner that the proportions of these subgroups in the final sample equals their proportions in the population.
Example: Again, suppose you want to find the average internet usage per person in India. Note that 70% of Indians live in rural areas and 30% live in urban areas. So, you would put the names of all the rural Indians in, say, hat A and the names of all the urban Indians in another hat, B. Then you would draw 70 names from hat A and 30 names from hat B. Now, you would again have a sample of 100 Indians, although this time, your sample would be more representative of the population as the proportions of rural and urban Indians would be the same as those in the population.
Cluster sampling: In this sampling method, a population is divided into clusters and then samples are drawn from some of these clusters using simple random sampling.
Difference between cluster sampling and stratified sampling
For a stratified random sample, a population is divided into strata, or subpopulations, before sampling. However, in cluster sampling, you will only study selected clusters. For example: Let us consider the problem of finding the average internet usage per person in India. It would be difficult to obtain a list of all Indians and collect data from a random sample across the country. However, a list of cities may be available, and you can collect data from a subset of these cities.
Systematic sampling: In this sampling method, sample units from a larger population are selected in accordance with a random starting point but with a fixed, periodic interval.
An example of this is when you select every third person on a list of a population, as shown in this image.
Nonrandom samples: When we cannot calculate the odds of any population unit being included in a sample, then such a sample is called a nonprobability or nonrandom sample. Let us discuss the two types of nonprobability sampling:
Convenience sampling: Convenience sampling, as the name suggests, is sampling based on the researcher’s convenience.
Example: You want to find the average internet usage per person in India. Only this time, you simply ask 100 people around you about their internet usage.
Clearly, this sampling method has potential to be extremely biased. The only good part here, probably, is that it is a relatively convenient sampling method.
Judgment sampling: This is a purely judgment-based sampling approach. Sample units are chosen based solely on the researcher’s knowledge and judgment.
Example: Suppose you wish to study the different faiths in India. It is advisable to use judgmental sampling to select people of different faiths as religion is considered to be a sensitive topic
i.e. In simple random sample doesn't matter each selected member belongs to which group
In Stratified and cluster sample first we divide all population to some groups based on some characteristic and then
If we randomly select some members from all groups, this called stratified random sample
If we randomly select one group and assume all members in that group , this called cluster sample
Airline Company Survey of Passengers
Let's say that you are the owner of a large airline company and you live in Los Angeles. You want to survey your L.A. passengers on what they like and dislike about traveling on your airline. For each of the methods, determine if a non-probability method or a probability method is used. Then determine the type of sampling.
a)Since you live in L.A. you go to the airport and just interview passengers as they approach your ticket counter.
b) You have your ticket counter personnel distribute a questionnaire to each passenger requesting they complete the survey and return it at end of the flight.
c) You randomly select a set of passengers flying on your airline and question those that you have selected.
d) You group your passengers by the class they fly (first, business, economy), and then take a random sample from each of these groups.
e) You group your passengers by the class they fly (first, business, economy) and randomly select such classes from various flights and survey each passenger in that class and flight selected.
Answers:
Non-probability method; convenience sampling.
Non-probability methods; Volunteer sampling
Probability method; Simple random sampling
Probability method: Stratified sampling
Probability method; Cluster sampling
There are advantages and disadvantages to both types of methods. Non-probability methods are often easier and cheaper to facilitate. When non-probability methods are used it is often the case that the sample is not representative of the population. If it is not representative, you can make generalizations only about the sample, not the population. The primary benefit of using probability sampling methods is the ability to make inference. We can assume that by using random sampling we attain a representative sample of the population The results can be “extended” or “generalized” to the population from which the sample came
A lot of thought has been put into the science of the design of survey questions.
Key design principles:
Keep the questionnaire as short as possible
As short, simple and clearly worded questions.
Start with demographic questions to help respondents get started comfortably
Use dichotomous (yes | no) and multiple-choice questions
Use open-ended questions cautiously
Avoid using leading – question
Pretest a questionnaire on a small number of people
Think about the way you intend to use the collected data when preparing the questionnaire.
In statistics, bias can affect the collection, analysis, interpretation, and publication of data. Here are some common types of bias in statistics:
Selection Bias: Occurs when the sample is not representative of the population being studied. For instance, you are interested in the attitude of female students regarding campus safety but when sampling you also include males. In this case, your population of interest was female students however your sample included subjects not in that population (i.e. males)
Sampling Bias: Results from using non-random sampling methods.
Volunteer Bias: When the sample consists of people who volunteer to participate, which might be different than the general population.
Non-response Bias: When those who participate in a study differ from those who did not respond.
Response Bias: when study participants either do not respond truthfully or give answers they feel the researcher wants to hear. For example, when students are asked if they ever cheated on an exam even those who have would respond with "no“
Measurement or Observation Bias: Systematic errors in measuring variables.
Instrument Bias: When a measuring instrument isn't calibrated correctly.
Observer or Examiner Bias: When the person making the observation influences the results, either consciously or subconsciously.
Additional Reading:
Types of Bias (cont.)
5. Information or Reporting Bias: Occurs when participants do not provide accurate information.
Recall Bias: When individuals remember past events inaccurately.
Response Bias: The tendency for participants to respond in a certain way regardless of their true status or opinion.
6. Confounding Bias: When an external factor (the confounder) affects both the independent and dependent variables, potentially producing a spurious association.
7. Publication Bias: The inclination to publish results that are positive or show a significant finding and neglect non-significant or negative results.
8. Lead-time Bias: In medical screening, when early detection is confused with increased survival time, even though early detection might not impact the course of the disease.
9. Length-time Bias: In medical screening, where slow-progressing cases are more likely to be detected than fast-progressing cases.
10. Overfitting: In modeling, when a statistical model describes random error or noise instead of the underlying relationship, leading to over-complicated models.
11. Bias due to omitted variables: When a model does not account for a variable that influences the outcome, leading to biased estimates.
Understanding and identifying potential sources of bias is crucial in statistics to ensure the validity and reliability of findings.
1.3. Descriptive Statistics
Major tasks in Statistical analysis:
Collecting data and Analyzing data
Data collection: Recording, sampling and also including the planning of data collection in terms of the design of surveys and experiments.
Data analysis: Two main statistical methods -- Descriptive statistics which summarizes data from a sample using indexes (e.g., mean or standard deviation), and Inferential statistics which draws conclusions(prediction) from data that are subject to random variation.
Assume we want to find average height of all grade 12 students in Canada
Population= all grade 12 students in Canada
Parameter= average height of all grade 12 students in Canada
We randomly select 200 grade 12 students in Canada and we measure height of all of them
Sample= 200 grade 12 students that we selected
Sample statistic=average height these 200 grade 12 students in Canada
Calculation of average height of these 200 students is descriptive statistics
Prediction of average height of all grade 12 students in Canada based on average of these 200 selected students is called inferential statistics
(prediction of population parameter based on sample statistic is called inferential statistics)
Descriptive Statistics:
Descriptive statistics are statistics that quantitatively describe or summarize features of a collection of data(sample dataset).
Descriptive statistics are distinguished from inferential statistics , in that descriptive statistics aim to summarize a sample, rather than use the data to learn about the population that the sample of data is thought to represent.
This generally means that descriptive statistics, unlike inferential statistics, are not developed on the basis of probability theory. Even when a data analysis draws its main conclusions using inferential statistics, descriptive statistics are generally also presented.
Statistical inference is the process of making as estimate, prediction, or decision about a population based on sample data. Because population are almost always very large, investigating each member of the population would be impractical and expensive.
It is far easier and cheaper to take a sample from the population of interest and obtain conclusions or make estimates about the population on the basis of information provided by the sample.
However, such conclusions and estimates are not always going to be correct for various reason. The results are very much depend on your samples.
The critical step in the statistical process is the choosing of the sample. A badly chosen sample will result in poor predictions or estimates.
To measure good or bad, we build into the statistical inference as the measure of reliability such as confidence level and significance level in the further study.
Hint: Each test has some assumptions that sample data set must satisfied them otherwise the prediction of that test is not reliable.
The following measures are commonly used to describe the observations:
a measure of location, or central tendency, such as the arithmetic mean, median, mode or interquartile
a measure of statistical dispersion, such as standard deviation, variance, range, interquartile range
a measure of the shape of the distribution, such as skewness or kurtosis
if more than one variable is measured, a measure of statistical dependence, such as the Pearson correlation coefficient, Spearman’s correlation coefficient, and so on
• When a population is believed to be normally distributed or close to normally distributed, parametric statistics is the best to be used. If not, it is best that a nonparametric method be used.
• Parametric statistics depend on normal distribution, but Non-parametric statistics does not depend on normal distribution.
• Parametric statistics make more assumptions than Non-Parametric statistics.
• Parametric statistics use simpler formulae in comparison to Non-Parametric statistics.
• Most of the commonly known elementary statistic methods belong to parametric statistics. Nonparametric statistics is sparingly used and applied for special cases.
I will talk about Parametric and Nonparametric in part 3 of this course.
Exploratory Data Analysis (EDA): A preliminary approach to analyzing data that focuses on visualizing, summarizing, and understanding its main characteristics and underlying patterns. The goal of EDA is to uncover relationships, anomalies, and insights within the data to inform further analysis and decision-making.
Key components and techniques of EDA include:
Visualization: Using various graphs and charts like histograms, scatter plots, box plots, and more to visually explore the data. Visualization aids in identifying patterns, outliers, and relationships between variables.
Summary Statistics: Computing statistics like mean, median, mode, standard deviation, and quantiles to get a sense of the distribution and spread of the data.
Handling Missing Values: Identifying and treating missing data. Decisions might involve filling them with a certain value (e.g., mean, median) or removing them.
Identifying Outliers: Detecting and deciding how to handle outliers, which are extreme values that deviate from other observations.
Checking for Duplicates: Ensuring that the data doesn't have unwanted duplicate rows or values.
Correlation Analysis: Checking how different variables correlate with each other. This can assist in feature selection for machine learning models and understanding relationships.
Dimensionality Reduction: If the dataset is vast with many variables, techniques like PCA (Principal Component Analysis) can be used to reduce the number of variables without losing much information.
Distribution Analysis: Examining the distribution of the data, like checking for normality or other distribution forms, to understand its spread and shape.
Through these techniques, EDA provides a comprehensive understanding of the dataset, helping analysts and data scientists make informed decisions about subsequent steps in the data analysis process.
What Is Exploratory Data Analysis in simple word?
American mathematician John Tukey promoted the use of EDA in his book, Exploratory Data Analysis (Pearson). Tukey emphasized that analysts need first to explore the data for potential research questions before jumping into confirming the answers with hypothesis testing and inferential statistics.
EDA is often likened to “interviewing” the data; it’s a time for the analyst to get to know it and learn about what interesting things it has to say. As part of our interview, we’ll want to do the following:
• Classify our variables as continuous, categorical, and so forth
• Summarize our variables using descriptive statistics
• Visualize our variables using charts
The first step in any data science project: exploring the data.
Exploratory data analysis, or EDA, is a comparatively new area of statistics.
EDA focuses more narrowly on checking assumptions required for model fitting and hypothesis testing, and handling missing values, detecting outliers and making transformations of variables as needed.
Stuff done during EDA:
Trends
Distributions
Mean
Median
Outliers
Variance
Correlations
Hypothesis testing
Visual explorations
EDA is a process to try to understand existing data by visualization, Summarization and testing hypothesises
Descriptive Statistics Cont.
Key terms for Rectangular Data
Structured Data Vs. Unstructured Data
Types of variables
Break
Univariate Analysis for Qualitative Variables
Classroom Exercise
Qualitative (Categorical)
Data that serves the function of a name only. Categorical values may be:
Ordinal – where the names imply levels with hierarchy or order of preference(possible to apply a rank order), i.e. a clear ordering to its categories, e.g. level of education, day of the week
Nominal – where no hierarchy is implied, e.g. purpose of loan, weather, eye color
Binary – where there are two choices, e.g. Male and Female, Yes and No, 0 and 1
For example, for coding purposes, you may assign Male as 0, Female as 1. The numbers 0 and 1 stand only for the two categories and there is no order between them.
Note: A binary attribute is a special case of a nominal attribute.
The best way to summarize categorical data is to use frequencies and percentages like in the table.
How can one graph qualitative variables?
Two common choices are the pie chart and the bar chart.
Pie charts may not be suitable for too many categories. Thus, if there are too many categories, you can either combine some categories or use a bar chart to represent the data. What is mean by "too many"? There is no clear cutoff, more of just a judgment on the appearance.
Bar Graph vs Pie Graph - When to Use What?
Generally, you should use a bar graph when you want to compare values between different categories or groups and a pie graph when you want to show the relative sizes of the different parts of a whole, They are the most useful when you are analyzing a small number of categories or parts that add up to a whole(1) and mutually exclusive. However, pie graph should not be preferred more than 6–7 categories, as viewers may not be able to differentiate among them immediately. For example: if you have conversion Rate for 3 different approach Approach Conversion-Rate A 5.6% B 4.2% C 5.2% We can't use pie chart since they are not part of the whole and they are not mutually exclusive
Mutually exclusive" is a term used to describe a situation in which two or more events or categories cannot occur simultaneously. In other words, when events or categories are mutually exclusive, the occurrence of one event or belonging to one category precludes the occurrence or membership of the others in the same context.
For example, if you have a set of categories A, B, and C, and they are mutually exclusive, it means that if an item or event falls into category A, it cannot simultaneously fall into categories B or C. These categories do not overlap in the context you are considering.
In statistics and probability theory, the concept of mutually exclusive events is important for calculating probabilities, especially when dealing with events that cannot happen at the same time. For example, when rolling a standard six-sided die, the events of getting a 1, 2, 3, 4, 5, or 6 are mutually exclusive because only one of these outcomes can occur in a single roll.
In a pie chart, the entire circle represents 100% of a whole, and each slice of the pie represents a portion or category of that whole. The sizes of the slices are relative to their proportions of the whole. For example, in a pie chart showing the distribution of a budget, each category represents a portion of the total budget, and the whole budget is 100%.
In the case of conversion rates for different approaches, there is no "whole" to represent as a sum of parts. Each approach's conversion rate is an independent percentage, and there is no combined or total percentage that makes up a whole.
Not Mutually Exclusive: Mutually exclusive events or categories cannot occur simultaneously. In a pie chart, the assumption is that each category is mutually exclusive; an item can belong to only one category. For example, in a pie chart showing the sources of website traffic (e.g., organic search, direct, referral), each visit is attributed to one and only one source.
In the context of conversion rates for different approaches, there is no exclusivity requirement. An individual or event could potentially be associated with multiple approaches simultaneously. For example, a website visitor might be exposed to both Approach A and Approach B, and their conversion may be influenced by both. This lack of mutual exclusivity makes a pie chart an inappropriate choice for visualizing the data.
Measures of central tendency, also known as measures of location are typically among the first statistics computed for the continuous variables in a new data set.
The main purpose of computing measures of central tendency is to give you an idea of what a typical or common value for a given variable is. The three most common measures of central tendency are :
the mean,
the median, and
the mode.
While measures of central tendency are important, they do not tell the whole story. For example, suppose the mean score on a statistics exam is 80%. From this information, can we determine a range in which most people scored? The answer is no. There are two other types of measures, measures of position and variability, that help paint a more concise picture of what is going on in the data. In this section, we will consider the measures of position and discuss measures of variability in the next one.
Measures of position give a range where a certain percentage of the data fall. The measures we consider here are percentiles and quartiles.
The five-number summary of a data set consists of the minimum, maximum, and quartiles written in increasing order: Min, Q1, Q2, Q3, Max.
There are several methods for finding percentile, and the result of them are a little difference . The previous method that I used is called M&M method
Example: Consider the following data set with 13 observations
(1, 2, 3, 5, 7, 8, 11, 12, 13, 15, 18, 18, 20):
First, we have to sort the data , but it is already sorted
we want to find the 25th percentile, so P= 25.
We have 13 observations, so n = 13.
R=𝑷/𝟏𝟎𝟎∗𝒏 = 0.25 *13 = 3.25, which is not an integer, so we will use the second option (#3 in the preceding list).( If it was 3 we had to get average between 3rd and 4th obs , but since it not integer we are calculating celling of that for finding location of 25th percentile)
Since celling of 3.25 is 4 , the 25th percentile is th 4th observation, which has the value 5.
We can follow the same steps to find the 75th percentile:
7. 𝑷/𝟏𝟎𝟎∗𝒏 = 0.75*13= 9.75, not an integer.
8. celling of 9.75 is 10
Therefore, the 75th percentile is the 10th observation, which has the value 15.
Therefore, the interquartile range is (15 − 5) or 10.
The resistance of the interquartile range to outliers should be clear. This data set has a range of 19 (20 − 1) and an interquartile range of 10; however, if the last value was 200 instead of 20,(1, 2, 3, 5, 7, 8, 11, 12, 13, 15, 18, 18, 200) the range would be 199 (200 − 1), but the interquartile range would still be 10, and that number would better represent most of the values in the data set.
e.g.
A sample with 10 obs.: 21,31,42,5,5,9,10,220,70,67 Find 25th , 30th, 50th and 75th percentile
after sorting the data: 5,5,9,10,21,31,42,67,70,220
25% (𝑄_1)location:R=(25/100)*(10+1)=0.25*11=2.75 (2.75 is the location of 𝑄_1after sorting data from smallest to largest so it is located between 2nd and 3rd obs more close to 3rd one)
That means
𝑄_1 =value of second observation(after sorting ascending) plus 0.75*(value of third obs. – value of second obs. )
𝑄_1=5+0.75(9-5)=5+0.75*4=8
Find 30th percentile?
R=(30/100)*(10+1)=3.3 That means 30th percentile located between 3rd obs and 4th obs so value of it equals
value of third obs.+0.3(value of fourth obs. – value of third obs.)
Value of 30% percentile=9+0.3(10-9)=9+0.3=9.3
Median=𝑄_2=50th percentile
Location of 𝑄_2= (50/100)*(10+1)=5.5
𝑄_2=21+0.5(31-21) or (21+31)/2=26
𝑄_3=75% 𝑝𝑒𝑟𝑐𝑒𝑛𝑡𝑖𝑙𝑒=?
Location of 𝑄_3=R=0.75*(10+1)=8.25
𝑄_3=67+0.25(70-67)=67+0.75=67.75
So for this data set the five number summary is Min=5, 𝑄_1 =8, 𝑄_2=26, 𝑄_3 =67.75, Max=220
As you saw there are three quartiles in a set of numbers:
The lower quartile Q1.
The middle quartile (called the median) Q2.
The upper quartile Q3.
The quartiles divide the set, when they are in numerical order, into four equal groups.
There are different methods for finding the quartiles, which give different values for them.
So Different software packages will produce different results. In SAS, you can use the PCTLDEF= option in PROC UNIVARIATE or the QNTLDEF= option in other procedures to control the method used to estimate quantiles.
Note: Within each method, the method is slightly different dependent on whether there are an odd or even number of numbers in the set
R statistical software (excellent, and free from www.r-project.org) gives you a choice of nine different algorithms: here are some sample printouts from my example above. Some 'types' insist on giving only exact sample values as quartiles, others can pick values between sample values.
If you have access to R, and are curious to pursue this further, you could look at the R help screen at ? quantile under type. But as a beginning statistician, you might do better just to understand that these differences do exist and decide not to worry much about them. Follow the rule in your textbook and expect your calculator or software to give different answers sometimes. Fortunately, in practice, quantiles are most often used for very large datasets. And for large datasets, differences among algorithms become less noticeable and are often unimportant.
A helpful summary of the data is called the five number summary. The five number summary consists of five values:
The minimum
The lower quartile, Q1
The median (also known as Q2)
The upper quartile, Q3
The maximum
Example:
SAS :98, 98, 97,95,94
Python:98, 96,95,94,93
You are new student ,in which course if you got 96 you did well?
answer: you have to find in each course your rank is better. So you have to order data
descending
SAS:98,98,97,95,94
Python:98,96,95,94,93
In Python your rank with 96 is better
After ordering data:
Python:98, 96,96,95,94,93 you are the 2nd student in Python
SAS: 98,98,97,96,95,94 you are the 4th student in SAS
Measures of Variability
Dispersion refers to how variable or spread out data values are.
For this reason, measures of dispersions are sometimes called measures of variability or measures of spread.
There are many ways to describe variability or spread including:
Range
Interquartile range (IQR)
Variance and Standard Deviation
Range
The range is the difference in the maximum and minimum values of a data set. The maximum is the largest value in the dataset and the minimum is the smallest value. The range is easy to calculate but it is very much affected by extreme values.
Range=maximum−minimum
e.g. 5,5,5,5,5,5,5 mean=5, median=5, Min=5, Max=5Range=5-5=0
0,1,2,5,8,9,10mean =5, median=5, Min=0, Max=10Range=10-0=10
Interquartile Range (IQR)
The interquartile range is the difference between upper and lower quartiles and denoted as IQR.
IQR=Q3−Q1=upper quartile−lower quartile=75th percentile−25th percentile
Note! The IQR is not affected by extreme values(outliers).
The most common measures of dispersion for continuous data are the variance and standard deviation.
Both describe how much the individual values in a data set vary from the mean or average value.
The variance and standard deviation are calculated slightly differently depending on whether a population or a sample is being studied, but basically the variance is the average of the squared deviations from the mean, and the standard deviation is the square root of the variance.
Unbiased and Biased Estimators
We want our estimator to match our parameter, in the long run. In more precise language we want the expected value of our statistic to equal the parameter. If this is the case, then we say that our statistic is an unbiased estimator of the parameter.
If an estimator is not an unbiased estimator, then it is a biased estimator.
In calculating the variance, we have changed from our original units to squared units, which might not be convenient to interpret.
For instance, if we were measuring weight in pounds, we would probably want measures of central tendency and dispersion expressed in the same units rather than having the mean expressed in pounds and variance in squared pounds.
To get back to the original units, we take the square root of the variance; this is called the standard deviation and is signified by σ for a population and s for a sample.
The CV cannot be calculated if the mean of the data is 0 (because you cannot divide by 0) and is most useful when the variable in question has only positive values.
If a variable has both positive and negative values, the mean can be close to zero although the data actually has quite a broad range, and this can produce a misleading CV value because the denominator will be a small number, potentially producing a large CV value even if the standard deviation is fairly moderate.
Solution for day 6 homework
Random Variables
3. Probability Distribution
3.1. Discrete probability distribution
Expected Value (or mean) of a Discrete Random Variable
Variance of a Discrete Random Variable
Binary (Bernoulli) distribution
Binomial distribution
Distributions and Statistical Tests
Random variables:
A random variable, is a variable whose value depends on possible outcomes.
Let us throw the die 10 times, the outcome of results may be:
(2 ,3 ,4 ,5 ,1 ,3, 6 , 4 ,3 ,4 )
Let define the results as variable X
Variable X has data value as above- Variable X is a random sample of size 10 from population (1-6)
X={1,2,3,…,6}
Let us throw the coin 10 times, the results may be:
H H T H T T H T H H
Let define the results as variable Y. Variable Y has data value as above- Variable Y is a random variable
e.g.1)Let us throw the coin 2 times, the sample space is:
S={HH,HT,TH,TT}
Let define variable X number of heads:
X is a random variable : X={0,1,2}
X(HH)=2
X(HT)=1
X(TH)=1
X(TT)=0
X: {HH,HT,TH,TT} {0,1,2}
e.g.2)Let us throw the coin 3 times, the sample space is:
S={HHH,HHT,…,TTT}
Let define variable X number of heads:
X is a random variable : X={0,1,2,3}
S={HHH,HHT,…,TTT} X={0,1,2,3}
As you can see by defining random variable, we converted Sample space to a subset of real numbers(ℝ)
e.g.3)Let us throw the die 2 times,
S={(1,1),(1,2),…,(6,6)}
Let define variable Y sum of the results
Y={2,3,4,5,..,12}
S={(1,1),(1,2),…,(6,6)} Y={2,3,4,5,..,12}
e.g.3)Let assume families with 3 children
S={BBB,BBG,…,GGG}
Let define variable X number of boys they have
S={BBB,BBG,…,GGG} X={0,1,2,3}
variable, we converted Sample space to a subset of real numbers(ℝ)
e.g.1)Let us throw the coin 2 times, the sample space is:
S={HH,HT,TH,TT}
Let define variable X number of heads:
X is a random variable : X={0,1,2}
X(HH)=2
X(HT)=1
X(TH)=1
X(TT)=0
X: {HH,HT,TH,TT} {0,1,2}
As you can see here by defining random variable, we converted Sample space to a subset of real numbers(ℝ)
e.g.2)Let us throw the coin 3 times, the sample space is:
S={HHH,HHT,…,TTT}
Let define variable X number of heads:
X is a random variable : X={0,1,2,3}
S={HHH,HHT,…,TTT} X={0,1,2,3}
As you can see by defining random variable, we converted Sample space to a subset of real numbers(ℝ)
e.g.3)Let us throw the die 2 times,
S={(1,1),(1,2),…,(6,6)}
Let define variable Y sum of the results
Y={2,3,4,5,..,12}
S={(1,1),(1,2),…,(6,6)} Y={2,3,4,5,..,12}
e.g.3)Let assume families with 3 children
S={BBB,BBG,…,GGG}
Let define variable X number of boys they have
S={BBB,BBG,…,GGG} X={0,1,2,3}
e.g.5)Let us throw a coin 4 times, the sample space is:
S={HHHH,HHHT,…,TTTT}n(S)=16
Let define variable X number of heads:
X is a random variable : X={0,1,2,3,4}
e.g.6)Let assume families with 2 children
S={(boy, boy),(boy, girl), (girl, boy), (girl, girl)} n(S)=4
Let define variable X number of boys they have
S={BB,BG,GB,GG} X={0,1,2}
Note: In previous example
Probability of {(boy, boy)} = Probability of {(boy, girl)}= Probability of {(girl, boy)}= Probability of {(girl, girl)}=1/4=25%
But probability of X=0 : P(X=0)=P( no boy)=P({GG})=1/4=25%
probability of X=1 : P(X=1)=P( one boy)=P({BG,GB})=2/4=50%
probability of X=2 : P(X=2)=P(two boys)=P({BB})=1/4=25%
Types of Random Variables:
There are two types of random variables, qualitative (or categorical) and quantitative (or numerical).
Qualitative Random Variables
The possible values vary in kind but not in numerical degree. They are also called categorical variables.
Quantitative Random Variables
There are two types of quantitative random variables.
Discrete Random Variable: When the random variable can assume only a countable, sometimes infinite, number of values.
Continuous Random Variable: When the random variable can assume an uncountable number of values in a line interval.
As an example, if you throw a coin 4 times, and define variable X number of heads:
X is a Discrete Random Variable : X={0,1,2,3,4}
But if you select a number between 0 and 1 this is Continuous Random Variable
A random variable is something that is used to quantify outcomes. The possible values of a random variable are the numerical outcomes of a particular experiment. It is commonly denoted with X.(UOA)
A probability distribution is a table , graph or an equation that links each outcome of a statistical experiment with its probability of occurrence.
Probability distribution is nothing but the graph obtained when we plot the probability of all random variables.(UOA)
Usually, X represents the random variable X. P(X) represents the probability of X. P(X=x) refers to the probability that the random variable X is equal to a particular value.
Note on notation! We use capitalized letters to represent the random variables and lowercase for the specific values of the variable.
Definition of Probability Distribution
For previous example
(Let assume families with 2 children
S={(boy, boy),(boy, girl), (girl, boy), (girl, girl)} ={BB,BG,GB,GG} n(S)=4
Let define variable X number of boys they have
S={BB,BG,GB,GG} X={0,1,2})
P(X=0)=probability of X=0 :P( no boy)=P({GG})=1/4=25%
P(X=1)= probability of X=1 : P( one boy)=P({BG,GB})=2/4=50%
P(X=2)= probability of X=2 : P(two boys)=P({BB})=1/4=25%
There are two classes of probability functions:
Probability Mass Functions(PMF) for discrete random variable and Probability Density Functions (PDF) for continuous random variable.
Probability Mass Function (PMF)
If the random variable is a discrete random variable, the probability function is usually called the probability mass function (PMF). If X is discrete, then 𝑓(𝑥_𝑖 )=𝑃(X=𝑥_𝑖 ) .In other words, the PMF for a constant, 𝑥_𝑖, is the probability that the random variable X is equal to 𝑥_𝑖. The PMF can be in the form of an equation or it can be in the form of a table. In previous example f(1)=P(X=1)=P( one boy)=0.5=50%
Properties of probability mass functions:
1) f(x)≥0 f(x)> 0 for x in the sample space and 0 otherwise.
2) ∑f(x)=1. In other words, the sum of all the probabilities of all the possible outcomes of an experiment is equal to 1.
Probability Density Function (PDF)
If the random variable is a continuous random variable, the probability function is usually called the probability density function (PDF). Contrary to the discrete case, f(x)≠P(X=x)
Properties of a probability density function:
1) f(x)≥0 f(x)> 0 for x in the sample space and 0 otherwise.
2)The area under the curve is equal to 1.
Cumulative Distribution Function (CDF)
A cumulative distribution function (CDF), usually denoted F(x), is a function that gives the probability that the random variable, X, is less than or equal to the value x.
F(x)=P(X≤x)
In previous example:
F(0)=P(X≤0)=0.25
F(1)=P(X≤1)=0.25+0.5=0.75
F(2)=P(X≤2)=0.25+0.5+0.25=1
Note! The definition of the cumulative distribution function is the same for a discrete random variable or a continuous random variable. For a continuous random variable, however, P(X= x)=0. Therefore, the CDF, F(x)=P(X ≤ x)=P(X<x), for the continuous case. For example, if I randomly select any float number between 0 and 5
P(X=1.3)=0 ( 1/∞=0 )
F(1.3)=P(X≤1.3)=P(X<1.3)=1.3/5
P(X=2.5)=0 ( 1/∞=0 )
F(2.5)=P(X≤2.5)=P(X<2.5)=2.5/5 =50%
As mentioned before there are two main types of random variables:
discrete and continuous.
Discrete: possible values of finite or countable infinite numbers for example number of boys in a family with 3 children
Continuous: possible values of interval or uncountable numbers for example average height of children in a family with 3 children
Correspondingly, most probability distributions can be classified:
Discrete probability distributions
Continuous probability distributions
One of the most important discrete random variables is the binomial distribution and the most important continuous random variable is the normal distribution.
For probability distribution, an important aspect of the distribution of a data set is its shape. There are some common distribution shapes:
bell-shaped
Triangular
Uniform
Right skewed
Left skewed
Bimodal
Multimodal
Reverse J-shaped
J-shaped
In considering the shape of a distribution, it is helpful to observe the number of peaks (highest points).
Unimodal - A distribution has one peak.
Bimodal – A distribution has two peaks(two peaks may have different height).
Multimodal – A distribution has three or more peaks.
Technically, a distribution is bimodal or multimodal only if the peaks are the same height. However, in practice, distributions with pronounced but not necessarily equal height peaks are often referred to as bimodal or multimodal.
Symmetry – the distribution can be divided into two pieces that are mirror images of one another. E.g. bell-shaped (normal), some triangular, rectangular(uniform), inverted u-shaped, etc. Bimodal and multimodal distributions sometimes are symmetry, but that are not always true.
For the discrete probability distributions, only the most popular ones –Bernoulli and Binomial probability distribution are introduced in the following sections.
For the continuous probability distributions, 4 distributions (uniform ,normal, t ,and chi_ square distribution) are introduced in the following sections.
Discrete probability distribution:
A table, formula or graph that lists all possible values a discrete random variable can assume, together with their associate probabilities.
Requirements of discrete probability distribution:
If a random variable X can take values 𝑥_𝑖, then the following must be true:
0<= p(xi)<=1 for all 𝑥_𝑖
Sum(p(xi))=1 for all 𝑥_𝑖
In practice, probabilities assigned to values of a random variable are often estimated from relative frequencies from random samples.
e.g. coin-tossing 100 times or 200 times
Frequency distribution table to estimate the population probability distribution
Let us throw a coin 3 times, and define variable X number of heads,
a. please show probability distribution in table , graph and equation
b. Please show cumulative probability distribution in table
As mentioned before, a cumulative probability refers to the probability that the value of a random variable falls within a specified range.
Q : What is the probability that the coin flips would be in one or fewer tails?
P(X ≤ 1) = P(X=0) + P(X=1) = 0.25 + 0.5 = 0.75
A cumulative probability distribution can be represented by the following table:
So the weighted average of possible values of a random variable, with weights given by their respective theoretical probabilities, is known as the expected value , usually represented by E(x) .
As an example, if we assume families with 2 children and define random variable number of boys
The mean of a discrete random variable X (expected value) is defined by
μx = E(X)=∑x * P(x) or μ=E(X)=
The formula means that we multiply each value, 𝑥_𝑖, in the support by its respective probability , pi or f(x), and then add them all together. It can be seen as an average value but weighted by the likelihood of the value.
Expected Value (or mean) of a Discrete Random Variable:
The expected value is the average value that one would expect to obtain for a random variable if an experiment was repeated an infinite number of times.
Laws of expected value
If X and Y are random variables and C is any constant then
E(C)=C
E(CX)=CE(X)
E(X+Y)=E(X)+E(Y)
E(X-Y)=E(X)-E(Y)
5) E(XY)=E(X)E(Y) , if X and Y are independent
Variance of a Discrete Random Variable
The variance of a discrete random variable is given by:
The formula means that we take each value of 𝑥_𝑖, subtract the expected value, square that value and multiply that value by its probability. Then sum of all those values.
There is an easier form of this formula we can use.
Var(X)=∑128▒〖(𝑥_𝑖−𝜇)^2 𝑓(𝑥_𝑖 ) 〗=∑128▒〖𝑥_𝑖^2 𝑓(𝑥_𝑖 ) 〗−𝜇^2=𝐸(𝑥^2 )−𝐸^2 (𝑥)
The formula means that first, we sum the square of each value times its probability then subtract the square of the mean. We will use this form of the formula in all of our examples.
Standard Deviation of a Discrete Random Variable
The standard deviation of a random variable, X, is the square root of the variance.
Laws of variance
If X and Y are random variables and C is a constant
V(C)=0
2) V(CX)= C2 V(X)
3) V(X+C)=V(X)
4) V(X+Y)=V(X)+V(Y) if X and Y are independent
V(X-Y)=V(X)+V(-Y)=V(X)+(−1)^2V(Y)=V(X)+V(Y) if X and Y are independent
Recall that a binary variable is a variable that has only two possible outcomes. For example, sex (male/female) or
having a cancer (yes/no) are both examples of a binary categorical variable.
A random variable can be transformed into a binary variable by defining a “success” and a “failure”.
Usually, we show probability of success by “p”, and probability of failure by “q”, so q=1-p (since sum of the probability of all possible outcomes is equal to 1: q + p=1 q= 1 –p )
For example, if the probability of having cancer is 1% (p=0.01) the probability of not having cancer is 99%
( q= 1- p=1- 0.01=0.99 )
or consider rolling a fair six-sided die and recording the value of the face. The random variable,
value of the face, is not binary(X={1,2,3,4,5,6}). If we are interested, however, in the event A={6 is rolled}, then the “success” is rolling a six. The failure would be any value not equal to six. Therefore, we can create a new variable with two outcomes, namely A = {6} and B = {not a six} = {1, 2, 3,4, 5}. This new variable is now a binary variable. In this example the probability of success (p) is 1/6 and probability of failure (q=1-p) is 5/6
Or If we are interested, however, in the event A={less than3 is rolled}, then the “success” is rolling a 1 0r 2. The failure would be any value bigger than 2. Therefore, we can create a new variable with two outcomes, namely
A = {1,2} and B = {3,4,5,6}. This new variable is now a binary variable. In this example the probability of success (p) is 2/6 and probability of failure (q=1-p) is 4/6
Expected Value, Variance, and Standard Deviation of Binary (Bernoulli) distribution:
Usually in Binary (Bernoulli) distribution random variable is o or 1 ( o for failure and 1 for success). What is expected value , variance , and standard deviation for Bernoulli distribution?
Failure X=0 P(X=0)=q
Success X=1 P(X=1)=p
So for Bernoulli distribution:
Expected value = probability of success 𝜇=𝐸(𝑥)= p
Variance= probability of success* probability of failure 〖𝑉𝑎𝑟(𝑋)=𝜎〗^2=pq
Standard Deviation=square root of (probability of success* probability of failure) Sd=√𝑝𝑞
The binomial distribution is a special discrete distribution where there are two distinct complementary outcomes, a “success” and a “failure”.
We have a binomial experiment if all of the following four conditions are satisfied:
1)The experiment consists of n identical trials. ( in previous example we had 3 trials)
2)Each trial results in one of the two outcomes, called success and failure. ( in previous example we had success ( getting 6) and failure ( not getting 6) in each trail )
3)The probability of success, denoted p, remains the same from trial to trial.
(in previous example probability of getting 6 in each trail is the same as 1/6 )
4)The n trials are independent. That is, the outcome of any trial does not affect the outcome of the others
(in previous example result of each trial was independent of other trails).
Example :An FBI survey shows that about 80% of all property crimes go unsolved. Suppose that in your town 3 such crimes are committed, and they are each deemed independent of each other. What is the probability that 1 of 3 of these crimes will be solved?
First, we must determine if this situation satisfies ALL four conditions of a binomial experiment:
Does it satisfy a fixed number of trials? YES the number of trials is fixed at 3 (n = 3.)
Does it have only 2 outcomes? YES (Solved and unsolved)
Do all the trials have the same probability of success? YES (q=0.8p=1-q = 0.2)
Are all crimes independent? YES (Stated in the description.)
To find the probability that only 1 of the 3 crimes will be solved we first find the probability that one of the crimes would be solved. With three such events (crimes) there are three sequences in which only one is solved:
P(SFF)=P(Solved First, Unsolved Second, Unsolved Third )= (0.2)(0.8)( 0.8) = 0.128
P(FSF)=P(Unsolved First, Solved Second, Unsolved Third) = (0.8)(0.2)(0.8) = 0.128
P(FFS)=P(Unsolved First, Unsolved Second, Solved Third )= (0.8)(0.8)(0.2) = 0.128
We add these 3 probabilities up to get 0.384. Looking at this from a formula standpoint, we have three possible sequences, each involving one solved and two unsolved events. Putting this together gives us the following: 3(0.2)(0.8)(0.8)=0.384=(■(3@1))(0.2)(0.8)(0.8)=(■(3@1))𝑝(1−𝑝)(1−𝑝)=(■(3@1))𝑝(1−𝑝)^2
So for Bernoulli distribution:
Expected value = probability of success 𝜇=𝐸(𝑥)= p
Variance= probability of success* probability of failure 〖𝑉𝑎𝑟(𝑋)=𝜎〗^2= pq
Standard Deviation=square root of (probability of success* probability of failure) SD=√𝑝𝑞
for Binomial distribution:
Expected value =n times of probability of success 𝜇=𝐸(𝑥)=np
Variance= n times of( probability of success* probability of failure) 〖𝑉𝑎𝑟(𝑋)=𝜎〗^2= npq
Standard Deviation=square root of (n times of (probability of success* probability of failure)) SD=√n𝑝𝑞
1. Sampling Distribution of the Sample Mean
2. Central Limit Theorem
3. Sampling Distribution of the Sample Proportion
In inferential statistics, we want to use the characteristics of the sample (i.e. a statistic) to estimate the
characteristics of the population (i.e. a parameter).
Sampling Distribution of the Sample Mean
Sampling Distribution of the Sample Proportion
If we obtain a random sample and calculate a sample statistic from that sample, the sample statistic
is a random variable (wow!). The population parameters, however, are fixed. For example lets assume
the average height of all grade 12 students in Canada is 172.8 cm. so if we want to find average
height of aIl grade 12 students in Canada, this number is population mean and it is parameter and
the value of that is a fixed number (here 172.8 cm.) .Now if we randomly select 100 students form
grade 12 in Canada to measure height of them and then calculate average of height of them , it is
sample mean and it is statistic and value of that is changing from one sample to another sample I
mean it could be 175.2 for one sample, 168.9 for another sample , and 171.4 for another sample
and so on, so this sample mean is random variable and it has its own distribution that we call it
sampling distribution of the sample mean.
The statistic is a random variable, can we find the distribution? The mean? The standard deviation?
The answer is yes! This is why we need to study the sampling distribution of statistics. So what is a
sampling distribution?
In this example if you randomly select 100 students 50 times and calculate the average height of
them(sample mean) , this sample mean has its own distribution
Assume 15 of them have sample mean equals to 174.5, 16 of them have
sample mean equals to 167.8, and 9 of them have
Sample mean equals to 181.2, and 8 of them have sample mean equals to
165.5 and 2 of them have sample mean equals to 185
So the above graph shows sampling distribution of the 50 sample means with size of 100.
Sampling Distribution
The sampling distribution of a statistic is a probability distribution based on a large number of samples of
size n from a given population
Sampling Distribution
The sampling distribution of a statistic is a probability distribution based on a large number of samples of
size n from a given population
As you can see population mean is a fixed number(=14), but sample mean depends on which two pumpkins are
selected , so it is changing from one sample to another, and it is random variable. In following table I show you
all possible outcomes for sample mean
the possible sample means cluster
more closely around the population
mean as the sample size increases.
Thus, the possible sampling error
decreases as sample size increases
For example we want to find average height of aIl grade 12 students in Canada, so this number is population mean
and it is parameter and the value of that is a fixed number assume it is 176 cm. If we randomly select 100 students
form grade 12 in Canada to measure height of them and then calculate average of height of them , it is sample mean
and assume we will get 178cm (we have 2cm error(178-176=2)), If we randomly select 100 students again and
calculate average height of them , it is sample mean and assume we will get 175cm (we have -1cm error(175-176=-
1)), )), If we randomly select 100 students again and calculate average height of them , it is sample mean and assume
we will get 174cm (we have -2cm error(175-176=-2)), until here the mean of sample mean equals to 178+175+174
3
=175.66 cm
If you repeat this experiment over and over again the mean of sample mean is going to very close to population
mean(176cm)
As you can see by increasing sample size standard error( standard deviation of sample mean tends to decrease
Sampling Error
The error resulting from using a sample characteristic to estimate a population characteristic.
Sample size and sampling error: the possible sample means cluster more closely around the population
mean as the sample size increases. Thus, the possible sampling error decreases as sample size increases
If Population is Normal:
If the population is normally distributed with mean μ and standard deviation σ, then the sampling
distribution of the sample mean is also normally distributed no matter what the sample size is and you are
allowed to find probabilty by converting it to z-distribtion. Recall that when the sampling is done with
replacement or if the population size is large compared to the sample size, then 𝑥ҧhas mean μ and
standard deviation 𝜎
𝑛
. We use the term standard error for the standard deviation of a statistic, and since
sample average, 𝑥ҧis a statistic, standard deviation of 𝑥ҧis also called standard error of 𝑥ҧ.
If Population is Normal:
If the population is normally distributed with mean μ and standard deviation σ, then the sampling
distribution of the sample mean is also normally distributed no matter what the sample size is and you are
allowed to find probabilty by converting it to z-distribtion. Recall that when the sampling is done with
replacement or if the population size is large compared to the sample size, then 𝑥ҧhas mean μ and
standard deviation 𝜎
𝑛
. We use the term standard error for the standard deviation of a statistic, and since
sample average, 𝑥ҧis a statistic, standard deviation of 𝑥ҧis also called standard error of 𝑥ҧ.
The mean of the sampling distribution is very close to the population mean. The standard deviation of the
sampling distribution is smaller than the standard deviation of the population since it is equal to
Population Standard deviation
𝑠𝑎𝑚𝑝𝑙𝑒 𝑠𝑖𝑧𝑒
Example: The engines made by Ford for speedboats have an average power of 220 horsepower (HP)
and standard deviation of 15 HP. You can assume the distribution of power follows a normal
distribution.
Consumer reports are testing the engines and will dispute the company's claim if the sample mean is
less than 215 HP. If they take a sample of 4 engines, what is the probability the mean is less than 215?
he Central Limit Theorem applies to a sample mean from any distribution. We could have a leftskewed or a right-skewed distribution for population , but sample mean is approximately noraammy
distributed as long as the sample size is large, the distribution of the sample means will follow an
approximate Normal distribution.
For the purposes of this course, a sample size of n>30 is considered a large sample.
Central Limit Theorem:
For a large sample size (we will explain this later), sample mean (𝑥ҧ) is approximately normally
distributed, regardless of the distribution of the population one samples from.
So if you know that your population is normal for any sample size sample mean is normally distributed, and if your
population is not normal but sample size is large enough sample mean is normally distributed based on CLT(Central
Limit Theorem)
Notes on the CLT for this demonstration:
•If the population is skewed and sample size small, then the sample mean won't be normal.
•If the population is normal, then the distribution of sample mean looks normal even if n=2.
•If the population is skewed, then the distribution of sample mean looks more and more normal
when n gets larger
So, the central limit theorem (CLT) states that the following properties hold true for
any type of data as long as the number of samples is high:
1.Mean of a sampling distribution (X) = Mean of the population (μ).
2.Standard deviation of a sampling distribution (standard error) = σ/√n.
3.For n > 30, a sampling distribution becomes normally distributed. This property of
sampling distributions is called the central limit theorem.
Example:
1.For a given discrete random variable X, the population parameters are μ = 5 and σ = 12. Now, if you take
10,000 sample means of size 100 from this population, what would the parameters of the distribution of
sample means be?
Mean = 5 and standard error = 1.2
2. For a given continuous random variable Y, the population parameters are μ = 55 and σ = 1.8. What is the
probability that a randomly chosen sample of size 36 will have a mean that lies between 54.7 and 55.3?
According to the central limit theorem, the mean of the distribution of sample means (X̄) will be equal to the
mean of the population (μ), and the standard error (SE) will be σ/√n, where n is the sample size. Hence, the
mean is 55 and the standard error is 0.3 in this case. Now, 54.7 lies one standard error to the left and 55.3
lies one standard error to the right of the mean. Hence, P(54.7 < X̄< 55.3) = 0.68, which is equal to 68%.
3. For a given discrete random variable Y, the population parameters are μ = 30 and σ = 10.5. What is the
probability that a randomly chosen sample of size 49 will have a mean that lies between 27 and 31.5?
Metro College Of Technology 28
. For a given continuous random variable Y, the population parameters are μ = 55 and σ = 1.8. What is the probability that a
randomly chosen sample of size 36 will have a mean that lies between 54.7 and 55.3?
According to the central limit theorem, the mean of the distribution of sample means (X̄) will be equal to the mean of the
population (μ), and the standard error (SE) will be σ/√n, where n is the sample size. Hence, the mean is 55 and the standard
error is 0.3 in this case. Now, 54.7 lies one standard error to the left and 55.3 lies one standard error to the right of the
mean. Hence, P(54.7 < X̄< 55.3) = 0.68, which is equal to 68%
81.5 : According to the central limit theorem, the mean of the
distribution of sample means (X̄) will be equal to the mean of
the population (μ), and the standard error (SE) will be σ/√n,
where n is the sample size. Hence, mean is 30 and standard
error is 10.5/√49 = 10.5/7 = 1.5 in this case. Now, 27 lies two
standard errors to the left of the mean and 31.5 lies one
standard error to the right of the mean. Hence, P(27 < X̄< 31.5)
= 13.5 + 0.68 = 0.815, which is equal to 81.5%
P(27<Xbar<31.5)=
27-30=-3
31.5-30=1.5
Standard deviation=1.5
With the Central Limit Theorem, we can finally define the sampling distribution of the sample mean.
The sampling distribution of the sample mean will have:
• The sample mean as the population mean, μ
• Standard deviation [standard error] of SE(Xbar)= 𝜎
𝑛
It will be Normal (or approximately Normal) if either of these conditions is satisfied
• The population distribution is Normal
or
• The sample size is large (greater than 30) so you can rely on CLt
Note:As you increase the sample size, the standard error, i.e., the standard deviation of the sampling distribution
Decrease
Example: The weights of baby giraffes are known to have a mean of mean of population=125 pounds and a
standard deviation of population of 15 pounds. If we obtained a random sample of n=40 baby giraffes,
a.what is the probability that the sample mean will be between 120 and 130 pounds?
b.what is the 75 percentile of the sample means of size n=40?
The sampling distribution of the sample mean will have:
• The sample mean as the population mean, μ
• Standard deviation [standard error] of SE(Xbar)= 𝜎
𝑛
It will be Normal (or approximately Normal) if either of these conditions is satisfied
• The population distribution is Normal
or
• The sample size is large (greater than 30) so you can rely on CLT
Whenever the population has a normal probability distribution, the sampling distribution of ¯X is a
normal probability distribution for:
any sample size
A population has a mean of 75 and a standard deviation of 8. A random sample of 800 is
selected. The expected value of ¯X is:
75
Answer
Does the problem indicate that the distribution of weights is normal? No, it does not. In order to
apply the Central Limit Theorem, we need a large sample. Since n=40>30, we can use the theorem.
The sampling distribution of the sample mean is approximately Normal with mean μ=125 and standard
error 𝜎Τ 𝑛 =15/ 40=2.37
a)We want P(120< 𝑥ҧ<130)=P(𝑥ҧ<130) - P(𝑥ҧ< 120
Note: If we don’t know the population standard deviation , instead of z-distribution we should use tdistribution that I will explain for you in this part.
Note: If we don’t know the population standard deviation , instead of z-distribution we should use tdistribution that I will explain for you in this part.
Let me first start by reviewing a Bernoulli random variable, Y
Bernoulli or Binary Random Variable Y
For an experiment that results in a success or a failure , let the random variable equal 1, if there is a success, and
0 if there is a failure. Therefore,
Y= 1 success
0 failure
and let p be the probability of a success. For example in previous example blue is success and because 2 of 5
children like blue P=2
5
= 0.4 = 40%
The Bernoulli random variable is a special case of the Binomial random variable, where the number of trials is
equal to one.
Suppose we have, say n, independent trials of this same experiment. Then we would have n values of Y,
namely Y1,Y2,...,Yn.
If we define X to be the sum of those values, we get X=Y1+Y2+…+Yn
X is then a Binomial random variable with parameters n and p.( X is the number of success)
For example in previous example
when you chose A and B
Y1=0 ,Y2=1
➔X=0+1=1
➔𝑃 =
1
2
for Binomial distribution:
Expected value =n times of probability of success ➔ 𝜇 = 𝐸 𝑥 = np
Variance= n times of( probability of success* probability of failure)➔ 𝑉𝑎𝑟 𝑋 = 𝜎
2= npq
Standard Deviation=square root of (n times of (probability of success* probability of failure))➔ SD= n𝑝q
We learned how to use the Central Limit Theorem to find the sampling
distribution for the sample mean and the sample proportion under certain
conditions.
We learned that the sampling distributions are centered around the
population parameter with variability. All of this theory was built knowing the
parameter. Can we use this information in situations where the parameter is
unknown?
The answer is yes! In the next lesson, we take our first step into inference and
into the “real world” where the population parameters are unknown and need
to be estimated
Introduction to Inferences
Estimation and Confidence Intervals
Inference for the Population Proportion
Inference for the Population Mean
Student's t-distribution
Chi-Square Distribution
Statistical Inference
The real power of statistics comes from applying the concepts of probability to situations where you have data but not necessarily the whole population. The results, called statistical inference, give you probability statements about the population of interest based on that set of data.
Types of Statistical Inference
There are two types of statistical inferences: Estimation and Statistical Tests.
Estimation
Use information from the sample to estimate (or predict) the parameter of interest of popultaion.
For instance, using the result of a poll about the president's current approval rating to estimate (or predict) his or her true current approval rating nationwide.
Statistical Tests (Hypothesis Tests)
Use information from the sample to determine whether a certain statement about the parameter of interest is true. Statistical tests are also referred to as hypothesis tests.
For instance, suppose a news station claims that the President’s current approval rating is more than 75%. We want to determine whether that statement is supported by the poll data.
Estimation
Two common estimation methods are point and interval estimates.
Point Estimates
An estimate for a parameter that is one numerical value. An example of a point estimate is the sample mean or the sample proportion.
Note: Point estimate of parameter is equal to value of sample statistic.
Interval Estimates
Interval estimates give an interval as the estimate for a parameter. This is a new concept which is the focus of this lesson. Such intervals are built around point estimates which is why understanding point estimates is important to understanding interval estimates.
In this course, the interval estimates we find are referred to as confidence intervals.
Confidence Interval
An interval of values computed from sample data that is likely to cover the true parameter of interest.
the center of our interval should be the point estimate for the parameter of interest.
e.g. a survey of the current approval rating of the President. In such surveys, you may hear reference to the "44% of those surveyed approved of the President's reaction" (this is the sample proportion), and "the survey had a 3.5% margin or error, or ± 3.5%." This latter number is the margin of error. In this example, the point estimate is 44% and
interval would be from 44% ± 3.5% i.e., 40.5%(=44%-3.5%) to 47.5% (44%+3.5%)
In statistical inference we have two methods: point estimates and confidence interval
What is point estimates: in this method we get one single value as an output: 1.sample proportion and sample mean
What is confidence interval: it is range that the center point is sample mean or sample proportion
Assume we randomly selected 200 students from grade 12 in Canada and measured height of them and find that mean of height of them is 175 cm and with 95% confidence margin of error is 2 cm
So we can say
175 is sample statistic= sample mean
Note: Point estimate of parameter is equal to value of sample statistic.
Average of height of all grade 12 students in Canada equals to 175 cm (this is a point estimate of Average of height of all grade 12 students in Canada )
Or we can say Average of height of all grade 12 students in Canada is between
(175-2, 175+2)
173cm and 177 cm (175±2) (this is interval estimate of" Average of height of all grade 12 students in Canada)"
Confidence interval = Interval Estimate= [173 , 177]
(173+177)/2=175= is your point estimate in this case it is sample mean
The point estimate equals to sample mean and based on sample might be changed so we need calculate margin of error. For finding margin of error based on we have to know confidence interval ,sample standard deviation and sample size. I will show and prove its formula
Point Estimates for a Population Mean
The point estimate of the population mean, μ , is qual to sample mean
Point Estimate of the Population Mean = 𝑥 ̅= sample mean
The notation that we are using for sample mean=Xbar
The notation for population mean= μ
Point estimate of the population proportion
The point estimate of the population proportion, p, is equal to sample proportion
Point Estimate of the Population Proportion = 𝑃 ̂ = sample proportion
Sample proportion=p
Population proportion= 𝑃 ̂
Confidence Interval
Example: We measure the heights of 40 randomly chosen men, and get a mean height of 175cm,
We also know the standard deviation of men's heights is 20cm.
The 95% Confidence Interval (we show how to calculate it later) is:
175cm ± 6.2cm=(168.8 cm,181.2 cm)
Sample statistic =175 cm
Margin of error (ME)=6.2 cm
Claim: we are claiming that the mean height of population is 175cm
(175-6.2,175+6.2)= (168.8,181.2)
Confidence level is: 95 percent
Confidence interval is: (168.8,181.2)
Significance level or aplha: 1-95%=0.05: the probability of rejecting null hypothesis in a test
We can say we are 95% confident that the true mean of ALL men (population Mean) is between 168.8cm and 181.2cm.
Mean= 171: this one is belong to 95 percent
Mean=182: this one is belong to those 5 percent
But it might not be!
The "95%" says that 95% of experiments like we just did will include the true mean, but 5% won't.
So there is a 1- 95%=(5%=5/100=1/20) that our Confidence Interval does NOT include the true mean.
In sampling distribution we are getting different samples from population, for some of the samples we are getting 175 cm as a mean for some of them not, so here we can say in 95 vases we got 175 and for the rest of them or in 5 cases we did not get 175 cm as a hight mean
General Format of a Confidence Interval :
the center of our interval should be the point estimate for the parameter of interest. With the estimated standard error of the point estimate, we can include a measure of confidence to our estimate by forming a margin of error.
Let us quickly get into the four terminologies that you just learned about:
The probability associated with a claim is called the confidence level.
The maximum error made in the sample mean is called the margin of error.
The final interval of values is called the confidence interval.
Significance level is estimated as Significance level (α) = (1 - Confidence level).
Recall that The point estimate for population mean equals to sample mean and based on sample might be changed so we need calculate margin of error and for finding margin of error based on what number you consider as your confident we must know sample standard deviation and sample size , I will show and prove its formula
And similar to it
that The point estimate for population proportion equals to sample proportion and based on sample might be changed so we need calculate margin of error and for finding margin of error based on what number you consider as your confident we must know sample standard deviation and sample size , I will show and prove its formula
Note!
Some might say, "Why not just be 100% confident?", but that does not make practical sense. For instance, what value comes from me saying I am 100% confident that the approval rating for the President is from 0% to 100%. That is the only interval in which one can be truly confident will capture the actual proportion. (for 100% confident your range must include all possible outcomes)
Similarly, if you were to ask your professor what they think your score I will be on an exam that has 20 marks total and they reply, "zero to twenty", what would you think of that answer? However, one does want to be as confident as reasonably possible. Most confidence levels use ranges from 90% confidence to 99% confidence, with 95% being the most widely used. In fact, when you read a report that includes a margin of error, you can usually assume this has a 95% confidence attached to it unless otherwise stated.
If I say your mark is between 15 to 18 with 95% confident that means, there is 5% possibility that you will get less than 15 or more than 18
1-alpha=0.95
Alpha=0.05
The value of the multiplier increases as the confidence level increases. This leads to wider intervals for higher confidence levels. We are more confident of catching the population value when we use a wider interval.
When the confidence level is 0.95 what is the significancy level? alpha=0.05
When the confidence level is 0.90, it is mostly use in a cases we have less restrict like: lifestyle and health
When the confidence level is 0.98 it means that the significancy level is 0.02, this level mostly use in critical situation like trying new or novel surgery because the interval will be narrower and it means we have much level of restrict
Commonly Used Alpha Levels
Now we learned how to calculate multiplier for constructiong confidence level:
As you know for creating CL= point estimate + or – ME (Multiplier*SD)
Point estimate: sample mean or sample proportion
Recall that:
If np and n(1−p) are greater than fifteen, then 𝑃 ̂" "is approximately normal with mean, p, standard error
So, z= (𝑃 ̂−𝑃)/√(𝑃(1−𝑃)/𝑛)
Recall that
if n≥30 0r if X comes from a normal distribution with mean, μ, and variance, 𝜎^2,
then the sampling distribution will be approximately normal with mean μ and
standard error, SE(𝑥 ̅ )=𝜎/√𝑛
So when the population is normal or when the sample size is large enough (n≥30) then,
Z=(𝑥 ̅−𝜇)/(𝜎/√𝑛)
where Z has a standard Normal distribution.
One way to estimate σ is by s, the standard deviation of the sample, and replace σ by s in the above Z-equation. However, this new quotient no longer has a Z-distribution. Instead, it has a t-distribution. We call the following a 'studentized' version of 𝑥 ̅ :
t= (𝑥 ̅−𝜇)/(𝑠/√𝑛)
So first you have to find out that is the population is known or unknown, for known population we must use Z-distribution for unknown population we must use t-distribution.
Now we learned how to calculate multiplier for constructiong confidence level:
As you know for creating CL= point estimate + or – ME (Multiplier*SD)
Point estimate: sample mean or sample proportion
Recall that:
If np and n(1−p) are greater than fifteen, then 𝑃 ̂" "is approximately normal with mean, p, standard error
So, z= (𝑃 ̂−𝑃)/√(𝑃(1−𝑃)/𝑛)
Margin of error
So the population mean is a subset of sample mean plus minus margin of error
And margin of error depend on confident , standard deviation and sample size:
By increasing confident (as an example using 98% confident instead of 95%)
〖 ( 𝑧〗_(𝛼_2/2) > 〖 𝑧〗_(𝛼_1/2) ) the margin of error increase as well so you will get wider interval
Multiplier for 0.95=1.96
Multiplier for 0.98=2.326 1-alpha=0.98 alpha=0.02 what is 𝒛_(𝜶∕𝟐)=0.01
If we increase sample size , Margin of error decrease
By increasing population standard deviation ,
Margin of error tends to be increased
If we want to estimate population mean with a single value( point estimate), we say population mean is 175 cm, but in this question, we want to find confidence interval
Step 1: start with
the number of observations n
the sample mean 𝑥 ̅
and the population standard deviation 𝜎
Using our example:
number of observations n = 40
the sample mean 𝑥 ̅ = 175
The population standard deviation 𝜎= 20
INTERPRET THE CONFIDENCE INTERVAL :
we can say we are (𝟏−𝛂)𝟏𝟎𝟎% confident that the
population Mean is between 𝑥 ̅+𝒛_(𝜶∕𝟐) 𝜎/√𝑛 and 𝑥 ̅−𝒛_(𝜶∕𝟐) 𝜎/√𝑛
Point Estimates for a Population Mean
The point estimate of the population mean, μ is:
Point Estimate of the Population Mean = 𝑥 ̅= sample mean
Point estimate of the population proportion
The point estimate of the population proportion, p, is:
Point Estimate of the Population Proportion = 𝑃 ̂ = sample proportion
Confidence Interval for the Population Proportion:
CHECK CONDITIONS: Check all conditions before using the sampling distribution of the sample proportion.
We previously used np and n(1−p). But p is not known. Therefore, for the confidence interval, we will use
n𝑃 ̂" ">15
and
n(1− 𝑃 ̂" ")>15
The general form of the confidence interval is ‘ ‘The point estimate is the
sample proportion, 𝑃 ̂" ",and the estimated standard error is . If the conditions are satisfied, then the sampling distribution is approximately normal. Therefore, the multiplier comes from the normal distribution. This interval is also known as the one-sample z-interval for p, or the Normal Approximation confidence interval for p.
We are using different name for this interval: 1. confidence interval 2. one-sample z-interval for p 3. Normal Approximation confidence interval for p.
confidence interval for the population proportion, p
Where 𝑧_(𝛼∕2) represents a z-value with α/2 area to the right of it.
General notes about the confidence interval...
The ± in the formula above means "plus or minus". It is a shorthand way of writing
It is centered at the point estimate, 𝑃 ̂" ".
The width of the interval is determined by the margin of error.
You must determine the multiplier
INTERPRET THE CONFIDENCE INTERVAL Applying the template from earlier in the lesson we can say we are (1−α)100% confident that the population proportion is between
Between and
( or
Z=(𝑃 ̂−𝑃)/𝑆𝐸 and -1.96 < Z< 1.96 -1.96< (0.44−𝑃)/0.0128<1.96 -1.96*0.0128< 0.44−𝑃 <1.96 *0.128
-0.0251< 0.44−𝑃 <0.251 -0.0251-0.44< −𝑃 <0.0251-0.44 -0.4651< −𝑃 <-0.4149
0.4651> 𝑃 >0.4149 0.4149<p<0.4651 0.415<p<0.465)
"We are 95% confident that the overall U.S. adult approval rating for the current president is from 41.5% to 46.5%." You could also see this written as, "The current U.S. approval rating for the president is 44% with a 95% margin of error of 2.5%." Commonly, the standard level of confidence is 95% so that reference is often left out as that is the assumed level of confidence unless otherwise stated. Also, the method calculates a proportion but often the reported values are converted to percentages. If you use the decimal formal (e.g. 0.415 and 0.465) then reference these as proportion and not percentage.
Precision
The wider the interval, the poorer the precision. Note that the higher the confidence level, the wider the width (or equivalently, half width) of the interval and thus the poorer the precision.
For 100% confidence interval we must cover all of possible outcomes
Since the confidence level reflects the success rate of the method we use to get the confidence interval, we like to have a narrower interval while keeping the confidence level at a reasonably higher level.
For most newspapers and magazine polls, it is understood that the margin of error is calculated for a 95% confidence interval (if not stated otherwise). A 3% margin of error is a popular choice also. For instance, you might see a television poll state that the "approval rating of the president is 72%; the margin of error of the poll is plus or minus 3%."
If we want the margin of error smaller (i.e., narrower intervals), we can increase the sample size. Or, if you calculate a 90% confidence interval instead of a 95% confidence interval, the margin of error will also be smaller. However, when one reports it, remember to state that the confidence interval is only 90% because otherwise, people will assume 95% confidence.
For 0.90 confidence level the critical value: 1.645
For 0.95 confidence level the critical value is: 1.96
When you have narrower interval you have more precision
Recall that when we are looking for population mean we have two scenario:
Case 1: σ is known
when the population is normal or when the sample size is large (n≥30) then, sample mean in approximately normally distributed
Z=(𝑥 ̅−𝜇)/(𝜎/√𝑛)
where Z has a standard Normal distribution.
Case 2: σ is unknown
One way to estimate σ is by s, the standard deviation of the sample, and replace σ by s in the above Z-equation. However, this new quotient no longer has a Z-distribution. Instead, it has a t-distribution. We call the following a 'studentized' version of 𝑥 ̅ :
t= (𝑥 ̅−𝜇)/(𝑠/√𝑛)
CHECK THE CONDITIONS
One of the following conditions need to be satisfied:
If the sample comes from a Normal distribution, then the sample mean will also be normal. In this case, (𝑥 ̅−𝜇)/(𝑠/√𝑛) will follow a t-distribution with n−1 degrees of freedom.
If the sample does not come from a normal distribution but the sample size is large (n≥30), we can apply the Central Limit Theorem and state that 𝑥 ̅ is approximately normal. Therefore, (𝑥 ̅−𝜇)/(𝑠/√𝑛) will follow a t-distribution with n−1 degrees of freedom.
When we do not know the standard deviation of population we are using t, In t distribution we have n-1 degree of freedom: it represents the number of values in the final calculation of a statistic, df
3,5,7,100= 115/4=28.47 in this example the degree of freedom or df is 3
in this example, once we know the mean of the first three data points, the last data point can not change freely, if we want to
Keep the overall mean equal to 28.47,
(𝟏−𝛂)𝟏𝟎𝟎%confidence interval for the population Mean, 𝜇
𝑥 ̅±𝒕_(𝜶∕𝟐) 𝑠/√𝑛
where the t-distribution has df=n−1. This interval is also known as the one-sample t-interval for the population mean.
Student's t-distribution
we are using t when the standard deviation for population is unknown and sample size is less than 1000, if it is more than 1000 we can use z distribution and use sample standard deviation as population standard deviation
In probability and statistics, Student's t-distribution (or simply the t-distribution) is any member of a family of continuous probability distributions that arises when estimating the mean of a normally distributed population in situations where the sample size is small and the population standard deviation is unknown. The t-distribution is different for different sample size, n. The graph below shows the t-distribution for degrees of freedom of 10 (blue) and 30 (red dashed).
The t-distribution plays a role in a number of widely used statistical analyses, including Student's t-test or T-test for assessing the statistical significance of the difference between two sample means, the construction of confidence intervals for the difference between two population means, and in linear regression analysis.
The degrees of freedom refers to the number of independent observations in a set of data which is equal to n – 1
Introduction to Inferences
2. Estimation and Confidence Intervals
3. Inference for the Population Proportion
4. Inference for the Population Mean
5. Student's t-distribution
6. Chi-Square Distribution
In probability and statistics, Student's t-distribution (or simply the t-distribution) is any member of a family of
continuous probability distributions that arises when estimating the mean of a normally distributed population in
situations where the sample size is small and the population standard deviation is unknown. The t-distribution is
different for different sample size, n. The graph below shows the t-distribution for degrees of freedom of 10
(blue) and 30 (red dashed).
The t-distribution plays a role in a number of widely used
statistical analyses, including Student's t-test or T-test for
assessing the statistical significance of the difference between
two sample means, the construction of confidence intervals for
the difference between two population means, and in
linear regression analysis.
The degrees of freedom refers to the number of independent
observations in a set of data which is equal to n – 1
Smaller than 1000 ( if n>1000 t distribution is very close to z
distribution)
Properties of the t-distribution:
1.t is symmetric about 0 (like Z-distribution )
2.t-distribution is more variable than the Standard Normal distribution(Z-distribution)
3.t-distributions are different for different degrees of freedom (d.f.=n-1).
4.The larger n gets (or as n goes to infinity), the closer the t-distribution is to the z-distribution.
5.Like 𝑧𝛼
2
The meaning of 𝑡𝛼
2
is the t-value having the area 𝛼
2
to the right of it.
`
7. You know that z has a mean of 0 and variance of 1
But for t distribution The variance is always
greater than 1,
although it is close to 1 when there are
many degrees of freedom. With infinite
degrees of freedom, the t distribution
is the same as the standard normal distribution (z-distribution).
Example: Find 𝒕𝟎.𝟎𝟓 where we have 21 observation .
The t-distribution is symmetric and bell-shaped, like the normal distribution, but has heavier
tails, meaning that it is more prone to producing values that fall far from its mean.
FYI:6. The variance is equal to 𝑣
𝑣−2
=
𝑛−1
𝑛−3
, where v the
degree of freedom and v ≥ 2 (v=n-1)
So, a T-distribution is not much different from a Z-distribution.
It is just that a T-curve is shorter and flatter than a Z-curve, as
shown in this diagram.
Method 1:Use this t
-table
𝒕
𝟎.𝟎𝟓
, 𝟐𝟎 =1.725
Note! When the corresponding
degree of freedom is not given in
the table, you can use the value for
the closest degree of freedom that
is smaller than the given one. We
use this approach since it is better
to err in a conservative manner (get a t-value that is slightly larger than
the precise
t-value).
Note! When the sample size is larger
than 30, the t
-values are not that
different from the z
-values. Thus, a
crude estimate for
𝒕
𝟎.𝟎𝟓 with 34
degrees of freedom is
𝒛
𝟎.𝟎𝟓=1.645.
Although it is a crude estimate, when
software is available, it is best to find
the
t values rather than use the z.
Example: You are interested in the average emergency room (ER) wait time at your local hospital.
You take a random sample of 50 patients who visit the ER over the past week. From this sample, the
mean wait time was 30 minutes and the standard deviation was 20 minutes. Find a 95% confidence
interval for the average ER wait time for the hospital.
1)Check conditions: Is the population data normal? We don't know. In order to
apply the Central Limit Theorem, we need a large sample. Since n=50>30, we can use the central
limit theorem.
2)The population standard deviation is unknown; we only know the sample standard deviation.
We know that 𝝁 ∈ 𝒙ഥ ± 𝒛𝜶Τ𝟐 ∗
𝝈
𝒏
but for using this formula we need population standard deviation (𝜎) , since here
we don’t know 𝜎 we estimate standard Error ( 𝝈
𝒏
) by sample standard deviation (estimated SE = 𝒔
𝒏
)
and in this scenario, we must use t-distribution 𝝁 ∈ 𝒙ഥ ± 𝒕𝜶
𝟐
,𝒏−𝟏
∗
𝒔
𝒏
Having satisfied the conditions we proceed by finding the proper multiplier from the t-table. With n of 50 the d.f. are
49, and for 95% confidence, the alpha value is 5% or 0.05.
From the t-table under the column 0.025 (remember we use α/2) and a d.f. of 40 (since 49 is not on the table), we arrive
at a t-value of 2.021 Completing our confidence interval formula,
From online calculator, we arrive at a t-value of |-2.010|=2.01
Shapiro-Wilk Test
Tests whether a data sample has a Gaussian distribution.
Assumptions
•Observations in each sample are independent and identically distributed (iid).
Interpretation
•H0: the sample has a Gaussian distribution.
•H1: the sample does not have a Gaussian distribution
Chi-Square Distribution
Sampling distribution of s
2
Suppose we have a sample with size n taken from normal N(u, σ)
The statistic
Chi-Square =Χ
2 = 𝑛 − 1
𝑠
2
𝜎2 =(n-1) * 𝑆𝑎𝑚𝑝𝑙𝑒 𝑉𝑎𝑟𝑖𝑎𝑛𝑐𝑒
𝑃𝑜𝑝𝑢𝑙𝑎𝑡𝑖𝑜𝑛 𝑉𝑎𝑟𝑖𝑎𝑛𝑐𝑒 with d.f =n-1
is called the chi-squared statistic which have chi-squared distribution
We use Chi_Square to estimate population standard deviation and fir test of independency
of two categorical variables
In the following picture,
✓ the red curve shows the distribution of chi-square values computed from all possible
samples of size 3 (v = 3-1 = 2).
✓ the green curve shows the distribution of chi-square values computed from all possible
samples of size 5 (v = 5-1 = 4).
✓ the blue curve shows the distribution of chi-square values computed from all possible
samples of size 11 (v = 11-1 = 10).
As the degrees of freedom increase, the chi-square
curve approaches a normal distribution
The properties of Chi-Square Distribution:
➢ The mean of the distribution is equal to the number of degrees of freedom:
E(Χ
2)=μ = n-1
➢ The variance is equal to two times the number of degrees of freedom:
Introduction to Hypothesis Testing
2. Hypothesis Test for One-Sample Mean
3. Hypothesis Test for One-Sample Proportion
As mentioned before, methods of making inferences about
parameters are either
• estimating the parameter
Or
• testing a hypothesis about the value of the parameter.
In this part, we will introduce the concepts of hypothesis testing.
Then we will discuss hypothesis testing for a population proportion ,
and we discuss inference for the population mean.
The hypotheses are the most important aspect. If the hypotheses are
incorrect, your conclusion will also be incorrect.
Suppose we have X ~ N(𝜇, 𝜎
2
), we don’t know the parameters
Take sample x1, x2, x3 ,…,xn from above population,
One way we can do is we can use the sample building statistics to estimate the
parameters
Another way we can do we can use statistical test called Hypothesis Testing
to answer question like
Example: Is average body temperature of adults really 37 degrees Celsius
Hypothes testing allows you to check the statistical credibility of the claims or assumptions, this
claim could be a claim by the customer or by a company, or just a form of the popular belief that
exists among people.
In using hypothesis testing you can gather evidence in the form of a sample and then prove
whether this claim should be rejected or not.
We all come across several claims every day. An airline company claims that 90% of its flights are on
time. A consultant claims that using just-in-time production can reduce your inventory cost per unit
by $10. A tyre manufacturer claims that its tyres last 50% longer than its competitors’. You are
probably left wondering how many of these are true.
Application of hypotheses testing:
• Campaign Analytics: Analyze if the campaign was effective or not based on the
response rate
• Let’s say an e-commerce company wants to decide between two variants of a button:
“Shop now” and “Buy now”. Although it may seem insignificant, practically speaking, it
has been observed that improving the UI/UX design can have enormous benefits for
the company. In such cases, we perform A/B testing, which is a subset of hypothesis
testing.
• Another important area where hypothesis testing is extensively used is checking the
validity of machine learning models. As the course progresses, you will learn how to
build and validate various machine learning models. This is where hypothesis testing
will come in handy.
• Any product that you buy comes with a warranty. Electronic products generally come
with a one-year warranty, whereas car manufacturers give a 3–10-year warranty
depending on the components of the car. These claims are majorly driven by
hypothesis testing.
Listed below are some important properties of null and alternate hypotheses:
•The null and alternate hypotheses are always mutually exclusive events.
•Null hypothesis (H0): This statement is the default assumption/belief (the status quo).
•Alternate hypothesis (Ha): This statement challenges the status quo, i.e., it states the opposite of
the null hypothesis.
•The null and alternate hypotheses must be collectively exhaustive event
•The null and alternate hypotheses must be collectively exhaustive events.
This means that at least one of these events must be true at any given time. The two events cover the complete
sample space. Since these are perfect opposites of each other, we can also call them complementary events.
Following are some important points to keep in mind about framing null and alternate hypotheses:
•The null and alternate hypotheses are perfect opposites of each other. Hence, they should cover the
entire range of possibilities.
•The null hypothesis always has the following signs: ‘=’ OR ‘≤’ OR ‘≥’.
•The alternate hypothesis always has the following signs: ‘≠’ OR ‘>’ OR ‘<’.
Note: The null hypothesis always includes an equal sign
So, whenever a new claim is made, the null hypothesis states that the claim is not true, and the
alternate hypothesis states that the claim is true. Hence, for the claim to be declared true, evidence
will have to be gathered to prove its statistical validity. In other words, the strategy is to frame the
alternate hypothesis as the hypothesis that you are trying to prove.
we always begin with the assumption that the null hypothesis is true. Then:
•If we have sufficient evidence to prove that the null hypothesis is false, we ‘reject’ it. In this case, the
alternate hypothesis is proved to be true.
•If we do not have sufficient evidence to prove that the null hypothesis is false, we ‘fail to reject’ it. In this
case, the assumption that the null hypothesis is true remains.
H0: all students are girls
We don’t have enough evidence to reject null hypotheses
you understood that we never prove the null hypothesis; we can only say that we ‘fail to reject’ the null hypothesis
based on the evidence that we have gathered.
If the evidence supports the claim, we accept the alternate hypothesis, i.e., we reject the null hypothesis. If the
evidence does not support the claim, we fail to reject the null hypothesis.
Formulating the Hypotheses
Suppose you are the COO of a shoe-manufacturing company. An employee has developed a new
sole and claims that incorporating it will decrease the wear of the shoe after three years of use
by more than 9%. Now, you want to test this claim.
What will be the null and alternative hypotheses for the sole developed by the employee in this
scenario?
Ho: Decrease in wear after 3 years ≤ 9%; Ha: Decrease in wear after 3 years > 9%
Ho: μ ≤ 35 minutes; Ha: μ > 35 minutes
Ho: μ > 35 minutes; Ha: μ ≤ 35 minutes
Ho: μ ≥ 35 minutes; Ha: μ < 35 minutes
Ho: μ < 35 minutes; Ha: μ ≥ 35 minutes
It is believed that the average commute time for an employee to and from their office in Toronto is
at least 35 minutes. Now, suppose you want to test this claim.
What will be the null and alternative hypotheses, considering the average commute time is
represented by μ?
Goodyear has launched a new tyre, which, it claims, can travel more than 7,500 miles
before it needs to be replaced.
Assuming that the ‘average distance travelled before replacement’ is given by μ, what
would be the null and alternative hypotheses in this case?
Ho: μ ≤ 7500 miles; Ha: μ ≠ 7500 miles
Ho: μ ≤ 7500 miles; Ha: μ > 7500 miles
Ho: μ = 7500 miles; Ha: μ ≠ 7500 miles
Ho: μ ≥ 7500 miles; Ha: μ < 7500 miles
As the claim that the average distance travelled before
replacement is ‘more than 7,500 miles’ is a ‘new’ claim, we
should state it as the alternative hypothesis. Hence, the
alternative hypothesis should be stated as ‘more than 7,500
miles’, and the null hypothesis should be stated as ‘less than or
equal to 7,500 miles’.
Null hypothesis
The null hypothesis is typically denoted as H0. This hypothesis is assumed to be
true until there is evidence to suggest otherwise .this is actually initial hypothesis.
Alternative hypothesis
The alternative hypothesis is typically denoted as Ha or H1. This is the statement
that one wants to conclude. It is also called the research hypothesis.
The goal of hypothesis testing is to see if there is enough evidence against the null
hypothesis. In other words, to see if there is enough evidence to reject the null
hypothesis. If there is not enough evidence, then we fail to reject the null hypothesis.
So, we have two options:
1)Reject the null hypothesis
2)Fail to reject the null hypothesis(not having enough evidence to reject null hypothesis)
So important
Example
A man, Mr. Orangejuice, goes to trial and is tried for the murder of his ex-wife. He is either
guilty or innocent. Set up the null and alternative hypotheses for this example.
Answer
Putting this in a hypothesis testing framework, the hypotheses being tested are:
1.The man is guilty
2.The man is innocent
Let's set up the null and alternative hypotheses.
H0: Mr. Orangejuice is innocent
Ha: Mr. Orangejuice is guilty
Remember that we assume the null hypothesis is true and try to see if we have evidence
against the null. Therefore, it makes sense in this example to assume the man is innocent
and test to see if there is evidence that he is guilty.
A common format for a hypothesis test is:
❖ Ho→ A statement of the null hypothesis, e.g.
population mean 𝜇=75 or
two population means are equal (𝜇1 = 𝜇2 or 𝜇1 − 𝜇2 = 0)
or data is normally distributed.
❖ Ha→ A statement of the alternative hypothesis, e.g.,
population mean 𝜇 ≠ 75 ,
two population means are not equal (𝜇1 ≠ 𝜇2 or 𝜇1 − 𝜇2 ≠ 0)
data is not normally distributed
❖ Test Statistic → The test statistic is based on the specific hypothesis test.
We want to know the answer to a research question. We determine our null and alternative
hypotheses. Now it is time to make a decision.
The decision is either going to be...
1.reject the null hypothesis or...
2.fail to reject the null hypothesis( there is not enough evidence to reject the null hypothesis)
Note! Why can’t we say we “accept the null”? The reason is that we are assuming the null
hypothesis is true and trying to see if there is evidence against it. Therefore, the conclusion
should be in terms of rejecting the null.
Assume health ministry of Canada country claims that children less that 8 years old don’t get One
particular disease
Null Hypothesis: H0: Children less than 8 years old don’t get that disease
Alternative Hypothesis: Ha: Less than 8 years old can get that disease
Example : A consumer test agency wants to see the whether the mean lifetime of a brand of tires is
less than 42,000 miles. The tire manufacturer advertises that the average lifetime is at least 42,000
miles. What is the Null Hypothesis and Alternative Hypothesis?
Ha: the mean lifetime of a brand of tires is less than 42,000 miles ( 𝜇 < 42000 )
H0: the average lifetime is at least 42,000 miles (𝜇 ≥ 42000)
Consider the following table. The table shows the decision/conclusion of the hypothesis test
and the unknown "reality", or truth. We do not know if the null is true or if it is false. If the null
is false and we reject it, then we made the correct decision. If the null hypothesis is true and
we fail to reject it, then we made the correct decision.
So what happens when we do not make the correct decision?
When doing hypothesis testing, two types of mistakes may be made and we call them Type I
error and Type II error. If we reject the null hypothesis when it is true, then we made a type I
error (False Alarm)(False Positive). If the null hypothesis is false and we failed to reject it, we
made another error called a Type II error.
So important
H0: Not being pregnant
As an example the null hypothesis for any one is that he/she doesn’t have cancer.
So if some one got positive for test of cancer that means we reject null hypothesis
And get this conclusion that he/ she has cancer but there is still chance that test was incorrect.
For some one who doesn’t have actually cancer, but by mistake got positive
(False Positive) , type I error happened .
For some one who has actually cancer , but by mistake got negative (False Negative),
Type II error happened
So important
Types of errors:
Type I error
When we reject the null hypothesis when the null hypothesis is true.
Type II error
When we fail to reject the null hypothesis when the null hypothesis is false.
The “reality”, or truth, about the null hypothesis is unknown and therefore we do not know if we have made the correct decision or if
we committed an error. We can, however, define the likelihood of these events.
α ('Alpha')
The probability of committing a Type I error(rejection of the null hypothesis when the null hypothesis is true). Also known as the
significance level.(significance level=Type I error = Alpha=False positive ) (As an example predicting cancer for some one who
doesn’t have cancer or predicting spam for a non-spam email)
β ('Beta')
The probability of committing a Type II error. (False Negative) (As an example predicting not having cancer for some one who has
cancer or predicting non-spam for a spam email)
Power
Power is the probability the null hypothesis is rejected given that it is false (ie. 1−β)
α and β are probabilities of committing an error so we want these values to be low. However, we cannot decrease both.
As α decreases, β increases.
Note! Type I error is also thought of as the event that we reject the null hypothesis GIVEN the null is true. In other words, Type I error is
a conditional event and α is a conditional probability. The same idea applies to Type II error and β.
(α)
(β)
•Type- I error: (FP)This error is committed when you reject a null hypothesis that is actually true. The
probability of committing a type-I error is denoted by the significance level (α) of the test.
•Type-II error:(FN) This error is committed when you fail to reject a null hypothesis that is actually false. The
probability of committing a type-II error is denoted by β.
• Power=1−β
Effect of confidence level on type-I and type-II errors:
•Increasing the confidence level of the test decreases the size of the rejection region, thereby lowering the
chances of rejecting the null hypothesis (irrespective of whether it is false or not). Hence, increasing the
confidence level of the test (say, from 95% to 99%) decreases the probability of committing a type-I error.
•Increasing the confidence level will make it more likely that you fail to reject the null hypothesis. This is
because it is difficult to reject the null hypothesis at higher confidence levels. Thus, the probability of
committing the type-II error will increase.
• By increasing confidence level:
1) type I error decreases
2) Type II error increases
• By increasing the sample size:
1) Type II error decreases
2) Type I doesn’t change
3) Evidence is increased, and you are more likely to get the correct decision.
Let's assume the following hypothesis experiment:
H0: The average battery life of life-saving equipment is at most 1 year.
Ha: The average battery life of life-saving equipment is more than 1 year.
a)Which type of error would you like to minimize here, Type I error or Type II error?
b) Should you choose a higher confidence interval or a lower confidence interval?
a)Type I error: Type I error means we conclude that the battery life is more than 1 year so it is
more dangerous since it is life-saving equipment, but type II error means conclude it is at most
one year so you may replace its battery sooner than you should and just increase the cost not
risking the life
b) Alpha decreases ➔ 1-alpha increases ➔ confidence interval increases
A test to screen for a serious but curable disease is similar to hypothesis testing. In this
instance, the null hypothesis would be that the person does not have the disease, and
the alternate hypothesis would be that the person has the disease. If the null
hypothesis is rejected, it means that the disease is detected and treatment will be
provided to the particular patient. Otherwise, it will not. Assuming the treatment does
not have serious side effects, in this scenario, it is better to increase the probability of:
type-I error would be providing treatment upon detecting the disease, when the
person does not actually have the disease. And type-II error would be not providing
treatment upon failing to detect the disease, when the person As an example the null hypothesis for any one is that he/she doesn’t have cancer.
So if some one got positive for test of cancer that means we reject null hypothesis
And get this conclusion that he/ she has cancer but there is still chance that test was incorrect.
For some one who doesn’t have actually cancer, but by mistake got positive
(False Positive) , type I error happened .
For some one who has actually cancer , but by mistake got negative (False Negative),
Type II error happened
So important
Types of errors:
Type I error
When we reject the null hypothesis when the null hypothesis is true.
Type II error
When we fail to reject the null hypothesis when the null hypothesis is false.
The “reality”, or truth, about the null hypothesis is unknown and therefore we do not know if we have made the correct decision or if
we committed an error. We can, however, define the likelihood of these events.
α ('Alpha')
The probability of committing a Type I error(rejection of the null hypothesis when the null hypothesis is true). Also known as the
significance level.(significance level=Type I error = Alpha=False positive ) (As an example predicting cancer for some one who
doesn’t have cancer or predicting spam for a non-spam email)
β ('Beta')
The probability of committing a Type II error. (False Negative) (As an example predicting not having cancer for some one who has
cancer or predicting non-spam for a spam email)
Power
Power is the probability the null hypothesis is rejected given that it is false (ie. 1−β)
α and β are probabilities of committing an error so we want these values to be low. However, we cannot decrease both.
As α decreases, β increases.
Note! Type I error is also thought of as the event that we reject the null hypothesis GIVEN the null is true. In other words, Type I error is
a conditional event and α is a conditional probability. The same idea applies to Type II error and β.
(α)
(β)
•Type- I error: (FP)This error is committed when you reject a null hypothesis that is actually true. The
probability of committing a type-I error is denoted by the significance level (α) of the test.
•Type-II error:(FN) This error is committed when you fail to reject a null hypothesis that is actually false. The
probability of committing a type-II error is denoted by β.
• Power=1−β
Effect of confidence level on type-I and type-II errors:
•Increasing the confidence level of the test decreases the size of the rejection region, thereby lowering the
chances of rejecting the null hypothesis (irrespective of whether it is false or not). Hence, increasing the
confidence level of the test (say, from 95% to 99%) decreases the probability of committing a type-I error.
•Increasing the confidence level will make it more likely that you fail to reject the null hypothesis. This is
because it is difficult to reject the null hypothesis at higher confidence levels. Thus, the probability of
committing the type-II error will increase.
• By increasing confidence level:
1) type I error decreases
2) Type II error increases
• By increasing the sample size:
1) Type II error decreases
2) Type I doesn’t change
3) Evidence is increased, and you are more likely to get the correct decision.
Let's assume the following hypothesis experiment:
H0: The average battery life of life-saving equipment is at most 1 year.
Ha: The average battery life of life-saving equipment is more than 1 year.
a)Which type of error would you like to minimize here, Type I error or Type II error?
b) Should you choose a higher confidence interval or a lower confidence interval?
a)Type I error: Type I error means we conclude that the battery life is more than 1 year so it is
more dangerous since it is life-saving equipment, but type II error means conclude it is at most
one year so you may replace its battery sooner than you should and just increase the cost not
risking the life
b) Alpha decreases ➔ 1-alpha increases ➔ confidence interval increases
A test to screen for a serious but curable disease is similar to hypothesis testing. In this
instance, the null hypothesis would be that the person does not have the disease, and
the alternate hypothesis would be that the person has the disease. If the null
hypothesis is rejected, it means that the disease is detected and treatment will be
provided to the particular patient. Otherwise, it will not. Assuming the treatment does
not have serious side effects, in this scenario, it is better to increase the probability of:
type-I error would be providing treatment upon detecting the disease, when the
person does not actually have the disease. And type-II error would be not providing
treatment upon failing to detect the disease, when the person actually has the
disease. Since the treatment has no serious side effects, type-I error poses a lower
health risk than type-II error, as not providing treatment to a person who actually has
the disease would increase his/her health risk
As you can see here, the Type I error (putting an innocent man in jail) is the more serious error. Ethically, it is more serious to
put an innocent man in jail than to let a guilty man go free. So to minimize the probability of a type I error we would choose a
smaller significance level.
Threshold=12
Assume you conduct binary classification for prediction of specific disease that means at the end of project
you want to predict for new person that he/she has that disease or not so having disease is positive class
and not having that disease is negative class based on all features you calculate y, then based on the value
of y you predict. Assume you consider y=12 as a threshold that means for people that have y less than 12
you predict negative and for people with value of y bigger than 12 you predict positive
By changing threshold, you can decrease one of them and increase the other one,
As an example, in spam detection project since FP is more dangerous, we may increase threshold
If we change threshold from 0.5 to 0.70 FP will be decrease but FN will be increase
In breast cancer prediction since FN is more dangerous , we may decrease threshold
If we decrease threshold FN will be decrease but FP will be incease
(𝛼)
(𝛽)
In Data mining course, you will learn confusion matrix and definition of accuracy , recall, Precision
and f measure.
For test of Normality: always
Null Hypotheses : Data is normally distributed
So, if p-value<0.05 ➔ we reject null hypotheses and conclude data is not normally distributed at 5%
significant level
For any test of independency( Chi-square, t-test, Anova, …):
Null Hypotheses: Those 2 variables are independent
So, if p-value<0.05 ➔ we reject null hypotheses and conclude that there is statistically association
between those 2 variables at 5% significant level
Hypothesis Testing
for one sample
• Hypothesis Test for One-Sample Mean
• Hypothesis Test for One-Sample Proportion
Let’s back to stat
Example: A political news company believes the national approval rating for the current
president has fallen below 40%. What is null hypotheses and what is alternate hypotheses?
Answer:
The response variable here is 'approval rating' and is a qualitative variable. Therefore, we
will set up the hypothesis in terms of p. In this case, the p0 value is 0.4 and the hypotheses
would be set up as a left-tailed test:
Ha: p<0.4 ➔H0:p=0.4 (or H0 :p≥ 0 ⋅ 4 )
40
Let’s back to stat
Example: A political news company believes the national approval rating for the current
president has fallen above 60%. What is null hypotheses and what is alternate hypotheses?
Answer:
Ha: p>0.60 ➔H0:p=0.6 (or H0 :p≤ 0 ⋅ 6 )
Right-tailed test
6Let’s back to stat
Example: A political news company believes the national approval rating for the current
president is not 60% any more . What is null hypotheses and what is alternate hypotheses?
Answer:
Ha: p ≠0.60 ➔H0:p=0.60
two-tailed test
Recall that for unknown population standard deviation if we have some conditions, we use t distribution
Hint: in two-tailed test for unknown population standard deviation we would reject the H0 in favor of Ha
if the t-value is less than -𝑡𝛼
2
or greater than 𝑡𝛼
2
We can find 𝑡𝛼
2
(we call it 𝑡𝑐𝑟𝑖𝑡𝑖𝑐𝑎𝑙) 𝑏𝑎𝑠𝑒𝑑 𝑜𝑛 𝛼 by using t-table or online calculator for t,
and based on sample mean we can find t-value (we call it 𝑡𝑠𝑡𝑎𝑡) by using following formula:
𝑡𝑠𝑡𝑎𝑡 =
𝑥ҧ− 𝜇
𝑠
𝑛
For whole of sample we calculate
𝑡𝑠𝑡𝑎𝑡 based on sample mean and sample standard deviation and sample size
Note: before using this formula make sure
1.the population is normal or
2.you have enough observation ( i.e. sample size =n ≥30) to rely on CLT(Central Limit Theorem)
Note: if you have population standard deviation you can use Z-formula
Comparing Two Population Means
Comparing Two Population Proportions
Chi-Square Test of Independence
Correlation
ANOVA
So far in our course, we have only discussed measurements taken in one variable for each sampling unit and comparing them with population or estimating for population . This is referred to as univariate analysis. In this lesson, we are going to talk about measurements taken in two variables for each sampling unit. This is referred to as bivariate analysis or bivariate data
Examples:
Sex and whether they smoke
GPA and the current degree level of a student
Dieting and the participant's weight before and after
Find out the relationship(association or correlation) between two variables
Look for association between variables at predefined significance level
Perform bi-variates analysis for any combination of categorical and continuous variables :
Continuous & Continuous
Categorical & Categorical
Categorical & Continuous
Different methods are used to describe these combinations
Continuous Vs. Continuous What is correlation?
Strength of association between two numerical variables.
Pearson correlation shows you the strength of linear relationship
A zero correlation does not imply that there is no relation between variables; it merely indicates that there will no linear relationship between them. Also, there can be a negative or positive correlation between variables. A negative correlation means that if the value of one variable increases, the value of another decreases, whereas it is the opposite for a positive correlation.
Now, the higher the coefficient of correlation between numerical variables, the higher the linear relation between them.
The pairs of scores can be listed in a table or presented in a scatterplot. Scatterplots can also tell us about the form (shape) of the relationship.
The correlation measure tells us about the direction of the relationship between the two variables. The direction can be positive or negative.
Positive: In a positive relationship both variables tend to move in the same direction: If one variable increases, the other tends to also increase. If one decreases, the other tends to also.
Negative: In a negative relationship the variables tend to move in the opposite directions: If one variable increases, the other tends to decrease, and vice-versa.
The direction of the relationship between two variables is identified by the sign of the correlation coefficient for the variables. Positive relationships have a "plus" sign, whereas negative relationships have a "minus" sign.
Sign of correlation is the same as sign of slope of best fit line but magnitude of correlation in not related to value of slop((𝑦_2−𝑦_1)/(𝑥_2−𝑥_1 ) ) ( 𝑠𝑙𝑜𝑝𝑒 𝑐𝑎𝑛 𝑏𝑒 𝑓𝑟𝑜𝑚 𝑎𝑛𝑦 𝑛𝑢𝑚𝑏𝑒𝑟 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 −∞ to +∞ but correaltion is between −1 to+1)
The Correlation coefficient r is in between -1 and +1.
+1 is the largest positive correlation when all points fall in exact one line with positive slope and -1 is the largest negative correlation that is possible when all points are located exactly on one line with negative slope
0.5 ≤ |r| High linear correlation
0.3 ≤ |r| < 0.5 Medium linear correlation
0 ≤ |r| < 0.3 Low linear correlation
r=0no linear correlation
r= +1 or -1perfect linear relationship
Strong Positive Correlation:
Description: Points on the scatter plot form a tight, upward-sloping cluster.
Weak Positive Correlation:
Description: Points on the scatter plot show a general upward trend but are more scattered.
No Correlation (Zero Correlation):
Description: Points on the scatter plot are scattered with no clear trend.
Remember that correlation strength is not only about the tightness of the pattern but also about its consistency.
Two specific strengths are:
Perfect Relationship: When two variables are exactly (linearly) related the correlation coefficient is either +1 or -1. They are said to be perfectly linearly related, either positively or negatively.
No relationship: When two variables have no linear relationship at all, their correlation is 0.
From the correlation matrix below, you can observe that petal length has a high correlation with sepal length, with a correlation coefficient of 0.87. Also, there is a very high correlation coefficient of 0.96 between petal width and petal length.
Note: The value of the correlation between Petal Length and Sepal Width is -0.43 and not 0.43 at both the places of the below correlation matrix.
However, the correlation matrix has its own limitations where you cannot see the exact distribution of a variable with another numeric variable. To solve this problem, we use pair plots. Pair plots are scatter plots of all numeric variables in a data set. It shows the exact variation of one variable with respect to the others. You can observe how one variable is varying with respect to another in the image below.
correlation is not causation
Correlation is a statistical technique which tells us how strongly the pair of variables are linearly related and change together. It does not tell us why and how behind the relationship but it just says the relationship exists. Causation takes a step further than correlation. It says any change in the value of one variable will cause a change in the value of another variable, which means one variable makes other to happen. It is also referred as cause and effect. Just after finding correlation, don’t draw the conclusion too quickly. Take time to find other underlying factors as correlation is just the first step. Find the hidden factors, verify if they are correct and then conclude. https://medium.com/@seema.singh/why-correlation-does-not-imply-causation-5b99790df07e Ice cream sales is correlated with homicides in New York (Study)
In some cases there are some hidden factors which are related on some level. Like in our example of ice cream sales and homicide rates , weather is the hidden factor which is causing both the things.Weather is actually causing the rise in ice cream sales and homicides. As in summer people usually go out, enjoy nice sunny day and chill themselves with ice creams. So when it’s sunny, wide range of people are outside and there is a wider selection of victims for predators.
How do we confirm causation between the variables?
By using experimental study or observational study
To calculate the Pearson product-moment correlation, one must first determine the covariance of the two variables in question. Next, one must calculate each variable's standard deviation. The Pearson correlation coefficient is determined by dividing the covariance by the product of the two variables' standard deviations.
A correlation coefficient measures the degree (strength) of the relationship between two numerical variables. The Pearson Correlation Coefficient measures the strength of the linear relationship between samples of two variables. The formula for Pearson Correlation Coefficient of variable X Y is:
There are several correlation coefficients.
Pearson Correlation Coefficient – is sensitive only to a linear relationship between two variables (which may exist even if one is a nonlinear function of the other)
Rank Correlation Coefficients – e.g., Spearman’s Rank Correlation Coefficient and Kendall Tau Rank Correlation Coefficient. These are the alternatives to Pearson’s coefficient, used either to reduce the amount of calculation or to make the coefficient less sensitive to non-normality in distribution.
The Pearson correlation technique works best with linear relationships: as one variable gets larger, the other gets larger (or smaller) in direct proportion. It does not work well with curvilinear relationships (in which the relationship does not follow a straight line). An example of a curvilinear relationship is age and health care. They are related, but the relationship doesn't follow a straight line.
H.R:
Sometimes the relationship is not linear, but x and y follows same direction or opposite direction
For example, in following data if x increases , y also increases although it is not linear relationship.
In this situation Rank correlation is better metric.
Rank correlation coefficients, (such as Spearman’s rank correlation) measure the extent to which as one variable increases, the other variable tends to increase, without requiring that increase to be represented by a linear relationship. If as the one variable increases, the other decreases, the rank correlation coefficients will be negative.
The Spearman’s rank correlation coefficient
is appropriate for both continuous and discrete variables including ordinal variables.
The rank correlation coefficient between 2 variables will be high when observations have a similar rank (or identical for a correlation of 1) between the two variables and low when observations have a dissimilar rank (or fully opposed for a correlation of -1) between the two variables.
That means if we want to predict PetalLength only based on SepalLength you can use following formula( this is SLR(simple linear regression))
That means we can only explain 76% of the variation in PetalLength by SepalLength That means there are some more features that can explain 24% of the variation in PetalLength but we don’t have them in our model
Since I used more than one predictor this is called MLR (Multiple Linear Regression)
R-Square is between 0 and 1, the closer to 1 ,the better model but for MLR you should better compare Adjusted R square
Chi-Square Test for Independence
How do we test the independence of two categorical variables? It will be done using the Chi-square test
of independence.
As with all prior statistical tests we need to define null and alternative hypotheses. Also, as we have
learned, the null hypothesis is what is assumed to be true until we have evidence to go against it. In this
lesson, we are interested in researching if two categorical variables are related or associated (i.e.,
dependent). Therefore, until we have evidence to suggest that they are, we must assume that they are
not. This is the motivation behind the hypothesis for the Chi-square Test of Independence:
•H0: In the population, the two categorical variables are independent.
•Ha: In the population, the two categorical variables are dependent.
Note! The are several ways to phrase these hypotheses. Instead of using the words "independent" and
"dependent" one could say "there is no relationship between the two categorical variables" versus "there
is a relationship between the two categorical variables." Or "there is no association between the two
categorical variables" versus "there is an association between the two variables." The important part is
that the null hypothesis refers to the two categorical variables not being related while the alternative is
trying to show that they are related.
Note! keep in mind that this Chi-square test, when significant, only provides statistical evidence of
an association or relationship between the two categorical variables. Do NOT confuse this result
with correlation which refers to a linear relationship between two quantitative variables
The primary method for displaying the summarization of categorical variables is called a contingency table. When we
have two measurements on our subjects that are both the categorical, the contingency table is sometimes referred to
as a two-way table.
How do we test the independence of two categorical variables? It will be done using the Chi-square test of
independence.
Once we have gathered our data, we summarize the data in the two-way contingency table. This table represents the
observed counts and is called the Observed Counts Table or simply the Observed Table.
Then calculate expected count for each cell under the null hypothesis.
Recall that if event A and event B are independent ➔P(A and B)=P(A) * P(B)
So we can calculate expected value by assuming they are independent
Under the null hypothesis and certain conditions (discussed below), the test statistic follows a Chi-square
distribution with degrees of freedom equal to (r-1)(c-1), where r is the number of levels for variable that
mentioned on rows and c is the number of levels for variable that mentioned on columns. We leave out
the mathematical details to show why this test statistic is used and why it follows a Chi-square distribution.
Note! We do not expect you to calculate the critical value or the p-value by hand. The p-value can be found
using software
Conditions for the Validity of Chi-Square Test:
The Chi-square test statistic can be used if the following conditions are
satisfied:(if you do not met the condition you can use fisher exact test)
1. N, the total frequency, should be reasonably large, say greater than 50.
2. The sample observations should be independent. This implies that no individual item
should be included twice or more in the sample.
3. No expected frequencies should be small. Small is a relative term. Preferably each
expected frequencies should be larger than 10 but in any case not less than 5.
Some statisticians hesitate to use the chi-square test if more than 20% of the cells have
expected frequencies below five, especially if the p-value is small and these cells give a large
contribution to the total chi-square value.
Are the results of my chi-square test of association invalid?
•If either variable has only 2 or 3 levels, then you can trust the results if either of the following is true: All cells have expected counts of at least 3.
•All cells have expected counts of at least 2, and 50% or fewer of the cells have expected counts of less than 5.
•If both variables have 4 to 6 levels, then you can trust the results if either of the following is true:All cells have expected counts of at least 2.
•All cells have expected counts of at least 1, and 50% or fewer of the cells have expected counts of less than 5.
Categorical & Categorical
(for example we want to find out is there any association between
canceling method( by phone or online) and churn(yes or no)(yes means
that customer canceled his/her contract)
Null hypothesis: there is no association between those two categorical
variables
• To find the relationship between two categorical variables, use
following methods:
• Two-way table: create two-way table of count and % count.
• The rows represent the category of one variables and the column
represent the category of the other variable.
• Show count or % count of observations available in each
combination of row and column categories.
Chi-Square Test : to derive statistical significance of relationship between two categorical variables.
• Also tests how strong the evidence in the sample to generalize that relationship for a larger population.
• Chi-square is based on the difference between the expected and observed frequencies in one or more
categories in the two-way table.
✓ Probability of 0:both categorical variable are dependent
✓ Probability of 1:both variables are independent
✓ Null Hypothesis : there is no association between those two categorical variables
If all assumption are met and p-value<0.05 ➔ you can reject null hypothesis at 5% significant level and
say there is association between those two columns
• use Chisq as an option with PROC FREQ to perform this test
Hint: Before interpreting of result of any test, you must be sure that all assumption for
that test are met.
Assumption of Chi-square test:
Tables Larger than 2x2
•Each observation is independent of all the others (i.e., one observation per subject);
(I mean you can’t have observation that counted in more than one levels as an example in
heart data set if we assume Moderate (6-20) and heavy(16 and more) you can’t rely on the
result of chi-square because if some one smokes 18 cigarettes he/she belongs to two groups)
•"No more than 20% of the expected counts are less than 5 and all individual expected counts
are 1 or greater" (Yates, Moore & McCabe, 1999, p. 734)
2x2 Tables: The Standard Advice
The standard advice for 2x2 tables dates back to Cochran (1952, 1954) or earlier,
and goes something like this:
Each observation is independent of all the others (i.e., one observation per subject)*
•All expected counts should be 10 or greater.
•If any expected counts are less than 10, but greater than or equal to 5, some
authors suggest that Yates' Correction for continuity should be applied. This is done
by subtracting 0.5 from the absolute value of O-E before squaring. However, the
use of Yates' correction is controversial, and is not recommended by all authors.
•If any expected counts are less than 5, then some other test should be used (e.g.,
Fisher exact test for 2x2 contingency tables)**
Imagine you have one categorical variable: like sex (male and female) and one continuous variable (weight)
Bivariate analysis for this scenario:ttest, here you are going to check:
H0: the mean of wight is equal between male and female
Ha: the mean of weight is not equal between male and female
Checking assumption or condition: normality 2. equality or homogeneity of variance: it means check the equality
Of wight variance between male and female
Continues variable: cholesterol (Anova) (analysis of variance)
Categorical variable: smoking (5 level)
H0: the mean of cholesterol is equal is the same in different categories of smoking
Ha: the mean of cholesterol is not equal or it is not same in different categories of smoking
Imagine you have one categorical variable: like sex (male and female) and one continuous variable (weight)
Bivariate analysis for this scenario:ttest, here you are going to check:
H0: the mean of wight is equal between male and female
Ha: the mean of weight is not equal between male and female
Checking assumption or condition: normality 2. equality or homogeneity of variance: it means check the equality
Of wight variance between male and female
Continues variable: cholesterol (Anova) (analysis of variance)
Categorical variable: smoking (5 level)
H0: the mean of cholesterol is equal is the same in different categories of smoking
Ha: the mean of cholesterol is not equal or it is not same in different categories of smoking
Exploratory Data Analysis (EDA):
Grouped Box Plots:
Draw box plots for each level of categorical variables.
Visualize the distribution of the continuous variable across different categories.
Check Assumptions:
Ensure the assumptions of the statistical tests are met (e.g., normality, homogeneity or equality of variances: equal variance labeled: pooled, d the other under the assumption of unequal variance (labeled Satterthwaite ).
Statistical Analysis:
Statistical Significance:
To assess statistical significance, perform z-test/t-test or ANOVA.
Determine the appropriate test based on the number of levels in the categorical variable.
For Categorical Variable with Two Levels (Binary):
Run a t-test to assess whether the means of the two groups are statistically different.
For Categorical Variable with More than Two Levels:
Run ANOVA (Analysis of Variance) to assess overall differences in means among the multiple groups.
Z-Test/T-Test:
1. Purpose:
Assess whether the mean of two groups are statistically different from each other.
2. Selection Criteria:
Choose between z-test and t-test based on sample size and whether the population standard deviation is known.
Z-Test for larger sample sizes with known population standard deviation.
T-Test for smaller sample sizes or when the population standard deviation is unknown.
In the output of t test you can see 4 table: the first one is descriptive statistics of continuous variable between different
Category of categorical variable, in that table you can see the difference of mean between two levels.
Move on to the forth table, it is equality of variance: check the p value if it is less than 0.05 it means you should reject
Null hypothesis, in other words the variance of bmi is not equal between stroke 0 and stroke 1
So you will move to table number three and report the p value for satterwaite test, it is less than 0.05 so it means the mean of Bmi is not equal between two levels of stroke so the difference of -1.77 is statistically significant.
Assumption: The independent two-sample t-test assumes variance of the data in both groups should be approximately equal. It means that the spread of data points around the mean should be similar for both groups.
Equality of Variances table - Since the p-value of the Equality of Variances test (0.9527) is greater than the significance level (e.g., 0.05), then we assume that the variances are equal. In this case we will use Pooled method. If the p-value had been less than 0.05, we would have used the Satterthwaite method.
ANOVA :Analysis of Variance or ANOVA is a statistical method that analyzes
variances to determine if the means from more than two populations are the
same. In other words, we have a quantitative response variable (y) and a categorical
explanatory variable (x) with more than two levels. (continuous dependent and categorical is independent
Use SAS PROC ANOVA
Lets assume you are a data scientist who work in data science department of Walmart
Scenario #1: you want to find is there any associate between amount of order and method of
buying if you offer two methods (online and instore) so you should run t-test because you want
to find relationship between categorical variable( method) and continuous variable (order
amount) and that categorical variables have only two levels
Scenario #2: you want to find is there any associate between amount of order and province so
you should run ANOVA because you want to find relationship between categorical variable(
Province) and continuous variable (order amount) and that categorical variables have more
than two levels
Comparing Two Population Parameters
taken from two distinct groups
1) Categorical - taken from two distinct groups
If the measurements are categorical and taken from two distinct groups, the analysis will involve
comparing two independent proportions
2)Quantitative - taken from two distinct groups
If the measurements are quantitative and taken from two distinct groups, the analysis will involve
comparing two independent means.
taken twice from same subject (paired)
3)Quantitative - taken twice from each subject (paired)
If the measurements are quantitative and taken twice from each subject, the analysis will involve
comparing two dependent means
4)Categorical - taken twice from each subject (paired)
If the measurements are categorical and taken twice from each subject, the analysis will involve
comparing two dependent proportions.
Outcome or y or dependent or response variable= weight (continuous variable)
Explanatory or x or independent variable= diet (categorical variable with two level)
To begin, one has to first decide whether the problem you are investigating requires the analysis of
categorical or quantitative data. In other words, you need to identify your response variable and
determine the type of variable. Next, one has to determine if the two measurements are from
independent samples or dependent samples.
Examples:
1)Sex and whether they smoke
Consider a case where we measure sex and whether they smoke. In this case, the response variable
(smoking status) is categorical and taken from two distinct groups, and the explanatory variable (sex) is
also categorical.
•Response variable(target): Yes or No to the Question “Do you smoke?”
•Explanatory variable: Sex (Female or Male)
2)GPA and the current degree level of a student(graduate or undergraduate)
In this case, the response variable (GPA) is Quantitative and taken from two distinct groups
, and the explanatory variable(degree level) is categorical.
• Response variable: GPA
• Explanatory variable: Graduate or Undergraduate
3)Dieting and the participant's weight before and after
In this case, the response (weight) is quantitative and taken twice from each subject (paired)
• Response variable: Weight
• Explanatory variable: Diet
In this situation we run paired t-test test because categorical variables has only two levels
The decision process will be the same as well:
Method #1: if the test statistic falls in the rejection region, we will reject the null hypothesis;
Method #2: if the p-value is less than the level of significance, we will reject the null hypothesis.
Method #3: The interpretation of confidence intervals in support of the hypothesis decision will
also be familiar:
• if the interval does not contain the null hypothesis value, then we will reject the null hypothesis;
• if the interval contains the null hypothesis value, then we will fail to reject the null hypothesis.
when comparing two proportions or two means, by default we will use a null value of 0 (i.e., "no
difference“ i.e. there is no association between those two variables).
For example, 𝜇1 - 𝜇2 = 0 would mean that 𝜇1 = 𝜇2, and there would be no difference between
the two population parameters. Similarly, for two population proportions.
Although we focus on the difference equaling zero, it is possible to test for specific values of the
difference using the methods presented. However, most applications research only for a difference
in the parameters (i.e., the difference is less than, greater than, or not equal to zero).
Comparing Two Population mean
(CATEGORICAL VS. CONTINUOUS)
For comparing equality or difference between the mean of two independent sample we use t-test ( for only two levels) Anova ( for more than two levels), now suppose we have two
Independent samples and the standard deviation or sigma for population is unknown and sample size is small and we can not rely on CLT, we want to know can we use the pooled variance method for estimating of standard deviation for population or we should use unpooled (or unequal) variances method in that case we should run leven’s or bartlet’s test, as leven’s is less sensitive to departures from normality we can use it in most cases.
PARAMETRIC ASSUMPTIONS
Parametric tests have the same assumptions, or conditions, that need to be met in
order for the analysis to be considered reliable.
Parametric test assumptions
1.Independence
2.Population distributions are normal(normality)
3.Samples have equal variances(HOMOGENEITY OF VARIANCES)
It is best to check the assumptions in the order above since some equal variance
tests are sensitive to the distribution being normal
INDEPENDENCE
This assumption is checked during the setup of the study. It means that each
observation is independent of another; if there are 2 or more groups being compared,
then it refers to that fact that groups are mutually exclusive, i.e. each individual
belongs to only 1 group; and that the data is not repeated over time.
If data is repeated measures, longitudinal/panel or time series, there are appropriate
methods (like paired t-test) to account for the repeated data points from individuals.
This is typically handled in the math of the statistical test that is designed to analyze
this type of data
NORMALITY
The normality assumption is applied differently depending on the statistical method being used. For example, it
applies to the shape of the sampling distribution for the dependent variable (outcome variable) if it's a
univariate test, to the difference scores if it's mean comparison (ex. independent sample t-test or paired t-test),
or to the residuals if it's a regression framework. This assumption can be checked with a formal test or
graphically. There are a few formal tests that can be used to check this assumption.
Before that gets touched, the Central Limit Theorem needs to be discussed. Ott and Longnecker (2010, p.188)
provide this overview about the Central Limit Theorem:
Note:
1.For parried t-test you should check normality for differences of each EU( experimental units)/ subject
2. For independent t-test and ANOVA you should check normality for each groups(But if you have many
groups (a 2x2x3 ANOVA has 12 groups) or if there are few observations per group (it’s hard to check normality
on only 20 data points), it’s often easier to just use the residuals and check them all together.
3. In regression you should check normality for residual
No specific shape was required for these measurements for the Central Limit Theorem to be
validated."
Great, what does it mean though? It means that if the sample size is large enough then normality
may not be a concern even if the test for normality indicates that normality is not present.
Simulation studies have been conducted and the general rule of thumb is n ≥ 30. However, if it's
clear that there is heavy skewness or kurtosis, one might consider transforming the data and/or
using a non-parametric statistical test.
There are a number of techniques that you can check if your data sample is Gaussian or
sufficiently Gaussian-like to use the standard techniques, or sufficiently non-Gaussian to instead
use non-parametric statistical methods.
This is a key decision point when it comes to choosing statistical methods for your data sample.
We can summarize this decision as follows
NORMALITY TESTS
There are many out there to choose from. These include 1.Shapiro-Wilk test, 2.Kolmogorov-Smirnov (K-S)
test, Lilliefors corrected K-S test, Anderson-Darling test, D'Agostino skewness test, D'AgostinoPearson omnibus test, and the Jarque-Bera test. This section will not cover all of these, but will
demonstrate common normality tests. Additionally, normality tests should be accompanied by a visual
test as well.
For Visualization you can use histogram (A sample of data has a Gaussian distribution of the histogram
plot, showing the familiar bell shape)or you can use q-q plot.
Quantile-Quantile Plot
Another popular plot for checking the distribution of a data sample is the quantile-quantile plot, Q-Q
plot, or QQ plot for short.
This plot generates its own sample of the idealized distribution that we are comparing with, in this case
the Gaussian distribution. The idealized samples are divided into groups , called quantiles. Each data
point in the sample is paired with a similar member from the idealized distribution at the same
cumulative distribution.
The resulting points are plotted as a scatter plot with the idealized value on the x-axis and the data
sample on the y-axis.
Shapiro-Wilk Test
The Shapiro-Wilk test evaluates a data sample and quantifies how likely it is that the data was
drawn from a Gaussian distribution, named for Samuel Shapiro and Martin Wilk.
In practice, the Shapiro-Wilk test is believed to be a reliable test of normality, although there is
some suggestion that the test may be suitable for smaller samples of data, e.g. thousands of
observations or fewer.
Tests whether a data sample has a Gaussian distribution.
Assumptions
•Observations in each sample are independent and identically distributed (iid).
Interpretation
•H0: the sample has a Gaussian distribution.
•H1: the sample does not have a Gaussian distribution. The Shapiro-Wilk test
So if p-value is less than 5% you can reject normality , and
If p-value is bigger than 5% normality assumption is met
D’Agostino’s K^2 Test
The D’Agostino’s K^2 test calculates summary statistics from the data, namely
kurtosis and skewness, to determine if the data distribution departs from the normal
distribution, named for Ralph D’Agostino.
•Skew is a quantification of how much a distribution is pushed left or right, a measure
of asymmetry in the distribution.
•Kurtosis quantifies how much of the distribution is in the tail. It is a simple and
commonly used statistical test for normality.
Assumptions
•Observations in each sample are independent and identically distributed (iid).
Interpretation
•H0: the sample has a Gaussian distribution.
•H1: the sample does not have a Gaussian distribution
Anderson-Darling Test
Anderson-Darling Test is a statistical test that can be used to evaluate whether a
data sample comes from one of among many known data samples, named for
Theodore Anderson and Donald Darling.
It can be used to check whether a data sample is normal. The test is a modified
version of a more sophisticated nonparametric goodness-of-fit statistical test called
the Kolmogorov-Smirnov test.
A feature of the Anderson-Darling test is that it returns a list of critical values rather
than a single p-value. This can provide the basis for a more thorough interpretation
of the result.
.
Assumptions
•Observations in each sample are independent and identically distributed (iid).
Interpretation
•H0: the sample has a Gaussian distribution.
•H1: the sample does not have a Gaussian distribution
As with testing the assumption of normality, there are a few statistical tests available to test the
assumption of equal variances. Some common methods are the Barlett test and Levene's test for
equality of variances. Choosing the correct test is also dependent on the assumption of normality. For
example, Barlett's test has been found to be sensitive to departures from normality whereas Levene's
test is less sensitive to this (Conover, 1981).
It doesn't matter if one is coming from a 2 sample approach or a regression/ANOVA approach, the
test steps are the same. One needs to test for equality of variance for each categorical variable on the
outcome.
HOMOGENEITY OF VARIANCES
WHAT IF HOMOGENEITY OF VARIANCES IS VIOLATED?
If there is not equal variances between groups there are a
few routes to consider, these are in no particular order:
•Use a non-parametric statistical test
•Transform the data
Common data transformation methods will be provided in
the next slides
Binning / discretization:
The process of transforming continuous variables into a smaller
number of “bins”.
➢ An example:
➢ Age grouped into categories such as 0-19,20-39, 40-59 and 60+.
➢ Numerical variables are usually discretized in the modeling methods
based on frequency tables
There are two common binning methods:
➢ Unsupervised – divide numeric variables into groups but do not use the target (class) information;
such as, Equal width, Equal frequency, etc.
➢ Supervised – divide numeric variables into groups and refer to the target (class) information when
selecting binning cut points; such as, Entropy based, Weight of evidence, etc
Unsupervised binning methods:
➢ Equal width (distance) binning
✓ It divides the range into N intervals of equal size (uniform grid).
✓ The width of intervals will be W = (Max. – Min.)/N.
✓ This method is simple and straightforward
✓ Outliers and skewed data are not handled well.
Equal frequency binning
✓ It divides the data into N groups which each group contains
approximately same number of values.
✓ This approach has good data scaling.
For these both methods, the best way of determining N is by looking at
the histogram and try different intervals or groups.
Supervised binning methods:
➢ Entropy based method – Entropy is also called expected
information. It is calculated based on the target (class). It finds the
best split so that the bins are as pure as possible that is the majority
of the values in a bin correspond to have the same target (class). It is
characterized by finding the split with the maximal information gain.
➢ Weight of evidence – it will be described in detail later.
Important Point
1.The WOE should be monotonic i.e. either growing or decreasing with the bins. You can
plot WOE values and check linearity on the graph.
2. Information value increases as bins / groups increases for an independent
variable. If WOE is monotonic with both smaller and larger bins / groups, we should
choose larger bins for an independent variable.
评论
发表评论