Calculate and interpret measures of centre, spread, frequency, and uncertainty; distinguish standard deviation from standard error; and summarize data with honest denominators.
You can download iris data from here or use the inbuilt data set as discussed in chapter Section 3.7.
To do any analysis we first import data, see Chapter Chapter 3. You can inspect your data using the functions head() and tail(), which will display the first and the last part of the data, respectively.
# import data set in to R# here we are using iris datasetmy_data<-iris # iris dataset is called in to variable my_data# display first 6 rowshead(my_data, 6)
# Compute the median valuemedian(iris$Sepal.Length)
[1] 5.8
# Compute the mode# install.packages("modeest")# The optional modeest::mfv() has a base-R equivalent for this lesson.mfv <-function(x) { x <- x[!is.na(x)]if (!length(x)) return(x) values <-unique(x) counts <-tabulate(match(x, values)) values[counts ==max(counts)]}mfv(iris$Sepal.Length)
# Range# minimum value and maximum value is displayedrange(iris$Sepal.Length)
[1] 4.3 7.9
# Inter quartile range IQR(iris$Sepal.Length)
[1] 1.3
# Compute the variancevar(iris$Sepal.Length)
[1] 0.6856935
# Compute the standard deviation sd(iris$Sepal.Length)
[1] 0.8280661
# Compute the median absolute deviationmad(iris$Sepal.Length)
[1] 1.03782
7.5 Overall summary
The overall summary statistics of One variable or an entire data frame can be displayed using the function summary().
7.5.1 Summary of single variable
The mean, median, 25th and 75th quartiles, minimum and maximum values are all returned in a single line call.
summary(iris$Sepal.Length)
Min. 1st Qu. Median Mean 3rd Qu. Max.
4.300 5.100 5.800 5.843 6.400 7.900
7.5.2 Summary of a Data frame
In this instance, each column receives an automated use of the summary() method. The kind of data in the column determines how the output is formatted. For instance:
The mean, median, min, max, and quartiles are returned if the column is a numeric variable.
The total number of observations in each group is returned if the column is a factor variable.
summary(iris)
Sepal.Length Sepal.Width Petal.Length Petal.Width
Min. :4.300 Min. :2.000 Min. :1.000 Min. :0.100
1st Qu.:5.100 1st Qu.:2.800 1st Qu.:1.600 1st Qu.:0.300
Median :5.800 Median :3.000 Median :4.350 Median :1.300
Mean :5.843 Mean :3.057 Mean :3.758 Mean :1.199
3rd Qu.:6.400 3rd Qu.:3.300 3rd Qu.:5.100 3rd Qu.:1.800
Max. :7.900 Max. :4.400 Max. :6.900 Max. :2.500
Species
setosa :50
versicolor:50
virginica :50
7.6 The sapply function
The sapply() method may also be used to apply a specific function on a list or vector. For example, we may use it to compute the mean, sd, var, min, quantile,… for each column in a data frame.
# The command direct to find the mean of all columns in the dataset iris except 5th column, which is species and is non numeric # Compute quartilessapply(iris[, -5], quantile)
The function stat.desc() [in pastecs package], provides lot of useful statistics in a single call
#Install 'pastecs' package#load the librarylibrary(pastecs) res <-stat.desc(my_data[, -5]) # results were stored to the variable 'res' # decimal digits were rounded to 2 using round()functionround(res, 2)
It should be noted that some R functions will return errors or NA even if just one item is missing or when the data contains missing values. For instance, if a vector has even a single missing value, the mean() method will return NA. The option na.rm = TRUE, which instructs the function to delete any NAs before computations, can be used to prevent this. Here’s an example of how to use the mean function:
mean(iris$Sepal.Length, na.rm =TRUE)
7.9 Frequency table
For more theoretical details see here Dataset below shows the number of children in 50 household collected as a part of survey. I will demonstrate how frequency table can be constructed.
# Create example vector set.seed(99)survey_data<-floor(runif(50, min=0, max=10)) survey_data
The frequency counts of the variables that appear in the chosen column of the dataframe are calculated using the table() function in R. The outcome is presented as a two-row tabular structure, with the first row displaying the value of each column and the second row displaying the matching frequencies.
# Frequency Tablefreq_table<-table(survey_data)freq_table
The total of all classes in a frequency distribution table, including this class below it, makes up the cumulative frequency distribution of a particular data set. The value at every cell location is calculated by adding the current value and all previous values encountered up to that point. This may be determined by using the cumsum() method.
# cumulative frequencycumsum(freq_table)
0 1 2 3 4 5 6 7 8 9
5 11 14 20 24 32 39 41 45 50
From the above results now you can easily answer the question: number of families having 8 or less than 8 children?. Verify it from sum(survey_data <= 8); the result depends on the simulated sample
# Constucting frequency tabledata<-iris$Sepal.Length # Decide how many classes you wish to use using struges formulaN<-length(data)k<-1+3.322*log(N,base=10)n_class<-ceiling(k) #Rounds value of k to nearest highest integer.# Equal-width boundaries must include every observed value.class_width <- (max(data) -min(data)) / n_classbreaks <-seq(min(data), max(data), length.out = n_class +1)data.cut <-cut(data, breaks = breaks, right =FALSE, include.lowest =TRUE)# Generate frequency table of classesdata.freq =table(data.cut)data.freq
This section extends the data-analysis examples into a reusable function. Read the calculation first, then see how the same steps are collected into one named operation. You do not need advanced programming to benefit from a small, clearly written function.
A function is a block of code which only runs when it is called. You can pass data, known as parameters, into a function. A function can return data as a result.
For example, here I’m creating a function named print_me(), on calling print_me(), it automatically prints “Hello world!”
# create a function with the name print_meprint_me <-function() { print("Hello World!")}
# calling function print_me()print_me()
[1] "Hello World!"
In the above section we have written at least 12 lines of code to construct a grouped frequency table from a data. Each time we need to run all these code individually, which is a difficult task. You can create a Function to perform this task in a single call. I’m creating a function called construct() to create a grouped frequency table.
construct <-function(x) {#This function will take in a numeric vector x and gives its frequency table x <- x[is.finite(x)]if (!length(x)) stop("No finite observations")if (length(unique(x)) ==1) return(table(x)) N <-length(x) n_class <-ceiling(1+3.322*log(N, base =10)) breaks <-seq(min(x), max(x), length.out = n_class +1) data.cut <-cut(x, breaks, right =FALSE, include.lowest =TRUE) data.freq <-table(data.cut)stopifnot(sum(data.freq) == N) data.freq}
Constructing frequency table of Petal.Length in iris dataset using the construct function we just created.
Find the descriptive statistics for the data using the functions above.
3.Construct grouped frequency table of Petal.Width column in iris data.
The boundary calculation above keeps the original Sturges-rule demonstration but ensures that the largest observation is counted. Rounding class boundaries too early can silently omit observations. Also, mode() in base R reports storage mode; it does not calculate the most frequent observation. The helper returns all tied most frequent values.
7.11 Agricultural applications, interpretation, and further practice
7.11.1 Centre and spread answer different questions
The arithmetic mean is the sum divided by the number of observations. The median splits ordered observations in half. The statistical mode is the most frequent value and may be nonunique. In R, mode() describes an object’s storage mode; it does not compute a statistical mode.
For measurements \(x_1,\ldots,x_n\), the sample variance and sample standard deviation are
Variance has squared units; standard deviation has the same units as the observations. The median and IQR are less sensitive to a few extreme values than the mean and standard deviation. Describe the distribution before choosing a summary.
crop <-read.csv("data/crop_trial.csv")y <- crop$yield_t_hac(n =length(y), mean =mean(y), median =median(y),variance =var(y), sd =sd(y), min =min(y), max =max(y), IQR =IQR(y))
n mean median variance sd min max
60.0000000 3.7720000 3.8750000 0.4259281 0.6526317 2.4000000 5.1600000
IQR
0.9475000
For continuously measured yield, the most frequent rounded value may have little scientific meaning; it depends on recording precision. A mode is often more useful for a categorical variable such as soil class.
7.11.2 Worked example: a small harvest sample
Consider four independent plot yields of 4.2, 4.8, 5.1, and 4.5 t/ha. Their sum is 18.6 t/ha and their mean is 4.65 t/ha. The deviations from the mean are −0.45, 0.15, 0.45, and −0.15. Squaring and adding gives 0.45 (t/ha)². Dividing by n - 1 = 3 gives a sample variance of 0.15 (t/ha)²; its square root is approximately 0.387 t/ha.
The denominator n - 1 reflects the loss of one degree of freedom after estimating the mean from these same observations. This is the sample-variance convention used by var(). If the four plots constitute an entire finite population of interest and the goal is its descriptive population variance, a denominator of four answers a different question. State the target rather than treating the two formulas as contradictory.
The median is the average of the two middle ordered values, 4.5 and 4.8, also 4.65 t/ha here. Equality of mean and median in this small example does not establish a symmetric population distribution.
7.11.3 Frequencies and grouped distributions
table(crop$treatment)
Compost Control Mineral
20 20 20
proportions(table(crop$treatment))
Compost Control Mineral
0.3333333 0.3333333 0.3333333
classes <-cut(y, breaks =pretty(range(y), n =6), include.lowest =TRUE)freq <-table(classes)data.frame(interval =names(freq), frequency =as.integer(freq),cumulative =cumsum(as.integer(freq)))
Class boundaries must cover every observation. This check avoids a weakness in hand-rounded class limits, where an extreme value may fall outside the final interval. Grouping loses within-class detail; retain the original measurements.
7.11.4 Standard deviation, standard error, and CV
Standard deviation describes variation among observations. For independent observations from a common population, the estimated standard error of a mean is \(s/\sqrt{n}\). It describes the precision of the estimated mean, not the spread of individual yields. Independence must be established from sampling or randomization, not from the software output.
The coefficient of variation is 100 * SD / mean. It is appropriate for ratio-scale measurements with a meaningful zero and a positive mean away from zero. It is inappropriate for comparing temperatures in degrees Celsius or pH. Also distinguish a raw-data CV from an experiment’s residual CV based on an ANOVA error mean square.
7.11.5 Grouped summaries
library(dplyr)
Attaching package: 'dplyr'
The following objects are masked from 'package:pastecs':
first, last
The following objects are masked from 'package:stats':
filter, lag
The following objects are masked from 'package:base':
intersect, setdiff, setequal, union
crop |>group_by(treatment, variety) |>summarise(n =n(), mean =mean(yield_t_ha), median =median(yield_t_ha),sd =sd(yield_t_ha), iqr =IQR(yield_t_ha), .groups ="drop")
# A tibble: 6 x 7
treatment variety n mean median sd iqr
<chr> <chr> <int> <dbl> <dbl> <dbl> <dbl>
1 Compost A 10 3.72 3.74 0.450 0.673
2 Compost B 10 3.95 3.86 0.339 0.567
3 Control A 10 2.92 2.93 0.381 0.535
4 Control B 10 3.32 3.34 0.518 0.498
5 Mineral A 10 4.32 4.19 0.386 0.408
6 Mineral B 10 4.39 4.35 0.302 0.312
Pooling groups can inflate spread because it combines between-group differences with within-group variation. Report the group definitions and sample sizes. Avoid presenting many decimal places beyond measurement precision.
7.11.6 Practical 8A and exercises
Calculate mean, median, variance, SD, and IQR for available soil nitrogen. State each unit.
Add one very large value to a copy of the yield vector. Which summaries change most?
Explain why doubling a sample size does not halve the SD of the underlying observations.
Compute a frequency table for the supplied soil classes and explain what its denominator represents.
Explain why ten leaves from one plant are not automatically ten independent experimental replicates.
Chapter review. Describe centre and spread together. Report sample size, missingness, units, and grouping so the summary can be interpreted.
7.12 Live R laboratory
Describe yield and its variability
Edit the code, then select Run R. The first run downloads R into your browser and may take a little time. Copy code also lets you use this example in RStudio.
yield <-c(3.2, 3.8, 4.0, 3.5, 4.6, 3.9)c(mean =mean(yield), median =median(yield),variance =var(yield), SD =sd(yield),CV_percent =100*sd(yield) /mean(yield))summary(yield)boxplot(yield, col ="#8ecab1", ylab ="Yield (t/ha)")
Investigate. Change 4.6 to 8.6 and compare the mean, median, and SD. Explain the units of variance and why the coefficient of variation is unitless.
From the browser to your practical notebook
Record the input you changed, the output you obtained, and your interpretation. These live examples use base R and work independently. The full chapter examples, including package-based analyses, remain available with copy buttons for RStudio. Browser objects last until this page is reloaded; Reset code restores the editor text. Internet access is needed on first use; long or interactive desktop tasks belong in RStudio.