Use selection, filtering, transformation, sorting, grouping, joins, and reshaping to answer agricultural questions while preserving observational units.
5.2 A grammar of data manipulation
select() chooses columns, filter() chooses rows, mutate() creates or changes columns, arrange() orders rows, and summarise() reduces observations to summaries. group_by() controls the groups within which later operations occur. These verbs express a sequence of data operations; the pipe passes each result to the next step.
library(dplyr)
Attaching package: 'dplyr'
The following objects are masked from 'package:stats':
filter, lag
The following objects are masked from 'package:base':
intersect, setdiff, setequal, union
plot_id treatment yield_t_ha
1 P045 Mineral 5.16
2 P036 Mineral 4.86
3 P047 Mineral 4.84
4 P046 Mineral 4.72
5 P023 Mineral 4.55
6 P041 Compost 4.52
filter() keeps rows where its condition is TRUE; an NA condition is not retained. State explicitly whether missing records should be included. mutate() generally preserves the number of rows; summarise() changes the observational level.
# A tibble: 3 x 5
treatment n_plots n_yield mean_yield sd_yield
<chr> <int> <int> <dbl> <dbl>
1 Compost 20 20 3.84 0.406
2 Control 20 20 3.12 0.488
3 Mineral 20 20 4.36 0.340
The treatment table has one row per treatment, whereas the original table has one row per plot. It cannot support analyses that require individual plot variation. .groups = "drop" prevents accidental retention of grouping in later operations. If every yield in a group is missing, mean(..., na.rm = TRUE) returns NaN; handle that group as having no usable estimate.
Summing monthly rainfall gives annual rainfall when all 12 months are present. Averaging monthly temperature means gives equal weight to months, not days; use day weights if the question requires a mean over days. A mean of yields is not a production-weighted regional yield unless the areas are equal.
5.4 Joining tables safely
A join links records by keys. Here the soil table has one record per plot. Verify key uniqueness and unmatched records before accepting a join. Many-to-many joins can silently multiply observations and change subsequent averages.
stopifnot(!anyDuplicated(soil$plot_id))anti_join(crop, soil, by ="plot_id")
left_join() retains all rows of the first table, adding NA when there is no match. inner_join() retains only matched rows. A row count check is necessary but not sufficient: inspect identifier meaning and measurement timing too.
5.5 Wide and long forms
Wide tables put several measurements in separate columns; long tables use a variable column and a value column. Long form is convenient for grouped graphics.
Do not average nitrogen and phosphorus merely because both are expressed in mg/kg. They are distinct quantities. Reshaping changes representation, not scientific meaning.
5.6 Understanding what happens inside a grouped operation
When we write group_by(treatment), R does not immediately replace the observations with treatment means. It adds grouping information to the table. The next operation then acts within each group. This distinction explains why mutate() and summarise() can give very different tables after the same grouping command.
# A tibble: 6 x 5
plot_id treatment yield_t_ha treatment_mean deviation_from_treatment
<chr> <chr> <dbl> <dbl> <dbl>
1 P001 Control 2.42 3.12 -0.704
2 P002 Control 2.82 3.12 -0.304
3 P003 Control 2.4 3.12 -0.724
4 P004 Control 2.86 3.12 -0.264
5 P005 Compost 3.52 3.84 -0.316
6 P006 Compost 3.14 3.84 -0.696
nrow(within_treatment)
[1] 60
nrow(treatment_summary)
[1] 3
The first result still has 60 plot rows. Every plot carries its treatment’s mean and its own deviation from that mean. The summary table has only three treatment rows. If the purpose is to investigate within-treatment variation, the first table is useful. If the purpose is a compact reporting table, the second is useful. Neither representation should be confused with additional independent replication.
The word n() counts rows in a group. It does not automatically count nonmissing yields. Thus a group can have ten rows but only eight observed yields. The quantities n(), sum(!is.na(yield_t_ha)), and mean(yield_t_ha, na.rm = TRUE) must be interpreted together.
5.7 Weighted means and unequal harvested areas
Let us suppose two farms harvest 1 ha and 9 ha, with yields of 2 and 6 t/ha respectively. Their arithmetic mean yield is 4 t/ha. However, total production is 56 tonnes on 10 ha, so the area-weighted yield is 5.6 t/ha. Both calculations are arithmetically correct; they answer different questions.
The arithmetic mean describes the average yield of these two farm records when each farm receives equal weight. The weighted mean describes production per hectare across their combined harvested area. Before reporting a district, season, or treatment summary, ask what should receive equal weight. A column of weights must have a scientific interpretation; arbitrary weights cannot improve an analysis merely by changing the result.
5.8 Joining is not matching by row position
If soil rows have been sorted by pH while crop rows remain sorted by plot number, using cbind() to combine them would attach soil measurements to the wrong plots. A join uses an identifier instead. We can demonstrate this by deliberately changing the soil row order.
soil_reordered <- soil[order(soil$ph), ]joined_by_id <-left_join(crop, soil_reordered, by ="plot_id")stopifnot(identical(joined_by_id$plot_id, crop$plot_id))all.equal(joined_by_id$ph, linked$ph)
[1] TRUE
The pH values remain correctly aligned because the join uses plot_id. A correct key is therefore more valuable than a visually tidy row order. For data collected at several depths or dates, plot_id alone may not be unique; the key might need plot_id, depth_cm, and sampling_date. A one-to-many relationship can be legitimate, but the resulting observational unit must be stated.
5.9 Practical 5 and exercises
Produce treatment-by-variety summaries with sample size, mean, and standard deviation.
Find the five highest-yielding plots. Why is choosing their treatment after looking at the data an exploratory decision?
Join crop and soil data, then summarize available nitrogen by treatment.
Calculate annual rainfall only for years with 12 recorded months. Explain why sum(..., na.rm = TRUE) could understate rainfall in an incomplete year.
Introduce a repeated key into a copy of the soil table and investigate its effect on the joined row count.
Chapter review. Grouping changes the level of a question. Check keys, denominators, units, and completeness whenever observations are combined.
5.10 Live R laboratory
Summarize treatment groups
Edit the code, then select Run R. The first run downloads R into your browser and may take a little time. Copy code also lets you use this example in RStudio.
Investigate. This base R example mirrors the grouped summaries explained with dplyr in the chapter. The simulated difference is descriptive; an experimental design is needed for causal interpretation.
From the browser to your practical notebook
Record the input you changed, the output you obtained, and your interpretation. These live examples use base R and work independently. The full chapter examples, including package-based analyses, remain available with copy buttons for RStudio. Browser objects last until this page is reloaded; Reset code restores the editor text. Internet access is needed on first use; long or interactive desktop tasks belong in RStudio.