Working with Imported Data - the Base R#

Learning Objectives#

Set up the working directory and import data into R.
Explore and manipulate data frames to extract insights.
Perform basic data cleaning, including handling dates and missing values.

Now you have mastered the basics of R. But we really want to learn how to import data! So let us do that.

We have a CSV file of the Atlantic Hurricane Database (HURDAT2) 1851-2022. The data is released by the National Hurricane Center (NHC) at NOAA. Use this link to download: https://tufts.box.com/shared/static/gdqc9tdv7334622tkco37oyg9t2fyeaf.csv

But how do we import this data so we can use it in R?

Working Directory#

All relative file paths in R are relative to the current working directory, which might or might not be the location of the R script, depending on how RStudio was originally launched. If RStudio was launched by double-clicking on the R script or by right-clicking on the R script and then selecting Open With > RStudio, the default working directory will be the location of the R script. Otherwise the default working directory will either be your Documents directory (Windows) or your home directory (macOS and Linux).

We can use getwd() and dir() to explore the current working directory.

getwd()     # The current location of the working directory.
dir()       # The files in the working directory.

'/home/runner/work/guides/guides/source/source/r'

'00_Getting_started_with_R.md'
'01_Intro_to_RStudio.md'
'02_intro-r_script excution and basics.Rmd'
'03_working_with_imported_data.Rmd'
'04_working_with_imported_data_tidyverse.Rmd'
'animation.gif'
'atlantic.csv'
'Fast_Loops_and_Parallel_Processing.md'
'index.md'
'r_data_dataviz'
'r-data-viz_files'

You can use the %in% operator to ensure that your working directory contains data file (atlantic.csv).

"atlantic.csv" %in% dir()

TRUE

If the statement above returns TRUE, you are all set. But if it returns FALSE, you need to change your working directory. Go to the top menu bar: Session > Set Working Directory > To Source File Location.

We can ensure the working directory is set correctly by either re-running the commands from above or clicking on the Files tab in the lower-right panel and then selecting More > Go To Working Directory and exploring the results.

Importing Data#

The atlantic.csv data file is in comma-separated values (CSV) format. (Note that the specifics of this format are outlined in standard RFC 4180.) This is a very common data format and you can easily import it in R as follows.

hurrdata <- read.csv("atlantic.csv")

The read.csv() function has numerous additional optional arguments that we can use to specify how exactly a data file should be read in and interpreted.

To investigate those, we can use the help() function or the ? operator.

?read.csv

Note that CSV files are different from Excel spreadsheets (XLS or XLSX files) and R does not contain the functionality to import the latter by default. An external community-developed package must be used to import Excel files. Installing and loading external packages is discussed later in this script.

Exploring Data#

We see that a new variable “hurrdata” has been added to the environment.

head(hurrdata)

A data.frame: 6 × 23
	ID	Name	Date	Time	Event	Status	Latitude	Longitude	Maximum.Wind	Minimum.Pressure	⋯	Radii.NW.34kt	Radii.NE.50kt	Radii.SE.50kt	Radii.SW.50kt	Radii.NW.50kt	Radii.NE.64kt	Radii.SE.64kt	Radii.SW.64kt	Radii.NW.64kt	Maximum.Wind.Radius
	<chr>	<chr>	<int>	<int>	<chr>	<chr>	<chr>	<chr>	<int>	<int>	⋯	<int>	<int>	<int>	<int>	<int>	<int>	<int>	<int>	<int>	<int>
1	AL011851	UNNAMED	18510625	0		HU	28.0N	94.8W	80	-999	⋯	-999	-999	-999	-999	-999	-999	-999	-999	-999	-999
2	AL011851	UNNAMED	18510625	600		HU	28.0N	95.4W	80	-999	⋯	-999	-999	-999	-999	-999	-999	-999	-999	-999	-999
3	AL011851	UNNAMED	18510625	1200		HU	28.0N	96.0W	80	-999	⋯	-999	-999	-999	-999	-999	-999	-999	-999	-999	-999
4	AL011851	UNNAMED	18510625	1800		HU	28.1N	96.5W	80	-999	⋯	-999	-999	-999	-999	-999	-999	-999	-999	-999	-999
5	AL011851	UNNAMED	18510625	2100	L	HU	28.2N	96.8W	80	-999	⋯	-999	-999	-999	-999	-999	-999	-999	-999	-999	-999
6	AL011851	UNNAMED	18510626	0		HU	28.2N	97.0W	70	-999	⋯	-999	-999	-999	-999	-999	-999	-999	-999	-999	-999

This shows us a preview of the fist couple rows of the file. You can also click on the data table under Environment > Data.

We can use summary() to get descriptive statistics.

summary(hurrdata)

      ID                Name                Date               Time       
 Length:53976       Length:53976       Min.   :18510625   Min.   :   0.0  
 Class :character   Class :character   1st Qu.:19180804   1st Qu.: 600.0  
 Mode  :character   Mode  :character   Median :19640609   Median :1200.0  
                                       Mean   :19554281   Mean   : 910.8  
                                       3rd Qu.:19961020   3rd Qu.:1800.0  
                                       Max.   :20221111   Max.   :2330.0  
    Event              Status            Latitude          Longitude        
 Length:53976       Length:53976       Length:53976       Length:53976      
 Class :character   Class :character   Class :character   Class :character  
 Mode  :character   Mode  :character   Mode  :character   Mode  :character  
                                                                            
                                                                            
                                                                            
  Maximum.Wind    Minimum.Pressure Radii.NE.34kt    Radii.SE.34kt   
 Min.   :-99.00   Min.   :-999.0   Min.   :-999.0   Min.   :-999.0  
 1st Qu.: 35.00   1st Qu.:-999.0   1st Qu.:-999.0   1st Qu.:-999.0  
 Median : 45.00   Median :-999.0   Median :-999.0   Median :-999.0  
 Mean   : 52.57   Mean   :-149.4   Mean   :-795.1   Mean   :-796.4  
 3rd Qu.: 70.00   3rd Qu.: 995.0   3rd Qu.:-999.0   3rd Qu.:-999.0  
 Max.   :165.00   Max.   :1024.0   Max.   : 780.0   Max.   : 720.0  
 Radii.SW.34kt    Radii.NW.34kt  Radii.NE.50kt    Radii.SE.50kt   
 Min.   :-999.0   Min.   :-999   Min.   :-999.0   Min.   :-999.0  
 1st Qu.:-999.0   1st Qu.:-999   1st Qu.:-999.0   1st Qu.:-999.0  
 Median :-999.0   Median :-999   Median :-999.0   Median :-999.0  
 Mean   :-801.4   Mean   :-799   Mean   :-806.1   Mean   :-806.4  
 3rd Qu.:-999.0   3rd Qu.:-999   3rd Qu.:-999.0   3rd Qu.:-999.0  
 Max.   : 660.0   Max.   : 600   Max.   : 360.0   Max.   : 300.0  
 Radii.SW.50kt    Radii.NW.50kt    Radii.NE.64kt    Radii.SE.64kt   
 Min.   :-999.0   Min.   :-999.0   Min.   :-999.0   Min.   :-999.0  
 1st Qu.:-999.0   1st Qu.:-999.0   1st Qu.:-999.0   1st Qu.:-999.0  
 Median :-999.0   Median :-999.0   Median :-999.0   Median :-999.0  
 Mean   :-807.7   Mean   :-807.1   Mean   :-809.2   Mean   :-809.3  
 3rd Qu.:-999.0   3rd Qu.:-999.0   3rd Qu.:-999.0   3rd Qu.:-999.0  
 Max.   : 360.0   Max.   : 390.0   Max.   : 180.0   Max.   : 250.0  
 Radii.SW.64kt    Radii.NW.64kt    Maximum.Wind.Radius
 Min.   :-999.0   Min.   :-999.0   Min.   :-999.0     
 1st Qu.:-999.0   1st Qu.:-999.0   1st Qu.:-999.0     
 Median :-999.0   Median :-999.0   Median :-999.0     
 Mean   :-809.7   Mean   :-809.5   Mean   :-973.2     
 3rd Qu.:-999.0   3rd Qu.:-999.0   3rd Qu.:-999.0     
 Max.   : 180.0   Max.   : 300.0   Max.   : 400.0     

We might wonder - what type of data is hurrdata? We imported it from csv, but how is it stored?

class(hurrdata)

'data.frame'

We can see that it is a “data.frame”, commonly referred to as a data frame. A data frame is a table where you have observations as rows and variables as columns. Data frames have some great features for working with data and are the go-to for R data storage. You can think of them almost as spreadsheets.

Working with Data Frames#

Let us say we want to access the maximum wind speed of the sixth observation. We can do this in multiple ways. Knowing that “Maximum.Wind” is the ninth column:

hurrdata[[6, 9]]    # [[row, column]]
hurrdata[[9]][6]    # [[column]][row]

NOTE: When using a data frame, we must use [[]] to access variable values. [] will just give us a subset of the data frame, not the values themselves.

Alternatively, we could use the column name:

hurrdata[["Maximum.Wind"]][6]
hurrdata$Maximum.Wind[6]

Note the dollar sign ($). This is a special operator that allows us to access data.frame variables (columns) based on their name.

The “$” operator is the preferred way of accessing data.frame variables based on their name, as it removes the complexity of when to use [[]] vs [] and does not require quotation marks … given your column names do not contain spaces.

Note how the read.csv() function automatically replaced spaces with periods in the column names to accommodate this. Underscores are also an acceptable alternative to spaces and other functions might use those instead of periods.

Selecting Data Based on Conditions#

Note how the summary() function from before did not provide much information on the columns containing textual (character/string) data. These columns can be analyzed further using the table() function. Can you guess what it does?

table(hurrdata$Name)

It creates a frequency table of all the unique values in the column! But what if we wanted to analyze hurricanes only with a specific name?

We can select rows based on a condition by combining logical operators with [] to subset all the rows where the logical operator returns TRUE. For example, all the hurricanes named Nicole can be extracted as follows.

hurrdata[hurrdata$Name == "NICOLE", ]

Note that we leave the column index blank to select all the columns.

Alternatively, we could extract values from one column based on the values of another column. For example, the maximum wind speed for all the hurricanes named Nicole could be obtained as follows.

max(hurrdata$Maximum.Wind[hurrdata$Name == "NICOLE"])

120

Data Cleaning: Dates & Strings#

Let us say we want to analyze maximum wind speed by year. Note how the date is stored as a number in YYYYMMDD format. This notation is great for sorting but very inconvenient for analysis.

We can use the “$” operator to easily extract the Data column as follows.

hurrdata$Date

Passing this column to the as.character() function will convert all the numbers into text (character data). Note the quotes around the new values.

as.character(hurrdata$Date)

Let us store these new character (string) representations of the dates in a new variable and use the substr() function to extract the year and the month.

date_strings <- as.character(hurrdata$Date)

Extract the year from the date string (position 1-4).

hurrdata$Year <- substr(date_strings, start = 1, stop = 4)

Extract the month from the date string (position 5-6).

hurrdata$Month <- substr(date_strings, 5, 6)

Convert both new variables to numeric to accommodate further analysis.

hurrdata$Month <- as.numeric(hurrdata$Month)
hurrdata$Year <- as.numeric(hurrdata$Year)

Note that this is a somewhat hack-y way of dealing with dates. It is suitable for simple conversions like this, but it is highly encouraged to use external packages specifically designed to work with dates for more complex tasks.

Data Cleaning: Missing Data#

Why are some wind speeds negative? If we looked into the metadata, we would find that we should have removed these! We can check on the negative data:

min(hurrdata$Maximum.Wind)

-99

We can combine [] with logical operators like before to find all the values less than zero (< 0) and replace them with “NA”, which means “No Data” in R.

hurrdata$Maximum.Wind[hurrdata$Maximum.Wind < 0] <- NA

We could also delete the whole observation, but this is bad practice! Let’s check on the results. It should print “NA”.

min(hurrdata$Maximum.Wind)

<NA>

To get the minimum wind speed excluding the NA values, we must call the min() function with na.rm = TRUE to instruct the function to ignore the NA values.

min(hurrdata$Maximum.Wind, na.rm = TRUE)

Sampling the Data#

We can also sample the data.frame to de-clutter the scatter plot. This is a two-step process in base R. First we use the sample() function to randomly select a desired quantity of row indexes/numbers for our data frame.

sample(nrow(hurrdata), 1000)

29809
42828
19469
44676
51631
8816
33764
39566
38865
6748
16676
2747
37083
5371
25897
10550
6601
23425
48811
51711
45674
37371
31033
10797
39389
9506
24126
25856
7330
38521
6717
12271
16017
11708
3726
4355
45986
51643
20018
24604
17351
15587
18603
33217
14021
5114
34986
7506
21412
28578
51166
51103
46346
38946
378
43649
50029
44458
15248
42514
37853
8363
37241
34348
29140
34227
10806
34590
18546
45639
53061
40667
18595
41656
17586
20846
42121
15774
2013
8379
8939
53638
37675
8964
26473
21753
53442
36196
39380
37209
24007
25242
25496
44024
22814
30956
44600
34028
34339
38687
30947
875
22189
8220
41616
12902
7367
48784
45228
38227
19924
42042
39645
52442
27658
17793
50201
29583
22887
49965
53183
5632
9952
25557
14816
15010
17012
24533
27857
41209
48213
45340
19908
28236
53807
23066
44758
21320
51081
32426
30597
18727
8380
34838
6685
23567
5233
43623
15677
45307
43741
41967
47240
53161
20214
13663
38809
24120
30847
23893
46365
31
7737
24260
22135
21512
52430
23900
24906
36120
31786
16099
36134
19860
50473
3448
46584
24085
11478
21241
21611
39790
834
22234
17107
36842
46955
53946
44300
48936
6504
48854
44857
52337
36448
47572
4659
18489
5591
19591
⋯
36869
9834
50984
28516
35385
11788
26971
51581
20684
21725
4161
50651
1772
40827
10903
38255
9018
17677
32959
3140
29099
30354
5329
34207
23829
21752
25842
35836
53901
52300
567
35208
49713
40731
41610
39582
27949
9984
2276
38011
36413
7100
28986
46917
48959
35906
50010
7896
33928
24486
49113
8629
45799
48738
15749
31127
40690
41908
16518
8049
48773
8544
11861
47220
33593
35885
9706
15833
5396
51772
23315
34032
23795
53863
48897
11877
16310
25714
40950
39913
11635
9373
48858
38789
4052
51049
27471
53002
20418
19170
7303
20236
10559
42361
17317
3131
50752
4274
21971
27695
42845
28781
2233
34351
16568
24953
43240
23172
12427
23245
22619
23310
12937
33389
12310
36012
42451
24287
22237
43807
42609
13035
33620
31824
51494
52453
35455
9431
25295
5697
25745
53023
1090
20119
31913
45107
26323
25062
8681
23514
31922
10554
1361
37394
16854
41730
40373
14187
8688
39165
18827
29363
35036
45524
51604
8717
2541
43999
47211
44576
1980
46078
8322
1898
24598
15726
19363
22817
13562
43472
28326
14275
24784
11501
19403
32328
18384
51783
44646
31445
40966
21023
4359
9321
33739
36293
20788
12081
41613
25747
25850
9976
31210
53243
27531
6420
18881
43749
50969
22580

And then we use [] to extract those rows from the data frame.

hurrdata2 <- hurrdata[sample(nrow(hurrdata), 1000), ]

To Get Support: - Please email tts-research@tufts.edu for questions and requests.