Welcome, Guest
Username: Password: Remember me

TOPIC: Extracting Seasonal Means

Extracting Seasonal Means 1 year 1 month ago #18

  • omdoherty
  • Offline
  • New Member
  • Posts: 2
  • Karma: 0
First, I want to thank Marcus and Todd for creating this awesome tool. SWMPr is so powerful, intuitive and easy to use. It's great for getting started looking at SWMP data.

I am struggling to extract individual seasons from SWMPr data frames. While I'm proficient in MATLAB, bash and such, R is new to me so I apologize if this is a novice question. I've taken two approaches, neither has worked.

The first approach is creating month, day and year arrays and then attaching them to the original SWMPr data frame.
#Grab a subset of the data, in this case the date-time stamp
ph <- subset(dat, select = 'ph')
dts = ph[['datetimestamp']]
datearray <- as.Date(dts, "%Y-%m-%D %H")
#Extract numeral date/time info from R date class
mm=as.numeric(format(dts, "%m"))
yyyy=as.numeric(format(dts, "%Y"))
dd=as.numeric(format(dts, "%d"))
#add month and year to swmp data frame for easy subsetting
dat$mm <- c(mm)
dat$yyyy <- c(yyyy)
dat$dd <- c(dd)
#create a seasonal subset of data, i.e. JFM
params <- c('temp','sal','ph')
winter_dat = subset(dat, mm<4, select = params)

This approach produces an error message of: "Error in as.POSIXct.default(subset, format = "%Y-%m-%d %H:%M", tz = timezone) :
do not know how to convert 'subset' to class “POSIXct”"

My second approach is to use the SWMPr aggregate function to produce quarterly globs, and then target the globs.
#Produce Seasonal Aggregates
q_ph=aggregate(dat, 'quarters', params = c('ph'), aggs_out = TRUE)
q_ph_jja <- subset(q_ph, format(q_ph$datetimestamp,'%m') %in% c('01','02','03'))

This produces an identical error. I think that this is occurring because the subset function is now 'pointing' to the SWMPr subset function, which is not built to handle this sort of approach. What then is the best way to grab individual seasons of data from a SWMP data frame?

I am interested in doing a comprehensive EDA on the data, so prefer keeping the raw data in tact and then calling R functions to assess spread, center, etc.

Thanks for both the great tool and for any thoughts on seasonal extractions!
Owen Doherty
Research Scientist
Eagle Rock Analytics
Sacramento, CA
The administrator has disabled public write access.

Extracting Seasonal Means 1 year 1 month ago #19

  • Marcus Beck
  • Offline
  • Administrator
  • Posts: 32
  • Thank you received: 5
  • Karma: 4
Hi Owen,

Thanks for your words of support for SWMPr! You've uncovered a tricky issue with R 'namespaces'. It's like an order of operations that R uses for calling methods for R classes.

Fortunately, the swmpr object class also inherits methods for data frames. The subset method for data frames also works for swmpr objects, unless there's a pre-existing swmpr method for a function as you've found for subset. You'll have to remove the swmpr class from the object to use the subset function in it's 'normal' mode.
# remove.packages('SWMPr')
# install.packages('SWMPr')
library(SWMPr)
data(apacpwq)
dat <- apacpwq
#Grab a subset of the data, in this case the date-time stamp
ph <- subset(dat, select = 'ph')
dts = ph[['datetimestamp']]
datearray <- as.Date(dts)
#Extract numeral date/time info from R date class
mm=as.numeric(format(dts, "%m"))
yyyy=as.numeric(format(dts, "%Y"))
dd=as.numeric(format(dts, "%d"))
#add month and year to swmp data frame for easy subsetting
dat$mm <- c(mm)
dat$yyyy <- c(yyyy)
dat$dd <- c(dd)
#create a seasonal subset of data, i.e. JFM
params <- c('temp','sal','ph')
dat <- as.data.frame(dat)
winter_dat = subset(dat, mm<4, select = params)
dat <- swmpr(dat, 'apacpwq')
q_ph=aggreswmp(dat, 'quarters', params = c('ph'), aggs_out = TRUE)
q_ph <- as.data.frame(q_ph)
q_ph_jja <- subset(q_ph, format(q_ph$datetimestamp,'%m') %in% c('01','02','03'))

The first change was coercing the swmpr object to the data frame class. Then I had to recreate the swmpr object to use aggreswmp (the latest release of SWMPr changed the function name), then back to data frame to use subset again. This is pretty klunky... you could also use functions from dplyr, which is a very useful data manipulation package.
# remove.packages('SWMPr')
# install.packages('SWMPr')
# install.packages('dplyr')
library(dplyr)
library(SWMPr)
data(apacpwq)
dat <- apacpwq
#Grab a subset of the data, in this case the date-time stamp
ph <- subset(dat, select = 'ph')
dts = ph[['datetimestamp']]
datearray <- as.Date(dts)
#Extract numeral date/time info from R date class
mm=as.numeric(format(dts, "%m"))
yyyy=as.numeric(format(dts, "%Y"))
dd=as.numeric(format(dts, "%d"))
#add month and year to swmp data frame for easy subsetting
dat$mm <- c(mm)
dat$yyyy <- c(yyyy)
dat$dd <- c(dd)
#create a seasonal subset of data, i.e. JFM
params <- c('temp','sal','ph')
winter_dat = filter(dat, mm<4)
winter_dat = select(winter_dat, one_of(c('datetimestamp', params)))
q_ph=aggreswmp(dat, 'quarters', params = c('ph'), aggs_out = TRUE)
q_ph_jja <- filter(q_ph, format(q_ph$datetimestamp,'%m') %in% c('01','02','03'))

Hope that helps.

-Marcus
Last Edit: 1 year 1 month ago by Marcus Beck.
The administrator has disabled public write access.
The following user(s) said Thank You: omdoherty

Extracting Seasonal Means 1 year 1 month ago #20

  • omdoherty
  • Offline
  • New Member
  • Posts: 2
  • Karma: 0
Hi Marcus,

Thanks very much for your prompt and thorough response. Each of the three solutions your response contains work perfectly. The dplyr package you suggested is very powerful, with a lot of great functions to play with.

Solution #2 (of filtering and then selecting) is a great option. Then one can simply aggregate data as desired in following steps. So thanks for that. After years of doing time-series analysis in MATLAB, R is a revelation. I got here kicking and screaming, but SWMPr makes it possible to get into some good stuff really fast.

Also wanted to comment on the plot_summary function. It's awesome, what a great way to get a quick overview of your data.
Owen Doherty
Research Scientist
Eagle Rock Analytics
Sacramento, CA
The administrator has disabled public write access.

Extracting Seasonal Means 1 year 1 month ago #21

  • Marcus Beck
  • Offline
  • Administrator
  • Posts: 32
  • Thank you received: 5
  • Karma: 4
Excellent, sounds like things are in working order! The initial hurdle learning R is a pain but it has immense payoffs, as you've found.
The administrator has disabled public write access.
Time to create page: 0.076 seconds
Powered by Kunena Forum