TOPIC: Adding station data not from CDMO

Several of my stations were operational prior to the data range that can be downloaded from CDMO and I am trying to add this data to what is run in the SWMPr package. To do so I copied and pasted the data into a water quality file that was downloaded from CDMO. None of the column headers, order, or number were changed. However, I keep getting this error message.
Error in rbind(deparse.level, ...) :
numbers of columns of arguments do not match

I even receive this error message if I copy say apacpwq2003 and rename it apacpwq2001. Does anyone know why I am receiving this error?

Because Excel SCREWS UP csvs! Seriously open a working and a non-working file in notepad, and you'll see differences in commas and quotation marks - nothing that you see in Excel, but stuff that's happening in the background. (incidentally, this is also helpful for troubleshooting CDMO upload problems.)

I believe Katie from GTM has gotten around this by opening up each CSV downloaded from the CDMO and resaving it. That makes all that hidden formatting the same as the other files you've saved.

I had this happen on a single file that I'd needed to modify, so I ended up altering just the one in notepad. With multiple files, just open, save as csv, close, move on.

EDIT - in case I wasn't clear - it's not so much that one format (the CDMO-formatted csv vs. Excel-formatted csv) is better than another for SWMPr. It's just that all the files you're reading into R need to be in the same format, and it takes a little work to get them that way.

Good luck!

Hi Ethan,

As a general rule, I would not mess with the CSV files that you've downloaded from CDMO. The SWMPr functions are written specifically for the format returned by the Zip Downloads feature. That's not to say that SWMPr won't be able to import altered CDMO files but it doesn't guarantee that strange things won't happen somewhere along the way.

My advice is to import both file types into R - CDMO first using import_local (SWMPr) and the other non-CDMO csv files using read.csv. Once all the files are in R, you can try to combine them by row using rbind. This is where you might have problems, if, for example, there are extra blank columns in the non-CDMO files. You can edit outside of R so it matches the format of the imported CDMO file, reimport the altered file, then try combining again. After you've successfully combined the files, you can make the combined object a 'swmpr' object using the swmpr function. This is used internally within import_local but it's also available as a stand-alone function to make your own 'swmpr' objects. Just make sure to include the 7 or 8 chr string for the station when you use 'swmpr'. You can use any of the other functions in SWMPr with the new 'swmpr' object. See the help file for 'swmpr' for more details.

Hope that helps.

-Marcus

I have heard about Excel and its formatting problems. Sadly R still gives the same error code even after saving it as a text file. The file from CDMO and the file that I created have the same number of columns. I created it by saving a CDMO file as a different name, then pasting in the parameter data. So I am at a loss.

Ethan, can you email me the files that are causing problems?

After some help and a whole lot of pirating from other scripts I managed to piece together a script that will allow you to add non CDMO data to CDMO data and run them together through the SWMPr package to get an estimate of the ecosystem metabolism. As well as, have R spit out a csv of the combined WQ NUT MET data, and a csv for the Ecotab values.

#Reset R's Brain
rm(list=ls())

# load any installed packages
library(SWMPr)
library(ggplot2)
library(dplyr)
library(tidyr)

#Find out where R is looking for files
getwd()

#Set where R ~should~ look for files
#note the forward slashes - you may have trouble if you just copy and paste from windows explorer, as it may use backslashes
setwd("C:/Users/bourque_e/Desktop/Metab/WQ NUT MET for CDMO sites")

#Make sure R is now looking in the right place
getwd()

# import data for multiple stations
# this is an example path with hypothetical csv files
# change as needed for actual data
path <- 'C:/Users/bourque_e/Desktop/Metab/WQ NUT MET for CDMO sites'

# import your data. Copy the format of the CDMO WQ files and name differently then the CDMO ones.
mydat <- read.csv('C:/Users/bourque_e/Desktop/Metab/WQ NUT MET for CDMO sites/aapacpwq2000.csv', header = T)
mydat1 <- read.csv('C:/Users/bourque_e/Desktop/Metab/WQ NUT MET for CDMO sites/aapacpwq1999.csv', header = T)
mydat2 <- read.csv('C:/Users/bourque_e/Desktop/Metab/WQ NUT MET for CDMO sites/aapacpwq1998.csv', header = T)

## rBIND now or later????
# rbind
alldat <- rbind(mydat, mydat1, mydat2)

##
# process mydat to match swmpr format

# remove extra columns
alldat <- select(alldat, -StationCode, -isSWMP, -Historical, -ProvisionalPlus, -F_Record)

# format datetimestamp to correct class and timezone DateTimeStamp DTS must be capitolized!!!!!!!!!!!
alldat <- mutate(alldat,
DateTimeStamp = as.POSIXct(DateTimeStamp, format = '%m/%d/%Y %H:%M', tz = 'America/Jamaica')
)

# columns are case-sensitive, convert to lower
names(alldat) <- tolower(names(alldat))

# import
wq_dat <- import_local(path, 'apacpwq')

# combine swmp data with your data
# rbind
alldat <- rbind(alldat, wq_dat)

# make swmpr object
alldat <- swmpr(alldat, 'apacpwq')

## HAVE TO GET alldat into wq_dat
wq_dat<-alldat

# import
nut_dat <- import_local(path, 'apacpnut')
met_dat <- import_local(path, 'apaebmet')
dat <- comb(wq_dat, nut_dat, met_dat)
dat <- subset(qaqc(dat), rem_cols = T)

# combine datasets that come with SWMPr
# qaqc and subset to remove empty columns

# names of all the attributes
names(attributes(dat))

# examples of retrieving attributes
attr(dat, 'station')
attr(dat, 'parameters')
attr(dat, 'qaqc_cols')
attr(dat, 'date_rng')
attr(dat, 'timezone')


#to see first 6 rows of the subsets data
head(dat)

# Summary of whole dataset
summary(dat)

# individual variables
summary(dat$do_mgl)

# mean, range, var, etc.
# note use of na.rm
mean(dat$do_mgl, na.rm = T)
range(dat$do_mgl, na.rm = T)
var(dat$do_mgl, na.rm = T)
sd(dat$do_mgl, na.rm = T)
min(dat$do_mgl, na.rm = T)
max(dat$do_mgl, na.rm = T)

# how many missing values?
sum(is.na(dat$do_mgl))

## estimate metabolism
res <- ecometab(dat)

# write to csv file
write.csv(res, file = 'FingersCrossed13.csv', na="")

## plot summaries of the metabolism data
plot_metab(res)

#res1 prevents res from being reassociated with a different data set
res1 <- attr(res, 'metabolism')

#Write CSV in R without NA in every empty cell.
write.csv(res1, file = 'FingersCrossed13b.csv', na="")

# save the whole workspace as a .RData file
# will be saved in the working directory
save(list = ls(), file = 'my_workspace.RData')