Pulling in my “Lost” Data

In my last post, I talked about finding and reviving my early Christianity data. Joyous occasion! Now I need to bring it in to RStudio.

My data is currently in several JSON files, which have been uploaded to a project space in Github. If you pull down that project and all of the files, then this code should work perfectly using the “here” package. The here package allows R to reference and construct file paths relative to where the project is saved, or the working directory you are working from. It has been a godsend since I have moved to using Quarto, where working inside project spaces is essential to making sure everything works and compiles correctly. It also makes working with Github much easier. Let’s take a look!

Before anything, you need your packages. The following code loads the required packages for this script, which include the aforementioned here, the always-essential tidyverse, and jsonlite, which will be used for loading and flattening the JSON files. I have included (commented-out) lines for installing those packages if you don’t already have them.

#install.packages("here")
#install.packages("tidyverse")
#install.packages("jsonlite")

# Load libraries
library(here)
library(tidyverse)
library(jsonlite)

In my case, I have the project, referred to as “blog-code-ver-usagi”, saved to a folder on my D: drive called “Early Xty”. When I call the here() function, it returns the path "D:/Early Xty/github/blog-code-ver-usagi". Accessing the folders inside the path is easy, just enter the name of the desired folder inside quotes.

here()

[1] "D:/Early Xty/github/blog-code-ver-usagi"

here("raw_data_files")

[1] "D:/Early Xty/github/blog-code-ver-usagi/raw_data_files"

I want to read in all of my files in at one time, and give them a name based on the file I read them from. I also want to apply jsonlite’s flatten to each of them. My JSON files are simple, with only 1 level of data, so flatten will easily convert each file into a simple, tabulated data frame.

A simple loop will give me exactly what I want. Below this chunk is a detailed explanation of what I am doing in each numbered step.

#1
raw_data_dir <- here("raw_data_files")
#2
raw_data_file_names <- list.files(raw_data_dir)
#3
for (i in raw_data_file_names) {
  #4
  current_file <- fromJSON(file.path(
    raw_data_dir,
    i
  ), flatten = T)
  #5
  simple_name <- i |> 
    str_remove(".json")
  #6
  assign(simple_name, current_file)
}

What am I doing here?

I created an object that contained the file path to where the raw files live and called it raw_data_files.
Using the base-R function list.files(), I created a vector with all of the file names that are in raw_data_files called raw_data_file_names.
I invoke the loop with for (i in raw_data_file_names), followed by curly brackets {}. The code inside the brackets is applied to i - or the current value being used in whatever set you have fed into the for (). You can actually use any letter you like - it is simply a placeholder inside your function. Not unlike “x” or “y” in a mathematical formula. Here, I am asking R to look at each item in raw_data_file_names vector.
Now the magic inside the loop’s curly brackets begins.
I apply the fromJSON() function to read the .json file. Inside the parenthesis in fromJSON(), file.path() tells R that I am putting together a file path based on the previously-constructed objects, inside of those parenthesis. In this case it is the raw_data_dir (which points to D:/Early Xty/github/blog-code-ver-usagi/raw_data_files) and i, which is the current value/file name in the vector, raw_data_file_names. This creates a complete file path that fromJSON() can read. After the file path, I apply the flatten with flatten = T. I ask that it store the file it is reading in to an object called current_file. With each iteration over the files in the loop, this object will be overwritten with whatever file is being referenced with i.
Because of that, we need to rename the set when the data has been loaded, and I think the self-descriptive names of the files work the best. So, again using the current value of i, I use the stringr command str_remove() to strip out the ".json" part of the file name, and leave me with just the simple name. For example: structures.json becomes structures.
Base-R function assign() lets me use that simple_name to name the current_file.

The loop will perform these steps for every file listed in raw_data_file_names, until it reaches the end of the vector. As a result, your Environment should show an object for every file, each name with the simplified name of that same file. If you did it right, you should see something like this:

Loops are incredibly handy for any code you want to apply repeatedly to a set of things. You just need to be sure you have that set of things stored somewhere, or otherwise accessible in your Environment. My example is very simple and straightforward, because that is what I needed, but you can get very fancy using them!

Now my data is loaded and ready to party.