Length 0 in index of make_choose_all_table

[llrs]

I'm having some problem with make_choose_all_table. It doesn't seem to handle missing values well:

ds <- structure(list(ma___1 = c("b", NA, NA, NA, NA, NA, NA, NA), 
                     ma___2 = c("a", NA, NA, NA, NA, NA, NA, NA), 
                     ma___3 = c(NA, NA, NA, NA, NA, NA, NA, NA), 
                     ma___4 = c(NA, NA, NA, NA, NA, NA, NA, NA)), 
                class = "data.frame", row.names = c(NA, -8L))
tidyREDCap::make_choose_all_table(ds, "ma")
#> Error in `purrr::map_chr()`:
#> ℹ In index: 1.
#> Caused by error:
#> ! Result must be length 1, not 0.

#> Backtrace:
#>      ▆
#>   1. ├─tidyREDCap::make_choose_all_table(ds, "ma")
#>   2. │ └─... %>% dplyr::select(.data$What, .data$Count)
#>   3. ├─dplyr::select(., .data$What, .data$Count)
#>   4. ├─dplyr::bind_cols(., counts)
#>   5. │ └─rlang::list2(...)
#>   6. ├─dplyr::rename(., What = .data$value)
#>   7. ├─tibble::enframe(., name = NULL)
#>   8. ├─purrr::map_chr(...)
#>   9. │ └─purrr:::map_("character", .x, .f, ..., .progress = .progress)
#>  10. │   ├─purrr:::with_indexed_errors(...)
#>  11. │   │ └─base::withCallingHandlers(...)
#>  12. │   └─purrr:::call_with_cleanup(...)
#>  13. └─base::.handleSimpleError(...)
#>  14.   └─purrr (local) h(simpleError(msg, call))
#>  15.     └─cli::cli_abort(...)
#>  16.       └─rlang::abort(...)

^{Created on 2023-02-20 with reprex v2.0.2}

An example like this would be very helpful to users and it could be used as a test (related to #15 ).

[RaymondBalise]

Hello Lluís (@llrs),
Thank you for reporting the problem. The issue is not with the NAs. Rather the function is unhappy because the data does not have the typical structure of a REDCap export. I am thinking we should change the documentation to make it clear that the function is expecting data coming out of REDCap. That is, it is expecting labeled variables and indicator variables in the format that REDCap exports ("checked" vs. "unchecked" or 1 vs. 0). We will add checks on the structure to avoid the bad error message that you got.

In theory we can tweak the function to work on any data but the labels are critical because that is how the function knows what values to print.

For now, try exporting your data using the tidyREDCap::import_instruments() function.

[RaymondBalise]

I just added a check. The development release now gives better feedback:

What do you think?

[llrs]

I can't test now, as I don't have access to the database, but this would prevent using data imported from the API directly instead of just exported and read into R with the R script REDCap provides.

There are many packages trying to simplify the data manipulation from REDCap. If the function could be made more general it would be great as it would help all the users using multiple packages to analyze the data.
Simply adding a branch for when the data is a data.frame would help a lot.
In this branch the code handling data like this could avoid looking for a labelled data.frame.

[RaymondBalise]

Thanks for the link to your blog. That is a useful summary. It mentions the need to specify the first and last variables in tidyREDCap. We greatly improved that in the new make_instruments() function. You just point to your REDCap project and it pulls in all the tables/instruments.

We are debating how much support we should supply for the manual/point-and-click export (instead of the API export). We will add your question/comment into the mix.

For the make_choose_all_table() function, if the data is not labeled, what would you use to "label" the responses? Would you use the variable names?

[llrs]

I'm glad the summary is useful and many thanks for the improvements! It will make it easier to work with it. I missed the release note, I'll update the package tomorrow (aka, I'm using version 0.2.2).

If the input data is not labeled you can use the characters or the labels of the factor. If there aren't it might be important to simply warn or stop (with an informative message like: "There is no data in the records"), as probably the user isn't expecting that too. Maybe make_choose_* functions are applied to questions that are not compulsory and might be always empty, so an option to disable the warn/stop can be useful too.

[RaymondBalise]

I just did some checks. The code will almost work with the manual export. The manual export includes both factor and not factor versions of the categorical variables. If the data is subset down to the the not .factor versions of a variable it does the counts but the labels are lost:

# get choose all ingredient list variables but not the factor versions
data |> select(starts_with("ingr")) |> select(!ends_with("or"))-> ingredients
make_choose_all_table(ingredients, "ingr")

We will need to mess around with the code that processes the labels to handle both the REDCapR labels and the manual export labels. I am adding this to the version 1.2 release milestones that we are working on.

[RaymondBalise]

We need to tweak

tidyREDCap:::getLabel2
function (data, aVariable) 
{
    variable <- {
        {
            data
        }
    }[aVariable]
    theLab <- dropTags(attributes(variable[, aVariable])$label)
   
###
# Check theLab for `=` or `:` 
# ERROR if both
# if `=` then parse with that (manual export)
# if `:` then parse with that (REDCapR export)
### 

   stringr::str_sub(theLab, stringr::str_locate(theLab, ":")[, 
        1] + 2, nchar(theLab))
}

Note: these are old functions that should be switched from camel to snake case

[RaymondBalise]

I just fixed the function so it will work on the manual export or REDCapR export format. It will need to tweak the vignette to explain this logic:

manual_export |>   # this is the data produced by exporting using the manual export
                 select(starts_with("ingr")) |>   # get all the ingredient variables
                 select(-ends_with("or")) |>   # drop the factor version of the ingredient variables
                 make_choose_all_table("ingr")  # make the table

[llrs]

Thank you for looking into this so quickly. I tried with the latest version in github in the manual_import branch (commit 85c8930 where you referenced this issue) and make_choose_all_table doesn't work or produce the expected error.

For a more longer explanation about how I get the data:

ds <- REDCapR::redcap_read(redcap_uri = "https://redcap.server.org/api/",
                  token = Sys.getenv("token_API"))$data
tidyREDCap::make_choose_all_table(ds, "ma__")
#> Error in `purrr::map_chr()`:
#> ℹ In index: 1.
#> Caused by error:
#> ! Result must be length 1, not 0.

As you can see the REDCapR function returns a data.frame without labelled data:

grep("ma__", colnames(ds), value = TRUE)
#> [1] "ma___1" "ma___2" "ma___3" "ma___4"
ds[, grepl("ma__", colnames(ds))] |> str()
#> 'data.frame':	8 obs. of  4 variables:
#>  $ ma___1: logi  NA NA NA NA NA NA ...
#>  $ ma___2: logi  NA NA NA NA NA NA ...
#>  $ ma___3: logi  NA NA NA NA NA NA ...
#>  $ ma___4: logi  NA NA NA NA NA NA ...

[RaymondBalise]

That helps me understand what is going on. I didn't realize you were pulling with REDCapR directly without labels. I thought this was a scenario where you only had permission to do a manual export.

Given that you have permission to use the API, please try using tidyREDCap::import_instruments(). It wraps around the REDCapR API and gives additional functionality (like the labels).

[llrs]

Using import_instrument it works:

a <- tidyREDCap::import_instruments(url = "https://redcap.server.org/api/",
                  token = Sys.getenv("token_API"))
#> Reading metadata about your project....
#> Reading variable labels for your variables....
#> Reading your data....                                                                                                                                       
#> ℹ This may take a while if your dataset is large.
#> Error:                                                                                                                                                      
#>          The first variable in df must be `record_id`;
#>          use option 'record_id=' to set the name of your custom id.
a <- tidyREDCap::import_instruments(url = "https://redcap.server.org/api/",
                  token = Sys.getenv("token_API"),
                  record_id = "participant_id")
#> Reading metadata about your project....
#> Reading variable labels for your variables....
#> Reading your data....                                                                                                                                       
#> ℹ This may take a while if your dataset is large.
a
#> NULL
tidyREDCap::make_choose_all_table(survey, "ma")
#> # A tibble: 4 × 2
#>   What                       Count
#>   <chr>                      <dbl>
#> 1 Choice One                     0
#> 2 Choice Two                     0
#> 3 Choice Three                   0
#> 4 Select as many as you like     0

I'm closing the issue. Thanks!

[RaymondBalise]

@llrs
I am very happy to hear that it worked as it should. Please continue to spread the word on tidyREDCap. I hate to see people getting stuck on the last steps of a project and I fear too many people don't know that help is a download away.