### Install DataQualityDashboard from GitHub Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/README.md Install the DataQualityDashboard package directly from its GitHub repository. This requires the 'remotes' package to be installed first. ```r install.packages("remotes") remotes::install_github("OHDSI/DataQualityDashboard") ``` -------------------------------- ### Install http-server Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/docs/articles/DataQualityDashboard.html Installs the http-server package globally using npm. This is required for launching the dashboard on a web server. ```bash npm install -g http-server ``` -------------------------------- ### Run http-server Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/docs/articles/DataQualityDashboard.html Starts a local web server in the specified directory. Navigate to the directory containing the _results.json file before running this command. ```bash http-server ``` -------------------------------- ### Install Local Dev Version Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/CLAUDE.md Command to install the local development version of the R package. ```r # Install local dev version devtools::install() ``` -------------------------------- ### Install DataQualityDashboard from CRAN Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/README.md Install the DataQualityDashboard package from the Comprehensive R Archive Network (CRAN). Ensure your R environment is properly configured. ```r install.packages("DataQualityDashboard") ``` -------------------------------- ### R Script for SQL-Only Data Quality Dashboard Execution Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/docs/articles/SqlOnly.html This R script demonstrates the complete workflow for running the Data Quality Dashboard using only SQL files. It covers database connection setup, creating the results table, executing individual SQL checks, and exporting results to JSON. ```r library(DatabaseConnector) cdmSourceName <- "" sqlFolder <- "./results_sql_only" jsonOutputFolder <- sqlFolder jsonOutputFile <- "sql_only_results.json" dbms <- Sys.getenv("DBMS") server <- Sys.getenv("DB_SERVER") port <- Sys.getenv("DB_PORT") user <- Sys.getenv("DB_USER") password <- Sys.getenv("DB_PASSWORD") pathToDriver <- Sys.getenv("PATH_TO_DRIVER") connectionDetails <- DatabaseConnector::createConnectionDetails( dbms = dbms, server = server, port = port, user = user, password = password, pathToDriver = pathToDriver ) cdmDatabaseSchema <- '' resultsDatabaseSchema <- '' writeTableName <- 'dqd_results' # or whatever you want to name your results table c <- DatabaseConnector::connect(connectionDetails) # Create results table ddlFile <- file.path(sqlFolder, "ddlDqdResults.sql") DatabaseConnector::renderTranslateExecuteSql( connection = c, sql = readChar(ddlFile, file.info(ddlFile)$size), resultsDatabaseSchema = resultsDatabaseSchema, writeTableName = writeTableName ) # Run checks dqdSqlFiles <- Sys.glob(file.path(sqlFolder, "*.sql")) for (dqdSqlFile in dqdSqlFiles) { if (dqdSqlFile == ddlFile) { next } print(dqdSqlFile) tryCatch( expr = { DatabaseConnector::renderTranslateExecuteSql( connection = c, sql = readChar(dqdSqlFile, file.info(dqdSqlFile)$size), cdmDatabaseSchema = cdmDatabaseSchema, resultsDatabaseSchema = resultsDatabaseSchema, writeTableName = writeTableName ) }, error = function(e) { print(sprintf("Writing table failed for check %s with error %s", dqdSqlFile, e$message)) } ) } # Extract results table to JSON file for viewing or secondary use DataQualityDashboard::writeDBResultsToJson( c, connectionDetails, resultsDatabaseSchema, cdmDatabaseSchema, writeTableName, jsonOutputFolder, jsonOutputFile ) jsonFilePath <- R.utils::getAbsolutePath(file.path(jsonOutputFolder, jsonOutputFile)) DataQualityDashboard::viewDqDashboard(jsonFilePath) ``` -------------------------------- ### Configure Database Connection Details Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/vignettes/DataQualityDashboard.rmd Set up the connection parameters for your database. Ensure all necessary fields are populated according to your environment. ```r connectionDetails <- DatabaseConnector::createConnectionDetails( dbms = "", user = "", password = "", server = "", port = "", extraSettings = "", pathToDriver = "" ) ``` -------------------------------- ### Build and Check Package Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/CLAUDE.md Commands to build and check the R package for development and distribution. ```r # Build and check devtools::build() devtools::check() R CMD check --as-cran --no-manual . ``` -------------------------------- ### Output Configuration Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/docs/articles/DataQualityDashboard.html Specifies the folder and file name for storing DQD results. ```R outputFolder <- "output" outputFile <- "results.json" ``` -------------------------------- ### Investigate High Condition Start Dates Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/docs/articles/checks/plausibleValueHigh.html Calculate the median difference in days between condition_start_date and current_date for future dates to identify potential issues. ```sql SELECT MEDIAN(DATEDIFF(day, condition_start_date, current_date)) FROM condition_occurrence WHERE condition_start_date > current_date ; ``` -------------------------------- ### Violated Rows Query Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/docs/articles/checks/plausibleStartBeforeEnd.html This SQL query identifies rows where the start date is after the end date, indicating a potential data quality issue. It selects the violating field and the entire row from the specified CDM table. ```sql SELECT '@cdmTableName.@cdmFieldName' AS violating_field, cdmTable.* FROM @schema.@cdmTableName cdmTable WHERE cdmTable.@cdmFieldName IS NOT NULL AND cdmTable.@plausibleStartBeforeEndFieldName IS NOT NULL AND cdmTable.@cdmFieldName > cdmTable.@plausibleStartBeforeEndFieldName ``` -------------------------------- ### Example SQL Query for ER Visit Length Check Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/docs/articles/AddNewCheck.html This SQL query specifically checks for ER visits where the duration between visit_start_date and visit_end_date exceeds 2 days. It demonstrates how to use input parameters and alias the count for violated rows. ```sql SELECT count(*) AS num_violated_rows FROM @cdmDatabaseSchema.@cdmTableName WHERE visit_concept_id = 9203 /* ER visit */ AND dateDiff(days, visit_start_date, visit_end_date) > 2 ``` -------------------------------- ### Query for Violated Rows Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/docs/articles/checks/withinVisitDates.html This SQL query identifies records where the event date is more than 7 days before the visit start date or more than 7 days after the visit end date. It requires specifying the CDM schema, table name, and the date field to check. ```sql SELECT '@cdmTableName.@cdmFieldName' AS violating_field, vo.visit_start_date, vo.visit_end_date, vo.person_id, cdmTable.* FROM @cdmDatabaseSchema.@cdmTableName cdmTable JOIN @cdmDatabaseSchema.visit_occurrence vo ON cdmTable.visit_occurrence_id = vo.visit_occurrence_id WHERE cdmTable.@cdmFieldName < dateadd(day, -7, vo.visit_start_date) OR cdmTable.@cdmFieldName > dateadd(day, 7, vo.visit_end_date) ``` -------------------------------- ### Launch Log Viewer Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/docs/articles/DataQualityDashboard.html Launches the log viewer to inspect the dashboard execution logs. Ensure the outputFolder and cdmSourceName are correctly defined. ```r ParallelLogger::launchLogViewer(logFileName = file.path(outputFolder, cdmSourceName, sprintf("log_DqDashboard_%s.txt", cdmSourceName))) ``` -------------------------------- ### Configure Connection Details for Data Quality Checks Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/docs/articles/DataQualityDashboard.html Set up connection details to your CDM database. This includes database type, credentials, server information, and schema names. The CDM version targeted is also specified. ```r # fill out the connection details ----------------------------------------------------------------------- connectionDetails <- DatabaseConnector::createConnectionDetails( dbms = "", user = "", password = "", server = "", port = "", extraSettings = "", pathToDriver = "" ) cdmDatabaseSchema <- "yourCdmSchema" # the fully qualified database schema name of the CDM resultsDatabaseSchema <- "yourResultsSchema" # the fully qualified database schema name of the results schema (that you can write to) cdmSourceName <- "Your CDM Source" # a human readable name for your CDM source cdmVersion <- "5.4" # the CDM version you are targetting. Currently supports 5.2, 5.3, and 5.4 ``` -------------------------------- ### Configure Execution Parameters for Data Quality Checks Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/docs/articles/DataQualityDashboard.html Specify the number of threads for concurrent SQL sessions and whether to execute queries directly or only generate the SQL scripts. Incremental insert option is also available. ```r # determine how many threads (concurrent SQL sessions) to use ---------------------------------------- numThreads <- 1 # on Redshift, 3 seems to work well # specify if you want to execute the queries or inspect them ------------------------------------------ sqlOnly <- FALSE # set to TRUE if you just want to get the SQL scripts and not actually run the queries sqlOnlyIncrementalInsert <- FALSE # set to TRUE if you want the generated SQL queries to calculate DQD results and insert them into a database table (@resultsDatabaseSchema.@writeTableName) ``` -------------------------------- ### Redshift Bulk Loading Configuration (Commented Out) Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/docs/articles/DataQualityDashboard.html Environment variables for configuring bulk loading in Redshift when writing to a table. ```R # Sys.setenv("AWS_ACCESS_KEY_ID" = "", # "AWS_SECRET_ACCESS_KEY" = "", # "AWS_DEFAULT_REGION" = "", # "AWS_BUCKET_NAME" = "", # "AWS_OBJECT_KEY" = "", # "AWS_SSE_TYPE" = "AES256", # "USE_MPP_BULK_LOAD" = TRUE) ``` -------------------------------- ### executeDqChecks Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/docs/reference/executeDqChecks.html Connects to the database, generates SQL scripts, and runs data quality checks. Results are written to a JSON file and a database table by default. ```APIDOC ## executeDqChecks ### Description This function will connect to the database, generate the sql scripts, and run the data quality checks against the database. By default, results will be written to a json file as well as a database table. ### Function Signature ```r executeDqChecks( connectionDetails, cdmDatabaseSchema, resultsDatabaseSchema, vocabDatabaseSchema = cdmDatabaseSchema, cdmSourceName, numThreads = 1, sqlOnly = FALSE, sqlOnlyUnionCount = 1, sqlOnlyIncrementalInsert = FALSE, outputFolder, outputFile = "", verboseMode = FALSE, writeToTable = TRUE, writeTableName = "dqdashboard_results", writeToCsv = FALSE, csvFile = "", checkLevels = c("TABLE", "FIELD", "CONCEPT"), checkNames = c(), checkSeverity = c("fatal", "convention", "characterization"), cohortDefinitionId = c(), cohortDatabaseSchema = resultsDatabaseSchema, cohortTableName = "cohort", tablesToExclude = c("CONCEPT", "VOCABULARY", "CONCEPT_ANCESTOR", "CONCEPT_RELATIONSHIP", "CONCEPT_CLASS", "CONCEPT_SYNONYM", "RELATIONSHIP", "DOMAIN"), cdmVersion = "5.3", tableCheckThresholdLoc = "default", fieldCheckThresholdLoc = "default", conceptCheckThresholdLoc = "default" ) ``` ### Arguments * **connectionDetails** (object) - A connectionDetails object for connecting to the CDM database * **cdmDatabaseSchema** (string) - The fully qualified database name of the CDM schema * **resultsDatabaseSchema** (string) - The fully qualified database name of the results schema * **vocabDatabaseSchema** (string) - The fully qualified database name of the vocabulary schema (default is to set it as the cdmDatabaseSchema) * **cdmSourceName** (string) - The name of the CDM data source * **numThreads** (integer) - The number of concurrent threads to use to execute the queries * **sqlOnly** (boolean) - Should the SQLs be executed (FALSE) or just returned (TRUE)? * **sqlOnlyUnionCount** (integer) - * **sqlOnlyIncrementalInsert** (boolean) - * **outputFolder** (string) - * **outputFile** (string) - * **verboseMode** (boolean) - * **writeToTable** (boolean) - * **writeTableName** (string) - * **writeToCsv** (boolean) - * **csvFile** (string) - * **checkLevels** (array) - * **checkNames** (array) - * **checkSeverity** (array) - * **cohortDefinitionId** (array) - * **cohortDatabaseSchema** (string) - * **cohortTableName** (string) - * **tablesToExclude** (array) - * **cdmVersion** (string) - * **tableCheckThresholdLoc** (string) - * **fieldCheckThresholdLoc** (string) - * **conceptCheckThresholdLoc** (string) - ``` -------------------------------- ### Run All Tests Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/CLAUDE.md Command to execute all tests within the R package. ```r # Run all tests devtools::test() ``` -------------------------------- ### Execute DQ Checks with sqlOnly=TRUE Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/tests/testthat/_snaps/executeDqChecks.md This snippet demonstrates executing DQ checks with `sqlOnly=TRUE`, `sqlOnlyUnionCount=4`, and `sqlOnlyIncrementalInsert=TRUE`. It selects check results and inserts them into the `dqdashboard_results` table. Note that SQL errors and performance are not included. ```sql WITH cte_all AS ( SELECT cte.num_violated_rows ,cte.pct_violated_rows ,cte.num_denominator_rows ,'' as execution_time ,'' as query_text ,'measurePersonCompleteness' as check_name ,'TABLE' as check_level ,'The number and percent of persons in the CDM that do not have at least one record in the SPECIMEN table' as check_description ,'SPECIMEN' as cdm_table_name ,'NA' as cdm_field_name ,'NA' as concept_id ,'NA' as unit_concept_id ,'table_person_completeness.sql' as sql_file ,'Completeness' as category ,'NA' as subcategory ,'Validation' as context ,'' as warning ,'' as error ,'table_measurepersoncompleteness_specimen' as checkid ,0 as is_error ,0 as not_applicable ,CASE WHEN (cte.pct_violated_rows * 100) > 100 THEN 1 ELSE 0 END as failed ,CASE WHEN (cte.pct_violated_rows * 100) <= 100 THEN 1 ELSE 0 END as passed ,NULL as not_applicable_reason ,100 as threshold_value ,NULL as notes_value FROM ( SELECT num_violated_rows, CASE WHEN denominator.num_rows = 0 THEN 0 ELSE 1.0*num_violated_rows/denominator.num_rows END AS pct_violated_rows, denominator.num_rows AS num_denominator_rows FROM ( SELECT COUNT_BIG(violated_rows.person_id) AS num_violated_rows FROM ( /*violatedRowsBegin*/ SELECT cdmTable.* FROM @yourCdmSchema.person cdmTable LEFT JOIN @yourCdmSchema.SPECIMEN cdmTable2 ON cdmTable.person_id = cdmTable2.person_id WHERE cdmTable2.person_id IS NULL /*violatedRowsEnd*/ ) ) violated_row_count, ( SELECT COUNT_BIG(*) AS num_rows FROM @yourCdmSchema.person cdmTable ) denominator ) cte ) ) INSERT INTO @yourResultsSchema.dqdashboard_results SELECT * FROM cte_all ; ``` -------------------------------- ### SQL Query Template with Parameters Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/vignettes/AddNewCheck.rmd A template for formatting SQL queries to be compatible with the Data Quality Dashboard. It includes placeholders for database schema, table, and field names, and defines the expected output fields. ```sql SELECT num_violated_rows , CASE WHEN denominator.num_rows = 0 THEN 0 ELSE 1.0*dqd_check.num_violated_rows/denominator.num_rows END AS pct_violated_rows, denominator.num_rows as num_denominator_rows FROM ( ) dqd_check CROSS JOIN ( SELECT COUNT_BIG(*) AS num_rows FROM @cdmDatabaseSchema.@cdmTableName cdmTable ) denominator; ``` -------------------------------- ### viewDqDashboard Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/docs/reference/viewDqDashboard.html Launches the Data Quality Dashboard Shiny application to visualize DQ check results. This function requires a JSON file path generated by `executeDqChecks`. ```APIDOC ## viewDqDashboard ### Description Launches the Data Quality Dashboard Shiny application to visualize DQ check results. This function requires a JSON file path generated by `executeDqChecks`. ### Function Signature `viewDqDashboard(jsonPath, launch.browser = NULL, display.mode = NULL, ...)` ### Parameters #### Path Parameters * **jsonPath** (string) - Required - The fully-qualified path to the JSON file produced by `executeDqChecks`. #### Other Parameters * **launch.browser** (boolean/string) - Optional - Passed on to `shiny::runApp`. Determines if the app should launch in a browser. * **display.mode** (string) - Optional - Passed on to `shiny::runApp`. Controls the display mode of the Shiny application. * **...** - Optional - Extra parameters for `shiny::runApp()` like "port" or "host". ### Value NULL (Launches Shiny application) ``` -------------------------------- ### Execute DQD with Custom Thresholds Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/docs/articles/Thresholds.html Use this function call to run the Data Quality Dashboard with specified locations for custom threshold files. Ensure the `connectionDetails`, `cdmDatabaseSchema`, `resultsDatabaseSchema`, and `cdmSourceName` are correctly set. The threshold location parameters should point to the fully qualified paths of your custom threshold files. ```r DataQualityDashboard::executeDqChecks(connectionDetails = connectionDetails, cdmDatabaseSchema = cdmSchema, resultsDatabaseSchema = resultsSchema, cdmSourceName = cdmSourceName, numThreads = numThreads, sqlOnly = sqlOnly, outputFolder = outputFolder, verboseMode = verboseMode, writeToTable = writeToTable, checkLevels = checkLevels, tablesToExclude = tablesToExclude, checkNames = checkNames, tableCheckThresholdLoc = location of the table check file, fieldCheckThresholdLoc = location of the field check file, conceptCheckThresholdLoc = location of the concept check file) ``` -------------------------------- ### Execute Data Quality Checks Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/vignettes/DataQualityDashboard.rmd Run the main data quality check function with specified connection details, schemas, source name, version, and output options. This is the core function to initiate the DQD process. ```R DataQualityDashboard::executeDqChecks(connectionDetails = connectionDetails, cdmDatabaseSchema = cdmDatabaseSchema, resultsDatabaseSchema = resultsDatabaseSchema, cdmSourceName = cdmSourceName, cdmVersion = cdmVersion, numThreads = numThreads, sqlOnly = sqlOnly, sqlOnlyUnionCount = sqlOnlyUnionCount, sqlOnlyIncrementalInsert = sqlOnlyIncrementalInsert, outputFolder = outputFolder, outputFile = outputFile, verboseMode = verboseMode, writeToTable = writeToTable, writeToCsv = writeToCsv, csvFile = csvFile, checkLevels = checkLevels, checkSeverity = checkSeverity, tablesToExclude = tablesToExclude, checkNames = checkNames) ``` -------------------------------- ### Execute DQ Checks for DEVICE_EXPOSURE Table Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/tests/testthat/_snaps/executeDqChecks.md SQL query to check the completeness of the DEVICE_EXPOSURE table. It calculates the number and percentage of persons without at least one record in the table. ```sql SELECT num_violated_rows, CASE WHEN denominator.num_rows = 0 THEN 0 ELSE 1.0*num_violated_rows/denominator.num_rows END AS pct_violated_rows, denominator.num_rows AS num_denominator_rows FROM ( SELECT COUNT_BIG(violated_rows.person_id) AS num_violated_rows FROM ( /*violatedRowsBegin*/ SELECT cdmTable.* FROM @yourCdmSchema.person cdmTable LEFT JOIN @yourCdmSchema.DEVICE_EXPOSURE cdmTable2 ON cdmTable.person_id = cdmTable2.person_id WHERE cdmTable2.person_id IS NULL /*violatedRowsEnd*/ ) violated_rows ) violated_row_count, ( SELECT COUNT_BIG(*) AS num_rows FROM @yourCdmSchema.person cdmTable ) denominator ) cte ) INSERT INTO @yourResultsSchema.dqdashboard_results SELECT * FROM cte_all ; ``` -------------------------------- ### Launch Log Viewer Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/vignettes/DataQualityDashboard.rmd Open the ParallelLogger log viewer to inspect the execution logs of the Data Quality Dashboard. This helps in debugging and understanding the process. ```R ParallelLogger::launchLogViewer(logFileName = file.path(outputFolder, cdmSourceName, sprintf("log_DqDashboard_%s.txt", cdmSourceName)))) ``` -------------------------------- ### Run Single Test File Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/CLAUDE.md Command to run a specific test file in the R package. ```r # Run a single test file testthat::test_file("tests/testthat/test-executeDqChecks.R") ``` -------------------------------- ### Investigate Source Concept ID Mapping Failures Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/docs/articles/checks/sourceConceptRecordCompleteness.html Run this SQL query to identify source codes that failed to map to an OMOP concept. Inspect the results to understand the scope of mapping issues. ```sql SELECT concept.concept_name AS standard_concept_name, cdmTable._concept_id, -- standard concept ID field for the table c2.concept_name AS source_value_concept_name, cdmTable._source_value, -- source value field for the table COUNT(*) FROM @cdmDatabaseSchema.@cdmTableName cdmTable LEFT JOIN @vocabDatabaseSchema.concept ON concept.concept_id = cdmTable._concept_id -- WARNING this join may cause fanning if a source value exists in multiple vocabularies LEFT JOIN @vocabDatabaseSchema.concept c2 ON concept.concept_code = cdmTable._source_value AND c2.domain_id = WHERE cdmTable.@cdmFieldName = 0 GROUP BY 1,2,3 ORDER BY 4 DESC ``` -------------------------------- ### Define Database Schemas and Source Name Source: https://github.com/ohdsi/dataqualitydashboard/blob/main/vignettes/DataQualityDashboard.rmd Specify the fully qualified names for your CDM and results schemas, and provide a human-readable name for your CDM source. Also, specify the CDM version being targeted. ```r cdmDatabaseSchema <- "yourCdmSchema" # the fully qualified database schema name of the CDM resultsDatabaseSchema <- "yourResultsSchema" # the fully qualified database schema name of the results schema (that you can write to) cdmSourceName <- "Your CDM Source" # a human readable name for your CDM source cdmVersion <- "5.4" # the CDM version you are targetting. Currently supports 5.2, 5.3, and 5.4 ```