sassoftware/ci360-download-client-sas

ci360 data download client program using Base SAS

ci360-download-client-sas's Issues

Truncation When Importing Data into SAS Programs

All data in the Unified Data Model (UDM) and Common Data Model (CDM) is stored in UTF-8 encoding, and the schema defines the length of columns based on characters. However, the SAS language does not natively support the VARCHAR data type.

If you use the download clients to download the data tables, some data might be truncated if your SAS data set does not have enough space to hold the values.

For a workaround, you can configure your SAS data sets with enough space to store any data from the tables in the UDM and CDM.

When you clone the scripts for the Download client, you can edit the sample code (in particular, dsc_create_attrib.sas and dsc_schema_sync.sas files) to add a buffer to account for the extra characters.

As a simple example for the MD_TASK table's task_nm column, here's a way to add a 50% buffer:

if( data_type= 'varchar' or data_type= 'char' )then do;
   if column_name='TASK_NM' then do;
      data_length_new=round(data_length*1.5,1);
      put @ident2 column_name 'FORMAT=$' data_length_new +(-1) '.';
   end;
   else put @ident2 column_name 'FORMAT=$' data_length +(-1) '.';
end;