2017 ICH CAHPS Fall Survey Site Visit Issues & 2017 ICH CAHPS Data Quality Reminders
folder_openOther Announcementscalendar_todayPosted January 22, 2018

As we approach the deadline for submitting data from the 2017 ICH CAHPS Fall Survey and the beginning of the 2018 Spring Survey, it is a good time to remind vendors of the importance of implementing quality control measures on all aspects of the survey.

This announcement includes a list of some of the data quality issues that the Coordination Team (CT) detected in the data from the 2017 Spring Survey and issues we observed during site visits to survey vendors during the 2017 Fall Survey. As you conduct final data cleaning and processing of 2017 Fall Survey data and prepare the XML file that will be submitted, please apply appropriate quality control measures so that you can detect and avoid these errors and data problems in all upcoming survey periods.

As a reminder, the XML files containing data from the 2017 Fall Survey must be submitted to the ICH CAHPS Data Center by 11:59 PM on January 31, 2018. Please contact the ICH CAHPS Coordination Team via email at ichcahps@rti.org or call (866) 245-8083 if you have any questions about any of the information provided below.

Data Quality Reminders

1. Check Assigned Disposition Codes. Survey vendors should make sure that the correct final disposition codes are assigned to all sample cases. Within 2 to 3 weeks after the data collection period begins and then after data collection ends, survey vendors should generate and review the interim and final disposition codes assigned to a sample of cases. Once final disposition codes are assigned, vendors should have a staff member (other than the one who assigned the disposition codes) conduct quality control on the disposition codes assigned. Note that the CT conducts quality control on each vendor’s data files after they are submitted. The following are some of the issues/errors that we found in data that were submitted for the 2017 ICH CAHPS Spring Survey period:
   1. Code 120 was incorrectly assigned to sample members who are ineligible for the survey. Code 120 should not be assigned if the answer to Q1 = 1 (at home) or 3 (am not currently receiving dialysis), or if the answer to Q2 = 1 (less than 3 months) or 5 (no longer receives dialysis at this facility).
   2. Code 160 was assigned to some mail survey cases that were completed by a proxy respondent. Remember that for all completed mail survey cases, vendors must first check the answer to Q62 (How did that person help you?) before assigning a final code to the case. If “Answered the questions for me” (response option 3) is marked in Q62, the case was completed by a proxy respondent. Proxy respondents are not allowed on the ICH CAHPS Survey. All cases completed by a proxy respondent must be assigned Code 199.
   3. Codes 140 (ineligible, no longer receiving dialysis), 160 (ineligible, receive dialysis at home or at facility for less than then three months or on hospice or institutionalized/nursing home), and 190 (ineligible, no longer receiving dialysis at sampled facility) should be assigned based on a respondent’s answers to Q1 and Q2, or via a note or phone call to the vendor.
   4. Code 130 should be assigned to a mail case if the respondent answered one or more of questions in Questions 2 through 44 AND one or more of the following applies:
• Q1—The answer to Q1 is “Receive dialysis care at home.”
• Q1—The answer to Q1 is “I am not currently receiving dialysis.”
• Q2—The answer to Q2 is “Less than 3 months.”
• Q2—The answer to Q2 is “No longer receives dialysis at this facility.”
• Q1 is blank.
• Q2 is blank.
• Q1and Q2 are both blank.
• Q1 and Q2 both indicate ineligibility
• Also assign Code 130 if both Q1 and Q2 are blank but then skipped to Q45.
2. Check to make sure that all completed survey cases meet the completeness criteria before assigning Code 110 (for completed mail survey cases) or Code 120 (for completed telephone interview cases). In addition, investigate cases that do meet the completeness criteria, but your program assigns a non-complete final code.
3. Make sure that all cases assigned Code 110 and 120 have survey data on the XML file. There were some cases on some of the data files submitted that were assigned Code 110 or 120 (completed survey) that did not have any survey response data. During the data collection period, survey vendors should check their raw data files to make sure that survey response data were captured via scanning or data entry. In addition, they should check telephone survey data files to make sure that survey response data is captured correctly.

   When preparing the XML files, vendors must compare the survey response data for a sample of cases assigned Code 110 with the survey responses on the hardcopy mail survey questionnaire to make sure the survey response data was exported correctly from the raw data files. Vendors must ensure that all cases assigned final code 110, 120, 130, or 210 have data (at least partial data for a 130 or 210). If not, vendors must investigate to determine whether an incorrect disposition code was assigned or whether data exist and did not get processed correctly. 

4. Check Data Collection and Data Processing Systems and Computer Programs. Survey vendors should check computer programs and systems used on the ICH CAHPS Survey prior to and periodically throughout each data collection period, to ensure that all programs are working correctly. Computer-assisted telephone interview (CATI) and data capture programs (data entry and scanning) should especially be checked on a periodic basis to ensure that programmed skip logic is working accurately. The following are just a couple examples of situations in which the quality of the data might be affected because computer system/programs are not working properly.
• Example 1: It was observed that a respondent had mentioned more than 1 race category in response to the race question, but only one race category was marked for that case on the XML file.
• Example 2: There was an error in the CATI skip logic, which resulted in a response being automatically applied for a question that the respondent never received.
5. Verify that the correct final disposition code has been assigned to each sample case. We observed that some vendors had discrepancies between the disposition code listed in their data system and the code entered in the XML file submitted to the Data Center. Survey vendors are required to check to make sure that the correct final disposition code has been assigned to each sample case. Vendors should ensure that the assigned final disposition code is correct and that the data included in the XML file are accurate based on the final code.
6. The correct use of Code X vs. Code M. Prior to submission of XML files, review data on each XML file to make sure that Code M and Code X are appropriately assigned. As a reminder, when follow-up questions are appropriately skipped by the respondent, the follow-up question response should be coded as “Not Applicable,” which is Code X. When follow-up questions should have been skipped (based on the response to the screening question) but are answered, scan or key the response that the respondent provides. If a question should have been answered but was not, assign Code M for “Missing” to the survey item.
7. Guidelines for coding both multiple responses and ambiguous responses. If the respondent marked two or more response options for a single-response option question, when coding the responses select the one that appears darkest. If it is not possible to make that determination, leave the response blank and code it as “Missing” (Code M) rather than guess. In addition, all responses or response boxes that are not circled, checked, underlined, or in some other way clearly designated by the respondent as his or her intended response must be coded M for “Missing”.

Other Observations during Site Visits

8. Obtain approval for any vendor-specific (or facility-specific) supplemental questions. During site visits we observed that some vendors had included questions in their surveys that had not been pre-approved by CMS. Remember that vendors must submit all vendor- or facility-specific supplemental questions to the CT for approval by CMS before they are added to the ICH CAHPS Survey.
9. Make sure mail survey cover letters include all required language of informed consent and meet ICH CAHPS requirements. Some vendors’ cover letters did not include required ICH CAHPS elements. Vendors should check all of their survey materials to ensure that all added questions/screens have been approved by CMS, and that all materials meet ICH CAHPS requirements and specifications. Remember that vendors must submit an Exceptions Request Form (ERF) for any survey materials that deviate from protocol and that do not currently have CMS approval.
10. Administer the correct version of the mail survey, the telephone survey, and other survey materials. Survey vendors must make sure that they are using the correct version of the ICH CAHPS mail survey and telephone interview script. During site visits we observed that some vendors had not implemented the changes that were included in the version of the mail survey and telephone script posted on the ICH CAHPS website. 
11. Make sure interviewers are trained on correct probing techniques. Survey vendors must train telephone interviewers on how and when to probe a respondent for more information or to clarify a response. Make sure that telephone interviewers do not probe if the respondent has already given a response to the question and the response given is clear. The Coordination Team observed some telephone interviewers who probed eligible respondents in a manner where they were recorded as ineligible. Telephone interview training should include a section on the use of effective neutral or nondirective probing techniques. Such techniques will assist the interviewer with obtaining a more complete or more specific answer from a respondent, when needed, without the unintended consequence of suggesting answers or leading the respondent.
12. Make attempts to contact a sample patient on different days of the week and at different times of the day. During the site visits we noted that some vendors are not making telephone call attempts during the evening hours while other vendors are not making attempts during the morning hours. In addition, some vendors were not contacting sample patients on weekend days, or were ending all call attempts well before the data collection period ended. Survey vendors must make a maximum of 10 telephone contact attempts for each sample patient, unless the sample patient refuses or the survey vendor learns that the sample patient is ineligible to participate in the survey. The 10 contact attempts must be made on different days of the week and at different times of the day, including nights and mornings, and spread over the data collection period.