Striving for interrater reliability with a patient counseling assessment rubric: Lessons learned

•University faculty attempted to develop and validate a patient counseling skills rubric.•Validation was unsuccessful due to several factors.•Although rubric validation was not achieved, several lessons were learned during this process.