"O would some power the giftie gie us to see ourselves as others see us."

Scottish poet Robert Burns wrote those words over 200 years ago. Today, through 360° Feedback programs, organizations have the ability to give that gift to their employees - the gift of seeing themselves through the eyes of others. The potential of this gift excites many and terrifies others. Crafting good 360° content is critical to fulfilling the promise of the program and calming any fears it creates.

While off-the-shelf instruments exist, many organizations prefer to develop their own 360°s. This approach takes time and care, but having content customized to your organization's values and key success factors may ultimately yield the best results. If this is the route you choose, here are some guidelines to make development easier.

Start with a good core competency model

A good core competency model is one that taps into your organizational goals and values. What would make someone successful in your organization? Although there are some "standard" competencies across all leadership models, different organizations often place different value on them (e.g., creativity may be more important in advertising than in accounting). And organizations often have unique priorities, e.g., one of our clients focuses heavily on "diversity" while another promotes "continuous learning." Determining the competencies that are most important to your success and values will help ensure a meaningful 360° tool.

It is very important to align your 360° content with your competency model. Results will only be meaningful to individuals if they can see the relationship between what they are being rated on, their job, and the organization's goals and values. Competency models can be complex; developing a good core competency model takes time and effort. If you don't have one, have an expert - like an I/O Psychologist - work with you to develop one.

Determine your focus

How you decide to use 360° Feedback should determine the details of your program. You likely had a goal in mind for your program - to develop managerial skills, to provide information for a performance management system, to identify individuals with "high potential," etc.

Start with your identified competencies; then pick and choose the behaviors on which you want to focus depending on your program objectives. For instance, if your goal is skill development, your focus is likely to be on what individuals DO and how well they do it. If your goal is identification of High Potentials, emphasis may be more on individuals' ability to learn, grow and adapt.

Is it all right to have more than one objective? Yes - but be aware that some goals may actually conflict; research shows that raters respond differently to 360°s that are used for development versus 360°s that are used as part of someone's performance evaluation. Combining these objectives into one survey will be difficult. Stick with one objective; the more focused your goal, the more likely you are to achieve it.

Target your participant group

It's a rare organization that has everyone go through 360° Feedback. Most often the target group is managers at various levels, although including individual contributors is not uncommon. It's very important to decide early on who will be participating in the program, because that has a direct bearing on content.

People should be assessed on behaviors they can logically be expected to exhibit on the job. For example, senior managers typically determine strategic direction and goals; lower-level managers communicate and drive action to achieve them. Asking people to rate a first-level supervisor on strategic goal-setting may result in a lot of missing data or "questionable" ratings. Individual contributors may not be expected to coach others. On the other hand, everyone may be expected to listen actively or be accountable for their decisions. 

Determine survey length

Say your goal is to provide feedback on a recurring basis (e.g., every 1-2 years). In order to gain buy-in, avoid resistance, and provide a manageable amount of data, the number of items should be limited. However, if your 360° is part of a one-time, in-depth development program, participants will expect to be given a wealth of information, so a longer survey will be expected.

The majority of programs that use a 360° as a recurring developmental tool typically have 3-5 items per competency. However, in your zeal to measure any and all knowledge/skills/ tasks, it is easy to get carried away - we have seen instruments with upwards of 200 items. Having too long an instrument will often impact response rates. Keep in mind your process and timing. Are all managers going through the 360° at the same time? Then know that some will be completing more than one assessment (assessing themselves, their boss, some of their direct reports, some of their peers) and will rebel against a long survey. In this case, try to keep your 360° to around 70 items or fewer.  

Consider your raters

Always keep in mind who will be rating participants. Peers may not know how well a manager coaches his or her employees. Direct reports can tell you how well decisions are communicated but may be less accurate in assessing how they were reached. This doesn't mean that you cannot include such items on the survey. It simply means that you should offer a "Don't Know/Not Applicable" (DK/NA) option. This is especially important if you do not have different versions of the survey for different groups of raters.

Even if you would naturally expect people to be able to respond to an item, it is normal for some to say they don't know. Look at your data after the fact to see if certain items received a preponderance of DK/NA responses. In that case, you may need to revise or eliminate the item.

Follow solid item-writing principles

Make sure items are behaviorally worded. Many organizations are fond of items that begin with "Understands" as in Understands the business model or Understands customer needs. But "understanding" is not an observable behavior - we can only infer that someone understands something by watching what they do - they must act in a way that tells us they understand the issue. 

Be wary of vague wording. Put yourself in the shoes of the manager who receives a low rating on an item - would you know what it means? Look at these items: Projects a leadership presence. Makes decisions as if he/she owns the business. Interpretation of the results for items like these can be difficult. If what you are after is Speaks with authority or Balances long-term and short-term priorities when making decisions, then say that.

Avoid double-barreled items. If you ask whether a manager selects and trains employees to meet departmental objectives, you insert confusion into the response. How would an employee respond who feels the right people are hired but not sufficiently trained? Look out for and's, or's, and but's - they often lead to double-barreled items.

Use definitions. To ensure that the rater understands exactly what you mean, it's often helpful to provide a definition. Look at these examples:

- Demonstrates awareness of external factors (e.g., political or economic trends) that could affect the business.

- Clearly defines the end results (in terms of quality, timelines, specific requirements) she or he expects.  

Keep it simple. Look at the following item: Focuses appropriate energy and resources on the most critical issues/areas (doesn't waste resources on things that are not important). While the parenthetical certainly helps explain what the item means, it very easily could have been used as the item itself, negating the need for an explanation.  

Don't overlook the scale 

Part of developing content is deciding on your response scale. There is NO scale that everyone agrees is best for a 360°. Common scales include Effectiveness [Very Effective to Very Ineffective], Agreement [Strongly Agree to Strongly Disagree], Satisfaction [Very Satisfied to Very Dissatisfied], Performance [Greatly Exceeds Requirements to Falls Below Requirements], Comparison [One of the Best to Far Below Average], Qualitative [Very Good to Very Poor], Frequency [Almost Always to Hardly Ever], and Extent [Very Great Extent to Very Little Extent]. Some people alter existing scales to meet their needs.

We confess a preference for the effectiveness scale; we've found that it works well with most content. A frequency scale may be less meaningful for some items, e.g., how frequently someone does something is not the same as whether they do it well. Extent scales are often open to confusion, e.g., the extent to which one communicates may imply frequency to some and comprehensiveness to others. However, the important thing is to determine what you want to know and then make sure your scale matches your intent and your item wording.  

Other scale issues to consider include:

Length: Use a 5-point or 7-point scale. Fewer than 5 data points doesn't allow for much variance (which helps tease out differences in perceptions); more than 7 points can become confusing. Also have a mid-point; research has shown that a mid-point is a valid response on a 360°, especially when you have a DK/NA option.

Labels: While it's common to label all data points, it is not absolutely necessary (especially if you have more than 5 points). However, you should definitely label both ends and your mid-point. If you do label all data points, be sure your labels show a logical progression and are not redundant. For example, don't have data point 4 labeled "Excellent" and data point 5 labeled "Outstanding" - those terms are synonymous to many people.

Balance: Balance means having an equal number of positive and negative options. Not all 360°° scales are balanced, e.g.:

» 1=Poor

» 2=Average

» 3=Good

» 4=Very Good

» 5=Outstanding

There are valid arguments on both sides of the balance issue. Because 360° ratings often tend to be inflated (i.e., there is a preponderance of high scores), having more positive options allows for greater differentiation, which is good.

On the other hand, some people use scale labels only to tell them which ends of the scale are positive and negative. They then assume a natural progression (if 1 is bad, then 3 is average and 5 is good). Making #2 "average" can confuse interpretation, since some people may have used an assumed progression rather than the label when they selected their response.

If you have questions about your scale, you can pilot your with a group of potential raters to see whether they interpret it the same way, and whether the scale provides adequate discrimination in the ratings. But don't obsess about this issue. In the final analysis, you will not be using literal interpretations of the labels but looking at results in context - high areas, low areas, trends and themes. 

*******

The gift of feedback is only valuable if it is perceived as meaningful. This means ensuring that the feedback relates to key organizational competencies, that intent is clear, that responses are complete, that the behaviors rated are observable by the raters, and that the data accurately differentiate good and bad performance. All of this depends on your content development.

 PDF version