The What/When/How/Why of Measuring Neutrals
Balance.
This critical component factors in every phase of dispute resolution. We view it as a critical component for fair, ethical dispute resolution.
The Neutral is the linchpin in dispute resolution. Qualified, capable, professional neutrals provide a valuable service to the parties, and for the dispute resolution process. Credibility depends on strong neutrals, and the parties deserve nothing less.
Balance presents some interesting questions when it’s applied to management tasks:
- How do we define balance in measuring the Neutral’s performance?
- When do we provide feedback?
- What do we use as a measuring device?
- When, and where, should this information be confidential?
- When, and how, should it be shared?
Example:
Company A (A) has an agreement with Green Dispute Resolution Services (Green) to provide arbitration services. Green randomly assigns cases to its Neutrals. After a period of time, A notifies Green that it would like to expand services to include disputes in other subject matters.
Green contacts the Neutrals, advises them of the new opportunity, and asks the Neutrals to reply indicating their interest and documenting the subject area where they have expertise. A high percentage of these experience Neutrals reply and Green finds it has several times more interested Neutrals than the expansion would potentially employ.
Green conducts an internal review of the Neutrals who applied, looking at performance issues such as:
- Decision made on time
- Decision includes an explanation of the reasoning, includes compelling factual evidence presented
- Decision indicates the Neutral conducted a thorough review of the information presented
- Neutral’s performance demonstrates compliance with rules and policies
- Neutral’s decisions, over time, demonstrate balance; reflecting a fluctuation or swing in the decisions. In other words, in some cases the Claimant prevails, in some cases the Respondent prevails.
Performance is evaluated as either “pass” or “fail”. Management selects several candidates from the Neutral pool who received “pass” marks in all the performance categories and assigns them to the expanded program
After the decisions were made, a Neutral (Chris) called to express disappointment at not being selected and asked for a reason. Management decided Chris deserved an explanation. Because Chris’s decisions nearly always favored one party, measurement #5 received a “fail”.
Management regretted not providing this feedback to all its Neutrals.
Initial Forum Question:
- Is it appropriate/ethical for Management to keep records that allow it to measure the Netural’s Performance?
- If no, why not?
- If yes,
- What should be measured?
- What could be measured?
Moderator Bios:
Jo DeMars is President of DeMars & Associates, Ltd., which manages online and face to face dispute resolution programs for companies such as eBay, Professional Warranty Services Corp, Porsche Cars North America, Workhorse Custom Chassis, Winnebago, and Thor Industries.
DeMars is a published author and frequent speaker on topics such as customer retention, ethics and online dispute resolution. In 1995 DeMars was recognized the Wisconsin Entrepreneurial Woman of the Year and in 2005 was recognized a Woman of Distinction by the Waukesha County Foundation. She graduated summa cum laude from the University of Wisconsin-Milwaukee.
Anne Foye
______________________________________________
Return to Cyberweek 2014 Homepage
Replies to This Discussion
-
Permalink Reply by Jo DeMars on
-
Good morning,
Welcome to our discussion about measuring neutrals.
Whether you are a professional mediator or arbitrator, a program administrator, or have an academic interest, we'd like to hear your thoughts and opinions.
We look forward to a lively discussion.
Thanks in advance for your contributions.
Jo
-
-
To provide some context for this discussion, I am linking a short demonstration video of our NetNeutrals Platform.
-
-
I believe tracking and measuring neutrals is absolutely essential in order to ensure that all parties are treated fairly and neutrality is maintained. ODR/ADR programs cannot be effectively managed with measurement policies in place. If data is not gathered, how then can neutrality be clearly judged? Examining long term trends can provide much more information than looking at decisions on a case by case basis. By tracking program Neutrals and trends, outliers can be identified. Administrators must have some tracking mechanisms in place to measure the effectiveness and neutrality of a neutral.
I believe that, on a program level, data should be collected on:
- Decision timeliness
- Customer perception of the Neutral's knowledge and professionalism
- Complaints against the Neutral
- Neutral's understanding and compliance with program rules
- Percentage of cases decided in each parties favor (slight bias toward one side may not be evident in individual cases but will be evident as a long term pattern)
By comparing all Neutral data, it can be easy to identify subtle issues such as slight bias, tendency to bend rules, or consistent late decisions.
I think an interesting follow up question is, once data has been collected - who has access to it? Is it shared with Neutrals? Would that influence their decisions in a way that would have an effect on their neutrality? I contend that it could and therefore such data should be shared only after significant thought.
-
Permalink Reply by Susan Nauss Exon on
-
Anne, you pose interesting information. I want to respond to your comment that "tracking and measuring neutrals is absolutely essential in order to ensure that all parties are treated fairly and neutrality is maintained." If you do not share your data with the neutral in question, how can that neutral treat participants fairly and neutrally? In other words, it seems that it would be essential to share the information with the neutral so that the neutral may conduct his or her own self-assessment regarding possible biases. In this way, the neutral can continually assess his or her ability to maintain impartiality.
Anne Foye said:I believe tracking and measuring neutrals is absolutely essential in order to ensure that all parties are treated fairly and neutrality is maintained. ODR/ADR programs cannot be effectively managed with measurement policies in place. If data is not gathered, how then can neutrality be clearly judged? Examining long term trends can provide much more information than looking at decisions on a case by case basis. By tracking program Neutrals and trends, outliers can be identified. Administrators must have some tracking mechanisms in place to measure the effectiveness and neutrality of a neutral.
I believe that, on a program level, data should be collected on:
- Decision timeliness
- Customer perception of the Neutral's knowledge and professionalism
- Complaints against the Neutral
- Neutral's understanding and compliance with program rules
- Percentage of cases decided in each parties favor (slight bias toward one side may not be evident in individual cases but will be evident as a long term pattern)
By comparing all Neutral data, it can be easy to identify subtle issues such as slight bias, tendency to bend rules, or consistent late decisions.
I think an interesting follow up question is, once data has been collected - who has access to it? Is it shared with Neutrals? Would that influence their decisions in a way that would have an effect on their neutrality? I contend that it could and therefore such data should be shared only after significant thought.
-
-
Thanks for your comments, Anne and Susan.
I agree that collecting data and using it for management purposes to help ensure fairness is an important aspect of Administrative oversight.
Susan's point about sharing the information with the Neutral, so they can self-assess regarding possible bias, points to another valuable use of the data.
I expect this information might provide additional insights if the individual neutral is provided the data on his/her activities and, then for comparison, the data on all activities of all the neutrals as a whole.
-
-
Today I'd like to expand the discussion, and look at how dispute resolution program managers might measure neutrals.
Here is one approach for an arbitration process:
Data is collected on performance issues such as:
- Decision made on time
- Decision includes an explanation of the reasoning, includes compelling factual evidence presented
- Decision indicates the Neutral conducted a thorough review of the information presented
- Neutral’s performance demonstrates compliance with rules and policies
- Neutral’s decisions, over time, demonstrate balance; reflecting a fluctuation or swing in the decisions. In other words, in some cases the Claimant prevails, in some cases the Respondent prevails.
Performance is evaluated as either “pass” or “fail”.
Other programs use a customer survey at the end of the process, in which the parties to the dispute rank (on a scale of 1-5) their opinions on:
-the neutral was prepared for the meeting
-the neutral conducted a professional meeting
-the neutral maintained independence.
I look forward to reading your comments.
-
Permalink Reply by Pablo Cortés on
-
Jo: thanks for sharing these thought-provoking ideas on evaluating neutrals.
I criteria seem very reasonable. Although, I am with Susan in that it would be important to treat neutrals in a fair manner, and so they should be informed about the criteria for renewal, it makes me wonder whether such criteria may affect their impartiality. In Jo’s example, if I were Chris, I’d try to meet the criteria, and that might influence my view, especially for the difficult cases, where I may be inclined to favour one of the parties if that might help me to reach the balanced outcomes required for my renewal. Also, many ethical rules state that the fees of neutrals should not depend on their outcomes –I wonder whether this criterion gets close to breaching such rules.
I very much like the idea of the customer surveys. One would have to be careful in how to gather this data though. It might be also interesting to see whether the adversarial principle was respected (i.e. whether the parties had similar opportunities to present their cases) and how fair they consider the final award to be –naturally, one would expect that the loser will be less happy with the outcome than the winner, but I would be interested to know whether all the arbitrators score the same points.
-
-
I believe that a manager should rely upon objective standards. Trying to create or measure a subjective standard is by definition an impossible task...assuming one wants to get it "right". You can share objective performance with the neutrals exactly because it is objective. ( The first 4 categories). You can have participant in put, which is at least know to come from an advocated source, and also shared with the neutral, with or without identification of the participant.( Do not worry the arbitrator will know who is reporting, for better or worst, since he/she will recall the case, based on the input.) The notion that a good neutral is authenticated by a balance of outcomes is a faulty premise, and highly unlikely when dealing with consumer contracts, that have carefully, and legally, been crafted to greatly advantage the drafter. We could not possible be doing a good job if we were striving to create a " balance" of outcomes. The 5th category cannot be applied to an arbitrator, any more than it could to a judge, who we hardly expect to balance her opinions just to make pretty.
-
-
Jo, thanks for the invite to participate. I do agree that neutrals should be evaluated. If they were informed of the criteria , rules, and standards, in writing, prior to accepting the position. I also think that the 5 performance issues you have laid out would be perfect for evaluation. I do believe that customer input is important, but remember that there will be a bias depending on who “won” the case In some of the examples I can think of, I’m not sure that the “losing” customer would be able to provide a fair analysis of the neutra’ls performance. Many times they are only seeing their side of the issue.
I also think that periodic feedback to neutrals is important for job growth and improvement. Maybe the data that is gathered could be shared anonymously with all neutrals, so that everyone gets to see the “big picture” which could help with impartiality as well as logical reasoning.
I’m not sure whether I agree with judging neutrals based on the percentage of cases decided in each parties favor without knowing the facts of the individual cases, and the statements submitted. Since assignment of cases is arbitrary, it is possible that a neutral could get assigned cases that would, on their face, be favorable to one party or the other. IE: cases where feedback is left 70 days after the purchase, or cases where payment was not completed by the buyer.
-
-
Great comments, everyone. Thanks for participating.
Today I'd like to open the discussion further.
Question: Should the data always be considered confidential?
If not, what information, when should it be available, and to whom?
Parties to the dispute, regulatory agencies, investigative reporters, others?
A. Prior to the decision?
- Individual Neutral performance or aggregate data?
- Other? Please provide examples or suggestion
B. Following the decision?
1. Individual Neutral performance or aggregate data?
2. Other? Please provide examples or suggestions
-
-
I don't believe that any data, individual or aggregate, should be shared prior to the decision. What would the reason be for sharing at that point?
I do believe however, that following the decision, aggregate, anonymous data should be available to parties to the dispute, and neutrals. The information shared should be factual, but very vague as to what cases it applies to.IE: Aggregate data about the time length not considered "submitted in a reasonable time frame". The aggregate data could be complied and released on a monthly basis.
-
-
Thank you, Jo, for inviting me to participate. I'll begin by stating that I agree with many of the sentiments posted above. In particular, I will opine that it is difficult to analyze a Neutral's potential bias based solely on the aggregate percentage of that Neutral's decisions in favor of the claimant or respondent. Not only does this percentage fail to take into account the nature of the fact patterns put before a Neutral, but it also fails to account for how a forum's standards are "slanted". It cannot be assumed that a forum is designed to produce "50/50" results even if that percentage intuitively appeals to our sense of fairness. The criminal justice system is designed to err on the side of "over-exoneration" of defendants because the cost of mistakenly convicting an innocent is thought to outweigh the cost of exonerating a guilty person. The bias is "baked into" the criminal justice system through the requirement that a guilty verdict be unanimous; we should examine the biases that are "baked into" the standards governing all forums before automatically attributing these biases to the Neutrals applying the standards.
All this is to say, I don't think it does much good to reveal aggregate data to parties to the dispute. Parties are uniquely predisposed to spotting bias where none exists, and as I detailed above, I think the appearance of bias in a forum is a nuanced issue that is ripe for misinterpretation. Regulators should certainly have the aggregate data at their disposal, but they should understand the purpose behind the forums that they're regulating. When done correctly, examining the biases that appear in a forum can be a wonderful method to get at the true purpose of the forum - why it exists, whom it protects, etc. However, beginning that examination with the assumption that any biases are those of the Neutral would be a mistake.
About
@ADRHub Tweets
ADRHub is supported and maintained by the Negotiation & Conflict Resolution Program at Creighton University
© 2024 Created by ADRhub.com - Creighton NCR.
Powered by
