SSA's Disability Quality Assurance Process
The Lewin report titled “Evaluation of SSA’s Disability Quality Assurance (QA) Processes and Development of QA Options That Will Support the Long-Term Management of the Disability Program” was summarized by the following comments:
“Given the challenges faced by SSA, the design of the Prototype Process, and the current performance of the existing QA system, no amount of retooling, refocusing, redesign, tinkering or the simple addition of resources to the existing QA processes will achieve SSA’s quality improvement goals .”
“The only way that SSA will achieve its quality objectives for the disability programs is to adopt a broad, modern view of quality management that includes efforts outside of the Office of Quality Assurance (OQA) and the current quality assurance process .”
Three specific challenges to performing high quality disability determinations and to quality management of the process were noted:
- Disability determinations are highly complex, and place extraordinary demands on individual adjudicators.
- The subjective nature of many determinations:
- Some decisions, and perhaps many, could legitimately go either way (“close-call” cases).
- Subjectivity contributes to disagreement at the initial and appellate levels and to inconsistency across jurisdictions.
- Subjectivity also makes detection of errors difficult.
3. The Federal/state split of responsibilities for conducting disability determinations adds to the complexity.
- DDSs with high-turnover and noncompetitive compensation.
- Field Office staffing in urban centers that appears out of alignment with work requirements.
- Budget and hiring/allocation restrictions that leave some offices critically short of qualified personnel.
- Insufficient time for special analysis or special quality studies.
- Dated information technology.
- Poor documentation of policy changes.
- For cases that could go either way, there is no wrong decision, per se.
- Even if all initial decisions are of very high quality, there will be some initial denials that are allowed by the Office of Hearings and Appeals (OHA) on the DDS record, because OHA considers each appealed case de novo (a complete reevaluation, including new evidence since the DDS decision) .
- The subjective nature of the process lends itself to substantial, systematic inconsistencies in decisional outcomes across individual adjudicators of the same type, and across groups of adjudicators of different types, or who are working in different organizations. The reason is that individuals, or the groups they belong to, form different habits in their treatment of subjective issues.
- Detecting incorrect decisions in the review process is problematic. Reviewers at all levels assess whether a decision is supported - not simply whether it is correct. When subjective issues are involved, it is essential for reviewers to understand the adjudicator’s rationale. This requires that the adjudicator adequately convey that rationale to the reviewer, which is a time-consuming and challenging task.
- The importance of subjectivity appears to have increased substantially over the last two decades due to policy changes, mostly mandated by legislation or court decisions that have introduced substantially more subjectivity into the process, which might help explain increases in the variability of initial allowance rates across states.
- The programmatic cost of an allowance error is the value of any benefits paid to the ineligible person, and can be very high because once an allowance is effectuated, it cannot be terminated unless SSA can demonstrate medical improvement or the beneficiary leaves the rolls for other reasons.
- The programmatic cost for denial errors is zero.
- The substantial administrative burden of a DDS denial error falls on the Office of Hearings and Appeals (OHA).
- The administrative burden of poor quality intake by a Field Office (FO) is likely to fall on a DDS.
- Erroneous denials or long delays in receiving much needed benefits. According to SSA, a claimant can wait as long as 1,153 days from initial claim through a decision from the Appeals Council.
- Attorney fees up to 25 percent of back benefits.
- Miscellaneous financial, time, and psychological costs associated with completing the process.
- In an adversarial process, either side could appeal an initial decision. In the context of the disability programs, this would mean that SSA could appeal an initial allowance. The appeals process would then be a balanced quality assurance mechanism, protecting against both allowance and denial errors. The pre-effectuation review of initial allowances can be viewed as a substitute for SSA’s ability to appeal an initial allowance.
- In an adversarial process, both sides can agree to a settlement prior to a decision by the adjudicator. In such cases, the two parties agree that the matter has been settled to their satisfaction and there is no further appeal. This is usually considered a high quality outcome.
- In an adversarial process, the adjudicator hears arguments from both sides, which allows the adjudicator to focus on the core of the parties’ differences. The adjudicator’s role is limited to conducting a fair process and judging the relative merits of the arguments.
- In an adversarial process, advocates for the program would learn about which cases are likely to be allowed on appeal and why. Advocates for the program would learn how to discriminate between appeals which should be settled before a hearing, and those which should not. Further, the advocates would become a source of feedback to the program. They would be able to identify systemic problems with the initial determination process that lead to inappropriate denials, or systemic problems with policy.
- Finally, in an adversarial process, either side can appeal the adjudicator’s decision to a higher appellate body. In the context of SSA’s process, this would mean that the program could appeal an allowance to the Appeals Council. In an adversarial process, the appeal board has a very strong interest in the quality of the adjudicator’s work, no matter what the decision is, because the poorer the quality, the more appeals they will receive.
SSA’s definition of quality is too narrow, and a broader definition is needed, encompassing timeliness, customer service, and efficiency, as well as accuracy. True quality is more than just decisional accuracy. At SSA, QA focuses on decision accuracy, or more correctly, whether the adjudicator’s decision is adequately supported.
In the quality literature, QA is the term that is commonly used for end-of-line inspection. Generally speaking, a quality management (QM) approach that relies solely on QA, or end-of-line inspection, is considered dated and relatively ineffective compared to more advanced quality management methods.
Lewin found little evidence that the current quality management system supports a quality-focused culture. Instead of taking responsibility for quality, there is a heavy reliance on the Office of Quality Assurance (OQA) to ensure quality via inspection. The current disability program QA system inspects in order to achieve a level of quality control by correcting error at the end of the process, which is known as a pre-effectuation review (PER) and scores the performance of state Disability Determination Services (DDS) offices without systematically using the system to support process improvement. This type of inspection process represents the most primitive and ineffective of all types of quality management systems.
Lewin encountered significant frustration from those involved in the daily work of the disability process, stemming from resource issues. These included:
Lewin feels that the biggest challenge to producing quality decisions is the complexity of the eligibility criteria. The amount of material that the disability examiner (DE) is required to master is truly impressive. Any DE no matter how well trained, is likely to make errors, some of which will result in incorrect decisions. While the DE might consult with others, and is even required to consult in some cases, ultimately it is the examiner’s responsibility to make the decision.
Individual determinations require the exercise of judgment on the following subjective factors: (1) restrictions of daily activities; (2) difficulties maintaining social functioning; (3) deficiencies in concentration, persistence, or pace that result in failure to complete tasks in a timely manner; (4) episodes of deterioration in work settings that cause the individual to have exacerbated signs and symptoms; (5) functional deterioration, especially as related to relevant abnormal signs and laboratory findings; and (6) pain.
Subjectivity enters into the decision process when the DE assesses the credibility of evidence. This includes considering the potentially conflicting medical evidence from the claimant’s provider(s), the consultative examiner(s), and the medical consultant(s).
Relating to the subjective nature of many cases, and the possibility that a substantial share of cases could go either way, Lewin outlined the following significant implications for quality assurance:
Not only do adjudicators have to deal with the subjectivity of interpretation of evidence, but the nature of medical evidence itself increases the challenge. While medical evidence is often described as “objective,” it is well known that for any given condition there may be substantial variation in how physicians collect, interpret, and report it. For example, one physician may interpret an MRI of the lumbar spine as showing a herniated disc, while another might interpret the imaging study as showing a mild disc bulge. Often interpretations of diagnostic testing are clearly reader dependent, meaning that the final impression is dependent on how the viewer sees it.
Currently, the DDSs have wide latitude with respect to the management of their processes. SSA monitors quality and productivity at the end of the line and intervenes only when a problem is identified. Federal efforts to improve quality assurance processes within the DDSs are constrained by SSA’s ability to influence DDS management via regulation or funding incentives.
The management of each DDS is subject to state civil service rules and other state requirements that can sometimes impede quality improvement efforts. The most obvious example is when DDSs are unable to provide sufficient compensation to retain qualified employees because of state civil service rules. In a January 2004 General Accounting Office report, more than three-quarters of DDS directors reported recruiting and hiring difficulties due to state imposed personnel restrictions including restrictive job classification systems for state employees and state limits on DE salaries and hiring.
DDSs are not required to have internal QA departments or formal QA processes in place. All DDSs, however, have some manner of internal QA procedures. They are generally characterized by an end-of-line review of cases conducted similarly to those conducted by the regional Disability Quality Branch (DQB). Samples sizes tend to be large, and error rates are, in some instances, used as a measure of examiner performance. More informal QA practices might also be undertaken by DDS unit supervisors. This might include a random review of cases in the process of being developed and a review of all decisions of new DEs prior to internal QA sampling. In addition to assessments of decision accuracy, DDSs also use other information to assess the quality and performance of DEs, including statistics on: processing time, workloads pending, age of the workloads, use of consultative examinations, and medical evidence of record (MER) requests. These statistics are shared with DDS supervisors and DEs to manage the claims process and address performance issues.
DDS QA activities often change in response to specific quality or workload issues. When work loads increase, QA staff are sometimes reassigned to processing claims and QA reviews are cut back. Similarly, when concerns about the DDS’s accuracy rate arise, QA staff may increase or target their review activity.
Lewin describes three types of costs to poor quality in disability determinations:
1. Programmatic costs:
2. Administrative costs:
3. Claimant costs:
Lewin noted that while SSA administrators are genuinely concerned about costs to claimants, these costs are not measured in any systematic way. As long as these costs stay below a level that draws the attention of the media or elected officials, then SSA administrators have little incentive to reduce them.
The asymmetry of programmatic costs for SSA for allowance and denial errors is the rationale behind pre-effectuation reviews (PER) of allowances. Pre-effectuation review of DDS Title II decisions is mandated by law: 50 percent of all Title II initial allowances must be reviewed prior to effectuation of payment. SSA uses a profiling system to select error-prone cases. The reviews are similar to QA reviews, except that only PER cases judged to have an error by the DQB examiner are required to have medical consultant review. Errors are returned to the DDS for corrective action. As a result, about 50 percent of all initial allowances are reviewed by SSA’s Regional Disability Quality Branches (DQBs), compared to just a small fraction of denials.
In addition, for Quality Assurance Review (QAR), the regional DQB draws a random sample of DDS decisions designed to capture 70 initial allowances and 70 initial denials per quarter for each DDS (reconsideration and CDR decisions are also sampled separately). Cases selected are reviewed before they are effectuated. The DQB review of the sampled decisions utilizes examiners, medical consultants, and, less frequently, vocational consultants. The DQB examiner conducts the review with the goal of identifying factors that have the potential to affect the decision, as well as the correctness of the decision. In almost all cases, a medical consultant also reviews the case. Errors found are classified into three groups: the decision is either wrong or not sufficiently supported (Group I); the period of disability is incorrect (Group II); or there are technical deficiencies which are unlikely to affect the decision or period of disability (Group III). Only Group I deficiencies are used in the calculation of performance accuracy. If a DDS has an accuracy rate of less than 90.6 percent on initial decisions for two consecutive quarters, SSA must provide management and performance support to the DDS.
OQA’s central office (CO) samples from, and re-reviews cases that have been reviewed by the ten regional DQBs, after effectuation. The (CO) reviewer reviews the case and assesses the review of the DQB reviewer, rather than conducting an independent review of the case.
Hearing Offices have no formal internal quality assurance processes in place for administrative law judge (ALJ) decisions. Regional Chief ALJs cannot change decisions, but they can educate ALJs for the purpose of future cases.
A sample of ALJ allowances with error-prone profiles is selected for screening and review before effectuation. This is known as the ALJ Pre-Effectuation Review. About 7,000 cases are reviewed each year. The OQA conducts the initial review, and the Appeals Council reviews cases where OQA disagrees with the ALJ decision, and returns those it agrees with to the ALJ. It is then up to the ALJ to take any further action. According to a July 2004 GAO report, from fiscal years 1998 through 2002, the OQA reviewed 27,148 ALJ allowances and of these, the OQA found fault with about 35 percent and referred them to the Appeals Council.
In addition, a random sample of ALJ decisions is selected for review at all levels of the adjudicative process, post effectuation. The review is conducted by an examiner and medical consultants, as well as ALJs (ALJ Peer Review). This is also known as the Disability Hearings Quality Review Process (DHQRP). The purpose of the review is to provide information to address broad program issues and to provide ALJs with feedback on their decisions.
A July 2004 GAO report noted that since 1993, the agency has conducted a biennial case review as part of its DHQRP. The reviews indicated that medical consultants and disability examiners have found that supportable ALJ allowances increased from 36 percent in fiscal year 1993-94 to 57 percent in fiscal year 1999-2000. OQA officials told the GAO that this increase suggests an improvement in consistency between adjudication levels; however, SSA’s assessment provides only a partial picture because it does not reflect trend information on the extent to which ALJs have found DDS decisions to be supportable, to ensure that both levels are making more consistent decisions.
The OHA has a much larger incentive to avoid denial errors than allowance errors, because of the high cost to OHA of an appeal to the court system. Lewin believes that the differences in incentives between DDSs and OHA help explain why so many DDS denials are appealed and reversed, although the evidence is not entirely definitive. Another issue to consider is that most ALJs don’t like their decisions to be remanded by the Appeals Council or overturned by a federal magistrate.
While both the initial determination and appellate processes have the objective of adjudicating claims, from an administrative perspective their objectives differ, and this difference has consequences for the conduct of determinations, quality, and quality assurance at the two levels.
A key implication of this difference is that the initial process must be staffed by large numbers of adjudicators (DEs) who are not required to obtain very expensive, advanced degrees, but who can be trained to follow a detailed set of policy and procedural guidelines (POMS), while the appeals process must be staffed by adjudicators (ALJs) who have advanced training and experience in ensuring due process and rely on statutes, regulations (HALLEX), and rulings for guidance. SSA should expect these two types of workers to disagree and to process cases in a different manner. This is a challenge to quality management because it is difficult to assess the extent to which differences in outcomes at the two levels are due to actual problems with one or both of the processes, or are due to the normal result of inherent differences in the process.
Another key implication of this difference is the importance of adjudicative independence for the appeals process: judges cannot protect the rights of individual claimants if the Agency can influence decisions through administrative oversight. Any quality assurance system must be consistent with the Administrative Procedures Act (APA), which exempts ALJs from certain management controls to help ensure that their judgments are independent. The line between management of the appeals process and infringement on judicial independence is very difficult to define, and this has been a source of friction between ALJs and SSA management.
A fundamental problem is that the ALJs must wear both the hat of an objective adjudicator, and the hat of the program. Judicial independence has to be preserved for the first hat, but management oversight is critical for the second hat. This appears to Lewin to be the root cause of difficulties that SSA has in managing the appeals process.
The appeals process is not adversarial in that the claimant is usually represented by a lawyer, but SSA is only represented by the ALJ, who is supposed to be an impartial adjudicator. This type of non-adversarial appeals processes is unusual.
The following features, absent from SSA’s appeals process, but found in adversarial appeals processes, help ensure quality:
In the context of maintaining an open record in the appeals process, one challenge to quality is that new information can be added to a claimant’s record at any point during the initial or appellate process. In some cases, this is new information about the claimant’s condition at the time the claim is filed, and in other cases it describes changes in the claimant’s impairment. Clearly, a situation where the known facts of a case can change at any time create a challenge to those charged with making quality decisions. It is a challenge in quality assessment because the right decision at a point in time must depend on information available at that point in time. For example, many ALJ allowances are based on information that was not available to the DDS and as such, might or might not be indicative of a poor initial decision.
Another reason that the open record is a challenge to quality is that the claimant’s attorney may be slow in providing evidence. Under a closed record system, the claimant and attorney would have a strong incentive to aid in the collection of evidence and ensure that all information is provided by a deadline. That incentive is not present in an open record system. As a result, a greater onus is put on SSA and the DDSs to collect the evidence. Further, Lewin reported hearing allegations from judges and others that attorneys withhold evidence early in the process to ensure a basis for appeal. Some further allege that attorneys have a less than ethical reason for such activity - delaying the decision up to a point increases the attorney’s fee, which is usually 25 percent of past due benefits up to a limit. Either way, the effect is a reduction in the availability of evidence early in the process.
Variation in process outcomes (e.g., allowance rates) will always exist across geographic regions due to variation in local factors that affect outcomes (applicant demographics, and the local economic, political, and cultural environment). Hence, such outcome measures alone are inadequate as indicators of uniformity.
SSA’s operational definition of quality is narrow and reflects an approach to quality management that is not considered best practice. There is a widespread belief in a trade-off between accuracy and productivity, and that SSA’s emphasis on improving productivity has resulted in reduced accuracy. While customer service and timeliness are stated objectives, in practice they appear to be a distant third in a trilogy of operational priorities.
Finally, Lewin encountered a high level of personal commitment to job performance at all levels of SSA, and within the DDSs. Staff believe that disability determinations are important, and need to be done well. But Lewin found little evidence that the current quality management system supports employees and management in their efforts to form a quality-focused culture.
If you are interested in reading the entire Lewin report, click on the link below to download it.