Vocational Rehabilitation Program (Combined or General)

IDVR is able to provide expected levels of performance for all primary performance indicators under this Combined State Plan.  

IDVR currently has strong relative performance falling in the first or second quartiles across all 116 performance accountability measures.  Idaho is experiencing record growth and one of the lowest unemployment rates in the nation heading into 2022.  The Division is engaging with partners in a number of ways to improve state performance:

The Department of Labor is a recipient of a Workforce Data Quality Initiative (WDQI) grant and continues working with IDVR and other core programs to develop policy and methods to automate data sharing to help all programs for performance reporting purposes. Additionally, the Division is participating in the DataLabs technical assistance project with the Idaho WDC and representation from WIOA programs.  This project is exploring methods to improve appropriate referral and indicate meaningful co-enrollment opportunities between programs.

IDVR has an agreement to access Idaho’s Statewide Longitudinal Data System (SLDS) in order to pull education related data elements. The Division is now able to impute limited SLDS data, however receiving data in time to meet reporting requirements continues to prohibit greater utilization of the SLDS system.

The Division has used the RSA Data Dashboards, RehabData Work Group data and data published by RSA as tools to better understand and improve performance under Section 116. 

Idaho Title-IV Expected Levels of Performance PY 2022 and PY 2023

RSA-TAC-20-02 notes: there are many ways to define continuous improvement as related to state or national program circumstances. For example, continuous improvement may reflect:

• an increase from the levels of performance previously attained;
• increases in percentile rankings of levels of performance either nationally or among similar states;
• a change in service strategy and delivery, including more progressive or innovative approaches;
• a change in the intensity or comprehensiveness with which customers are served; or
• a maintenance of previous performance for the top performing states.

Idaho Title IV programs are exploring various data-driven processes to utilize for establishing expected levels of performance prior to negotiation with RSA.

Two years ago, the Idaho Title-IV programs employed a data-driven comparative benchmarking exercise utilizing current data published by the RehabData Workgroup to establish expected levels for the last state plan update.  As new RehabData products have not yet been issued to replicate the benchmarking exercise, Title IV programs are exploring other data-driven approaches to target setting. The CDC’s National Center for Health Statistics has “developed statistical methods and analytical tools to assist in setting transparent, data-driven targets” (NCHS, 2020, p. 1).

Target-Setting Methods in Healthy People 2030 outlines the following approaches in setting data-driven targets:

• Percent improvement
• Percentage point improvement
• Projection
• Minimal statistical significance
• Maintain consistency with national programs, regulations, policies, or laws
• Maintain the baseline

While developed to promote stronger target setting for the Healthy People project, this systematic approach to target setting provides a framework for Idaho to set data-driven targets for all primary performance indicators.  Target-Setting Methods in Healthy People 2030 document presents the decision tree for target setting from this project.  

Additionally, some recent comparative information on states is available via the November 2021 CSAVR presentation by RSA.  Idaho Title-IV programs include this comparative data to characterize relative performance when establishing expected levels of performance to meet continuous improvement expectations defined by RSA. 

Projections are based on the performance data for PY 2020 Idaho Title IV programs from the ETA-9169.

Rationale for Expected Targets for Idaho Title-IV Programs:

Employment Rate-2nd Quarter after Exit

• TAC-20-02 notes a maintenance of previous performance for the top performing states is an acceptable target.
  • Idaho ranks 4^th Nationally/1^st Quartile (RSA, 2021)
• HP2030 notes that the target-setting method of maintaining the baseline is used when an objective is already at a desired level nationally or because the objective is moving or expected to move in an undesirable direction.
  • Idaho unemployment is at a historic low

Idaho proposes a baseline performance extension for employment rate 2^nd quarter after exit.  PY 2020 data from RSA’s CSAVR presentation show Idaho is 4^th in the nation in both 2^nd quarter after exit (58.9 percent).  December 2021 preliminary unemployment figures show Idaho with the 4^th lowest unemployment rate at 2.4 percent (BLS.gov, 2022).  This is a series low unemployment rate for Idaho and the series began in 1976.  Because of these conditions, Idaho believes maintaining performance across 2^nd and 4^th quarters is a reasonable expectation.

Employment Rate-4th Quarter after Exit

• TAC-20-02 notes a maintenance of previous performance for the top performing states is an acceptable target.
  • Idaho ranks 4^th Nationally/1^st Quartile (RSA, 2021)
• HP2030 notes that the target-setting method of maintaining the baseline is used when an objective is already at a desired level nationally or because the objective is moving or expected to move in an undesirable direction.
  • Idaho unemployment is at a historic low

Idaho proposes a baseline performance extension for employment rate 4^th quarter after exit.  PY 2020 data from RSA’s CSAVR presentation show Idaho is 4^th in the nation in 4^th quarter after exit (57.3 percent).  December 2021 preliminary unemployment figures show Idaho with the 4^th lowest unemployment rate at 2.4 percent (BLS.gov, 2022).  This is a series low unemployment rate for Idaho and the series began in 1976.  Because of these conditions, Idaho believes maintaining performance across 2^nd and 4^th quarters is a reasonable expectation.

Median Earnings-2nd Quarter after Exit

Median earnings have been benefiting from low unemployment in Idaho and the state can expect to see this trend continue in the near term as employers compete for talent.  Idaho has seen some vacillation in median wage values across quarters now available for analysis.  Data available is not yet suitable for projections however with confidence intervals ranging between ±11.1-13.2%.  Idaho comparative performance in Median Earnings suggests an outperformance at $4,259, placing the state in the second quartile (RSA, 2021). Idaho is traditionally a 4^th quartile median earnings state, although continues moving in the right direction. Regardless of this outperformance, Idaho Title IV programs are proposing expected earnings increase of 2.5 percent a year following the TSM percent improvement method.

Credential Attainment

Idaho performance was in the first quartile and 7^th overall for PY 2020 (RSA, 2021). Regardless of this performance, the Idaho Title IV programs are committing to significant improvement targets within this two-year update.

Idaho has recently corrected reporting issues with accurately reporting Credential Attainment and consequently has realized a substantial gain as reporting has come online.  Since accurate trend data for this measure is unavailable, Idaho is utilizing the TSM percentage point improvement method.  This method articulates 10-year goal setting for both target effect sizes of 0.1 and 0.2 to correspond with 10 percent and 20 percent improvement, respectively.  The benefit of using this approach is that it is sensitive to projections closer to the top or bottom of the spectrum.

The NCHS Statistical Note on Target-Setting Methods states:     

Targets based on effect size of 0.1 For percentages, when the target is set based on a directional effect size of h = 0.1, the result is a 1–5 percentage point improvement from baseline, depending on the value at baseline:

• For baseline values in the range 25%–75%, the targeted change is 4–5 percentage points. Note that for baseline values around 50%, a targeted change of 5 percentage points corresponds to a 10% change from baseline (over a 10-year period).

Targets based on effect size of 0.2 When the target is set based on a directional effect size of h = 0.2, the result is a 1–10 percentage point improvement from baseline, depending on the value at baseline: 

• For baseline values in the range 25%–75%, the targeted change is 8–10 percentage points. Note that for baseline values around 50%, a targeted change of 10 percentage points corresponds to a 20% change from baseline (over a 10-year period).

The equivalent annual performance gain using the percent point improvement guidelines provided would indicate that a two-percent annual percentage point gain is suitable for utilization in credential improvement.  The Idaho Title IV programs believe they can achieve even better performance and will strive for 2.5 percent percentage point improvement each year of this plan. Note: This corresponds to the higher h=0.2 effect size approach listed above.

Measurable Skill Gains

National data suggest Idaho Title IV programs are performing at the top of the second quartile in MSG rate. Regardless of this strong relative performance, Idaho Title IV programs are committing to significant improvement targets within this two-year update.

*See NCHS Statistical Note on Target-Setting Methods in the Credential Attainment section above for rationale on percentage used for MSG expectations.    

While Idaho Title-IV programs have historical data on MSG now available, the target setting approach for MSG mirrors the rationale contained above in Credential Attainment.  With a two-percent annual percentage point gain indicated by the TSM approach, Idaho Title-IV programs expect to outpace that performance by a full percent across the next two Program Years.  Note: This corresponds to the higher h=0.2 effect size approach listed above.