WIOA State Plan Common Elements

Assessment of Title I-B Adult, Dislocated Worker and Youth Program Services

Maine Bureau of Employment Services has established a schedule of formal monitoring, which includes a program and fiscal review of each of the Title I-B programs. Maine’s Monitoring Policy and Guidance Manual covers oversight and monitoring of WIOA Title I-B and Title III programs and spells out the roles and responsibilities of local area monitoring and identifies criteria to be reviewed annually and on an ongoing basis.

Financial Risk Assessment

Fiscal reviews ensure that expenditures meet the programmatic, performance, cost category, and compliance requirements of WIOA. Fiscal reviews assess the quality of financial administration by reviewing and assuring that service providers have adequate internal controls and fiscal policies and procedures in place. Financial reviews are conducted using tools provided by the USDOL Employment and Training Administration. In addition to on-site fiscal reviews, the Department of Administrative and Financial Services (DAFS) also reviews the audit reports completed on each local area and their subrecipients. Bureau staff work very closely with DAFs staff to ensure that costs for which funds are being requested are allowable and documentation of such are clear and allocated appropriately.

Fiscal monitoring results in an informal exit interview at which identified findings and concerns are discussed and best practices lauded. It is followed up with a formal report that articulates specific findings, cites the regulations and/or policies pertaining to the findings, outlines required action steps that must be taken to resolve the findings and the timelines in which the action steps must be completed.

Procurement practices and policies are also reviewed to ensure that the local area has a clearly identified process for competitive procurement and appropriate procurement practices are in place for pass-through awards. Contract documents are reviewed to ensure they contain the required citations and protocols and procurement activities are also reviewed to ensure contract award decision makers have followed controls pertaining to conflict of interest.

Finally, the fiscal monitoring that the local areas conduct on their subrecipients is also reviewed for the same purposes listed above.

Local Board and Program Assessment

The Bureau of Employment Services’ monitoring tool is fashioned after the “Core Monitoring Tool” published by the Employment and Training Administration. Each local area and its Title I-B sub-recipients are monitored annually. Monitoring is conducted to ensure the one-stop system is in compliance with the intent and substance of the rules governing funding streams and to identify whether the systems are operating to achieve state and local strategic workforce system goals. The monitoring tool is also designed to explore the working relationships between required workforce system partners. Monitoring provides an opportunity to identify best practices that can be replicated and to identify performance and compliance issues that need addressing.

Each year, the Bureau of Employment Services conducts on-site monitoring of the local board during which local board governance is evaluated and local area subrecipient monitoring activity is assessed. The local area governance review includes examination of board policies, board membership, and formal “Memoranda of Understanding” with required partners. It also includes a review of board minutes, requests for proposal, service and subrecipient contracts, quarterly reports and most recent progress in achieving planned service levels and performance goals. Inadequate policies and/or lack of required board membership are identified as findings and technical assistance is provided upon request.

The local area plan is reviewed against plan guidelines issued by the USDOL and Maine. Plans that do not contain all the required elements are not approved until they comply with all the requirements. Local areas are evaluated on the process used to develop and create the local plan, to ensure adequate involvement of system partners in identifying and implementing strategies outlined in the plan and that the plan contains steps for implementation of identified strategies. Local board certification is approved only if the local board has an approved plan in place, meets negotiated performance measures, maintains required local board membership, addresses any outstanding findings or policy requirements, and sustains fiscal integrity.

Methods the local board uses to communicate, educate and inform sub-recipients and system partners is also evaluated to understand local area effectiveness in meeting plan goals and promoting continuous improvements.

Local area subrecipients are also monitored annually to ensure that the local board oversight is adequate and results in subrecipient compliance. Subrecipient monitoring includes intensive file reviews to ensure that adequate documentation of eligibility, participant services, and performance outcomes are in place. Program staff interviews are utilized to gauge whether program design and delivery is being conducted according to requirements and local area plans and participant interviews are conducted to gain insight on the participant’s perspective and satisfaction with the service being provided. Service providers are required to upload all validation and eligibility documentation eliminating the need for paper files. This allows  staff to conduct file reviews through the central office on an ongoing basis.

Sub-recipient interviews allow the monitors to assess whether the service providers understand and are implementing service delivery according to the local area plan and whether they are familiar with and appropriately applying federal, state and local board policies and meeting the requirements of specific programs. Tools that service providers use to conduct initial, comprehensive, and academic assessments are also appraised, as are the methods for providing and documenting provision of required services, including information about nondiscrimination and customer complaint process, labor market and career information, and job search assistance processes.

Program Youth services staff are asked to explain how they assess and document each youth participant’s need for individualized or training services or need for any of the required youth service elements. File reviews and the number of enrolled, look to identify whether youth have access to each of the elements. Youth program case managers are interviewed to showcase service delivery methods. Staff interviews help reviewers understand how they develop employment and training plans or individual service strategies for youth and whether they partner with outside entities to provide any of the fourteen service elements, such as adult mentoring. Work experience files and agreements are reviewed, and staff members are asked to explain how work experience sites are monitored to determine that they are safe and are asked to describe how any issues identified at such sites are addressed. Service providers must also provide an overview of and discuss how work experience employers are educated about child labor law and safety requirements. Youth service policies and protocols are discussed to identify whether and how they differ from adult program policies and protocols.

Assessment of Employer Services

One of the fundamental methods for assessing quality of employer services is to identify if the information and services being provided are being coordinated with other one-stop system partners. Providers are asked to explain how they promote the services of the Maine JobLink and how they collaborate with other Maine business assistance entities, such as the Department of Economic and Community Development or the Small Business Development Centers, the business and industry team members of Maine Community Colleges and other local and regional education and workforce partners. Core partners are required to document services to employers in the Maine JobLink employer account system. This system is used to create the Effectiveness Serving Employers annual report, documentation requires that the employer have an account in the Maine JobLink and staff are educated to explain the many benefits to an employer of having a Maine JobLink account.

Quarterly Performance Assessment

Monitoring is an ongoing activity that requires local areas to submit quarterly reports that identify actual to planned service levels and levels of attainment of negotiated performance goals. Local areas submit both quantitative and qualitative reports which must include at least one service success story from each program each quarter. Planned spending is also reviewed against actual using quarterly fiscal reports to ensure spending thresholds are met and spending caps not exceeded.

Quarterly reporting allows us to address and understand issues regarding service or spending levels that are below or above planned levels. Bureau of Employment Services’ responses to quarterly reports may be in written or oral form; whenever there are common issues across multiple local areas at once, a group conference call is initiated to share data and brainstorm action steps to address the issues directly with local board staff. The Bureau of Employment Services has recently added another component to monitoring, which is to conduct random checks of backup documentation for weekly drawdowns; this practice allows a just-in-time review of fiscal practices and internal controls.

In addition to monitoring of one-stop partner programs conducted by their own administrative entity, the level and ability of the partner programs to adhere to the activities identified in the local area Memorandum of Understanding, that pertain to provision of and access to services and cross-agency referrals, will be reviewed as part of the review process identified by each local board and as part of the annual monitoring process conducted by the MDOL.

Local areas are required to provide an annual report on the progress of their local/regional strategic plan activities, including partner-related activities identified in in the plan. These reports will also be used to evaluate and assess the efficacy of one-stop partner roles in the local one-stop systems.

Data Validation

The BES has developed a Data Element Validation Policy Manual that explains how data will be validated for Title I-B and Title III programs. The policy requires that all validating documents be uploaded into Maine JobLink at specific times and under specific document upload headings. It requires local areas to conduct a sample file validation exercise once quarterly to ensure staff are keyed in on what is required for data validation and includes both a quarterly and annual data element validation file review to be conducted by bureau staff as well. The manual identifies acceptable data element validation documentation and tools and report forms to be used by service provider staff. The policy includes a requirement for annual staff training on data validation that will explain the purpose of data element validation and that will address any areas in which validation checks have failed throughout the year. Files selected for validation each quarter are those of participants who have exited from the adult, dislocated worker, in-school and out-of-school youth programs.

The State Board has a subcommittee known as the program partner committee made up of core, required, and additional one-stop partners. The committee meets to discuss implementation of the local area Memorandum of Understanding and how to integrate services and implement cost sharing agreements. The mission for this subcommittee is to provide leadership to their service providers in the local areas on how to implement and improve a customer-centric, seamless, integrated workforce system that meets the needs of workers, including those with barriers to employment and employers.

One-Stop Certification

A key tool for assessing how well the system functions is the one-stop certification process. The SWB reviewed the criteria that must be considered for one-stop certification and identified two additional items to be included. The first additional component they identified was outreach; specifically, the level, methods, and outcomes of outreach efforts to both employers and target populations, but particularly to employers offering high-demand, high-wage job openings to targeted populations that cannot readily access one-stop services because they live-in isolated areas or are reentry civilian life after incarceration.

In addition to outreach, the SWB identified the capacity to meet or exceed negotiated performance measures by one-stop, so one-stop center performance can be compared, best practices identified and replicated, corrective action steps be implemented, and additional resources and technical assistance can be applied to one-stops that are having difficulty meeting or exceeding performance measures. State policy PY16-04 One Stop Certification outlines the requirements for implementing a methodology and certifying one-stops.

The certification process is important to setting a minimum level of quality and consistency of services in one-stop centers across Maine. The certification criteria allow the state to set standard expectations for customer-focused, seamless services from a network of partners that will help individuals overcome barriers to becoming and staying employed. In order to be eligible to receive infrastructure funding, one-stop centers must be assessed and certified by the local board at least once every three years using criteria established under WIOA Section 121(g) and identified by the SWB in consultation with the chief elected officials and local boards, as outlined in the policy.

The SWB, in consultation with chief elected officials and local boards, will review the one-stop certification criteria at least once every two years and will update it as necessary to assure continuous improvement of the system. The SWB will ensure that one-stop certification criteria is in alignment with the Unified Plan and that any revisions to the certification criteria will be formalized in a policy issuance and included as a modification to the Unified Plan.

If a local board also acts as a one-stop operator, the SWB will be the entity that evaluates and certifies the one-stops in that local area.

The criteria identified in this guidance will be used to evaluate one-stop centers for effectiveness, customer satisfaction, physical and programmatic accessibility, and alignment and integration of resources for the purpose of continuous improvement.

Evaluation Criteria must include assessment of how well the one-stop center:

• Integrates available services for participants and businesses in a way that is tied to locally negotiated performance goals
• Meets the workforce development needs of participants through provision of services and leverage of resources
• Meets the employment needs of local employers
• Operates in a cost-efficient manner
• Coordinates services among and between one-stop programs in a way that is seamless to the customer and eliminates duplication of services
• Provides access to partner program services to the maximum extent possible, including providing services outside of regular business hours where and when there is a workforce need identified by the local board
• Ensures equal opportunity for all individuals, including individuals with barriers to employment, to participate in or benefit from one-stop center services
• Acts to comply with disability-related regulations implementing WIOA Section 188 set forth in 29 CFR 38
  • Achieves or exceeds state negotiated levels of performance and other performance measures established by the local board for the local area
  • Has a process for identifying and responding to technical assistance needs of staff and partners
  • Has a system of ensuring professional staff have the requisite abilities, knowledge and skills required to administer services, including a system for provision of continuing professional development activities on behalf of professional staff, as necessary
  • Has a system in place to capture and respond to customer feedback and to ensure customer-centric service delivery and customer satisfaction (workers, seekers and employers)
  • Has a system in place to assess itself regarding these requirements and to implement continuous improvements

State Criteria: Per the SWB, local boards must also evaluate one-stop centers on the effectiveness of outreach strategies and efforts, including:

• Outreach to employers to provide information about the types of services, information and sector initiatives offered by and through the system

• Outreach to individuals who cannot easily access the services at the physical one-stop centers, including:
  • Individuals in remote areas
  • Individuals with disabilities
  • Individuals with limited English proficiency or literacy
  • Individuals who are currently incarcerated and preparing for release

Procedures and Methods of Evaluation: Local boards may adopt locally identified methods of assessment which may include:

• Assessment through a recognized certification mechanism, like the Malcolm Baldridge Award, that incorporates the above criteria
• Assessment using a specific evaluation tool designed to review and evaluate the above criteria
• Assessment using a combination of the above or other method as determined by the local board

Local Board Requirements and Deadlines:

• A draft of the local area process and evaluation instruments to be used must be submitted to the Bureau of Employment Services
• Local boards must ensure local one-stop system service providers and partners have been made aware of the process and evaluation instruments that will be used to evaluate and certify one-stops
• All local area one-stops must be evaluated and certified at least once every three years
• Documentation of certification of each one-stop must be provided to the Bureau of Employment Services

Local Area Criteria: Local boards may identify criteria in addition to that identified in state guidance. They must inform local system partners of the additional criteria via dissemination of formal policy or guidance and must include the additional criteria as part of the local/regional plan and must assess based on the criteria as part of the overall one-stop certification process.

Assessment of Title III Wagner-Peyser Programs –Accessibility - EEO Practices

Wagner-Peyser programs are assessed at the same time Title I-B program reviews are being conducted. Staff members are interviewed regarding knowledge and practice of explaining job order procedures and job seeker registration services and are asked to explain the ways in which they provide employer assistance and help in creation and resolution of jobs orders. An assessment of staff knowledge regarding equal opportunity and affirmative action requirements is also conducted along with a review of staff knowledge Wagner-Peyser regulations. Processes to provide initial assessment and appropriate referrals to center customers and front-end procedures are also reviewed. In some instances, participants may also be interviewed either in person or via telephone. Monitors use the checklist provided under Section 188 to conduct a portion of the accessibility review.

Staff review the local board’s annual assessment of accessibility of the centers and conduct a brief accessibility assessment while monitoring specific one-stops. Staff members must explain how customers can access the assistive technology in the centers; all required posted information is examined to ensure it reflects the most up-to-date version of the regulations and sites are checked to determine whether information is provided in Braille and other languages besides printed English.

At least once annually, a separate equal employment opportunity review is conducted by the State EO officer. The EO officer reviews subrecipient compliance with universal access and nondiscrimination regulations through examination of participant applications and enrollments against demographic data. Likewise, participant files are reviewed to ensure that all staff-assisted participants have been provided with the required EO statement and understand their right to file a complaint. Upon completion of the review, providers are supplied with a formal report of review outcomes and a corrective action plan to address findings.

Customer Satisfaction

Job Seeker and employer customer satisfaction surveys are conducted at least once annually in addition to any local area customer satisfaction activities. These are conducted using survey monkey and may include short phone interviews as well. In general, the results provide us with information about the perceived value of required services and whether the services are meeting the needs of the customer. The quality of staff assistance is also ascertained through the questionnaire. The results of the service are explained in the Annual Report.

Walk-in customers are requested to fill out a customer satisfaction survey at each visit, this tool has resulted in reconfiguring info center space, reviewing customer guides to ensure they are easy to understand, and other efforts toward continuous improvements.

Performance Results

Every two years, state performance goals for the Adult, Dislocated Worker, Youth and Wagner- Peyser programs are negotiated with USDOL. In turn, the Maine Bureau of Employment Services negotiates performance goals with each of the local areas. Technical assistance is provided to local area board and service provider staff regarding how performance is negotiated, measured and reported. The ability to achieve proposed performance levels can be affected by numerous factors, including unanticipated mass layoffs, layoffs of low-skill workers in a high-skill job market, and lack of job openings in rural areas. Most recently, low unemployment and an increase in minimum wage each year for the last three years have had an effect on negotiated measures.

Assessment of Core Program Activities Using Annual Performance Accountability Assessment Results.

Now that all four core partners are required to report on the same performance outcomes, the State will be better able to compare outcomes for quality improvement purposes,  based not only on the barriers of the individuals served, but on the type and level of services received through each core program.

Using specific reporting elements, such as co-enrollment, level and type of career service provided, level and type of training service provided (or not), amounts spent on each participant for each service type, participant demographics and local economic factors, the state will be able to develop a quality assessment that more deeply explores all of the data that may affect performance achievement.

The additional data will allow the state to compare the programs, service packages, co-enrollments and specific approaches of local areas (and individual service providers) that meet or exceed planned and negotiated measures with those of local areas that are having difficulty doing so. In doing this, the performance staff will be able to promote and/or require local areas that are struggling to achieve negotiated measures to replicate the practices of local areas that are exceeding goals. Such strategies may include, expanding the number of participants taking part in multiple core programs services simultaneously, such as adult education and Title I-B services. It may also identify that professional staff development pertaining to participant and/or employer outreach needs to be addressed. Finally, it may be that state, local area or service provider policies are restricting service blending possibilities and that by adjusting these the providers will be better able to address participant needs and provide supports for improved employment opportunities and longer-term employment success.

Once the state becomes adept at utilizing the Statistical Adjustment Model it will have the ability to identify factors that are beyond the local area’s control, such as major downsizings, extreme rurality, higher levels of non-English speaking residents etc. and will be able to recommend proactive responses by core and other service partners in those areas that can begin to address those issues. Such as by directing collaborative investments to improved English literacy programs that combine English proficiency and occupational training, or that establish partnerships that link individuals without access to one-stops or affiliate sites (because they reside at extreme distances) through technology or new partnerships with adult education providers or municipal libraries.

Local Areas are conducting research around high-growth career pathways and training staff to utilize pathway models whenever providing labor market guidance to, or developing individual services strategies with, participants. It is likely there is some direct correlation between clearly defined pathways (both at the individual and local area level) and improved employment attainment and retention outcomes.

Maine reviews performance on a quarterly and annual basis using various methods, including program and fiscal report review, annual and ongoing desk top monitoring, and anecdotal information sharing. By using all available assessment data, monitors have been able to identify service models that result in better outcomes for high-risk youth that could be replicated in some areas but not all. If the state achieves at least 90 percent of the negotiated performance rate they are considered to have met the individual measure; however, when reviewing the overall measures for the state, the adjusted levels attained by all four core partners are considered.