What's New

PDE notified ASG today that it has been selected to assist the Department in a number of services related to its upcoming assessment procurement. The company will be working with PDE to perform pre-RFP design and pricing of its new assessment system, draft the scope of work for the RFP for assessment services, develop the RFP evaluation criteria, and evaluate the vendor technical and cost responses. The work will be conducted over the next several months.

ASG is delighted to announce that Wayne Ostler has joined the company as a partner and will head up ASG’s technology consulting area. Wayne’s decades of experience in assessment technology adds much needed expertise to ASG’s consulting services. Wayne had been working as a technology consultant to ASG for the past several years and the two parties decided to make the relationship permanent after seeing the value both bring to the relationship.

Wayne and ASG will continue to work with clients to advise them in all areas of assessment technology including online test delivery systems, test administration systems, and content development and item banking technology. ASG is thrilled to add Wayne to the team.

Wayne’s bio is below.

Wayne Ostler is a seasoned technology specialist known for leading large scale software development projects effectively and efficiently. Wayne is very skilled at analyzing business requirements and transforming them into an effective strategy for technology implementation. Wayne is able to clearly communicate complex technology strategies and solutions to business executives and customers.

Wayne worked at Pearson from 1994-2018 where his most recent position was Vice President, Content Systems and Publishing Services. In this role he headed a global organization of 140 full-time and contract software engineers, web publishers and quality support staff. In this position he was responsible for research and development activities for next generation online digital assessment content management systems and led digital content publishing activities in support of paper and online test delivery systems.

Prior to that, Wayne was Vice President, Product Development where he directed all software product development activities for Pearson’s flagship assessment platforms including Pearson Access and TestNav. He was responsible for all research and development activities, technology innovations and customer deployments associated with these solutions.

Prior to joining Pearson, Wayne worked at Texas Instruments from 1981-1994 where he started as a software engineer in the defense systems group. During his tenure at Texas Instruments, Wayne advanced into several leadership and architecture roles.

Wayne received his Bachelor of Science degree in computer science from Indiana University, Bloomington, Indiana.

ASG completed a review of one of the assessment industry’s leading vendor’s proposal response process today, delivering a detailed summary report of findings and recommendations. The project included a detailed review and analysis of several of the vendor’s proposals using proprietary rubrics ASG has developed over the years in assisting vendors with their proposal responses, as well as the final summary report.

ASG has developed rubrics it uses to evaluate a vendor’s response to each RFP requirement, as well as a section (development, psychometrics, delivery, etc.) summary rubric. Rubrics and responses are assigned point values and a statistical analysis of vendor proposals is completed showing areas of strength and weakness. Recommendations are provided to improve the vendor proposals.

ASG has now performed this work for four industry vendors.

ASG completed its 11th annual state assessment survey today – sending the final database of state responses and summary presentation to the 44 participating states. The purpose of the survey is to enhance communication among the states regarding how each state is responding to current assessment industry trends.

The survey consists of a Survey Monkey form that is completed at the convenience of the state, an Excel file capturing the details of the state’s assessment program, and a short follow-up interview. ASG consolidates all the information in an Access database it distributes to participating states, as well as a detailed summary report. With 11 years of data, ASG has the most complete assessment database in the country and many states have reported to us on their using the survey data to answer questions from state Chiefs, legislatures and the public.

ASG developed the Scope of Work for New Jersey’s Student Learning Assessment – Mathematics/English Language Arts upcoming RFP. During the multi-month process, ASG provided assessment design and consulting services to the New Jersey Department of Education while drafting the scope of work for the RFP. ASG also provided approximate cost estimates for the assessment system based on the RFP requirements. The State’s new assessment system will include tests in grades 3-8 Mathematics, grades 3-9 ELA, Algebra I, Algebra II, Geometry and the New Jersey Graduation Proficiency Assessment.

With the passage of ESSA, states were given approval to use a College Entrance Exam (CEE) in lieu of their statewide assessments provided the CEE aligned with their state standards. Over a dozen states have elected to replace their prior high school accountability assessments with a CEE since that time. ASG surveyed states on their use of CEE’s to determine the factors that went into states making the decision to adopt a CEE, the driving forces behind the decision to switch tests, and the satisfaction with their new assessments. The results of the survey were shared with participating states.

ASG finalized the scope of work for Washington D.C.s upcoming RFP for statewide general assessments in English language arts/literacy (ELA) and mathematics in grades 3-8 and high school, and science in grades 5 and 8 and high school biology. Throughout the assignment, ASG assisted District personnel with assessment design and general consulting services.

ASG will also be providing assistance in analyzing vendor technical proposals and evaluating vendor cost proposals using its Assessment Cost Model. ASG has saved states over $100 million in evaluating assessment cost proposals since 2009.

ASG completed its analysis of the vendor cost proposals for WIDA’s RFP for Printing, Distribution, Administration, Scoring and Reporting of WIDA ACCESS, Screener, and Alternate ACCESS Operational Assessments. ASG used its Assessment Cost Model to evaluate vendor cost proposals. ASG has saved states over $100 million in evaluating assessment cost proposals since 2009.

ASG had previously drafted the RFP Scope of Work and has assisted WIDA with numerous procurements over the past several years.

ASG completed its review of vendor technical proposals in response to WIDA’s RFP for Printing, Distribution, Administration, Scoring and Reporting of WIDA ACCESS, Screener, and Alternate ACCESS Operational Assessments.

ASG created a detailed, custom rubric for the analysis which assigned point values to each functional area and key requirement of the RFP, scored each vendor response per the rubric and derived the technical score for each vendor. Using the ASG developed rubric eliminated much of the subjectivity in the proposal scoring and allowed for a more objective analysis of the vendor proposals.

ASG completed the scope of work for WIDA’s RFP for Printing, Distribution, Administration, Scoring and Reporting of WIDA ACCESS, Screener, and Alternate ACCESS Operational Assessments. ASG delivered the final Scope of Work and Vendor Cost Input Sheets for WIDA’s RFP that will be issued shortly.

ASG drafted the RFP Scope of Work, provided its proprietary Vendor Cost Input Sheets for inclusion in the RFP, developed a custom rubric to analyze vendor technical proposals and will be performing vendor cost analysis services of the various bids using its Assessment Cost Model. ASG provided a similar set of services to WIDA for its previous procurement in 2014.

ASG completed its 10th annual state assessment survey today – sending the final database of state responses and summary presentation to the 44 participating states. The purpose of the survey is to enhance communication among the states regarding how each state is responding to current assessment industry trends.

The survey consists of a Survey Monkey form that is completed at the convenience of the state, an Excel file capturing the details of the state’s assessment program, and a short follow-up interview. ASG consolidates all the information in an Access database it distributes to participating states, as well as a detailed summary report. With 10 years of data, ASG has the most complete assessment database in the country and many states have reported to us on their using the survey data to answer questions from state Chiefs, legislatures and the public.

ASG assisted a major assessment provider in reviewing vendor technical proposals to provide it with an item authoring, item banking and test creation system. ASG created a unique rubric to evaluate the various proposals. The rubric weighted the various sections of the RFP, and individual requirements within the sections on a multi-point scale. This allowed the reviewers to score RFP requirements and generate a total score for each vendor’s proposal quickly and objectively. ASG previously drafted the actual RFP for the assessment provider.

ASG assisted a major assessment provider in reviewing vendor technical proposals to provide it with an online assessment delivery system. ASG created a unique rubric to evaluate the various proposals. The rubric weighted the various sections of the RFP, and individual requirements within the sections on a multi-point scale. This allowed the reviewers to score RFP requirements and generate a total score for each vendor’s proposal quickly and objectively. ASG had previously drafted the actual RFP for the assessment provider.

The Smarter-Balanced Assessment Consortium contracted with ASG to provide it with an estimate of the costs of the University of California creating an organization and assessment based on the SBAC assessment system to use as a college entrance exam. Earlier this year, the University of California voted to no longer require the use of the SAT or ACT for admission to the UC system.

In order to provide the UC with an approximation of the costs of creating its own college entrance assessment, SBAC asked ASG to update the calculations it made for the consortium when it estimated the costs of the SBAC assessment system 6 years ago. ASG used the great majority of assumptions behind the original development of the SBAC assessment system and organization in providing the cost estimate.

ASG completed the scope of work for 3rd party psychometric services for a major assessment vendor today. The vendor provides a complex, high quality assessment and desires an outside 3rd party to assist with various psychometric services. ASG worked with the vendor to define the scope of services required, vendor experience requirements and corporate qualifications to perform the services in the SOW.

ASG completed the scope of work for an item authoring and test development RFP for a major assessment provider today. It is critical that test developers have a current, fully functional system to develop and store content and create actual assessment instruments if they are to create quality assessments at a reasonable costs.

Item banking and content creation systems have come a long way over the past several years. ASG’s technical and development consultants are current with the technology and functionality of the best systems, as well as test developer requirements to create quality assessments. ASG will also develop the scoring rubric and assist in the evaluation of the vendor technical proposals.

ASG submitted a report today to a major assessment provider on the complexity of the technology required to deliver the provider’s assessment, as well as an evaluation of the current platform used to deliver the assessment.

ASG worked with the assessment provider to gather and review technical information on the assessment, technical information on the technology system and submitted a series of questions to the test delivery vendor regarding the technologies it uses to develop, maintain, and host the delivery platform.

The final report outlined ASG’s findings and provided recommendations to improve the various technology and assessment systems to make them more efficient.

ASG completed a Request for Information for a test delivery system for a major assessment vendor today. The assessment vendor wants ASG to take the findings gained from this RFI and include the relevant requirements in the RFP for assessment services it will write for the vendor later this year. The RFI was sent to all major online assessment delivery providers.

ASG’s technology consultants are well versed in assessment delivery systems (having been responsible for the development of leading industry systems) and worked with the vendor to develop the appropriate RFI for the customer’s needs.

ASG completed its 9th annual state assessment survey today – sending the final database of state responses and summary presentation to the 44 participating states. The purpose of the survey is to enhance communication among the states regarding how each state is responding to current assessment industry trends.

The survey consists of a Survey Monkey form that is completed at the convenience of the state, an Excel file capturing the details of the state’s assessment program, and a short follow-up interview. ASG consolidates all the information in an Access database it distributes to participating states, as well as a detailed summary report. With 9 years of data, ASG has the most complete assessment database in the country and many states have reporting using the survey data to answer questions from state Chiefs, legislatures and the public.

ASG completed a multi-month process to assist an assessment vendor develop a labor tracking system for its employees. The assessment vendor is entering into new lines of business and called on ASG’s expertise in assessment financial systems and cost tracking to help it develop requirements for a new time tracking system. ASG developed a time coding framework and time keeping systems requirements document for the customer to use in drafting the RFP for vendor services.

The process ASG used develop the framework consisted of interviewing department leaders to gain an understanding of departmental processes and workflows, as well as workflow mapping. ASG then used its knowledge of assessment costs to develop a timekeeping system that records key work activities without creating an undue burden on employee time tracking requirements.

A major assessment vendor selected ASG to help it update its financial systems and reports as it implements new company strategies. ASG will perform several important services for the company including analyzing its bidding process, implementing systems and procedures to track labor costs, defining requirements for a new time-tracking system, and developing financial reporting systems and capabilities. ASG has helped several assessment vendors better understand their costs, establish metrics, and implement financial reporting systems to better manage their businesses.

The process for conducting ASG’s 9th annual state assessment survey officially started today with ASG sending the participation letter to 50 state departments of education and the District of Columbia. Last year, 46 states and DC participated in the assessment survey which is designed to increase communication regarding issues of interest among the states. Participating states receive a database with all of the survey responses and can contact their peers in other states for further discussions. The complete database and a summary report will be issued by ASG in early 2020.

ASG completed its review of a major assessment vendor’s proposal writing process and delivered its report of findings today. ASG reviewed both winning and losing vendor proposals, as well as proposals from competitors in arriving at its findings. ASG developed a rubric to score proposals and provided both qualitative and quantitative feedback to the customer. The customer received a list of areas of strength, areas for improvement and red flag areas related to its proposals, as well as ways to improve the quality of its bids.

ASG completed its work assisting an assessment vendor in creating a metrics framework for its item development function. Item development is a major component of assessment system costs. The assessment vendor wanted to develop metrics for the functional area and selected ASG to assist it with the effort. ASG worked with the company to develop metrics frameworks for item development, forms construction and forms development.

ASG assisted the Tennessee Department of Education by reviewing the technical RFP submissions for its new Tennessee assessment system. ASG had previously assisted the department in creating the RFP for assessment services so was highly aware of the RFP content and the perfect choice to review vendor bids. The vendor bids were evaluated based on a defined rubric outlined in the RFP.

The Mississippi Department of Education selected ASG and its partner, WebbAlign, to conduct independent alignment studies for both its general and alternate assessments. ASG has worked with the department several times to develop its RFPs for assessment services and is intimately familiar with the state’s assessment system. WebbAlign is the premier organization conducting alignment studies according to the framework its founder, Norman Webb, developed years ago. The WebbAlign team conducted the alignment studies using Mississippi educators and independent experts and ASG reviewed the results and issued the final reports for the Department. The reports will be used by MDE in its federal peer review submission.

ASG completed a review of the item development function for a major assessment consortium. The review consisted of a complete evaluation of the development process steps, as well as a review of item development costs vs. industry averages. ASG documented each step in the development process and compared the process to industry best practices. Process improvements were recommended and potential cost reduction activities delineated.

ASG delivered a cost evaluation of the vendor bids for its new KPREP Assessment System to KDE. ASG used its industry unique Assessment Cost Model to complete the analysis. ASG also assisted KDE in writing the RFP for vendor services and during this process developed the vendor cost sheets for submission with the bid. ASG input the information contained in the vendor cost sheets into its assessment cost model to derive the appropriate vendor costs to develop and administer the assessment. The appropriate assessment costs are then compared against the vendor bid costs, on a function by function and line item basis, to complete the analysis. The final report provided Kentucky with solid evidence it could use in negotiating the final price with the selected vendor.

ASG completed three item development projects for a major assessment vendor. The projects consisted of a) an evaluation of its item development procedures, b) a cost analysis of its item development function, and c) creation of a custom bid template the organization can use to calculate its item development costs for customer bids. ASG used its knowledge of best practices in item development to review the vendors item development processes and its Assessment Cost Model to evaluate the vendor’s item development costs against industry metrics. Then, ASG created an interactive template the customer can use to develop its item development pricing for customer bids. The template provides the vendor with multiple pricing methodologies for developing its cost estimates including current vendor costs, anticipated future vendor costs and industry standards costs, each on a line item basis.

The Tennessee Department of Education (TDOE) selected ASG to assist it in preparing an RFP for the administration, scoring and reporting of its new assessment system, provide its proprietary Cost Sheets for inclusion in the RFP, and provide technical evaluations of the vendor responses to the RFP. ASG staff has written, evaluated and reviewed hundreds of state RFPs for assessment services and looks forward to working with TDOE in developing the RFP and evaluating the vendor responses.

Update – In December ASG completed its work for TDOE and the RFP is scheduled to be released in January 2019.

ASG recently completed two different cost analysis services for KDE. The first was an estimate of the costs for its new KPREP assessment system. KDE is operating under a firm budget for assessment services and wanted to make sure that the design of its new KPREP assessment system would fall beneath its assessment services cost cap. ASG evaluated the appropriate costs for the KPREP system and after recommending some minor changes to the assessment design, determined that the state could afford the new assessment system. Once this work was completed, ASG developed the vendor cost sheets for inclusion in the state’s RFP for assessment services. The cost sheets provide ASG with the information it needs to evaluate the vendor cost bids in response to the RFP. ASG had previously assisted KDE in developing the RFP.

ASG recently completed two important cost analyses for an assessment consortium. The first analysis consisted of building a fixed-variable pricing model the consortium will use to determine the appropriate amount to pay its assessment vendor as states and students are added to the consortium. The model separates the assessment into fixed and variable price components so only the variable cost components are calculated in determining the appropriate price increase as states and students are added to the consortium. The second analysis provided the consortium with a pricing model to determine appropriate customer service costs. The model uses inputs such as average handle time, average speed of answer, occupancy rates, availability rate and staff shrinkage rate to calculate appropriate customer service costs. The number of customer service representatives are calculated using the above variables and call center standard Erlang equations.

The process for conducting ASG’s 8th annual state assessment survey officially started today with ASG sending the participation letter to 50 state departments of education and the District of Columbia. Last year, 45 states and DC participated in the assessment survey which is designed to increase communication regarding issues of interest among the states. Participating states receive a database with all of the survey responses and can contact their peers in other states for further discussions. The complete database and a summary report will be issued by ASG in early 2019.

ASG completed its work with KDE and delivered the final draft of an RFP for assessment services for its new KPREP assessment. ASG worked with KDE and other industry experts to help design the KPREP assessment system and write the RFP for assessment services. The RFP included ASG’s vendor costs sheets. ASG has worked with KDE on numerous occasions over the past decade to perform cost analysis and RFP writing services.

The Florida Department of Education partnered with Assessment Solutions Group (ASG) and others to conduct an in-depth analysis of the feasibility and impact of using the ACT and/or SAT in lieu of Florida's statewide assessments as provided in ESSA. Would such a system meet the federal requirements for technical quality, adequate accommodations, and alignment to standards? Would it provide comparable data and fit into the accountability system? Representatives from Florida and the ASG team discussed the results of the study and implications for states.

The Every Student Succeeds Act (ESSA) has given states more flexibility and authority for implementing new assessment and accountability systems. Many states have taken advantage of this opportunity by creating Assessment Task Forces (ATFs) and/or stakeholder committees to review their current systems and solicit input around how to implement improved assessment/accountability systems. Some states have taken a very deliberate approach to the assessment/accountability system redesign and engaged a well-thought-out process to solicit feedback from educators, administrators, and the public on how to best assess their students. These states have used ATFs to gather extensive feedback and review their options for assessment/accountability systems. In this session, two states provided details on how they have worked recently with ATFs and ASG to gather input from key stakeholders and rethink how their assessment systems might be improved.

The Kentucky Department of Education (KDE) announced today that it has retained the services of ASG to assist it in writing two RFPs for assessment services, provide cost sheets for the RFP, and perform cost analysis of the vendor RFP responses for its next summative assessment. Kentucky and ASG have a long history of working together starting in 2010. The state has hired ASG to write each of its assessment RFPs and provide cost analysis services of vendor RFP responses since 2010. ASG completed its work in early 2018 on an RFP for College Entrance Exams and more will complete work on a major RFP for a new assessment program, to be issued by KDE in fall 2018.

ASG completed and delivered its report to the Florida Department of Education (FDOE) today on the Feasibility and Use of ACT and SAT in Lieu of Statewide Assessments on schedule earlier today. This important study included analyses of comparability, accountability, accommodations, alignment, and whether such an assessment system would be likely to pass UDED peer review. ASG managed the process and authored the peer review section of the report. Subcontractors University of Wisconsin (WebbAlign), Kansas University (Center for Assessment Accountability Research and Design), and University of Minnesota (National Center on Education Outcomes) completed the alignment, comparability/accountability and accommodations studies for the project.

The final report can be found at the following link: Final Report

The WIDA Assessment Consortium and ASG announced an agreement today to have ASG assist WIDA with several services in support of its ACCESS 2.0 English language proficiency assessment. The multi-year agreement will see ASG provide consulting services in a number of technical assessment areas, Information Technology consulting, RFP review services, and several different assessment cost analyses in support of WIDA’s ACCESS 2.0 assessment.