Research, consultation, planning, communications

The Monitoring and Evaluation System (MES) project for the New Zealand Community Funding Agency (NZCFA).  was designed to help enhance the effectiveness of services provided under the NZCFA’s Family/Whanau Resource Development funding programme.  It also served as a pilot for the introduction of outcome monitoring of the NZCFA’s programmes.


The NZCFA administered the Family/Whanau Resource Development programme on behalf of the Government.  The programme funded a wide range of parenting and lifeskills support and educational activities aimed at strengthening family relationships and parenting skills.  These activities were provided to individuals, families and whanau through NZCFA contracts with community-based service providers.


The MES computer-based monitoring system was developed in two stages:

  1. Bulletan initial design and development in which a preliminary monitoring/ evaluation system was developed and tested with a sample of cases; and

  2. Bulleta second stage which refined and implemented the system, and undertook an assessment of over 2900 cases covering over 5100 clients completed in the latter part of 1995.

The monitoring/evaluation system was initially developed and piloted through a series of intensive interviews/discussions with a selected range of service providers and NZCFA staff.  The design of the system was strongly influenced by the  need to:

  1. Bulletbe culturally appropriate;

  2. Bulletintegrate output and outcome monitoring to avoid unreasonable compliance costs on the providers and on the NZCFA; and

  3. Bulletavoidance of  interference in the relationship between the provider and client.


The MES was client-focused, but provider-based.  It contained a nested hierarchy of assessments reflecting the interests of the various stakeholders.  At the core it evaluated client goal satisfaction.  Then followed assessments of issue/problem resolution, and family strength which also related to the particular service provided to the family or whanau and its outcome.  Outside this was the assessment of the extent to which the services met the needs identified in the service plans of the NZCFA area office.  These all contributed to the final level of assessment, an evaluation of the effectiveness of the services within the funding programme in meeting the programme’s objectives.  Contributions to the assessments were made by the clients, the service providers, the outreach workers, and the area office teams.


Design of the monitoring programme was based on a was based on the ‘stakeholder service’ model.  This contemporary approach to programme evaluation is founded on the understanding that evaluations will better aid programme management when tailored to the information needs of stakeholders who are close to the specific programmes being evaluated.   The key to this approach is the way in which stakeholders play major roles in specifying problems, questions, and interventions, with the evaluator acting as a coordinator of this information.  One of the principles of the approach is close contact with stakeholders to respond to changing needs and to maximise the use of the results.


The stakeholder interest in the system operated at four different levels:




 
















Interviews revealed the pervasiveness of what might be called a “diagnostic model” that structured perceptions and discussions of the Family/Whanau Resource Development NDOC and the services provided.  The core of this model was the notion of the diagnosis of a deficiency or dysfunction in the family, which is to be remedied with some sort of intervention, resulting in restoration of a “healthy” situation, characterised by  a minimum of deficiency or dysfunction.   Dysfunctionality was variously defined but included the following attributes:

  1. Bulletinability to relate to peers;

  2. Bulletinability to relate to authority;

  3. Bulletlack of lifeskills;

  4. Bulletinability to communicate and negotiate; and

  5. Bulletinability to manage resources in the broadest sense.


This dysfunctionality is expressed in a range of “symptoms” including truancy, petty crime, and stressed relationships within the family.


This underpinning structure, and its associated notions of causality, provided a basis for a client-focused system evaluating the outcomes of the services in this NDOC.  It basically structured the requirements of the evaluation system and defined its coverage into three components:









While most of the monitoring information was initially gathered by the service provider, the monitoring system can in fact be seen as a nested hierarchy of evaluations, each with a different focus, reflecting the interests of different stakeholders in the funding programme and the evaluation comprising:

  1. Bulletclient assessment of goal satisfaction and service;

  2. Bulletassessment of issue severity;

  3. Bulletfamily strength assessment;

  4. BulletOutreach worker assessment;

  5. BulletArea team assessment; and

  6. BulletNational Office assessment.

 

The MES was computer based but with minimal requirements so as not to tax the equipement used by the various providers. The program, written in Microsoft Visual Basic, operated on any computer capable of running MS Windows 3.0 or higher.  While the program would run on a 80286 machine with 1MB of RAM, performance improved with more capable machines (a 80386 with 4MB was the preferred minimum configuration).  The program itself was quite small (426Kb) although over time data storage requirements increase as more cases are entered. 


Data downloading onto floppy disk was done automatically with a small program.  The only decision required of the computer operator was to choose whether to download the extra screens which were included in the program to enable service providers to  record other information about their cases which is not required by the MES.


The primary data analysis for the ongoing monitoring of Family/Whanau Resource Development services was undertaken through MS Access.  A wide range of queries, tables and reports were prepared for the analysis.  Filters were also developed to enable the results from specific service types and individual providers to be extracted easily.