Results-based monitoring and achieving desired impacts

The perceived ineffectiveness of development aid over the last thirty years has prompted the donor community to aspire to a new development approach that can use financial inputs constructively and progressively. A core component of this contemporary developmental approach is monitoring and evaluation - an approach that allows financing institutions and implementing agencies to assess and evaluate both inputs and outputs. This has been internationally identified as a way in which funds can be used to achieve goals and objectives successfully. ProBEC is fully committed to such as an approach and has recently, in association with relevant experts, implemented an easily accessible, flexible evaluation and knowledge management system to facilitate this - Results Based Monitoring (RBM).

ProBEC has set itself goals and objectives for all its activities.  These are influenced by external factors that may bring about different impacts. While these impacts may not be directly attributed to ProBEC, there may still be positive impacts emerging from ProBEC activities, particularly around sustainable socio-economic growth. Desired impacts will occur only after necessary outcomes are achieved, and ProBEC is fully committed to ensuring these. 

• Goal – e.g. to improve health in homes
• Indicators – e.g. incidence of burns
• Data variables – e.g. number of occasions in the last year that a member of the household has suffered burns requiring medical attention
• Proxy data may prove more cost-effective
• 'Story' data is at least as important as numerical data – with a focus on the 'Most Significant Change' approach

Such information is captured via annual impact assessment surveys, incorporating both qualitative and quantitative research methods.

Surveys begin and database of stove users being developed

Each year, ProBEC undertakes impact assessment surveys in each country to establish user acceptance of the improved devices, monies and fuel saved, correct use of the devices, replacement of devices, effectiveness of marketing activities, and general demographic information on our target groups.

An M&E planning session for 2010 took place in Johannesburg with the regional M&E manager, Erika Schutze, and the recently appointed M&E officer from Tanzania, Mr Elias Simfukwe.

Mr Simfukwe has extensive M&E experience and hence will be performing regional support roles to those countries that need help with data analysis and SPSS.

As before, user impact assessments will be conducted on all household and institutional stoves. An added dimension to this year's surveys, particularly for producer surveys, will be the collection of some basic information related to carbon offset programmes. Examples include ascertaining what incentives are needed for accurate record keeping and participation in carbon offset programmes. Surveys will commence in March and are expected to be completed by July. A 5% sample is being used for all products.

Besides establishing overall user acceptance and correct use of the stoves, the overall objectives of the surveys are:

• To ensure that they reflect the new indicators
• Assessment of whether or not stoves are still in use, including quality assurance exercises
• More detailed fuel measurement information collection, as well as fuel savings
• Carbon compliance and more detailed assessment of biomass reduction, based on the recommendations of the monitoring plan submitted Atmosfair gGmbh, in preparation for the PDD development
• Gender specific questions to ascertain exactly how women benefit from the reduction in fuel and what the monetary savings are spent on, and what is done with their free time.

ProBEC is formalising its monitoring record keeping to prepare the project for the stringent requirements of carbon reporting and verification. In terms of monitoring, ProBEC is piloting the introduction of receipt books in order to formalise sales and to track stove users more accurately, at the point of sale. Receipt books were printed in Johannesburg and consist of carbonated paper that records three copies of each receipt so that one goes to the customer, one is kept by the seller and the final copy will then be collected by ProBEC.

A sample of receipt books has been posted to each country, to pilot along the value chain, from producer to wholesalers to retailer and to end user. Staff have been asked to train partners in this record keeping system, and to brainstorm with them what incentives would work to make such a system convincing.

The monitoring of a CDM PoA occurs in two layers, the PoA and the CPA layer. On the CPA level, national offices continuously gather primary distribution data (sales figures, serial numbers and distribution dates; full user information for a master sample). The managing entity processes and stores the primary distribution data (PoA Level). In every monitoring period the managing entity generates a sampling list for each CPA. The CME national offices conduct sample surveys and appliance tests following the sampling lists. The managing entity processes data obtained from the CPA sampling campaigns and produces one monitoring report per CPA and aggregates all CPA monitoring reports to a PoA monitoring report. The PoA monitoring report is submitted to and verified by the DOE.

In order to develop representative master samples, the users need to be traceable and stove sales figures must be credible. Information to be reported in the master sample databases includes serial number of appliances sold to user, the date of deployment, name of customer, address of customer, cellphone number of customer, if available. To this end, serial numbers for stoves have been introduced in most countries, and staff are trying to find innovative ways for partners and users to part with such information. Ideas have ranged from the introduction of competitions, to cellphone credits to outright cash payment for recordkeeping. It is hoped that the information and practices generated in this way could inform the monitoring and verification procedures of the new SADC Regional Carbon Facility that will be appointed to take over ProBEC.

Controlled Cooking Tests and Water Boiling Tests are being conducted on ProBEC improved stoves to complete ProBEC's Stove Technology Profile – a record of stove efficiencies – which will form part of the value proposition that ProBEC offers the potential private sector carbon partner.

For more information, contact Erika Schutze: Erika.schutze@gtz.de

Monitoring and evaluation

Report on the results of the surveys from 2009

All countries except for Lesotho, have dedicated monitoring and evaluation officers who submit monthly stove production figures to the regional monitoring and evaluation officer in the central office in Johannesburg. In addition, annual impact assessment surveys are conducted comprising the following:

• Household stove user impact assessment surveys (rural and urban)
• Institutional stove user impact assessment survey
• Producer impact assessment survey

Correct use of stoves ensures their efficiency and hence six criteria for correct use were also surveyed.

In many cases, we do not have specific figures on how much money is saved through wood fuel savings since there was no information on how much wood costs provided in the surveys. Fuel pricing surveys are being conducted in 2010 to obtain such information. M&E officers will be tasked with gathering the required fuel information that is outstanding, as well as collecting wood fuel pricing data by visiting three vendors of each fuel type and get prices per kilogram of wood / charcoal, so that we can obtain a "typical” price on the street.

The impact assessment survey process of 2009 revealed that the research skills level in each country differs significantly, with only Malawi staff being able to compile information at the required level. To a lesser extent the Zambian reports provided more comprehensive information. In the rest of the countries, data analysis had to be outsourced to university-based consultants, and even then, the resulting reports were weak. Nonetheless, conducting the impact assessment surveys was a valuable capacity building exercise and revealed some interesting as well as unexpected results. Some M&E officers will receive training in M&E research methods is a priority for 2010.

Obligatory Indicator 1: "The number of improved energy devices promoted has increased from 60 000 in 2008 to 125 000 in December 2009.”

Indicator 1.1: "Annual sales figures of improved basic energy devices increase from 97,000 hh stoves, 1322 institutional stoves and 193 SME devices in May 2008 to 120,000 stoves, 4179 institutional stoves and 478 SME devices in December 2009.”

Indicator 1.2: "The correct use of improved basic energy devices increases from 40% in May 2008 to 50% in December 2009.”

Indicators 1.3: 40% of women involved increase their disposable income by 10% through the sale and use of energy efficient stoves from September 2005 to December 2009.

	Stoves produced 1 June 2008 – 31 Dec 2009
Mozambique	26988
Lesotho	632
Malawi	31348
Tanzania	26381
Zambia	24851
TOTAL	110200

SUMMARY OF STOVE PRODUCTION FIGURES

1 JUNE 2008 – 31 DECEMBER 2009

Tanzania

Household stoves

90% of households use the rocket stove everyday for cooking, however, there is a demand for charcoal stoves as charcoal is more easily available, and many beneficiaries are using the improved stoves with charcoal even though they are designed for firewood. This is because government discouraged the use of charcoal and ProBEC was therefore prohibited from promoting charcoal stoves. This changed in the latter half of 2009 and ProBEC is now promoting charcoal stoves. In the case of tea estates that provide firewood, this is not a problem.

70% of the household stoves were in good condition with most of the damages being due to incorrect use, which calls for more regular user training. Posters have been disseminated across the region with graphical instructions on how to use the stoves correctly, and follow-up training is planned.

In Tanzania there are no specific figures available on how much money is saved by wood saving, however 94.3% of households experience fuel saving as the most important advantage when using the improved stoves. Most households buy wood from vendors except for Njombe where 80% collect it from the bush or alternatively, it is supplied by tea companies.

Institutional stoves

In Tanzania, institutional stoves are promoted but have not reached the target to sell 400 institutional rocket stoves by December this year, however orders are rising and 50 schools have already placed orders for rocket stoves. The main reason for poor performance in sales of institutional rocket stoves is lack of stove marketing campaigns in schools outside Dar es Salaam where the higher demand for wood stoves. Majority of school chefs (60%) use the institutional rocket stove everyday preparing meals for the students. The highest usage rates of rocket stove are found in almost all private owned schools, government schools and few donor funded institutions. In these schools rocket stoves are used four times a day because students are served four meals per day including porridge, breakfast, lunch and supper. This was observed by all schools because these schools have large number of boarding and day scholars that eat regularly at school.

In Tanzania only 60% of school chefs had been trained in how to use the stoves correctly and so more training is needed. With institutional stoves, at least 3 out of 6 good cooking practices are being observed in 100% of cases.

In 75% of schools, wood collection is primarily done by private individuals (wood vendors) that have been given tenders by the school administration to supply wood in schools.

Increase in income

13 stove producers experienced an increase in their income as a result of their involvement in stove businesses, while only 3 (one portable stove producer and two fixed stove builders) did not have an increase in income. The reasons for the latter were that the stoves are too expensive, and a general lack of awareness of the stoves pointing to the need for stronger stove promotion.

For example, four fixed stove builders in Tabora region building fixed household Changu stoves and institutional stoves said since they became involved in building stoves their income has increased gradually as they build stoves. Before, they earned money from building houses and rocket barns only during tobacco curing season, but with introduction of stoves they have increased their income.

Zambia

Household stoves Zambia

Overall, the Pulumusa is popular and regularly used, however it is supplemented by use of less efficient devices. At least 80% of the respondents use a Pulumusa every day.

Out of the 143 households interviewed 137 were using the Pulumusa stove, 94 respondents had both the Pulumusa and an mbaula; 43 respondents had only a Pulumusa and no Mbaula; 4 had an Mbaula but no Pulumusa; 2 had no Mbaula and no Pulumusa.

In terms of correctly using the stoves, at least 80% of the households fulfilled 4 of the 6 criteria required.

Income saving

81% the users reported saving money through using less fuel, and 45% of the users spent their savings on groceries and 17% where able to make investments in income generating activities. Before the introduction of the pulumusa stove the average expense on charcoal per day was K 3078.00 and on the with the Pulumusa stove this was almost halved to K1558.00. Likewise, charcoal consumption dropped from 3.3 Kg to 1.7 Kg per week with the Pulumusa stove. 70% of Zambian households save at least 51.5% of biomass because of correct use of the stove and 40% of women involved have increased their disposable income by at least 50% through the use of the stove.

There is a high demand for the Pulumusa stoves from those who have heard about them and even from those who have one already; unfortunately the rate production does not meet the demand, because producers claim not to have material and so cannot build up stock. Producers of the pulumusa stove reported an increase in their income due to the addition of the stove to their production line. However, since the majority of the groups have been producing the Pulumusa for less than a year, the business is still in its infancy and so far only 30% of their income can be attributed to the sales of Pulumusa stoves. Nonetheless, the producer groups sell an average of 26 stoves a month, and 58% of the producers trained have continued to produce Pulumusa stoves. 83% of the producers said they use their profits to reinvest in their business. Concerning the indicator requiring an increase in income: 91.7% of the Pulumusa producers claim that its production is economically profitable, and out of these 50% say the Pulumusa sales contributes to 20% of their income.

Institutional stoves in Zambia:

91.5% of the schools used the stoves correctly and followed at least 3 of the correct use criteria. With the money saved through the introduction of efficient institutional stoves, the schools reported being able to balance the diet for the children with money saved from fuel. Overall, the schools have reduced the expenses on fuel and have reduced the number of cooks from many to one and have appointed permanent cooks. ProBEC has also carried out user training on Institutional stoves at World Food Programme (WFP) schools. The purpose of the training was to sensitize school and community members on the fuel efficient stove project and conduct practical demonstrations on the use of the stove since so much of a stove's efficiency is based its correct use. The objective of the programme includes orienting teachers, parents, as well as School Food Management Committee members and cooks to the background of the project, benefits of practicing energy conservation technologies in general, and of using the fuel efficient stoves.

Tobacco barns Zambia

Tobacco cured using the ProBEC-promoted Rocket barns fetches more money on the market as a huge sales of this tobacco was recently recorded on the auction floors in May 2009 in the Southern Province town of Choma. The farmers were delighted to see that tobacco which was cured using the traditional barns was fetching an average of U$2.2 per Kg while tobacco cured in the ProBEC-promoted rocket barns fetched between US$2.8 and US$3.3 per Kg.

Malawi

Household clay stoves and fixed stoves (Changu and Esperanza)

There is a significant uptake of the ProBEC-introduced improved stoves. In Malawi, households save a lot of money through firewood savings. The clay stove producer groups consist mainly of women. The production rates per producer differ considerably between two and ten stoves per month. Stove businesses tend to be a part-time activity with potential for scaling-up. Nevertheless, it is the most important income for 30% of the interviewed producers and the second most important for 35%. Most producers have no expenses for stove production, as clay is available for free.

Without the use of the improved stove, households that do purchase their firewood spend 2.82 US dollars per week or 11.30 US dollars per month on wood which is a lot for the target group. In Malawi the clay stove offers considerable monetary savings and users spend around 600MK while those households that cook the traditional way spend between 200MK up to 1500MK per month.

The clay stove is the primary device used for cooking food by the household on daily basis. 79.2% of rural households use clay stoves for cooking, followed by the three-stone fire. Of these, 85% of the households use clay stoves for more than two thirds of their cooked meals. Firewood collection is reduced by roughly 36% due to the frequent use of the clay stove. 73% of the three-stone fire users collect wood once a week or more often, while only 45% of the households cooking on clay stoves collect that frequently. Households frequently cooking on the clay stoves spend 11% less time on firewood collection than those mainly cooking on three-stone fires. Since women or young girls are responsible for this task, they benefit in particular.

While before clay stove producers relied heavily on ProBEC to do marketing, they are now realizing that it is their responsibility to create the market for the stoves and ProBEC will provide a document on marketing of clay stoves to them and provide training on how to be a ‘seller' as well as a ‘producer'.

Of the fixed stoves installed in tea estates' staff houses, 85% of the households use them for more than two thirds of all cooked meals. In the staff houses, 55.6% of the households are perfectly loading their fuel and in all the observed kitchens, 3 of the 5 criteria that were being assessed to establish correct use are practiced: correct storage and preparation of wood fuel and the use of pot lids while cooking.

88% of households collected their own firewood, and 3% are given wood by the company.

Institutional Rocket stoves (IRS)

An average of 1355 people is being served by institutional stoves in a day, using an average pot size of 121 litres. The highest numbers of people are served in schools where donor feeding programmes occur: World Food Programme WFP serves 1048 pupils and Mary's Meals serves 1851 pupils. In the case of the school feeding programme, stoves were donated by the donors whose interest, among other things, for the intervention has been to increase enrollment and retention rate of the pupils in most of the primary schools.

88% of users cited fuel saving as the main advantage, followed by 83% who cited the speed of cooking, and this is probably so because the bulk of the responsibility for obtaining wood falls on the cooks. Firewood is collected on a daily basis by the volunteer cooks, but 8% of institutions buy wood and spend on average US$86.16 per week. Private institutions and some public colleges provide the wood. In the case of the primary schools, the communities surrounding the school in collect it. There are instances where volunteer cooks and pupils are expected to bring firewood and so alternative ways of making firewood available for cooking for the school feeding programme must be explored.

However, concern was expressed about the weight of these stoves since they must be moved from where they are stored to where the cooking actually takes place. In some cases, the distance between the two is about 20 meters and in most cases the stove is just dragged to the cooking location, thus weakening stove parts and hastening damage. Therefore it is proposed that a storeroom should be attached to the kitchen.

Now that Malawi has been declared a ‘food secure' country, one of the main buyers of the IRS - the WFP - is leaving the country, and so the sustainability of this technology is threatened unless another approach is pursued, such as approaching private schools to sell them stoves.

Lesotho

Lion/Tau institutional stoves

37 schools were covered in the survey, including private and donor-funded schools. A total of 10316 students receive food in the schools covered, another 254 including teachers and cooks also get food giving a total of 10570 people having food in the schools. Of these, the students are served with one meal, lunch only in 26 schools and they have breakfast and lunch in 7 schools.

When the Tau stove is used, firewood expenditures are reduced by 67%. It is the cook's responsibility to source fuel as a requirement for the contract with the Ministry of Education. No schools have budgets for firewood and do not contribute in any way to source for firewood. If cooks buy firewood, they have to use part of the money they receive for the catering services to buy it. Firewood is expensive and not readily available in many areas. All financial benefits are therefore realized by the cooks.

Most schools are served by caterers or cooks that are registered with The Ministry of Education. The cooks follow a prescribed menu and supply both the food as per the menu and the firewood to do the cooking. The Government pays them per daily pupil consumption. The teachers and cooks gave M1.00 and M1.50/pupil for the WFP feeding where two meals are served per day. For catering they gave M3.00 and M3.50 / pupil and only one meal is served per day.

Overall there is a slow uptake of the stoves' use. The lion stoves may be found in use in 9 schools (25%) whilst the rocket stove is in 5 schools (14%). This is because of the conflicted messages around responsibility for cooking with the ministry having changed the programme from one of cooks preparing meals at the schools to cooks preparing meals at home and then bringing them to school. Furthermore, due to the high turnover of cooks in the school, many cooks do not know how to use stoves and hence a training plan is underway. In another school, both cooks and the school head claim that they did not know that the stoves were built for them to use. They claim to be waiting for clarification from the Ministry of Education. In another school, the school head claims that the stoves were built without his permission and the school did not need the Tau stoves. Results however also indicate a low level of awareness on the benefits of using the stoves and the need for awareness campaigns on the benefits and correct use of the stoves. All cooks responded positively in all cases where the cooks were interviewed on their willingness for training.

Most of the stoves are outside without adequate shelter and therefore in cold weather they are not desirable for use. Likewise some schools only have access to wet wood and do not have shelters to dry it out.

In general, the monthly costs for providing one meal to one person every school day rises sharply when fuel is bought as compared to when fuel is partly bought and partly collected for free. Firewood, twigs and shrubs are either collected for free from private plantations, the forest or veld. This is collected by the cooks and their families or they hire people to collect for them. Apart from collecting as head loads, firewood is also collected by means of wheelbarrows, ox drawn carts, donkeys or trucks. The firewood is collected from areas varying from 3km away to distances of 8km away. The frequency of collection varies from daily to half yearly where firewood is stock piled. Depending on the distances, the quantities collected and availability, collection time varies from 3hrs to two days.

Mozambique

Bakery ovens

The promotion of bakery ovens was reintroduced into programme activities in 2009 after it was established that the reason for the slow uptake in the past was the prohibitive cost of the bakeries which was due to the high mark-up costs being applied by intermediary NGOs responsible for building them. A new dedicated ProBEC team thus built 10 ovens this year and the survey covered 4 of them. Overall, users were happy with the stoves and reported wood savings and hence an increase in income. The money saved by using improved stove is used to buy more wheat which in turn increases income, to pay the workers and to save the money for future investment.

Clay & mud household stoves

There has been a sudden surge in production of these stoves and a high level of acceptance, due to the dedicated efforts of the community based activists who are given bicycles and cellphone credits by the partner organizations to promote the stoves and train people how to make them. This remains a self-help approach which is taking off. 55% of the stoves were given as a gift, 15% bought them, and around the rest are self-made.

Out of a sample of 320 families, most of the respondents have fixed mud stoves (96, 7% in Sofala and 73,6% in Manica), while clay stoves represent 5, 6% in Sofala and 35,8% in Manica. They cook on them 3 times a day.

Users have experienced a drastic reduction in money and wood costs – before using the stoves they were spending between 20 and 25 Meticais (Mts) per week and after it dropped to 11Mts.

They also experienced a reduction in frequency of collecting wood which now lasts twice as long as before the stove introduction. Likewise, time taken to collect wood diminishes: before 3 days and after between 1,5 days and 2,5 days.

In terms of correct use, 53% are loading the fuel in the clay stove perfectly, 70% use correct pot sizes, 60% of the users always use lids, and therefore 3 out of 6 criteria are being fulfilled.

The rural programme is expanding in 2010 in a new collaboration with Heifer International and World Vision. A baseline study is being conducted to assess the requirements for the implementation of a rural programme in Gaza and Inhambane provinces, which will expand on such activities currently taking place in Manica and Sofala. ProBEC, in collaboration with its new partners, Heifer International and World Vision, has been running stove training sessions in Nampula province to teach potential stove producers of household stoves.

Institutional stoves

Due to the recent introduction of this intervention, and its near collapse due to the abandonment by the World Food Programme, only 10 institutions were sampled. Most of the visited institutions are open center (57%) followed by boarding school (29%) and at end mixed institutions (14%). Mixed schools are those where some students stay at school and other stay at school part of the day.

67% of the schools use the stoves every day. 88% of the cooks are women. Most of the visited institutions said that they buy wood (75%) while 25% said that they collect wood from the bush.

Regarding the number of meals prepared, 72% prepare two meals for students going home, 14% prepare three meals and the same number said that they prepare one meal.

70% of the respondents said that improved stoves save wood while 50% of the respondents said that improved stove is rapid.

Schools have experienced wood reduction with the use of the stoves - before using improved stoves, school used to spend about 3 hours per day on wood collection but after using improved stoves, schools spend 2 hours on wood collection. Since this is mainly the responsibility of pupils, they benefit from having more time and more energy to attend to school work.

In those schools that bought wood, on average, before using improved stoves, the schools were spending 594 Mts per week on wood and after, the expenditures on wood were reduced to 392 Mts per week. The money saved as a result of using improved stoves, is used for buying more other cooking ingredients, to increase the quantity of food for students, and to buy medicine.

In terms of correct use, all respondents said that they usually cut wood before using it, 90% of the users use dry wood for cooking, and all use the lid while cooking, therefore 3 out of 6 criteria for correct use are being fulfilled.