Effective Vaccine Management (EVM) Global Data Analysis 2010-2013 EVM setting a standard for the vaccine supply chain Update: WHO EVM database, December 2014 1
The Effective Vaccine Management (EVM) Assessmentg What does EVM assess? EVM assesses each level of the supply chain Primary level (PR): Vaccine stores that receive vaccine direct from an international vaccine manufacturer or distributors or a local vaccine manufacturer. Sub-national level (SN): Vaccine stores that receive vaccine from a primary store or higher level sub-national store. There may be 0, 1, or more SN levels. Lowest distribution level (LD): Vaccine stores that receive vaccine from a primary level or a subnational store and supply vaccine to one or more health facilities. Service point level (SP): Facilities that receive vaccine from any higher level store and supply immunization services. How does EVM assess? Vaccine Manufacturer or UNICEF National store (Level PR) Provincial stores (Level SN) District stores (Level LD) Health facilities (Level SP) EVM assesses 9 areas of vaccine management the 9 EVM Criteria E9 Information systems & supportive functions E7 Distribution E5 Maintenance E8 Vaccine management E6 Stock management E4 Buildings, equipment, transport E1 Pre-shipment & E2 arrival Storage temperature E3 Capacity A representative sample of sites is selected at each level of the supply chain. Each of the 9 EVM Criteria is assessed at each supply chain level by observation, inspection of infrastructure and records, and by interview of staff. Process and performance indicators are evaluated in each of the 9 areas at each level. Indicator scores are combined to give criterion scores for each area at each level. An area of vaccine management is considered Effective if its criterion score is greater than or equal to 80% - the EVM standard. 2
Assessments (2010-2013) 75 countries have conducted at least one EVM assessment between 2010 and 2013. 4 countries, Senegal, Vietnam, Cameroon and Sierra Leone, have conducted 2 assessments. For those countries that have conducted 2 assessments, only the years of the most recent assessments are shown. Data Source: WHO EVM database, November 2014 Map production: Immunization Vaccines and Biologicals, (IVB), World Health Organization Date of slide: 10 September 2014 2010 (5 assessments) 2011 (30) 2012 (20) 2013 (11) The boundaries and names shown and the designations used on this map do not imply the expression of any opinion whatsoever on the part of the World Health Organization concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted lines on maps represent approximate border lines for which there may not yet be full agreement. WHO 2014. All rights reserved 0 875 1,750 3,500 Kilometers 3
EVM Criterion Scores 2010-2013 EVM setting a standard for the vaccine supply chain Update: WHO EVM database, December 2014 4
EVM Criterion Scores (2010-2013) Primary (70 assessments) Sub-national (53) Lowest distribution (70) Service point (70) Maximum score 75 th percentile Median score 25 th percentile Minimum score E1 E2 E3 E4 E5 E6 E7 E8 E9 Vaccine arrival Temperature monitoring Storage capacity Buildings & equipment Maintenance Stock management Vaccine distribution Vaccine management Information systems Globally, there is a broad range of performance in each criterion at each level; only E3 at the primary level has a median score above 80%. E3 Storage and transport capacity and E4 buildings and equipment are relatively strong at each level of the supply chain. Among the weakest areas are E2 Temperature monitoring at the national level, E5 Maintenance at the lower levels, E6 Stock management 5 at the lower levels, and E7 Distribution to the lower levels.
EVM Country Composite Scores (2010-2013) 0 850 1,700 3,400 Kilometers Data Source: WHO EVM database. November 2014 Map production: Immunization Vaccines and Biologicals, (IVB), World Health Organization Date of slide: 23 February 2015 The boundaries and names shown and the designations used on this map do not imply the expression of any opinion whatsoever on the part of the World Health Organization concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted lines on maps represent approximate border lines for which there may not yet be full agreement. WHO 2014. All rights reserved Bottom 25% (18 countries) 0 875 1,750 3,500 Kilometers Percentiles 26 to 50 (17 countries) Percentiles 51 to 75 (17 countries) Top 25% (18 countries) 6
EVM Criterion Scores by Region 2010-2013 EVM setting a standard for the vaccine supply chain Update: WHO EVM database, December 2014 7
E1 Vaccine arrivals (2010-2013) Maximum score 75 th percentile Median score 25 th percentile Minimum score AFR EMR EUR AMR SEAR WPR (n=37 assessments) (n=4) (n=10) (n=1) (n=8) (n=6) EMR and EUR score higher on average than the other regions. More than half of their primary stores assessed surpass the EVM standard. 8 Less than 25% of the primary stores in the other regions meet the standard.
E2 Temperature monitoring (2010-2013) Primary (PR) Sub-national (SN) Lowest distribution (LD) Service point (SP) Maximum score 75 th percentile Median score 25 th percentile Minimum score AFR EMR EUR AMR SEAR WPR (37 assessments) (n=4) (n=10) (n=1) (n=8) (n=6) More than 75% of countries in EMR & EUR meet the standard at LD & SP; more that 75% of EUR countries also meet the standard at SN. Otherwise, with very few exceptions, temperature monitoring is below the EVM standard of >= 80%. 9 The PR E2 scores are low in all countries in all regions.
E3 Storage capacity (2010-2013) Primary Sub-national Lowest distribution Service point Maximum score 75 th percentile Median score 25 th percentile Minimum score AFR EMR EUR AMR SEAR WPR (37 assessments) (n=4) (n=10) (n=1) (n=8) (n=6) At each level of the supply chain in AFR, about 75% of the countries assessed have significant storage capacity shortfalls. The situation is slightly better in SEAR and WPR; at each level about 50% of countries have significant shortfalls. At each level in EMR and EUR, less than 25% of the countries have shortfalls. 10
E4 Infrastructure (2010-2013) Primary Sub-national Lowest distribution Service point Maximum score 75 th percentile Median score 25 th percentile Minimum score AFR EMR EUR AMR SEAR WPR (37 assessments) (n=4) (n=10) (n=1) (n=8) (n=6) AFR has slightly poorer infrastructure overall than in other regions; at each level less than 25% of countries meet the EVM standard of >= 80%. More than 75% of primary stores assessed in the other regions meet the EVM standard. 11 More than 75% of countries in EMR and EUR also meet the standard at SN level, and 3 of the 4 EMR countries meet the standard at LD.
E5 Maintenance (2010-2013) Primary Sub-national Lowest distribution Service point Maximum score 75 th percentile Median score 25 th percentile Minimum score AFR EMR EUR AMR SEAR WPR (37 assessments) (n=4) (n=10) (n=1) (n=8) (n=6) Only the PR and SN median scores in EMR exceed the 80% EVM standard for preventive maintenance of equipment and buildings; all 4 PR stores and 3 of the 4 SN levels assessed meet the EVM standard. 12 The maintenance of cold chain infrastructure gets progressively weaker as vaccine flows through the supply chain from PR to SP.
E6 Stock management (2010-2013) Primary Sub-national Lowest distribution Service point Maximum score 75 th percentile Median score 25 th percentile Minimum score AFR EMR EUR AMR SEAR WPR (37 assessments) (n=4) (n=10) (n=1) (n=8) (n=6) With a couple of exceptions, none of the regions meet the EVM standard of >= 80% in stock management. The exceptions are the PR, SN and LD levels in EMR; all 4 PR stores and 3 of the 4 SN and LD levels assessed score above 80%. 13 Stock management procedures and performance gets progressively weaker as vaccines flow through the supply chain from PR to SP.
E7 Distribution (2010-2013) Primary Sub-national Lowest distribution Service point Maximum score 75 th percentile Median score 25 th percentile Minimum score AFR EMR EUR AMR SEAR WPR (37 assessments) (n=4) (n=10) (n=1) (n=8) (n=6) In general, all regions perform poorly in the organization of vaccine distribution between the different levels of the supply chain. EMR and EUR countries score slightly higher in general than the countries of other regions. 14
E8 Vaccine management (2010-2013) Primary Sub-national Lowest distribution Service point Maximum score 75 th percentile Median score 25 th percentile Minimum score AFR EMR EUR AMR SEAR WPR (37 assessments) (n=4) (n=10) (n=1) (n=8) (n=6) EMR and EUR score highly in vaccine management; at each level more than 75% of the countries meet the EVM standard of >= 80%. 15 In AFR, PAH, SEAR and WPR, at each level of the supply chain, less than 25% of countries meet the EVM standard (exception WPR PR).
E9 Information systems (2010-2013) Primary Sub-national Lowest distribution Service point Maximum score 75 th percentile Median score 25 th percentile Minimum score AFR EMR EUR AMR SEAR WPR (37 assessments) (n=4) (n=10) (n=1) (n=8) (n=6) With a few exceptions, information systems are weak at each level in each region. 16 EMR PR and SN and EURO SN and LD are exceptions; more than 75% of countries assessed meet the EVM standard of >= 80% at these levels.
Selected EVM Indicator Scores 2010-2013 EVM setting a standard for the vaccine supply chain Update: WHO EVM database, December 2014 17
Selected EVM Indicator Scores (2010-2013) Explanatory note: There are 400 indicators at PR level, 300 at SN, 200 at LD and 150 at SP. Most indicators assess compliance with EVM standards (Eg. % of cold rooms with continuous temperature monitoring). Some indicators measure performance (Eg. % of facilities with accurate stock records). Some indicators describe the context (Eg. % of stores that store vaccine at -20 C). Such context indicators are not scored, but are used to customize the questionnaire they turn OFF non-applicable indicators. The following slides present global results for selected indicators. The first 3 slides show results for selected context indicators. The remaining slides present results for selected scoring indicators. The scoring indicators are grouped into 3 categories: Availability:those indicators likely to have a direct impact on the availability of vaccine at the service delivery level. Quality: those indicators likely to have a direct impact on the quality (potency) of vaccine at the service delivery level. Efficiency: those indicators likely to have an impact on the operational cost of the immunization supply chain. Indicator data exist for 58 countries across 6 WHO regions: 67 primary stores 406 sub-national stores 896 lowest distribution stores 949 immunization service facilities. 18
Context 1: Infrastructure (2010-2013) % of vaccine stores that store vaccine at -20 C % of facilities that store vaccine in cold rooms or freezer rooms Primary Sub-national Lowest distribution Service point % of refrigeration units that are CFC-free % of facilities with voltage fluctuations greater than +/-15% % of facilities that deliver or collect vaccine using refrigerated vehicles *The data was collected between 2010 and 2013 in 58 countries across all 6 WHO regions. 19
Context 2: Policy (2010-2013) % of health facilities that have adopted the MDVP Primary Sub-national Lowest distribution Service point % of health facilities that bury used and discarded vials on site % of health facilities that incinerate safety boxes on site *The data was collected between 2010 and 2013 in 58 countries across all 6 WHO regions. 20
Context 3: Outsourcing (2010-2013) % of national stores that outsource customs clearance Primary Sub-national Lowest distribution Service point % of national stores that outsource transport services % of national stores that outsource equipment maintenance services *The data was collected between 2010 and 2013 in 58 countries across all 6 WHO regions. 21
Availability indicators (2010-2013) % of stock record templates that contain all required fields % of facilities with matching vaccine and diluent stock levels (mismatch <1%) Primary Sub-national Lowest distribution Service point % of facilities with accurate vaccine stock records (mismatch with actual stock level <1%) % of facilities that were able to fully satisfy all requests from lower level facilities or for outreach in the past year *The data was collected between 2010 and 2013 in 58 countries across all 6 WHO regions. 22
Quality indicators (2010-2013), 1 % of countries that have carried out a systematic temperature monitoring study within the past 5 years % of cold rooms for which a fully documented temperature mapping report is available Primary Sub-national Lowest distribution Service point % of facilities in which all vaccine refrigerators comply with WHO specifications % of facilities for which kerosene or gas is always available (where absorption refrigerators are used ) *The data was collected between 2010 and 2013 in 58 countries across all 6 WHO regions. 23
Quality indicators (2010-2013), 2 % of facilities in which all cold and freezer rooms have continuous temperature recorders % of facilities in which all vaccine refrigerators have continuous temperature recorders or freeze indicators % of facilities that pack freeze indicators with deliveries of freeze-sensitive vaccines Primary Sub-national Lowest distribution Service point % of health facilities in which all VVMs are before the discard point *The data was collected between 2010 and 2013 in 58 countries across all 6 WHO regions. 24
Quality indicators (2010-2013), 3 % of storekeepers and health workers that know which vaccines on the schedule can be damaged by temperatures below 0 C % of storekeepers and health workers that can condition icepacks or cool chilled water packs in accordance with WHO guidelines % of storekeepers and health workers that know when and how to conduct the shake test Primary Sub-national Lowest distribution Service point % of storekeepers and health workers that know how to read VVMs *The data was collected between 2010 and 2013 in 58 countries across all 6 WHO regions. 25
Efficiency indicators (2010-2013), 1 % of facilities with an up to date cold chain equipment inventory % of refrigerators that are fully functional Primary Sub-national Lowest distribution Service point % of facilities with a functional telephone, radio or internet connection % of facilities with a computerised stock management system *The data was collected between 2010 and 2013 in 58 countries across all 6 WHO regions. 26
Efficiency indicators (2010-2013), 2 % of facilities that calculate vaccine wastage rates for each vaccine % of facilities that use vaccine wastage rate data to forecast vaccine needs Primary Sub-national Lowest distribution Service point % of facilities that lost less than 1% of vaccine stock in the past year due to temperature damage % of vaccine arrivals for which there is a correctly completed VAR *The data was collected between 2010 and 2013 in 58 countries across all 6 WHO regions. 27
Efficiency indicators (2010-2013), 3 % of facilities with a Standard Operating Procedures (SOP) manual % of facilities with a satisfactory SOP describing a contingency plan in the event of equipment failure or other emergency Primary Sub-national Lowest distribution Service point % of facilities that receive regular supportive supervision % of health workers that know how to apply the MDVP *The data was collected between 2010 and 2013 in 58 countries across all 6 WHO regions. 28
Links EVM public access website ( http://www.who.int/immunization/programmes_systems/supply_chain/evm/en/ ) A one-stop shop for all things EVM: Download EVM tools and user guides: Background EVM documents EVM assessor training materials EVM assessment report template EVM site selection tool and user guide EVM Assistant tool and user guide. This tool facilitates the calculation of available and required storage and transport capacities. EVM Analysis & Recommendations tool. This tool facilitates the identification of gaps and drafting of recommendations. Link to EVM Assessors website ( https://extranet.who.int/evm/ ) Download latest versions of the offline assessment tool and questionnaire. Upload completed EVM assessment reports and datasets. This website is for registered EVM assessors only. To register you must first complete a WHO EVM assessor training course. Link to EVM e-learning website ( http://apps.who.int/immunization_delivery/systems_policy/logistics/evmlearning/index_0_1_1.php ) An online training course on how to use the EVM assessment tools. This course compliments the formal WHO training course, it does not replace it. It is highly recommended that you complete this training BEFORE you start using the EVM tools. 29
Slide Notes EVM setting a standard for the vaccine supply chain 30
Slide Notes Slide 2 Each criterion is assessed at each level by observation, inspection and interview: Inspection of cold chain equipment, transport vehicles and buildings Inspection of records (temperature, stock, wastage, ) for the previous 12 months Interview of responsable staff to assess knowledge, understanding and practice Note that E1 is assessed at the Primary level only, and E9 was not assessed at the Service Point level in the original version of the questionnaire (version 1.0), but is in the new questionnaire (version 2.1) Indicators are evaluated in each of the 9 EVM criterias at each level. 400 indicators at PR level, 300 at SN, 200 at LD, 150 at SP Critical indicators are given a weight of 5, non-critical indicators are given a weight of 1 Each criterion at each level is scored out of 100%. Slide 5 Each criterion score is the weighted average of its constituent indicator scores Countries are assessed in 9 areas of vaccine management, the 9 EVM criteria: E1 Vaccine arrivals, E2 Temperature monitoring, E3 Storage and transport capacity, E4 Buildings and equipment, E5 Maintenance, E6 Stock management, E7 Vaccine distribution, E8 Vaccine management, E9 Information systems. Note that E1 is assessed at the Primary level only Criterion score data is available for 70 of the 75 countries that have conducted at least one assessment. There are 70 assessment scores in the PR, LD and SP distributions, and 53 in the SN distributions because 17 of the national supply chains assessed have only 3 levels. The box and whisker plot shows the median scores, the inter-quartile range (the range of scores of the central half) and the maximum and minimum scores. Consider the E1 PR box and whisker for example: the median score is about 70%, the central half of countries have a score between about 60% and 80%, about a quarter of countries have scores above the target score of 80%, a quarter have scores below 60%. At least one country scored 100% (maximum), and the lowest score was about 36% (minimum). 31
Slide Notes Slide 6 A country s EVM composite score is defined as the geometric mean of the country s 32 individual criterion scores (PR E1, PR E2,, SP E9). The EVM composite score is used to represent the overall strength of a country s immunization supply chain. In the understanding that an immunization supply chain is only as strong as its weakest link, the geometric mean is preferred to the arithmetic mean, as low criterion scores have a stronger effect on the geometric mean (weak links have a more significant effect on the strength of the overall chain). This map presents the overall distribution of countries' EVM composite scores by quartiles. The bottom 25% are the poorest performing relative to other countries who have conducted EVM assessments over the period 2010-2013, while the top 25% can be considered those countries who have the highest performing immunisation supply chain systems relative to other countries. If a country has completed more than one assessment, the score of the most recent assessment is used. 14 countries (Afghanistan, Armenia, Moldova, Rwanda, Albania, Burkina Faso, Malawi, Sri Lanka, Tanzania, Uzbekistan, Vietnam, Zimbabwe, Yemen, Sudan) have country composite scores above the target score of 80%. Slide 7 In summary: Slide 8 EMR and EUR perform significantly better on average than the other regions in the largely procedural areas of vaccine arrivals (E1), temperature monitoring (E2), stock management (E6), vaccine management (E8) and information systems (E9). AFR has the most room for improvement in the capacity, quality and condition of vaccine storage buildings and refrigeration equipment (E3 & E4). AFR = Africa Region, EMR = Easter Mediterranean Region, EUR = Europe Region, AMR = Americas Region, SEAR = South East Asia Region, WPR = Western Pacific Region. The assessment of vaccine arrivals procedures applies to the primary level only. 32
Slide Notes Slide 9 There are more stringent temperature monitoring requirements at the PR level than at the lower levels due to the higher volume of vaccine stored at that level. This partially explains the relatively low scores at the PR level compared to the other levels. Slide 15 The SP level has stricter requirements in vaccine management than the other levels, given that vaccines are administered at this level. Slide 16 7 of the 66 countries were assessed using the new questionnaire (version 2.1), in which E9 is assessed at SP level (3 in AFR, 3 in EMR and 1 in PAH). Slide 18 The data was collected between 2010 and 2013 in 58 countries in 6 WHO regions (31 AFR, 4 EMR, 10 EUR, 1 AMR, 7 SEAR, 5 WPR). Slides 19-28 The data was collected between 2010 and 2013 in 58 countries in 6 WHO regions (31 AFR, 4 EMR, 10 EUR, 1 AMR, 7 SEAR, 5 WPR). Results are based on 67 primary stores, 406 sub-national stores, 896 lowest distribution stores, and 949 immunization service facilities. Slide 19 The infrastructure context questions reveal that more than 10% of national stores assessed do not have cold or freezer rooms, that more than 10% do not store any vaccine at sub-zero temperatures, and that refrigerated vehicles are used in only a quarter of countries. 33
Slide Notes Slide 20 The policy context questions reveal that the multi dose vial policy has yet to be implemented in about 20% of health facilities assessed. MDVP = Multi-dose vial policy. Slide 21 The outsourcing context questions reveal that many countries outsource services such as customs clearance, vaccine transport and equipment maintenance. Slide 22 The availability indicators reveal that more than half of the stock record templates in use do not contain all of the recommended fields, that more than a third of facilities have inaccurate stock records, have mismatched vaccine and diluent stocks, and have been unable to meet demand on at least one occasion in the year preceding assessment Slide 23 The quality related indicators reveal poor temperature monitoring practice and equipment. Less than a fifth of countries have conducted a temperature monitoring study in the past 5 years, less than a fifth of cold rooms have been temperature mapped. Slide 24 Less than a half of all cold rooms and refrigerators have the recommended temperature monitoring equipment, and monitoring of exposure to freezing temperatures during transport in passive containers with ice-packs is almost non-existent VVM = Vaccine Vial Monitor 34
Slide Notes Slide 25 On the bright side, the quality related questions do reveal a high level (>80%) of health worker and store manager knowledge and understanding of vaccine temperature sensitivities and recommended vaccine storage temperatures, and use of VVMs. The shake test is however not well understood, especially by health workers at the service delivery level (40%). VVM = Vaccine Vial Monitor Slide 26 The efficiency related questions reveal that about 50% of the countries assessed do not have an up to date cold chain equipment inventory. In addition, the data reveal that about 25% and 50% of national and sub-national stores respectively still use paper based stock management systems. Slide 27 The efficiency related questions reveal that more than a third of vaccine storage facilities do not monitor vaccine wastage. VAR = Vaccine arrival Report Slide 28 The efficiency related questions reveal that less than 50% of national stores do not have a set of standard operating procedures, and that more than a third of health workers and store managers do not receive regular supportive supervision. MDVP = Multi-dose vial policy 35