USAID extends its thanks to CSIS and its survey partners, especially Afghan citizens, for undertaking and participating in this far reaching and comprehensive study.

• Strong Economic Progress Noted: Study conclusions suggest that Afghanistan has made considerable progress on the economic front since the last CSIS Study (2005) in terms of its gross domestic product (GDP) with sizable increases in foreign direct investment, trade, tax and customs collection, etc. Sustained, long- term economic viability is essential to Afghan economic and political development.
  
• Major Gains for Afghan Women: Other positive study findings indicate that the lives of women, the object of harsh Taliban attack, ridicule, and punitive restriction, have improved significantly. Since the last CSIS survey for example, women have become more optimistic about their government meeting their long-term needs. And, most Afghan women now report feeling safer at home than in 2005.
  
• Capacity Strengthening Underway: Still other positive findings suggest that the government's institutional and human capacity shows signs of improvement since 2005 and reconstruction projects (e.g. schools, roads, clinics) have helped but individual access, according to the Study, needs to be improved. Efforts to increase access are a top priority in USAIDs programming on the ground.
  
• Potential Setbacks in Some Areas: At the same time, the CSIS Report notes slippage from 2005 in several areas such as security, governance, and the provision of justice. While the Report's authors acknowledge the increasing complexity of challenges facing Afghanistan (e.g. security, drugs and corruption) they conclude that many of the trends are negative and pushing the country into what they call "the danger zone."
  
• Recommended Methodology Improvements: USAID applauds the wide range of methods and unique approaches used by CSIS to aggregate and analyze their data, display their findings, and reach conclusions. However, a number of methodological issues raise serious concerns about how to interpret the CSIS data. For example:
  
  
  
  
  • Sampling - While the researchers attempted to obtain an interview sample with a "good mix" of ethnic, geographic, and linguistic characteristics, it is not clear that the sample is scientifically representative and therefore, if any of the findings can be generalized beyond the interview sample.
    
  • Data Pooling - The Report notes that over 6,000 "data points" served as a basis for analysis, this pool was actually a m�lange of one-on-one qualitative interviews, selected public opinion surveys, proscribed media (newspapers, periodicals, newswires and websites), etc…. The authors no doubt were trying to be inclusive by tapping many sources but calling all these "data points" has been called into question.
    
  • Data Weighting - The Report assigned an "equal" weight to each so-called data point in an effort to be fair and to minimize the bias associated with any one source. One straightforward interpretation of that weighting scheme is that the impact of a website is equivalent to a one-on-one interview, or to a locally published article, etc. Under or over-weighting any one source could dramatically change the results -in either direction. Without a better weighting rationale, it is hard to see that "equivalence" is the best methodological option.
    
  • Qualitative Data - The Report uses a "multiple methods" approach by using a mix of quantitative and qualitative data meant to enrich the readers understanding. USAID is grateful to the researchers for this sometimes accepted approach. However, it is unclear how the qualitative data (derived mainly from in depth-interviews and written documents) was reviewed to minimize inherent interviewer biases nor how the qualitative data were systematically and rigorously "quantified" to produce valid and reliable results.
    
  • Graphic Display - The Report, in a genuine effort to simplify results and present "quick impact" or headline grabbing findings to policy makers, uses a unique graphic display featuring two "scales". One scale purports to depict longer-term effects (Afghan Capacity) while the other scale claims to reflect shorter-term effects (Need Fulfillment). Scores are then "plotted" against this grid as a heuristic device, with positive or negative study outcomes judged solely by how far and in what direction coordinates have "moved" from 2005 to 2006. Without knowing some of the very basic measurement properties and underlying assumptions of these scales, it is very difficult for anyone to accurately judge whether a data point has been reliably plotted to begin with, how much of a so-called "change" is within the normal range (e.g., noise), why the scales are plotted as "equal intervals", etc.
  
  
• Independent Study: USAID respects independent research and that is why the Agency funded this study. Further, USAID is not unmindful that CSIS and it partners went to great lengths to include multiple data sources and create an appealing graphic display for decision/policy makers. From the start, USAID has been clear that it values quantitative and qualitative data to provide the best, fullest picture possible.
  
• Alternative Analysis: Had CSIS presented the final data for each category (i.e., source) separately many of methodological problems cited above would have been obviated or at least minimized. CSIS needs to resolve these methodological issues so the public can receive a more valid and reliable assessment of change-both negative and positive-within Afghanistan. And, as a result, make appropriate evidence-based policy recommendations.