1: Short answers
· Identify and make explicit the criteria applied to support decisions
in economic evaluation made on others behalf, using measurement of
consequences in natural units, such as life years gained, biochemical
parameters, etc. The results are presented as incremental costs per unit of effect;
· In some settings, the natural units, such as life years gained, are
more favored instead of Quality Adjusted Life Years (QALYs) (insurance
companies in health systems with private funding, such as, for example).
· Only evaluates the costs related to a single and common effect of
interest to both alternatives;
· Tends to focus on the specific impact of the intervention as opposed
to the broader health of the patient;
· Difficult to assess the opportunity costs (i.e. benefits forgone) in
other programs covered by the same budget, due to the specific measures of
effect used for a given intervention.
· Identify and make explicit the criteria applied to support decisions
in economic evaluation made on others behalf, using measurement of
consequences in healthy years (mainly QALYs);
· Can evaluate single or multiple effects from interventions in the
· Allows decision-makers to compare benefits gained from introducing a
new intervention with those lost from any existing alternatives
(interventions) that could be displaced, through generic measure of benefits;
· Offers the potential to compare programs in different areas of health
care and to assess opportunity costs of adopting programs (their budget
· The estimation of preferences for health states is a particularly
useful technique because allows for health-related quality of life adjustments
to a given set of treatment outcomes, while simultaneously provides generic
outcome measures for the comparative assessment of costs and outcomes among
· The generic outcome measure for comparison is partially based on
individual preference. Health state value is preference based, but the
duration is not;
· Weights are often the average of population values (EQ5D, for example),
and not individual preferences;
· A QALY, for instance, is regarded as being of equal value,
irrespective of who receives it, but on a societal level QALYs may be valued
differently depending on the situation;
· QALYs do not encapsulate all the relevant attributes of health care;
· By giving a cost per generic outcome measure (cost per QALY, for
example) it’s not addressing other non-health benefits.
Visual analog scales (Condition specific descriptions)
· Easy to use;
· Acceptable response rates;
· Adjusted to the specific conditions of the disease under assessment;
· Allows to give health utility values to concrete stages of disease
· Gives a steady indication of the health outcomes prioritization and
some information on the intensity of those preferences.
· Measurement bias – subjects tend to avoid using the end of the scale;
· Context bias – subjects tend to space out the outcomes over the scale
regardless of how good or bad the states are;
· No notion of opportunity costs;
· Respondents may not be indicating strength of preference.
Multi-attribute health status (EQ-5D)
· Multiatribute health status classification allows to work on several
dimensions of quality of life;
· Defines 243 different health states;
· Continues to be used for reference case analysis;
· Generally used and widely accepted;
· Extensively tested and validated method;
· Simplicity, because it’s easy to complete by respondents, and its
available in several languages;
· Values reflect the general population’s health preferences.
· Country-specific value sets.
· Ceiling effect risk, particularly in general populations surveys but also
in some patient populations;
· Generic preference-based methods (such as EQ-5D) isn’t often included
in clinical trials of new therapies;
· Not disease-specific (for example, EQ-5D do not cover most of the
concerns for patients with mental health problems – mistakes with depression);
· In this case EQ-5D wouldn’t allow to attribute different health state
preferences for Lyme disease stages (Erythema migrans, Arthritic sequelae, Cardiac
sequelae, and Neurologic sequelae) (Shadick, 2001);
· Suboptimal sensitivity to change or difference;
· Not all important health aspects covered.
The uncertainty of interventions in a
cost-effectiveness and cost-utility analysis can be assessed through parameter
and/or structural analysis. In the parameter analysis, sensitivity could be
assessed through deterministic (one-way or multiway sensitivity analysis)
and/or probabilistic analysis (Drummond, 2015).
Shadick et al performed a parameter analysis (Shadick, 2001). Particularly, a deterministic
sensitivity analysis, using both one-way and two-way sensitivity analysis:
one-way sensitivity analysis were performed for most of the parameters:
clinical probabilities (seasonal infection rate), treatment efficacy (between
20% and 100%), vaccine efficacy (between 0.43-0.74 for partially compliant
persons and 0.67-0.86 for fully compliant persons), and cost estimates (between
50$ to 300$ of vaccination costs) (Shadick, 2001).
Although one-way sensitivity analysis indicates how
sensitive the model outputs might be to changes in particular inputs, it cannot
indicate how uncertain a decision might be. Nor can indicate which parameters
contribute most to this decision uncertainty, so this is not exclusively recommended
to represent uncertainty. The one-way analysis can’t provide the combined
effect of uncertainty in the value of all parameters, and all the parameters can
be simultaneously uncertain. It underestimates the uncertainty surrounding the
decision and only provides a qualitative conclusion.
two-way sensitivity analysis was also performed by Shadick et al for the Lyme
disease attack rate and the amount of time for which the vaccine is effective
(since the persistence of the vaccination effect has not been demonstrated).
The acceptability curves for the two-way sensitivity analysis were presented in
figure 4 of the article (Shadick, 2001).
If the range is arbitrary chosen or based on some implicit
assumptions, the results may be misleading. Upper and lower bounds can be
justified based on the evidence used to estimate the parameter and the
distribution that might be assigned to represent the uncertainty associated
with its estimation. Shadick et al reported valid assumptions for range values,
based on estimates from the literature (Shadick, 2001). Nevertheless, the references were not
The threshold values that parameters would need to
take for cost and effects to change sufficiently to alter the decision about
which alternative offered the highest expected net benefits.
else could be done?
vaccination efficacy has some uncertainty (there is evidence reporting risk
reduction of Lyme Disease due to vaccination between 79% to 92%), a multiway
sensitivity analysis considering: i) vaccination efficacy range; ii) the
disease attack rate; and iii) the time for which the vaccine is effective,
might give better cost-effectiveness acceptability curves (or frontiers) to aid
probability analysis that parameters would take values more extreme than theirs
threshold values based on how they were estimated, reflecting the amount and
quality of existing evidence, was not performed. Shadick et al stated a
“clinically plausible range” with estimates obtained from the literature for
sensitivity analysis, without detailing the source of information;
overcome this, a Probabilistic Sensitivity Analysis (PSA) could be performed,
assigning distributions to parameters, providing better estimates on
uncertainty and giving more accurate estimates of expected costs, effects and
providing the probability that each alternative might be cost-effective. It
allows quantitative analysis.
might be represented as distributions of possible mean values. PSA forces the
analyst to be more explicit;
most of the data comes from trials and surveys (patient-level data), the
analyst can use bootstrapping as a non-parametric alternative for describing
the distribution of possible mean values;
simulated set of costs and effects for each of the alternatives into expected
net benefit, using a cost-effectiveness threshold, might overcome difficulties.
Otherwise it’s is not possible to know which alternative gives the highest
expected net benefit;
there is no data of vaccination efficacy beyond two years of vaccination (Shadick, 2001), the structural
uncertainty should be parametrized. A way to perform this could be by assigning
a parameter describing the proportion of treatment effect between time 0 and
the end of the trial. This would make possible to perform a distribution and
represent uncertainty in a PSA analysis (Drummond, 2015). Shadick et al assumed an additional
year of vaccination efficacy as an assumption without any reference that
supports this assumption (Shadick, 2001);
et al assumed that patients who do not present with or do not receive treatment
for erythema migrants or in whom treatment for early Lyme disease has failed,
develop disseminated Lyme disease (Shadick, 2001). Despite the fact that there is a reference
for this assumption in the article, a sensitivity analysis for this should be
and effects were not evaluated within the same group, which might lack some
Brief description of
methods used by Shadick et al
Comment on the extent
to how methods meet guidelines
A societal perspective was adopted, following reference-case
recommendations which describes a base-case as reference for other
comparisons and incorporates QALYs (cost-effectiveness ratios) – important
for public health and resource allocation.
Nevertheless, other less frequent long-term sequelae, the resulting
loss of work from disability, and the pain incurred from the illness were not
Shadick et al article is not indicated to support reimbursement in a
particular health system.
The societal perspective, as recommended, was followed.
A separate payer perspective, to deliver therapeutic benefits and
related costs by the health system through a willingness to pay, was not
The payers’ perspective, recommended as a primary analysis, was not
performed. The authors conducted an optional perspective from the guidelines
10 years (including a 2-year vaccination schedule with an additional
year of vaccine effectiveness).
The time horizon chosen captures all relevant differences in costs and
in effects of health treatments and resources.
There is no clear statement regarding the 10 year chosen for time
horizon. AMCP suggests an ideal time horizon between 1 to 5 years, and there
is no plausible reason to extend the time horizon for more than 5 years.
Nevertheless, the model is appropriate to reflect all important
differences in costs and outcomes, and applies a discount rate for both costs
and life years, as recommended. Since there is no payers’ perspective
(recommended in guidelines), the time horizon doesn’t consider financial and
Preferred analytical technique
Decision-analytic model to evaluate the cost-effectiveness and
cost-utility of vaccination compared with no vaccination, in terms of cost
per case of Lyme disease adverted and cost per QALY gained, respectively.
The techniques – cost-effectiveness and cost-utility analysis – are
recommended in the guidelines and suitable methods to address the decision
problem. There is a clinically significant effect and an improvement in
health related quality of life as a result of vaccination.
AMCP guidelines consider cost-utility as a “subtype” of
cost-effectiveness analysis. In this particular case, the model complies with
the requirements because it is disease-based, considers related direct costs
(including adverse events), outcomes related to vaccination, and incremental
cost and outcomes analysis are presented as cost- effectiveness ratios.
Nevertheless, it lacks a budget impact analysis. As already stated,
It’s not focused on the payers’ perspective, on the opposite of guidelines
Preferred method to derive utility
Visual analogue scale with rating scores (0 to 1) for each
hypothetical clinical scenario, converted to utilities using a power
The health-related quality of life weights were estimated from a
random sample of 105 residents from an area with one of the highest
incidences of Lyme disease in the USA.
The estimated utilities were multiplied by the expected duration in
each health state to estimate QALYs over the 10-year time horizon period and
produce the ICERs.
In a cost-utility analysis, the recommended unit of measurement is
QALYs, which were used in this analysis. The methods to estimate QALYs are
also clearly described.
Utilities are based in preference estimates derived from surveying
patients, using a direct elicitation method.
The analytic framework from the AMCP Guidelines recommends QALYs as a
valid outcome for economic evaluation.
One-way and two-way deterministic sensitivity analysis were performed.
Several one-way sensitivity analysis were performed for the
parameters: clinical probabilities, treatment efficacy, vaccine efficacy, and
A two-way sensitivity analysis was performed for the Lyme disease
attack rate and the amount of time for which the vaccine is effective. The
acceptability curves for this analysis were presented as figure 4.
Probabilistic sensitivity analysis were not performed.
One-way sensitivity analysis was performed for all the relevant
parameters for uncertainty accountability.
A two-way sensitivity analysis was performed to generate acceptability
curves and better aid decision by varying two main parameters.
The main parameters subjected to uncertainty were identified and a
sensitivity analysis (deterministic) was performed.
Nevertheless, PSA was not performed, neither scatter plots for
assessing decision uncertainty. Only acceptability curve for a 2-way
sensitivity analysis was presented.
The lower and upper levels were, in general, not arbitrary, which is
positive. Nevertheless, there’s only a reference to “clinically plausible”
ranges, which is not enough to justify the options.
Deterministic (univariate) sensitivity analysis, as recommended, was
conducted. One-way sensitivity analysis was used for all the relevant
parameters in the model, with assessment of QALYs and ICERs, which complies
with the guidelines priority recommendations.
The evidence is a crucial feature for any model-based
economic evaluation. The guidelines for economic evaluation should address, at
least, three main dimensions in order to ensure quality evidence for decision
making (Drummond, 2015):
nature of the populations (and subpopulations) being considered in the
where the resource allocation decision is to be made, types of medical practice
and the nature of the health care system.
In these three dimensions should be ensured that
relevant evidence is identified in an unbiased way and that decisions are
justified by the evidence produced (Drummond, 2015). In particular, there are some
principles that can be compared between Spanish guidelines and the AMCP
Comparison between Spanish
Guidelines and AMCP guidelines
2010) (Committee, 2016)
The uncertainty analysis should be assessed very
clearly and evidence should not be selected to suit a particular point of
Both requires formal assessments of uncertainty and
the impact of uncertainty in results. Also, both recommends confidence levels
(probabilistic analysis) when suitable.
AMCP goes further and requires cost-effectiveness
scatter plots and acceptability curves to better present options and aid
If possible, a hierarchy of evidence should be
recommended and the measure of relative effectiveness would ideally be taken
from a randomized trial (well designed and implemented), due to the risk of
selection bias in an observational study and the process of identifying
suitable resource use estimates for a decision model may also benefit from
nationally available data.
AMCP prioritizes head-to-head clinical studies
between alternatives and recommends summaries and evidence tables for
critical evidence from key studies.
Spanish guidelines recommend efficacy and
effectiveness studies without prioritizing. There is no priority suggestion
in collecting evidence. There’s a reference for greater internal validity of
Synthesizing evidence should also be a concern, due
to heterogeneity (appropriate synthesis of estimates from different sources
reflecting heterogeneity as far as possible).
AMCP gives high importance to comparative evidence
and recommends the inclusion of indirect treatment comparison and
meta-analysis studies. It requires a clear understating heterogeneity of
Spanish guidelines requires description of methods
used to combine efficacy data and quantitative estimates of overall effect of
the intervention under analysis.
Generalizability of treatment effects estimates from
RCTs should be addressed with caution, given that patients are usually
selected from a wider population and the observed and/or unobserved
characteristics of that sample may be different from those of the wider
Transparency is also a very important factor.
AMCP guidelines require transparency in the
modelling report, presenting evidence, and presenting results.
Spanish guidelines require transparency in overall
terms and specifically in combining data, model selection for the analysis
and the key parameters, statistical distribution of the variables analyzed,
methods data and results of the report, presenting data and assumptions, and
when presenting strengths and weakness aspects of the results.
A ‘reference case’, as defined by a set of methods,
should be followed to provide evidence for submission, whenever possible.
Both guidelines recognize the reference case as an
important asset for economic evaluation.
These principles are compliant with the checklist for
assessing quality in decision-analytic models (Philips, 2004), and represent the
considerations that should be addressed to provide quality evidence.
Both guidelines address the main dimensions for
generating quality evidence. Nevertheless, AMCP guidelines recommend to follow
ISPOR-SMDM (ISPOR and Society for Medical Decision Making (SMDM) Modeling Good
Research Practices Task Force) best practices and, in addition, CHEERS
guidance, which are international consensus on economic evaluation, which makes
them more widely accepted, following fundamental principles for each dimension (Committee, 2016).
Also, AMCP favors the payers perspective, which indicates
a requirement for more ‘tailor made’ evidence for the provision of
cost-effectiveness analysis. The recommendation of a budget impact analysis,
following ISPOR recommendations, is also a factor for more detailed and quality
evidence, since it requires estimates specifically for the target population, intervention
costs in the setting under assessment, health care cost offsets, and adverse
event costs, as well as the expected utilization in the health care system, to
derive projected per member per month costs (Committee, 2016).
Spanish guidelines don’t include consensual techniques
for assessment (e.g. ISPOR recommendations), are less detailed and lacks
recommendations in specific aspects, such as preference methods of modelling,
collecting evidence, promoting transparency, and sensitivity analysis
presentation (López-Bastida, 2010).
All thing considered, the AMCP guidelines would give
better quality evidence.