Forsdyke's 1992 HIV Funding Scheme
bgold at itsa.ucsf.edu
Tue Jun 18 16:14:29 EST 1996
Accountability in Research (1992) 3:1-5
BICAMERAL GRANT REVIEW: HOW A SYSTEMS ANALYST WITH AIDS WOULD
REFORM RESEARCH FUNDING
D.R. FORSDYKE, M.B., Ph.D.
Department of Biochemistry, Queen's University,
Kingston, Ontario, Canada K7L 3N6
A systems analyst (SA) with AIDS has applied his professional
skills to determine whether available research funds are being
spent optimally. After an initial briefing by the director (D) of
a major funding organization and visits to various research
laboratories, he now returns to suggest to the director a novel
"bicameral" method of reviewing research proposals. The
"retrospective" and "prospective" parts of research proposals
should be separated and independently routed. Peer-review should
be entirely retrospective and concerned with past performance
relative to funds received. Prospective review, concerned solely
with budget, should be performed in house by the funding bodies.
The director is not entirely in agreement.
SA-Thank you for agreeing to see me again. When I first learned
that I was seropositive for the AIDS virus, I was overwhelmed by a
feeling of utter helplessness. However, I've worked for many years
as a systems analyst advising organizations how to make their
operations more efficient. It occurred to me that I might apply my
professional skills to medical research organizations, such as that
which you direct.
D-When we last met I told you how the research funding system works
(Forsdyke, 1989a). You thought that competition between
researchers for research funds might be delaying progress by
impeding collaboration. Have your studies led you to modify that
SA-On the contrary, I agree with Lewis Thomas that the degree of
competition may now be counterproductive (Angier, 1988). The
system as we know it today was established in the late 1940s. By
all accounts, it worked very well as long as sufficient funds
chased the available talent (Apirion, 1979; Mandel and Vesell,
1989). Then in the late 60s financial cutbacks began to reveal
serious structural problems which were not apparent at the outset
(Osmond, 1983; Forsdyke, 1983a,b; 1989a,b,c; Lederberg, 1989).
D-I'd like to hear what problems you have identified. But first
let's make sure we agree on basics. What do you understand to be
the mission of the organization which I direct?
SA-Your mission is to advance medical knowledge. This will result
in better methods of preventing, diagnosing and treating diseases
such as AIDS. To this end you have a system for allocating funds
to medical researchers.
D-It's not that simple. We have to have the funds before we can
allocate them. How do we persuade the government to put funds into
medical research rather than into other areas? How do we persuade
individuals making charitable donations to choose medical charities
rather than other charities? As part of our mission, we have to be
concerned with public relations. This can affect our allocation of
funds to researchers. For example, the recent discovery of a gene
defective in cystic fibrosis patients represents an immense advance
(Koshland, 1989). Many of us foresaw this decades ago and wanted
to put more funds into basic research in molecular biology.
Instead we had to pour funds into various "quick fix" approaches to
satisfy the cystic fibrosis lobby.
SA-I accept that, to keep up the global level of funding, some
funds have to be allocated in that way.
D-It is also very important for fund-raising that our system for
allocating funds not only be sound, but be perceived as sound.
SA-It is that thought which may be muffling a lot of the dissent I
hear in the laboratories which I have visited. It is apparent that
the current peer review system is overburdened and is working very
inefficiently. Yet researchers are reluctant to express their
discontent publicly. They may even fear retaliation from the
funding bodies if they were to do so.
D-Do you have any reforms in mind?
SA-We expect too much from the peer review process. First, we
expect it to provide ratings on the qualities of the applicants.
Second, we expect it to provide information on whether the proposed
budgets are realistic. Third, we expect it to provide feed-back to
the applicants so that they can improve the research proposed.
Only the first two of these are really essential. There are
numerous ways in which researchers can and do get constructive
criticism of their ideas for future research. Provision of such
criticism by the funding bodies is redundant. Abandonment of this
would allow a major restructuring of the peer review process
D-A frequent complaint from researchers is that we do not provide
sufficient feedback so that unsuccessful applicants can improve
their next proposals. Now you tell me this should be abandoned
SA-I would reform the peer review process by separating grant
applications into two distinct parts, a "retrospective" part and a
"prospective" part. These would be routed separately. The
retrospective part would describe what had been achieved with the
available funds. This part alone would be sent out for peer
review. The reviewers would evaluate performance in terms of the
funds received. This would be difficult, but it would be more
objective and less error-prone that the "prospective" evaluation of
an applicant's ideas for future research (Forsdyke, 1983b, 1989a).
D-Would you allow on-site visits to ensure that the research
results reported had actually been obtained?
SA-Yes, there would be some random auditing both of results and
expenditure. Knowledge of this possibility should ensure accurate
reporting. Positive research results would score highly, but
discriminating reviewers would also be looking at the logic of the
overall approach and how the researcher had marshalled the
available resources. A big problem, requiring a long-term
approach, might produce no publishable results within a given
funding period, yet might still score highly. An important part of
this review procedure would be that reviewers would be evaluating
the ratio of performance to funds received. There would be an
incentive to be economical. Indeed, only the funds actually
expended would be taken into account. These might be less than the
funds awarded at the beginning of the funding period.
D-How would you deal with people who were just entering the system
and did not have a research track record?
SA-Most future independent researchers will have gained some sort
of track record during their apprenticeships. Their initial
funding would be modest. Within a few years they would have an
independent track record which could be evaluated.
D-Would it be good politics for the funding bodies to ask
politicians and private donors to support research ideas which had
not been independently evaluated?
SA-No. But from the point of view of the politics of fund raising,
the emphasis of the retrospective part of the grant application on
accountability for past performance should be a plus. There is a
middle ground between giving a researcher carte blanche on what
research is done and scrupulously evaluating the cogency of that
research. Here we come to the "prospective" part of the grant
application. This would be routed to a new class of specialist
financial officers within the funding organizations. These
individuals would have professional expertise in evaluating
research budgets. This "in house" part of the application would
contain sufficient information on the proposed research to allow a
financial officer to determine if the budget was realistic. Thus
the granting body would know the research plan, even though it
would not be evaluating that plan directly. Obviously, if some
quite bizarre line of research was proposed, there would be the
option of a veto.
D-Would the granting body require that the previous research plan
be included in the retrospective part of the applicant's next grant
application? Would the applicant be criticized if the results
achieved did not match the plan?
SA-No. Peers would be concerned with evaluating the quality
(relatively objective) and the value (more subjective) of the
results. It would probably be prudent for the applicant to
describe the path, serendipitous or otherwise, which had led to
those results. But that would be for the applicant to decide,
given the need for conciseness.
D-How does all this relate to the sliding scale of fund allocation
which you suggested when we last met (Forsdyke, 1989)? You
proposed that first a decision would be made as to how many
projects were of sufficient merit to justify support. Then a
sliding scale would be applied to the approved projects. Only
those at the very top of the funding scale would get all the funds
they needed to complete the work in a reasonable time. Those just
below the top would get 90% of what they needed, and so on down to
the approved projects of lowest merit which might receive only 10%.
SA-When the retrospective review by peers of past performance was
completed, a rating would be available, just as under the present
system. When the prospective in-house review of the proposed
budget was completed, a budget figure would be available. After
this "bicameral" review, all that would remain would be to rank the
applicants by rating, decide on a rating cut-off point and then
allocate funds on the sliding scale to those above that point.
Obviously, the rate of progress and scope of a project which only
received 10% funding would be severely compromised. But 10% funding
is enormously different from 0% funding. With imagination and
fortitude I believe many projects could limp along, even with 10%
D-What sort of feedback would an applicant get?
SA-From the peer review he or she might get a critique of past
strategy. For example, a researcher might be criticized for not
having used a method capable of giving more definitive results or
for not adequately justifying the introduction of an expensive new
procedure. This might have been circumvented by collaboration with
a neighbouring laboratory. The researcher's interpretation or
evaluation of the results might be challenged by the reviewers.
>From the prospective budget review, a researcher might learn of
less expensive ways of carrying out the research.
D-It would be expensive to recruit and train more specialist
financial officers. Yet such people could play an important role
in keeping down the prices of equipment and supplies. The medico-
industrial establishment would not like it! But do you think that
the financial officers would be able to detect those applications
inwhich the budgets had been inflated in anticipation of receiving,
through the sliding scale, less than the optimum budget?
SA-I think the task of financial officers would, in many respects,
be far easier than the task of peer reviewers. The officers, after
all, would be dealing with numbers. If there was any doubt they
could demand to see past accounting records, seek justification for
past expenditures and ask for a better justification of future
expenditures. It could be difficult to pull the wool over their
D-You have more faith in the skill of your proposed budgetary gate-
keepers than I have. What would you do, for example, in the case of
a researcher who, knowing he had achieved a great deal in the
previous granting cycle on, say, a budget of $100,000, decides to
propose a new, well justified project, which would cost $1,000,000?
The peer review process you propose would give him a very high
rating for his productivity relative to dollars received, so that
he could expect to receive funding close to the 100% level.
SA-Two answers. First, the researcher would know that, down-the-
line, he or she would have to justify the $1,000,000 expenditure in
terms of results received. This would act as a restraining force
providing pressure towards realistic budget-making. Second, I do
not think we should impose the bicameral review system rigidly, to
the exclusion of other approaches. Bicameral review should be
applicable to most on-going projects which have a relatively stable
level of expenditure...perhaps 90% of the total. Those researchers
who propose a substantial departure from previous expenditures
could submit their applications for conventional peer review as now
D-The present system may have its faults, but at least it is
relatively simple and well understood. You are proposing a far more
complicated two level review process, bicameral review, and, on top
of that, you now say that we will still maintain conventional peer
review for special cases. Running such a system would be a
SA-That is exactly how I have heard researchers describe the
present system, a bureaucratic nightmare. Surely we can do better?
Let us at least give bicameral review a try. It is remarkable that
the funding bodies, dedicated to the pursuit of truth through
experimentation, have themselves for so long neglected to
experiment with different mechanisms of fund allocation. The
optimum harnessing of the expertise, energy and enthusiasm of the
nation's biomedical work force is critical for the conquest of AIDS
and of the many other diseases that inflict humankind. My
proposals for reform of the funding system could result not only in
a better distribution of research funds, but could also influence
in a positive manner the conduct of those engaged in research
(Forsdyke, 1983a,b; 1989a). Let us experiment!
Angier, N. (1988) Natural Obsessions: The Search for the Oncogene.
Boston: Houghton-Mifflin. pp. 1-4.
Apirion, D. (1979) Research funding and the peer review system.
Fed. Proc. 38, 2649-50.
Forsdyke, D.R. (1983a) Canadian medical research strategy for the
eighties. I. Damage-limitation or superelitism as the basis for
the distribution of research funds. Medical Hypothesis 11, 141-
Forsdyke, D.R. (1983b) Canadian medical research strategy for the
eighties. II. Promise or performance as the basis for the
distribution of research funds. Medical Hypothesis 11, 147-156.
Forsdyke, D.R. (1989a) A systems analyst asks about AIDS research
funding. Lancet 2, 1382-84.
Forsdyke, D.R. (1989b) Peer review policy. The Scientist 3, 16:13.
Forsdyke, D.R. (1989c) Sudden-death funding system. FASEBJ. 3,
Forsdyke, D.R. (1991) Bicameral grant review: an alternative to
conventional peer review. FASEBJ. 5, 2312-2314.
Koshland, D.E. (1989) The cystic fibrosis gene story. Science 245,
Lederberg, J. (1989) Does scientific progress come from projects,
or people? Current Contents, Life Sciences 32, 48:5-12.
Mandel, H.G. and Vesell, E.S. (1989) NIH funding. FASEBJ. 3, 2322-
Osmond, D. (1983) Malice's wonderland. Research funding and peer
review. J. Neurobiol 14, 95-112.
More information about the Bioforum