There are two points worth noting about the IPCC WG2 AR2 lead authors meeting in San Francisco.
First, a conflict of interest is now in place. Essentially, we all signed a form declaring that we have no conflicts of interest. This is defined narrowly: pecuniary, personal, direct benefit from deliberate bias. There is no audit of these declarations, and they will not be made public. The IPCC pretends that its authors operate in their personal capacity, even if people work on their chapters in their bosses' time. Conflicts of interest that arise because IPCC authors are also journal editors, PhD advisors, researchers, fund raisers, referees, and what nots are deemed irrelevant.
Energy Economics, an academic journal, has decided that the IPCC poses a conflict of interest and a potential risk for the reputation of the journal. The journal editors will refrain from handling papers that are relevant to the IPCC chapters they are involved with as authors.
Second, the IPCC member states have ruled on freedom of information legislation. Specifically, it has been decided that FoI does not apply to IPCC material. This is false. FoI is national legislation. These laws can only be interpreted by the relevant courts. These laws can only be changed by the relevant parliaments. The civil servants that speak on behalf of their countries have no right to usurp FoI legislation, and the IPCC has no say in this matter.
Showing posts with label procedures. Show all posts
Showing posts with label procedures. Show all posts
Tuesday, December 13, 2011
Tuesday, January 11, 2011
First lead authors meeting, Tsukuba, January 2011
The first lead authors meeting is this week in Tsukuba (near Tokyo). The aim is to agree on an outline and allocate tasks.
The starting point is an outline that was written by a jet-lagged, non-expert committee. It is full of holes and overlaps. It is hard to set this straight, as most everyone here is jetlagged too -- and there are 30 chapters and therefore 60+ CLAs. Chapter 10 needs to be coordinated with no less than 11 other chapters.
Two things were discussed that will come to haunt us.
First, everything we write in an IPCC context is potentially subject to a Freedom of Information request -- either under national or EU law. A substantial share of authors is not used to working under those circumstances.
Second, a decision was made to disclose potential conflicts of interest -- but details are given verbally, no preparation time was given, and statements are not checked. Only a summary will be committed to paper.
The starting point is an outline that was written by a jet-lagged, non-expert committee. It is full of holes and overlaps. It is hard to set this straight, as most everyone here is jetlagged too -- and there are 30 chapters and therefore 60+ CLAs. Chapter 10 needs to be coordinated with no less than 11 other chapters.
Two things were discussed that will come to haunt us.
First, everything we write in an IPCC context is potentially subject to a Freedom of Information request -- either under national or EU law. A substantial share of authors is not used to working under those circumstances.
Second, a decision was made to disclose potential conflicts of interest -- but details are given verbally, no preparation time was given, and statements are not checked. Only a summary will be committed to paper.
Wednesday, September 1, 2010
Regulating Knowledge Monopolies: The Case of the IPCC
The Intergovernmental Panel on Climate Change has a monopoly on the provision of climate policy advice at the international level and a strong market position in national policy advice. This may have been the intention of the founders of the IPCC. I argue that the IPCC has a natural monopoly, as a new entrant would have to invest time and effort over a longer period to perhaps match the reputation, trust, goodwill, and network of the IPCC. The IPCC is a not-for-profit organization, and it is run by nominal volunteers; it therefore cannot engage in the price-gouging that is typical of monopolies. However, the IPCC has certainly taken up tasks outside its mandate; the IPCC has been accused of haughtiness; innovation is slow; quality may have declined; and the IPCC may have used its power to hinder competitors. There are all things that monopolies tend to do, against the public interest. The IPCC would perform better if it were regulated by an independent body which audits the IPCC procedures and assesses its performance; if outside organizations would be allowed to bid for the production of reports and the provision of services under the IPCC brand; and if policy makers would encourage potential competitors to the IPCC.
Full paper
Full paper
Wednesday, June 23, 2010
Selection of authors
The authors of IPCC WG2 AR5 have now been announced.
The selection process is something as follows.
As a first step, the IPCC member states nominate people, and indicate their role (convening lead author, lead author, review editor). Every member state does this differently. I know of one case in which somebody was nominated by his government, without him knowing. I know of one case in which someone was nominated by a junior civil servant, against the will of more senior civil servants. I know of one case in which someone agreed to be be nominated, but declined the invitation by the IPCC.
As a second step, the IPCC Technical Support Unit (TSU) and WG Bureau select the Convening Lead Authors (CLA). For some chapters, there is a choice, but for other chapters there is one candidate only -- or none.
As a third step, the CLAs select a team of Lead Authors from the nominations (having access to summary and full CVs), and submit the list to the TSU.
As a fourth step, the TSU and WG Bureau reconcile the conflicts between the lists of authors. Strange things happen in this phase. Apparently, a number of people were nominated for multiple roles -- and the good ones were selected for multiple roles. This makes a mockery of the notions that the IPCC has a clear outline and that IPCC authors are experts in their fields. Other candidates were excluded because there were too many people from country X -- even if country X has a long tradition in climate research and therefore a large pool of experts, and even if there is no expert from another country as a substitute. Finally, some people were selected as a lead author without being nominated for that chapter.
As a fifth step, TSU and CLA agree on the list, adding some people and removing others.
Because the nomination and selection process is about people, it cannot be fully open and transparent. I think that the current process can be improved, however. Particularly, it strikes me that there should be a strict limit on the number of chapters for which someone can be nominated. The structure of the reports should be much more crisp, so that people would clearly recognize where they can best contribute. While there should be a soft limit on the number of participants from a particular country, it cannot be that there are gaps of expertise because of such constraints.
The selection process is something as follows.
As a first step, the IPCC member states nominate people, and indicate their role (convening lead author, lead author, review editor). Every member state does this differently. I know of one case in which somebody was nominated by his government, without him knowing. I know of one case in which someone was nominated by a junior civil servant, against the will of more senior civil servants. I know of one case in which someone agreed to be be nominated, but declined the invitation by the IPCC.
As a second step, the IPCC Technical Support Unit (TSU) and WG Bureau select the Convening Lead Authors (CLA). For some chapters, there is a choice, but for other chapters there is one candidate only -- or none.
As a third step, the CLAs select a team of Lead Authors from the nominations (having access to summary and full CVs), and submit the list to the TSU.
As a fourth step, the TSU and WG Bureau reconcile the conflicts between the lists of authors. Strange things happen in this phase. Apparently, a number of people were nominated for multiple roles -- and the good ones were selected for multiple roles. This makes a mockery of the notions that the IPCC has a clear outline and that IPCC authors are experts in their fields. Other candidates were excluded because there were too many people from country X -- even if country X has a long tradition in climate research and therefore a large pool of experts, and even if there is no expert from another country as a substitute. Finally, some people were selected as a lead author without being nominated for that chapter.
As a fifth step, TSU and CLA agree on the list, adding some people and removing others.
Because the nomination and selection process is about people, it cannot be fully open and transparent. I think that the current process can be improved, however. Particularly, it strikes me that there should be a strict limit on the number of chapters for which someone can be nominated. The structure of the reports should be much more crisp, so that people would clearly recognize where they can best contribute. While there should be a soft limit on the number of participants from a particular country, it cannot be that there are gaps of expertise because of such constraints.
Friday, May 28, 2010
Submission to the IAC (second draft)
I was a lead author of two chapters in the AR2 WG3 report (1995), a convening lead author in the WG2 Special Report on Regional Impacts (1998), a contributing author of one chapter in the AR3 WG1 report (2001), a lead author of one chapter in the AR3 WG2 report (2001), and a contributing author of one chapter in the AR4 WG2 report (2007). I am designated to be a convening lead author for AR5 WG2 (2014).
The IPCC is a victim of its own success. Policy makers trust the IPCC reports as neutral and authoritative assessments of climate research. Therefore, people with a political agenda have tried to influence the IPCC. Such attempts were largely in vain in AR2 and AR3, but this is not true for AR4. Working Group 2 systematically portrays climate change as a bigger problem than is scientifically acceptable. (Examples include the date of disappearance for glaciers in the Himalayas, the mix-up on weather and climate for agriculture in Africa, and the projected number of people at risk from water stress.) Working Group 3 systematically portrays climate policy as easier and cheaper than can be responsibly concluded from academic research. (Examples include the attribution of market-driven and welfare-improving improvements in energy efficiency to climate policy, the omission of the opportunity costs of energy research and development, and the use of gross (rather than net) estimates of job creation.) These biases can be found in the chapters, the technical summaries, the summaries for policy makers, and the synthesis report.
The most important problem of the IPCC is the nomination and selection of authors and Bureau Members. Some experts are included or excluded because of their political allegiance rather than their academic quality. Sometimes, the “right” authors are put in key positions with generous government grants to support their IPCC work, while the “wrong” authors are sidelined to draft irrelevant chapters and sections without any support.
AR5 should be put on hold until the IPCC can prove that its author teams indeed have the relevant expertise, and its authors are at least as qualified as their peers. If needed, the IPCC should request additional nominations.
The IPCC member states are represented by their environment departments. For AR6, this responsibility should be transferred to their research departments or their academies. The selection process for authors and Bureau Members should be made transparent. The IPCC should commission a scientometric assessment of its Bureau Members, authors, and chapters.
IPCC assessment reports do not cover the entire literature. For instance, the IPCC does not fully assess the literature on trade-offs between the various greenhouse gases and the literature on international environmental treaties. Both areas are actively researched and conclusions are typically critical of current policy, but were largely omitted from previous IPCC reports. The literature on comparing the impacts of climate change to the costs of emission reduction falls outside WG2 and WG3, and thus outside the Synthesis Report as well. There is a risk that the nascent literature on ex-post evaluations of climate policy (in Europe and Japan) will be ignored in AR5, and particularly those papers that show that climate policy is ineffective and needlessly expensive.
The IPCC should submit the proposed outline for AR5 for peer review, and ensure that it covers all aspects of climate change and climate policy, including those parts of the academic literature that are potentially embarrassing to policy makers.
The IPCC Plenary has delegated its supervisory powers to the IPCC Bureau, which is also its executive. This implies that there is no mechanism to correct the IPCC Chair (if, for example, he oversteps his mandate and gives policy advice on behalf of the IPCC, if he says embarrassing things to journalists, or if he uses the IPCC to raise funds for his home institution). There is no mechanism to reconcile differences between IPCC Working Groups on the treatment of interdisciplinary topics. (The main example is the IPCC treatment of global warming potentials as a pure physical issue.) Therefore, the Chairs of the IPCC and its Working Groups should leave the IPCC Bureau, and the Bureau should adopt a supervisory role under a strong and independent chairperson.
In contrast to the procedures for governing the IPCC and selecting its authors and officials, the procedures for drafting IPCC reports are acceptable on paper but not in practice. Particularly, these procedures were not enforced in AR4 WG2 and WG3. (Some chapters rely heavily on gray literature while ignoring peer-reviewed literature on the same matter (e.g., Ch 7 WG2). Other chapters cite papers published after the deadline (e.g., Ch 15 WG2). Incomplete drafts were sent for peer-review (e.g., Ch 11 WG3). Substantial material was added after the final review (e.g., Ch 20 WG2).) In order to avoid repetition, Review Editors should be more independent, and return chapters for further revision if needed. The Review Editors rather than the Convening Lead Authors should have the final say about chapters. Review Editors should actively seek the input of referees. Review Editors should have the right to reject sections of the report that do not meet basic quality standards at the deadline. In short, Review Editors should have the same powers as journal editors. Review Editors should be selected from the ranks of (former) journal editors.
The IPCC should engage the wider public in the review process. After the publication of AR4, the “blogosphere” uncovered a large number of procedural and substantial errors. It would be preferable to discover this before publication. Engaging with “the public” is difficult. Therefore, the IPCC should do this in an experimental fashion in AR5 so as to prepare guidelines for AR6. Experiments could include posting draft chapters on blogs for public comment (A draft of the current statement was discussed at http://rogerpielkejr.blogspot.com/2010/05/richard-tols-draft-submission-to-iac.html), and posting draft chapters on wikis for public amendment.
The IPCC is a victim of its own success. Policy makers trust the IPCC reports as neutral and authoritative assessments of climate research. Therefore, people with a political agenda have tried to influence the IPCC. Such attempts were largely in vain in AR2 and AR3, but this is not true for AR4. Working Group 2 systematically portrays climate change as a bigger problem than is scientifically acceptable. (Examples include the date of disappearance for glaciers in the Himalayas, the mix-up on weather and climate for agriculture in Africa, and the projected number of people at risk from water stress.) Working Group 3 systematically portrays climate policy as easier and cheaper than can be responsibly concluded from academic research. (Examples include the attribution of market-driven and welfare-improving improvements in energy efficiency to climate policy, the omission of the opportunity costs of energy research and development, and the use of gross (rather than net) estimates of job creation.) These biases can be found in the chapters, the technical summaries, the summaries for policy makers, and the synthesis report.
The most important problem of the IPCC is the nomination and selection of authors and Bureau Members. Some experts are included or excluded because of their political allegiance rather than their academic quality. Sometimes, the “right” authors are put in key positions with generous government grants to support their IPCC work, while the “wrong” authors are sidelined to draft irrelevant chapters and sections without any support.
AR5 should be put on hold until the IPCC can prove that its author teams indeed have the relevant expertise, and its authors are at least as qualified as their peers. If needed, the IPCC should request additional nominations.
The IPCC member states are represented by their environment departments. For AR6, this responsibility should be transferred to their research departments or their academies. The selection process for authors and Bureau Members should be made transparent. The IPCC should commission a scientometric assessment of its Bureau Members, authors, and chapters.
IPCC assessment reports do not cover the entire literature. For instance, the IPCC does not fully assess the literature on trade-offs between the various greenhouse gases and the literature on international environmental treaties. Both areas are actively researched and conclusions are typically critical of current policy, but were largely omitted from previous IPCC reports. The literature on comparing the impacts of climate change to the costs of emission reduction falls outside WG2 and WG3, and thus outside the Synthesis Report as well. There is a risk that the nascent literature on ex-post evaluations of climate policy (in Europe and Japan) will be ignored in AR5, and particularly those papers that show that climate policy is ineffective and needlessly expensive.
The IPCC should submit the proposed outline for AR5 for peer review, and ensure that it covers all aspects of climate change and climate policy, including those parts of the academic literature that are potentially embarrassing to policy makers.
The IPCC Plenary has delegated its supervisory powers to the IPCC Bureau, which is also its executive. This implies that there is no mechanism to correct the IPCC Chair (if, for example, he oversteps his mandate and gives policy advice on behalf of the IPCC, if he says embarrassing things to journalists, or if he uses the IPCC to raise funds for his home institution). There is no mechanism to reconcile differences between IPCC Working Groups on the treatment of interdisciplinary topics. (The main example is the IPCC treatment of global warming potentials as a pure physical issue.) Therefore, the Chairs of the IPCC and its Working Groups should leave the IPCC Bureau, and the Bureau should adopt a supervisory role under a strong and independent chairperson.
In contrast to the procedures for governing the IPCC and selecting its authors and officials, the procedures for drafting IPCC reports are acceptable on paper but not in practice. Particularly, these procedures were not enforced in AR4 WG2 and WG3. (Some chapters rely heavily on gray literature while ignoring peer-reviewed literature on the same matter (e.g., Ch 7 WG2). Other chapters cite papers published after the deadline (e.g., Ch 15 WG2). Incomplete drafts were sent for peer-review (e.g., Ch 11 WG3). Substantial material was added after the final review (e.g., Ch 20 WG2).) In order to avoid repetition, Review Editors should be more independent, and return chapters for further revision if needed. The Review Editors rather than the Convening Lead Authors should have the final say about chapters. Review Editors should actively seek the input of referees. Review Editors should have the right to reject sections of the report that do not meet basic quality standards at the deadline. In short, Review Editors should have the same powers as journal editors. Review Editors should be selected from the ranks of (former) journal editors.
The IPCC should engage the wider public in the review process. After the publication of AR4, the “blogosphere” uncovered a large number of procedural and substantial errors. It would be preferable to discover this before publication. Engaging with “the public” is difficult. Therefore, the IPCC should do this in an experimental fashion in AR5 so as to prepare guidelines for AR6. Experiments could include posting draft chapters on blogs for public comment (A draft of the current statement was discussed at http://rogerpielkejr.blogspot.com/2010/05/richard-tols-draft-submission-to-iac.html), and posting draft chapters on wikis for public amendment.
Subscribe to:
Posts (Atom)