wiki:ReviewProcessRevisions

Context Navigation

Version 3 (modified by Gordon Woodhull, 8 years ago) ( diff )
minor formatting

The Boost Formal Review Process has evolved since it was described on the website.

This wiki page documents proposed changes to the Review Schedule, Library Submission Process, and Formal Review Process pages.

The goals are:

demystify the process for new library authors
encourage more review managers to step forward
remove cruft

In addition, this documents the following new processes:

propose the Incubator as a replacement for the queue-which-is-not-a-queue
the Community Maintenance Team for orphaned libraries
mini-reviews for reviewing conditions on acceptance
passing on of orphaned reviews

Except for the new Boost Library Incubator, there is no attempt to improve the process. The purpose is to document the process as it exists today.

For context, most of the original text of the pages is quoted

Subjective advice is in italics; part of the proposal is to include this or put it into an 'Advice for Authors and Review Managers' page.

Boost Formal Review Schedule

http://www.boost.org/community/review_schedule.html

Reviews are usually scheduled --on a first-come-first-served basis, and normally last ten days.-- See Formal Review Process for more information.

Reviews are scheduled when the review wizards approve a review manager and agree with the manager and author on dates. Reviews normally last ten days...

Boost Library Submission Process

http://www.boost.org/development/submissions.html

Determine interest

Potential library submitters should --use the Boost developers mailing list as a forum-- to gauge interest a possible submission.

... use the Boost developers mailing list and Boost Library Incubator as forums ...

A message might be as simple as "Is there any interest in a library which solves Traveling Salesperson problems in linear time?" A bit of further description or snippet of code may be helpful. Messages should be plain text; not rich text, HTML, etc. Please don't post lengthy descriptions, documentation, or code to the mailing list, and no attachments, even small ones. Please post lengthy material in the YahooGroups boost Files section (formerly called the " vault").

Please post your code to a version control system such as Github, and make your documentation available in HTML format on a public website. An issue tracker is also highly recommended.

Release your code under the (Boost Software License).

(Move from Formal Review section)

Please verify that your submission compiles and runs under at least two compilers. This flushes out obvious portability problems.

While participation in prior reviews is not a prerequisite for submitting a library to Boost, it is highly recommended because it will acquaint you with the process and the emotional demands of a formal review.

Preliminary submission

If response to an initial query indicates interest, then make your preliminary submission publicly available if you haven't already done so.

(Remove this step. Everyone seems to be fine with putting their code in version control from the get-go.)

Discuss, refine, resubmit. Repeat until satisfied. The exact details of this process varies a lot. Sometimes it is public, on the mailing list, sometimes a lot of discussion happens in private emails. For some libraries the process is over quickly, for others it goes on for months. It's often challenging, and sometimes leads off in completely unexpected directions. The archive of past messages is one way to see how this process worked for other Boost libraries.

(Nothing changed here!)

Seek a Review Manager

(new step)

In order to have a formal review, the author must find a capable volunteer to manage the review. This should be someone with knowledge of the library domain, and experience with the review process. See (Formal Review Process) for the responsibilities of the review manager. Review Managers are approved by the review wizards based on their level of participation in the Boost community.

Some reviews are managed more by the author, with the manager signing off on the results, and other reviews are managed more by the manager. This depends on the personalities of author and manager. In some cases, the author can be trusted to synthesize requirements from the review, whereas in other cases, the manager must take a more active role.

Hopefully someone will step forward to manage the review during the refining stage; if not, it may be appropriate to contact members of the community off-list to request their help. However, please respect that managing a review is a significant amount of work and one should not ask another to undertake this without first being sure they have the motivation to complete the obligation.

If there is any question that a review manager might not be approved (e.g. because they are new to the community), the author and volunteer manager should contact the review wizards off-list and seek approval.

Submission for review

All of the files which make up the library should be combined and compressed into a single submission file using the .zip format. Free encoders and decoders for this format running on many different platforms are available at the Info-ZIP web site, which includes a FAQ and much other useful information about the .zip format. Many commercial compressor-archiver utilities also support this format.

The submission file should contain material as if on the boost.org web site. The closer the submission file mirrors the final directory structure and format of the web site, the better.

Like a preliminary submission, make final submission .zip publicly available.

(Pure cruft! Remove section, move directory structure requirement to Formal Review step or Determine Interest step.)

Formal Review

Before asking for formal review, your submission should be easilly accessible to all. Please verify that your submission compiles and runs under at least two compilers. This flushes out obvious portability problems. If you don't have access to a second compiler, ask for help on the Boost mailing list.

(Remove: these requirements are covered earlier.)

Once a library author feels a submission has matured enough for formal review, the author sends a message requesting a formal review to the mailing list. Please use a subject in the form "Review Request: library" where library is replaced by the library name.

Once a library author feels a submission has matured enough for the formal review, and has found someone to manage the review, the author sends a message requesting a formal review to the mailing list and the review wizards. Please use a subject in the form "Review Request: library" where library is replaced by the library name.

The review wizards will schedule the review at a date convenient for the author and review manager.

At the time of submission, the directory structure of the library should mirror Boost's directory structure.

See Formal Review Process for details.

Formal Review schedules are posted on the web site.

Boost Formal Review Process page

http://www.boost.org/community/reviews.html

Before Requesting a Formal Review

Read and follow the Boost submission process. There are --at least four steps-- a library author must take before a formal review is requested.

... several steps ...

Introduction

Proposed libraries are accepted into Boost only after undergoing a formal review, where Boost mailing list members comment on their evaluation of the library.

The final "accept" or "reject" decision is made by the Review Manager, based on the review comments received from boost mailing list members.

Boost mailing list members are encouraged to submit Formal Review comments:

Publicly on the mailing list.

or on the Boost Library Incubator

Privately to the Review Manager.
Private comments to a library submitter may be helpful to her or him, but won't help the Review Manager reach a decision, so the other forms are preferred.

What to include in Review Comments

(This hasn't changed.)

Results

At the conclusion of the comment period, the Review Manager will post a message to the mailing list saying if the library has been accepted or rejected. A rationale is also helpful, but its extent is up to the Review Manager. If there are suggestions, or conditions that must be met before final inclusion, they should be stated.

Notes for Review Managers

Before a library can be scheduled for formal review, an active boost member not connected with the library submission must volunteer to be the "Review Manager" for the library.

The Review Manager:

Checks the submission to make sure it really is complete enough to warrant formal review. --See the Boost Library Requirements and Guidelines--. If necessary, work with the submitter to verify the code compiles and runs correctly on several compilers and platforms.

See the Boost Library Requirements and Guidelines - or submitting through the Boost Library Incubator will

Decides if there is consensus to accept the library, and if there are any conditions attached.

Consensus is not the same as a vote. The Review Manager has discretion to weigh opinions based on authority or thoughtfulness.

Notes for Library Submitters

(Unchanged.)

Library Maintainer's Rights and Responsibilities

By submitting a library to boost, you accept responsibility for maintaining your library or finding a qualified volunteer to serve as maintainer. You must be willing to put your library and documentation under a Boost-compatible license.

You will be expected to respond to reasonable bug reports and questions in a timely manner and to participate as needed in discussions of your library on the boost mailing lists.

You are free to change your library in any way you wish, and you are encouraged to actively make improvements. However, peer review is an important part of the Boost process and as such you are also encouraged to get feedback from the boost community before making substantial changes to the interface of an accepted library.

If at some point you no longer wish to serve as maintainer of your library, it is your responsibility to make this known to the boost community and to find another individual to take your place.

Libraries which have been abandoned will be put in care of the Community Maintenance Team.

Review Wizard

(Responsibilities have changed considerably here.)

The Review Wizard coordinates the formal review schedule:

(Ok)

Maintains a list of review manager volunteers, in the form of a queue, so that volunteers who least recently managed reviews become the prime candidates for upcoming reviews.

(Nix. There is no manager volunteer queue, because domain knowledge and interest are required.)

When a formal review is requested for a library:
--Assign a review manager-- and

Approve the review manager based on their participation in the Boost community, including the mailing list. and previous reviews? Stack Overflow participation?

Suggest a schedule, after checking (via private email) availability of the volunteers

Finalize the schedule, once the review manager verifies the library is actually ready for review.
Resolve schedule slips or other issues with review managers and submitters.

(Ok)

(Bikeshed question: do we continue to call it a review queue, but say "the queue is not a queue"? Or simply replace the Queue with the Incubator?)

Maintains a schedule of both past and pending reviews, in the form of the Review Schedule web page.
Resolves questions from review managers and library submitters, who sometimes want a third opinion on questions such as "Should we extend the review period because ...?"
Monitors the general review process, and makes minor adjustments as needed, or queries the list about possible major adjustments.

The role of Boost Review Wizard is currently played by John Phillips (johnphillipsithaca at gmail dot com) and Ronald Garcia (rxg at cs dot ubc dot ca).

(Still current.)

Fast Track Reviews

To qualify for fast track review:

The component must be small.
The technique must be already in use in Boost libraries and the new component provides a common implementation.
A full Boost-conformant implementation is available in the sandbox.
The Review Wizard determines that the proposal qualifies for fast track review.

Procedure:

The Boost Review Wizard posts a review announcement to the main Boost developer's list. The review period will normally last for 5 days. No two fast track reviews will run in parallel. Fast track reviews may run during full reviews, though generally this is to be avoided.
After the review period ends, the submitter will post a review summary containing proposed changes to the reviewed implementation.
The Review Wizard will accept or reject the proposed library and proposed changes.
After applying the proposed changes, the component is checked into the repository like any other library.

Mini-Reviews

(add section)

If a review results in conditions on acceptance, the review manager may request a Mini-Review to determine if the conditions have been met. The Mini-Review is usually conducted by the same review manager.

Orphaned Reviews