Guacamol Benchmarking Models for De Novo Molecular Design: A Practical Guide

GuacaMol benchmarks molecular-generation models using two types of tasks: distribution learning and goal-directed design. For a meaningful comparison, evaluate models on the same tasks, with the same benchmark version and sampling budget, and report task-level scores alongside any aggregate.

Introduced to make comparisons between generative models and established algorithms more consistent, GuacaMol provides benchmark definitions and scoring methods for these evaluations (GuacaMol: Benchmarking Models for De Novo Molecular Design, 2019).

Choose a benchmark type

GuacaMol’s tasks assess different capabilities:

• Distribution-learning tasks assess whether a model can learn and reproduce features of a reference molecule distribution. The model generates samples from its learned distribution.
• Goal-directed tasks assess whether a model can generate molecules that meet a defined objective. Depending on the task, the objective may involve molecular properties, similarity to a reference, or a combination of scoring criteria.

The required model interface depends on the task. Distribution-learning evaluations use generated samples; goal-directed evaluations require a model interface that can receive the benchmark’s request and return candidate molecules. Check the documentation for the specific task and version you plan to run.

GuacaMol’s benchmark definitions and scoring code are available in its open-source repository. Follow the instructions for the version you install. Use the output format required by the task runner—commonly SMILES strings—and apply consistent handling to invalid outputs.

Before running an evaluation, choose the tasks and set the number of samples or optimization attempts. Use the same settings for every model. Record random seeds where applicable, but note that matching seeds alone does not make runs equivalent: model behavior, sampling procedures, and benchmark configuration also matter.

Interpret task scores before the aggregate

GuacaMol reports scores for individual tasks and may provide an aggregate across them. Task-level scores show performance on specific objectives. An aggregate is a summary, but it can hide differences between tasks and depends on which tasks are included.

Interpret each score in light of its task definition. Distribution-learning and goal-directed evaluations ask different questions, so their scores are not interchangeable measures of overall molecular quality. A high score on one objective does not establish strong performance on another.

Validity, uniqueness, and novelty are useful concepts when reviewing generated molecules, but they are not part of every GuacaMol score. Validity concerns whether the relevant chemistry toolkit can interpret an output as a molecule. Uniqueness concerns repeated outputs within a sample. Novelty concerns whether outputs are absent from a specified reference set. Their calculation depends on the evaluation setup, so check the task definition before reporting them.

For goal-directed tasks, inspect how candidates meet the objective and how the task combines its criteria. A model’s performance may vary across components; task-level results help explain its aggregate score and reveal its limits.

Example: comparing two distribution-learning runs

Suppose two models are evaluated on the same distribution-learning task, benchmark version, and sample budget. If Model A achieves a higher task score than Model B, that supports a comparison on that task under those settings. It does not show that Model A is better at goal-directed optimization or that its molecules suit a particular research purpose. Evaluate the relevant tasks before making those claims.

Sample budgets matter because generating more candidates may give a model more chances to meet an evaluation criterion. Different budgets can therefore blur the effect of the model itself. Report the sampling procedure and budget alongside the score.

Make comparisons reproducible

A useful benchmark report gives readers enough information to understand how scores were produced. Include the GuacaMol version or repository commit, the tasks evaluated, and configuration choices that affect scoring. Report the number of generated candidates or optimization attempts for each task, along with random seeds where applicable.

Describe the model and evaluation conditions, including relevant details such as the model checkpoint, training data, sampling settings, and output preprocessing. If you filter, canonicalize, or remove duplicate molecules, explain what you did and whether you applied the same procedure to every model. Preprocessing can affect counts and scores.

Present per-task results alongside any aggregate. If you omit a task, state which one and why. This helps readers distinguish results across the selected tasks from results based on a subset. Avoid treating aggregate scores from different task lists or benchmark versions as directly equivalent.

Where possible, share generated outputs and evaluation configuration so others can inspect or rescore the candidates without repeating model training. This also helps separate differences caused by the model from those caused by scoring or data handling.

GuacaMol provides a common set of evaluations, not a complete test of every intended use. A model designed for a particular target or workflow may require additional evaluation suited to that setting. Report those results separately from standardized benchmark scores.

Key Takeaways

• GuacaMol evaluates distribution learning and goal-directed molecular design through task-specific benchmarks.
• Choose tasks that match your research question and follow each task’s input and scoring requirements.
• Interpret task-level results before relying on an aggregate; different tasks measure different objectives.
• Keep benchmark versions, task selections, sample budgets, and evaluation procedures consistent when comparing models.
• Report enough detail for others to understand the evaluation, and use domain-specific tests when the research question requires them.