Active Learning in Machine Learning: How Models Learn More from Less Data

Active Learning in Machine Learning: Teaching Models What to Learn Next

Machine learning systems often need large amounts of labelled data to make accurate predictions. But labelling data can be slow, expensive or require specialist knowledge. Active learning offers a way to make this process more efficient: instead of labelling examples at random, a model helps identify which examples would be most useful to label next.

What is active learning?

In conventional supervised learning, a model is trained on a fixed collection of examples, each paired with a known answer or label. For example, a dataset of photographs might be labelled as “cat”, “dog” or “neither”. The model learns patterns from these examples and uses them to make predictions about new data.

In active learning, the model can select examples from a larger pool of unlabelled data and request labels for them. A person, or another trusted source, supplies the answers. The model then uses the newly labelled examples to improve its predictions and chooses further examples to label. This creates an iterative loop:

Train an initial model using a small labelled dataset.
Use the model to identify unlabelled examples that may be informative.
Ask an annotator or expert to label those examples.
Add the new labels to the training data and retrain or update the model.
Repeat until the model performs well enough or the labelling budget is reached.

Why choose examples actively?

Not every example provides the same amount of useful information. A model may already be highly confident that a straightforward image shows a dog. Having someone confirm that prediction is unlikely to teach it much. A blurry image that could plausibly show either a dog or a fox may be more valuable, because its label could help the model distinguish between the two.

By prioritising useful examples, active learning can reduce the number of labels needed to reach a given level of performance. This is particularly helpful when labels are costly, data is plentiful, or annotation requires specialist expertise—for instance, identifying conditions in medical images or classifying technical documents.

Common ways to select data

The process of choosing which examples to label is known as a query strategy. Several approaches are widely used:

Uncertainty sampling: Select examples for which the model is least confident. This may mean choosing the prediction with the lowest confidence or examples where the model’s leading predictions are close together.
Query by committee: Train several models and select examples on which they disagree. Disagreement can indicate that an example would help resolve uncertainty in the model’s predictions.
Diversity sampling: Choose examples that represent different parts of the data, rather than selecting many similar cases. This can help prevent the labelled dataset from becoming too narrow.
Expected improvement: Estimate which label would be most likely to improve the model’s future performance, and prioritise examples accordingly.

These strategies can be combined. For example, a system might first find uncertain examples, then select a diverse subset so that annotators do not receive a batch of near-duplicates.

Different active learning settings

In pool-based active learning, the model chooses from a fixed collection of unlabelled examples. This is common when an organisation has a large archive of documents, images or customer messages.

In stream-based active learning, examples arrive one at a time or in a continuing stream. The system decides whether each new example should be sent for labelling. This can be useful when data is generated continually, although the model must make decisions quickly.

In membership-query synthesis, a system can request labels for examples it has generated itself, rather than selecting only from existing data. This approach can be challenging in practice: generated examples may be unrealistic or difficult for a person to label consistently.

A practical example

Imagine a team building a system to sort incoming support messages into categories such as “billing”, “technical issue” and “account access”. The team begins by labelling a modest set of messages and training an initial classifier. The classifier then identifies messages for which its predictions are uncertain. A support specialist reviews those messages, provides the correct categories, and the model is updated.

Over several rounds, the system can learn from the kinds of messages it finds difficult. The team still needs to check the model’s overall performance, especially on messages that were not selected for labelling. Active learning helps allocate annotation effort; it does not remove the need for careful testing or human oversight.

Benefits and limitations

The main potential benefit is efficiency. When annotation is a bottleneck, selecting informative examples can help a team make better use of limited time and expertise. Active learning can also make it easier to adapt a model when a new category or type of data appears.

However, the approach has important limitations:

Uncertainty is not the same as usefulness. A model may be uncertain about an unusual or corrupted example that does not help it perform better on real-world cases.
Selection can create bias. If the model repeatedly chooses examples from groups or topics it already recognises, other parts of the data may be under-represented.
Labels can be inconsistent. Human annotators may disagree, particularly when categories are subjective or instructions are unclear.
The process adds complexity. Teams need tools for selecting examples, managing annotation, tracking versions of the data and monitoring model performance.
It may not always outperform simpler methods. A well-designed random sample can be effective, and some active-learning strategies require considerable computing or implementation effort.

How to use active learning responsibly

A sound active-learning workflow starts with a clear goal and a reliable evaluation set. The evaluation data should be kept separate from the examples selected during training, so that performance is measured on data the model has not actively chosen. Teams should also track where the labelled examples come from and check whether important groups, categories or edge cases are missing.

Annotation guidance matters just as much as the selection strategy. Clear definitions, examples and a way to resolve disagreements can improve label quality. For high-impact applications, expert review and human decision-making may remain essential, even when a model is confident.

Conclusion

Active learning changes the question from “How much data can we label?” to “Which data is most worth labelling next?” By directing human attention towards potentially informative examples, it can make supervised machine learning more economical and responsive. Its success depends on thoughtful data selection, dependable labels and rigorous evaluation. Used with those safeguards, active learning can be a valuable part of building machine-learning systems when labelled data is in short supply.

8 Practical Tips for Active Learning in Machine Learning

Start with a small, representative labelled set.
Choose uncertain examples for expert labelling.
Add diverse samples to avoid redundant queries.
Set a clear labelling budget and stopping point.
Check label quality before adding examples.
Track performance on a fixed validation set.
Watch for class imbalance and rare cases.
Compare against random sampling as a baseline.

Start with a small, representative labelled set.

Start with a small, representative labelled set that reflects the range of examples the model is likely to encounter, including different categories, formats and edge cases. This gives the model a more balanced foundation for its initial predictions and helps avoid early bias towards common or easy examples. Check the labels for accuracy and consistency, then use the first model to identify which additional examples would be most valuable to label.

Choose uncertain examples for expert labelling.

Choose uncertain examples for expert labelling to focus annotation effort where it may be most valuable. When a model is unsure which prediction is correct, an expert’s label can help clarify a difficult case and improve future predictions. This approach is especially useful when labelling is costly, but uncertainty alone is not a guarantee that an example will be informative. Combine it with checks for data quality and diversity, and keep a separate evaluation set to measure whether the model is genuinely improving.

Add diverse samples to avoid redundant queries.

When selecting examples for labelling, include a diverse mix rather than several near-identical samples. A model may be uncertain about many similar cases, but labelling them all can provide little new information. Choosing examples from different categories, sources or regions of the data helps reduce redundant queries and gives the model a broader range of examples to learn from.

Set a clear labelling budget and stopping point.

Set a clear labelling budget and stopping point before you begin active learning. Decide how many examples can be labelled, or how much time and money can be spent, and agree what success looks like—for example, reaching a target level of accuracy or seeing only small improvements after each round. A defined limit helps teams manage resources, compare active learning with simpler approaches and avoid continuing to label data when the benefits are no longer worth the cost.

Check label quality before adding examples.

Before adding newly labelled examples to your training data, check that the labels are accurate, consistent and based on clear guidelines. Active learning often selects difficult or ambiguous cases, so these examples may be especially prone to mistakes or disagreement between annotators. Reviewing uncertain labels, resolving inconsistencies and documenting any corrections helps prevent poor-quality data from misleading the model and undermining its performance.

Track performance on a fixed validation set.

Track your model’s performance on a fixed validation set throughout the active-learning process. Because the model repeatedly selects new examples to label, its training data changes with each round; a consistent, separate validation set makes it possible to compare results fairly and see whether each batch is genuinely improving performance. Keep this set out of both training and example selection, and make sure it represents the data the model will encounter in practice.

Watch for class imbalance and rare cases.

Watch for class imbalance and rare cases when selecting examples to label. An active-learning strategy may favour common categories or cases the model finds uncertain, leaving rare but important examples under-represented. This can result in a model that performs well overall but misses critical edge cases. Track how often each class is selected, deliberately include examples from less common groups, and assess performance separately across classes—not just with an overall accuracy score.

Compare against random sampling as a baseline.

When using active learning, compare its results with random sampling as a baseline. Randomly selecting examples to label is simple, but it provides a useful reference for judging whether your query strategy is genuinely improving efficiency. Test both approaches under the same labelling budget and evaluate them on the same held-out dataset, comparing performance after each batch of labels. If active learning does not consistently outperform random sampling, its added complexity may not be worthwhile.

behaveannual.org

Driving Positive Change through Behavioral Science