Deep Learning / Model Evaluation

Optimiser Comparison: Adam vs RMSprop

Two optimisers compared on a small neural network for historical passenger-survival data, then regularisation, dropout, early stopping and cross-validation explored.

Type

Binary Classification

Domain

Model Evaluation Methodology

Methods

Adam, RMSprop, Regularisation

Origin

Cambridge DS Career Accelerator coursework

OPTIMISER EVALUATION

4-STAGE METHODOLOGY

BASELINE COMPARISON

REGULARISATION + DROPOUT

EARLY STOPPING

5-FOLD CROSS-VALIDATION

The Challenge

On a held-out test set, a small network trained with RMSprop scored 86.0% accuracy against 83.2% for Adam.

Optimiser choice is often left at the default. The aim was to compare Adam and RMSprop on the same architecture and data, then test whether regularisation, dropout and early stopping change the picture.

Approach

Data

891 passenger records, 10 features after encoding; split 72% train, 8% validation, 20% test.

Baseline comparison

Identical 64-32 networks trained for 10 epochs with Adam and with RMSprop.

Regularisation and early stopping

L2 regularisation, dropout and early stopping added to both.

Cross-validation

5-fold cross-validation set up to compare the optimisers more robustly than a single split.

Results

86.0%

RMSprop test accuracy (baseline network)

83.2%

Adam test accuracy (baseline network)

891

Records; 179 in the test set

In the baseline comparison RMSprop finished ahead. On a test set of this size a difference of about 3 points is roughly five passengers, so a single split cannot separate the optimisers reliably, which is why cross-validation was the next step.

Note: the regularisation, early-stopping and cross-validation cells were not saved with outputs, so their results are not quoted here. They will be added after a re-run.

Technology Stack

Python TensorFlow / Keras scikit-learn pandas NumPy Matplotlib

Interested in this work or something similar?

Get in Touch View All Projects