Problem Statement: Predict Titanic survivors using a neural network, comparing Adam and RMSprop optimisers to determine the most effective approach.
Approach: Preprocessed Titanic data, built a simple neural network, trained with both optimisers, and evaluated performance to assess their effectiveness.
Loaded 891 records, dropped 'PassengerId', 'Name', 'Ticket', 'Cabin'; filled missing 'Age' and 'Embarked' with mode.
One-hot encoded 'Sex', 'Embarked'; split into 80% train (10% validation), 20% test with stratification.
Scaled features with StandardScaler post-split to prevent data leakage.
891 passenger records
Clean, encoded features
Initialise project to compare optimiser effectiveness for Titanic survival prediction.
Load TensorFlow, Keras, scikit-learn, pandas, and the Titanic dataset.
Clean data, handle missing values, encode categorical features, split and scale data.
Create a neural network with 64-32-1 architecture, using ReLU for hidden layers and Sigmoid for output.
Compile and train the model using Adam optimiser with binary cross-entropy loss.
Compile and train an identical model architecture using RMSprop optimiser.
Compare loss, accuracy, and training behaviour between the two optimisation approaches.
Draw conclusions about optimiser performance and make recommendations.
Adaptive Moment Estimation combines the advantages of AdaGrad and RMSProp by storing both moving averages of past gradients and squared gradients.
Root Mean Square Propagation maintains a moving average of the squared gradient for each weight, dividing the gradient by the square root of this average.
Trained for 10 epochs with batch size of 32. Validation split of 10% used to monitor performance and prevent overfitting.
Identical training settings to Adam. Monitored validation loss and accuracy over 10 epochs for comparison.
Adam converged faster initially, showing a steeper loss reduction in the first few epochs. However, it showed signs of slight overfitting in later epochs with validation loss increasing whilst training loss continued to decrease.
RMSprop demonstrated more stable validation loss throughout training, suggesting better generalisation potential despite slightly higher overall loss values.
Both optimisers reached accuracy levels between 0.80-0.85 on the test set. Adam achieved slightly higher peak accuracy (0.85 vs 0.82), but RMSprop showed more consistent performance across validation and test sets.
Adam's faster convergence makes it suitable for quicker training, whilst RMSprop's stability suggests better performance for production deployment.
| Metric | Adam | RMSprop | Notes |
|---|---|---|---|
| Final Test Accuracy | 0.85 | 0.82 | Adam slightly higher |
| Final Test Loss | 0.41 | 0.45 | Lower is better |
| Convergence Speed | Fast | Medium | Adam reached target accuracy earlier |
| Validation Stability | Medium | High | RMSprop showed less overfitting |
| Training Time | 1.2x | 1.0x | Relative computation time |
Adam and RMSprop performed similarly in predicting Titanic survivors, both achieving accuracy levels between 0.80-0.85. Adam converged faster and achieved slightly higher peak accuracy (0.85), but showed signs of overfitting in later epochs. RMSprop offered better validation stability, suggesting superior generalisation potential despite slightly lower overall accuracy (0.82).
Select Adam when training speed is a priority and when implementing regularisation techniques like early stopping to prevent overfitting.
Prefer RMSprop for production models where generalisation and stability are more important than raw performance metrics.
For the Titanic dataset, the differences between optimisers were modest. This suggests that for small-to-medium datasets with binary classification tasks, both Adam and RMSprop are viable choices. The final selection should consider specific project requirements regarding training speed, model stability, and deployment context.
Future work could explore combining the advantages of both optimisers with techniques like learning rate scheduling or hybrid approaches in more complex neural network architectures.