Titanic Survivor Prediction: Neural Network Optimisation Study

Titanic Survivor Prediction

Neural Network Optimisation Study: Comparing Adam vs RMSProp with Advanced Regularisation Techniques

Complete Process Flowchart

Comparing Optimisers (Adam vs RMSProp)

1
Data Loading & Preprocessing
Loaded Titanic dataset, dropped irrelevant columns (PassengerId, Name, Ticket, Cabin), handled missing values with mode imputation
2
Feature Engineering
Applied one-hot encoding to categorical variables (Sex, Embarked), standardised numerical features after train-test split
3
Model Architecture Design
Built neural network with input layer, one hidden layer (64→32 neurons), ReLU activation, sigmoid output layer
4
Optimiser Comparison
Trained identical models with Adam and RMSProp optimisers, binary crossentropy loss, 10 epochs training
5
Performance Evaluation
Adam showed faster convergence but slight overfitting; RMSProp demonstrated more stable validation performance

Implementing Regularisation Techniques

6
L2 Regularisation Implementation
Added L2 regularisation (0.01) to prevent overfitting by penalising large weights in the neural network
7
Dropout Integration
Implemented dropout (0.1 rate) after hidden layers to reduce co-adaptation and improve generalisation
8
Enhanced Model Training
Trained models with both optimisers using combined L2 regularisation and dropout techniques
9
Regularisation Impact Analysis
Observed improved generalisation with slightly higher training loss but better validation stability

Implementing Early Stopping

10
Early Stopping Configuration
Implemented early stopping with patience=1, monitoring validation loss, learning rate=0.001
11
Training Optimisation
Combined early stopping with L2 regularisation for both Adam and RMSProp optimisers
12
Cross-Validation Implementation
Applied Stratified K-Fold (5-fold) cross-validation for robust performance estimation
13
Comprehensive Metrics Analysis
Evaluated models using accuracy, precision, recall, and F1-score across all configurations

Key Findings & Conclusions

📊
Cross-Validation Results
Optimiser Accuracy Confidence Interval
Adam 82.83% ±4.78%
RMSProp 81.93% ±4.30%

Adam demonstrates slightly superior average performance with marginally higher variance.

🎯
Detailed Performance Metrics
Model Configuration Precision Recall F1-Score
4.2.4 Adam 83.85% 67.54% 74.74%
4.2.4 RMSProp 84.28% 66.67% 74.39%
4.3.3 Adam 82.35% 68.43% 74.64%
4.3.3 RMSProp 82.88% 67.26% 74.13%
⚡
Optimiser Characteristics
Adam Optimiser:
  • Faster initial convergence
  • Better recall performance (68.43%)
  • Slight tendency towards overfitting
  • Higher F1-score balance
RMSProp Optimiser:
  • More stable validation performance
  • Superior precision (84.28%)
  • Better generalisation capabilities
  • Lower variance across folds
🔬
Regularisation Impact

L2 Regularisation (0.01):

Successfully prevented overfitting by penalising large weights, resulting in more stable validation performance across both optimisers.

Dropout (0.1 rate):

Reduced neuron co-adaptation, improving model robustness and generalisation to unseen data.

Early Stopping:

Enhanced training efficiency by preventing unnecessary epochs, maintaining optimal model performance without overfitting.

Business Implications & Recommendations

🎯 Model Selection Strategy

  • For Balanced Performance: Model with Adam optimiser offers the highest F1-score (74.74%) and excellent overall accuracy
  • For High Precision Requirements: Model with RMSProp achieves superior precision (84.28%) for minimising false positives
  • For Maximum Survivor Identification: Model with Adam provides highest recall (68.43%) for comprehensive survivor detection
  • For Consistency: RMSProp demonstrates lower variance and more stable performance across different data splits

📈 Implementation Recommendations

  • Production Deployment: Implement Model with Adam as primary model due to balanced performance metrics
  • Regularisation Best Practices: Continue using L2 regularisation (0.01) and dropout (0.1) for optimal generalisation
  • Training Efficiency: Maintain early stopping with patience=1 to prevent overfitting and reduce computational costs
  • Cross-Validation Protocol: Use Stratified K-Fold validation for all future model evaluations to ensure robust performance estimates
  • Hyperparameter Tuning: Consider grid search optimisation for learning rates and regularisation strengths in future iterations

Final Recommendation:

Deploy Model with Adam optimiser for production use, implementing comprehensive regularisation techniques and early stopping for optimal survivor prediction performance.