Challenges in Deploying Machine Learning: a Survey of Case Studies, Paleyes et al., 2020

Paper, Tags: #nlp

Machine learning deployment workflow

1. Data management

Data collection
Data preprocessing
Data augmentation
Data analysis

2. Model learning

Model selection
Training
Hyper-parameter selection

3. Model verification

Requirement encoding
- Performance metrics, business driven metrics
Formal verification
- Regulatory frameworks
Test-based verification, checking that the model generalizes well to the previously unseen data.
- Simulation-based testing

4. Model deployment

Integration
- Operational support
- Reuse of code and models
- Software engineering anti-patterns
- Mixed team dynamics
Monitoring
- Feedback loops
- Outlier detection
- Custom design tooling
Updating
- Concept drift
- Continuous delivery

5. Cross-cutting aspects

Ethics
- Country-level regulations
- Focus on technical solution only
- Aggravation of biases
- Authorship
- Decision making
End user's trust
- Involvement of end users
- User experience
- Explainability score
Security
- Data poisoning
- Model stealing
- Model inversion