ML Engineering / MLOps
Deployment of PySpark AI-Applications
Deployment of locally developed PySpark AI-applications into a production-ready Kubernetes cluster
- Industry
- Public Administration
- Client
- Stiftung Zentrale Stelle Verpackungsregister, Osnabrück
- Engagement period
- January 2024 – February 2024
- Delivery mode
- Hybrid
Situation
The client had built several distributed PySpark AI applications that ran only in local mode. They needed these applications moved into a production setting on a dedicated Kubernetes compute cluster, with the reliability guarantees that entails.
Objective
Templefield Technologies was engaged to productionize the local PySpark applications and stand up the supporting Kubernetes infrastructure required for reliable, scalable operation.
Approach
- Containerized the local PySpark applications
- Established the required Kubernetes infrastructure — deployments, external secrets, network policies, and Helm charts
- Implemented model monitoring and serving via MLflow
- Configured pod autoscaling for efficient resource use
- Orchestrated batch jobs through Apache Airflow
Results
The applications were successfully deployed to production and made accessible to end users, while maintaining fault tolerance, scalability, and reliability.