Data Platform Services
In the ever-evolving landscape of digital services, GOV.UK required a critical change: building a robust data platform from scratch to power the service-deliveries and search functionalities across multiple microservice verticals. This platform needed to be highly secure, reliable, and resilient,running seamlessly on AWS. The key performance indicators (KPIs) focused on these aspects, demanding a solution that could handle Big Data efficiently.
Key Challenges Solved
- Building from scratch
- Ensuring high security standards & reliability
- Processing large volume of data(big data)
- Decoupled development from dev to production
- Shift-left testing and fail-fast strategy
- Seamless environment transitions
My Key Contributions
To address these requirements, I spearheaded the design and development of the data platform, ensuring a foundation built on best practices with a decoupled development environment and a fail-fast strategy. This approach aimed to deliver a high-quality product while minimizing defects, during environment transitioning until go-live.
Created reference implementations for the microservice verticals, leveraging Kafka Data Streaming APIs and Postgres DB APIs, adhering to clean code principles and industry best practices.
Additionally, I developed a Micro-Simulator developer-tool with custom annotations which empowered developers, QAs, and business analysts to collaborate efficiently and helped in accurate analysis of business scenarios using simple, human-readable JSON DSLs.
Tech Stack Used
+ DATA STREAMING:
Kafka
+ CONTAINERISTION:
Docker,K8s
+ DATABASES:
Postgres,Oracle,HBase
+ LANGUAGES:
Java,Python
+ INFRA:
Skyscape,AWS
+ CI CD:
Jenkins
+ SCM :
Git,Bitbucket
+ SCRIPTS:
tf,cf,bash