Streamnow
Objective:
- Develop data-ingestion engine to downloaded semi-structured data from streaming and batch sources
- Scalable and Extensible plugin based components
Approach:
- Understood the scope of data model necessary for tactical solution
- Constructed a design with future ambitions as strategic solution
- Built an in-house near real-time data-ingestion engine within 3 months, with additional features and high run-time configurability
- Plugins for attaching other services like parsing multiple formats(JSON, CSV, Fixed Width,..),tokenization of data, masking and other ETL processes as need arrives.
Results:
- First kind of bigdata ingestion engine, capable of loading terabytes of data in 1-2 days, within Barclays
- Completely open source driven API and metadata driven workloads
- Scalable service with capacity to cater demands from multiple sources at once