Software Engineer (Hadoop, Realtime, DataLake, Big Data, ETL, Business Intelligence, Advanced Analytics, Cloud , Spark)

• Designed and developed multiple containerized spark jobs to integrate the data sources (csv, hbase, hive and SQLServer) across big data landscape hosted in Azure.
• Schema designing for batch layer storage in hive and real time layer storage in Hbase.
• Data collection from Ecommerce platform: Integrated scriptlets for gathering behavioural, device tracking and PII data.
• Containerized the R based model to expose as an API, enabled the model for TCP call
• Learned the modelling techniques with H2o.
• Built a tag manager for the ecommerce platform and designed Rest APIs for data automated data collection using microservices architecture with NodeJs/Seneca
• Integrated APIs of various vendors in E Commerce platform for data collection and device identification

• Data architecture setup using Hadoop, Spark and Hive for in house clusters
• Gathered and processed raw production data at scale from relational sources using Sqoop.
• Performed data modelling for storage in HDFS (Serialization, Compression, Retrieval, Partitioning, Bucketing and de-normalizing aspects)
• Work closely with Testing and Quality Teams to integrate the data context into Hadoop storage layer.
• Learned and applied concepts surrounding spark data abstraction using APIs of RDDs and DataStream
• Used Statistics knowledge to do analysis.
• Organized presentations to bring project stakeholders’ understanding at same level.

• Designed and developed multiple Map Reduce jobs using locally configured Hadoop clusters for dataset hosted in HDFS as well as in Amazon S3 storage
• Created and configured [S3 triggered] AWS Elastic MapReduce cluster [using Spot-Instances] for chained jobs with common crawl data of terabytes.
• To achieve high speed up, integrated the key-value store of Infinispan (memcache)
• Designed complex charts to analyse the impact of bucket items and bucket size on distribution storage(used R statistical Language)
• Designed web services using python and wrote algorithms for web graph's optimal cost calculation

• Analysis, design and implementation of features for an analytic search service using SQL script
• Analysis and correction of complex Software Errors
• Setting up an environment for shared development process based on HANA Studio
• Development of features in a Javascript and HTML5 based interface
• Build real time analytic utility for customers to find their business directions using a natural query interface. It is a generic open domain and open language tool provided with language formalization.

Reporting/Analytics Migration Project

• Developed a utility using J2SE to map data model from IBM Cognos Catalog to SQL Server Reporting Service SSRS Report Model.
• Designed, developed and tested SSRS Reports/SSAS Cubes with user acceptance criteria.
• Business and technical Requirements engineering using Cognos Catalog Reports,
• Impromptu Catalog, DB2 schema. (The catalog consist of 6 DB schema and 200+ table)
• Delivered report model,100 reports and 6 cubes of acceptable quality within given deadlines.
• Deployed and tested reports, cubes over Sharepoint servers(development & production)
• Speeded up the development of XML Generation utility using Java StAX API, to generate SSRS model components of data source View and Semantic Model files

Data Migration in ETL Project

• Design data flows for an integration project to synchronize the multi systems data from different stores to central repository.
• Requirement engineering from legacy infrastructure of Oracle DB packages to overcome performance issues, faults in the design and unavoidable functional errors.
• Enhanced and newly design & developed 10 packages in SQL Server Integration Services SSIS to complete ETL iteration.
• Cleanse data from 5 item store databases to central repository and loaded under strict analytic requirement of transformations.
• Administered the running Scheduled for given packages, deployed in testing & production environments of SQL Server Integration services.
• Performed rigorous Testing and achieved speed up of overall execution by 38% by tuning data flows with standard SSIS practices
• Enhanced performance using automatic scheduling of data flows.

Revlon ETL Project (Staging, Migration and Integration phases)

• Analysed, designed and developed data workflows for a data consolidation and synchronization project to enable integration of Enterprise data
• Developed packages for data from 4 legal entities into one ERP system's consolidation base using MDS staging and bringing back the direct ERP data of Dynamic AX.
• Analyzed and designed 4 usecases of SSIS workflows which made use of Windows Communication Foundation WCF Webservices in both directions while staging data in MDS Database and integration services of SQL server 2008 R2 and 2012 as data workflow engine.
• Managed the data stores interfaces of Oracle, SQL Server.
• Managed models, entities, attribute (domain attributes), used modelling capability along with version management in MDS. Defined access levels for entity data and hierarchies.
• Quality metrics maintained for SSIS packages at acceptable production level
• Our work recognized for fostering ultimate analytic by2 months period earlier and more risk free. Utilized the medium capacity model to improve upon the space.