Methodologies used: SCRUM

• Designed and built data model for big data software application handling the billing of billions of items for German governmental health insurance service
  

• • Translation of functional requirements into tech. requirements & data model

  • conceptualizing of in-memory data structure to handle billions of entries with high throughput

  • selected suitable tech. components and built tech. stack for software solution

  • Implemented and ran benchmarks for different processing and workflow control technologies (Camnuda 8, Spring Statemachine, etc.)

  • Evaluation of performance tests for millions of data points to validate performance of tech. stack

• Software design of microservice in-memory application and proof of concept implementation

• Data architecture and persistence of historical data

• Implementation & proof of concepts with different technologies, measuring of access times to evaluate if solution would work on hundreds of millions of data points.

Duration: ongoing (started in August 2022)

Role: Data architect, software architect for big data application solution

Size of team: 4 software developers, 10+ stakeholders representing functional requirements

Link(s): not public, internal software

Technologies used: Java 17, Quarkus Framework, Camunda 8 Workflow Engine, OpenAPI, Junit 5, Hazelcast InMemory Grid, Grafana, Apache Kafka, Spring Statemachine Kubernetes, Helm, Docker, Confluence

Methodologies used: SCRUM

• Worked on concept of data model for new software application handling billing of German governmental health insurance service

  • Translation of functional requirements into data model and suitable tech. Stack for software solution

  • Comparison of different technologies for processing and workflow control

• • Software design of in memory application and proof of concept implementation

  • Data architecture and persistence of historical data, microservice architecture

  • Implementation of proof of concepts with different technologies, measuring of access times to evaluate if solution would work on hundreds of millions of data points.

Methodologies used: -

• Optimizing performance for HBase access of complex data structures

  • identifying performance bottlenecks

  • optimizing access patterns and data processing flows

  • using fastutil big data performance library to reduce memory footprint and shorten run time of spark jobs processing of millions of data rows

• Optimizing existing Spark jobs to reduce run time and memory footprint by 50%

• Implementing Spark jobs accessing HBase data structures and Kafka messaging systems to meet functional requirements

• Exploration of source data: Planning and designing a data pipeline solution to clean and process massive data-set (~40 billion data points per day) to form a graph structure for easy clustering of nodes
• Building a scalable java application (Framework: Spring Boot) to build up a graph structure with (weighted) edges between vertices/nodes
  • Main challenge is to build up graph structure in memory from massive data-set (50+ Gigabytes per day)
  • Graph traversal has super-fast to identify known nodes when a new event is analysed only in memory solution is feasible ? only in memory solution is feasible
  • Implementation of solution on premise at customer?s data center (running Hortonworks distribution 2.6 and later newer 3.1)
  • Development of algorithms handling ~700 GB of in memory data
  • Handling fail-proof connection to HDFS data lake, ORC files on HDFS contain source data
  • Exporting (sub-)graph structures in different formats for ad targeting engine(s) of international media companies
  • Continuously measuring and evaluating graph quality based on heuristics since no labeled data exists for given real-world data
  • Technology was used to run several large ad campaigns
• Visualization of data for insight generation for marketing department
• Git workflow; merge-requests signed off by team members
• setup and use of CI/CD pipeline
• TDD ? test driven development based on JUnit for core algorithms
• Size of Hadoop cluster: 20 nodes with 300 TB HDFS storage and 1.2 TB of memory running Hortonworks HDP 2.6

• Migrating multiple complex Apache Hive and Apache Spark jobs from a 20 node HDP 2.6 cluster with a ~300 TB HDFS storage to a new HDP 3.1.4.0 cluster of similar size
  • Transferring tables from HDP 2.6 to new cluster and setting up new ACID tables in HDP 3.1
  • Migrating complex Apache Spark jobs with multiple stages to use data warehouse connector of HDP-3.1.4.0
  • Migrating Airflow jobs and DAGs
  • Adjusting ORC file formats of tables on new cluster
  • Troubleshooting, performance tracking and improvements, Kerberos configuration adjustments for 3rd party applications
  • Monitoring of Services (Apache Ambari) and health checking
  • Configuration adjustments on HDP 3.1 cluster to make sure all jobs are running
  • Verification and validation of migrated data after transfer to HDP 3.1 cluster

• Maintaining and Extending a Java Enterprise web-application (Java EE 6, Hibernate) with Adobe Flex Frontend (based on Adobe Flash technology)
• Performance tuning of Glassfish application server and ORM database model (removal of ineffective triggers and indexes)
• Migrating the application from Java 6 to Java 8 JVM for security and performance reasons
• Extending the rights and permissions functionality to allow ?shared? editors to work in collaboration across multiple editorial offices (Spring, Hibernate, Database)
• Implementation of 3 management dashboard views ? realized as Angular JS single page applications ? which show real-time information about planned media elements on big screens in different editorial offices using kiosk mode
• DevOps: developing, testing, integrating, deploying in production and monitoring the application

• Built web application (Spring Boot Framework) for the ad sales and media research department which visualizes historical data and shows forecasts of estimated reach of ad campaigns

  • help ad customers to make decisions about their campaign bookings

  • supported decision making process and optimized ad-campaigns and increase ad-revenue

  • get reach estimations based on tv-show or tv-ad criteria

  • identify uplift potential for increased ad-revenue

• Built Apache Spark (v2.3) Scala (v2.11) program to per-aggregate data using bloom filters with open source java library (Twitter Algebird)

• Storing pre-aggregated data in ClickHouse OLAP DB for fast access

• Java web application accesses aggregated data in ClickHouse SQL database for HTML aggregation and display

• Migration of existing website using CoreMedia Government Site Builder technology to CoreMedia 9 Blueprint

• Development of Freemarker newsletter templates for bundesregierung.de

• Development of Spring Boot application for validation of migrated newsletter content between old production system and newly developed system

• Customization of Mailman 2 (Python); migrating custom extension database for newsletter delivery from Oracle DB to MySQL DB; integrating legacy application via REST microservice into CoreMedia ecosystem

• Unit testing of Spring Boot microservices with Junit 5 and H2 in-memory database engine

• Integration of Docker containers to extend CoreMedia functionality (PDF generation with pdfreactor)

• Implementation of backend search functionality (Apache SOLR based)

• Frontend development:
  • development of a very large and complex Single Page Application with AngularJS framework and TypeScript, FLUX
  • Implementation of Angular Services, Directives, Controllers, Templates, Filters, Data Binding, Dependency Injection, Modules etc.
• Backend development:
  • extension of backend microservice API (C# / .NET 4.6), usage of Entity Framework and Migrations for MSSQL Database access and updates, Elasticsearch was integrated as search engine
  • Backend and Frontend communicate via REST calls. Backend component is implemented using microservice architecture.
• Implementation of windows server services which retrieve, clean and process data from different sources (DB of 3rd party systems, serial connections, XML files)

[on request] is a free platform where friends and family can organize their joint gifts. We have done much development offshore. I had to write code for certain parts myself which would haven taken to much time by the offshore team. The product was developed with private money with the help of an offshore team.

• Development of a Java Spring web-application which was integrated into the CoreMedia Government Site Builder technology
• Database, ORM and OO-Architecture (DAO, MVC), design of software interfaces, ensuring interoperability between a multitude of different software components, Usage of JPA Criteria API
• Implementation of core business logic
• Implementation of full-text-search based on Oracle Text technology
• SCRUM Agile development methodology used throughout entire project
• Testing and integration of software components into CoreMedia production system

The Android app wraps around the responsive gift blog and adds a home screen (branding), local notifications (for user retention) and native sharing functionality (message distribution). The app is available for free from the Google Play Store to increase brand reach.

I wrote an E-Book about “Product Design Sprints” (German) - a methodology to validate innovative ideas for new or existing products within 5 days. The process was originally invented by Google Ventures and used for startups within their own portfolio. The patterns can be used to validate a great variety of product ideas and improvements leading to the creation of a prototype which allows getting valuable direct customer feedback after only one workweek. In the book, the whole process is covered and illustrated with examples. 

Phase I – Implementation (08/2013 – 12/2013): build and launch fitmio.de

• Setting up complete development environment (ticket system - redmine) and tools for collaboration
• Responsible for research, planning, technical evaluation and building the hosting infrastructure for the fitmio.de project
• Contract negotiations with multiple hosting partners
• Responsible for technical evaluation and software platform decisions for fitmio.de project
• Responsible for software testing during development, lead through alpha and beta test phase with first customers
• Responsible for KPI metrics and tracking concept, implementation of internal (Adobe Site Catalyst, former Omniture) and external tracking components (Google Analytics), testing
• Concept and implementation of fitmio.de video hosting
• Concept and implementation of newsletter mailing solution for fitmio.de project
• Research, selection and integration of suited affiliate partner tracking software

Phase II – B2C focus (2014): implementation of additional features, tracking data aggregation

• Developed software features and tools for back office processes for fitmio.de
• Tracking of user behavior and Data Mining on current data to learn more about users needs and develop features accordingly

Phase III – B2B focus (since 02/2015): fitmio.de is now sold to other companies part of their company health offering to their employees

• Management of software development contractors for backend (Hosting, PHP, Symfony2) and frontend (HTML5, CSS3, JS)
• Agile development process supervision
• Responsible for operations & monitoring of infrastructure
• Test & Release management
• Emergency bug fixes

• Karma King is a mobile app to encourage people to share their good deeds done by themselves everyday with others (preferably on facebook) in order to gain karma points. If their facebook friends “like” their good deeds they are awarded additional karma points and compete to become the Karma King.
• The project aimed at creating a marketing tool with a positive “viral coefficient”. Meaning if you can get enough people to share their good deeds on facebook, this will get more people into 1) seeing those and 2) downloading and using the app. These new users will in turn share more good deeds and bring in even more downloads.
• A positive viral coefficient means that for every app user, you will get more than one additional user – resulting in hyper growth which is the ultimate form of marketing (cheap and perfectly scalable).
• Technically, the project consists of two apps (Android/ Java and iOS / Objective-C) for the major mobile platforms and a backend component (PHP, MySQL) for storing good deeds, user data and karma points.

• Development of feature backlog with customers (local sports newspaper editors)
• building an SOAP interface for XML content transmission between the DuMont CoreMedia Java CMS and the [on request] partner company
• Technology used was Java Spring MVC framework
• Responsible for software architecture and implementation of the FuPa.net exporter with Java Spring technology
• SCRUM Agile development methodology used throughout entire project
• Testing and integration of software into CoreMedia production system

• Laying out detailed plans and options for a project that will consume about 1000 man-days of work – the plan was accepted and the project started in January 2013
• Comparing architectures of two mayor complex CMS systems aiming towards the development of an optimal migration strategy
• Development of small (Java) tools, mostly database driven, which were needed to migrate from CoreMedia CMS 2008 to CoreMedia CMS Version 7 (released in 2012)
• Migration of user comments from classical relational database design to MongoDB schema-free database

Role:

software architect, CoreMedia software developer, Full-Stack development, systems administrator / DevOps

[on request] is a small company that offers small businesses very high quality and responsive web design for desktop PCs, tablets and smartphones. This is possible because we have invented a very lean process to provide customers with HTML templates created in advance by designers. We then fixed their pages online, while customers watched via screen sharing and could instruct changes.

• Goal of the project “Innovationsmanagement Technology”: integrate technically motivated innovations into the mainline company development processes
• Responsible for creation and implementation of a process to bring forward technical ideas for innovation within the development team

• Team leadership: conduction of job-interviews, coordination and technical supervision of multiple software development projects
• support and advice for developers and bringing new colleagues up to productivity
• Leadership of the team for maintenance tasks (refactoring, small features, increase test coverage...)
• Proofreading of requirements from other departments or clients, laying out architecture and plans for components to be implemented by the development team
• Coordination and development of software components with external contractors or service providers
• Coordination of content delivery to the central DuMont CoreMedia Java CMS via XML interfaces

• Responsible for flawless transition of mission-critical CoreMedia Java CMS system (hosting 5 newspaper websites at that time, 3 were among the top 10 German newspaper websites) to off-site data center without downtime or data loss.
• Creating architecture for transition and evaluation of different migration strategies – the most suited one was later selected for the execution of the transition
• Total data which had to be kept constantly in sync between two data centers was close to 2 Terabytes.
• Lots of BASH scripting for testing and automating migration of services and servers, night shift work (because of zero downtime requirement) on oracle based databases
• Build up Apache SOLR replication cluster for smooth migration of all search servers