Optimization of Big Data Processing

Customer: SwissRe

Tasks:

• SwissRe has a very big collection of data from varous sources. That data would be useless without the ability to analyze it, and extract intelligence from it. Therefore SwissRe has a sophisticated data processing pipeline built with PySpark on DataBricks, running in an Azure cloud environment. Though that is a particularly powerfull data processing environment, the process still took over eight hours for each run. To improve the experience of the users, the goal was to speed up that process significantly.
• Usually only data from the current quarter, and sometimes the previous, or the next are changed. I implemented the necessary logic to detect which quarters are affected, and process only those. I also took some measurements, to optimize the cluster size, and the node size for the specific workload. Apart from only performance oriented optimizations I also developed several approaches to improve the over all code quality.
• Together with some changes, other team members implemented in parallel, my changes reduced the processing time to around two hours.

Skills:

Python, PySpark, Azure, DataBricks

Wealth Management

Customer: Credit Suisse

• Many Credit Suisse customers seek advice for their wealth management. To support them, we developed a portfolio optimization system that uses mathematical optimizations with high dimensional functions to determine an optimal investment strategy.
• We developed a microservice system for reliability and scalability. I was responsible for some of the more mathematically challenging quantitative computing services, as well as for the framework and deployment infrastructure for those services written in Python.
• The project was successfully delivered.

Skills:

Python, CVXpy, numPy, FastAPI, Microservices, OpenShift

BI system

Customer: PPRO

Tasks:

• PPRO had a lot of data spread over several systems and aspired to analyze it in order to improve its marketing and product development decisions.
• As technical lead, I set up a data warehouse with my team in PostgreSQL because PostgreSQL allows to leverage remote tables from other database systems such as Cassandra. This in turn allowed for a smooth transition at a later date when PostgreSQL was no longer sufficient. For scheduling, I decided to use Azkaban, and the collection of data from all the systems was done with Pentahoo Kettle.
• The database had three schemata: a staging area, the data warehouse itself, and several data marts. With that setup we were able to feed Tableau Dashboards for decision support and allow the data scientists to run their data mining without disturbing the production processes.
• The BI system, as a data-driven decision support tool, was highly regarded by management.

Skills:

PostgreSQL, Azkaban, Pentahoo Kettle, Tableau

Sanction management

Customer: PPRO

Tasks:

• As a financial institution, PPRO is obliged to obey the sanction regimes in all jurisdictions it is operating in. The authorities distribute the list of sanctioned people as XML files, which contain the sometimes sparse information available, to identify sanctioned persons, or organizations.
• In this project I developed not only the import and update of these sanction lists, but also the matching of customer requests to entries on these lists. The result was a risk score, which reflected the probability that the customer might match an entry on one of the lists. With that risk score, the system either decided to allow, or deny the customer, or to delegate the decision to a human in unclear cases.
• The system was applied to every new customer, and regularly to the whole customer base. For the customer base, I stored a list of human decisions, so that we didn?t have human staff members decide again.
• With this decision support system, PPRO was able to reduce the work load on the staff, while at the same time reducing the risk of failing to comply with the sanction regimes.

Skills:

Python, XML, MySQL

Direct Debit Payment System

Customer: PPRO

Tasks:

• PPRO decided to develop a payment service for the German market to complete their portfolio. The idea was that the user would enter his login for his online banking, and then the service would trigger a transaction to the shop. Since the service could view historical transactions, we were able to tell the shop, if he can be sure that the money is coming, or if he should wait for it to arrive at his account.
• Together with a second developer, I analyzed the online banking systems of the biggest banks, banking conglomerates, and online banking providers for small banks. That was enough to cover over 90% of all German bank customers.
• Since my coworker and I implemented this system in a short time, PPRO was able to improve its position in the market significantly.

Skills:

Python, Flask, BeautifulSoup

Big Data Low Latency Complex Event Processing System

Customer: SIX Telekurs

• SIX Telekurs (now SIX Financial Information) is one of only a few companies that can provide marked data from almost every exchange around the world to customers in a standardized format with low latency. The goal of this project was to replace the central market data processing system with a decentralized one, for even lower latency.
• We were able to leverage the domain-specific language for parser generators from the existing system. These parsers had to interpret the different stream formats from the different exchanges, and therefore there were several hundreds of them already implemented.
• Not only have I written big parts of the code. I was also responsible for the design of the main components and, together with the technical lead, the main architecture decisions. We developed an asynchronous system that used a reliable multicast protocol (LBM by 29West) for hot failover as well as for delivery to the customer. The code had to be very close to the bare metal, considering aspects like the NUMA structure, memory layout and cache strategies, and user space direct network card access to reduce the number of context switches. The applications I designed basically took over most of the machine from the operating system with CPU binding and CPU shielding. The whole system was designed lock-free to avoid any unnecessary latency, and handled all issues that were not part of the processing itself asynchronously, like e.g. logging, or metrics collection. To give the compiler the best chances to optimize the code, I relied on the most modern C++ techniques of the time, like template meta programming.
• While I was on the project, we managed to successfully deploy a prototype to production.

Skills:

C++, Template Metaprogramming, NUMA aware, Lock Free, Linux, Reliable IP Multicast, Parser Generators