Engineering at GS: Overcoming the Challenges of Legacy Systems

Iza and Damian work as software engineers in Post Trade Equities Enricher in Global Banking and Markets. Iza, an associate software engineer, first joined the firm as a summer intern and transitioned to a full-time role in 2020. With expertise as a full stack developer, she focuses on creating and refining tools that enhance the platform’s capabilities. Damian, a vice president, became part of the team in 2021 and has since been involved in building the core backend components that power the new platform’s infrastructure.

Legacy systems are a common challenge in the fast-paced world of business technology. Our team's equities processing system, initially built to enrich and handle order and execution data, was reliable in its early years. However, it was not designed with the flexibility required to keep pace with the exponential growth of data as the business expanded. As the volume of data surged and complexity of business requirements increased, the system required an uplift. The older technology was inflexible and struggled with scaling efficiently, revealing performance bottlenecks and making it evident that a new approach was necessary for sustaining future growth.

Recognizing these issues, we made a strategic decision to develop a new, modern framework. Drawing inspiration from an existing solution for distributed stateful stream processing, initially tailored for a specific use case, we embarked on a journey to generalize and build upon this concept. The goal was to create a system capable of handling complex data transformations and large-scale transactional workloads while maintaining flexibility and robustness. This decision was pivotal, as it positioned us to address the underlying challenges of the legacy system and embrace a more adaptive, forward-looking approach.

The core function of the legacy system—data transformation—was fragmented, hard to debug, and dependent on outdated technologies. Our new framework, developed in Java, adopts a declarative Model-to-Model transformation approach. This shift has made the system cleaner, more understandable, and easier for developers to work with. Additionally, the framework is versatile because transformations are separated from runtime technology, allowing it to be run on various streaming execution platforms such as Apache Flink and Apache Spark. Robust exception handling and replay mechanisms were implemented for SRE. To support this, we built a suite of tools: React and TypeScript web interfaces supported by Spring Boot APIs for exception management, a graph-based lineage visualization of fields, and centralized monitoring tools to observe distributed pipelines.

The new framework brought enhanced flexibility and scalability, critical for adapting to evolving business demands. It provided low-latency, real-time data processing capabilities with high throughput, ensuring the system could manage surges in transaction volumes. This resiliency also proved invaluable in handling market volatility, where rapid data processing and error handling are crucial. Our business operations benefited from improved speed and stability, creating a foundation ready for future advancements.

For our engineering team, this project was more than just an upgrade; it represented significant growth and opportunity. Working on this project exposed developers to cutting-edge technologies and presented a platform for continuous learning and exploration. Our voices were valued in decision-making, empowering us to contribute ideas and enhancements that shaped the final product. The company’s commitment to fostering development is evident, providing access to learning platforms like O'Reilly and Pluralsight, which reinforce our engineering tenet: to keep learning and evolving. With such projects and support, engineering life here is vibrant and inspiring—a space where our skills grow as rapidly as the technology we build. And, of course, the breathtaking view from the office isn’t too bad either.

 

Read this blog post in Polish.