In an exclusive interview with Shadaj Laddad, a third-year PhD student at UC Berkeley, we delve into the intricacies of software engineering in academia, the challenges and rewards of open-source contributions, and the groundbreaking work on Hydroflow—a project aimed at optimizing distributed systems. Shadaj, who began programming at the age of 6 and later found his passion in Scala, shares his journey from being a hobbyist programmer to contributing to startups and finally diving deep into academic research.
So, dive into the ocean of insights and learn how we can contribute to Hydroflow and be a part of this revolutionary project aimed at redefining the landscape of distributed systems.
1. Let's start with your background. Can you tell us about your career journey so far?
I got my start programming pretty early, when I was 6, but I started seriously building systems a bit later when I got introduced to Scala, around 2012. From there, I was primarily a hobbyist programmer, building games and simple websites (using then cutting-edge technologies like Scala.js!), which eventually led into several internships at startups like Coursera and Apollo GraphQL where I started focusing more on distributed systems and developer tools. That and my experience contributing to open-source projects like Scala Native got me quite excited about working on programming languages, which I decided to pursue in a PhD program at UC Berkeley. I'm now in the third year of my PhD, where I've been working on programming languages and optimizers for distributed systems as part of the Hydro project! 2. What potential downsides need to be considered and addressed when developing new technologies in software engineering in academia?
The biggest challenge with developing systems in academia is the disconnect between the goals of a research project and the needs of real-world developers. It's often impossible to satisfy both in a single project, since academia is more interested in identifying elegant solutions and rigorous proofs/evaluations while industry is looking for easy-to-adopt solutions that can provide an immediate business benefit. Academic projects tend to introduce a lot of new concepts, which ends up hurting the user experience since developers now have to study papers to understand how to use your library. On the other hand, focusing too much on industrial use cases can lock your work down to incremental improvements and limits the discovery of paradigm shifts. To be fair, my own research is stuck between these two worlds, as it is grounded in academic foundations but focused on delivering meaningful improvements to real-world developers. But I've found that a tick-tock model of developing core research and then working on making it accessible to a wider audience has helped balance these two sides. 3. Tell us about Hydroflow. What problems is it solving? How does one contribute to it?
Hydroflow is the foundation of the Hydro programming stack, and provides a low-level IR for programming distributed dataflow systems with an optimizer that aims to automatically discover more scalable versions of your logic and a type system that helps ensure that developers guard against sources of non-determinism in a distributed system. The second piece in particular builds on a lot of past research from our group on ways to reason about "application consistency" -- making sure that your business logic executes correctly regardless of failures or network latency. We are applying many theoretical results such as the CALM (Consistency As Logical Monotonicity) theorem to ensure that Hydroflow programs can be automatically optimized without impacting the correctness of the underlying logic. We've even used Hydroflow to auto-optimize protocols like Paxos, since it can rewrite general distributed protocols without needing users to reason about the correctness of the transformations. The hope is that by programming backend logic in Hydroflow, developers can more efficiently utilize the availability of cloud-scale compute by focusing on their business logic and letting our optimizer figure out how to distribute it. Hydroflow is open-source at https://github.com/hydro-project/hydroflow and we encourage anyone interested to contribute optimization opportunities, challenging protocols, and new deployment targets! 4. You are a prolific OSS contributor, on your website, you say: "I explore the boundaries of programming". In your opinion, how important OSS is for pushing the boundaries of programming?
OSS software has a unique role in the development of new systems, especially those that affect how other developers build their own applications. While industry often leads in the development of systems with higher performance and cost efficiency, these systems are often specialized for specific use cases and deployment scenarios. The beauty of open-source is that it forces the system developers to create the abstractions necessary for a wide audience to use the system, and opens up the software to contributions from the community that increase its flexibility by aggregating use cases from around the world. I think for modern systems, being open-source (or at least source-available) is quite critical if you want adoption from a wide range of developers. It naturally creates a community of people excited about the technology and making it more accessible, which is the first step towards stretching the boundaries of what we can do with programming!
If you've found this conversation with Shadaj Laddad as enlightening as we have, you won't want to miss the chance to learn even more. Shadaj will be speaking at the upcoming SBTB conference this November. It's a golden opportunity to delve further into the intricacies of Hydroflow, distributed systems, and the intersection of academia and industry. You'll even have the chance to engage with Shadaj personally, asking questions and discussing the future of programming languages and distributed systems.
As a special offer for our readers, if you share this blog post on your social media platforms and send an email with the subject "Shadaj" and the link to your post to sbtb@konfy.care, you'll receive a 15% discount on your SBTB ticket.
Don't miss this unique opportunity to be at the forefront of revolutionary changes in the software engineering landscape.
Comments