We previously wrote about Scale By the Bay's long history with Spotify and how much we adore and respect this brand not only because it's one of the most beloved brands in the world but also a data powerhouse that helps hundreds of millions around the world listen to their favorite music daily.
At Scale By the Bay 2019, Spotify both is present as a sponsor and its team shares the insights on eliminating surprises in data. In advance of this highly anticipated talk by Anne and Idrees Khan, we caught up with Idrees to get his perspective on what it's like to work at Spotify and the exciting things he and Anne will be sharing at Scale By the Bay. Read on!
How did you get interested in Data Infrastructure? What do you find to be the most exciting and interesting about it?
I previously worked at a couple of small companies in Canada on teams focused on Data Engineering, Analytics, and Data Science. While there, I had a variety of responsibilities there which fell into buckets of either Data Engineering or Data Infrastructure. In addition, I have always had a lingering interest is building generic tools and systems which solve issues across a wide problem space. These two things combined created a natural progression towards Data Infrastructure. Working in the data infrastructure space is exciting because I get to solve some complex problems, provide value to co-workers, and work with some interesting technologies along the way.
You are a Senior Data infrastructure Engineer at Spotify: can you please tell us more about your role and what exciting things you are working on at the moment?
At Spotify I work in the Data Infrastructure department on a team which focuses on data quality. We build a variety of tools for other engineers to help them detect, troubleshoot, or avoid quality issues. Right now we have a core set of tools which have proven to be useful, and we are working on democratizing those tools to make it easier for additional employees and engineers in the company to utilize them. This is exciting because we are working towards integrating what we have built with core parts of Spotify’s data processing, and onboarding new users. And it’s always exciting to see others derive value from your work.
What do you enjoy most about your work?
The engineering culture at Spotify supports a lot of grassroots style work, which is enjoyable as someone that tries to be very self-driven. It creates a lot of opportunities to influence work on your team or in the organization as a whole.
Do you find that Spotify has a different approach to Data Infrastructure Engineering? Can you tell us more about it?
The work culture at Spotify is very decentralized. Though it varies in different areas of the company, we typically have cross-functional teams which focus on a specific feature, area, or product. This means that our approach to Data Infrastructure Engineering needs to not only support a wide variety of use cases, but also focus on making things as easy as possible for our customers (Spotify Engineers). It’s important to make things easy so that we can develop standardization across the company and avoid fragmentation.
What are the most surprising learnings/discoveries you can share that stem from the work you have been doing?
Sometimes the most boring thing can be the most useful. I still think there can be tonnes of value provided by building or providing complex tools, but I think simplicity is often underrated in software development.
What are the key challenges that you think developers working on Data Infrastructure face and what are the best ways to address these challenges?
I think data infrastructure has not yet matured as much as other disciplines, which means there is a lot of opportunity for developers in the space. However it can also be difficult to build the right thing in an area that keeps changing. I think it’s important to have a good understanding of what others are doing in the space, as well as having a close connection with your intended users. Otherwise, you risk building something that provides little to no value.
Who should attend your Scale By the Bay talk and why?
Anyone who is interested in understanding or detecting data quality issues or avoiding pitfalls in the data pipeline development lifecycle.
Whom would you like to connect with at the conference?
Anyone who would like to share their own experiences in the data quality space or who has a shared interest in RPGs, surfing, travelling, or food!
Anything else you'd like to add?
If you have anything you’d like to discuss regarding data quality, data infrastructure, or Scala in general feel free to reach out to me on Twitter @idreesxkhan or via GitHub idreeskhan.
Don't miss Idrees Khan and Anne DeCusatis at Scale By the Bay in Oakland this November. Book your ticket now.
Kommentarer