Tell me about a time where you had to adapt to an unexpected change
Senior Data Engineer Interview Questions
2,562 senior data engineer interview questions shared by candidates
Context Given a night with X ride requests and Y available drivers in a fictional city, you have to develop a batch processing application that would aggregate and expose data coming from the matching engine. Every Z seconds, the matching engine tries to match every pair of request and driver that are available in the city. Some are matched, some are not. The results of each matching tick are stored in a set of files. Given this data we want to be able to get a overview of the marketplace health and multiple applications could follow such as heat maps. The metrics that we want to use for heat mapping are driver match rate and request match rate. Exercice Develop a microservice that will: Aggregate matching data by fetching new matching data and exploiting this data in order to aggregate driver match rates and request match rates by geo-spatial units of your choice. Expose an endpoint which return the adjusted values of request and driver match rate, following this formula.. Bonus Plot the driver and request match rate for one night
Technical - you need to be strong in fundamentals. Syntax is not important Problem solving/ solutioning - approach is important. Not syntax Managerial - You should be able to articulate your projects well. Project goals, scope, value ads are important - not technologies Behavioral - Be transparent
In First Round Mostly questions related to basics of spark and Hadoop questions. In Second Round Question related to optimization and fine tuning of spark jobs, Skew Joins
Topics: 1. SQL- Window functions 2. Bigdata Ecosystem 3. Design questions (ETL design)
Design pipeline that is relevant to the company
Tell me the time when you had worked with SQL
I had 4 rounds of interview.
Kimbal model types AWS Python SQL
How do you automate your day to day tasks as part of Data Platform team?
Viewing 1491 - 1500 interview questions