What is datbricks used for and can we use indexes on delta table What is dag and lineage graph
Senior Data Engineer Interview Questions
2,556 senior data engineer interview questions shared by candidates
It's a nicely designed panel where all the questions are around the same ETL pipeline. The question is asked in a confusing way so try to focus on the specific asks. In the sys design part, they assess the depth of your understanding of the process while asking you to design an ETL pipeline. You'd want to revise concepts such as CDC, queues, batch and stream processing frameworks and things you can do when pipelines fail. In the coding part, you're asked to implement the transform part of the pipeline. You have to aggregate the data together and transform it etc. It's no algorithmic challenge, it's about how comfortable you are with python data structures to manipulate a whole bunch of data very quickly during an interview. You start with JSON input and massage the dataset to get it into the right shape. Finally the SQL interview- you're asked to come up with and write several queries to check whether the data was copied properly. You'd want to do checks like row count checks etc
We receive data in a certain format [provided example]. How would you write a query that shows a full dimensional history, slowly changing dimension style?
Q1) How about your past data engineering experience?
Explain spark optimizations which you have worked on?
What are your strengths and weaknesses as a software engineer?
As per test assignment. Showing, that the interviewers have not been prepared to the interview
Did not ask any questions at all.
Modeling types Snowflake Star schema
Present how data can help the council.
Viewing 1731 - 1740 interview questions