I have been asked following set of questions for Big Data engineer position Please find below all questions Forgive my spelling mistakes, hope you got an idea. Spark: Working with Partitions, Memory size selection Choosing Executor memory? Optimize it? which parameters to consider, selecting block size of HDFS? Optimize Spark Job? What steps needed? Updating 3 copies of a file at once? deciding partition size/number of partitions? while configureing spark cluster, what parameters to fix? have you worked with joins in Spark? how many of them and thier purpose? what is implict in Spark? usage? while shuffle and sort? how can you improve preformance? Diff betwen Functional and OOP language? any 3? What are serialzable objects? are they on heap or off heap? What is CDC, change data capture, give example SCD ? how many types of SCD and thier purpose? What are ACID properties? what is A ? what is C, I and D? scala uses, which design pattern? what is singleton pattern and its example Rank Vs Dense Rank Monarch functions how to resolve skewness of a query/table what things to condsider while Scaling up system? horizontal scaling? linux, Crontab difference btw Map Vs Flatmap
