One point I answered was "reduce Tensor Parallel (TP) unless it is really necessary because TP requires all-reduce and all-gather operations, which can be very expensive." The interviewer was unwilling to accept this answer and looked like suspicious. However, what I said was correct, as mentioned in a recent Nvidia Post on serving the Llama 3.1 405B model. Apparently, the interviewer might not know the details of tensor parallel.