The correct answers are: He must have been sleeping soundly to completely miss the sound of shattering glass. He should have been patrolling the main gallery at the exact time the thief entered. He might have been listening to loud music on his headphones, which would explain a lot.
Must/might have been + verb-ing is used to make deductions about continuous actions happening at a specific time in the past.
Should have been + verb-ing expresses an unfulfilled expectation or obligation—something that was supposed to be happening in the past but wasn't.
Why the others are wrong:
Modals must always be followed by the base form of the auxiliary verb (have, not had). Additionally, the active past continuous form requires a present participle (patrolling, not patrolled).