The correct answers are I tried restarting the computer, but it didn't help and I tried to turn it off, but the button was stuck.

Try + gerund (-ing) = experiment with a method to see if it works. Try + to-infinitive = attempt / make an effort (often implying difficulty). When you successfully restart but it doesn't solve the problem, use the gerund. When you can't even complete the action, use the infinitive!