Although it seems as if we need to use the negative form use to, the second clause is actually an affirmative sentence: I used to be.... We can only use would for an action in the past, not a state in the past.