That is somewhat difficult. Probably the best approach would be for you to sign a resignation of your role and provide that to your brother so he can act without you. Another alternative would be for you to delegate the role solely to your brother so that you could take back the mantle if anything happened to your brother and he could no longer act under the durable power of attorney.
Harry S. Margolis practices elder law, estate, and special needs planning in Boston and Wellesley, Massachusetts. He is the founder of ElderLawAnswers.com and answers consumer questions about estate planning issues here and at AskHarry.info.