Yes, get rid of it or do a thorough search. Our family got a tablet as a christmas from an inactive family member. My tbm inlaws got to it first and downloaded their mormon scriptures onto it. Mormons will stoop to all kinds of dishonest behavior to indoctrinate people.
It's not "church issued" it's a gift from someone who doesn't know how to give gifts.
Did they transfer your phone number to the new phone? I say unless you have no need, keep it and do a reset on the phone to put it back to factory settings.
If you don't know what to do, take it to the nearest phone store and ask them how to find out exactly what the deal is. At the very least they will direct you to someone who will help you find out what the phone settings are.