So sad. What you are experiencing is "normal" in the sense of "not unexpected".
I imagine these kids have been through hell, as some of the behaviors you describe seem almost feral. They need stability. And at least one good parent.
Sorry, but it sounds like your brother is essentially MIA here. Does he know what he's doing? That is, is he a parent or merely the paycheck? Is the mother still living at home?
It is common for the kids to want to protect the weaker parent, in this case the mother. Even is she has abused them (physically, emotionally, etc.), they'll miss her and want her. They probably need counseling to help process their inappropriate home life.
It can get better.
Your rule of action is Slow, Calm, Stable.
It doesn't matter if they say hurtful things. That is what they have been taught. You are the Catholic adult, and you see the bigger picture.
Tell them you love them (without condition). But don't expect an appropriate response, because they don't understand love.
If they are crying, let them cry. They need it to process their grief. You might be able to help them understand it by asking if they need to cry. (Usual answer: yes.) You might be able to help them control it by asking if 5 minutes (or even 10 minutes) will be enough. Tell them "OK. Then cry as hard as you want for 5 minutes." They should stop well before that.
Use "time outs" if they get out of hand. Time outs should be in a nearby area, but one that has little sensory stimuli (away from the others, on the stairs, or in a hallway, or on a chair, but no TV, no toys, no books, etc.). I'm not suggesting extreme isolation or sensory deprivation, but just pick a quiet place. The general rule of thumb is 30 seconds to 1 minute for every year of age, but you may adjust that depending on the situation. They can come out of "time out" early if they promise to behave. Back into "time out" for a longer period if they don't.
Be clear on roles. Their "job" is to be children. Your "job" is to be the adult.
The oldest is messed up because the parent-child roles in her life are messed up. It would not be surprising if some aspects of her emotional development are stuck at an earlier age due to ongoing trauma at home. You cannot force her to develop faster than she is able. Let her be a child. Let her keep her blanket and teddy. They represent stability and comfort; things that have probably been sorely lacking in her life. Counseling is probably necessary to help with her emotional development. See if a local Catholic Charities center offers such.
Your "job authority" might be enhanced if your brother is willing to tell his kids (with some authority?), and with you next to him, that they need to follow your directions while they are with you. Clarity helps. Equivocation doesn't.
Give them some light responsibilities (e.g., putting away toys, setting the table, clearing the table). See how they handle it. Time outs and/or loss of privileges if they don't.
You might be able to get the older one involved in menu planning (where you provide a number of healthy options and she makes a choice and maybe even helps out). If she doesn't want to eat dinner, fine. But no dessert. No snacks.
Car trick: books on tape (or CD, or MP3, or whatever). There really are some marvelous and complicated stories that are child appropriate (Black Beauty, Treasure Island, Charlotte's Web, etc.). Turn up the volume, if necessary. Or try a child-safe sing-along collection. Veggie-tales has several. The old Disney tunes are safe.
Car trick: Refocus, redirect. Get their attention, and then ask a question of the troublemaker. How was daycare? Do you need to go to the bathroom? Can you tell me how much is 3x7? How many hamburgers can you eat? How many hours until you see your dad? How many minutes is that? Whatever works. You want to get their focus off the fight and redirected to something safer, quieter.
Also, put the troublemaker in the front seat with you if it is safe to do so.
Little successes will soon become big successes.
Share what works with your brother. Unfortunately, any progress you make by the end of summer will likely be lost if the role of parent is not properly filled in their home.
Bless you for doing the hard work that these kids need.