Land issue

Hai, Respected Sir/Madam, My grandfather got 4 acres land as a gift from his father and 4 acres land for his brother too, one 4 acres land is near to village and other 4 acers land is 1KM from village, they divided land and cultivated jointly but due to frequent fightings and arguments my grandfather's brother only decided to take total 4 acres Land which is 1KM far from village in which 2 acres my grandfather have registration and 2 acres on his brothers name. In the same he gave 4 acres of land near to village to my grandfather in which my grandfather have 2 acres registration and his brother have 2 acres registration. They written one agreement like if anybody wants again to change their decision and opposes they need to pay penalty of 50000 rupees this agreement was done before 23 years. Everything is going well and good till now we were farming on the same land without any issues, but now again is arguing with us to return back his 2 acres land which is near to village in our part and saying to take 2 acres land which is 1KM far from village. Can you pls help with legal advice to not to return his 2 acres because if we return our 2 acres and take their 2 acres land then again fightings will happen and they will not let us leave happily.