Is it hard to survive IPM IIM Indore for someone who's not good in maths?

If you have studied Maths till 12th, it cannot be counted as an advantage as it usually fades off after a month of classes and tests. The CBSE/ICSE + JEE type of Maths that you have studied in your school is useless, unfortunately, because it does not have any use here and offers no advantage whatsoever.

Math is definitely very important if you go through the IPM curriculum because of its correlation with subjects like Economics and Statistics which require you to face some number crunching. You will have to put a little effort during the first couple of months or so just as the non-math students do. It will be fruitful because even they score full marks in Math. 

You just have to deal with your Math anxiety. Get over it by making math fun and not scary and IPM will feel like a dream.