There are two aspects to the question. You asked.
To answer this part, I will say with certainty that IIT Kharagpur is an excellent institute. However, as far as architecture is concerned, your first choice should always be the elite schools of architecture like SPA or CEPT. This is mainly because IITs are known for their technology and not architecture.
If you are genuinely interested in physics and math, then this will be one of your strengths and will prove to be useful in the journey of architecture.
Architecture requires problem-solving skills apart from artistic skills. There are many students who are average in drawing but have excellent ideas and designs. To broaden your knowledge about art and architecture you can read books.