Does working in a BPO count as work experience for getting into IIMs and XLRI?

Yes, as long as it is full-time and you have documentation to show it, it will be counted.

Any type of job teaches you a lot, and if you put in the effort, you can advance anywhere. You should be fine as long as you think carefully about what you've learned and present it as part of a cohesive life story during the selection process.

One of my friend’s first jobs was at a call center, and it was a lot of fun for him. His sales skills were lacking, but his quality scores were excellent, and he was even recognized for them at an annual event. He was selected to be part of a small pilot team because the organization decided to show the client that they could do more challenging work as well. 

Diversity in the batch is a wonderful thing to have from the admissions committee's viewpoint, along with other factors like good grades.