First Baptist Church Madison Church of Madison, MS. The location of First Baptist Church Madison Church is 39110, 2100 Main St - marked with a pin on Google Map.