Tommy Rowe's 26th-minute goal saw Posh lead at half-time but two goals in four minutes from Michael Gash and Jack Byrne turned the game on its head.
A Britt Assombalonga penalty levelled the scores with 16 minutes to go but parity lasted just two minutes as Joe Lolley grabbed a third for Harriers, who are now the only non-league side left in the competition.
Fellow Skrill Premier outfit Macclesfield had their cup run ended as three late goals saw them beaten 4-1 at Hillsborough by Sheffield Wednesday.
Chris Maguire delivered an early blow to the non-league team's hopes as he fired the Owls ahead after just four minutes, but a 65th-minute penalty from Scott Boden had the Silkmen level.
Jacques Maghoma settled the nerves at Hillsborough, though, as he netted with 12 minutes to go and Jermaine Johnson and Miguel Llera sealed the victory late on.
A Joe Garner hat-trick handed Preston a shock 3-2 win over Championship side Ipswich at Deepdale.
Franck Nouble put Ipswich into the lead after 58 minutes but two goals in as many minutes from Garner put Preston in control before substitute David McGoldrick grabbed an equaliser.
Garner then completed his hat-trick with two minutes remaining and Preston will now go on to face Nottingham Forest.
Watford had no such problems overcoming Bristol City as goals from Marco Davide Faraoni and Lewis McGugan gave them a 2-0 win.
Port Vale will face Brighton in round four after they claimed a dramatic 3-2 win over Plymouth.
Argyle were ahead after just two minutes through Andres Gurrieri. Jordan Hugill equalised on the half-hour mark before Conor Hourihane restored the lead nine minutes before the break.
Ben Williamson then levelled things up again and the game took another twist as Jennison Myrie-Williams put Vale ahead.
There was time for more drama as Chris Lines was sent off for the visitors but Vale goalkeeper Chris Neal saved Reuben Reid's resulting penalty to ensure his side's progression.
Championship teams Birmingham and Bournemouth both made serene progress into round four with comfortable wins.
Brett Pitman gave the Cherries the lead five minutes in against Burton but Jimmy Phillips had Albion level before Tommy Elphick restored the lead on the stroke of half-time.
Ryan Fraser opened up a two-goal cushion in the 86th minute and two minutes later a Pitman penalty secured a 4-1 win and a fourth-round tie against Liverpool.
At St Andrews, Paul Robinson scored for the Blues after 35 minutes and two late goals from Chris Burke sealed a comfortable 3-0 triumph over Bristol Rovers.
Charlton and Oxford will have to play again after their re-arranged third-round tie ended 2-2.
The first game had been called off as the Addicks continue to have problems with the pitch at the Valley and their woes were deepened when Johnny Mullins fired United ahead after 13 minutes and Scott Davies doubled the lead 11 minutes later.
Michael Morrison pulled one back in the 54th minute to give the home side hope, though, and they were level in the 82nd minute through Yann Kermorgant.
Source : PA
Source: PA