The Sky Bet League One side fell twice behind to goals from Saido Berahino, who was making his first start for three months and possibly his final Albion appearance ahead of Monday's transfer deadline.
But Shaquile Coulthirst levelled 11 minutes from time and, then after Berahino scored what looked set to be a winner in the 85th minute for the Barclays Premier League side, Posh substitute Jon Taylor levelled again within seconds.
Shrewsbury also stunned higher-ranked opponents with defender Jack Grimmer's stoppage time header clinching a 3-2 win over Sheffield Wednesday.
Lewis McGugan had twice put the Championship side ahead but Jean-Louis Akpa Akpro and Shaun Whalley, following up Akpa Akpro's saved penalty, levelled before Grimmer sent the League One side through to the last 16.
Cup holders Arsenal beat Burnley 2-1 with Alexis Sanchez's 53rd-minute winner his first goal after two months out through injury.
Calum Chambers had earlier given Arsenal a 19th-minute lead before Wales international Sam Vokes equalised on the half-hour mark with a firm header.
Teenager Kelechi Iheanacho was the Manchester City hero as the 2011 winners enjoyed a 4-0 romp at Aston Villa
Iheanacho put City ahead inside three minutes and there was no way back for Villa after his 24th-minute penalty punished Micah Richards' push on Raheem Stering.
Richards' poor back pass allowed Iheanacho to complete his hat-trick after 74 minutes and Sterling completed the rout moments later.
Tottenham cruised into the fifth round with a 4-1 win at Colchester.
Nacer Chadli scored twice with Eric Dier and Tom Carroll also on target as Spurs chalked up a fourth straight win, while League Two's bottom club managed a consolation when Ben Davies put through his own net.
Bournemouth were in danger of tumbling out to their south coast neighbours Portsmouth after Gary Roberts struck for the 2008 winners two minutes before the break.
But Joshua King (69) restored parity and substitute Marc Pugh (83) struck the winner to secure local bragging rights as well as a place in the last 16.
Wilfried Zaha's 17th-minute strike decided the all-Premier League battle with Stoke and one goal was also enough for Watford at Nottingham Forest.
Odion Ighalo, on as a substitute in the final quarter, settled the tie in the final minute with his 15th goal of the season.
The Championship will be well represented in the fifth round with Leeds, Blackburn, Hull and Reading all going through
Leeds won 2-1 at Championship rivals Bolton with Souleymane Doukara and new signing Toumani Diagouraga scoring first-half goals, Darren Pratley managing a late reply for the hosts.
Blackburn coasted home 3-0 at League Two Oxford with Ben Marshall scoring twice, one from the penalty spot, and Tony Watt adding another on his debut following his loan move from Charlton.
Oxford had beaten Premier League Swansea in the third round but were 2-0 down by half-time and never looked like producing another upset.
Reading reached the semi-finals last season and might be eyeing another cup run after thrashing League One Walsall 4-0.
Matej Vydra scored twice after Hal Robson-Kanu had opened the scoring and Danny Williams was also on the scoresheet.
On-loan striker Chuba Akpom scored his first senior hat-trick as Championship leaders Hull eased through 3-1 at Bury.
Akpom opened the scoring after 14 minutes and converted his second from the penalty spot close to the hour-mark after Sone Aluko had been tripped.
The Arsenal man completed his treble after 69 minutes for the 2014 finalists, with Craig Jones finding the net late on for the League One Shakers.
Source : PA
Source: PA