The former York City striker picked himself up after being floored by Mark Arber in the fifth of six minutes of added time and sent home goalkeeper Mark Tyler wrong way with a sweetly struck right-footed shot.
Bury's Kasper Schmeichel was the busier goalkeeper in the opening half and life was made more difficult for the Manchester City loanee by a swirling wind.
Schmeichel was rescued by defenders after completely failing to punch a Dean Holden corner. Another fumble gave away an unnecessary corner, but Schumacher made amends by tipping over a fiercely struck Holden free-kick.
Lloyd Opara squandered a glorious Posh opening, blazing over the bar from inside the six-yard box and in added time Schmeichel saved impressively when substitute Justin Richards let fly.
Bury created little going forward and on a rare occasion when Tom Youngs found space to cross, Arber's well-timed tackle denied Bishop as he was about to shoot.
Arber's timing also proved decisive just after the break when Bury skipper Dave Challinor hurled in one of his trademark long throws.
Home keeper Tyler denied David Buchanan, turning a shot over the crossbar, but was saved by the woodwork on 58 minutes when Richard Baker got forward to send in an angled header from Youngs' cross.
After little over an hour Posh boss Keith Alexander had lost all of his starting attacking line-up. Trevor Benjamin exited early with a knee injury, Opara was replaced after the break and then Simon Yeo was stretchered off with a leg injury.
Play was even held up while repairs supervised by referee Fred Graham were made to the goal net being defended by Posh, although a wait for a step-ladder to be brought on was ended when lanky home defender Jude Sterling came to the rescue.
Outstanding defender Guy Branston almost snatched the win for Posh, seeing a storming header from a Danny Crow cross smack against the Bury crossbar before Bishop struck.