Argyle had Paul Bignot sent off in the first half for kicking out at Sean Evans, who had just fouled him.
Bricklayer Paul McCone scored the opener after the break, heading home Sean Gebbis's corner into the bottom corner from six yards out.
And Evans sealed it when he converted following Jake Cole's parry.
The scoreline did not flatter seventh-tier side Stourbridge, who were unlucky to need a replay following Onismor Bhasera's 88th-minute equaliser for Plymouth in the first game at Home Park.
And the West Midlands side's win over the Football League's basement club means they will host League One side Stevenage in the second round.
Goalscorer Paul McCone won the West Midlands final of the Guild of Bricklayers junior competition this year
Stourbridge played excellently throughout the game and could have taken a first-half lead when Evans' centre found Ryan Rowe and his shot was well saved by Cole.
And things were looking ominous for Plymouth when Bignot retaliated to Evans' foul, which sparked a melee.
The Glassboys continued to dominate and Ben Billingham saw his top corner-bound shot saved by Cole before the break.
They did not let up after half-time and McCone, who works by day for his father's company, headed them into a fully deserved lead.
Plymouth's chances were few and far between and Durrell Berry's weak effort which was easily saved by Lewis Solly summed up their evening.
Stourbridge, sensing they were on the verge of something special, should have had their second when Paul Lloyd's cross hit the bar and Leon Broadhurst's headed the rebound off the crossbar with keeper Cole beaten.
But they had their clinching goal when the lively Rowe saw his shot saved and Evans found the corner from the rebound.
Plymouth never looked like getting back into it and had Billingham's late shot crept in rather than flew wide, there could have been few complaints in Devon at the scoreline.
For the latest updates throughout the day follow SportsDayLive. Get involved on Twitter using the hashtag #bbcsportsday.
Full Time   The final whistle is blown by the referee.
Drew Canavan takes a short corner.
Durrell Berry is adjudged to have handled the ball. Paul Lloyd takes the direct free kick.
Unfair challenge on Isaac Vassell by James Dyson results in a free kick. Simon Walton takes the direct free kick.
Durrell Berry takes a shot. Lewis Solly makes a comfortable save.
Ryan Rowe takes a weak shot and went wide of the left-hand post.
The referee blows for offside against Ryan Rowe.  Indirect free kick taken by Jake Cole.
Paul McCone restarts play with the free kick.
Booking      Conor Hourihane is given a yellow card.
Free kick awarded for an unfair challenge on Josh Craddock by Conor Hourihane.
Shot from just outside the box by Ben Billingham misses to the left of the goal.
Substitution      (Stourbridge) makes a substitution, with James Dyson coming on for Sean Gebbis.
The ball is swung over by Sean Gebbis.
Durrell Berry produces a cross, Nathan Bennett makes a clearance.
The referee penalises Drew Canavan for handball. Jake Cole restarts play with the free kick.
Ben Gibson challenges Paul McCone unfairly and gives away a free kick. Lewis Solly takes the free kick.
Jake Cole restarts play with the free kick.
Substitution      Sean Evans goes off and Josh Craddock comes on.
The assistant referee signals for offside against Ryan Rowe.
Corner taken by Will Atkinson from the left by-line to the near post, clearance made by Nathan Bennett.
Foul by Paul McCone on Matt Lecointe, free kick awarded. Conor Hourihane crosses the ball from the free kick left-footed from left channel.
Free kick awarded for a foul by Simon Walton on Ben Billingham. The free kick is swung in right-footed by Sean Gebbis.
Substitution      Isaac Vassell joins the action as a substitute, replacing Warren Feeney.
Assist on the goal came from Ryan Rowe.
Goal!  - Sean Evans -    Stourbr'ge 2 - 0 Plymouth        Sean Evans grabs a goal from the edge of the area to the bottom left corner of the goal. Stourbridge 2-0 Plymouth.
Ryan Rowe takes a shot. Save made by Jake Cole.
Free kick awarded for a foul by Durrell Berry on Sean Evans. Sean Evans crosses the ball, Ryan Rowe takes a shot. Save by Jake Cole.
Substitution      Drew Canavan on for Sam Rock.
Substitution      Matt Lecointe replaces Carl Fletcher.
Sean Gebbis concedes a free kick for a foul on Durrell Berry. Simon Walton takes the free kick.
Sean Evans produces a cross, clearance made by Ben Gibson. The ball is crossed by Paul Lloyd. Leon Broadhurst takes a shot.
Sean Gebbis produces a cross, Leon Broadhurst has a headed effort at goal from deep inside the area missing to the left of the target.
Sean Evans produces a cross, Curtis Nelson manages to make a clearance.
Shot by Conor Hourihane. Lewis Solly makes a comfortable save.
Onismor Bhasera sends in a cross, clearance made by Nathan Bennett.
Unfair challenge on Carl Fletcher by Leon Broadhurst results in a free kick. Direct free kick taken by Conor Hourihane.
Free kick awarded for a foul by Nathan Bennett on Warren Feeney. Free kick crossed left-footed by Conor Hourihane from right channel, Sam Smith manages to make a clearance.
Onismor Bhasera gives away a free kick for an unfair challenge on Sam Rock. Lewis Solly takes the direct free kick.
Shot by Durrell Berry. Save made by Lewis Solly.
Assist by Sean Gebbis.
Goal!  - Paul McCone -    Stourbr'ge 1 - 0 Plymouth        Paul McCone scores a headed goal from inside the six-yard box. Stourbridge 1-0 Plymouth.
Sean Gebbis takes a inswinging corner to the near post,
Simon Walton concedes a free kick for a foul on Ryan Rowe. Strike comes in from Ben Billingham from the free kick.
Centre by Durrell Berry, Lewis Solly makes a save.
Unfair challenge on Simon Walton by Ben Billingham results in a free kick. Simon Walton crosses the ball in from the free kick.
Sean Evans is penalised for handball and concedes a free kick. Durrell Berry takes the free kick.
Sean Evans gives away a free kick for an unfair challenge on Durrell Berry. Jake Cole restarts play with the free kick.
The second half has started.
Half Time    The match has reached half-time.
Unfair challenge on Simon Walton by Ben Billingham results in a free kick. Onismor Bhasera restarts play with the free kick.
The ball is sent over by Nathan Bennett, Ben Gibson manages to make a clearance.
Will Atkinson is penalised for a handball. Sean Gebbis takes the free kick.
Centre by Will Atkinson, save made by Lewis Solly.
Foul by Paul Lloyd on Warren Feeney, free kick awarded. Jake Cole takes the free kick.
Ben Billingham takes a shot. Save by Jake Cole.
Free kick awarded for an unfair challenge on Simon Walton by Paul McCone. Direct free kick taken by Jake Cole.
Outswinging corner taken right-footed by Sean Gebbis from the right by-line, clearance by Curtis Nelson.
Direct free kick taken by Jake Cole.
Substitution      Durrell Berry replaces Luke Daley.
Booking      The referee shows Curtis Nelson a yellow card for dissent.
Booking      Booking for Ryan Rowe for dissent.
Dismissal      Paul Bignot is sent off by the referee.
Sean Evans gives away a free kick for an unfair challenge on Paul Bignot.
Sean Evans takes a shot. Curtis Nelson gets a block in.
A cross is delivered by Sean Evans.
Free kick awarded for an unfair challenge on Nathan Bennett by Will Atkinson. Direct free kick taken by Lewis Solly.
Short corner worked by Will Atkinson.
Corner taken left-footed by Conor Hourihane, Lewis Solly makes a save. Luke Daley challenges Paul McCone unfairly and gives away a free kick. Lewis Solly takes the direct free kick.
Free kick awarded for a foul by Ryan Rowe on Jake Cole. Jake Cole takes the direct free kick.
Centre by Sean Evans, Ryan Rowe takes a shot. Brilliant save by Jake Cole.
Free kick awarded for an unfair challenge on Simon Walton by Paul McCone. Simon Walton takes the free kick.
Foul by Sean Evans on Paul Bignot, free kick awarded. Direct free kick taken by Jake Cole.
Short corner worked by Paul Bignot.
Centre by Luke Daley, blocked by Paul Lloyd.
Ryan Rowe gives away a free kick for an unfair challenge on Curtis Nelson. Free kick taken by Jake Cole.
Paul Bignot gives away a free kick for an unfair challenge on Sean Evans. Direct free kick taken by Paul Lloyd.
Carl Fletcher concedes a free kick for a foul on Leon Broadhurst. Direct free kick taken by Sean Gebbis.
Paul McCone concedes a free kick for a foul on Will Atkinson. The free kick is delivered left-footed by Conor Hourihane from left wing, clearance made by Nathan Bennett.
Effort from inside the area by Conor Hourihane misses to the left of the goal.
The ball is crossed by Leon Broadhurst, save by Jake Cole.
Unfair challenge on Nathan Bennett by Warren Feeney results in a free kick. Sean Gebbis sends in a cross, Nathan Bennett takes a shot. Save by Jake Cole.
Carl Fletcher gives away a free kick for an unfair challenge on Nathan Bennett. Free kick taken by Sean Gebbis.
The match gets underway.
Live data and text provided by our data suppliers
Live text commentary
Southern Premier side Stourbridge reached the FA Cup second round for the first time in their history with a shock replay win over 10-man Plymouth.