The past few times I've had this happen, refreshing the page just dropped me right where I'd left off before trying to seek backwards.