You can accelerate so slow down. The flyby of a planet creates an accelleration that, if in the correct direction, results in a slower orbital velocity in the sun's reference frame. In really simple terms, if you approach the planet from the right (in a counterclockwise solar system) you speed up, if you aim to the left you slow down. The velocity is transferred to/from the planet.
Notice how the probes orbit gets more eccentric but still smaller each time it approaches a planet on the left. The gravity assist is slowing it down, again in the solar reference frame.
Gravitational slingshots are a method were you accelerate and end up slowing down but I don't think it's fair to say that you "just accelerate to reach a lower orbit", since your acceleration ultimately ends up being either retrograde for very low fly-by orbits or on the normal/radial axis. Or possibly even prograde.
Basically, using a flyby to apply retrograde speed is not much more than accelerating prograde and then using the gravity of a planet to A) move your orbital position without a speed change and/or B) redirect your acceleration.
https://youtu.be/hKjcbmlAaYY
Notice how the probes orbit gets more eccentric but still smaller each time it approaches a planet on the left. The gravity assist is slowing it down, again in the solar reference frame.