>How do you know the ballot box you put your vote into is being taken to be counted and recorded, and not simply destroyed and another ballot box introduced?
You can watch it.
With a "ballot box" system, you could get there, put your vote in the box, then sit there and watch it all day until it's counted. Hell if you want you could get there and inspect the box before they start voting.
And it's not just you that can do that, everyone can do that regardless of profession, age, race, background, education, etc...
To clarify on top of Klathmon's point: this actually happens in Switzerland, it's not just a theoretical argument.
Actually, the government is required to have some people watching the counting (specific rules vary on a per-canton basis). Some states select random citizens for that (similar to jury duty in the US - you get fined if you don't show up, but get paid for your time). Other states have a pool of people they choose from randomly (you volunteer for the pool).
EDIT: Furthermore, Swiss people vote in their community. It's not like you need to go out of your way to watch the counting - it happens in your town hall or (sometimes) the school. It always happens on Sundays (which is a required day off in Switzerland except in some rare cases).
If you don't mind me asking, how does Switzerland handle those which are required to work on election days?
That's one of the ugliest parts of the US election system to me, is the fact that many people need to fit the election around their work schedule, and I know of at least 3 people that couldn't get time to vote this year, and my state makes it so you basically need to be actively deployed in the military, or be hospitalized to get an absentee ballot.
Like I added in my edit, there is almost no such case: Sundays are a mandatory day off for most. That means you can't go shopping on a Sunday, except in very few shops (train station shops have exceptions). Doctors, pharmacies and most other emergency things operate on a reduced schedule.
I'm not certain what would happen if say a nurse working a shift in an ICU was selected. I'm pretty sure it's either announced early enough that the hospital would have to switch shiftees or they would be given a pass for free. I suspect a person working in a train station shop on a Sunday would tell their boss, and would simply be shifted to another time (it's not like the boss or the employee can do anything about it - it's law).
EDIT: to clarify once again - I'm pretty sure it does happen, but the case is probably rare enough for it not to matter too much. The rule works for a good 90% of cases, exceptions can certainly be done on a case-by-case basis at community level.
In Spain, where the system is similar to what you described (staff for the day is chosen at random from all citizens in that voting district, and similar to jury duty), some professionals (doctors, nurses, air controllers, etc.) can decline the request if they will be on watch, and there are assigned replacements for them.
A non-critical worker cannot refuse but:
a) Chosen citizens are paid, even if not much (60€ I think).
b) Chosen citizens have the following Monday morning free, should they need to travel back to their job site, etc.
c) Chosen citizens have strong legal protections should their boss try anything.
In practice, when an employee is chosen for election duty the boss just deals with it as best as they can, since it's a case where judges and police are super strict: messing with election duty will get you heavy fines and/or jail time. So no-one is going to fire you or retaliate if you are assigned election duty.
In all the German elections I participated the only people watching (apart from the regular citizens that are appointed official poll workers) were the few people showing their children how voting works - and they leave after a few minutes because let's be honest, counting ballots is rather boring. It would be nice if a few more people were watching the process.
So is pulling down a lever and watching spinning wheels. Permit placing bets on the outcome. Turn it into entertainment. That always works. And if money is involved, people will triple check. [oh the Humanity. I forget now if this comment started as an /s.]
To add to your point, also with ballot system it's very hard to change results. You can by a few votes, though it's a drop in the water compared how much you can buy and how many vote. If you want to change the outcome of the vote - you need to have a lot of resources (look at Crimea's occupation by Russia). Where's with e-voting, there could be few north koreans who would be able to compromise e-voting system in the country, which is on the other side of the world, just for the kicks.
I suppose that I could, in theory, watch it. It's a physical ballot, so we could do that. But does anyone actually do this? And more importantly, are all the votes watched?
It's completely impractical to try to watch millions of physical ballots. Or to even simply count to a million without making a mistake.
But if the voting was done (correctly, securely) in software, each person could independently audit the complete results. And to be clear, it's the overall election being audited that's important, not my vote in particular.
Perhaps it would be more difficult to understand the algorithm and code than following a physical ballot, but not a million times more difficult.
paper voting is so great because it scales so well.
There is no one person counting millions of votes. There are people counting hundreds or even thousands of votes in every precinct. Then from there they are publicly announced, and then anyone and everyone can tally up the totals.
The more people that vote, the more precincts there are, so the amount that any one person or place is responsable for (even only partially) is kept small so that mistakes or "bad actors" can only do a very small amount of damage on their own.
And if you read down in this submission a bit, people are pointing out that some countries (namely Switzerland) not only encourage people to watch ballot boxes and count along, but actually "draft" people to do it similar to a jury-duty kind of thing in the US.
Also I think you are severely underestimating the difficulty of auditing code. For starters, it completely removes the ability for the vast majority of the population to audit even if they wanted to. non-programmers can't read programming. Not to mention the illiterate, the elderly, and anyone else that doesn't have the knowledge to be able to read and follow a programming language.
Then there's the amount of time it takes. Watching a ballot box takes the duration of the election day. A day at most. Auditing every inch of a codebase? Months, years, even longer. You can point out that not everyone has to review every single line, but then you are back to letting a fraction of a percent of the population validate the code (and in a ballot box system, you can have everyone validate the process, adding magnitudes more redundancy that something shady will be caught). And what percentage of programmers do you think have a good enough grasp on cryptographic protocols and voting systems to be able to correctly and securely vet a system like this? I'd be willing to bet good money there are less than a thousand of them in the united states, and by sheer (un)luck at least one of them will be a "bad actor".
But probably the biggest thing is that to sway a "paper and pencil" election, you need materials, you need a LOT of people, and all it takes is one person to see what's going on and it all collapses. If you're really clever, you might be able to get away with changing the outcome of a single precinct, but what about the rest of the county? the state? the nation? In 2012 there were 2712 voting precincts in virginia alone. Orchestrating a system where you can put a person at a majority of them that will do something nefarious to sway the election means you are looking at 1000 people spread across a not-insignificant amount of land. And that's just for one state.
With an electronic voting system, one bug is all it takes. One vulnerability, one crypto flaw, one guy somewhere that can put the backdoor in the manufacturing of the CPUs for the vote counters. A single person can (if the are very smart and lucky) completely change the outcome of the election. And they might get caught, but the chances are much smaller than if it required thousands of people at the least to do so.
You can watch it.
With a "ballot box" system, you could get there, put your vote in the box, then sit there and watch it all day until it's counted. Hell if you want you could get there and inspect the box before they start voting.
And it's not just you that can do that, everyone can do that regardless of profession, age, race, background, education, etc...