Why everything takes longer than planned
Ask somebody how long a piece of work will take and the answer will usually be too short. Ask the same person how long the last comparable piece of work took and they will often give the right figure. The two answers sit in the same head, and the gap between them is one of the most reliable findings in the study of planning.
The reason is in what a person imagines when they estimate. Asked for a figure, we construct a picture of the work going as intended: the information arrives, the supplier answers, nobody falls ill. Each of those assumptions is individually reasonable and the chance that all of them hold is small. We are not estimating the likely duration. We are estimating the duration of the version in which nothing goes wrong, which is a different quantity and always a smaller one.
This explains a detail that puzzles people, which is that experience does not cure the problem. An experienced builder knows that previous projects overran, and estimates the next one optimistically anyway, because the estimate is made by imagining the work rather than by consulting the record. The knowledge is in memory but it is not in the method.
The most effective correction is therefore not to think harder but to think about something else. Instead of imagining this project, find how long similar projects actually took and start from that number, adjusting only for differences you can name. Organisations that adopt this produce estimates that are less precise, considerably more accurate, and unpopular, because an estimate derived from a dull average feels like an admission that you have not thought about the specifics.
There is one more pressure that is often left out of the psychological account. In most settings a long estimate competes against a short one, and the short estimate wins the work. The optimistic figure is therefore selected for, not merely produced, and no amount of better reasoning by individuals will fix a problem that is being rewarded.