The software engineering discipline contains numerous myths and over-simplifications. Some of them may be harmless, but others may hamper evidence-based practices and contribute to a fashion- and myth-based software engineering discipline. In this article we give examples of software engineering myths and over-simplifications and discuss how they are created and spread. One essential mechanism of the creation and spread of myths and over-simplifications are, we argue, people's tendency towards searching for confirming and neglecting disconfirming evidence. We report from a study examining this tendency. The study demonstrated that the developers who believed in a positive effect of agile methods tended to interpret randomly generated (neutral) project data as evidence confirming the benefit of agile methods. For the purpose of supporting evidence-based practice and avoiding unwanted influence from myths and over-simplifications, we provide a checklist to be used to evaluate the validity of software engineering claims.