My father never went to synagogue. He made an exception for my bar mitzvah, but he must have suffered a lot. He did fast on the Day of Atonement. He explained to us that he did so out of solidarity with the rest of his family.
As a child I had to fast only half a day. I remember eating herring salad in our garden on the Day of Atonement.