There are many reasons for low charpy values. Here are the most obvious ones in order of importance.
Discontinuities such as shrinkage/porosity or macro-inclusions at the location near the notch.
Normalized and tempered condition rather than quenched and tempered condition. Q&T gives much...