Here we present a novel atlas-based geometry pipeline for constructing three-dimensional cubic Hermite finite element meshes of the whole human heart from tomographic patient image data. To build the cardiac atlas, two superior atria, two inferior ventricles as well as the aorta and the pulmonary trunk are first segmented, and epicardial and endocardial boundary surfaces are extracted and smoothed. Critical points and skeletons (or central-line paths) are identified, following the cardiac topology. The surface model and the path tree are used to construct a hexahedral control mesh via a skeleton-based sweeping method. Derivative parameters are computed from the control mesh, defining cubic Hermite finite elements. The thickness of the atria and the ventricles is obtained using segmented epicardial boundaries or via offsetting from the endocardial surfaces in regions where the image resolution is insufficient. We also develop a robust optical flow approach to deform the constructed atlas and align it with the image from a second patient. This registration method is fully-automatic, and avoids manual operations required by segmentation and path extraction. Moreover, we demonstrate that this method can also be used to deformably map diffusion tensor MRI data with patient geometries to include fiber and sheet orientations in the finite element model.