The risk of pollen-induced allergies can be determined and predicted based on data derived from pollen monitoring. Hirst-type samplers are sensors that allow airborne pollen grains to be detected and their number to be determined. Airborne pollen grains are deposited on adhesive-coated tape, and slides are then prepared, which require further analysis by specialized personnel. Deep learning can be used to recognize pollen taxa based on microscopic images. This paper presents a method for recognizing a taxon based on microscopic images of pollen grains, allowing the pollen monitoring process to be automated. In this research, a deep CNN (convolutional neural network) model was built from scratch. Publicly available deep neural network models, pre-trained on image data (not including microscopic pictures), were also used. The results show that even a simple deep learning model produces quite good results when the classification of pollen grain taxa is performed directly from the images. The best deep learning model achieved 97.88% accuracy in the difficult task of recognizing three types of pollen grains (birch, alder, and hazel) with similar structures. The derived models can be used to build a system to support pollen monitoring experts in their work.