Modern machine learning methods were used to automate and improve the determination of an effective quality index for coffee beans. Machine learning algorithms can effectively recognize various anomalies, among others factors, occurring in a food product. The procedure for preparing the machine learning algorithm depends on the correct preparation and preprocessing of the learning set. The set contained coded information (i.e., selected quality coefficients) based on digital photos (input data) and a specific class of coffee bean (output data). Because of training and data tuning, an adequate convolutional neural network (CNN) was obtained, which was characterized by a high recognition rate of these coffee beans at the level of 0.81 for the test set. Statistical analysis was performed on the color data in the RGB color space model, which made it possible to accurately distinguish three distinct categories of coffee beans. However, using the Lab* color model, it became apparent that distinguishing between the quality categories of under-roasted and properly roasted coffee beans was a major challenge. Nevertheless, the Lab* model successfully distinguished the category of over-roasted coffee beans.