This model study aimed to evaluate the effect of phenolic–food matrix interactions on the in vitro bioaccessibility and antioxidant activity of selected phenolic compounds (gallic acid, ferulic acid, chlorogenic acid, quercetin, apigenin, and catechin) as well as protein and starch digestibility in fortified white bean paste. The magnitude of food matrix effects on phenolics bioaccessibility and antioxidant activity was estimated based on “predicted values” and “combination indexes”. Furthermore, the protein–phenolics interactions were investigated using electrophoretic and chromatographic techniques. The results demonstrated phenolic–food matrix interactions, in most cases, negatively affected the in vitro bioaccessibility and antioxidant activity of phenolic compounds as well as nutrient digestibility. The lowest in vitro bioaccessibility of phenolic compounds in fortified paste was found for quercetin (45.4%). The most negative impact on the total starch digestibility and relative digestibility of proteins was observed for catechin–digestibility lower by 14.8%, and 21.3% (compared with control), respectively. The observed phenolic–food matrix interactions were strictly dependent on the applied phenolic compound, which indicates the complex nature of interactions and individual affinity of phenolic compounds to food matrix components. In conclusion, phenolic–food matrix interactions are an important factor affecting the nutraceutical and nutritional potential of fortified products.