This study describes the performance of a randomly coupled erbium-doped optical fiber amplifier (EOFA) using machine learning. A reinforcement learning-based artificial neural network (RL-ANN) is used to study various parameters of EOFA, such as gain, noise figure, and inter-core crosstalk for the 12-core EOFA. In addition to this, the relationship between mode-dependent loss, a penalty of the EOFA, and the average bending loss (BL) for three operating wavelengths, such as 1550, 1580, and 1600 nm, has been studied. To predict the performance of RL-ANN, three different machine learning algorithms, namely, K-nearest neighbor (KNN), ANN, and support vector machine (SVM) are used. The performance of the proposed RL-ANN is evaluated using five statistical error criteria: mean absolute error, root mean square error, scatter index, coefficient of variation, and overall index measurement. The statistical test result shows that the proposed RL-ANN provides the best performance compared to KNN, ANN, and SVM. The performance of the RL-ANN-based erbium-doped fiber amplifier (RA-EOFA) is studied and the results show that the gain and estimated NF based on a wavelength range from 1555 to 1565 nm provides a maximum gain. A comparison study was carried out for the proposed RA-EOFA with other learning methods. The results show that the proposed RA-EOFA provides the best performance in terms of less BL (0.25 dB), mode-dependent loss (0.5 dB), computational time (36.8 s), and a success rate of 98.39%.