As a coin collector and technology nerd, in late 2023 I developed Numi,an AI-powered chatbot that leverages the advanced capabilities ofOpenAI's GPT-4 vision model to assist coin collectors in identifyingand grading their coins. It's been fascinating seeing the exponentialgrowth of Artificial Intelligence, so I created Numi to test AI's abilities to tackle one of the biggest barriers to new collectors in coincollecting. Throughout Numi's development, I became more and moreconvinced that AI is going to fundamentally change the future of thehobby.

Testing Numi With OpenAI's Latest GPT-4 Update

I had a hypothesis that more data given to the AI would yield moreaccurate grading results. In December 2023 I ran a series of tests oneach grade using 2 to 10 photos per coin. Following OpenAI's recentApril 2024 update to their GPT-4 model, which powers Numi's AIcapabilities, I conducted another series of tests on Numi's gradingaccuracy.

I then ran statistical analyses to assess the impact on Numi'sperformance and compared its grading accuracy between theDecember 2023 and April 2024 test results.

Determining the Optimal # of Photos for Accurate Grading

A key aspect of my analysis focused on identifying the optimal numberof coin photos users should upload to achieve the most accurategrading results. In December 2023, my tests indicated that uploading10 photos yielded the best accuracy across all coin grades. Thisaligned with my hypothesis that more data = better. However, after theGPT-4 update in April 2024, that number had changed, with just 4photos now providing the most precise grading outcomes.

Just How Much Did Numi Improve?

To measure Numi's accuracy and any improvements, I calculated theMean Absolute Deviation (MAD) - a metric that represents theaverage deviation between Numi's predicted grades and the actual,expert-assigned grades. In December 2023, Numi's MAD was 5.39,indicating that, on average, its predictions deviated by approximately 5points from the actual coin's grade (for example, coin grade is AU-55 but Numi predicted MS-60). By April 2024, following the GPT-4 update, Numi's MAD score decreased to 3.64, representing asubstantial 32.47% increase in overall accuracy.

I suspected that Numi would be more accurate given the updates, butI was not expecting this much of a change. While the GPT-4 visionmodel still struggles immensely with medium-graded coins [AroundXF-40], Numi performed exceptionally well for very low and very high-graded coins, with the biggest improvements seen for very low-graded coins.

The Future of AI in Numismatics

After seeing these results, I am even more convinced that ArtificialIntelligence will revolutionize the field of coin collecting. As models like GPT-4 continue to improve, AI tools will become increasingly valuablefor collectors seeking to expand their knowledge and make informeddecisions about their collections. While Numi itself will most likely notend up being the go-to tool for collectors in the future, it serves aspowerful evidence of where the hobby is heading.

The progress Numi has made in a short time is encouraging, and Ilook forward to testing its capabilities as AI models advance. Bymaking coin grading more accessible and user-friendly, AI has thepotential to attract new enthusiasts to the hobby and help experiencedcollectors deepen their understanding and appreciation fornumismatics.

To read the earlier E-Sylum articles, see:
NUMI: AI-POWERED COIN IDENTIFICATION APP(https://www.coinbooks.org/v26/esylum_v26n47a10.html)
https://www.coinbooks.org/v26/esylum_v26n47a11.html(https://www.coinbooks.org/v26/esylum_v26n47a11.html)
UPDATE: AI-POWERED APP NUMI(https://www.coinbooks.org/v26/esylum_v26n52a12.html)
NOTES FROM E-SYLUM READERS: NOVEMBER 26, 2023 : Numi on the Coin World Podcast(https://www.coinbooks.org/v26/esylum_v26n48a15.html)