GPT-4o’s Chinese Training Data Contaminated by Pollution, Poses AI Challenge in Astronomy

Recently, OpenAI released its latest version of the chatbot, GPT-4o. While it was expected to be an improvement over previous versions, some Chinese speakers started to notice that the text processed by GPT-4o contained spam and porn phrases. This issue arises because GPT-4o reads in tokens, which are distinct units in a sentence, rather than in words. These tokens have consistent and significant meanings in the context of the sentence. However, the new tokenizer used by GPT-4o introduced a disproportionate number of meaningless phrases in the Chinese language.

Experts believe that this problem stems from insufficient data cleaning and filtering before training the tokenizer. Without proper cleaning of the data, the tokenizer becomes prone to including irrelevant or inappropriate content. If this issue is not resolved, it can lead to hallucinations, poor performance, and misuse of the chatbot. This highlights the importance of ensuring data quality and thorough preprocessing before training language models like GPT-4o.

Furthermore, this incident serves as a reminder of the challenges faced in developing AI models that can handle multi-language tasks. Language models like GPT-4o aim to provide accurate and contextually relevant responses in various languages. However, language nuances, cultural differences, and diverse linguistic patterns pose significant difficulties in achieving this goal. The incident with GPT-4o shows that even a state-of-the-art model can struggle when it comes to processing certain languages, leading to potentially misleading or inappropriate outputs.

In a different domain, astronomers are grappling with a data challenge of their own. With the upcoming Square Kilometer Array (SKA) Observatory, astronomers are preparing to process an enormous amount of cosmological data. The SKA Observatory will utilize hundreds of thousands of dishes and antennas to gather information about the universe’s first stars and the evolution of galaxies. However, this endeavor will result in nearly 300 petabytes of data per year, equivalent to the storage capacity of a million laptops.

To handle this data deluge, astronomers are turning to AI for assistance. They are exploring the application of AI algorithms and techniques to analyze and extract meaningful insights from the massive volumes of data. The use of AI can aid in automating data processing tasks, identifying patterns, and discovering new phenomena that may have otherwise gone unnoticed. By leveraging AI’s capabilities, astronomers hope to accelerate their research and gain a better understanding of the cosmos.

Nevertheless, integrating AI into the field of astronomy also presents its own challenges. Astronomical data is complex and often requires domain-specific expertise to interpret. The AI algorithms employed must be trained on relevant datasets and tailored to address the specific requirements of astronomical research. Additionally, ensuring the accuracy and reliability of AI-generated results is crucial, as any inaccuracies or biases could lead to misleading conclusions.

Despite these challenges, the collaboration between AI and astronomy holds immense potential. AI can aid astronomers in tackling the vast amounts of data generated by cutting-edge observatories like the SKA. It can assist in filtering noise, extracting meaningful signals, and even facilitating the discovery of celestial phenomena that were previously unknown. The combination of human expertise and AI capabilities can push the boundaries of astronomical research and contribute to our understanding of the universe.

In conclusion, the incident with GPT-4o highlights the importance of data cleaning and filtering in language model development. It demonstrates the complexities of handling multi-language tasks and emphasizes the need for appropriate preprocessing to ensure accurate and contextually relevant outputs. On the other hand, the collaboration between AI and astronomy showcases the potential of AI in handling massive volumes of data and accelerating scientific research. By harnessing AI’s capabilities, astronomers can overcome data challenges and gain deeper insights into the mysteries of the universe.



Source link