Meta’s new large language model, Llama 3, is powering the company’s chatbot, Meta AI. This chatbot is integrated into various Meta platforms and apps, including Instagram, Facebook, and WhatsApp. In terms of functionality, Meta AI falls short in comparison to other general-purpose conversational AI systems. It often regurgitates web search results and lacks excellence in any particular area. However, considering that it is free to use, this may be overlooked by users.
Currently, Meta AI can be accessed on the web at Meta.ai, as well as on platforms like Instagram, Facebook, and WhatsApp. These platforms were already available, but the recent releases of Llama 3 and the Imagine image generator have prompted Meta to promote Meta AI as a go-to AI assistant. It is worth noting that Meta AI is now the default search box replacement, which means that users may stumble upon it accidentally.
Meta AI holds a strong ambition to become the most widely used and best AI assistant globally, according to Mark Zuckerberg. While this is an admirable goal, it remains to be seen if Meta AI can live up to this expectation.
In terms of evaluating Meta AI, a casual and informal approach was taken. Ordinary questions that regular people might ask were posed to the model, and the responses were compared with other AI models or the desired outcomes. This evaluation process aims to be accessible and replicable for anyone, but it should not be considered comprehensive.
Notably, the process for evaluation is subject to change and adjustment, and there may be occasional unusual findings. For instance, during this evaluation, it was observed that the Imagine model demonstrated biases related to Indian people. These observations will be explored further in a separate article.
It is essential to highlight a bug encountered on Instagram that prevented the deletion of the queries sent to Meta AI. Therefore, caution is advised when asking questions that users may not want appearing in their search history. Additionally, there were compatibility issues with the web version of Meta AI on Firefox.
When asked about current news and events, Meta AI provided concise and up-to-date answers. However, its responses often included links for further search, which suggested possible search promotion partnerships. To determine if Meta AI relied on Bing’s AI model, an examination was performed, and the answers provided by Copilot were compared. The two models presented different information, indicating that they are distinct systems.
When inquiring about recent trends on TikTok, Meta AI provided a high-level summary of creators’ activities on the platform without referencing recent trends. In contrast, asking about trends on Instagram yielded a response that appeared to be derived directly from an SEO bait Instagram trends post by Hootsuite. This reliance on regurgitating content diminishes the value of using Meta AI for obtaining genuinely interesting and informative insights on social media trends.
Concerning historical research, Meta AI struggled to provide primary sources related to Supreme Court decisions in the late 19th century. Instead, it relied on an SEO-optimized post that listed famous 19th-century decisions. The inclusion of an irrelevant founding document from the People’s Party further undermined the accuracy and relevance of the response. In comparison to other models, Meta AI did not offer the same level of context and summarization.
For factual questions such as medal counts in the 1984 Olympics and notable events of that year, Meta AI provided satisfactory answers with appropriate citations. However, the presentation of citations at the top and links at the bottom of the response seemed unnecessary and inconvenient. Other models that provide in-line citations offer a more user-friendly approach for research and fact-checking.
When discussing controversial topics such as the age and racial demographics of Donald Trump’s supporters, Meta AI provided an even-handed response that pushed back on the assertion inherent in the question. Unfortunately, no sources or links were provided for further exploration. Similarly, when asking about the rise of white nationalism, Meta AI provided a comprehensive list of reasons without offering any sources. It is speculated that Meta may be avoiding citations on certain topics, but citations are particularly important in such contexts for promoting transparency and learning.
In terms of medical inquiries, Meta AI initially encountered a bug that prevented it from providing a response about a fictitious 9-year-old developing a rash after eating a cupcake. Upon re-posing the question, Meta AI offered reasonable and general advice for handling potential allergic reactions. The issue seemed to stem from a delayed realization by the model, resulting in an awkward response. Similarly, when inquiring about supplements, Meta AI provided a fair and well-sourced answer.
When addressing mental health, Meta AI presented predictable advice for anxiety and medication, emphasizing the importance of consulting a professional. It also shared helpline numbers for individuals facing serious difficulties. However, no sources or links were provided alongside these responses, highlighting a missed opportunity for further information.
Meta AI struggled to effectively summarize articles, often selecting important-sounding sentences without providing a cohesive summary. In some instances, the model slightly altered the meaning of sentences, potentially leading to misunderstandings. When asked for a summary in less than 100 words, the response incorporated information from external articles, which was not an expected outcome.
Regarding content creation, Meta AI generated typical marketing copy suggestions for an imaginary clothing brand. This demonstrated the AI’s ability to generate captions commonly seen in marketing materials. However, the repetitive nature of these suggestions highlighted the formulaic nature of the task, which AI systems have become proficient at. On the bright side, marketers might benefit from AI-generated suggestions, potentially reducing their workload.
Lastly, when Meta AI was prompted for jokes related to farmers, the responses were subpar, displaying the limitations of AI-generated humor. These questions do not necessarily serve to provide genuinely funny material; instead, they aim to test the AI model and ensure it does not exhibit concerning behavior or repeat content from specific communities.
In conclusion, Meta AI positions itself as a convenient tool for answering casual questions. However, its search-centric approach, with extensive quoting and reliance on external search engines, raises the question of why users should not simply use Google or Bing directly. Although Meta AI can offer up-to-date information through online searches, comparable results can be obtained outside of a social media app.
Meta AI’s responses tended to be straightforward and minimalistic. While this can be acceptable in certain cases, users may expect a more intuitive and informative experience when engaging in an AI-powered conversation. Nevertheless, Meta AI’s status as a free and current AI model does make it a viable option for quick queries within Meta’s platforms. Nevertheless, it is crucial to consider potential usage limits that Meta may impose on Meta AI.
For users who prefer the convenience of in-app search, Meta AI may serve as a useful tool. However, it is important to recognize that other AI models, like Copilot on Bing, often deliver better results. Users should assess their specific needs and requirements when deciding which AI assistant to employ.
Source link
