Machine learning is a multi-billion dollar business that seems to have a lot of potential, but there are some risks, and here's how to avoid the most common machine learning mistakes.
隨著機器學習技術的應用越來越廣泛，它正在許多領域佔據一席之地。研究公司 Fortune Business Insights 預測，全球機器學習市場將從 2023 年的 260.3 億美元擴大到 2030 年的 2259.1 億美元。機器學習的用例包括產品推薦、圖像識別、欺詐檢測、語言翻譯、診斷工具等。
As a subset of artificial intelligence, machine learning refers to the process of utilizing algorithms on large data sets to make predictive decisions. The potential benefits of machine learning may seem limitless, but it also comes with some risks.
We asked technology leaders and analysts about the most common ways they have seen machine learning projects fail. Here's what they told us.
10 Ways Machine Learning Projects Fail :
Artificial intelligence hallucinations
Model bias
Legal and moral hazard
Poor data quality
Model overfitting and underfitting
Legacy system integration issues
Performance and scalability issues
Lack of transparency and trust
Lack of knowledge in specific areas
There is a shortage of machine learning skills
Artificial intelligence hallucinations
In machine learning, hallucinations are when large language models (LLMs) perceive patterns or objects that don't exist or are not perceptible to humans. When hallucinations manifest themselves in generated code or chatbot responses, they result in useless output.
"In today's environment, problems like hallucinations are at an all-time high." Camden Swita, head of AI/machine learning at unified data platform vendor New Relic, said he noted that recent studies have shown that the vast majority of machine learning engineers are observing signs of hallucinations.
Swita says that to eliminate illusions, you can't just focus on generating content. "Instead, developers must emphasize summarizing tasks and take advantage of advanced technologies such as Retrieval Enhanced Generation (RAG), which can greatly reduce hallucinations." In addition, pinning the output of AI to real, verified, and canonical data sources reduces the likelihood of misleading information.
Model bias
Businesses need to be aware of model bias, which is the presence of systematic errors in the model that can lead to incorrect predictions on an ongoing basis. These errors can come from the selection of the algorithm used, the training data, the selection of features used when creating the model, or other issues.
"The data used to train machine learning models must contain accurate population representation and diverse datasets," said Sheldon Arora, CEO of StaffDNA, a company that uses AI to help match job seekers in the healthcare industry. "Overrepresentation of any one particular group can lead to an inaccurate representation of the entire group. Continuous monitoring of model performance ensures fair representation of all demographic groups. ”
Tackling bias is key to success in the modern AI landscape, and best practices include implementing continuous monitoring, alerting mechanisms, and content filtering to help proactively identify and correct biased content, Swita said. "With these approaches, businesses can develop AI frameworks that prioritize proven content."
Tackling bias requires a dynamic approach, including continuous improvement of systems, to keep up with rapidly evolving patterns, Swita says, and a well-tailored strategy to eliminate bias is needed.
Legal and moral hazard
There are certain legal and ethical hazards associated with machine learning. Legal risks include discrimination, data privacy violations, security breaches, and intellectual property rights violations due to model bias. These and other risks can have an impact on developers and users of machine learning systems.
Moral hazard includes potential harm or exploitation, misuse of data, lack of transparency, and lack of accountability. Decisions made based on machine learning algorithms can have a negative impact on individuals, even if they weren't intended.
Swita reiterates that models and outputs must be built on trusted, validated, and regulated data. By complying with regulations and standards regarding data use and privacy, businesses can reduce the legal and ethical risks associated with machine learning, he said.
Poor data quality
Like any technology that relies on data to produce positive results, machine learning requires high-quality data to be successful. Poor data quality can lead to model flaws and unacceptable results.
Market analysis by research firm Gartner shows that most organizations have problems with their data, with many citing unreliable and inaccurate data as the number one reason for not trusting AI. Peter Krensky, senior director and analyst in the Analytics and Artificial Intelligence team at Gartner, said, "Leaders and practitioners are struggling between preparing data for prototypes and ensuring they are ready for the real world. ”
"To meet these challenges, businesses must be pragmatic and adopt a management approach that is consistent with the intended purpose of the data, promoting trust and adaptability," says Krensky.
Marin Cristian-Ovidiu, CEO of online gaming site Online Games, said machine learning relies heavily on data quality. Bad data [leads to] inaccurate predictions, he says, just as a recommender system promotes irrelevant content because of biased input.
To solve this problem, organizations must adopt robust data cleansing processes and diverse data sets, Cristian-Ovidiu says. Arora added that high-quality data is essential for building reliable machine learning models. He said that data should be cleaned regularly and pre-processing techniques should be employed to ensure accuracy, and that good data is the key to effectively training models and obtaining reliable outputs.
In addition to data that is inaccurate or otherwise flawed, businesses may also find themselves dealing with data points that don't make sense for a particular task. Teams can identify irrelevant data with features such as data visualization and statistical analysis. Once this data is identified, it can be removed from the dataset before the model is trained.
Model overfitting and underfitting
In addition to the data used, the model itself can also be a source of failure in a machine learning project.
Overfitting occurs when the model is trained too close to the training set. This causes the model to perform poorly on new data. Models are often trained on known datasets to make predictions on new data, but overfitted models don't generalize well to new data and therefore fail to accomplish the expected tasks.
Elvis Sun, a software engineer at Google and founder of PressPulse, said: "If a model performs well on training data but not well on new data, then the model is called an overfit model. "PressPulse is a company that uses artificial intelligence to help journalists and experts connect." When a model becomes too complex, it 'memorizes' the training data instead of figuring out patterns. ”
Underfitting is when the model is too simplistic to accurately capture the relationships between input and output variables. The result is that the model performs poorly on training data and new data. "Underfitting occurs when a model is too simplistic to represent the true complexity of the data. ”
Sun says the team can address these issues using cross-validation, regularization, and appropriate model architecture. Cross-validation, he says, can demonstrate a model's ability to generalize by assessing how well it performs on retained data. Businesses can strike a balance between the complexity and generalization of their models to produce reliable, accurate machine learning solutions. Regularization techniques such as L2 or L0 discourage overfitting by penalizing the complexity of the model and promoting simpler, more widely applicable solutions, he said.
Legacy system integration issues
Integrating machine learning into legacy IT systems may require assessing the adaptability of existing infrastructure to machine learning, creating integration processes, using application programming interfaces (APIs) for data exchange, and other steps. Whatever the involving, it's critical to ensure that existing systems can support new machine learning-based products.
Damien Filiatrault, founder and CEO of Scalable Path, a software talent agency, said: "Legacy systems may not be able to meet the infrastructure requirements of machine learning tools, which can lead to inefficiencies or incomplete integrations. ”
"For example, a demand forecasting machine learning model may not be compatible with the inventory management software currently used by retail companies," Filiatrault says. Therefore, for such an implementation to take place, the system must be thoroughly evaluated. ”
According to Filiatrault, machine learning models can be integrated with legacy systems through APIs and microservices that enable them to interact with each other. "In addition, data scientists and IT teams collaborate cross-functionally and roll out in phases to ensure smoother adoption."
Performance and scalability issues
Scalability is another concern, especially as machine learning is used over time. If the system is unable to maintain its performance and efficiency while handling significantly larger data sets, increased complexity, and higher computational demands, the results may be unacceptable.
Machine learning models must be able to handle growing volumes of data without significant degradation in performance or speed. "Unless companies use scalable cloud computing resources, they won't be able to handle fluctuating data volumes," Arora said. Depending on the size of the dataset, a more complex model may be required. Distributed computing frameworks allow parallel computation of large data sets. ”
Lack of transparency and trust
Machine learning applications tend to operate like a "black box," which makes it challenging to interpret their results, Filiatrault said.
"In healthcare and other environments where confidentiality is important, this lack of transparency can undermine user confidence," Filiatrault said. Whenever possible, using explainable models or employing explanatory frameworks such as SHAP (SHapley Additive exPlanations) may help to address this issue. ”
Filiatrault said that proper documentation and visualization of the decision-making process can also help build user trust and comply with regulations to guarantee the ethical use of AI.
Cristian-Ovidiu says, "Models often only give results, but don't explain why. For example, a player engagement model might improve retention, but it doesn't know what played a role. Use an easy-to-understand model and ask an expert to check the results. ”
Lack of knowledge in specific areas
Effective use of machine learning often requires a deep understanding of the problem or domain being solved, Sun says. Companies that lack the right talent on their teams may find this domain expertise a significant problem.
"Depending on factors such as industry-specific data structures, business procedures, and laws and regulations, machine learning solutions may or may not succeed," Sun said. ”
To bridge this gap, machine learning professionals must work closely with people in the field. "By combining the technical expertise of the machine learning team with the contextual knowledge of domain experts, organizations can create better machine learning models," he says, "and this collaboration can take the form of problem definition, training dataset creation, or establishing a continuous feedback loop during model development and deployment." ”
There is a shortage of machine learning skills
Like many other areas of technology, organizations face a shortage of needed machine learning skills.
"Talent challenges often stem from skills shortages and the need to bridge the gap between technical and non-technical stakeholders," Krensky said. Many organizations struggle with change management, which is critical to driving adoption and aligning teams with evolving capabilities. ”
Organizations are overcoming these challenges by focusing on reskilling and facilitating cross-discipline collaboration, and embracing new roles, such as AI translators, Krensky said.