自然语言处理 Paddle NLP - 词法分析技术及其应用

自然语言,处理,paddle,nlp,词法,分析,技术,及其,应用 · 浏览次数 : 89

小编点评

## Natural Language Processing (NLP) with PaddleNLP: A Practical Introduction This document provides an introduction to Natural Language Processing (NLP) using PaddleNLP, a pre-trained language model with self-supervised learning capabilities. **Key Points:** * **PaddleNLP** is a framework built on top of transformers that focuses on self-supervised NLP tasks. * **Pre-trained Language Models**: PaddleNLP provides pre-trained models like **BERNIE-CRF** and **ERNIE-Gram** for various NLP tasks. * **Tokenization and Morpheme Analysis**: The framework offers tools for tokenization, part-of-speech tagging, and entity recognition. * **Named Entity Recognition (NER)**: It provides a pipeline for named entity recognition, including question-answering and classification. * **Text Similarity and Question Answering**: The framework includes functionalities for text similarity calculations using **ERNIE-Gram** and **WordNet-based** approaches. * **Pre-training on Special Datasets**: PaddleNLP can be fine-tuned on various datasets like **MSR** for improved performance. * **Sentence and Question Answering**: The framework offers pipelines for structured data and translation tasks, including question-answering and translation. **Benefits of using PaddleNLP:** * **Self-Supervised Learning**: Requires minimal human annotation, making it efficient for large-scale NLP tasks. * **Pre-trained Models**: Provides high-quality pre-trained models for fast and accurate model building. * **Modular Architecture**: Allows customization and tailoring for specific NLP tasks. * **Comprehensive Toolset**: Includes tokenization, POS tagging, NER, text similarity measures, and more. **Overall, PaddleNLP is a powerful and flexible framework for building and training effective NLP models. Its self-supervised learning approach and pre-trained models make it ideal for various NLP tasks, especially when requiring minimal manual annotation.**

正文

词法分析就是利用计算机对自然语言的形态(morphology) 进行分析，判断词的结构和类别等。”简单而言，就是分词并对每个词进行分类，包括：分词、词性标注、实体识别三个任务

问答
知识图谱，类似一张表，里面放了姚明（人）、妻子（属性）对应的是谁
这张表来源于网页挖掘，也是词法分析，进行对应识别

对话
词法分析得到标签，根据动作进行具体操作

对每个字打标签：B 是一个词的开始，I 是一个词的持续，通过一个公式转换成序列标注

OOV(out of vocabulary) 问题：新词，没有见过的词
将预训练模型插进来进行替换，解决OOV问题

预测有4个词，只有3个和标签标注的一样，它的准确就是 3/4
只有一个错了应该是8/9 实际比它要小，所以说以词为力度去计算是一个更严格的标准。

基于上面的评估方法得到的一个准确率和效果，MSR数据集（分词数据集）BERNIE-CRF效果最好。
最长匹配，准确率还可以，还是有很多地方在使用，高效简单