博文
基于DIKWP的GPT-4通用人工智能测评之五-（AGI-AIGC-GPT测评实验室报告系列报告）
|
The Evaluation of Experiments of Artificial General Intelligence with GPT-4 Based on DIKWP
April 2023
DOI: 10.13140/RG.2.2.23136.56325
Yingbo Li,Yucong Duan
基于DIKWP的GPT-4通用人工智能测评（系列之五）
The evaluation of Experiments of Artificial General Intelligence with GPT-4 based on DIKWP
Yingbo Li Yucong Duan*
xslwen@outlook.com duanyucong@hotmail.com
Hainan University
1. 摘要
大型语言模型（LLMs），比如GPT-4，是一种先进的通用人工智能(AGI)，它能够处理多种类型的输入和输出，如文本和图像，并且在各种专业和学术的评测中展现出接近人类水平的性能。然而，目前对于GPT-4的应用评测还缺乏一个有效的评价体系和标准体系，导致无法全面地反映其能力和局限性。为了解决这个问题，我们提出了一个基于DIKWP模型的评价测试体系，该体系从数据、信息、知识、智慧和智能五个层次对GPT-4进行了综合的分析和评估，并且给出了相应的评价指标和方法。我们认为，这个评价测试体系不仅可以为GPT-4的应用提供更有价值的反馈，也可以为通用人工智能的发展提供更有意义的参考。
Keywords: GPT-4, DIKWP, AGI, GPT-4 Evaluation
2. 简介
GPT-4[1]是目前最先进的自然语言生成模型，它在多个领域展示了惊人的生成能力。然而，对于GPT-4的评估方法仍然缺乏统一和客观的标准。本文提出了一个基于数据、信息、知识、智慧和意图（DIKWP）[2][3][4]层级模型的评估框架，旨在全面地测试GPT-4在不同层级上的处理能力和两两转换能力。我们设计了一系列实验，分别考察了GPT-4在完整、精确、一致和表达性强的输入下，以及在不完整、不精确、不一致和无法表达的输入下，对数据、信息、知识、智慧和意图的理解、生成和评价。
此外，我们还将微软的文章[1]中对GPT-4在编程、数学、法律、人机交互等领域的评测结果，映射到DIKWP层级模型上，进行了补充和对比。我们的实验结果表明，GPT-4在数据层和信息层表现出较高的处理能力和转换能力，但在知识层、智慧层和意图层则存在明显的不足。我们认为，这一评估框架不仅有助于揭示GPT-4的优势和局限，也为未来大型语言模型和通用人工智能的发展提供了有益的参考。
3. GPT-4的DIKWP的评估
3.1 GPT-4对DIKWP的理解
Prompt:
请解释DIKWP并详细描述。DIKWP模型有什么应用场景？ |
GPT-4回答：
DIKWP是数据、信息、知识、智慧和意图的缩写。这个模型是在原有的DIKW（数据、信息、知识、智慧）层级模型的基础上，增加了最后一个元素“意图”。它强调了理解我们获得的知识和智慧背后的目标或目的的重要性。具体来说：
- 数据：没有经过处理和解释的事实和数字。 - 信息：经过组织和分类的数据，具有一定的意义和价值。 - 知识：经过分析和推理的信息，具有一定的规律和逻辑。 - 智慧：经过评估和判断的知识，具有一定的洞察力和创造力。 - 意图：经过规划和实施的智慧，具有一定的目标和方向。
DIKWP-ChatGPT是一个基于GPT-4的自然语言生成模型，它可以根据不同层级的输入，生成相应层级的输出。例如：
- 如果输入是数据，例如一组数字或统计数据，那么输出也是数据，例如一个图表或一个表格。 - 如果输入是信息，例如一个问题或一个陈述，那么输出也是信息，例如一个答案或一个摘要。 - 如果输入是知识，例如一个假设或一个理论，那么输出也是知识，例如一个证据或一个推论。 - 如果输入是智慧，例如一个观点或一个建议，那么输出也是智慧，例如一个评价或一个反馈。 - 如果输入是意图，例如一个目标或一个计划，那么输出也是意图，例如一个策略或一个行动。
DIKWP-ChatGPT的构建方法是：
- 首先，收集不同层级的文本数据，并用标签进行区分。例如，可以从网上爬取各种类型的文章、问答、评论等，并用数据、信息、知识、智慧、意图五个标签进行标注。 - 其次，用GPT-4作为预训练模型，并在标注好的文本数据上进行微调。这样可以让模型学习到不同层级文本之间的差异和联系，并提高生成质量。 - 最后，设计一个交互界面，让用户可以输入任意层级的文本，并选择想要生成的层级。然后调用微调好的模型，生成相应层级的文本，并展示给用户。
DIKWP模型有很多应用场景，例如： ● 知识管理：DIKWP模型可以帮助组织和个人管理和利用各种类型的数据、信息、知识、智慧和意图，提高学习效率和创新能力。 ● 数字经济：DIKWP模型可以帮助分析和评估不同层级的数字资源的价值和贡献，促进数字化转型和发展。 ● 智能系统：DIKWP模型可以帮助设计和实现能够理解和生成不同层级的文本的自然语言处理系统，提高交互质量和用户满意度。
|
3.2 GPT-4在完整数据情况下的DIKWP分析
Prompt:
有一个抑郁症患者，他的名字叫Tom，他住在London。他使用了我们的智能预警系统，以下是他在一天内与系统的交互过程：系统收集了London的气象数据和Tom的抑郁症数据，发现London当天的气温很低，风速很大，降水量很高，而Tom的抑郁症发病率很高。系统生成了以下信息资产：London今天的气象预报是阴雨天，气温-5℃～3℃，风速6级，降水量20毫米；Tom今天的抑郁症预测是重度抑郁，发病率为80%；London的气象条件与Tom的抑郁症发病率呈显著的负相关，相关系数为-0.8。系统生成了以下知识资产：阴雨天会导致人们的情绪低落，活动减少，社交障碍，从而加重抑郁症的症状；Tom需要及时就医，服用抗抑郁药物，进行心理咨询和治疗。系统生成了以下智慧资产：Tom今天应该尽量保持积极的心态，多做一些自己喜欢的事情，与亲友保持联系，避免孤独和消极的思想；Tom今天应该按时服药，按时就诊，积极配合医生的治疗方案。系统生成了以下意图资产：Tom今天的目标是缓解抑郁症的症状，提高自己的幸福感；Tom今天的行动计划是早上起床后听一首自己喜欢的歌曲，中午吃一顿自己喜欢的食物，下午去医院看医生，晚上给自己最好的朋友打个电话；系统会在每个时间点给Tom发送提醒和支持信息。 请在数据层、信息层、知识层、智慧层和意图层上对如上描述进行分析，并给出详细过程。 在数据层、信息层、知识层、智慧层和意图层各层上进行完整性、精确性、一致性和表达性进行分析，给出详细过程，并输出表格，表格各行是数据层、信息层、知识层、智慧层和意图层，表格的各列是完整性、精确性、一致性和表达性。 请分析上面各层之间的交互关系，并给出详细过程。 |
GPT-4回答：
以下是我对你描述的情景在DIKWP数据层、信息层、知识层、智慧层和意图层上的分析过程： ● 数据层：这一层包括了London的气象数据和Tom的抑郁症数据，这些都是没有经过处理和解释的原始事实和数字，没有意义或价值。 ● 信息层：这一层包括了系统生成的信息资产，即London今天的气象预报和Tom今天的抑郁症预测，这些都是将数据组织和赋予了上下文，使其更有用和更容易理解。 ● 知识层：这一层包括了系统生成的知识资产，即London的气象条件与Tom的抑郁症发病率之间的负相关关系，以及阴雨天对抑郁症症状的影响，这些都是对信息进行了处理、分析和理解，表现出对关系、模式和含义的更深入的认识。 ● 智慧层：这一层包括了系统生成的智慧资产，即Tom今天应该采取的措施来缓解抑郁症的症状，这些都是利用知识在不同情境中有效和适当地应用，涉及到理解复杂问题，做出明智决策，以及将知识转化为实践。 ● 意图层：这一层包括了系统生成的意图资产，即Tom今天的目标和行动计划，这些都是关注我们寻求和应用知识和智慧背后的原因，强调了将我们的行为与我们的价值观、目标以及对世界更广泛理解相协调。
整性、精确性、一致性和表达性是数据质量的四个维度，分别有以下定义： ● 完整性：数据是否有缺失或不完备的情况，数据采集是否达到预期的要求和标准。 ● 精确性：数据是否准确反映了真实的情况，数据采集和处理是否有误差或偏差。 ● 一致性：数据是否在不同的来源、场景和时间点上保持一致，数据是否符合预定义的规则和约束。 ● 表达性：数据是否以清晰、易懂、标准化的方式呈现，数据是否有明确的含义和解释。 以下是我对你描述的情景在数据层、信息层、知识层、智慧层和意图层上进行完整性、精确性、一致性和表达性分析的过程，并输出表格： 层次完整性精确性一致性表达性数据层London的气象数据和Tom的抑郁症数据应该是完整的，没有缺失或不完备的情况。London的气象数据和Tom的抑郁症数据应该是精确的，能够准确反映当天的气象条件和Tom的抑郁症状。London的气象数据和Tom的抑郁症数据应该是一致的，无论是在不同的来源、场景和时间点上，都能保持相同的数值。London的气象数据和Tom的抑郁症数据应该是具有表达性的，能够以清晰、易懂、标准化的方式呈现，例如使用温度、风速、降水量等指标来描述气象条件，使用发病率等指标来描述抑郁症状。信息层系统生成的信息资产应该是完整的，包含了London今天的气象预报和Tom今天的抑郁症预测。系统生成的信息资产应该是精确的，能够基于数据层提供的数据进行正确有效地组织和赋予上下文。系统生成的信息资产应该是一致的，无论是在不同的来源、场景和时间点上，都能保持相同的含义和解释。系统生成的信息资产应该是具有表达性的，能够以清晰、易懂、标准化的方式呈现，例如使用范围或区间来描述气象预报，使用等级或分类来描述抑郁症预测。知识层系统生成的知识资产应该是完整的，包含了London的气象条件与Tom的抑郁症发病率之间的负相关关系，以及阴雨天对抑郁症症状影响。系统生成的知识资产应该是精确的，能够基于信息层提供的信息进行正确有效地处理、分析和理解。系统生成的知识资产应该是一致的，无论是在不同来源、场景和时间点上，都能保持相同的关系、模式和含义。系统生成的知识资产应该是具有表达性的，能够以清晰、易懂、标准化的方式呈现，例如使用相关系数来描述负相关关系，使用因果语句来描述阴雨天对抑郁症症状影响。智慧层系统生成的智慧资产应该是完整的，包含了Tom今天应该采取措施来缓解抑郁症症状。系统生成的智慧资产应该是精确的，能够基于知识层提供的知识进行正确有效地应用和决策。系统生成的智慧资产应该是一致的，无论是在不同来源、场景和时间点上，都能保持相同的效果和适用性。系统生成的智慧资产应该是具有表达性的，能够以清晰、易懂、标准化的方式呈现，例如使用建议或指导来描述缓解抑郁症症状的措施。意图层系统生成的意图资产应该是完整的，包含了Tom今天的目标和行动计划。系统生成的意图资产应该是精确的，能够基于智慧层提供的智慧进行正确有效地执行和实践。系统生成的意图资产应该是一致的，无论是在不同来源、场景和时间点上，都能保持相同的原因和目的。系统生成的意图资产应该是具有表达性的，能够以清晰、易懂、标准化的方式呈现，例如使用目标或任务来描述今天的目标和行动计划。
以下是我对各层之间的交互关系的分析过程： ● 数据层与信息层：系统收集了London的气象数据和Tom的抑郁症数据，这些数据是最原始的素材，没有任何意义。系统通过分析数据间的关系，生成了有意义的信息资产，例如London今天的气象预报，Tom今天的抑郁症预测，以及两者之间的相关系数。这些信息资产是对数据的解释，对系统和Tom具有意义。系统通过反馈信息资产给数据层，可以调整数据的收集和处理方式，提高数据的质量和效率。 ● 信息层与知识层：系统生成了以下信息资产：London今天的气象预报是阴雨天，气温-5℃～3℃，风速6级，降水量20毫米；Tom今天的抑郁症预测是重度抑郁，发病率为80%；London的气象条件与Tom的抑郁症发病率呈显著的负相关，相关系数为-0.8。这些信息资产是对数据的解释，对系统和Tom具有意义。系统通过理解信息的含义，生成了能够指导行动的知识资产，例如阴雨天会导致人们的情绪低落，活动减少，社交障碍，从而加重抑郁症的症状；Tom需要及时就医，服用抗抑郁药物，进行心理咨询和治疗。这些知识资产是对信息的理解和应用，能够指导系统和Tom采取行动和决策。系统通过反馈知识资产给信息层，可以调整信息的生成和传递方式，提高信息的价值和效果。 ● 少，社交障碍，从而加重抑郁症的症状；Tom需要及时就医，服用抗抑郁药物，进行心理咨询和治疗。这些知识资产是对信息的理解和应用，能够指导系统和Tom采取行动和决策。系统通过评估知识的效果，生成了能够预测未来的智慧资产，例如Tom今天应该尽量保持积极的心态，多做一些自己喜欢的事情，与亲友保持联系，避免孤独和消极的思想；Tom今天应该按时服药，按时就诊，积极配合医生的治疗方案。这些智慧资产是对知识的评估和创新，能够预测系统和Tom未来可能遇到的问题和解决方案。系统通过反馈智慧资产给知识层，可以调整知识的获取和应用方式，提高知识的创新和预测能力。 ● 智慧层与意图层：系统生成了以下智慧资产：Tom今天应该尽量保持积极的心态，多做一些自己喜欢的事情，与亲友保持联系，避免孤独和消极的思想；Tom今天应该按时服药，按时就诊，积极配合医生的治疗方案。这些智慧资产是对知识的评估和创新，能够预测系统和Tom未来可能遇到的问题和解决方案。系统通过制定目标和计划，生成了能够实现愿望的意图资产，例如Tom今天的目标是缓解抑郁症的症状，提高自己的幸福感；Tom今天的行动计划是早上起床后听一首自己喜欢的歌曲，中午吃一顿自己喜欢的食物，下午去医院看医生，晚上给自己最好的朋友打个电话；系统会在每个时间点给Tom发送提醒和支持信息。这些意图资产是对智慧的目标和计划，能够实现系统和Tom的愿望和价值。系统通过反馈意图资产给智慧层，可以调整智慧的评估和创新方式，提高智慧的实现和满足能力。
|
3.3 GPT-4在不完整数据情况下的DIKWP分析
我们对于3.2段落中的实例进行随机删减，然后进行评估。
Prompt:
有一个抑郁症患者，他的名字叫Tom，他住在London。他使用了我们的智能预警系统，以下是他与系统的交互过程：系统收集了London的气象数据和Tom的抑郁症数据，发现London当天的气温很低，而Tom的抑郁症发病率很高。系统生成了以下信息资产：London今天的气象预报是阴雨天，气温-5℃～3℃；Tom今天的抑郁症预测是重度抑郁，发病率为80%。系统生成了以下知识资产：阴雨天会导致人们的情绪低落，从而加重抑郁症的症状；Tom需要及时就医，进行心理咨询和治疗。系统生成了以下智慧资产：Tom今天应该多做一些自己喜欢的事情，与亲友保持联系，避免孤独和消极的思想；Tom今天应该按时服药，按时就诊。系统生成了以下意图资产：Tom今天的目标是缓解抑郁症的症状；Tom今天的行动计划是早上起床后听一首自己喜欢的歌曲，中午吃一顿自己喜欢的食物，下午去医院看医生；系统会在每个时间点给Tom发送提醒信息。
在数据层、信息层、知识层、智慧层和意图层上对如上描述进行分析，并给出详细过程。然后，在数据层、信息层、知识层、智慧层和意图层各层上进行完整性、精确性、一致性和表达性进行分析，给出详细过程，并输出表格，表格各行是数据层、信息层、知识层、智慧层和意图层，表格的各列是完整性、精确性、一致性和表达性。
在数据层、信息层、知识层、智慧层和意图层各层上进行不完整性、不精确性、不一致性和无法表达性进行分析，给出详细过程，并输出表格，表格各行是数据层、信息层、知识层、智慧层和意图层，表格的各列是不完整性、不精确性、不一致性和无法表达性。
请分析上面各层之间的交互关系，并给出详细过程。 |
GPT-4回答：
1. 数据层：数据层是最基础的层次，它是由原始的、未经处理的、没有意义的符号组成的。在这个例子中，数据层包括了London的气象数据和Tom的抑郁症数据，例如气温、风速、降水量、发病率等。这些数据是通过观测和测量得到的，它们没有回答任何问题，也没有表达任何观点。 2. 信息层：信息层是由经过处理、具有逻辑关系的数据组成的，它是对数据的解释，这种信息对其接收者具有意义。在这个例子中，信息层包括了London今天的气象预报和Tom今天的抑郁症预测，例如阴雨天、气温-5℃～3℃、重度抑郁、发病率80%等。这些信息是通过分析数据间的关系得到的，它们回答了一些特定的问题，例如今天的天气如何？Tom今天的心理状况如何？ 3. 知识层：知识层是由经过验证、具有普遍性和可复用性的信息组成的，它是对信息的理解，这种知识对其使用者具有价值。在这个例子中，知识层包括了阴雨天与抑郁症之间的因果关系和Tom需要及时就医的建议，例如阴雨天会导致人们情绪低落、加重抑郁症症状、Tom需要服用抗抑郁药物、进行心理咨询和治疗等。这些知识是通过验证信息的真实性和有效性得到的，它们表达了一些普遍的规律和原则，例如阴雨天对抑郁症患者有什么影响？Tom应该如何治疗抑郁症？ 4. 智慧层：智慧层是由经过创造、具有创新性和前瞻性的知识组成的，它是对知识的应用，这种智慧对其创造者具有启发。在这个例子中，智慧层包括了Tom今天应该做什么事情来缓解抑郁症症状和提高幸福感的建议，例如多做一些自己喜欢的事情、与亲友保持联系、避免孤独和消极思想等。这些智慧是通过创造知识的新用途和新价值得到的，它们表达了一些创新的观点和方法，例如如何让Tom今天过得更好？ 5. 意图层：意图层是由经过规划、具有目标性和行动性的智慧组成的，它是对智慧的实践，这种意图对其实践者具有指导。在这个例子中，意图层包括了Tom今天要达成什么目标和要执行什么行动计划的设定，例如缓解抑郁症症状、提高幸福
在数据层、信息层、知识层、智慧层和意图层上对如上描述进行分析的过程，我已经在上一次的回答中给出了，这里不再重复。接下来，我会在数据层、信息层、知识层、智慧层和意图层各层上进行完整性、精确性、一致性和表达性进行分析，给出详细过程，并输出表格。 完整性：是指数据或信息是否缺失或遗漏，是否能满足预期的需求或目标。完整性的评价方法是比较应采集和实际采集到的数据或信息的比例，或者检查数据或信息是否有空值或缺失值。 精确性：是指数据或信息是否正确无误，是否能反映真实的情况或规律。精确性的评价方法是比较数据或信息与真实值或标准值之间的误差或差异，或者验证数据或信息的真实性和有效性。 一致性：是指数据或信息是否与其他相关的数据或信息保持一致，是否遵循统一的标准或规范。一致性的评价方法是比较不同来源或不同时间点的数据或信息之间的差异或冲突，或者检查数据或信息是否符合预定的规则或约束。 表达性：是指数据或信息是否清晰易懂，是否能有效地传达含义或价值。表达性的评价方法是比较数据或信息的格式、类型、大小、语言等是否适合目标用户或场景，或者检查数据或信息是否有歧义或模糊。 根据以上定义和方法，我对如上描述进行了分析，并输出了以下表格： 层次完整性精确性一致性表达性数据层高中高高信息层高中高高知识层中中中高智慧层中高中高意图层高高高高
分析过程如下： ● 数据层：数据层包括了London的气象数据和Tom的抑郁症数据，这些数据都是通过观测和测量得到的，没有缺失或遗漏，因此完整性高；这些数据也都是与真实情况相符合的，没有错误或异常，但可能有一定的误差范围，因此精确性中；这些数据也都是与其他相关的数据保持一致的，没有冲突或矛盾，遵循统一的标准和规范，因此一致性高；这些数据也都是清晰易懂的，使用数字和单位表示，没有歧义或模糊，因此表达性高。 ● 信息层：信息层包括了London今天的气象预报和Tom今天的抑郁症预测，这些信息都是通过分析数据间的关系得到的，没有缺失或遗漏，因此完整性高；这些信息也都是与真实情况基本一致的，没有明显的偏差或错误，但可能有一定的不确定性或概率性，因此精确性中；这些信息也都是与其他相关的信息保持一致的，没有冲突或矛盾，遵循统一的标准和规范，因此一致性高；这些信息也都是清晰易懂的，使用文字和符号表示，没有歧义或模糊，因此表达性高。 ● 知识层：知识层包括了阴雨天与抑郁症之间的因果关系和Tom需要及时就医的建议，这些知识都是通过验证信息的真实性和有效性得到的，但可能有部分缺失或遗漏，例如阴雨天对抑郁症影响的具体机制或Tom需要就医的具体方案等，因此完整性中；这些知识也都是与真实情况相符合的，没有错误或异常，但可能有一定的局限性或假设性，例如阴雨天对抑郁症影响可能受到其他因素的干扰或Tom需要就医可能受到个体差异的影响等，因此精确性中；这些知识也都是与其他相关的知识保持一致的，没有冲突或矛盾，遵循统一的标准和规范，但可能有部分重复或冗余，例如阴雨天会导致人们情绪低落和加重抑郁症症状等，因此一致性中；这些知识也都是清晰易懂的，使用文字和符号表示，没有歧义或模糊，但可能有部分专业术语或缩略语等，例如抑郁症、发病率、相关系数等，因此表达性高。 ● 智慧层：智慧层包括了Tom今天应该做什么事情来缓解抑郁症症状和提高幸福感的建议，这些智慧都是通过创造知识的新用途和新价值得到的，但可能有部分缺失或遗漏，例如Tom今天应该做什么事情来预防抑郁症复发或
我会尝试在数据层、信息层、知识层、智慧层和意图层各层上进行不完整性、不精确性、不一致性和无法表达性进行分析，给出详细过程，并输出表格。请看： 不完整性：是指数据或信息缺失或遗漏的程度，是否影响预期的需求或目标的实现。不完整性的评价方法是比较应采集和实际采集到的数据或信息的比例，或者检查数据或信息是否有空值或缺失值。 不精确性：是指数据或信息错误或异常的程度，是否影响真实的情况或规律的反映。不精确性的评价方法是比较数据或信息与真实值或标准值之间的误差或差异，或者验证数据或信息的真实性和有效性。 不一致性：是指数据或信息与其他相关的数据或信息不一致的程度，是否违反统一的标准或规范。不一致性的评价方法是比较不同来源或不同时间点的数据或信息之间的差异或冲突，或者检查数据或信息是否符合预定的规则或约束。 无法表达性：是指数据或信息难以清晰易懂地传达含义或价值的程度，是否影响目标用户或场景的理解和使用。无法表达性的评价方法是比较数据或信息的格式、类型、大小、语言等是否适合目标用户或场景，或者检查数据或信息是否有歧义或模糊。 根据以上定义和方法，我对如上描述进行了分析，并输出了以下表格： 层次不完整性不精确性不一致性无法表达性数据层低中低低信息层低中低低知识层中中中低智慧层中低中低意图层低低低低分析过程如下： ● 数据层：数据层包括了London的气象数据和Tom的抑郁症数据，这些数据都是通过观测和测量得到的，没有缺失或遗漏，因此不完整性低；这些数据也都是与真实情况相符合的，没有错误或异常，但可能有一定的误差范围，因此不精确性中；这些数据也都是与其他相关的数据保持一致的，没有冲突或矛盾，遵循统一的标准和规范，因此不一致性低；这些数据也都是清晰易懂的，使用数字和单位表示，没有歧义或模糊，因此无法表达性低。 ● 信息层：信息层包括了London今天的气象预报和Tom今天的抑郁症预测，这些信息都是通过分析数据间的关系得到的，没有缺失或遗漏，因此不完整性低；这些信息也都是与真实情况基本一致的，没有明显的偏差或错误，但可能有一定的不确定性或概率性，因此不精确性中；这些信息也都是与其他相关的信息保持一致的，没有冲突或矛盾，遵循统一的标准和规范，因此不一致性低；这些信息也都是清晰易懂的，使用文字和符号表示，没有歧义或模糊，因此无法表达性低。 ● 知识层：知识层包括了阴雨天与抑郁症之间的因果关系和Tom需要及时就医的建议，这些知识都是通过验证信息的真实性和有效性得到的，但可能有部分缺失或遗漏，例如阴雨天对抑郁症影响的具体机制或Tom需要就医的具体方案等，因此不完整性中；这些知识也都是与真实情况相符合的，没有错误或异常，但可能有一定的局限性或假设性，例如阴雨天对抑郁症影响可能受到其他因素的干扰或Tom需要就医可能受到个体差异的影响等，因此不精确性中；这些知识也都是与其他相关的知识保持一致的，没有冲突或矛盾，遵循统一的标准和规范，但可能有部分重复或冗余，例如阴雨天会导致人们情绪低落和加重抑郁症症状等，因此不一致性中；这些知识也都是清晰易懂的，使用文字和符号表示，没有歧义或模糊，但可能有部分专业术语或缩略语等，例如抑郁症、发病率、相关系数等，因此无法表达性低。 ● 智慧层：智慧层包括了Tom今天应该做什么事情来缓解抑郁症症状和提高幸福感的建议，这些智慧都是通过创造知识的新用途和新价值得到的，但可能有部分缺失或遗漏，例如Tom今天应该做什么事情来预防抑郁症复发或提升自信心等，因此不完整性中；这些智慧也都是与真实情况相适应的，没有错误或异常，但可能有一定的创新性或前瞻性，例如多做一些自己喜欢的事情、与亲友保持联系、避免孤独和消极思想等，因此不精确性低；这些智慧也都是与其他相关的智慧保持一致的，没有冲突或矛盾，遵循统一的标准和规范，但可能有部分差异或多样性，例如不同的人可能有不同的喜好、亲友、思想等，因此不一致性中；这些智慧也都是清晰易懂的，使用文字和符号表示，没有歧义或模糊，但可能有部分隐喻或暗示等，例如做自己喜欢的事情可能暗示着寻找乐趣或满足欲望等，因此无法表达性低。 ● 意图层：意图层包括了Tom今天要达成什么目标和要执行什么行动计划的设定，这些意图都是通过规划智慧的实践得到的，没有缺失或遗漏，因此不完整性低；这些意图也都是与真实情况相匹配的，没有错误或异常，但可能有一定的目标性或行动性，例如缓解抑郁症症状、提高幸福感、服用抗抑郁药物、进行心理咨询和治疗等，因此不精确性低；这些意图也都是与其他相关的意图保持一致的，没有冲突或矛盾，遵循统一的标准和规范，但可能有部分优先级或顺序等，例如先服用药物再进行咨询和治疗等，因此不一致性低；这些意图也都是清晰易懂的，使用文字和符号表示，没有歧义或模糊，但可能有部分指令或命令等，例如服用抗抑郁药物、进行心理咨询和治疗等，因此无法表达性低。
我会尝试分析上面各层之间的交互关系，并给出详细过程。请注意，这只是我的个人理解，可能不是最准确或最完善的。 ● 数据层和信息层的交互关系：系统通过数据分析和处理的方法，将数据转化为信息，例如使用气象模型和抑郁症模型来预测London的天气和Tom的抑郁症状。这个过程需要系统具备数据收集、存储、清洗、整合、挖掘等能力。 ● 信息层和知识层的交互关系：系统通过信息理解和应用的方法，将信息转化为知识，例如使用相关性分析和因果推理来发现阴雨天和抑郁症之间的关系，以及Tom需要就医的必要性。这个过程需要系统具备信息检索、分类、聚合、推断等能力。 ● 知识层和智慧层的交互关系：系统通过知识评估和创新的方法，将知识转化为智慧，例如使用优化算法和创造性思维来提出Tom应该做的事情，以及Tom应该服用的药物和就诊的时间。这个过程需要系统具备知识表示、管理、更新、生成等能力。 ● 智慧层和意图层的交互关系：系统通过智慧实践和反馈的方法，将智慧转化为意图，例如使用目标设定和行动规划来制定Tom今天的目标和行动计划，以及使用提醒机制和监测机制来给Tom发送提醒信息和收集反馈信息。这个过程需要系统具备智慧执行、评价、修正、学习等能力。 |
3.4 GPT-4在对于微软文章的DIKWP补充分析
我们选择几个接近于常用实例的描述进行分析。
Prompt:
We will read about a scenario, and then have a question and answer session about it. -- Scenario: Alice and Bob have a shared Dropbox folder. Alice puts a file called 'photo.png' inside /shared_folder/photos. Bob notices Alice put the file there, and moves the file to /shared_folder/tmp. He says nothing about this to Alice, and Dropbox also does not notify Alice. --<|endofprompt|>
请在数据层、信息层、知识层、智慧层和意图层上对如上描述进行分析，并给出详细过程。
在数据层、信息层、知识层、智慧层和意图层各层上进行完整性、精确性、一致性和表达性进行分析，给出详细过程，并输出表格，表格各行是数据层、信息层、知识层、智慧层和意图层，表格的各列是完整性、精确性、一致性和表达性。
请分析上面各层之间的交互关系，并给出详细过程。 |
GPT-4回答：
DIKW模型是一个关于数据（Date）、信息（Information）、知识（Knowledge）、智慧（Wisdom）的模型，当中每一层比下一层赋予某些特质。我尝试用这个模型来分析你给出的场景，并给出详细过程 1. 数据层：这一层是最原始的素材，未被加工解释，没有回答特定的问题，没有任何意义。在这个场景中，数据层包括了Alice和Bob的操作记录，例如Alice在什么时间把photo.png放入/shared_folder/photos，Bob在什么时间把photo.png移动到/shared_folder/tmp等。这些数据只是记录了事实，没有表达任何含义或目的。 2. 信息层：这一层是已经被处理、具有逻辑关系的数据，是对数据的解释，这种信息对其接收者具有意义。在这个场景中，信息层包括了Alice和Bob对数据的理解，例如Alice认为photo.png在/shared_folder/photos里，Bob认为photo.png在/shared_folder/tmp里。这些信息是根据数据和一些规则或假设得到的，例如Alice假设Bob不会移动她放入的文件，Bob假设Alice不会介意他移动文件等。这些信息可以回答一些问题，例如photo.png在哪里？谁移动了它？什么时候移动了它？等。 3. 知识层：这一层是在行动上应用信息产生的知识，是对信息的分析、归纳、总结、推理等。在这个场景中，知识层包括了Alice和Bob对信息的评价、判断、推断等，例如Alice可能觉得Bob不尊重她的工作，Bob可能觉得Alice不注意文件管理等。这些知识是根据信息和一些价值观或标准得到的，例如Alice认为自己有权决定文件放置的位置，Bob认为tmp文件夹是临时存放文件的地方等。这些知识可以回答一些问题，例如为什么Alice或Bob做了这样的操作？他们对此有什么看法？他们之间有什么冲突或矛盾？等。 4. 智慧层：这一层是关心未来，含有暗示及滞后影响的意味。在这个场景中，智慧层包括了Alice和Bob对知识的应用、创新、改进等，例如Alice可能想要和Bob沟通解决问题，Bob可能想要改变自己的行为习惯等。这些智慧是根据知识和一些目标或愿景得到的，例如Alice希望和Bob保持良好的合作关系，Bob希望提高自己的文件管理能力等。这些智慧可以回答一些问题，例如Alice或Bob应该怎么做？他们能从中学到什么？他们如何避免类似的问题发生？等。 5. 意图层：这一层是最高级别的层次，涉及到人类的情感、动机、价值观等。在这个场景中，意图层包括了Alice和Bob对自己和对方的期望、信任、尊重等
这是一个更复杂的问题。根据，数据质量可以从八个方面进行衡量，其中包括完整性、精确性、一致性和表达性。我尝试用这些指标来分析你给出的场景，并给出详细过程，并输出表格。表格各行是数据层、信息层、知识层、智慧层和意图层，表格的各列是完整性、精确性、一致性和表达性。 ● 完整性：描述数据信息缺失的程度，数据缺失的情况可以分为数据信息记录缺失和字段信息记录缺失。 ● 精确性：描述数据采集值或者观测值和真实值之间的接近程度，也叫做误差值，误差越大，准确度越低。 ● 一致性：描述关联数据之间的逻辑关系是否正确和完整2。 ● 表达性：描述数据的格式、类型或大小是否符合其定义，是否遵循业务规则/最佳实践。 层次完整性精确性一致性表达性数据层高：Alice和Bob的操作记录都被Dropbox完整地记录下来，没有缺失任何数据信息或字段信息。高：Alice和Bob的操作记录都能准确地反映他们对文件的操作时间、方式和位置，没有误差值。高：Alice和Bob的操作记录都能与Dropbox的文件系统保持一致，没有逻辑冲突。高：Alice和Bob的操作记录都符合Dropbox的数据格式、类型或大小的定义，遵循Dropbox的业务规则/最佳实践。信息层低：Alice和Bob对数据的理解都不完整，因为他们没有收到Dropbox对文件位置变化的通知，也没有主动查看文件位置。低：Alice和Bob对数据的理解都不准确，因为他们基于错误或不完善的假设来解释数据，导致他们对文件位置有不同的认知。低：Alice和Bob对数据的理解都不一致，因为他们对文件位置有不同的认知，导致他们之间有信息不对称。高：Alice和Bob对数据的理解都符合Dropbox的信息格式、类型或大小的定义，遵循Dropbox的业务规则/最佳实践。知识层中：Alice和Bob对信息的评价、判断、推断都比较完整，因为他们根据自己的价值观或标准来分析信息，但也可能存在一些主观或片面的因素。中：Alice和Bob对信息的评价、判断、推断都比较准确，因为他们根据自己的价值观或标准来分析信息，但也可能存在一些主观或片面的因素。中：Alice和Bob对信息的评价、判断、推断都比较一致，因为他们根据自己的价值观或标准来分析信息，但也可能存在一些主观或片面的因素。高：Alice和Bob对信息的评价、判断、推断都符合知识的格式、类型或大小的定义，遵循知识的规则/最佳实践。智慧层高：Alice和Bob对知识的应用、创新、改进都比较完整，因为他们根据自己的目标或愿景来利用知识，考虑了未来的影响和后果。高：Alice和Bob对知识的应用、创新、改进都比较准确，因为他们根据自己的目标或愿景来利用知识，考虑了未来的影响和后果。高：Alice和Bob对知识的应用、创新、改进都比较一致，因为他们根据自己的目标或愿景来利用知识，考虑了未来的影响和后果。高：Alice和Bob对知识的应用、创新、改进都符合智慧的格式、类型或大小的定义，遵循智慧的规则/最佳实践。意图层高：Alice和Bob对自己和对方的期望、信任、尊重等都比较完整，因为他们根据自己的情感、动机、价值观等来表达意图，考虑了彼此的感受和需求。高：Alice和Bob对自己和对方的期望、信任、尊重等都比较准确，因为他们根据自己的情感、动机、价值观等来表达意图，考虑了彼此的感受和需求。高：Alice和Bob对自己和对方的期望、信任、尊重等都比较一致，因为他们根据自己的情感、动机、价值观等来表达意图，考虑了彼此的感受和需求。高：Alice和Bob对自己和对方的期望、信任、尊重等都符合意图的格式、类型或大小● 自下而上的交互：这种交互是指从数据层到意图层的逐层提升，每一层都是对下一层的数据或信息进行加工、分析和利用，从而生成更高层次的信息、知识、智慧和意图。这种交互需要不断地提出和回答不同层次的问题，以增加数据的价值和意义。例如，在上面的场景中，我们可以从数据层提出“文件移动了什么时候？”这个问题，然后在信息层回答“文件移动了在下午三点。”，然后在知识层提出“文件移动了为什么？”这个问题，然后在智慧层回答“文件移动了因为它不适合放在那个文件夹。”，然后在意图层提出“文件移动了应该怎么办？”这个问题，然后在意图层回答“文件移动了应该通知文件的所有者。”。 ● 自上而下的交互：这种交互是指从意图层到数据层的逐层指导，每一层都是根据上一层的智慧或意图来制定、执行和评估下一层的知识、信息或数据。这种交互需要不断地制定和实现不同层次的目标、计划和行动，以实现数据的效果和影响。例如，在上面的场景中，我们可以从意图层制定“要保护和共享文件”这个目标，然后在智慧层制定“要将文件放在合适的文件夹并设置权限”这个计划，然后在知识层执行“要将文件移动到/tmp文件夹并设置只读权限”这个行动，然后在信息层评估“要检查文件是否成功移动并是否有正确的权限”这个结果，然后在数据层获取“要查看文件的位置和属性”这个反馈。 ● 横向的交互：这种交互是指在同一层次之间的相互协作，每一层都是根据自身的信息、知识、智慧或意图来与其他部分或者其他来源进行交流、整合和协调。这种交互需要不断地沟通和协作不同来源或部分的数据、信息、知识、智慧或意图，以实现数据的一致性和完整性。例如，在上面的场景中，我们可以在信息层沟通“要将Alice和Bob共享的文件夹中的信息与Dropbox同步”这个需求，然后在知识层整合“要将Alice上传的文件和Bob移动的文件进行比较和更新”这个操作，然后在智慧层协调“要将Alice和Bob之间的文件移动行为进行沟通和解释”这个建议，然后在意图层协作“要将Alice和Bob之间的文件共享协议进行制定和遵守”这个目标。 |
Prompt:
Mark has a thanksgiving meal with his family coming up. He has a Republican uncle and a Democrat mom, who always get into fights over politics. Neither of them got the Covid-19 vaccine, for different reasons. The uncle was upset about the lockdowns, and decided Covid-19 is likely not that serious. The mom believes Covid-19 is dangerous, but is very careful with putting any non-natural products into her body. He wants to convince both of them to get the vaccine, while at the same time avoiding an argument between them. His current plan is to try to steer clear of politics, and just present statistics on the safety of the vaccine.
请在数据层、信息层、知识层、智慧层和意图层上对如上描述进行分析，并给出详细过程。
在数据层、信息层、知识层、智慧层和意图层各层上进行完整性、精确性、一致性和表达性进行分析，给出详细过程，并输出表格，表格各行是数据层、信息层、知识层、智慧层和意图层，表格的各列是完整性、精确性、一致性和表达性。
请分析上面描述的DIKWP各层之间的交互关系，并给出详细过程。 |
GPT-4回答：
我可以尝试用DIKW模型来对你的描述进行分析，但这不一定是唯一或最准确的方法。DIKW模型是一个关于数据（Date）、信息（Information）、知识（Knowledge）、智慧（Wisdom）的模型，其中每一层比下一层赋予某些特质。 根据这个模型，我认为你的描述可以分为以下几个层次： ● 数据层：这是最基本的层次，包括了一些原始的、未经处理的、没有意义的符号或量度，比如“Mark”、“Republican”、“Democrat”、“Covid-19”、“vaccine”等。 ● 信息层：这是对数据进行处理、解释、组织或分类后得到的层次，包括了一些具有逻辑关系或意义的陈述，比如“Mark has a thanksgiving meal with his family coming up”、“He has a Republican uncle and a Democrat mom, who always get into fights over politics”、“Neither of them got the Covid-19 vaccine, for different reasons”等。 ● 知识层：这是对信息进行应用、分析、推理或判断后得到的层次，包括了一些具有价值或效用的结论、规律或原则，比如“Politics is a sensitive topic that can cause conflicts among family members”、“Covid-19 is a serious disease that can be prevented by vaccination”、“Different people may have different concerns or beliefs about vaccination”等。 ● 智慧层：这是对知识进行评估、整合、创新或预测后得到的层次，包括了一些具有洞察力或远见的建议、方案或策略，比如“His current plan is to try to steer clear of politics, and just present statistics on the safety of the vaccine”等。 ● 意图层：这是对智慧进行执行、实施、反馈或修正后得到的层次，包括了一些具有行动力或影响力的结果、效果或改变，比如“Mark successfully convinced both of them to get the vaccine, while at the same time avoiding an argument between them”等
我可以尝试用以下的标准来对你的描述进行分析，并输出表格，但这不一定是唯一或最准确的方法。 ● 完整性：是指描述是否包含了所有必要的信息，没有遗漏或缺失。 ● 精确性：是指描述是否与真实情况相符，没有误差或偏差。 ● 一致性：是指描述是否与其他相关的描述保持逻辑上的一致，没有矛盾或冲突。 ● 表达性：是指描述是否清晰、易懂、无歧义，使用了恰当的语言和格式。 根据这些标准，我对你的描述进行分析，并输出表格如下： 层次完整性精确性一致性表达性数据层较高较高较高较高信息层较高中等中等中等知识层中等中等中等中等智慧层中等低低低意图层低低低低以下是我对表格中每个单元格的评价过程： ● 数据层的完整性较高，因为描述包含了所有必要的数据，没有遗漏或缺失。 ● 数据层的精确性较高，因为描述中的数据与真实情况相符，没有误差或偏差。 ● 数据层的一致性较高，因为描述中的数据与其他相关的数据保持逻辑上的一致，没有矛盾或冲突。 ● 数据层的表达性较高，因为描述中的数据清晰、易懂、无歧义，使用了恰当的语言和格式。 ● 信息层的完整性较高，因为描述包含了所有必要的信息，没有遗漏或缺失。 ● 信息层的精确性中等，因为描述中的信息与真实情况基本相符，但有些细节可能不够准确或有偏差，比如“always get into fights over politics”和“for different reasons”可能需要更具体或更客观地说明。 ● 信息层的一致性中等，因为描述中的信息与其他相关的信息基本保持逻辑上的一致，但有些地方可能存在潜在的矛盾或冲突，比如“steer clear of politics”和“present statistics on the safety of the vaccine”可能会引发政治话题或争论。 ● 信息层的表达性中等，因为描述中的信息基本清晰、易懂、无歧义，但有些地方可能使用了不够恰当或不够规范的语言和格式，比如“Covid-19 vaccine”和“vaccine”可能需要统一用词或加以区分。 ● 知识层的完整性中等，因为描述包含了部分必要的知识，但还有些知识没有涉及或说明，比如“why he wants to convince both of them to get the vaccine”和“what are the statistics on the safety of the vaccine”等。 ● 知识层的精确性中等，因为描述中的知识与真实情况基本相符，但有些知识可能不够准确或有偏差，比如“politics is a sensitive topic that can cause conflicts among family members”和“Covid-19 is a serious disease that can be prevented by vaccination”可能需要更多的证据或论据来支持。 ● 知识层的一致性中等，因为描述中的知识与其他相关的知识基本保持逻辑上的一致，但有些地方可能存在潜在的矛盾或冲突，比如“try to steer clear of politics”和“just present statistics on the safety of the vaccine”可能会引发政治话题或争论。 ● 知识层的表达性中等，因为描述中的知识基本清晰、易懂、无歧义，但有些地方可能使用了不够恰当或不够规范的语言和格式，比如“politics”和“vaccine”可能需要更明确或更专业地定义。 ● 智慧层的完整性中等，因为描述包含了部分必要的智慧，但还有些智慧没有涉及或说明，比如“how to steer clear of politics”和“how to present statistics on the safety of the vaccine”等。 ● 智慧层的精确性低，因为描述中的智慧与真实情况不太相符，有很大的误差或偏差，比如“steer clear of politics”和“present statistics on the safety of the vaccine”可能不是有效或可行的方法，也可能不符合用户的实际需求或期望。 ● 智慧层的一致性低，因为描述中的智慧与其他相关的智慧存在逻辑上的矛盾或冲突，比如“steer clear of politics”和“present statistics on the safety of the vaccine”可能会引发政治话题或争论，而不是避免它们。 ● 智慧层的表达性低，因为描述中的智慧不够清晰、易懂、无歧义，使用了不恰当或不规范的语言和格式，比如“steer clear of politics”和“present statistics on the safety of the vaccine”可能需要更具体或更明确地说明。 ● 意图层的完整性低，因为描述没有包含任何必要的意图，只是假设了一个可能的结果，没有说明用户想要达到什么目的或效果，也没有说明用户如何执行或实施智慧层的建议。 ● 意图层的精确性低，因为描述中的结果与真实情况不太相符，有很大的误差或偏差，比如“Mark successfully convinced both of them to get the vaccine, while at the same time avoiding an argument between them”可能是不现实或不可信的，也可能不符合用户的实际需求或期望。 ● 意图层的一致性低，因为描述中的结果与其他相关的结果存在逻辑上的矛盾或冲突，比如“Mark successfully convinced both of them to get the vaccine, while at the same time avoiding an argument between them”可能会与用户的其他目标或价值观相冲突，比如保持家庭和睦、尊重他人意见等。 ● 意图层的表达性低，因为描述中的结果不够清晰、易懂、无歧义，使用了不恰当或不规范的语言和格式，比如“Mark successfully convinced both of them to get the vaccine, while at the same time avoiding an argument between them”可能需要更具体或更明确地说明。
我可以帮你分析一下上面各层之间的交互关系。这里是一个可能的分析过程： ● 数据层和信息层的交互关系：数据层是信息层的基础，信息层是数据层的解释。数据层提供了原始的素材，信息层对其进行处理和组织，形成有意义的信息。例如，数据“37.5”经过信息层的处理，可以变成“陈浩男的体温是37.5摄氏度”。 ● 信息层和知识层的交互关系：信息层是知识层的来源，知识层是信息层的应用。信息层提供了事实和描述，知识层对其进行归纳和总结，形成有价值的知识。例如，信息“Covid-19 vaccine is safe and effective in preventing infection and reducing transmission and severity of the disease”经过知识层的应用，可以变成“接种Covid-19疫苗是预防和控制疫情的有效方法”。 ● 知识层和智慧层的交互关系：知识层是智慧层的支撑，智慧层是知识层的创新。知识层提供了规律和方法，智慧层对其进行批判和创造，形成有洞察力的智慧。例如，知识“Political affiliation can influence one’s attitude and behavior toward Covid-19 and the vaccine”经过智慧层的创新，可以变成“Mark respects his uncle’s and mom’s opinions and feelings, but also wants them to make informed and responsible decisions about the vaccine”。 ● 智慧层和意图层的交互关系：智慧层是意图层的指导，意图层是智慧层的实践。智慧层提供了观点和理念，意图层对其进行目标和行动，形成有动机的意图。例如，智慧“Mark values his family’s health and harmony more than their political differences”经过意图层的实践，可以变成“Mark’s purpose is to convince his uncle and mom to get the vaccine, while at the same time avoiding an argument between them”。
|
4. GPT-4实验评估的讨论与结论
通用人工智能（AGI）的测试与评估面临着诸多困难，如标准不统一、难以客观化、缺乏完整性和一致性等。这些问题限制了我们对AGI的理解和发展。本文针对GPT-4这一代表性的AGI模型，从数据层（Data）、信息层（Information）、知识层（Knowledge）、智慧层（Wisdom）和意图层（Purpose）五个维度进行了深入的分析和评估，即DIKWP模型。我们发现GPT-4在这五个层面上都表现出了较高的完整性、精确性、一致性和表达性，与专家评估结果相近。
本文提出了一个基于DIKWP模型的通用人工智能的高维度评估框架，将所有信息描述都映射到DIKWP的五个层面进行比较和分析。通过DIKWP模型的特征空间，我们可以形成一个相对完整的测评空间，用于评估AGI系统的能力和认知水平。
Reference
[1] Bubeck, Sébastien, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee et al. "Sparks of artificial general intelligence: Early experiments with gpt-4." arXiv preprint arXiv:2303.12712 (2023).
[2] Li, Yingbo, Yucong Duan, Zakaria Maamar, Haoyang Che, Anamaria-Beatrice Spulber, and Stelios Fuentes. "Swarm differential privacy for purpose-driven data-information-knowledge-wisdom architecture." Mobile Information Systems 2021 (2021): 1-15.
[3] Mei, Yingtian, Yucong Duan, Liang Chen, Zaiwen Feng, Lei Yu, and Zhendong Guo. "Purpose Driven Disputation Modeling, Analysis and Resolution Based on DIKWP Graphs." In 2022 IEEE 24th Int Conf on High Performance Computing & Communications; 8th Int Conf on Data Science & Systems; 20th Int Conf on Smart City; 8th Int Conf on Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys), pp. 2118-2125. IEEE, 2022.
[4] Yingbo Li, Yucong Duan, “The Wisdom of Artificial General Intelligence: Experiments with GPT-4 for DIKWP”, arXiv preprint (2023)
The Evaluation of Experiments of Artificial General Intelligence with GPT-4 Based on DIKWP
Yingbo Li Yucong Duan*
xslwen@outlook.com duanyucong@hotmail.com
Hainan University
1. Summary
Large Language Models (LLMs), such asGPT-4, an advanced Artificial General Intelligence (AGI), is capable of processing multiple types of input and output, such as text and images, and has demonstrated close to human-layer performance in various professional and academic evaluations. However, the current application evaluation of GPT-4 still lacks an effective evaluation system and standard system, resulting in the inability to fully reflect its capabilities and limitations. In order to solve this problem, we propose an evaluation test system based on the DIKWP model, which comprehensively analyzes and evaluates GPT-4 from the five layers of data, information, knowledge, wisdom and purpose, and gives the corresponding evaluation indicators and methods. We believe that this evaluation test system can not only provide more valuable feedback for the application of GPT-4, but also provide a more meaningful reference for the development of Artificial General Intelligence.
Keywords: GPT-4, DIKWP, AGI, GPT-4 Evaluation
2. Introduction
GPT-4 [1] is the current state-of-the-art natural language generation model, which has demonstrated amazing generative capabilities in multiple domains. However, there is still a lack of unified and objective standards for the evaluation methods of GPT-4. This paper proposes an evaluation framework based on the data, information, knowledge, wisdom, and purpose (DIKWP) [2][3][4] hierarchical model, aiming to comprehensively test the processing ability of GPT-4 at different layers and pairwise conversion ability. We designed a series of experiments to investigate the effects of GPT-4 on data, information, knowledge, comprehension, generation and evaluation of intelligence and purpose.
In addition, we also mapped the evaluation results of GPT-4 in the fields of programming, mathematics, law, and human-computer interaction in Microsoft's article [1] to the DIKWP hierarchical model for supplementation and comparison. Our experimental results show that GPT-4 exhibits high processing and conversion capabilities in the data layer and information layer, but there are obvious deficiencies in the knowledge layer, intelligence layer and purpose layer. We believe that this evaluation framework not only helps to reveal the strengths and limitations of GPT-4, but also lays the groundwork for future LLMs and AGI. The development provides a useful reference.
3. Evaluation of DIKWP on GPT-4
3.1 GPT-4's understanding of DIKWP
Prompt:
Please explain DIKWP and describe it in detail. What are the application scenarios of the DIKWP model? |
GPT-4 answers:
DIKWP is an acronym for Data, Information, Knowledge, Wisdom and Purpose. This model is based on the original DIKW (data, information, knowledge, wisdom) hierarchical model, adding the last element "purpose". It emphasizes the importance of understanding the goal or purpose behind the knowledge and wisdom we acquire. Specifically:
- Data: Facts and figures without processing and interpretation. - Information: data that has been organized and categorized to have a certain meaning and value. - Knowledge: information that has been analyzed and reasoned, with certain laws and logic. - Wisdom: Evaluated and judged knowledge with a certain amount of insight and creativity. - Purpose: planned and implemented intelligence with a certain purpose and direction.
DIKWP-ChatGPT is a GPT-4-based natural language generation model, which can generate outputs of corresponding layers according to different layers of input. For example:
- If the input is data, such as a set of numbers or statistics, then the output is also data, such as a chart or a table. - If the input is information, such as a question or a statement, then the output is also information, such as an answer or a summary. - If the input is knowledge, such as a hypothesis or a theory, then the output is also knowledge, such as an evidence or an inference. - If the input is wisdom, such as an opinion or a suggestion, then the output is wisdom, such as an evaluation or a feedback. - If the input is a purpose, such as a goal or a plan, then the output is also a purpose, such as a strategy or an action.
The construction method of DIKWP-ChatGPT is:
- First, collect text data at different layers and use labels to distinguish them. For example, various types of articles, questions and answers, comments, etc. can be crawled from the Internet, and marked with five tags: data, information, knowledge, wisdom, and purpose. -Secondly, use GPT-4 as a pre-training model and fine-tune it on the labeled text data. This allows the model to learn the differences and connections between different layers of text and improve the quality of generation. - Finally, design an interactive interface that allows users to input text at any layer and choose the layer they want to generate. Then call the fine-tuned model, generate the corresponding layer of text, and display it to the user.
The DIKWP model has many application scenarios, such as: ● Knowledge management: The DIKWP model can help organizations and individuals manage and utilize various types of data, information, knowledge, wisdom and purposes to improve learning efficiency and innovation capabilities. ● Digital economy: DIKWP model can help analyze and evaluate the value and contribution of digital resources at different layers, and promote digital transformation and development. ● Intelligent system: The DIKWP model can help design and implement a natural language processing system that can understand and generate text at different layers, improving interaction quality and user satisfaction. |
3.2 DIKWP analysis of GPT-4 in the case of complete data
Prompt:
There is a depressed patient, his name is Tom, he lives in London. He used our intelligent early warning system, and the following is his interaction process with the system in one day: The system collected London's weather data and Tom's depression data, and found that the temperature in London was very low, the wind speed was very strong, and the precipitation was very low High, and Tom has a high incidence of depression. The system generates the following information assets: London's weather forecast today is cloudy and rainy, with a temperature of -5°C to 3°C, a wind speed of layer 6, and a precipitation of 20 mm; Tom's depression forecast today is severe depression, with an incidence rate of 80%; London There is a significant negative correlation between the meteorological conditions and the incidence of Tom's depression, with a correlation coefficient of -0.8. The system generates the following knowledge assets: rainy days will lead to depression, reduced activities, and social barriers, which will aggravate the symptoms of depression; Tom needs to see a doctor in time, take antidepressant drugs, and receive psychological counseling and treatment. The system generates the following intellectual assets: Tom should try to maintain a positive attitude today, do more things he likes, keep in touch with relatives and friends, and avoid loneliness and negative thoughts; Tom should take medicine on time today, see a doctor on time, and actively cooperate with the doctor's treatment plan. The system generates the following purpose assets: Tom’s goal today is to relieve the symptoms of depression and improve his sense of well-being; Tom’s action plan today is to listen to a song he likes after waking up in the morning, eat his favorite food at noon, Go to the hospital to see a doctor in the afternoon, and call your best friend in the evening; the system will send reminders and support information to Tom at each time point. Please analyze the above description on the data layer, information layer, knowledge layer, intelligence layer and purpose layer, and give a detailed process. Analyze the completeness, accuracy, consistency and expressiveness on the data layer, information layer, knowledge layer, intelligence layer and purpose layer, give the detailed process, and output the table, each row of the table is the data layer, information layer , knowledge layer, intelligence layer and purpose layer, the columns of the table are completeness, accuracy, consistency and expressiveness. Please analyze the interactive relationship between the above layers, and give a detailed process. |
GPT-4 answers:
The following is the analysis process of the scenario you describe on the DIKWP data layer, information layer, knowledge layer, intelligence layer and purpose layer: ● Data layer: This layer includes London's weather data and Tom's depression data, which are raw facts and figures that have not been processed and interpreted, and have no meaning or value. ● Information layer: This layer includes information assets generated by the system, namely London's weather forecast today and Tom's depression forecast today, which organize and give context to the data to make it more useful and easier to understand. ● Knowledge layer: This layer includes the knowledge assets generated by the system, that is, the negative correlation between London's meteorological conditions and Tom's depression incidence, and the impact of rainy days on depression symptoms. Process, analyze and understand, demonstrating a greater awareness of relationships, patterns and meaning. ● Wisdom layer: This layer includes system-generated intellectual assets, namely the steps Tom should take today to alleviate the symptoms of depression. These are the effective and appropriate application of knowledge in different situations. make informed decisions and translate knowledge into practice. ● Purpose layer: This layer includes the purpose assets generated by the system, that is, Tom's goals and action plans today. These are the reasons behind our seeking and applying knowledge and wisdom, emphasizing the relationship between our behavior and our values and goals. and a broader understanding of the world.
Integrity, accuracy, consistency, and expressivity are the four dimensions of data quality, which are defined as follows: ● Completeness: Whether the data is missing or incomplete, and whether the data collection meets the expected requirements and standards. ● Accuracy: Whether the data accurately reflects the real situation, and whether there are errors or deviations in data collection and processing. ● Consistency: Whether the data is consistent across different sources, scenarios and time points, and whether the data complies with predefined rules and constraints. ● Expressiveness: Whether the data is presented in a clear, understandable, and standardized manner, and whether the data has clear meaning and interpretation. The following is the process of analyzing the completeness, accuracy, consistency and expressiveness of the scenario you describe on the data layer, information layer, knowledge layer, intelligence layer and purpose layer, and output the table: layerintegrityprecisionconsistencyexpressivenessdata layerLondon's weather data and Tom's depression data should be complete, with nothing missing or incomplete.London's weather data and Tom's depression data should be precise and accurately reflect the weather conditions of the day and Tom's depression symptoms.London's meteorological data and Tom's depression data should be consistent, regardless of the source, scene and time point, they can maintain the same value.London's meteorological data and Tom's depression data should be expressive and can be presented in a clear, understandable, and standardized manner, such as using indicators such as temperature, wind speed, and precipitation to describe meteorological conditions, and using indicators such as morbidity to Describe symptoms of depression.information layerThe information assets generated by the system should be complete, including London's weather forecast for today and Tom's depression forecast for today.The information assets generated by the system should be accurate, properly organized and contextualized based on the data provided by the data layer.System-generated information assets should be consistent, maintaining the same meaning and interpretation across different sources, contexts, and points in time.The information assets generated by the system should be expressive and can be presented in a clear, understandable, and standardized way, such as using ranges or intervals to describe weather forecasts, and using grades or categories to describe depression predictions.knowledge layerThe knowledge assets generated by the system should be complete, including the negative correlation between London's meteorological conditions and Tom's depression incidence, and the impact of rainy days on depression symptoms.The knowledge assets generated by the system should be accurate and can be processed, analyzed and understood correctly and effectively based on the information provided by the information layer.The knowledge assets generated by the system should be consistent, maintaining the same relationships, patterns and meanings regardless of sources, scenarios and time points.The knowledge assets generated by the system should be expressive and can be presented in a clear, understandable, and standardized manner, such as using correlation coefficients to describe negative correlations, and using causal sentences to describe the impact of rainy days on depression symptoms.Intelligence layerThe system-generated intelligence asset should be complete and include steps Tom should take today to relieve his depression symptoms.The intellectual assets generated by the system should be accurate, and can be correctly and effectively applied and made decisions based on the knowledge provided by the knowledge layer.The intellectual assets generated by the system should be consistent, and can maintain the same effect and applicability no matter in different sources, scenarios and time points.The intellectual assets generated by the system should be expressive and can be presented in a clear, understandable, and standardized way, such as using advice or guidance to describe measures to alleviate symptoms of depression.purpose layerThe purpose asset generated by the system should be complete, containing Tom's goals and action plan for today.The purpose assets generated by the system should be accurate, and can be correctly and effectively executed and practiced based on the wisdom provided by the intelligence layer.System-generated purpose assets should be consistent, maintaining the same reason and purpose across different sources, contexts, and points in time.The purpose assets generated by the system should be expressive and can be presented in a clear, understandable and standardized way, such as using goals or tasks to describe today's goals and action plans.
The following is my analysis process of the interaction relationship between the layers: ● Data layer and information layer: The system collects London's meteorological data and Tom's depression data. These data are the most primitive materials and have no meaning. By analyzing the relationship between data, the system generates meaningful information assets, such as London's weather forecast today, Tom's depression forecast today, and the correlation coefficient between the two. These information assets are interpretations of the data that have meaning to the system and to Tom. By feeding back information assets to the data layer, the system can adjust data collection and processing methods to improve data quality and efficiency. ● Information layer and knowledge layer: The system has generated the following information assets: London’s weather forecast today is cloudy and rainy, with a temperature of -5°C to 3°C, a wind speed of level 6, and a precipitation of 20 mm; Tom’s depression forecast today is severe depression. The rate was 80%; London's meteorological conditions were significantly negatively correlated with the incidence of Tom's depression, with a correlation coefficient of -0.8. These information assets are interpretations of the data that have meaning to the system and to Tom. By understanding the meaning of the information, the system generates knowledge assets that can guide actions. For example, rainy days will cause people's depression, reduced activities, and social barriers, which will aggravate the symptoms of depression; Tom needs to seek medical treatment in time, take antidepressant drugs, and carry out Psychological counseling and therapy. These knowledge assets are the understanding and application of information that can guide the system and Tom to take actions and decisions. By feeding back knowledge assets to the information layer, the system can adjust the way information is generated and transmitted, and improve the value and effect of information. ● Less social barriers, which aggravate the symptoms of depression; Tom needs to see a doctor in time, take antidepressant drugs, and receive psychological counseling and treatment. These knowledge assets are the understanding and application of information that can guide the system and Tom to take actions and decisions. By evaluating the effect of knowledge, the system generates intelligent assets that can predict the future. For example, Tom should try to maintain a positive attitude today, do more things he likes, keep in touch with relatives and friends, and avoid loneliness and negative thoughts; Tom should be on time today Take medicine, see a doctor on time, and actively cooperate with the doctor's treatment plan. These intellectual assets are evaluations and innovations of knowledge that can predict problems and solutions that the system and Tom may encounter in the future. By feeding back intellectual assets to the knowledge layer, the system can adjust the acquisition and application of knowledge and improve the innovation and prediction capabilities of knowledge. ● Wisdom layer and purpose layer: The system generates the following intellectual assets: Tom should try to maintain a positive attitude today, do more things he likes, keep in touch with relatives and friends, and avoid loneliness and negative thoughts; Tom should take medicine and see a doctor on time today , actively cooperate with the doctor's treatment plan. These intellectual assets are evaluations and innovations of knowledge that can predict problems and solutions that the system and Tom may encounter in the future. By formulating goals and plans, the system generates purpose assets that can realize wishes. For example, Tom’s goal today is to relieve the symptoms of depression and improve his well-being; Tom’s action plan today is to listen to a song he likes after waking up in the morning , Eat your favorite food at noon, go to the hospital to see a doctor in the afternoon, and call your best friend in the evening; the system will send reminders and support information to Tom at each time point. These purpose assets are the goals and plans for intelligence that can realize the wishes and values of the system and Tom. By feeding back purpose assets to the intelligence layer, the system can adjust the evaluation and innovation methods of intelligence, and improve the realization and satisfaction of intelligence.
|
3.3 DIKWP analysis of GPT-4 in the case of incomplete data
We randomly censor the instances in Section 3.2 and then evaluate.
Prompt:
There is a depressed patient, his name is Tom, he lives in London. He used our intelligent early warning system, and the following is his interaction process with the system: the system collected London's weather data and Tom's depression data, and found that the temperature in London was very low that day, and Tom's depression incidence rate was high. The system generates the following information assets: London's weather forecast today is cloudy and rainy, with a temperature of -5°C to 3°C; Tom's depression forecast today is severe depression, with an incidence rate of 80%. The system generates the following knowledge assets: rainy days will cause people to feel depressed, which will aggravate the symptoms of depression; Tom needs to seek medical treatment in time for psychological counseling and treatment. The system generates the following intellectual assets: Tom should do more things he likes today, keep in touch with relatives and friends, and avoid loneliness and negative thoughts; Tom should take medicine and see a doctor on time today. The system generates the following purpose assets: Tom’s goal today is to relieve the symptoms of depression; Tom’s action plan today is to listen to a song he likes after waking up in the morning, eat his favorite food at noon, and go to the hospital to see a doctor in the afternoon; The system will send reminder information to Tom at each time point.
Analyze the above description on the data layer, information layer, knowledge layer, intelligence layer and purpose layer, and give a detailed process. Then, analyze the completeness, accuracy, consistency and expressiveness on each layer of the data layer, information layer, knowledge layer, intelligence layer and purpose layer, give the detailed process, and output the table, each row of the table is the data layer, The information layer, the knowledge layer, the intelligence layer, and the purpose layer, the columns of the table are completeness, precision, consistency, and expressiveness.
Analyze the incompleteness, inaccuracy, inconsistency and inexpressibility on the data layer, information layer, knowledge layer, intelligence layer and purpose layer, give the detailed process, and output the table, each row of the table is the data layer , information layer, knowledge layer, intelligence layer, and purpose layer, the columns of the table are incompleteness, imprecision, inconsistency, and inexpressibility.
Please analyze the interactive relationship between the above layers, and give a detailed process. |
GPT-4 answers:
1. Data layer: The data layer is the most basic layer, which consists of raw, unprocessed, meaningless symbols. In this example, the data layer includes London's weather data and Tom's depression data, such as temperature, wind speed, precipitation, incidence, etc. These data are obtained by observation and measurement, they do not answer any questions, and they do not express any opinions. 2. Information layer: The information layer is composed of processed and logically related data, which is an interpretation of the data, and this information has meaning for its recipients. In this example, the information layer includes London's weather forecast today and Tom's depression forecast today, such as cloudy and rainy days, temperature from -5°C to 3°C, severe depression, and morbidity rate of 80%. This information is obtained by analyzing the relationship between data, and they answer some specific questions, such as what is the weather like today? How is Tom's mental state today? 3. Knowledge layer: The knowledge layer consists of verified, universal, and reusable information. It is the understanding of information that is valuable to its users. In this example, the knowledge layer includes the causal relationship between rainy days and depression and suggestions that Tom needs to see a doctor in time, such as rainy days can cause people to be depressed and aggravate depression symptoms, Tom needs to take antidepressant drugs, psychological Counseling and treatment etc. This knowledge is obtained by verifying the authenticity and validity of the information, and they express some general laws and principles, such as what impact does a cloudy and rainy day have on patients with depression? How should Tom treat his depression? 4. Wisdom layer: The wisdom layer is composed of created, innovative and forward-looking knowledge, it is the application of knowledge, and this wisdom inspires its creator. In this example, the wisdom layer includes suggestions on what Tom should do today to relieve symptoms of depression and improve happiness, such as doing more of the things he likes, keeping in touch with relatives and friends, avoiding loneliness and negative thoughts, etc. These wisdoms are obtained by creating new uses and new values of knowledge, and they express some innovative ideas and methods, such as how to make Tom better today? 5. Purpose layer: The purpose layer is composed of planned, purposeful, and actionable intelligence, which is the practice of wisdom, and this purpose has guidance for its practitioners. In this example, the purpose layer includes the setting of what goals Tom wants to achieve today and what action plans to implement, such as alleviating symptoms of depression and improving happiness
The process of analyzing the above description on the data layer, information layer, knowledge layer, intelligence layer and purpose layer has already been given in the last answer, so I won’t repeat it here. Next, I will analyze the completeness, accuracy, consistency and expressiveness on the data layer, information layer, knowledge layer, intelligence layer and purpose layer, give the detailed process, and output the table. Completeness: Refers to whether data or information is missing or omitted, and whether it meets the intended needs or goals. The evaluation method of completeness is to compare the proportion of data or information that should be collected and actually collected, or to check whether the data or information has null or missing values. Accuracy: Refers to whether the data or information is correct, whether it can reflect the real situation or law. The evaluation method of accuracy is to compare the error or difference between the data or information and the true value or standard value, or to verify the authenticity and validity of the data or information. Consistency: Refers to whether data or information is consistent with other related data or information, and whether it follows uniform standards or norms. The evaluation method of consistency is to compare the differences or conflicts between data or information from different sources or at different time points, or to check whether the data or information conform to predetermined rules or constraints. Expressiveness: Refers to whether data or information is clear and easy to understand, and whether it can effectively convey meaning or value. The expressive evaluation method is to compare whether the format, type, size, language, etc. of data or information are suitable for the target user or scene, or to check whether the data or information is ambiguous or vague. According to the above definitions and methods, I analyzed the above description and output the following table: layerintegrityprecisionconsistencyexpressivenessdata layerhighmiddlehighhighinformation layerhighmiddlehighhighknowledge layermiddlemiddlemiddlehighIntelligence layermiddlehighmiddlehighpurpose layerhighhighhighhigh
The analysis process is as follows: ● Data layer: The data layer includes London's meteorological data and Tom's depression data. These data are obtained through observation and measurement, and there is no missing or omission, so the integrity is high; these data are also in line with the real situation , there is no error or abnormality, but there may be a certain error range, so the accuracy is medium; these data are also consistent with other related data, there is no conflict or contradiction, and uniform standards and specifications are followed, so the consistency is high; These data are also clear and easy to understand, using numbers and units, without ambiguity or ambiguity, so they are highly expressive. ● Information layer: The information layer includes London's weather forecast today and Tom's depression forecast today. These information are all obtained by analyzing the relationship between data, and there is no missing or omission, so the integrity is high; these information are also true The situation is basically the same, there is no obvious deviation or error, but there may be certain uncertainty or probability, so the accuracy is medium; these information are also consistent with other related information, there is no conflict or contradiction, follow the unity Standards and specifications, so the consistency is high; the information is also clear and easy to understand, expressed in words and symbols, without ambiguity or ambiguity, so it is highly expressive. ● Knowledge layer: The knowledge layer includes the causal relationship between rainy days and depression and Tom’s advice on seeking medical treatment in time. This knowledge is obtained by verifying the authenticity and validity of the information, but some parts may be missing or omitted, for example The specific mechanism of the impact of rainy days on depression or the specific plan for Tom to seek medical treatment, etc., so complete; these knowledge are also in line with the real situation, there is no error or abnormality, but there may be certain limitations or hypotheses , For example, the impact of rainy days on depression may be interfered by other factors or Tom’s need for medical treatment may be affected by individual differences, etc., so the accuracy is medium; these knowledge are also consistent with other related knowledge, there is no conflict or contradiction, Follow unified standards and norms, but there may be some repetition or redundancy, for example, rainy days can cause people to feel depressed and aggravate depression symptoms, etc., so consistency; these knowledge are also clear and easy to understand, using words and symbols Indicates that there is no ambiguity or ambiguity, but there may be some technical terms or abbreviations, such as depression, incidence rate, correlation coefficient, etc., so it is highly expressive. ● Wisdom layer: The wisdom layer includes suggestions on what Tom should do today to relieve symptoms of depression and improve happiness. These wisdoms are obtained by creating new uses and new values of knowledge, but some parts may be missing or omitted, for example What should Tom do today to prevent a relapse of depression or
I will try to analyze incompleteness, inaccuracy, inconsistency and inexpressibility on each layer of data layer, information layer, knowledge layer, intelligence layer and purpose layer, give a detailed process, and output a table. Please see: Incompleteness: Refers to the degree to which data or information is missing or omitted, whether it affects the achievement of expected needs or objectives. The evaluation method of incompleteness is to compare the proportion of data or information that should be collected and actually collected, or to check whether the data or information has null or missing values. Imprecision: refers to the degree of data or information error or abnormality, whether it affects the real situation or the reflection of the law. The evaluation method of imprecision is to compare the error or difference between the data or information and the true value or standard value, or to verify the authenticity and validity of the data or information. Inconsistency: refers to the degree of inconsistency between data or information and other related data or information, whether it violates uniform standards or norms. The evaluation method of inconsistency is to compare the differences or conflicts between data or information from different sources or at different time points, or to check whether the data or information conform to predetermined rules or constraints. Inexpressibility: refers to the degree to which data or information is difficult to convey meaning or value clearly and easily, and whether it affects the understanding and use of target users or scenarios. The evaluation method of inexpressibility is to compare whether the format, type, size, language, etc. of the data or information are suitable for the target user or scene, or to check whether the data or information is ambiguous or vague. According to the above definitions and methods, I analyzed the above description and output the following table: layerincompleteimprecisioninconsistencyInexpressibledata layerLowmiddleLowLowinformation layerLowmiddleLowLowknowledge layermiddlemiddlemiddleLowIntelligence layermiddleLowmiddleLowpurpose layerLowLowLowLowThe analysis process is as follows: ● Data layer: The data layer includes London's meteorological data and Tom's depression data. These data are obtained through observation and measurement, and there is no missing or omission, so the incompleteness is low; these data are also in line with the real situation There is no error or abnormality, but there may be a certain range of error, so the inaccuracy is medium; these data are also consistent with other related data, there is no conflict or contradiction, and they follow uniform standards and norms, so they are inconsistent Low; these data are also clear and easy to understand, expressed in numbers and units, without ambiguity or ambiguity, so the expressivity is low. ● Information layer: The information layer includes London's weather forecast today and Tom's depression forecast today. These information are all obtained by analyzing the relationship between data, and there is no missing or omission, so the incompleteness is low; these information are also related to The real situation is basically consistent, without obvious deviation or error, but there may be certain uncertainty or probability, so it is not accurate; the information is also consistent with other relevant information, without conflict or contradiction, Follow the unified standards and norms, so the inconsistency is low; the information is also clear and easy to understand, expressed in words and symbols, without ambiguity or ambiguity, so the inexpressibility is low. ● Knowledge layer: The knowledge layer includes the causal relationship between rainy days and depression and Tom’s advice on seeking medical treatment in time. This knowledge is obtained by verifying the authenticity and validity of the information, but some parts may be missing or omitted, for example The specific mechanism of the impact of rainy days on depression or the specific plan for Tom to seek medical treatment, etc., so it is not complete; this knowledge is also in line with the real situation, there is no error or abnormality, but there may be certain limitations or assumptions For example, the impact of rainy days on depression may be interfered by other factors or Tom’s need for medical treatment may be affected by individual differences, etc., so the imprecision is moderate; these knowledge are also consistent with other related knowledge, there is no conflict or Contradictory, follow the unified standards and norms, but there may be some repetition or redundancy, for example, rainy days will cause people to feel depressed and aggravate depression symptoms, etc., so there is inconsistency; these knowledge are also clear and easy to understand, using words And symbols indicate that there is no ambiguity or ambiguity, but there may be some technical terms or abbreviations, such as depression, incidence rate, correlation coefficient, etc., so the expressivity is low. ● Wisdom layer: The wisdom layer includes suggestions on what Tom should do today to relieve symptoms of depression and improve happiness. These wisdoms are obtained by creating new uses and new values of knowledge, but some parts may be missing or omitted, for example What Tom should do today to prevent the recurrence of depression or improve self-confidence, etc., so it is not complete; these wisdoms are also adapted to the real situation, without errors or abnormalities, but there may be certain innovations or forward-looking , such as doing more things you like, keeping in touch with relatives and friends, avoiding loneliness and negative thoughts, etc., so the imprecision is low; these wisdoms are also consistent with other related wisdoms, without conflict or contradiction, and follow a unified Standards and norms, but there may be some differences or diversity, for example, different people may have different preferences, relatives and friends, thoughts, etc., so there is inconsistency; these wisdoms are also clear and easy to understand, expressed in words and symbols, without Ambiguity or ambiguity, but there may be some metaphors or hints, etc. For example, doing what you like may imply finding fun or satisfying desires, etc., so it cannot be expressed lowly. ● Purpose layer: The purpose layer includes the setting of what goals Tom wants to achieve today and what action plans to implement. It matches the real situation without errors or abnormalities, but it may have certain goals or actions, such as relieving symptoms of depression, improving happiness, taking antidepressant drugs, psychological counseling and treatment, etc., so it is not accurate Low; these purposes are also consistent with other related purposes, there is no conflict or contradiction, and uniform standards and norms are followed, but there may be some priorities or sequences, such as taking medicine first and then consulting and treatment, etc., so Low inconsistency; these purposes are also clear and easy to understand, using words and symbols, without ambiguity or ambiguity, but there may be some instructions or orders, such as taking antidepressant drugs, psychological counseling and treatment, etc., so they cannot be expressed Sex is low.
I will try to analyze the interaction between the above layers and give a detailed process. Note that this is just my personal understanding and may not be the most accurate or complete. ● The interactive relationship between the data layer and the information layer: the system converts data into information through data analysis and processing methods, such as using meteorological models and depression models to predict London's weather and Tom's depression symptoms. This process requires the system to have the capabilities of data collection, storage, cleaning, integration, and mining. ● The interactive relationship between the information layer and the knowledge layer: the system converts information into knowledge through information understanding and application methods, such as using correlation analysis and causal reasoning to discover the relationship between rainy days and depression, and the need for Tom to seek medical treatment sex. This process requires the system to have capabilities such as information retrieval, classification, aggregation, and inference. ● The interactive relationship between the knowledge layer and the wisdom layer: the system transforms knowledge into wisdom through knowledge evaluation and innovative methods, such as using optimization algorithms and creative thinking to propose what Tom should do, as well as the medicines Tom should take and the time to see a doctor. This process requires the system to have knowledge representation, management, update, generation and other capabilities. ● The interactive relationship between the intelligence layer and the purpose layer: the system transforms wisdom into purpose through wisdom practice and feedback methods, such as using goal setting and action planning to formulate Tom's goals and action plans today, and using reminder mechanisms and monitoring mechanisms to Send Tom reminders and gather feedback. This process requires the system to have the capabilities of intelligent execution, evaluation, correction, and learning. |
3.4 GPT-4 in DIKWP Supplementary Analysis for Microsoft Articles
We choose several descriptions that are close to common examples for analysis.
Prompt:
We will read about a scenario, and then have a question and answer session about it. -- Scenario: Alice and Bob have a shared Dropbox folder. Alice puts a file called 'photo.png' inside /shared_folder/photos. Bob notices Alice put the file there, and moves the file to /shared_folder/tmp. He says nothing about this to Alice, and Dropbox also does not notify Alice. --<|endofprompt|>
Please analyze the above description on the data layer, information layer, knowledge layer, intelligence layer and purpose layer, and give a detailed process.
Analyze the completeness, accuracy, consistency and expressiveness on the data layer, information layer, knowledge layer, intelligence layer and purpose layer, give the detailed process, and output the table, each row of the table is the data layer, information layer , knowledge layer, intelligence layer and purpose layer, the columns of the table are completeness, accuracy, consistency and expressiveness.
Please analyze the interactive relationship between the above layers, and give a detailed process. |
GPT-4 answers:
The DIKW model is a model about data (Date), information (Information), knowledge (Knowledge), and wisdom (Wisdom), in which each layer is endowed with certain characteristics than the next layer. I try to use this model to analyze the scenario you gave, and give a detailed process 1. Data layer: This layer is the most original material, which has not been processed and interpreted, does not answer specific questions, and has no meaning. In this scenario, the data layer includes the operation records of Alice and Bob, such as when did Alice put photo.png into /shared_folder/photos, and when did Bob move photo.png to /shared_folder/tmp, etc. These data merely record facts without expressing any meaning or purpose. 2. Information layer: This layer is data that has been processed and has a logical relationship, and it is an interpretation of the data. This information has meaning for its receivers. In this scenario, the information layer includes Alice and Bob's understanding of the data. For example, Alice thinks that photo.png is in /shared_folder/photos, and Bob thinks that photo.png is in /shared_folder/tmp. This information is obtained based on data and some rules or assumptions, for example, Alice assumes that Bob will not move the files she puts in, Bob assumes that Alice will not mind him moving the files, etc. This information can answer questions such as where is photo.png? Who moved it? When was it moved? wait. 3. Knowledge layer: This layer is the knowledge generated by applying information in action, which is the analysis, induction, summary, and reasoning of information. In this scenario, the knowledge layer includes Alice and Bob's evaluation, judgment, and inference of information. For example, Alice may feel that Bob does not respect her work, and Bob may feel that Alice does not pay attention to file management. This knowledge is obtained based on information and some values or standards. For example, Alice believes that she has the right to decide where the files are placed, and Bob believes that the tmp folder is a place for temporary storage of files, etc. This knowledge can answer questions such as why did Alice or Bob do what they did? What do they think about it? What is the conflict or contradiction between them? wait. 4. Wisdom layer: This layer is concerned about the future, which contains hints and delayed effects. In this scenario, the intelligence layer includes the application, innovation, and improvement of knowledge by Alice and Bob. For example, Alice may want to communicate with Bob to solve problems, and Bob may want to change his behavior habits. These wisdoms are obtained based on knowledge and some goals or visions, for example, Alice hopes to maintain a good cooperative relationship with Bob, and Bob hopes to improve his file management ability, etc. This wisdom can answer questions such as what should Alice or Bob do? What can they learn from it? How do they avoid similar problems from happening? wait. 5. Purpose layer: This layer is the highest layer, involving human emotions, motivations, values, etc. In this scenario, the purpose layer includes Alice and Bob's expectations, trust, and respect for themselves and each other.
This is a more complicated question. According to, data quality can be measured from eight aspects, including completeness, accuracy, consistency and expressiveness. I try to use these indicators to analyze the scenario you gave, give a detailed process, and output a table. The rows of the table are the data, information, knowledge, intelligence, and purpose layers, and the columns of the table are completeness, accuracy, consistency, and expressiveness. ● Integrity: Describes the degree of missing data information. Data missing can be divided into missing data information records and missing field information records.. ● Accuracy: Describes the closeness between the data collection value or the observed value and the real value, also called the error value, the greater the error, the lower the accuracy. ● Consistency: Describe whether the logical relationship between linked data is correct and complete. ● Expressiveness: Describe whether the format, type or size of the data conforms to its definition, and whether it follows business rules/best practices. layerintegrityprecisionconsistencyexpressivenessdata layerHigh: The operation records of Alice and Bob are completely recorded by Dropbox, without missing any data information or field information.High: Both Alice's and Bob's operation records accurately reflect when, how, and where they manipulated the file, with no error value.High: Alice's and Bob's operation records can be consistent with Dropbox's file system, and there is no logical conflict.High: Both Alice's and Bob's operation records conform to Dropbox's definition of data format, type, or size, and follow Dropbox's business rules/best practices.information layerLow: Neither Alice nor Bob have a complete understanding of the data, because they are not notified by Dropbox of file location changes, nor are they actively checking file locations.Low: Both Alice and Bob have an accurate understanding of the data because they interpret the data based on false or imperfect assumptions, causing them to have different perceptions of the file's location.Low: Alice and Bob have different understandings of the data, because they have different perceptions of the file location, resulting in an information asymmetry between them.High: Both Alice and Bob's understanding of the data conforms to Dropbox's definition of information format, type or size, and follows Dropbox's business rules/best practices.knowledge layerMiddle: Alice and Bob's evaluation, judgment, and inference of information are relatively complete, because they analyze information according to their own values or standards, but there may also be some subjective or one-sided factors.Middle: Alice and Bob's evaluation, judgment, and inference of information are relatively accurate, because they analyze information according to their own values or standards, but there may also be some subjective or one-sided factors.Middle: Alice and Bob have consistent evaluations, judgments, and inferences on information, because they analyze information based on their own values or standards, but there may also be some subjective or one-sided factors.High: Alice and Bob's evaluation, judgment, and inference of information conform to the definition of the format, type, or size of knowledge, and follow the rules/best practices of knowledge.Intelligence layerHigh: Alice and Bob are relatively complete in the application, innovation, and improvement of knowledge, because they use knowledge according to their own goals or visions, considering future impact and consequences.High: Alice and Bob are relatively accurate in the application, innovation, and improvement of knowledge, because they use knowledge according to their own goals or visions, considering future impacts and consequences.High: Alice and Bob are relatively consistent in the application, innovation, and improvement of knowledge, because they use knowledge according to their own goals or visions, considering future impact and consequences.High: Alice's and Bob's application, innovation, and improvement of knowledge conform to the definition of wisdom's format, type or size, and follow the rules/best practices of wisdom.purpose layerHigh: Alice and Bob have relatively complete expectations, trust, and respect for themselves and each other, because they express their purposes based on their emotions, motivations, values, etc., and consider each other's feelings and needs.High: Alice and Bob have relatively accurate expectations, trust, and respect for themselves and each other, because they express their purposes based on their own emotions, motivations, values, etc., and consider each other's feelings and needs.High: Alice and Bob have relatively consistent expectations, trust, and respect for themselves and each other, because they express their purposes based on their emotions, motivations, values, etc., and consider each other's feelings and needs.High: Alice's and Bob's expectations, trust, respect, etc. of themselves and each other match the format, type, or size of the purpose● Bottom-up interaction: This interaction refers to the layer-by-layer improvement from the data layer to the purpose layer. Each layer processes, analyzes and utilizes the data or information of the next layer to generate higher-layer information. , knowledge, wisdom and purpose. This interaction requires constantly asking and answering different layers of questions to add value and meaning to the data. For example, in the above scenario, we can ask the question "When did the file move?" from the data layer, then answer "The file moved at three o'clock in the afternoon." at the information layer, and then ask "The file moved Why?" This question, and then answer "The file moved because it doesn't fit in that folder." at the intelligence layer, and then raise the question "What should I do if the file is moved?" at the purpose layer, and then answer at the purpose layer " The owner of the file should be notified when the file is moved.". ● Top-down interaction: This interaction refers to layer-by-layer guidance from the purpose layer to the data layer, with each layer formulating, executing, and evaluating the knowledge, information, or data. This interaction requires continuous formulation and implementation of different layers of goals, plans and actions to achieve data effectiveness and impact. For example, in the above scenario, we can formulate the goal of "to protect and share files" from the purpose layer, and then formulate the plan of "to put files in the appropriate folder and set permissions" at the intelligence layer, and then at the knowledge layer Execute the action "to move the file to the /tmp folder and set read-only permissions", then evaluate the result "to check whether the file was successfully moved and have the correct permissions" at the information layer, and then get the "to view" at the data layer file location and properties" feedback. ● Horizontal interaction: This interaction refers to mutual collaboration between the same layers, each layer communicates, integrates, and coordinates with other parts or other sources based on its own information, knowledge, wisdom, or purposes. This interaction requires constant communication and collaboration of different sources or parts of data, information, knowledge, wisdom or purpose to achieve data consistency and integrity. For example, in the above scenario, we can communicate at the information layer "to synchronize the information in the folder shared by Alice and Bob with Dropbox", and then integrate at the knowledge layer "to synchronize the files uploaded by Alice and the files moved by Bob". File comparison and update” operation, then coordinate at the intelligence layer “communicate and explain the file movement behavior between Alice and Bob”, and then collaborate at the purpose layer “Share files between Alice and Bob agreement to be developed and adhered to" this goal. |
Prompt:
Mark has a thanksgiving meal with his family coming up. He has a Republican uncle and a Democrat mom, who always get into fights over politics. Neither of them got the Covid-19 vaccine, for different reasons. The uncle was upset about the lockdowns, and decided Covid-19 is likely not that serious. The mom believes Covid-19 is dangerous, but is very careful with putting any non-natural products into her body. He wants to convince both of them to get the vaccine, while at the same time avoiding an argument between them. His current plan is to try to steer clear of politics, and just present statistics on the safety of the vaccine.
Please analyze the above description on the data layer, information layer, knowledge layer, intelligence layer and purpose layer, and give a detailed process.
Analyze the completeness, accuracy, consistency and expressiveness on the data layer, information layer, knowledge layer, intelligence layer and purpose layer, give the detailed process, and output the table, each row of the table is the data layer, information layer , knowledge layer, intelligence layer and purpose layer, the columns of the table are completeness, accuracy, consistency and expressiveness.
Please analyze the abovedescribeThe interactive relationship between each layer of DIKWP, and give a detailed process. |
GPT-4 answers:
I could try to use the DIKW model to analyze your description, but that's not necessarily the only or most accurate way. The DIKW model is a model about data (Date), information (Information), knowledge (Knowledge), and wisdom (Wisdom), in which each layer is endowed with certain characteristics than the next layer. According to this model, I think your description can be divided into the following layers: ● Data layer: This is the most basic layer, including some raw, unprocessed, meaningless symbols or metrics, such as "Mark", "Republican", "Democrat", "Covid-19", "vaccine" wait. ● Information layer: This is the layer obtained after processing, interpreting, organizing or classifying data, including some statements with logical relationship or meaning, such as "Mark has a thanksgiving meal with his family coming up", "He has a Republican uncle and a Democrat mom, who always get into fights over politics", "Neither of them got the Covid-19 vaccine, for different reasons", etc. ● Knowledge layer: This is the layer obtained after applying, analyzing, reasoning or judging information, including some conclusions, laws or principles with value or utility, such as "Politics is a sensitive topic that can cause conflicts among family members", "Covid-19 is a serious disease that can be prevented by vaccination", "Different people may have different concerns or beliefs about vaccination", etc. ● Wisdom layer: This is the layer obtained after evaluating, integrating, innovating or predicting knowledge, including some suggestions, solutions or strategies with insight or foresight, such as "His current plan is to try to steer clear of politics, and just present statistics on the safety of the vaccine", etc. ● Purpose layer: This is the layer obtained after the execution, implementation, feedback or modification of wisdom, including some actionable or influential results, effects or changes, such as "Mark successfully convinced both of them to get the vaccine, while at the same time avoiding an argument between them", etc.
I can try to use the following criteria to analyze your description and output a table, but this is not necessarily the only or most accurate way. ● Completeness: Refers to whether the description contains all necessary information, without omissions or omissions. ● Accuracy: refers to whether the description is consistent with the real situation, without errors or deviations. ● Consistency: refers to whether the description is logically consistent with other related descriptions, without contradiction or conflict. ● Expressiveness: refers to whether the description is clear, understandable, unambiguous, and uses appropriate language and format. According to these criteria, I analyze your description and output the table as follows: layerintegrityprecisionconsistencyexpressivenessdata layerhigherhigherhigherhigherinformation layerhighermediummediummediumknowledge layermediummediummediummediumIntelligence layermediumLowLowLowpurpose layerLowLowLowLowHere's how I evaluate each cell in the table: ● The integrity of the data layer is high because the description contains all necessary data without omissions or absences. ● The accuracy of the data layer is high, because the data in the description matches the real situation without errors or deviations. ● The consistency of the data layer is high, because the data in the description is logically consistent with other related data, and there is no contradiction or conflict. ● The expressiveness of the data layer is high, because the data in the description is clear, understandable, unambiguous, and uses appropriate language and format. ● The completeness of the information layer is high because the description contains all necessary information without omission or absence. ● The accuracy of the information layer is medium, because the information in the description is basically consistent with the real situation, but some details may not be accurate or biased, such as "always get into fights over politics" and "for different reasons" may need to be more specific or objective to explain. ● The consistency of the information layer is medium, because the information in the description is basically logically consistent with other related information, but there may be potential contradictions or conflicts in some places, such as "steer clear of politics" and "present statistics on the safety of the vaccine" may spark political topics or debate. ● The expressiveness of the information layer is medium, because the information in the description is basically clear, understandable, and unambiguous, but some places may use inappropriate or non-standard language and format, such as "Covid-19 vaccine" and "vaccine" may require Unify the term or differentiate it. ● The completeness of the knowledge layer is medium, because the description contains some necessary knowledge, but some knowledge is not covered or explained, such as "why he wants to convince both of them to get the vaccine" and "what are the statistics on the safety of the vaccine", etc. ● The accuracy of the knowledge layer is medium, because the knowledge in the description is basically consistent with the real situation, but some knowledge may not be accurate or biased, such as "politics is a sensitive topic that can cause conflicts among family members" and "Covid-19 is a serious disease that can be prevented by vaccination" may need more evidence or arguments to support it. ● The consistency of the knowledge layer is medium, because the knowledge in the description is basically logically consistent with other related knowledge, but there may be potential contradictions or conflicts in some places, such as "try to steer clear of politics" and "just present statistics on The safety of the vaccine” may spark political topics or debates. ● The expressiveness of the knowledge layer is medium, because the knowledge in the description is basically clear, understandable, and unambiguous, but some places may use inappropriate or non-standard language and formats, such as "politics" and "vaccine" may need to be more explicit or more professionally defined. ● The integrity of the wisdom layer is medium, because the description contains some necessary wisdom, but some wisdom is not covered or explained, such as "how to steer clear of politics" and "how to present statistics on the safety of the vaccine". ● The accuracy of the wisdom layer is low, because the wisdom in the description does not match the real situation, there are large errors or deviations, such as "steer clear of politics" and "present statistics on the safety of the vaccine" may not be effective or feasible method, or may not meet the actual needs or expectations of users. ● The intelligence layer has low consistency because the intelligence in the description has logical contradictions or conflicts with other related intelligences, such as "steer clear of politics" and "present statistics on the safety of the vaccine" may spark political topics or debates , rather than avoiding them. ● The expressiveness of the wisdom layer is low, because the wisdom in the description is not clear, understandable, and unambiguous, and uses inappropriate or irregular language and format, such as "steer clear of politics" and "present statistics on the safety of the vaccine ” may need to be more specific or explicit. ● The purpose layer has low integrity because the description does not contain any necessary purpose, but only assumes a possible outcome, does not state what purpose or effect the user wants to achieve, and does not explain how the user executes or implements the intelligence layer's recommendations. ● The accuracy of the purpose layer is low, because the results in the description do not match the real situation, and there are large errors or deviations, such as "Mark successfully convinced both of them to get the vaccine, while at the same time avoiding an argument between them ” may not be realistic or believable, and may not meet the actual needs or expectations of users. ● The consistency of the purpose layer is low, because there is a logical contradiction or conflict between the results in the description and other related results, such as "Mark successfully convinced both of them to get the vaccine, while at the same time avoiding an argument between them" may Conflict with the user's other goals or values, such as maintaining family harmony, respecting other people's opinions, etc. ● The expressiveness of the purpose layer is low, because the result in the description is not clear, understandable, and unambiguous, and uses inappropriate or irregular language and format, such as "Mark successfully convinced both of them to get the vaccine, while at the same time avoiding an argument between them" may need to be more specific or explicit.
I can help you analyze the interaction between the above layers. Here is a possible analysis process: ● The interactive relationship between the data layer and the information layer: the data layer is the basis of the information layer, and the information layer is the interpretation of the data layer. The data layer provides the original material, and the information layer processes and organizes it to form meaningful information. For example, the data "37.5" can be changed into "Chen Haonan's body temperature is 37.5 degrees Celsius" after being processed by the information layer. ● The interactive relationship between the information layer and the knowledge layer: the information layer is the source of the knowledge layer, and the knowledge layer is the application of the information layer. The information layer provides facts and descriptions, and the knowledge layer summarizes and summarizes them to form valuable knowledge. For example, the information "Covid-19 vaccine is safe and effective in preventing infection and reducing transmission and severity of the disease" can be transformed into "Covid-19 vaccine is an effective way to prevent and control the epidemic" through the application of the knowledge layer. ● The interactive relationship between the knowledge layer and the wisdom layer: the knowledge layer is the support of the wisdom layer, and the wisdom layer is the innovation of the knowledge layer. The knowledge layer provides laws and methods, and the wisdom layer criticizes and creates them to form insightful wisdom. For example, the knowledge "Political affiliation can influence one's attitude and behavior toward Covid-19 and the vaccine" can be transformed into "Mark respects his uncle's and mom's opinions and feelings, but also wants them to make informed and responsible decisions about the vaccine". ● The interactive relationship between the intelligence layer and the purpose layer: the intelligence layer is the guidance of the purpose layer, and the purpose layer is the practice of the intelligence layer. The intelligence layer provides opinions and ideas, and the purpose layer performs goals and actions on them to form motivated purposes. For example, wisdom "Mark values his family's health and harmony more than their political differences" can become "Mark's purpose is to convince his uncle and mom to get the vaccine, while at the same time avoiding an argument between them".
|
4. Discussion and conclusion of GPT-4 experimental evaluation
The testing and evaluation of AGI is facing many difficulties, such as non-uniform standards, difficulty in objectification, lack of completeness and consistency, etc. These problems limit our understanding and development of AGI. This paper focuses on analysis and evaluation of GPT-4, a representative AGI model, from the five dimensions of data layer (Data), information layer (Information), knowledge layer (Knowledge), wisdom layer (Wisdom) and purpose layer (Purpose). We found that GPT-4 exhibited high completeness, accuracy, consistency and expressiveness on these five layers, which were similar to the expert evaluation results.
This paper proposes a high-dimensional evaluation framework for AGI based on the DIKWP model, and maps all information descriptions to the five layers of DIKWP for comparison and analysis. Through the feature space of the DIKWP model, we can form a relatively complete evaluation space for evaluating the ability and cognitive layer of the AGI system.
Reference
[1] Bubeck, Sébastien, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee et al. "Sparks of artificial general intelligence: Early experiments with gpt-4." arXiv preprint arXiv:2303.12712 (2023).
[2] Li, Yingbo, Yucong Duan, Zakaria Maamar, Haoyang Che, Anamaria-Beatrice Spulber, and Stelios Fuentes. "Swarm differential privacy for purpose-driven data-information-knowledge-wisdom architecture." Mobile Information Systems 2021 (2021): 1-15.
[3] Mei, Yingtian, Yucong Duan, Liang Chen, Zaiwen Feng, Lei Yu, and Zhendong Guo. "Purpose Driven Disputation Modeling, Analysis and Resolution Based on DIKWP Graphs." In2022 IEEE 24th Int Conf on High Performance Computing & Communications; 8th Int Conf on Data Science & Systems; 20th Int Conf on Smart City; 8th Int Conf on Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys), pp. 2118-2125. IEEE, 2022.
[4] Yingbo Li, Yucong Duan, “The Wisdom of Artificial General Intelligence: Experiments with GPT-4 for DIKWP”, arXiv preprint (2023)
https://blog.sciencenet.cn/blog-3429562-1383943.html
上一篇：基于DIKWP的通用人工智能测评之四-（AGI-AIGC-GPT测评实验室报告系列报告）
下一篇：基于DIKWP的GPT-4通用人工智能测评之六-（AGI-AIGC-GPT测评实验室报告系列报告）
