什么是LangChain

LangChain是一个强大的框架，旨在帮助开发人员使用语言模型构建端到端的应用程序。它提供了一套工具、组件和接口，可简化创建由大型语言模型 (LLM) 和聊天模型提供支持的应用程序的过程。LangChain 可以轻松管理与语言模型的交互，将多个组件链接在一起，并集成额外的资源，例如 API 和数据库。
官方文档：https://python.langchain.com/en/latest/

如何使用LangChain

要使用 LangChain，开发人员首先要导入必要的组件和工具，例如 LLMs, chat models, agents, chains, 内存功能。这些组件组合起来创建一个可以理解、处理和响应用户输入的应用程序。
LangChain 为特定用例提供了多种组件，例如个人助理、文档问答、聊天机器人、查询表格数据、与 API 交互、提取、评估和汇总。

LangChain的模型

LangChain model 是一种抽象，表示框架中使用的不同类型的模型。LangChain 中的模型主要分为三类：
1. LLM（大型语言模型）：这些模型将文本字符串作为输入并返回文本字符串作为输出。它们是许多语言模型应用程序的支柱。
2. 聊天模型( Chat Model)：聊天模型由语言模型支持，但具有更结构化的 API。他们将聊天消息列表作为输入并返回聊天消息。这使得管理对话历史记录和维护上下文变得容易。
3. 文本嵌入模型(Text Embedding Models)：这些模型将文本作为输入并返回表示文本嵌入的浮点列表。这些嵌入可用于文档检索、聚类和相似性比较等任务。

LangChain 的主要特点

LangChain 旨在为六个主要领域的开发人员提供支持：
1. LLM 和提示：LangChain 使管理提示、优化它们以及为所有 LLM 创建通用界面变得容易。此外，它还包括一些用于处理 LLM 的便捷实用程序。
2. 链(Chain)：这些是对 LLM 或其他实用程序的调用序列。LangChain 为链提供标准接口，与各种工具集成，为流行应用提供端到端的链。
3. 数据增强生成：LangChain 使链能够与外部数据源交互以收集生成步骤的数据。例如，它可以帮助总结长文本或使用特定数据源回答问题。
4. Agents：Agents 让 LLM 做出有关行动的决定，采取这些行动，检查结果，并继续前进直到工作完成。LangChain 提供了代理的标准接口，多种代理可供选择，以及端到端的代理示例。
5. 内存：LangChain 有一个标准的内存接口，有助于维护链或代理调用之间的状态。它还提供了一系列内存实现和使用内存的链或代理的示例。
6. 评估：很难用传统指标评估生成模型。这就是为什么 LangChain 提供提示和链来帮助开发者自己使用 LLM 评估他们的模型。

使用示例

提示模板（PromptTemplate）

支持我们来自定义提示模板，不会将用户输入直接发送到 LLM，而是接收用户的输入，然后构造一个prompt将其发送给 LLM。

from langchain.prompts import PromptTemplate

prompt = PromptTemplate(
    input_variables=["paperTitle", "area", "nums", "keywords"],
    template="我想让你充当{area}领域的专家，针对于{paperTitle}这个主题，帮我撰写一篇{nums}字的文献综述。包含的关键字如下：{keywords}"
)



print(prompt.format(paperTitle="大学生网红产品消费行为研究", area="经济学", nums=500, keywords="大学生、网红、消费、行为研究"))
# 输出：我想让你充当经济学领域的专家，针对于大学生网红产品消费行为研究这个主题，帮我撰写一篇500字的文献综述。包含的关键字如下：大学生、网红、消费、行为研究

LLM Chain

我们可以把初始化好的LLM和Template，组合成一个chain

from langchain_openai import ChatOpenAI

# 1. 初始化llm
llm = ChatOpenAI()

# 这里其实就可以直接调用llm了
llm.invoke("how can langsmith help with testing?")

# 2. 这里可以初始化一个模板
from langchain_core.prompts import ChatPromptTemplate
prompt = ChatPromptTemplate.from_messages([
    ("system", "你是世界上最好的科幻小说作者"),
    ("user", "{input}")
])

# 3. 采用这种方式可以将llm和template组合成一个chain
chain = prompt | llm

# 4. 调用组合后的chain
chain.invoke({"input": "你对LangSmith了解吗？"})

Retrieval Chain（基于检索的chain）

当你需要让LLM回答一个问题时（比如 “how can langsmith help with testing?”），有时候你要提供的提示信息可能太多，直接传递给语言模型可能不够有效。为了提供更多上下文，LangChain提供了通过检索（retrieval）的方式获取相关的文档，并将这些文档传递给语言模型。
在这个过程中，我们使用一个检索器（Retriever）来查找相关的文档，然后将这些文档传递给提示模板。检索器可以由各种数据支持，比如一个 SQL 表、互联网上的数据等，但在官网的Demo里，将使用一个向量存储（vector store）来作为检索器。
首先，我们需要加载要索引的数据。为了做到这一点，我们将使用 WebBaseLoader（这个应该是LangChain内部实现的一个爬虫工具，要依赖于BeautifulSoup
pip install beautifulsoup4
引入WebBaseLocder来爬取内容

from langchain_community.document_loaders import WebBaseLoader

loader = WebBaseLoader("https://docs.smith.langchain.com/overview")
docs = loader.load()
print(docs)
# [Document(page_content='\n\n\n\n\nLangSmith Overview and User Guide | 🦜️🛠️ LangSmith\n\n\n\n\n\nSkip to main content🦜️🛠️ LangSmith DocsPython DocsJS/TS DocsSearchGo to AppLangSmithOverviewTracingTesting & EvaluationOrganizationsHubLangSmith CookbookOverviewOn this pageLangSmith Overview and User GuideBuilding reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.Over the past two months, we at LangChain have been building and using LangSmith with the goal of bridging this gap. This is our tactical user guide to outline effective ways to use LangSmith and maximize its benefits.On by default\u200bAt LangChain, all of us have LangSmith’s tracing running in the background by default. On the Python side, this is achieved by setting environment variables, which we establish whenever we launch a virtual environment or open our bash shell and leave them set. The same principle applies to most JavaScript environments through process.env1.The benefit here is that all calls to LLMs, chains, agents, tools, and retrievers are logged to LangSmith. Around 90% of the time we don’t even look at the traces, but the 10% of the time that we do… it’s so helpful. We can use LangSmith to debug:An unexpected end resultWhy an agent is loopingWhy a chain was slower than expectedHow many tokens an agent usedDebugging\u200bDebugging LLMs, chains, and agents can be tough. LangSmith helps solve the following pain points:What was the exact input to the LLM?\u200bLLM calls are often tricky and non-deterministic. The inputs/outputs may seem straightforward, given they are technically string → string (or chat messages → chat message), but this can be misleading as the input string is usually constructed from a combination of user input and auxiliary functions.Most inputs to an LLM call are a combination of some type of fixed template along with input variables. These input variables could come directly from user input or from an auxiliary function (like retrieval). By the time these input variables go into the LLM they will have been converted to a string format, but often times they are not naturally represented as a string (they could be a list, or a Document object). Therefore, it is important to have visibility into what exactly the final string going into the LLM is. This has helped us debug bugs in formatting logic, unexpected transformations to user input, and straight up missing user input.To a much lesser extent, this is also true of the output of an LLM. Oftentimes the output of an LLM is technically a string but that string may contain some structure (json, yaml) that is intended to be parsed into a structured representation. Understanding what the exact output is can help determine if there may be a need for different parsing.LangSmith provides a straightforward visualization of the exact inputs/outputs to all LLM calls, so you can easily understand them.If I edit the prompt, how does that affect the output?\u200bSo you notice a bad output, and you go into LangSmith to see what\'s going on. You find the faulty LLM call and are now looking at the exact input. You want to try changing a word or a phrase to see what happens -- what do you do?We constantly ran into this issue. Initially, we copied the prompt to a playground of sorts. But this got annoying, so we built a playground of our own! When examining an LLM call, you can click the Open in Playground button to access this playground. Here, you can modify the prompt and re-run it to observe the resulting changes to the output - as many times as needed!Currently, this feature supports only OpenAI and Anthropic models and works for LLM and Chat Model calls. We plan to extend its functionality to more LLM types, chains, agents, and retrievers in the future.What is the exact sequence of events?\u200bIn complicated chains and agents, it can often be hard to understand what is going on under the hood. What calls are being made? In what order? What are the inputs and outputs of each call?LangSmith\'s built-in tracing feature offers a visualization to clarify these sequences. This tool is invaluable for understanding intricate and lengthy chains and agents. For chains, it can shed light on the sequence of calls and how they interact. For agents, where the sequence of calls is non-deterministic, it helps visualize the specific sequence for a given run -- something that is impossible to know ahead of time.Why did my chain take much longer than expected?\u200bIf a chain takes longer than expected, you need to identify the cause. By tracking the latency of each step, LangSmith lets you identify and possibly eliminate the slowest components.How many tokens were used?\u200bBuilding and prototyping LLM applications can be expensive. LangSmith tracks the total token usage for a chain and the token usage of each step. This makes it easy to identify potentially costly parts of the chain.Collaborative debugging\u200bIn the past, sharing a faulty chain with a colleague for debugging was challenging when performed locally. With LangSmith, we\'ve added a “Share” button that makes the chain and LLM runs accessible to anyone with the shared link.Collecting examples\u200bMost of the time we go to debug, it\'s because something bad or unexpected outcome has happened in our application. These failures are valuable data points! By identifying how our chain can fail and monitoring these failures, we can test future chain versions against these known issues.Why is this so impactful? When building LLM applications, it’s often common to start without a dataset of any kind. This is part of the power of LLMs! They are amazing zero-shot learners, making it possible to get started as easily as possible. But this can also be a curse -- as you adjust the prompt, you\'re wandering blind. You don’t have any examples to benchmark your changes against.LangSmith addresses this problem by including an “Add to Dataset” button for each run, making it easy to add the input/output examples a chosen dataset. You can edit the example before adding them to the dataset to include the expected result, which is particularly useful for bad examples.This feature is available at every step of a nested chain, enabling you to add examples for an end-to-end chain, an intermediary chain (such as a LLM Chain), or simply the LLM or Chat Model.End-to-end chain examples are excellent for testing the overall flow, while single, modular LLM Chain or LLM/Chat Model examples can be beneficial for testing the simplest and most directly modifiable components.Testing & evaluation\u200bInitially, we do most of our evaluation manually and ad hoc. We pass in different\ninputs, and see what happens. At some point though, our application is performing\nwell and we want to be more rigorous about testing changes. We can use a dataset\nthat we’ve constructed along the way (see above). Alternatively, we could spend some\ntime constructing a small dataset by hand. For these situations, LangSmith simplifies\ndataset uploading.Once we have a dataset, how can we use it to test changes to a prompt or chain? The most basic approach is to run the chain over the data points and visualize the outputs. Despite technological advancements, there still is no substitute for looking at outputs by eye. Currently, running the chain over the data points needs to be done client-side. The LangSmith client makes it easy to pull down a dataset and then run a chain over them, logging the results to a new project associated with the dataset. From there, you can review them. We\'ve made it easy to assign feedback to runs and mark them as correct or incorrect directly in the web app, displaying aggregate statistics for each test project.We also make it easier to evaluate these runs. To that end, we\'ve added a set of evaluators to the open-source LangChain library. These evaluators can be specified when initiating a test run and will evaluate the results once the test run completes. If we’re being honest, most of these evaluators aren\'t perfect. We would not recommend that you trust them blindly. However, we do think they are useful for guiding your eye to examples you should look at. This becomes especially valuable as the number of data points increases and it becomes infeasible to look at each one manually.Human Evaluation\u200bAutomatic evaluation metrics are helpful for getting consistent and scalable information about model behavior;\nhowever, there is still no complete substitute for human review to get the utmost quality and reliability from your application.\nLangSmith makes it easy to manually review and annotate runs through annotation queues.These queues allow you to select any runs based on criteria like model type or automatic evaluation scores, and queue them up for human review. As a reviewer, you can then quickly step through the runs, viewing the input, output, and any existing tags before adding your own feedback.We often use this for a couple of reasons:To assess subjective qualities that automatic evaluators struggle with, like creativity or humorTo sample and validate a subset of runs that were auto-evaluated, ensuring the automatic metrics are still reliable and capturing the right informationAll the annotations made using the queue are assigned as "feedback" to the source runs, so you can easily filter and analyze them later.\nYou can also quickly edit examples and add them to datasets to expand the surface area of your evaluation sets or to fine-tune a model for improved quality or reduced costs.Monitoring\u200bAfter all this, your app might finally ready to go in production. LangSmith can also be used to monitor your application in much the same way that you used for debugging. You can log all traces, visualize latency and token usage statistics, and troubleshoot specific issues as they arise. Each run can also be assigned string tags or key-value metadata, allowing you to attach correlation ids or AB test variants, and filter runs accordingly.We’ve also made it possible to associate feedback programmatically with runs. This means that if your application has a thumbs up/down button on it, you can use that to log feedback back to LangSmith. This can be used to track performance over time and pinpoint under performing data points, which you can subsequently add to a dataset for future testing — mirroring the debug mode approach.We’ve provided several examples in the LangSmith documentation for extracting insights from logged runs. In addition to guiding you on performing this task yourself, we also provide examples of integrating with third parties for this purpose. We\'re eager to expand this area in the coming months! If you have ideas for either -- an open-source way to evaluate, or are building a company that wants to do analytics over these runs, please reach out.Exporting datasets\u200bLangSmith makes it easy to curate datasets. However, these aren’t just useful inside LangSmith; they can be exported for use in other contexts. Notable applications include exporting for use in OpenAI Evals or fine-tuning, such as with FireworksAI.To set up tracing in Deno, web browsers, or other runtime environments without access to the environment, check out the FAQs.↩PreviousLangSmithNextTracingOn by defaultDebuggingWhat was the exact input to the LLM?If I edit the prompt, how does that affect the output?What is the exact sequence of events?Why did my chain take much longer than expected?How many tokens were used?Collaborative debuggingCollecting examplesTesting & evaluationHuman EvaluationMonitoringExporting datasetsCommunityDiscordTwitterGitHubDocs CodeLangSmith SDKPythonJS/TSMoreHomepageBlogCopyright © 2024 LangChain, Inc.\n\n\n\n', metadata={'source': 'https://docs.smith.langchain.com/overview', 'title': 'LangSmith Overview and User Guide | 🦜️🛠️ LangSmith', 'description': 'Building reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.', 'language': 'en'})]
# 这里输出的docs就是网页上的内容

然后我们需要将这些文档索引到一个向量存储中。这需要一些组件，包括嵌入模型（embedding model）和向量存储。
1. 检索器（Retriever）：想象一下你在图书馆找书的时候，你可能不会把整个图书馆的书都拿去看。相反，你可能会咨询图书管理员或者查找图书馆的目录，找到与你问题相关的书籍。在这里，检索器就像图书馆的目录，它帮助我们找到与问题相关的信息。
2. 向量存储（Vector Store）：这就像一个大仓库，里面存放着各种各样的信息。每个信息都用一个向量表示，就像是仓库中的一个箱子。通过这些向量，我们可以快速找到存储库中与我们关心的主题相关的信息。
3. 嵌入模型（Embedding Model）：这就像是一个翻译工人，它将文档中的文字翻译成计算机能够理解的形式，也就是向量。这样，计算机就能够更好地处理和比较文档，找到最相关的信息。

根据文档构建向量

使用 OpenAIEmbeddings 初始化一个嵌入模型，用于将文档转化为向量。

1
2
3

from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings()

使用一个简单的本地向量存储 FAISS 来建立索引.

1	pip install faiss-cpu

构建向量存储

from langchain_community.vectorstores import FAISS
from langchain.text_splitter import RecursiveCharacterTextSplitter

# 文本分割器
text_splitter = RecursiveCharacterTextSplitter()
# 将刚刚获取的docs分割，输出是一个List<String>的类型
documents = text_splitter.split_documents(docs)
# 根据嵌入模型和分割后的文档，转换为向量
vector = FAISS.from_documents(documents, embeddings)

构建链

构建一个链，该链接受问题和检索到的文档，生成答案：

from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_template("""Answer the following question based only on the provided context:

<context>
{context}
</context>

Question: {input}""")

document_chain = create_stuff_documents_chain(llm, prompt)

使用检索器动态选择最相关的文档并传递给链：

from langchain.chains import create_retrieval_chain

# 这里的vector是我们第一步构建的向量，将其转换为检索器
retriever = vector.as_retriever()
retrieval_chain = create_retrieval_chain(retriever, document_chain)

调用这个链，来获取响应结果，这个答案应该更加准确，因为它结合了检索到的相关文档和原始问题。这整个过程就是为了使得语言模型在回答问题时能够基于更全面的上下文信息。

1	response = retrieval_chain.invoke({"input": "how can langsmith help with testing?"})

完整demo

from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import WebBaseLoader
from langchain_community.llms.ollama import Ollama
from langchain_community.vectorstores.faiss import FAISS
from langchain_core.prompts import ChatPromptTemplate
from langchain_community.embeddings import OllamaEmbeddings


llm = Ollama(model="llama2")
# 初始化嵌入模型

embeddings = OllamaEmbeddings()

# 爬取内容
loader = WebBaseLoader("https://docs.smith.langchain.com/overview")
docs = loader.load()
print(docs)
# 文本分割器
text_splitter = RecursiveCharacterTextSplitter()
# 将刚刚获取的docs分割
documents = text_splitter.split_documents(docs)
print(documents)
# 根据嵌入模型和分割后的文档，转换为向量
vector = FAISS.from_documents(documents, embeddings)
print("转换成向量完毕")
# 将向量转换为检索器
retriever = vector.as_retriever()
print("将向量转换为检索器")


# 构建问题链
prompt = ChatPromptTemplate.from_template("""Answer the following question based only on the provided context:

<context>
{context}
</context>

Question: {input}""")

document_chain = create_stuff_documents_chain(llm, prompt)
print("构建问题完毕")

# 使用检索器动态选择最相关的文档并传递给链：
retrieval_chain = create_retrieval_chain(retriever, document_chain)
# 最终调用
print("开始调用")

response = retrieval_chain.invoke({"input": "how can langsmith help with testing?"})

print("调用完毕")

print(response)

输出如下

[Document(page_content='\n\n\n\n\nLangSmith Overview and User Guide | 🦜️🛠️ LangSmith\n\n\n\n\n\nSkip to main content🦜️🛠️ LangSmith DocsPython DocsJS/TS DocsSearchGo to AppLangSmithOverviewTracingTesting & EvaluationOrganizationsHubLangSmith CookbookOverviewOn this pageLangSmith Overview and User GuideBuilding reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.Over the past two months, we at LangChain have been building and using LangSmith with the goal of bridging this gap. This is our tactical user guide to outline effective ways to use LangSmith and maximize its benefits.On by default\u200bAt LangChain, all of us have LangSmith’s tracing running in the background by default. On the Python side, this is achieved by setting environment variables, which we establish whenever we launch a virtual environment or open our bash shell and leave them set. The same principle applies to most JavaScript environments through process.env1.The benefit here is that all calls to LLMs, chains, agents, tools, and retrievers are logged to LangSmith. Around 90% of the time we don’t even look at the traces, but the 10% of the time that we do… it’s so helpful. We can use LangSmith to debug:An unexpected end resultWhy an agent is loopingWhy a chain was slower than expectedHow many tokens an agent usedDebugging\u200bDebugging LLMs, chains, and agents can be tough. LangSmith helps solve the following pain points:What was the exact input to the LLM?\u200bLLM calls are often tricky and non-deterministic. The inputs/outputs may seem straightforward, given they are technically string → string (or chat messages → chat message), but this can be misleading as the input string is usually constructed from a combination of user input and auxiliary functions.Most inputs to an LLM call are a combination of some type of fixed template along with input variables. These input variables could come directly from user input or from an auxiliary function (like retrieval). By the time these input variables go into the LLM they will have been converted to a string format, but often times they are not naturally represented as a string (they could be a list, or a Document object). Therefore, it is important to have visibility into what exactly the final string going into the LLM is. This has helped us debug bugs in formatting logic, unexpected transformations to user input, and straight up missing user input.To a much lesser extent, this is also true of the output of an LLM. Oftentimes the output of an LLM is technically a string but that string may contain some structure (json, yaml) that is intended to be parsed into a structured representation. Understanding what the exact output is can help determine if there may be a need for different parsing.LangSmith provides a straightforward visualization of the exact inputs/outputs to all LLM calls, so you can easily understand them.If I edit the prompt, how does that affect the output?\u200bSo you notice a bad output, and you go into LangSmith to see what\'s going on. You find the faulty LLM call and are now looking at the exact input. You want to try changing a word or a phrase to see what happens -- what do you do?We constantly ran into this issue. Initially, we copied the prompt to a playground of sorts. But this got annoying, so we built a playground of our own! When examining an LLM call, you can click the Open in Playground button to access this playground. Here, you can modify the prompt and re-run it to observe the resulting changes to the output - as many times as needed!Currently, this feature supports only OpenAI and Anthropic models and works for LLM and Chat Model calls. We plan to extend its functionality to more LLM types, chains, agents, and retrievers in the future.What is the exact sequence of events?\u200bIn complicated chains and agents, it can often be hard to understand what is going on under the hood. What calls are being made? In what order? What are the inputs and outputs of each call?LangSmith\'s built-in tracing feature offers a visualization to clarify these sequences. This tool is invaluable for understanding intricate and lengthy chains and agents. For chains, it can shed light on the sequence of calls and how they interact. For agents, where the sequence of calls is non-deterministic, it helps visualize the specific sequence for a given run -- something that is impossible to know ahead of time.Why did my chain take much longer than expected?\u200bIf a chain takes longer than expected, you need to identify the cause. By tracking the latency of each step, LangSmith lets you identify and possibly eliminate the slowest components.How many tokens were used?\u200bBuilding and prototyping LLM applications can be expensive. LangSmith tracks the total token usage for a chain and the token usage of each step. This makes it easy to identify potentially costly parts of the chain.Collaborative debugging\u200bIn the past, sharing a faulty chain with a colleague for debugging was challenging when performed locally. With LangSmith, we\'ve added a “Share” button that makes the chain and LLM runs accessible to anyone with the shared link.Collecting examples\u200bMost of the time we go to debug, it\'s because something bad or unexpected outcome has happened in our application. These failures are valuable data points! By identifying how our chain can fail and monitoring these failures, we can test future chain versions against these known issues.Why is this so impactful? When building LLM applications, it’s often common to start without a dataset of any kind. This is part of the power of LLMs! They are amazing zero-shot learners, making it possible to get started as easily as possible. But this can also be a curse -- as you adjust the prompt, you\'re wandering blind. You don’t have any examples to benchmark your changes against.LangSmith addresses this problem by including an “Add to Dataset” button for each run, making it easy to add the input/output examples a chosen dataset. You can edit the example before adding them to the dataset to include the expected result, which is particularly useful for bad examples.This feature is available at every step of a nested chain, enabling you to add examples for an end-to-end chain, an intermediary chain (such as a LLM Chain), or simply the LLM or Chat Model.End-to-end chain examples are excellent for testing the overall flow, while single, modular LLM Chain or LLM/Chat Model examples can be beneficial for testing the simplest and most directly modifiable components.Testing & evaluation\u200bInitially, we do most of our evaluation manually and ad hoc. We pass in different\ninputs, and see what happens. At some point though, our application is performing\nwell and we want to be more rigorous about testing changes. We can use a dataset\nthat we’ve constructed along the way (see above). Alternatively, we could spend some\ntime constructing a small dataset by hand. For these situations, LangSmith simplifies\ndataset uploading.Once we have a dataset, how can we use it to test changes to a prompt or chain? The most basic approach is to run the chain over the data points and visualize the outputs. Despite technological advancements, there still is no substitute for looking at outputs by eye. Currently, running the chain over the data points needs to be done client-side. The LangSmith client makes it easy to pull down a dataset and then run a chain over them, logging the results to a new project associated with the dataset. From there, you can review them. We\'ve made it easy to assign feedback to runs and mark them as correct or incorrect directly in the web app, displaying aggregate statistics for each test project.We also make it easier to evaluate these runs. To that end, we\'ve added a set of evaluators to the open-source LangChain library. These evaluators can be specified when initiating a test run and will evaluate the results once the test run completes. If we’re being honest, most of these evaluators aren\'t perfect. We would not recommend that you trust them blindly. However, we do think they are useful for guiding your eye to examples you should look at. This becomes especially valuable as the number of data points increases and it becomes infeasible to look at each one manually.Human Evaluation\u200bAutomatic evaluation metrics are helpful for getting consistent and scalable information about model behavior;\nhowever, there is still no complete substitute for human review to get the utmost quality and reliability from your application.\nLangSmith makes it easy to manually review and annotate runs through annotation queues.These queues allow you to select any runs based on criteria like model type or automatic evaluation scores, and queue them up for human review. As a reviewer, you can then quickly step through the runs, viewing the input, output, and any existing tags before adding your own feedback.We often use this for a couple of reasons:To assess subjective qualities that automatic evaluators struggle with, like creativity or humorTo sample and validate a subset of runs that were auto-evaluated, ensuring the automatic metrics are still reliable and capturing the right informationAll the annotations made using the queue are assigned as "feedback" to the source runs, so you can easily filter and analyze them later.\nYou can also quickly edit examples and add them to datasets to expand the surface area of your evaluation sets or to fine-tune a model for improved quality or reduced costs.Monitoring\u200bAfter all this, your app might finally ready to go in production. LangSmith can also be used to monitor your application in much the same way that you used for debugging. You can log all traces, visualize latency and token usage statistics, and troubleshoot specific issues as they arise. Each run can also be assigned string tags or key-value metadata, allowing you to attach correlation ids or AB test variants, and filter runs accordingly.We’ve also made it possible to associate feedback programmatically with runs. This means that if your application has a thumbs up/down button on it, you can use that to log feedback back to LangSmith. This can be used to track performance over time and pinpoint under performing data points, which you can subsequently add to a dataset for future testing — mirroring the debug mode approach.We’ve provided several examples in the LangSmith documentation for extracting insights from logged runs. In addition to guiding you on performing this task yourself, we also provide examples of integrating with third parties for this purpose. We\'re eager to expand this area in the coming months! If you have ideas for either -- an open-source way to evaluate, or are building a company that wants to do analytics over these runs, please reach out.Exporting datasets\u200bLangSmith makes it easy to curate datasets. However, these aren’t just useful inside LangSmith; they can be exported for use in other contexts. Notable applications include exporting for use in OpenAI Evals or fine-tuning, such as with FireworksAI.To set up tracing in Deno, web browsers, or other runtime environments without access to the environment, check out the FAQs.↩PreviousLangSmithNextTracingOn by defaultDebuggingWhat was the exact input to the LLM?If I edit the prompt, how does that affect the output?What is the exact sequence of events?Why did my chain take much longer than expected?How many tokens were used?Collaborative debuggingCollecting examplesTesting & evaluationHuman EvaluationMonitoringExporting datasetsCommunityDiscordTwitterGitHubDocs CodeLangSmith SDKPythonJS/TSMoreHomepageBlogCopyright © 2024 LangChain, Inc.\n\n\n\n', metadata={'source': 'https://docs.smith.langchain.com/overview', 'title': 'LangSmith Overview and User Guide | 🦜️🛠️ LangSmith', 'description': 'Building reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.', 'language': 'en'})]
[Document(page_content='LangSmith Overview and User Guide | 🦜️🛠️ LangSmith', metadata={'source': 'https://docs.smith.langchain.com/overview', 'title': 'LangSmith Overview and User Guide | 🦜️🛠️ LangSmith', 'description': 'Building reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.', 'language': 'en'}), Document(page_content="Skip to main content🦜️🛠️ LangSmith DocsPython DocsJS/TS DocsSearchGo to AppLangSmithOverviewTracingTesting & EvaluationOrganizationsHubLangSmith CookbookOverviewOn this pageLangSmith Overview and User GuideBuilding reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.Over the past two months, we at LangChain have been building and using LangSmith with the goal of bridging this gap. This is our tactical user guide to outline effective ways to use LangSmith and maximize its benefits.On by default\u200bAt LangChain, all of us have LangSmith’s tracing running in the background by default. On the Python side, this is achieved by setting environment variables, which we establish whenever we launch a virtual environment or open our bash shell and leave them set. The same principle applies to most JavaScript environments through process.env1.The benefit here is that all calls to LLMs, chains, agents, tools, and retrievers are logged to LangSmith. Around 90% of the time we don’t even look at the traces, but the 10% of the time that we do… it’s so helpful. We can use LangSmith to debug:An unexpected end resultWhy an agent is loopingWhy a chain was slower than expectedHow many tokens an agent usedDebugging\u200bDebugging LLMs, chains, and agents can be tough. LangSmith helps solve the following pain points:What was the exact input to the LLM?\u200bLLM calls are often tricky and non-deterministic. The inputs/outputs may seem straightforward, given they are technically string → string (or chat messages → chat message), but this can be misleading as the input string is usually constructed from a combination of user input and auxiliary functions.Most inputs to an LLM call are a combination of some type of fixed template along with input variables. These input variables could come directly from user input or from an auxiliary function (like retrieval). By the time these input variables go into the LLM they will have been converted to a string format, but often times they are not naturally represented as a string (they could be a list, or a Document object). Therefore, it is important to have visibility into what exactly the final string going into the LLM is. This has helped us debug bugs in formatting logic, unexpected transformations to user input, and straight up missing user input.To a much lesser extent, this is also true of the output of an LLM. Oftentimes the output of an LLM is technically a string but that string may contain some structure (json, yaml) that is intended to be parsed into a structured representation. Understanding what the exact output is can help determine if there may be a need for different parsing.LangSmith provides a straightforward visualization of the exact inputs/outputs to all LLM calls, so you can easily understand them.If I edit the prompt, how does that affect the output?\u200bSo you notice a bad output, and you go into LangSmith to see what's going on. You find the faulty LLM call and are now looking at the exact input. You want to try changing a word or a phrase to see what happens -- what do you do?We constantly ran into this issue. Initially, we copied the prompt to a playground of sorts. But this got annoying, so we built a playground of our own! When examining an LLM call, you can click the Open in Playground button to access this playground. Here, you can modify the prompt and re-run it to observe the resulting changes to the output - as many times as needed!Currently, this feature supports only OpenAI and Anthropic models and works for LLM and Chat Model calls. We plan to extend its functionality to more LLM types, chains, agents, and retrievers in the future.What is the exact sequence of events?\u200bIn complicated chains and agents, it can often be hard to understand what is going on under the hood. What calls are being made? In what", metadata={'source': 'https://docs.smith.langchain.com/overview', 'title': 'LangSmith Overview and User Guide | 🦜️🛠️ LangSmith', 'description': 'Building reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.', 'language': 'en'}), Document(page_content="retrievers in the future.What is the exact sequence of events?\u200bIn complicated chains and agents, it can often be hard to understand what is going on under the hood. What calls are being made? In what order? What are the inputs and outputs of each call?LangSmith's built-in tracing feature offers a visualization to clarify these sequences. This tool is invaluable for understanding intricate and lengthy chains and agents. For chains, it can shed light on the sequence of calls and how they interact. For agents, where the sequence of calls is non-deterministic, it helps visualize the specific sequence for a given run -- something that is impossible to know ahead of time.Why did my chain take much longer than expected?\u200bIf a chain takes longer than expected, you need to identify the cause. By tracking the latency of each step, LangSmith lets you identify and possibly eliminate the slowest components.How many tokens were used?\u200bBuilding and prototyping LLM applications can be expensive. LangSmith tracks the total token usage for a chain and the token usage of each step. This makes it easy to identify potentially costly parts of the chain.Collaborative debugging\u200bIn the past, sharing a faulty chain with a colleague for debugging was challenging when performed locally. With LangSmith, we've added a “Share” button that makes the chain and LLM runs accessible to anyone with the shared link.Collecting examples\u200bMost of the time we go to debug, it's because something bad or unexpected outcome has happened in our application. These failures are valuable data points! By identifying how our chain can fail and monitoring these failures, we can test future chain versions against these known issues.Why is this so impactful? When building LLM applications, it’s often common to start without a dataset of any kind. This is part of the power of LLMs! They are amazing zero-shot learners, making it possible to get started as easily as possible. But this can also be a curse -- as you adjust the prompt, you're wandering blind. You don’t have any examples to benchmark your changes against.LangSmith addresses this problem by including an “Add to Dataset” button for each run, making it easy to add the input/output examples a chosen dataset. You can edit the example before adding them to the dataset to include the expected result, which is particularly useful for bad examples.This feature is available at every step of a nested chain, enabling you to add examples for an end-to-end chain, an intermediary chain (such as a LLM Chain), or simply the LLM or Chat Model.End-to-end chain examples are excellent for testing the overall flow, while single, modular LLM Chain or LLM/Chat Model examples can be beneficial for testing the simplest and most directly modifiable components.Testing & evaluation\u200bInitially, we do most of our evaluation manually and ad hoc. We pass in different", metadata={'source': 'https://docs.smith.langchain.com/overview', 'title': 'LangSmith Overview and User Guide | 🦜️🛠️ LangSmith', 'description': 'Building reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.', 'language': 'en'}), Document(page_content='inputs, and see what happens. At some point though, our application is performing\nwell and we want to be more rigorous about testing changes. We can use a dataset\nthat we’ve constructed along the way (see above). Alternatively, we could spend some\ntime constructing a small dataset by hand. For these situations, LangSmith simplifies\ndataset uploading.Once we have a dataset, how can we use it to test changes to a prompt or chain? The most basic approach is to run the chain over the data points and visualize the outputs. Despite technological advancements, there still is no substitute for looking at outputs by eye. Currently, running the chain over the data points needs to be done client-side. The LangSmith client makes it easy to pull down a dataset and then run a chain over them, logging the results to a new project associated with the dataset. From there, you can review them. We\'ve made it easy to assign feedback to runs and mark them as correct or incorrect directly in the web app, displaying aggregate statistics for each test project.We also make it easier to evaluate these runs. To that end, we\'ve added a set of evaluators to the open-source LangChain library. These evaluators can be specified when initiating a test run and will evaluate the results once the test run completes. If we’re being honest, most of these evaluators aren\'t perfect. We would not recommend that you trust them blindly. However, we do think they are useful for guiding your eye to examples you should look at. This becomes especially valuable as the number of data points increases and it becomes infeasible to look at each one manually.Human Evaluation\u200bAutomatic evaluation metrics are helpful for getting consistent and scalable information about model behavior;\nhowever, there is still no complete substitute for human review to get the utmost quality and reliability from your application.\nLangSmith makes it easy to manually review and annotate runs through annotation queues.These queues allow you to select any runs based on criteria like model type or automatic evaluation scores, and queue them up for human review. As a reviewer, you can then quickly step through the runs, viewing the input, output, and any existing tags before adding your own feedback.We often use this for a couple of reasons:To assess subjective qualities that automatic evaluators struggle with, like creativity or humorTo sample and validate a subset of runs that were auto-evaluated, ensuring the automatic metrics are still reliable and capturing the right informationAll the annotations made using the queue are assigned as "feedback" to the source runs, so you can easily filter and analyze them later.', metadata={'source': 'https://docs.smith.langchain.com/overview', 'title': 'LangSmith Overview and User Guide | 🦜️🛠️ LangSmith', 'description': 'Building reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.', 'language': 'en'}), Document(page_content="You can also quickly edit examples and add them to datasets to expand the surface area of your evaluation sets or to fine-tune a model for improved quality or reduced costs.Monitoring\u200bAfter all this, your app might finally ready to go in production. LangSmith can also be used to monitor your application in much the same way that you used for debugging. You can log all traces, visualize latency and token usage statistics, and troubleshoot specific issues as they arise. Each run can also be assigned string tags or key-value metadata, allowing you to attach correlation ids or AB test variants, and filter runs accordingly.We’ve also made it possible to associate feedback programmatically with runs. This means that if your application has a thumbs up/down button on it, you can use that to log feedback back to LangSmith. This can be used to track performance over time and pinpoint under performing data points, which you can subsequently add to a dataset for future testing — mirroring the debug mode approach.We’ve provided several examples in the LangSmith documentation for extracting insights from logged runs. In addition to guiding you on performing this task yourself, we also provide examples of integrating with third parties for this purpose. We're eager to expand this area in the coming months! If you have ideas for either -- an open-source way to evaluate, or are building a company that wants to do analytics over these runs, please reach out.Exporting datasets\u200bLangSmith makes it easy to curate datasets. However, these aren’t just useful inside LangSmith; they can be exported for use in other contexts. Notable applications include exporting for use in OpenAI Evals or fine-tuning, such as with FireworksAI.To set up tracing in Deno, web browsers, or other runtime environments without access to the environment, check out the FAQs.↩PreviousLangSmithNextTracingOn by defaultDebuggingWhat was the exact input to the LLM?If I edit the prompt, how does that affect the output?What is the exact sequence of events?Why did my chain take much longer than expected?How many tokens were used?Collaborative debuggingCollecting examplesTesting & evaluationHuman EvaluationMonitoringExporting datasetsCommunityDiscordTwitterGitHubDocs CodeLangSmith SDKPythonJS/TSMoreHomepageBlogCopyright © 2024 LangChain, Inc.", metadata={'source': 'https://docs.smith.langchain.com/overview', 'title': 'LangSmith Overview and User Guide | 🦜️🛠️ LangSmith', 'description': 'Building reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.', 'language': 'en'})]
转换成向量完毕
将向量转换为检索器
构建问题完毕
开始调用
调用完毕
{'input': 'how can langsmith help with testing?', 'context': [Document(page_content="You can also quickly edit examples and add them to datasets to expand the surface area of your evaluation sets or to fine-tune a model for improved quality or reduced costs.Monitoring\u200bAfter all this, your app might finally ready to go in production. LangSmith can also be used to monitor your application in much the same way that you used for debugging. You can log all traces, visualize latency and token usage statistics, and troubleshoot specific issues as they arise. Each run can also be assigned string tags or key-value metadata, allowing you to attach correlation ids or AB test variants, and filter runs accordingly.We’ve also made it possible to associate feedback programmatically with runs. This means that if your application has a thumbs up/down button on it, you can use that to log feedback back to LangSmith. This can be used to track performance over time and pinpoint under performing data points, which you can subsequently add to a dataset for future testing — mirroring the debug mode approach.We’ve provided several examples in the LangSmith documentation for extracting insights from logged runs. In addition to guiding you on performing this task yourself, we also provide examples of integrating with third parties for this purpose. We're eager to expand this area in the coming months! If you have ideas for either -- an open-source way to evaluate, or are building a company that wants to do analytics over these runs, please reach out.Exporting datasets\u200bLangSmith makes it easy to curate datasets. However, these aren’t just useful inside LangSmith; they can be exported for use in other contexts. Notable applications include exporting for use in OpenAI Evals or fine-tuning, such as with FireworksAI.To set up tracing in Deno, web browsers, or other runtime environments without access to the environment, check out the FAQs.↩PreviousLangSmithNextTracingOn by defaultDebuggingWhat was the exact input to the LLM?If I edit the prompt, how does that affect the output?What is the exact sequence of events?Why did my chain take much longer than expected?How many tokens were used?Collaborative debuggingCollecting examplesTesting & evaluationHuman EvaluationMonitoringExporting datasetsCommunityDiscordTwitterGitHubDocs CodeLangSmith SDKPythonJS/TSMoreHomepageBlogCopyright © 2024 LangChain, Inc.", metadata={'source': 'https://docs.smith.langchain.com/overview', 'title': 'LangSmith Overview and User Guide | 🦜️🛠️ LangSmith', 'description': 'Building reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.', 'language': 'en'}), Document(page_content='inputs, and see what happens. At some point though, our application is performing\nwell and we want to be more rigorous about testing changes. We can use a dataset\nthat we’ve constructed along the way (see above). Alternatively, we could spend some\ntime constructing a small dataset by hand. For these situations, LangSmith simplifies\ndataset uploading.Once we have a dataset, how can we use it to test changes to a prompt or chain? The most basic approach is to run the chain over the data points and visualize the outputs. Despite technological advancements, there still is no substitute for looking at outputs by eye. Currently, running the chain over the data points needs to be done client-side. The LangSmith client makes it easy to pull down a dataset and then run a chain over them, logging the results to a new project associated with the dataset. From there, you can review them. We\'ve made it easy to assign feedback to runs and mark them as correct or incorrect directly in the web app, displaying aggregate statistics for each test project.We also make it easier to evaluate these runs. To that end, we\'ve added a set of evaluators to the open-source LangChain library. These evaluators can be specified when initiating a test run and will evaluate the results once the test run completes. If we’re being honest, most of these evaluators aren\'t perfect. We would not recommend that you trust them blindly. However, we do think they are useful for guiding your eye to examples you should look at. This becomes especially valuable as the number of data points increases and it becomes infeasible to look at each one manually.Human Evaluation\u200bAutomatic evaluation metrics are helpful for getting consistent and scalable information about model behavior;\nhowever, there is still no complete substitute for human review to get the utmost quality and reliability from your application.\nLangSmith makes it easy to manually review and annotate runs through annotation queues.These queues allow you to select any runs based on criteria like model type or automatic evaluation scores, and queue them up for human review. As a reviewer, you can then quickly step through the runs, viewing the input, output, and any existing tags before adding your own feedback.We often use this for a couple of reasons:To assess subjective qualities that automatic evaluators struggle with, like creativity or humorTo sample and validate a subset of runs that were auto-evaluated, ensuring the automatic metrics are still reliable and capturing the right informationAll the annotations made using the queue are assigned as "feedback" to the source runs, so you can easily filter and analyze them later.', metadata={'source': 'https://docs.smith.langchain.com/overview', 'title': 'LangSmith Overview and User Guide | 🦜️🛠️ LangSmith', 'description': 'Building reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.', 'language': 'en'}), Document(page_content='LangSmith Overview and User Guide | 🦜️🛠️ LangSmith', metadata={'source': 'https://docs.smith.langchain.com/overview', 'title': 'LangSmith Overview and User Guide | 🦜️🛠️ LangSmith', 'description': 'Building reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.', 'language': 'en'}), Document(page_content="retrievers in the future.What is the exact sequence of events?\u200bIn complicated chains and agents, it can often be hard to understand what is going on under the hood. What calls are being made? In what order? What are the inputs and outputs of each call?LangSmith's built-in tracing feature offers a visualization to clarify these sequences. This tool is invaluable for understanding intricate and lengthy chains and agents. For chains, it can shed light on the sequence of calls and how they interact. For agents, where the sequence of calls is non-deterministic, it helps visualize the specific sequence for a given run -- something that is impossible to know ahead of time.Why did my chain take much longer than expected?\u200bIf a chain takes longer than expected, you need to identify the cause. By tracking the latency of each step, LangSmith lets you identify and possibly eliminate the slowest components.How many tokens were used?\u200bBuilding and prototyping LLM applications can be expensive. LangSmith tracks the total token usage for a chain and the token usage of each step. This makes it easy to identify potentially costly parts of the chain.Collaborative debugging\u200bIn the past, sharing a faulty chain with a colleague for debugging was challenging when performed locally. With LangSmith, we've added a “Share” button that makes the chain and LLM runs accessible to anyone with the shared link.Collecting examples\u200bMost of the time we go to debug, it's because something bad or unexpected outcome has happened in our application. These failures are valuable data points! By identifying how our chain can fail and monitoring these failures, we can test future chain versions against these known issues.Why is this so impactful? When building LLM applications, it’s often common to start without a dataset of any kind. This is part of the power of LLMs! They are amazing zero-shot learners, making it possible to get started as easily as possible. But this can also be a curse -- as you adjust the prompt, you're wandering blind. You don’t have any examples to benchmark your changes against.LangSmith addresses this problem by including an “Add to Dataset” button for each run, making it easy to add the input/output examples a chosen dataset. You can edit the example before adding them to the dataset to include the expected result, which is particularly useful for bad examples.This feature is available at every step of a nested chain, enabling you to add examples for an end-to-end chain, an intermediary chain (such as a LLM Chain), or simply the LLM or Chat Model.End-to-end chain examples are excellent for testing the overall flow, while single, modular LLM Chain or LLM/Chat Model examples can be beneficial for testing the simplest and most directly modifiable components.Testing & evaluation\u200bInitially, we do most of our evaluation manually and ad hoc. We pass in different", metadata={'source': 'https://docs.smith.langchain.com/overview', 'title': 'LangSmith Overview and User Guide | 🦜️🛠️ LangSmith', 'description': 'Building reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.', 'language': 'en'})], 'answer': 'LangSmith can assist with testing by providing various features to simplify dataset uploading, running chains over the data points, visualizing the outputs, and evaluating the results. Here are some ways LangSmith can help with testing:\n\n1. Easy dataset uploading: LangSmith simplifies dataset uploading, making it easier to construct a small dataset by hand or use an existing one.\n2. Running chains over data points: Once you have a dataset, LangSmith allows you to run the chain over the data points and visualize the outputs. You can review the outputs directly in the web app by assigning feedback to runs and marking them as correct or incorrect.\n3. Evaluating results: LangSmith adds a set of evaluators to the open-source LangChain library, which can be specified when initiating a test run. These evaluators will evaluate the results once the test run completes, providing valuable information for guiding your eye to examples you should look at.\n4. Manual testing and annotation: LangSmith makes it easy to manually review and annotate runs through a visualization of the sequence of events, helping identify subjective qualities that automatic evaluators struggle with.\n5. Tracking token usage: LangSmith tracks the total token usage for a chain and the token usage of each step, making it easier to identify potentially costly parts of the chain.\n6. Collecting examples: LangSmith includes an "Add to Dataset" button for each run, enabling you to add examples for an end-to-end chain, an intermediary chain (such as a LLM Chain), or simply the LLM or Chat Model. This feature is available at every step of a nested chain, making it easier to test the overall flow and individual components.\n\nBy leveraging these features, LangSmith can help streamline your testing process and provide valuable insights into your LLM application\'s performance.'}

Conversation Retrieval Chain（对话检索链）

我们刚刚创建的链只能回答单一的问题，但有些应用场景需要构建对话型的聊天机器人。
为了实现这一点，我们仍然可以使用create_retrieval_chain函数，但是我们需要改变两件事：
- 检索方法现在不应仅适用于最近的输入，而应考虑整个历史记录。
- 最终的 LLM 链同样应该考虑整个历史
  更新检索（retrieval）
  我们创建了一个新的链，这个链将接受最近的用户输入（input）和整个对话历史（chat_history），然后使用语言模型生成一个搜索查询。这里使用了 create_history_aware_retriever 函数来实现这个新链。

from langchain.chains import create_history_aware_retriever
from langchain_core.prompts import MessagesPlaceholder

# 定义一个提示模板，用于根据对话历史生成搜索查询
prompt = ChatPromptTemplate.from_messages([
    MessagesPlaceholder(variable_name="chat_history"),  # 对话历史的占位符
    ("user", "{input}"),  # 用户输入的占位符
    ("user", "在上述对话的基础上，生成一个搜索查询，以获取与对话相关的信息")  # 生成搜索查询的指令，基于对话内容
])

# 创建一个检索器链，整合语言模型、检索器和定义的提示
retriever_chain = create_history_aware_retriever(llm, retriever, prompt)

我们可以传入后续的问题实例来测试这一点

from langchain_core.messages import HumanMessage, AIMessage

# 定义对话历史，包括人类消息和AI消息
chat_history = [HumanMessage(content="LangSmith可以帮助我测试App吗？"), AIMessage(content="Yes!")]

# 调用检索器链，传入对话历史和用户输入
retriever_chain.invoke({
    "chat_history": chat_history,
    "input": "告诉我怎么做"
})

此时的输出应该是会返回有关 LangSmith 中测试的文档。这是因为LLM生成了一个新查询，将聊天历史记录与后续问题相结合。
现在我们有了这个新的检索器，我们可以创建一个新的链来继续与这些检索到的文档进行对话。

prompt = ChatPromptTemplate.from_messages([
    ("system", "根据上下文回答用户的问题\n\n{context}"),
    MessagesPlaceholder(variable_name="chat_history"),
    ("user", "{input}"),
])

document_chain = create_stuff_documents_chain(llm, prompt)
retrieval_chain = create_retrieval_chain(retriever_chain, document_chain)

现在进行整体的测试：

chat_history = [HumanMessage(content="LangSmith可以帮助我测试App吗？"), AIMessage(content="Yes!")]

retriever_chain.invoke({
    "chat_history": chat_history,
    "input": "告诉我怎么做"
})

会出了一个连贯的答案，表明我们已经成功地将检索链变成了聊天机器人

Agent

构建代理时要做的第一件事就是决定它应该有权访问哪些工具。在此示例中，我们将为代理提供两个工具的访问权限：
1. 我们刚刚创建的检索器。这将让它轻松回答有关 LangSmith 的问题
- 首先，让我们为刚刚创建的检索器设置一个工具：

from langchain.tools.retriever import create_retriever_tool

# 这个检索器就是刚刚爬取LongSmith文档的内容，生成的检索器
retriever_tool = create_retriever_tool(
    retriever,
    "langsmith_search",
    "搜索有关LangSmith的信息或者有关LangSmith的任何问题，必须使用这个工具！",
)

一个搜索工具。这将使它能够轻松回答需要最新信息的问题。
- 官网示例中使用的搜索工具是Tavily，需要一个 API 密钥（有免费套餐）。在他们的平台上创建后，需要将其设置为环境变量

1
2
3
from langchain_community.tools.tavily_search import TavilySearchResults

search = TavilySearchResults()

- 我们现在可以创建我们想要使用的工具的列表：
1
tools = [retriever_tool, search]

- 现在我们有了工具，我们可以创建一个代理来使用它们。
1
2
3
4
5
6
7
8
9
10
11
12
13
# 这个可以拉取预定义好的prompt
pip install langchainhub

from langchain_openai import ChatOpenAI
from langchain import hub
from langchain.agents import create_openai_functions_agent
from langchain.agents import AgentExecutor

# 拉取别人预定义好的prompt
prompt = hub.pull("hwchase17/openai-functions-agent")
llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)
agent = create_openai_functions_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

- 调用
1
agent_executor.invoke({"input": "LangSmith可以帮助我测试App吗？"})

构建服务

LangServe 可以帮助开发人员将 LangChain 链部署为 REST API。

from typing import List

from fastapi import FastAPI
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
from langchain_community.document_loaders import WebBaseLoader
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.tools.retriever import create_retriever_tool
from langchain_community.tools.tavily_search import TavilySearchResults
from langchain_openai import ChatOpenAI
from langchain import hub
from langchain.agents import create_openai_functions_agent
from langchain.agents import AgentExecutor
from langchain.pydantic_v1 import BaseModel, Field
from langchain_core.messages import BaseMessage
from langserve import add_routes

# 1. 加载检索器（Retriever）
# 首先，通过 WebBaseLoader 从 LangChain 文档网站加载文档数据。然后，使用 RecursiveCharacterTextSplitter 将文档分割为文档片段，并使用 OpenAIEmbeddings 将文档片段嵌入为向量。最后，使用 FAISS 创建一个向量存储作为检索器。
loader = WebBaseLoader("https://docs.smith.langchain.com/overview")
docs = loader.load()
text_splitter = RecursiveCharacterTextSplitter()
documents = text_splitter.split_documents(docs)
embeddings = OpenAIEmbeddings()
vector = FAISS.from_documents(documents, embeddings)
retriever = vector.as_retriever()

# 2. 创建工具（Tools）
# 创建两个工具：一个是之前创建的检索器工具 retriever_tool，另一个是 Tavily 搜索工具 search。
retriever_tool = create_retriever_tool(
    retriever,
    "langsmith_search",
    "搜索有关LangSmith的信息或者有关LangSmith的任何问题，必须使用这个工具！",
)
search = TavilySearchResults()
tools = [retriever_tool, search]


# 3. 创建代理
# 使用 LangChain Hub 中提供的预定义提示（自己构建也可以），创建一个使用 OpenAI 模型和工具的代理。这个代理被设置为能够决定在处理问题时使用哪些工具。
prompt = hub.pull("hwchase17/openai-functions-agent")
llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)
agent = create_openai_functions_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)


# 4. FastAPI 定义
app = FastAPI(
  title="LangChain Server",
  version="1.0",
  description="A simple API server using LangChain's Runnable interfaces",
)

# 5. 添加链路由

# 通过调用 add_routes 函数，将代理链添加为 FastAPI 应用的一个路由，从而可以通过 /agent 路径访问代理链。

class Input(BaseModel):
    input: str
    chat_history: List[BaseMessage] = Field(
        ...,
        extra={"widget": {"type": "chat", "input": "location"}},
    )


class Output(BaseModel):
    output: str

add_routes(
    app,
    agent_executor.with_types(input_type=Input, output_type=Output),
    path="/agent",
)

if __name__ == "__main__":
    import uvicorn

    uvicorn.run(app, host="localhost", port=8000)

链的概念（补充）

链（Chains）通常将大语言模型（LLM）与提示（Prompt）结合在一起，基于此，我们可以对文本或数据进行一系列操作。链（Chains）可以一次性接受多个输入。例如，我们可以创建一个链，该链接受用户输入，使用提示模板对其进行格式化，然后将格式化的响应传递给 LLM 。我们可以通过将多个链组合在一起，或者通过将链与其他组件组合在一起来构建更复杂的链。

顺序链（SequentialChains）

是按预定义顺序执行其链接的链。具体来说，我们将使用简单顺序链（SimpleSequentialChain），这是顺序链的最简单类型，其中每个步骤都有一个输入/输出，一个步骤的输出是下一个步骤的输入。

import json
import os
from langchain_community.llms import Tongyi
from langchain.chains import LLMChain, SimpleSequentialChain
from langchain_core.prompts import ChatPromptTemplate

os.environ["DASHSCOPE_API_KEY"] = "sk-1903ddb520ac4637a1d36b2f93e0aa4c"

llm = Tongyi()

prompt1 = ChatPromptTemplate.from_template("你现在是写作领域的大师，现在为{title}这篇文章生成一些大纲吧。")

chain_one = LLMChain(llm=llm, prompt=prompt1)

prompt2 = ChatPromptTemplate.from_template("根据这个大纲：{toc}，生成每个章节对应的内容吧。")

chain_two = LLMChain(llm=llm, prompt=prompt2)

chain = SimpleSequentialChain(chains=[chain_one, chain_two], verbose=True)

while True:
    title = input("请输入题目：")
    res = chain.invoke({"input": title})
    print(json.dumps(res, indent=4))

执行结果

当然可以，以下是一个关于“大学生网红消费行为分析”的文章大纲：

I. 引言
   A. 网红经济的崛起与大学生消费者的概述
   B. 大学生网红消费现象的重要性及研究背景
   C. 文章的研究目的和意义

II. 大学生网红消费者群体特征分析
   A. 年龄、性别、专业等人口统计学特征
   B. 大学生的网络使用习惯与社交媒体偏好
   C. 大学生对网红的认可度与信任度

III. 大学生网红消费行为的特点
   A. 消费动机：从娱乐消遣到购物决策的影响
   B. 消费品类偏向：时尚穿搭、美妆护肤、电子产品等方面的消费情况
   C. 跟风消费与个性化消费需求的平衡
   D. 情感因素在购买决策中的作用

IV. 网红对大学生消费行为的影响机制
   A. 网红的影响力来源：内容创作能力、人格魅力、口碑效应等
   B. 网红营销策略对大学生消费心理的影响：情感共鸣、生活方式塑造、意见领袖角色
   C. 网红广告与软性推广对大学生消费选择的引导

V. 大学生网红消费行为的社会文化影响
   A. 对大学生价值观塑造的影响
   B. 社会审美趋势与消费观念的变化
   C. 对校园文化和市场经济发展的启示

VI. 结论
   A. 大学生网红消费行为的主要特点及其成因总结
   B. 对教育部门、企业、网红自身以及社会各方面应对大学生网红消费行为进行合理引导和规范的建议
   C. 展望未来研究方向与可能性

这个大纲旨在全面探讨大学生网红消费行为的现象、特点、成因以及其背后的社会文化影响，并针对相关各方提出有针对性的思考与建议。


# 下面就是每个章节对应的内容

I. 引言
A. 网红经济的崛起与大学生消费者的概述
随着互联网技术的发展和社交媒体平台的普及，网红经济作为一种新型商业模式在全球范围内迅速崛起。大学生作为网络主力军和新兴消费群体，他们对于网红的追捧和消费行为成为当前市场关注的热点。这一现象不仅反映出了网红经济的巨大潜力，也揭示了大学生在消费观念和行为上的新变化。

B. 大学生网红消费现象的重要性及研究背景
大学生网红消费现象日益普遍，其背后的消费心理、消费行为模式以及对社会文化的影响具有重要研究价值。通过深入探究这一现象，有助于理解新一代消费者的消费趋势，为相关企业和政策制定者提供针对性的策略指导。

C. 文章的研究目的和意义
本文旨在通过系统地研究大学生网红消费行为，揭示其特征、成因及其社会文化影响，为教育部门、企业、网红自身乃至整个社会提供理性应对这一现象的思路和建议，同时对未来相关研究领域的发展方向进行展望。

II. 大学生网红消费者群体特征分析
A. 年龄、性别、专业等人口统计学特征
以在校大学生为主体的年轻消费人群主要集中在18-24岁年龄段，男女比例相对均衡，不同专业的学生可能由于兴趣爱好、审美取向等因素，在网红消费上表现出一定的差异。

B. 大学生的网络使用习惯与社交媒体偏好
当代大学生是互联网的重度使用者，他们在日常生活中广泛依赖于社交媒体平台获取信息、交流互动和娱乐休闲。尤其偏爱短视频、直播、微博等新媒体形式，这为网红传播内容提供了广阔的受众基础。

C. 大学生对网红的认可度与信任度
大学生普遍认为网红具备较高的潮流引领力和社交影响力，对于具有一定专业素养和亲和力的网红更容易产生认同感和信赖感，从而将其推荐的产品或服务纳入自己的消费考虑范围。

III. 大学生网红消费行为的特点
A. 消费动机：从娱乐消遣到购物决策的影响
大学生在关注网红的过程中，最初可能出于娱乐消遣的需求，但逐渐地，网红所展示的生活方式、时尚品味以及产品体验等内容会影响他们的购物决策，使得消费行为由单纯的观赏转变为实际购买行为。

B. 消费品类偏向：时尚穿搭、美妆护肤、电子产品等方面的消费情况
大学生网红消费行为中，时尚穿搭、美妆护肤和个人电子产品是最常见的消费类别。这些商品往往与年轻人追求个性化、时尚化和高品质生活的需求相契合。

C. 跟风消费与个性化消费需求的平衡
大学生在网红消费过程中，既存在跟风模仿的现象，也注重个性表达和自我风格的塑造。他们会在一定程度上接受网红推荐的商品和服务，同时也根据个人喜好和实际情况做出选择，实现跟风与个性需求的动态平衡。

D. 情感因素在购买决策中的作用
大学生消费者在网红购物决策中情感因素占据较大比重，包括对网红本人的情感认同、对网红所构建生活方式的向往以及与其他粉丝间的社交关系等因素都会影响到购买行为。

IV. 网红对大学生消费行为的影响机制
A. 网红的影响力来源：内容创作能力、人格魅力、口碑效应等
网红的影响力源于其独特的内容创作能力、鲜明的个性特质以及良好的口碑效应。他们通过高质量的内容输出，吸引并维系粉丝群体，进而转化为商业价值。

B. 网红营销策略对大学生消费心理的影响：情感共鸣、生活方式塑造、意见领袖角色
网红运用情感共鸣、生活方式展示以及意见领袖的角色定位，巧妙地将产品融入其中，激发大学生消费者的心理共鸣，形成潜在购买意愿，并推动实际消费行为的发生。

C. 网红广告与软性推广对大学生消费选择的引导
网红借助广告植入、品牌合作等形式进行商品推广，这种隐性且易于接受的营销手段在很大程度上左右了大学生的消费选择。

V. 大学生网红消费行为的社会文化影响
A. 对大学生价值观塑造的影响
大学生网红消费行为促进了他们对美的追求、个性化意识的觉醒以及对品质生活的向往，进一步塑造了新一代消费者的消费价值观。

B. 社会审美趋势与消费观念的变化
随着大学生对网红消费的积极参与，他们的审美情趣和消费观念逐渐成为社会审美趋势的重要组成部分，进一步推动着社会整体消费观念的变革与发展。

C. 对校园文化和市场经济发展的启示
大学生网红消费行为不仅反映了当前市场经济环境下消费结构和消费行为的新变化，也为校园文化的多元化发展提供了新的视角与启示。

VI. 结论
A. 大学生网红消费行为的主要特点及其成因总结
通过对大学生网红消费行为的研究，我们发现其主要特点包括高度活跃的社交媒体参与、强烈的情感驱动消费、紧跟潮流且兼顾个性化的消费需求以及深受网红营销策略的影响。这些特点的背后成因主要包括大学生独特的年龄特征、网络环境下的信息接收方式以及网红影响力的多重作用。

B. 对教育部门、企业、网红自身以及社会各方面应对大学生网红消费行为进行合理引导和规范的建议
教育部门应加强对大学生消费观的引导教育，培养他们的理性消费意识；企业应充分认识到网红营销的价值，合理利用网红资源，同时严格遵守相关法律法规，确保营销活动的合法合规；网红自身则需保持职业操守，传递正能量，发挥好榜样作用；社会各界也应对网红经济持开放包容态度，强化监管与自律相结合，共同营造健康有序的网红消费环境。

C. 展望未来研究方向与可能性
未来的研究可以从更微观的层面深入剖析大学生网红消费行为的内在机理，探究不同个体之间消费差异的具体原因；也可以着眼于全球视野，对比国内外大学生网红消费现象的异同和发展趋势；此外，还可以探讨如何结合新技术应用，如大数据、人工智能等，更好地理解和引导大学生网红消费行为。

多链组合

from langchain.chains import LLMChain, SequentialChain
from langchain_community.llms.tongyi import Tongyi
from langchain_core.prompts import ChatPromptTemplate
import os

os.environ["DASHSCOPE_API_KEY"] = "sk-1903ddb520ac4637a1d36b2f93e0aa4c"

llm = Tongyi()

# 子链1
# prompt模板 1: 翻译成英语（把下面的review翻译成英语）
first_prompt = ChatPromptTemplate.from_template(
    "把下面的评论review翻译成中文:"
    "\n\n{Review}"
)
# chain 1: 输入：Review    输出：中文的 Review
chain_one = LLMChain(llm=llm, prompt=first_prompt, output_key="Chinese_Review")

# 子链2
# prompt模板 2: 用一句话总结下面的 review
second_prompt = ChatPromptTemplate.from_template(
    "请你用一句话来总结下面的评论review:"
    "\n\n{Chinese_Review}"
)
# chain 2: 输入：中文的Review   输出：总结
chain_two = LLMChain(llm=llm, prompt=second_prompt, output_key="summary")

# 子链3
# prompt模板 3: 下面review使用的什么语言
third_prompt = ChatPromptTemplate.from_template(
    "下面的评论review使用的什么语言:\n\n{Review}"
)
# chain 3: 输入：Review  输出：语言
chain_three = LLMChain(llm=llm, prompt=third_prompt, output_key="language")

# 子链4
# prompt模板 4: 使用特定的语言对下面的总结写一个后续回复
fourth_prompt = ChatPromptTemplate.from_template(
    "使用特定的语言对下面的总结写一个后续回复:"
    "\n\n总结: {summary}\n\n语言: {language}"
)
# chain 4: 输入： 总结, 语言    输出： 后续回复
chain_four = LLMChain(llm=llm, prompt=fourth_prompt, output_key="followup_message")

#输入：review
#输出：中文review，总结，后续回复
overall_chain = SequentialChain(
    chains=[chain_one, chain_two, chain_three, chain_four],
    input_variables=["Review"],
    output_variables=["Chinese_Review", "summary", "followup_message", "language"],
    verbose=True
)

while True:
    text = input("请输入内容：")
    print(overall_chain.invoke({"Review": text}))

路由链

前面的链的输入顺序基本上都是固定的，如果想做更复杂的事情，就需要根据输入将其路由到特定的链。假设你有多个子链，每个子链都专门用于特定类型的输入，那么可以组成一个路由链
路由器由两个组件组成：
1. 路由链（Router Chain）：路由器链本身，负责选择要调用的下一个链
2. destination_chains：路由器链可以路由到的链

import os

from langchain.chains import LLMChain, MultiPromptChain
from langchain.chains.router.llm_router import RouterOutputParser, LLMRouterChain
from langchain.chains.router.multi_prompt_prompt import MULTI_PROMPT_ROUTER_TEMPLATE
from langchain_community.llms.tongyi import Tongyi
from langchain_core.prompts import ChatPromptTemplate, PromptTemplate

os.environ["DASHSCOPE_API_KEY"] = "sk-1903ddb520ac4637a1d36b2f93e0aa4c"

llm = Tongyi()


# 定义多种提示模板
# 第一个提示适合回答物理问题
physics_template = """你是一个非常聪明的物理专家。 \
你擅长用一种简洁并且易于理解的方式去回答问题。\
当你不知道问题的答案时，你承认\
你不知道.

这是一个问题:
{input}"""

# 第二个提示适合回答数学问题
math_template = """你是一个非常优秀的数学家。 \
你擅长回答数学问题。 \
你之所以如此优秀， \
是因为你能够将棘手的问题分解为组成部分，\
回答组成部分，然后将它们组合在一起，回答更广泛的问题。

这是一个问题：
{input}"""

# 第三个适合回答历史问题
history_template = """你是以为非常优秀的历史学家。 \
你对一系列历史时期的人物、事件和背景有着极好的学识和理解\
你有能力思考、反思、辩证、讨论和评估过去。\
你尊重历史证据，并有能力利用它来支持你的解释和判断。

这是一个问题:
{input}"""

# 第四个适合回答计算机问题
computerscience_template = """ 你是一个成功的计算机科学专家。\
你有创造力、协作精神、\
前瞻性思维、自信、解决问题的能力、\
对理论和算法的理解以及出色的沟通技巧。\
你非常擅长回答编程问题。\
你之所以如此优秀，是因为你知道  \
如何通过以机器可以轻松解释的命令式步骤描述解决方案来解决问题，\
并且你知道如何选择在时间复杂性和空间复杂性之间取得良好平衡的解决方案。

这还是一个问题：
{input}"""

# 为每个模板命名，并且给出具体描述，将这些信息传递给路由链，路由链决定何时调用子链
prompt_infos = [
    {
        "名字": "物理学",
        "描述": "擅长回答关于物理学的问题",
        "提示模板": physics_template
    },
    {
        "名字": "数学",
        "描述": "擅长回答数学问题",
        "提示模板": math_template
    },
    {
        "名字": "历史",
        "描述": "擅长回答历史问题",
        "提示模板": history_template
    },
    {
        "名字": "计算机科学",
        "描述": "擅长回答计算机科学问题",
        "提示模板": computerscience_template
    }
]

# 基于提示模版信息创建相应目标链
destination_chains = {}
for p_info in prompt_infos:
    name = p_info["名字"]
    prompt_template = p_info["提示模板"]
    prompt = ChatPromptTemplate.from_template(template=prompt_template)
    chain = LLMChain(llm=llm, prompt=prompt)
    destination_chains[name] = chain

destinations = [f"{p['名字']}: {p['描述']}" for p in prompt_infos]
destinations_str = "\n".join(destinations)

# 构建默认目标链，当路由器无法决定使用哪个子链时调用的链。
default_prompt = ChatPromptTemplate.from_template("{input}")
default_chain = LLMChain(llm=llm, prompt=default_prompt)

# 构建路由链
router_template = MULTI_PROMPT_ROUTER_TEMPLATE.format(destinations=destinations_str)

router_prompt = PromptTemplate(
    template=router_template,
    input_variables=["input"],
    output_parser=RouterOutputParser(),
)

router_chain = LLMRouterChain.from_llm(llm, router_prompt)

# 构建整体链路
chain = MultiPromptChain(router_chain=router_chain,  # 路由链路
                         destination_chains=destination_chains,  # 目标链路
                         default_chain=default_chain,  # 默认链路
                         verbose=True
                         )

while True:
    que = input("请输入问题：")
    print(chain.invoke({"input": que}))

后续我们可以预定义非常多的prompt，然后通过路由链来进行任务分发，尽可能把任务细分，这样出现问题时，只需要关注对应的某一个prompt，而不需要整体的改动。

代理的概念（补充）

大型语言模型（LLMs）非常强大，但它们缺乏“最笨”的计算机程序可以轻松处理的特定能力。LLM 对逻辑推理、计算和检索外部信息的能力较弱，这与最简单的计算机程序形成对比。例如，语言模型无法准确回答简单的计算问题，还有当询问最近发生的事件时，其回答也可能过时或错误，因为无法主动获取最新信息。这是由于当前语言模型仅依赖预训练数据，与外界“断开”。要克服这一缺陷，LangChain框架提出了“代理”(Agent)的解决方案。
代理作为语言模型的外部模块，可提供计算、逻辑、检索等功能的支持，使语言模型获得异常强大的推理和获取信息的超能力。

要使用代理 (Agents) ，我们需要三样东西：

一个基本的 LLM
我们将要进行交互的工具 Tools
一个控制交互的代理 (Agents) 。

现在尝试使用代理来解决数学问题

import os

from langchain.agents import load_tools, initialize_agent, AgentType
from langchain_community.llms.tongyi import Tongyi

os.environ["DASHSCOPE_API_KEY"] = "sk-1903ddb520ac4637a1d36b2f93e0aa4c"

llm = Tongyi()

tools = load_tools(
    ["llm-math", "wikipedia"],
    llm=llm
)
agent = initialize_agent(
    tools,  # 第二步加载的工具
    llm,  # 第一步初始化的模型
    agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION,  # 代理类型
    handle_parsing_errors=True,  # 处理解析错误
    verbose=True  # 输出中间步骤
)

agent.invoke({"input": "2084的25%是多少？还有，你知道Palworld这款游戏吗"})

输出
Thought: 第一个问题需要计算2084的25%，可以使用计算器工具；第二个问题询问关于Palworld这款游戏的信息，需要用到Wikipedia工具。

Action:
1
2
3
4
{
"action": "Calculator",
"action_input": "2084 * 25%"
}
Observation: Answer: 521.0
Thought:对于第二个问题，我需要使用Wikipedia工具来查询Palworld这款游戏的信息。
Action:
1
2
3
4
{
"action": "Wikipedia",
"action_input": "Palworld"
}
Observation: Page: Palworld
Summary: Palworld is an action-adventure survival game by Japanese developer Pocket Pair. The game is set in an open world populated with animal-like creatures known as “Pals”. The players can battle and capture Pals in order to use them for base building, traversal, and combat. Palworld can be played either solo, or online by up to 32 players on one server. Announced in 2021, it was launched via early access for Windows, Xbox One, and Xbox Series X/S in January 2024.
The game’s comedic premise, which involves using firearms and equipping Pals with them, earned it the nickname “Pokémon with guns”. Other elements, such as using creatures for food or placing them to work in mines and factories, have also garnered attention. It was generally well received, with praise for its gameplay, content, and satirical premise, but criticism for its reliance on shock humor and use of unoriginal designs and mechanics.Palworld sold eight million units in the first six days of early access and reached two million concurrent players on Steam, making it the second-highest played game of all time on the platform.

Page: Brotato
Summary: Brotato is a 2023 shoot 'em up video game created by French independent developer Thomas Gervraud under the studio name Blobfish. It was first released via Steam early access in 2022, during which it sold over one million copies. Brotato received positive reviews from critics and players, and it was later ported to multiple platforms.

Page: List of video games in development
Summary: This is a confirmed list of video games in development, but are scheduled for release beyond 2024 or currently carry no release date at all.
总结一下上面的流程