开源类GPT4多模态模型项目-OpenFlamingo-转型开源模型项目集合页

Machinery

Nomic Embed

可重现的长上下文文本嵌入器(Long Context Text Embedder)

技术报告:https://arxiv.org/abs/2402.01613

github项目主页:https://github.com/nomic-ai/contrastors

这份技术报告描述了nomic-embed-text-v1的训练过程，它是第一个可完全复现的、开源、开放权重、开放数据、上下文长度为8192的英文文本嵌入模型，在短文本和长文本任务上均优于OpenAI的Ada-002和OpenAI text-embedding-3-small

同时，以Apache 2许可证发布了训练代码和模型权重，与其他开源模型不同，还发布了一个训练数据加载器，其中包含了2.35亿个经过筛选的文本对，可以完全复制nomic-embed-text-v1的训练过程


文本嵌入模型的基准测试结果，nomic-embed-text-v1、OpenAI text-embedding-ada、OpenAI text-embedding-3-small和jina-embedding-base-v2在短文本和长文本基准测试中的综合性能

Nomic Embed是唯一一个可以完全审计的长文本模型，它在短文本和长文本基准测试中都超过了OpenAI text-embedding-ada、OpenAI text-embedding-3-small和Jina的性能，X轴的单位因基准套件而异


将nomic-embed-text-v1与OpenAI模型和其他顶级长文本开源模型进行基准测试，Nomic-embed-text-v1是唯一一个具有1亿参数级别的开源模型，它在短文本和长文本任务上均优于OpenAI text-embedding-ada和text-embedding-3-small

Nomic-embed-text-v1-ablated是指第5.4节中描述的训练设置，其中省略了HotpotQA和FEVER数据，“Seq”表示模型的上下文长度，Jina LC是Jina长上下文基准测试中任务的平均值


GLUE Dev Set集结果，除了2048模型之外，以与nomic-bert-2048相同的方式进行评估


MTEB基准测试结果，每个类别的数据均取平均值


Jina长上下文评估基准测试

—— 来自 S1Fun

Machinery

PokéLLMon

通过大型语言模型在宝可梦对战中实现人类水准的代理者

github项目主页:https://github.com/git-disl/PokeLLMon

PokéLLMon，它是第一个在战略战斗游戏中实现人类水平表现的LLM具身代理者，就像在宝可梦战斗中所展示的，PokéLLMon的设计包含三个关键策略:
(i)上下文强化学习，即时利用从战斗中产生的基于文本的反馈来迭代改进策略
(ii)知识增强生成，即检索外部知识以抵消幻觉，并使代理者能够及时正确地行动
(iii)一致的动作生成，以减轻代理者面对强大对手并想要躲避战斗时的“恐慌切换(panic switching)”现象

展示了与人类进行的在线战斗，证明了PokéLLMon的类人战斗策略和即时决策能力，其在排行比赛中胜率达到49％，在邀请战斗中胜率达到56％


在每一轮中，玩家被要求决定采取哪个行动，例如，是让快龙出招还是切换到场外的另一个宝可梦


两个代表性的宝可梦:喷火龙和妙蛙花，每个宝可梦都有(复数)类型、能力、属性和四个战斗招式


类型相克关系，"+"表示超有效/2倍伤害；"-"表示无效/0.5倍伤害；"×"表示没有效果/0倍伤害，未标记的则为标准1倍伤害


使LLMs能够与人类玩家进行战斗的框架:它解析从战斗服务器接收到的消息，并将状态日志转换为文本，LLMs将这些状态描述和历史回合日志作为输入，并为下一步生成一个行动，然后将该行动发送到战斗服务器，并与对手选择的行动一起执行


LLMs在与bot对战中的表现


类型相克预测的混合矩阵


POKELLMON配备了三种策略:
(1)ICRL利用战斗中的即时反馈来迭代改进生成
(2)KAG检索外部知识来对抗幻觉，并及时正确地行动
(3)一致的行动生成以预防恐慌切换问题


代理者重复使用相同的攻击招式，但由于其能力“干燥皮肤”，对对方宝可梦没有任何效果


在第3回合中，代理者使用“精神冲击”，对对方宝可梦造成了零伤害，通过ICRL，代理者切换到另一个宝可梦


ICRL在与bot对战中的表现


KAG在与bot对战中的表现


代理者理解了招式的效果并正确使用:钥圈儿对钻角犀兽的地面属性攻击很脆弱，代理者没有交换宝可梦，而是使用了“电磁飘浮”这个招式，可以保护自己免受地面属性攻击的影响，持续五回合，从而使对方的钻角犀兽的地面属性攻击“地震”无效


提示方法在与bot对战中的表现


当面对一个强大的宝可梦时，具有CoT的代理者连续三次换宝可梦来逃避战斗，这给了对手三个免费回合来四倍增加攻击属性，并迅速击败了代理者的整个队伍


恐慌切换宝可梦的统计分析


POKELLMON每回合都选择了有效的招式，使对手的整个队伍都倒下了，只用了一个宝可梦


POKELLMON对抗人类玩家的表现


POKELLMON受到了消耗战策略的困扰:对手玩家经常恢复高防御的宝可梦，要打破这个困境需要跨越多个回合的联合效果


战斗表现受到消耗战策略的影响


一个经验丰富的人类玩家误导代理者使用龙属性攻击，首先派出一个龙属性宝可梦，然后立即换成另一个免疫龙属性攻击的宝可梦

yesicant

先暂停更新，等泥潭把bug修好，大家新年快乐

李少卿

大佬不更新了吗？

—— 来自 OnePlus GM1910, Android 10上的 S1Next-鹅版 v2.5.4

yesicant

李少卿 发表于 2024-3-19 07:24
大佬不更新了吗？

—— 来自 OnePlus GM1910, Android 10上的 S1Next-鹅版 v2.5.4

感觉最近还沉浸在春节

实际上最近没啥大的进展，生态也逐渐起来了，这楼也许可以转讨论楼？

simpleyan

昨天sd3发了api，今天llama3发了两个小的，感觉又回到了去年底那种进展日新月异新闻看不完的时期，期待楼主重启更新。

—— 来自 OPPO PFFM20, Android 14上的 S1Next-鹅版 v2.5.4

ZetaGo

楼主在哪收集到这些有趣的项目呀，是刷gayhub和论文吗

seedees3

我记得泥潭是不是还有一个stable diffusion的专楼，怎么找不到了

诚司

dbrx开源，command R plus开源，llama3开源
qwen1.5之后本来以为开源已经稳定了，结果这段时间模型都下不过来，gpt4级别模型遍地走了是吧

不过8k context确实不大行，我还是选择command R+

两袖清风和中堂

没有好的显卡，跑起来太痛苦了

—— 来自 HUAWEI VOG-AL10, Android 10上的 S1Next-鹅版 v2.5.4

诚司

这llama3 70B tm的绝了，lmsys上它绝对被低估了，打榜上它和command R+差不多，但实测它比command R+强多了，哪怕是中文prompt，只不过llama3不喜欢说中文而已

llama3 70B完爆 llama3 8B，8B用英文提示词就能答上来一些东西，中文就不行，但70B的没这个问题，而Command R+参数比70B多也一样有这种问题

多轮工具调用我也试了试，llama3 70B至少是大杯claude水平，Sonnet claude是被llama3完爆的
这玩意开源了真tm绝了

ps:
试了两个类似TPTU文章里那样，但是更复杂一点的例子，GPT4和claude opus都容易答错，llama3在中文问题的debuff情况下都能答对……以前根本不敢想这种程度的tool planning

1. You are a strategy model and given a problem and a set of tools,you need to generate a sequence of executable tools to determine the solution to the problem.
   
2. 
   
3. Each tool in the toolset is defined as follows:
   
4. SQL Generator: Given an input problem and a database,create a syntactically correct SQLite query statement. Note that the table here, only contrain the following field: 'name'： this is the name of the book, 'price' this is the price of the book and the number of the price in the table is represented in the price of RMB(Chinese yuan), 'number': this is the number of books in this library
   
5. PythonREPL:Given an input problem and some information,generate a syntactically correct Python code.  In Python code. there is a  global variable 'rate' which represents  the exchange rate between US dollars and Chinese yuan.
   
6. 
   
7. Please use the following format:
   
8. 
   
9. Question: Here is the question
   
10. Error: Here is the previously generated error output
    
11. Tasks:Here is a Python List type,where each item in the List is a dictionary.The key of the dictionary represents the selected tool, and the value is the query input when calling the tool. Please note that the generated Tool and Query should be different from those in the Error.
    
12. 
    
13. Here are some examples mapping the question to the tools:
    
14. 
    
15. Question: What is the the number of albums by Jolin Tsai?
    
16. Error: None
    
17. 
    
18. Tasks:[{{SQL Generator:"What is the number of albums by Jolin Tsai?"}}]
    
19. 
    
20. Question: What is the square of the number of albums by Jolin Tsai?
    
21. Error: None
    
22. 
    
23. Tasks:[{{SQL Generator:"What is the number of albums by Jolin Tsai?"}},
    
24. {{PythonREPL:"What is the square of the number of albums by Jolin Tsai?"}}]
    
25. 
    
26. Question: How many books are cheaper than 10× 20 dolars? Please find the number of books and output the square of the number.
    
27. Error: None
    
28. 
    
29. Tasks:[{{PythonREPL:"What is 10× 20 ?"}}
    
30. {{SQL Generator:"How many books are cheaper than 10× 20 dolars?"}},
    
31. {{PythonREPL:"Output the square of the number above"}}]
    
32. 
    
33. Question:First,calculate the square of 40 and denote it as A.Then,find the names of all artists with a total number of fans less than A.
    
34. Error: None
    
35. 
    
36. Tasks:[{{PythonREPL:"Let A be the square of 40.What is the value of A?"}},{{SQL Generator:"Find the names of all artists with a total number of fans less than A"}}]
    
37. 
    
38. Note that you must ensure that the generated Tasks strictly adhere to the format requirements: they must be in Python List type,where each item is a dictionary.The key of the dictionary represents the selected tool, and the value is the query input when calling the tool.
    
39. 
    
40. Now,let's proceed:
    
41. 
    
42. Question: 从文件 'a.txt' 中读取一个数字。找出那些书名长度大于这个数字的减去20的所有书籍，将这些书名字符串写入到文件 'book_name.txt' 中。然后再查询那些书的价格以美元计算比之前从'a.txt’里读取的数字更贵的那些书的书名，写入到'test.txt'中
    
43. Error: None
    
44. Tasks:

复制代码

1. You are a strategy model and given a problem and a set of tools,you need to generate a sequence of executable tools to determine the solution to the problem.
   
2. 
   
3. Each tool in the toolset is defined as follows:
   
4. SQL Generator: Given an input problem and a database,create a syntactically correct SQLite query statement. Note that the table here, only contrain the following field: 'name'： this is the name of the book, 'price' this is the price of the book and the number of the price in the table is represented in the price of RMB(Chinese yuan), 'number': this is the number of books in this library.  'rate' : This  is the exchange rate between US dollars and Chinese yuan.
   
5. PythonREPL:Given an input problem and some information,generate a syntactically correct Python code.
   
6. 
   
7. Please use the following format:
   
8. 
   
9. Question: Here is the question
   
10. Error: Here is the previously generated error output
    
11. Tasks:Here is a Python List type,where each item in the List is a dictionary.The key of the dictionary represents the selected tool, and the value is the query input when calling the tool. Please note that the generated Tool and Query should be different from those in the Error.
    
12. 
    
13. Here are some examples mapping the question to the tools:
    
14. 
    
15. Question: What is the the number of albums by Jolin Tsai?
    
16. Error: None
    
17. 
    
18. Tasks:[{{SQL Generator:"What is the number of albums by Jolin Tsai?"}}]
    
19. 
    
20. Question: What is the square of the number of albums by Jolin Tsai?
    
21. Error: None
    
22. 
    
23. Tasks:[{{SQL Generator:"What is the number of albums by Jolin Tsai?"}},
    
24. {{PythonREPL:"What is the square of the number of albums by Jolin Tsai?"}}]
    
25. 
    
26. Question: How many books are cheaper than 10× 20 dolars? Please find the number of books and output the square of the number.
    
27. Error: None
    
28. 
    
29. Tasks:[{{PythonREPL:"What is 10× 20 ?"}}
    
30. {{SQL Generator:"How many books are cheaper than 10× 20 dolars?"}},
    
31. {{PythonREPL:"Output the square of the number above"}}]
    
32. 
    
33. Question:First,calculate the square of 40 and denote it as A.Then,find the names of all artists with a total number of fans less than A.
    
34. Error: None
    
35. 
    
36. Tasks:[{{PythonREPL:"Let A be the square of 40.What is the value of A?"}},{{SQL Generator:"Find the names of all artists with a total number of fans less than A"}}]
    
37. 
    
38. Note that you must ensure that the generated Tasks strictly adhere to the format requirements: they must be in Python List type,where each item is a dictionary.The key of the dictionary represents the selected tool, and the value is the query input when calling the tool.
    
39. 
    
40. Now,let's proceed:
    
41. 
    
42. Question: 从文件 'a.txt' 中读取一个数字。找出那些书名长度大于这个数字的减去20的所有书籍，将这些书名字符串写入到文件 'book_name.txt' 中。然后再查询那些书的价格以美元计算比之前从'a.txt’里读取的数字更贵的那些书的书名，写入到'test.txt'中
    
43. Error: None
    
44. Tasks:

复制代码