Bastet：AI 找合約漏洞，怎麼排除誤報？

0:00靜態工具抓不到邏輯洞，Bastet 要公開基準

0:00大家好，很高興來到 ETHTaipei。我是 Alice。很高興介紹我們的專案 Bastet，主題是評估 Web3 裡用 LLM 工具找智能合約漏洞。我現在是 OneSavie Labs 的資深資安研究員，目前研究聚焦 LLM 推理和漏洞偵測。我在一些研討會分享過資安研究，例如 Black Hat USA 2026、Black Hat Asia 2026，以及其他資安會議。Hi everyone, it's a pleasure to be here at ETHTaipei. And my name is Alice. I'm excited to introduce our project, Bastet. It's on evaluating LLM tools for smart contract vulnerability identification in Web3. And I'm now a senior security researcher at OneSavie Labs, and my current research is focused on LLM reasoning and vuln detection. I have shared my security research at some conferences, such as Black Hat USA 2026, Black Hat Asia 2026, and various security conferences.

0:49這是今天的大綱。先從背景開始：為什麼要做這個工具、它要解什麼問題。然後快速介紹 Bastet。接著談我們用 Bastet 的資源在 Kaggle 辦的比賽，我們想找出什麼、評分怎麼設計才量得到真正重要的事。再來深入看比賽裡最強的解法，並現場走一遍。結論會收在：這場比賽讓我們看到什麼，以及接下來要做什麼。And this is today's outline. I will start with the background: why we needed to build this tool and the problem it's meant to solve. Then I will give a quick introduction to Bastet. From there, we will talk about the Kaggle competition we ran using Bastet's resources, what we were trying to find out, and how we designed the scoring to actually measure what matters. And then we will go deep on the strongest solution that came out of it and walk through it live. And in the conclusion section, we will close with what this competition showed us and what we are building toward next.

1:36好，先從背景開始。Bastet 要解兩個問題。第一，靜態工具會漏掉邏輯漏洞；第二，AI 審計工具沒人能驗證、也沒人能接著做。技術面上，多數現有工具靠靜態分析，像 AST 解析、正規表示式規則。它們擅長抓已知樣式，但沒辦法推理執行邏輯。這表示 DeFi 協議裡造成最大駭案的一些邏輯漏洞，完全沒被抓到。生態面上，現有的 AI 審計工具多半閉源，你無法重現結果、無法驗證宣稱、也無法站在他們的工作上繼續做。Okay, so let's get started with some background. Bastet was built to solve two problems. First is static tools that miss logic flaws, and AI audit tools that no one can verify or build on. So on the technical side, most existing tools rely on static analysis, like AST parsing, regex rules. They are good at catching known patterns, but they cannot reason about execution logic. That means... some logic flaws in DeFi protocols that caused the biggest hacks lay completely undetected. And on the ecosystem side, the AI audit tools that do exist are mostly closed source, so you cannot reproduce their results, you cannot verify their claims, and you cannot build on their work.

2:35兩個問題都在：工具不夠好，就算好的也不能分享、不能改進。所以才有 Bastet。來看 Bastet。這張圖是整體架構。跟我們剛談的最相關的兩塊，一是資料集，來自真實的 Code4rena 發現，標了漏洞根因；二是評測框架，給你混淆矩陣，衡量 AI 工具到底有沒有用。開源資料集加上可驗證基準，正是這個領域缺的。We had the two problems now, so the tools were not good enough, and even the good ones could not be shared or improved. That is why Bastet was born. Now let's talk about Bastet. This diagram shows the overall architecture of Bastet. The two components most relevant to what we discussed are the datasets, sourced from real Code4rena findings, tagged with vulnerability root cause, and the evaluation framework, giving you a confusion matrix to measure whether an AI tool actually works. Having an open-source dataset and a verifiable benchmark is exactly what the field was missing.

3:28Kaggle 賽：330 人、221 隊、超過 7,500 份提交

3:28我們問自己，AI 研究者在哪、誰能處理這些問題？答案是 Kaggle。AI 人才在那裡比賽。我們看到機會，在 Web3 資安和 AI 人才之間搭橋，於是把資料集和基準帶到 Kaggle，辦比賽邀請 LLM 研究者進來，激發更強的方法。比賽從 2026 年 4 月開到 7 月，在 Kaggle 上。任務很單純：給一批智能合約倉庫，找出漏洞並用 Bastet 分類法正確分類，參賽者要標出 tag、subtag、嚴重程度，以及一段短描述。So we ask ourselves, where are the AI researchers and who can tackle those problems? And the answer is Kaggle. That is where AI talent competes. We saw an opportunity to build a bridge between Web3 security and AI talent, so we brought the dataset and benchmark to Kaggle and ran a competition to invite LLM researchers in and inspire stronger methodologies. And our competition ran from April to July 2026 on Kaggle. The task was very simple. Given a set of smart contract repositories, find the vulnerabilities and classify them correctly using the Bastet taxonomy, and the participant needs to identify the tag, subtag, severity, and a short description.

4:24來看比賽。這其實是至今最大的開放基準之一，用來測 LLM 在真實世界智能合約漏洞上的表現。我們有 330 位參賽者、221 隊，三個月衝刺從 2026 年 4 月到 7 月，超過 7,500 份提交。非常感謝支持這件事的組織，包括以太坊基金會的 Ecosystem Support Program、IEEE Computational Intelligence Society 新加坡分會，以及 Global Chinese Community of Universal Digital Commons。So now let's talk about the competition. This is actually one of the largest open benchmarks ever built for testing LLMs on real-world smart contract vulnerabilities. We had 330 participants across 221 teams, bringing in over 7,500 submissions over a three-month sprint from April to July 2026. And we are incredibly grateful to have the backing of some amazing organizations that made this possible, including the Ethereum Foundation's Ecosystem Support Program, the IEEE Computational Intelligence Society Singapore Chapter, and the Global Chinese Community of Universal Digital Commons.

5:13我們真心感謝他們的支持。這種 backing 對 Web3 生態的長期安全很重要。那我們到底要人做什麼？資料集來自與 DeFiHackLabs 的合作，是真實智能合約倉庫裡的真實漏洞，由有經驗的資安研究員標註。參賽者掃描程式碼庫、找出漏洞，每一個都要精確分類：tag、subtag、嚴重程度、描述。全部對照 Bastet 自己的 tag 定義，精度在這裡很重要。We really, truly appreciate their support. This kind of backing matters for the long-term security of the Web3 ecosystem. So, what exactly were we asking people to do? The dataset came from a partnership with DeFiHackLabs, real vulnerabilities from real-world smart contract repositories, and tagged by experienced security researchers. Participants scan a codebase, find vulnerabilities, and for each one need to classify it precisely: tag, subtag, severity, and a description. All of this against Bastet's own tag definition, and the precision matters here.

6:01多報會被扣分。評分獎勵分類正確、解釋清楚的發現，因為真實審計師為誤報付出代價。我們從第一天就把這個成本寫進指標。好。來看頂尖隊伍怎麼做。第二名走比較傳統的機器學習路徑。這很好提醒我們，扎實基礎還是能競爭。但冠軍是 Specter。他們走完全不同的路。我們來走一遍他們做了什麼。Overreporting will get penalized. The scoring rewards findings that are correctly classified and well explained, because real auditors pay a cost for false alarms. We build the cost into the metric from day one. Okay. Now let's see how the top teams actually approach it, and the second-place solution took a more traditional machine learning approach. That is a great reminder that strong foundations still compete, but the winner is Specter. They took a different path entirely. Let's work through what they built.

6:51冠軍 Specter：八階段多代理，GPT-5.5 找洞、Sonnet 4.5 當法官

6:51冠軍是 Specter，公開分數 0.688。有意思的不只是分數。它是八階段多代理管線，骨架是 Cloudflare 的八個階段。智能合約特有的延伸，加上驗證、補洞、根因分析這些子階段，那是 Specter 自己的工作。Specter 建成兩個分開的層。把層分開是有意的。第一層是引擎：GPT-5.5 去找，Sonnet 4.5 當法官。兩個模型，兩個角色。The winner is Specter, with a public score of 0.688. What makes it interesting is not just the score. It is an eight-stage multi-agent pipeline, and its spine is Cloudflare's eight stages. The smart-contract-specific extensions add substages of validation, gap-filling, and root-cause analysis. Those are Specter's own work. And Specter is built as two separate layers. And keeping the layers separate is intentional. Layer one is the engine: GPT-5.5 hunts, and Sonnet 4.5 judges: two models, two different roles.

7:37因為管線跟模型無關，你可以換成開源模型，像 Gemma 4 或 Qwen，走同一套介面。第二層是後處理。原始輸出在這裡被清理、轉成可讀的東西。第一層衝召回，第二層衝精確率。你沒辦法在同一步同時做兩件事，所以才拆成兩層。管線跑八個階段。從 recon 開始，搞懂合約在做什麼；第二步 hunt，由 GPT-5.5 找漏洞。And because the pipeline is model-agnostic, you can swap in open-source models like Gemma 4 or Qwen over the same interface. And layer two is post-processing. This is where the raw output gets cleaned up and converted into something readable. Layer one is optimized for recall, and layer two is optimized for precision. So you cannot do both in the same step. That is why they are separated into two layers. And the pipeline runs in eight stages. It starts with recon, understanding what the contract does, then step two, hunt, where GPT-5.5 looks for vulnerabilities.

8:31Validate 檢查發現是不是真的，gap-fill 找有沒有漏掉的。然後 dedupe 去重，reach 檢查漏洞實務上到不到得了，feedback 再繞一圈，最後是 report。其中兩個階段會繞回 hunt。Gap-fill 和 feedback 都會在往下走之前再看一次。這才是管線，不是清單。八階段骨架來自 Cloudflare 的開放專案 Glasswing。Specter 的貢獻是疊在上面的子階段。Validate checks whether the findings are real, and the gap-fill looks for anything that got missed. Then dedupe deduplicates, reach checks whether the vulnerability is actually reachable in practice, and feedback loops back for one more pass, and finally is report. And two of these stages loop back into hunt. Gap-fill and feedback both trigger a second look before moving on. That is what makes it a pipeline and not just a checklist. The eight-stage spine comes from Cloudflare's open project, Glasswing. And Specter's contribution is the substages on top of that.

9:22兩個真的把分數拉開的設計決策。第一是對抗審查。獵人找到東西之後，第二個模型進來，被提示成像辯護律師。它的唯一工作是留下或丟掉發現，絕不新增。重點就是讓模型和提示去跟獵人不一致。張力才把誤報砍掉。第二：把鏈拆開。因為「bug 存在」和「bug 到得到」被當成兩個問題。Bug 躺在內部函式裡、沒有外部函式叫得到，那就不是可利用的漏洞。Two design decisions that actually move the score. The first one is adversarial review. After the hunter finds something, a second model comes in and is prompted like a defense attorney. Its only job is to keep or drop findings and never add new ones. The whole point is to make the model and the prompt disagree with the hunter. The tension is what cuts false positives. And the second one: split the chain. Because "bug exists" and "the bug is reachable" are treated as two separate questions. A bug sitting in an internal function that no external function can call, that is not an exploitable vulnerability.

10:17先廣掃再收斂，分數從 216 走到 1,088

10:17所以那條發現會被丟掉或降級。這兩件事都是精確率濾網。提交分數才往排行榜爬。策略很單純：先撒開，再收斂。掃描器故意產出過多候選人。邏輯是：寧可收太多，也不要漏掉真 bug。那是撒開；管線再收斂：PoC 驗證、去重、可達性檢查、描述修煉，把清單收成只剩真實、可執行的。So that finding will get dropped or downgraded. Both of these are precision filters. That is why the submission climbed the leaderboard. And the strategy is simple: they cast wide first, then converge. The scanner deliberately overproduces candidates. The logic is, better to collect too much than miss a real bug. That is cast wide, and the pipeline converges: POC verification, dedupe, reachability checks, description refinement, all of that collapses the list down to only what is real and actionable.

11:06分數上真的看得到。早期版本只有 216。加上後處理，跳到 409。全部覆蓋之後，分數落到 1,088。沒有覆蓋的召回只是雜訊，所以兩者彼此需要。多數 LLM 審計停在描述：問題是什麼、為什麼重要。Specter 多走一步，交出可運作的 exploit。And you can actually see how that played out in the scores. The early build came in at just 216. Once post-processing was added, it jumped to 409. And when everything was fully covered, the score landed at 1,088. Recall without coverage is just noise, so they kind of need each other. And most LLM auditors stop at a description: here is what the problem is. Here is why this matters. But Specter goes one step further and ships a working exploit.

11:55發現被確認之後發生四件事。第一是 PoC 驗證。它編譯一份真的能在鏈上重現發現的 PoC。編不過，發現就丟掉。不編譯，就不報告。第二是環境測試。它對協議自己的規則做 fuzz，確認攻擊下真的會破，不是我們覺得會破，是證明真的破。第三是無力償債和壞帳偵測。它直接檢查無力償債和壞帳狀態。Four things happen after a finding is confirmed. First is PoC verification. It compiles an actual PoC that reproduces the finding on-chain. And if it doesn't compile, the finding gets dropped. So no compile, no report. And the second is environment testing. It fuzzes the protocol's own rules to confirm they actually break under attack, not just, we think this breaks. We prove it really breaks. And the third is insolvency and bad debt detection. It checks for insolvency and bad debt states directly.

12:42那種靜態工具完全漏掉的經濟型攻擊。最後是破壞呼叫序列。它報告觸發漏洞的精確交易序列。不是只說這有漏洞。這是你得怎麼做，因為這是證明層。光有描述不夠。你得把整個根因、漏洞怎麼發生秀出來。Specter 還加了一個東西：SCAR。漏洞在程式碼庫裡不是平均分布。The kind of economic exploits the static tools missed entirely. And the last is breaking call sequences. It reports the exact sequences of transactions that trigger the vulnerability. Not just, this is vulnerable. Here is exactly how you need to do it, because this is the proof layer. The description is not enough. You need to show the whole root cause and how the vulnerability happened. And... One more thing Specter adds is SCAR. Vulnerabilities are not spread evenly across a codebase.

13:30SCAR 先篩檔案，示範一輪從 37 條收到 21 條

13:30有些檔案有風險，多數沒有。有些智能合約程式碼只是函式庫或介面。所以不是把所有東西餵進 LLM，而是先用一個小型、訓練過的檢索器把檔案排序，只把可疑的、或含重要經濟邏輯的，送進 LLM。以一個倉庫為例，範圍內 42 個檔案，其實只要讀 8 個。token 少 80%，重點是 top-3 召回接近 1.0，幾乎沒有風險被濾掉。Some files are risky, but most are not. Some smart contract code is just library or interface. So instead of feeding everything into the LLM, a small, trained retriever ranks the files first and only sends the suspicious ones, or ones that contain the important economic logic code, into the LLM. And in one repository, as an example, they have 42 files in scope, but actually, they only have eight they actually need to read. That is 80% fewer tokens, and the key point is recall at top three stays near 1.0, meaning almost nothing at risk gets filtered out.

14:23有了 SCAR，LLM 讀得更少，但該看的不會漏。這就是 Specter 解法。來看 demo。它有對抗法官，另一個模型平行跑，再加上證明層做出證人。這是 Black Hat Arsenal 那一刀：八個階段，一輪。Recon 讀專案、建呼叫圖，打獵時不盲目。然後 Hunt 散出狹窄的專家，平行各自追一類 bug。每個主張都面對對抗法官，另一個被提示成辯護律師的模型。So with SCAR, LLMs can read less, but miss nothing that matters. That is the Specter solution. Now let's see the demo. It has an adversarial judge, a different model running in parallel, plus it builds witnesses with a proof layer. This is the Black Hat Arsenal cut: eight stages, one pass. Recon reads the project and builds the call graph so nothing hunts blind. Then Hunt fans out narrow specialists. In parallel, each chasing one class of bug. Every claim faces an adversarial judge, a different model prompted as a defense attorney.

15:15它可以留下或丟掉，絕不發明。Gapfill 對沉默再探一次。Dedup 把重複收到根因，可達性把「bug 存在」跟「攻擊者到得到」分開。確認的發現繞回去，在周圍再播一次 hunt。迴圈穩定後，第八階段交出報告。靠近看，每個階段都會散開。光 recon 就在建模攻擊面、調度專家。Validate 正規化 tag、濾掉結構性誤報、對照圖驗證。Dedupe 跑五層。骨架是 Cloudflare 的開放研究，裡面的操作是我們的。這是它在一個 live repo 上。It can keep or drop, never invent. Gapfill reprobes the silence. Dedup collapses duplicates to root causes, and reachability separates a bug that exists from a bug an attacker can reach. Confirmed findings loop back and reseed the hunt around them. When the loop settles, stage eight ships the report. Look closer, and every stage fans out. Recon alone models the attack surface and routes the specialists. Validate normalizes tags, filters structural false positives, and verifies against the graph. Dedupe runs five distinct layers. The spine is Cloudflare's open research. The operations inside it are ours. Here it is on a live repo.

16:02Recon 剖析 128 個檔案，Slither 抽出呼叫圖，攻擊面模型在還沒花一個 token 打獵前就調度專家。六個專家平行找，回 37 條原始發現。然後法官開工：16 條因無法證明或結構雜訊被丟掉，21 條留下。Gapfill 對保持沉默的專家再探，重複收到單一根因，可達性走呼叫圖。一輪，從原始合約到乾淨報告。還有一件事：不要全掃。SCAR 是漏洞訓練過的檢索器，Qwen Coder 加 BM25，先把程式碼庫排序。Recon profiles 128 files, Slither extracts the call graph, and an attack-surface model routes the specialists before a single token is spent hunting. Six specialists hunt in parallel and return 37 raw findings. Then the judge goes to work: 16 dropped as unprovable or structural noise, 21 survive. Gapfill reprobes the specialists that stayed silent, duplicates collapse to single root causes, and reachability walks the call graph. One pass, raw contract to clean report. One more thing: don't scan everything. SCAR is a vulnerability-trained retriever, Qwen Coder plus BM25, that ranks the codebase first.

16:49公開榜 688、私有榜 307，過基準不等於能上鏈

16:49昂貴模型只讀可疑的五分之一。範圍內 42 個檔案變成 8 個，token 少 80%，Recall@3 接近 1。報告同時機器可讀、人類可稽核。同一份 JSON 渲染成審計發現：程式碼錨點、根因、攻擊路徑、以及落地在真實程式碼裡的影響。然後證明層讓每條發現都得掙到位置。Foundry 的概念驗證重現 exploit，不變量 fuzz 確認真的破。重現得出來的是確認真陽性；重現不出來的是誤報，報告交出去前就砍掉。發現只有標對才有意義，所以報告層是開源的。The expensive model reads only the suspicious one-fifth. Forty-two in-scope files become eight, 80 percent fewer tokens, and Recall@3 stays near one. The report is machine-readable. And human-auditable at once. The same JSON renders as an audit finding: a code anchor, the root cause, the exploit path, and the impact grounded in real code. Then the proof layer makes every finding earn its place. A Foundry proof of concept reproduces the exploit, and invariant fuzzing confirms the break. What reproduces is a confirmed true positive. What does not is a false positive, cut before the report ships. Findings only matter if they are labeled right, so the reporting layer is open source.

17:41Bastet Taxonomy 是給 Claude Code 或 Codex 的 skill：42 個正規 tag、69 個 subtag，加上一份定義每對有效組合的正規檔。它驗證每一列，拒絕不存在的配對，並把描述對照參考風格：程式碼錨點、根因、攻擊路徑、影響。一個可丟進 CI 的確定性檢查器。免費放在 GitHub。github.com/SunWeb3Sec/Bastet-Taxonomy。實戰測過：107 個倉庫、21,000 條發現。好，最終公開排行榜分數是 688，私有分數是 307。Bastet Taxonomy is a skill for Claude Code or Codex: 42 canonical tags, 69 subtags. And one canonical file that defines every valid pair. It validates every row, rejects pairs that do not exist, and links descriptions against a reference style: code anchors, root cause, exploit path, impact. A deterministic checker you can drop into CI. Free at GitHub. github.com/SunWeb3Sec/Bastet-Taxonomy. It is battle-tested: 107 repositories, 21,000 findings. Okay, so the final score is 688 on the public leaderboard, and the private score is 307.

18:33第一名是由 DeFiHackLabs 創辦人 SunSec 帶領的隊伍。收一下今天學到的。通過基準，跟準備好部署上鏈，是兩件事。基準能告訴你工具有多好，但不能告訴你它在還沒人看過的合約上表現如何。這裡誤報讓你扣分，但在鏈上……這裡誤報讓你扣分，但在鏈上漏報可能讓你損失數百萬。First place goes to a team led by SunSec, founder of DeFiHackLabs, and let's wrap up what we have learned today. Passing the benchmark and being ready for on-chain deployment are two different things. A benchmark can tell you how good your tool is, but it cannot tell you how it performs on a contract nobody's looked at yet. So a false positive here costs you points, but on-chain... A false positive here costs you points, but on-chain a false negative can cost millions.

19:24LLM 剛改寫了 Web3 資安裡什麼是可能的，但故事沒完。下一個挑戰不是贏基準，是在野外、在別人之前抓到漏洞。重點是：幫防禦方更快找 bug 的同一套工具，也幫攻擊方更快做出 exploit。以前要花好幾天人工審查的，LLM 現在幾分鐘就能掃。兩邊都成立。如果防禦方找到一個漏洞，攻擊方找到同一個，他們會用 LLM 在幾小時而不是幾週內做出可運作的 exploit。這就是新現實。幫你審計更快的同一套工具，已經在找你漏掉什麼的人手上。LLMs just rewrote what is possible in Web3 security, but the story does not end there. The next challenge is not beating a benchmark; it is catching vulnerabilities in the wild before someone else does. And that is the thing. The same tools that help defenders find bugs faster help attackers build exploits faster. So what took days of manual review, LLMs can now scan in minutes. That is true for both sides. If a defender finds a vulnerability and an attacker finds the same one, then they will use LLMs to generate a working exploit in hours instead of weeks. That is the new reality. The same tool that helps you audit faster is already in the hands of people looking for what you missed.

20:16留下資料集、基準，也第一次把 AI 圈跟 Web3 資安接上

20:16所以這場比賽留下的東西才重要：三件我們要放回生態的事。第一是資料集。這是跟 DeFiHackLabs 專家的合作，LLM 找到的品質，只跟它被評測的資料一樣好。所以我們把這個基礎做扎實。第二是基準，Kaggle 上的開放結構。任何人都能跑、重現結果、再往上做。放上 Kaggle，表示它會被更廣的 AI 社群壓測，不只已經在圈內的人。最後是橋。第一次，AI 研究者和 Web3 資安的人在同一個地方解同一類問題。That is exactly why what this competition left behind matters: three things we are putting back into the ecosystem. The first is the dataset. This is a collaboration with the experts at DeFiHackLabs, and the quality of what an LLM finds is only as good as the data it is evaluated against. So we made sure that foundation was solid. And the second is the benchmark, an open structure on Kaggle. Anyone can run it, reproduce the results, and build on top of it. Putting it on Kaggle meant it gets pressure-tested by the broader AI community, not just people already inside the space. And the last is the bridge. For the first time, AI researchers and Web3 security people were solving the same problems in the same place.

21:02那種合作才會長出新想法。所以今天要帶走的是：LLM 已經改變漏洞發現裡什麼是可能的。第二，頂尖提交裡看到的精確率，來自刻意設計：對抗審查、可達性檢查、exploit 驗證，不是只換更大的模型。最後，通過基準不是終點。上鏈代表未知合約、會適應的攻擊者、以及真實資產在線上。我們這裡用的樣本，其實已經在另一邊。下一步工作往那裡走。希望今天的分享能帶回去一點有用的東西，也許還有值得接著做的。謝謝。That kind of collaboration is where new ideas come from. So this is today's takeaway. LLMs have already changed what is possible in vulnerability discovery. And the second is: the precision we saw in the top submission came from deliberate design, adversarial review, reachability checks, exploit verification, not just a bigger model. And the last thing is, passing the benchmark is not the finish line. On-chain means unknown contracts, attackers who adapt, and real assets on the line. So the samples we used here are already on the other side. That is where the work goes next. We hope today's talk can give you something useful to take back, and maybe something worth building on. Thank you.

21:52謝謝今天的時間。謝謝 Alice 的精彩分享。現在開放向 Alice 提問。好。如果有問題，現場可以找 Alice。我想她還有更多可以談。謝謝。Thank you for your time today. Thank you, Alice, for that wonderful presentation. We'll now open up to questions for Alice. Okay. Well, if you have questions, feel free to reach out to Alice here. I'm sure she'll have more for you. Thank you.