Spider 1.0

Nov 12, 2024: We have released Spider 2.0 full paper, data and code. Follow the guideline to submit your scores to the leaderboard!

Aug 28, 2024: The early access version of Spider 2.0 (a more realistic and challenging text-to-SQL task) is now available! We expect to release the whole dataset in 1-2 weeks. As this is a preliminary release, there may be errors. Your feedback would be invaluable in refining the dataset!

Spider is a large-scale complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 Yale students. The goal of the Spider challenge is to develop natural language interfaces to cross-domain databases. It consists of 10,181 questions and 5,693 unique complex SQL queries on 200 databases with multiple tables covering 138 different domains. In Spider 1.0, different complex SQL queries and databases appear in train and test sets. To do well on it, systems must generalize well to not only new SQL queries but also new database schemas.

---

Why we call it "Spider"? It is because our dataset is complex and cross-domain like a spider crawling across mutiple complex(with many foreign keys) nests(databases). XLANG Lab for Building LLM/VLM Agents Spider Paper (EMNLP'18) Spider Post

---

Related Works from XLANG Lab: Spider 2.0 Text-to-SQL ('24) Spider2-V ('24) OSWorld ('24) DS-1000 Challenge (ICML'23) Binder Framework (ICLR '23) UnifiedSKG Framework (EMNLP'22) SParC Challenge (ACL'19) CoSQL Challenge (EMNLP'19)

• 11/12/2024 We have released Spider 2.0 full paper, data and code. Follow the guideline to submit your scores to the leaderboard.
• 08/28/2024 The early access version of Spider 2.0 (a more realistic and challenging text-to-SQL task) is now available! As this is a preliminary release, there may be errors. Your feedback would be invaluable in refining the dataset!
• 07/15/2024 Spider 2.0-vision (Benchmarking Multimodal Agents on Automating Data Science and Engineering Workflows) is out! Spider 2.0-SQL (much more realistic and challenging than Spider 1.0!) will be released in August.
• 02/05/2024 We will no longer accept submissions for Spider 1.0 evaluations or update its leaderboard. The test set of Spider 1.0 has already been released (check the Spider dataset link below). Look forward to the release of Spider 2.0, a more realistic and challenging benchmark in the era of LLMs, expected this March June. Stay tuned!
• 08/10/2023 Please check out XLANG Lab for Building LLM/VLM Agents!
• 05/27/2023 Please check out Dr.Spider, a robustness evaluation benchmark based on Spider, from AWS AI Lab for studying robustness in semantic parsing!
• 11/20/2022 Please check out our recent work DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation. Please check out examples, data, and code on the DS-1000 project site!!
• 10/18/2022 Please check out our recent work Binder: an easy but sota neural-symbolic built on GPT-3 Codex & SQL/Python interpreter. It injects GPT-3 Codex prompt API calls in programming languages! Please check out Binder demo, code, paper, and video on the Binder project site!!
• 01/18/2022 Please check out our recent work UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models. We open-sourced simple but SOTA/strong models for 21 tasks including text-to-SQL! Please check out our code in the UnifiedSKG repo!!
• 03/11/2021 Please check out a nice work from Google Research (including new Spider splits) for studying compositional generalization in semantic parsing!
• 11/15/2020 We will use Test Suite Accuracy as our official evaluation metric for Spider, SParC, and CoSQL. Please find the evaluation code from here. Also, Notice that Test results after May 02, 2020 are reported on the new release (collected some annotation errors).
• 08/03/2020 Corrected "column_name" and "column_name_original" mismatches in 2 dbs ("scholar" and "formula_1") in tables.json, and reparsed SQL queries (this only affects some models (e.g. RATSQL) which use our parsed SQL as the SQL input). Please download the Spider dataset from this page again.
• 06/07/2020 We corrected some annotation errors and label mismatches (not errors) in Spider dev and test sets (~4% of dev examples updated, click here for more details). Please download the Spider dataset from this page again.
• 01/16/2020 For value prediction (in order to compute the execution accuracy), your model should be able to 1) copy from the question inputs, 2) retrieve from the database content (database content is available), or 3) generate numbers (e.g. 3 in "LIMIT 3").
• 9/24/2019 (Min et al., EMNLP 2019) translated Spider to Chinese! Check out the Chinese challenge page.
• 5/17/2019 Our paper SParC: Cross-Domain Semantic Parsing in Context with Salesforce Research was accepted to ACL 2019! It introduces the context-dependent version of the Spider challenge: SParC!
• 5/17/2019 Please report any annotation errors here, we really appreciate your help and will update the data release in this summer!
• 1/14/2019 The submission tutorial is out!.
• 12/17/2018 We updated 7 sqlite database files (issue 14). Please download the Spider dataset from this page again.
• 10/25/2018 The evaluation script and results were updated (issue 5). Please download the lastest versions of the script and papers. Also, please follow instructions in issue 3 to generate the latest SQL parsing results (fixed a bug).

As the above spider chart shows, Spider 1.0 is distinct from most of the previous semantic parsing tasks because:

• ATIS, Geo, Academic: Each of them contains only a single database with a limited number of SQL queries, and has exact same SQL queries in train and test splits.
• WikiSQL: The numbers of SQL queries and tables are significantly large. But all SQL queries are simple, and each database is only a simple table without any foreign key.

Spider 1.0 spans the largest area in the chart, making it the first complex and cross-domain semantic parsing and text-to-SQL dataset! Read more on the blog post.

The data is split into training, development, and test sets. Download a copy of the dataset (distributed under the CC BY-SA 4.0 license):

Some examples look like the following:

Ask us questions at our Github issues page or contact Tao Yu, Rui Zhang, or Michihiro Yasunaga.

We expect the dataset to evolve. We would greatly appreciate it if you could donate us your non-private databases or SQL queries for the project.

We thank Graham Neubig, Tianze Shi, Catherine Finegan-Dollak, and the anonymous reviewers for their precious comments on this project. Also, we thank Pranav Rajpurkar for giving us the permission to build this website based on SQuAD.

Our team at the summit of the East Rock park in New Haven (The pose is "NLseq2SQL"):

Our current models do not predict any value in SQL conditions so that we do not provide execution accuracies. However, we encourage you to provide it in the future submissions. For value prediction, your model should be able to 1) copy from the question inputs, 2) retrieve from the database content (database content is available), or 3) generate numbers (e.g. 3 in "LIMIT 3"). Notice: Test results after May 02, 2020 are reported on the new release (collected some annotation errors).

For exact matching evaluation, instead of simply conducting string comparison between the predicted and gold SQL queries, we decompose each SQL into several clauses, and conduct set comparison in each SQL clause. Please refer to the paper and the Github page for more details. Notice: Test results after May 02, 2020 are reported on the new release (collected some annotation errors).

Other papers used Spider (evaluated on the dev but not test set):

1. (Wang et al., KDD 2022), Alibaba DAMO
2. (Ma et al., VLDB 2022), HKUST
3. (Qin et al., COLING 2022), Alibaba DAMO
4. (Usta et al., VLDB 2021), Bilkent University
5. (Min et al., EMNLP 2019), Westlake University, Spider in Chinese
6. (Yao et al., EMNLP 2019), OSU & Facebook AI Research
7. (Shaw et al., ACL 2019), Google
8. (Shin et al., NeurlPS 2019), UC Berkeley & MSR
9. (Weir et al., SIGMOD 2019), Brown University & TU Darmstadt
10. (Baik et al., ICDE 2019), U of Michigan & IBM