Spider 1.0 test image

Yale Semantic Parsing and Text-to-SQL Challenge

What is Spider?

Spider is a large-scale complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 Yale students. The goal of the Spider challenge is to develop natural language interfaces to cross-domain databases. It consists of 10,181 questions and 5,693 unique complex SQL queries on 200 databases with multiple tables covering 138 different domains. In Spider 1.0, different complex SQL queries and databases appear in train and test sets. To do well on it, systems must generalize well to not only new SQL queries but also new database schemas.
Why we call it "Spider"? It is because our dataset is complex and cross-domain like a spider crawling across mutiple complex(with many foreign keys) nests(databases). XLang Lab for building LM agents! Spider Paper (EMNLP'18) Spider Post
Related works: DS-1000, Binder, UnifiedSKG, multi-turn SParC and conversational CoSQL text-to-SQL tasks. DS-1000 Challenge ('22) Binder Framework (ICLR '23) UnifiedSKG Framework (EMNLP'22) SParC Challenge (ACL'19) CoSQL Challenge (EMNLP'19)

News

Why Spider?

test image

As the above spider chart shows, Spider 1.0 is distinct from most of the previous semantic parsing tasks because:
  • ATIS, Geo, Academic: Each of them contains only a single database with a limited number of SQL queries, and has exact same SQL queries in train and test splits.
  • WikiSQL: The numbers of SQL queries and tables are significantly large. But all SQL queries are simple, and each database is only a simple table without any foreign key.
Spider 1.0 spans the largest area in the chart, making it the first complex and cross-domain semantic parsing and text-to-SQL dataset! Read more on the blog post.

Getting Started

The data is split into training, development, and unreleased test sets. Download a copy of the dataset (distributed under the CC BY-SA 4.0 license):

Spider Dataset Details of baseline models and evaluation script can be found on the following GitHub site: Spider GitHub Page

Once you have built a model that works to your expectations on the dev set, you can submit it to get official scores on the dev and a hidden test set. To preserve the integrity of test results, we do not release the test set to the public. Instead, we request you to submit your model so that we can run it on the test set for you. Here's a tutorial walking you through official evaluation of your model:

Submission Tutorial

Data Examples

Some examples look like the following:

test image

Have Questions or Want to Contribute ?

Ask us questions at our Github issues page or contact Tao Yu, Rui Zhang, or Michihiro Yasunaga.

We expect the dataset to evolve. We would greatly appreciate it if you could donate us your non-private databases or SQL queries for the project.

Acknowledgement

We thank Graham Neubig, Tianze Shi, Catherine Finegan-Dollak, and the anonymous reviewers for their precious comments on this project. Also, we thank Pranav Rajpurkar for giving us the permission to build this website based on SQuAD.

Our team at the summit of the East Rock park in New Haven (The pose is "NLseq2SQL"):

test image

Leaderboard - Execution with Values

Our current models do not predict any value in SQL conditions so that we do not provide execution accuracies. However, we encourage you to provide it in the future submissions. For value prediction, your model should be able to 1) copy from the question inputs, 2) retrieve from the database content (database content is available), or 3) generate numbers (e.g. 3 in "LIMIT 3"). Notice: Test results after May 02, 2020 are reported on the new release (collected some annotation errors).

Rank Model Test

1

Aug 20, 2023 		DAIL-SQL + GPT-4 + Self-Consistency

Alibaba Group

(Gao and Wang et al.,'2023) code 		86.6

2

Aug 9, 2023 		DAIL-SQL + GPT-4

Alibaba Group

(Gao and Wang et al.,'2023) code 		86.2

3

Apr 21, 2023 		DIN-SQL + GPT-4

University of Alberta

(Pourreza et al.,'2023) code 		85.3

4

July 5, 2023 		Hindsight Chain of Thought with GPT-4

Kainos Software Ltd.

Code and paper coming soon 		83.9

5

Jun 1, 2023 		C3 + ChatGPT + Zero-Shot

Zhejiang University & Hundsun

(Dong et al.,'2023) code 		82.3

6

July 5, 2023 		Hindsight Chain of Thought with GPT-4 and Instructions

Kainos Software Ltd.

Code and paper coming soon 		80.8

7

Feb 7, 2023 		RESDSQL-3B + NatSQL (DB content used)

Renmin University of China

(Li et al., AAAI'23) code 		79.9

8

Nov 21, 2022 		SeaD + PQL (DB content used)

Anonymous

 78.5

9

Apr 21, 2023 		DIN-SQL + CodeX

University of Alberta

(Pourreza et al.,'2023) code 		78.2

10

August 10, 2023 		T5-3B+NatSQL+Token Preprocessing (DB content used)

George Mason University & MIT

(Rai et al., ACL '23) code 		78.0

10

Sep 14, 2022 		CatSQL + GraPPa (DB content used)

Anonymous

 78.0

12

Sep 13, 2022 		Graphix-3B+PICARD (DB content used)

Alibaba DAMO & HKU STAR & SIAT

(Li et al., AAAI'2023) code 		77.6

13

Sep 1, 2022 		SHiP+PICARD (DB content used)

AWS AI Labs

(Zhao et al.,'22) 		76.6

14

Dec 15, 2022 		N-best List Rerankers + PICARD (DB content used)

Alexa AI

(Zeng et al., IEEE SLT 2023) 		75.9

15

Jun 4, 2022 		RASAT+PICARD (DB content used)

SJTU LUMIA & Netmind.AI

(Qi et al., EMNLP'22) code 		75.5

16

May 8, 2022 		T5-SR (DB content used)

Anonymous

 75.2

17

Aug 12, 2022 		RESDSQL+T5-1.1-lm100k-xl (DB content used)

Anonymous

 75.1

17

Jul 14, 2021 		T5-3B+PICARD (DB content used)

Element AI, a ServiceNow company

(Scholak et al., EMNLP'21) code 		75.1

19

Aug 12, 2022 		RESDSQL+T5-1.1-lm100k-large (DB content used)

Anonymous

 74.8

20

May 18, 2022 		SeaD + SP (DB content used)

Anonymous

 74.1

21

May 4, 2021 		RATSQL+GAP+NatSQL (DB content used)

Queen Mary University of London

(Gan et al., EMNLP Findings'21) code 		73.3

22

August 10, 2021 		T5-Base+NatSQL+Token Preprocessing (DB content used)

George Mason University & MIT

(Rai et al., ACL '23) code 		71.1

22

Mar 10, 2021 		SmBoP + GraPPa (DB content used)

Tel-Aviv University & Allen Institute for AI

(Rubin and Berant, NAACL'21) code 		71.1

24

Aug 05, 2021 		RaSaP + ELECTRA (DB content used)

Ant Group, ZhiXiaoBao & Ada

(Huang et al.,'21) 		70.0

25

Nov 24, 2020 		BRIDGE v2 + BERT(ensemble) (DB content used)

Salesforce Research

(Lin et al., EMNLP-Findings '20) code 		68.3

26

Jan 16, 2021 		COMBINE (DB content used)

Novelis.io Research

(Youssef et al.,'21) 		68.2

27

Jul 22, 2022 		T5QL-Base (DB content used)

Anonymous

 66.8

28

Nov 24, 2020 		BRIDGE v2 + BERT (DB content used)

Salesforce Research

(Lin et al., EMNLP-Findings '20) code 		64.3

29

May 30, 2020 		AuxNet + BART (DB content used)

Anonymous

 62.6

30

May 30, 2020 		BRIDGE + BERT (DB content used)

Salesforce Research

(Lin et al., EMNLP-Findings '20) code 		59.9

31

May 20, 2020 		GAZP + BERT (DB content used)

University of Washington & Facebook AI Research

(Zhong et al., EMNLP '20) 		53.5

Leaderboard - Exact Set Match without Values

For exact matching evaluation, instead of simply conducting string comparison between the predicted and gold SQL queries, we decompose each SQL into several clauses, and conduct set comparison in each SQL clause. Please refer to the paper and the Github page for more details. Notice: Test results after May 02, 2020 are reported on the new release (collected some annotation errors).

Rank Model Dev Test

1

Sep 13, 2022 		Graphix-3B + PICARD (DB content used)

Alibaba DAMO & HKU STAR & SIAT

(Li et al., AAAI'2023) code 		77.1 74.0

1

Sep 14, 2022 		CatSQL + GraPPa (DB content used)

Anonymous

 78.6 73.9

3

Sep 1, 2022 		SHiP + PICARD (DB content used)

AWS AI Labs

(Zhao et al.,'22) 		77.2 73.1

4

May 23, 2022 		G³R + LGESQL + ELECTRA (DB content used)

Southeast University & Tencent Cloud Xiaowei

(Xiang et al., ACL-Findings '23) 		78.1 72.9

6

Aug 12, 2022 		RESDSQL+T5-1.1-lm100k-xl (DB content used)

Anonymous

 78.1 72.4

6

May 8, 2022 		T5-SR (DB content used)

Anonymous

 77.2 72.4

7

Dec 15, 2022 		N-best List Rerankers + PICARD (DB content used)

Alexa AI

(Zeng et al., IEEE SLT 2023) 		76.4 72.2

8

Sep 1, 2021 		S²SQL + ELECTRA (DB content used)

Alibaba DAMO

(Hui et al., ACL-Findings '22) code 		76.4 72.1

9

Feb 7, 2023 		RESDSQL-3B + NatSQL (DB content used)

Renmin University of China

(Li et al., AAAI'23) code 		80.5 72.0

10

Jun 1, 2021 		LGESQL + ELECTRA (DB content used)

SJTU X-LANCE Lab & AISpeech

(Cao et al., ACL'21) code 		75.1 72.0

11

Jul 14, 2021 		T5-3B+PICARD (DB content used)

Element AI, a ServiceNow company

(Scholak et al., EMNLP'21) code 		75.5 71.9

12

Aug 12, 2022 		RESDSQL+T5-1.1-lm100k-large (DB content used)

Anonymous

 76.6 71.1

13

Jun 4, 2022 		RASAT+PICARD (DB content used)

SJTU LUMIA & Netmind.AI

(Qi et al., EMNLP'22) code 		75.3 70.9

14

Nov 19, 2020 		DT-Fixup SQL-SP + RoBERTa (DB content used)

Borealis AI

(Xu et al., ACL'21) code 		75.0 70.9

15

Nov 19, 2020 		RAT-SQL + GraPPa + Adv (DB content used)

Anonymous

 75.5 70.5

16

Oct 18, 2021 		RATSQL++ + ELECTRA (DB content used)

Anonymous

 75.7 70.3

17

Nov 19, 2020 		SADGA + GAP (DB content used)

DMIR Lab

(Cai and Yuan et al., NeurIPS'21) code 		73.1 70.1

18

Dec 25, 2020 		RATSQL + GraPPa + GP (DB content used)

OCFT Gamma Big Data Lab

(Zhao et al.,'21) 		72.8 69.8

19

Sep 08, 2020 		RATSQL + GAP (DB content used)

University of Waterloo & AWS AI Labs

(Shi et al., AAAI'21) code 		71.8 69.7

20

Aug 18, 2020 		RATSQL + GraPPa (DB content used)

Yale & Salesforce Research

(Yu et al., ICLR'21) code 		73.4 69.6

21

Mar 10, 2021 		SmBoP + GraPPa (DB content used)

Tel-Aviv University & Allen Institute for AI

(Rubin and Berant, NAACL'21) code 		74.7 69.5

22

Aug 05, 2021 		RaSaP + ELECTRA (DB content used)

Ant Group, ZhiXiaoBao & Ada

(Huang et al.,'21) 		74.7 69.0

23

May 4, 2021 		RATSQL+GAP+NatSQL (DB content used)

Queen Mary University of London

(Gan et al., EMNLP Findings'21) code 		- 68.7

24

Nov 20, 2020 		RAT-SQL + STRUG (DB content used)

Microsoft Research & OSU

(Deng et al., NAACL '21) 		72.6 68.4

25

Jun 1, 2021 		LGESQL + BERT (DB content used)

SJTU X-LANCE Lab & AISpeech

(Cao et al., ACL'21) code 		74.1 68.3

26

Jan 16, 2021 		COMBINE (DB content used)

Novelis.io Research

(Youssef et al.,'21) 		71.4 67.7

27

Nov 24, 2020 		BRIDGE v2 + BERT(ensemble) (DB content used)

Salesforce Research

(Lin et al., EMNLP-Findings '20) code 		71.1 67.5

28

Sep. 8, 2020 		ShadowGNN + RoBERTa (DB content used)

SJTU X-LANCE Lab & AISpeech

(Chen et al., NAACL'21) 		72.3 66.1

29

Jul. 22, 2022 		T5QL-Base (DB content used)

Anonymous

 69.3 65.9

30

May 02, 2020 		RATSQL v3 + BERT (DB content used)

Microsoft Research

(Wang and Shin et al., ACL '20) code 		69.7 65.6

31

Dec. 07, 2020 		DuoRAT + BERT (DB content used)

Anonymous

 - 65.4

32

Sep. 8, 2020 		YCSQL + BERT (DB content used)

Anonymous

 - 65.3

33

Jan. 29, 2021 		ETA + BERT (DB content used)

Microsoft Research Asia

(Liu et al., ACL-Findings '21) 		70.8 65.3

34

Nov 24, 2020 		BRIDGE v2 + BERT (DB content used)

Salesforce Research

(Lin et al., EMNLP-Findings '20) code 		70.0 65.0

35

Sep. 8, 2020 		GP-RATSQL + BERT (DB content used)

Anonymous

 - 64.5

36

Nov. 25, 2020 		RATSQL-HPFT + BERT (DB content used)

Anonymous

 - 64.4

37

Feb 2, 2021 		LGESQL + GLOVE (DB content used)

SJTU X-LANCE Lab & AISpeech

(Cao et al., ACL'21) code 		67.6 62.8

38

May 31, 2020 		AuxNet + BART (DB content used)

Anonymous

 70.0 61.9

39

Dec 13, 2019 		RATSQL v2 + BERT (DB content used)

Microsoft Research

(Wang and Shin et al., ACL '20) code 		65.8 61.9

40

May 31, 2020 		AuxNet + BART

Anonymous

 68.0 61.3

41

Feb 18, 2020 		RYANSQL v2 + BERT

Kakao Enterprise

(Choi et al., '20) 		70.6 60.6

42

Oct 19, 2020 		SmBoP + BART

Tel-Aviv University & Allen Institute for AI

(Rubin and Berant '20) 		66.0 60.5

43

Dec 18, 2019 		IRNet++ + XLNet (DB content used)

Anonymous

 65.5 60.1

44

Apr 21, 2023 		DIN-SQL + GPT-4

University of Alberta

(Pourreza et al.,'2023) code 		60.1 60

45

May 30, 2020 		BRIDGE + BERT (DB content used)

Salesforce Research

(Lin et al., EMNLP-Findings '20) code 		65.5 59.2

46

Nov 12, 2019 		RYANSQL + BERT

Kakao Enterprise

(Choi et al., '20) 		66.6 58.2

47

Dec 13, 2019 		RATSQL v2 (DB content used)

Microsoft Research

(Wang and Shin et al., ACL '20) code 		62.7 57.2

48

Apr 21, 2023 		DIN-SQL + CodeX

University of Alberta

(Pourreza et al.,'2023) code 		57.2 57

49

Dec 13, 2019 		SLSQL + BERT + Data Annotation

National University of Singapore

(Lei and Wang et al., EMNLP '20) code 		60.8 55.7

50

Dec 13, 2019 		EditSQL+LSL + BERT

Anonymous

 57.9 55.2

51

June 24, 2019 		IRNet v2 + BERT

Microsoft Research Asia

 63.9 55.0

52

Sep 20, 2019 		GIRN + BERT

Anonymous

 60.2 54.8

53

May 19, 2019 		IRNet + BERT

Microsoft Research Asia

(Guo and Zhan et al., ACL '19) code 		61.9 54.7

54

Nov 4, 2019 		GNN + Bertrand-DR

Got It R&D

(Kelkar et al., '20) code 		57.9 54.6

55

Apr 8, 2020 		CNSQL

Anonymous

 58.0 54.0

56

Sep 19, 2019 		RATSQL

Anonymous

 60.6 53.7

57

Sep 1, 2019 		EditSQL + BERT

Yale University & Salesforce Research

(Zhang et al., EMNLP '19) code 		57.6 53.4

58

May 21, 2020 		GAZP + BERT

University of Washington & Facebook AI Research

(Zhong et al., EMNLP '20) 		- 53.3

59

May 21, 2020 		NatSQL v3

Anonymous

 - 53.2

60

May 28, 2020 		IRNET+ GNN

Anonymous

 - 49.6

61

June 24, 2019 		IRNet v2

Microsoft Research Asia

 55.4 48.5

62

Aug 30, 2019 		Global-GNN (DB content used)

Tel-Aviv University & Allen Institute for AI

(Bogin et al., EMNLP '19) code 		52.7 47.4

63

Dec 13, 2019 		LSL

Anonymous

 56.8 47.0

64

Apr 5, 2020 		GraphSQL

Anonymous

 52.8 46.9

65

May 19, 2019 		IRNet

Microsoft Research Asia

(Guo and Zhan et al., ACL '19) code 		53.2 46.7

66

Mar 17, 2020 		SG-IRNet

Anonymous

 - 46.6

67

Dec 13, 2019 		NatSQL v2

Anonymous

 52.0 46.4

68

June 11, 2019 		HSRNet

Anonymous

 51.5 45.6

69

June 12, 2019 		CFGN

Anonymous

 48.7 44.1

70

Aug 31, 2019 		NatSQL

Anonymous

 52.9 42.5

71

May 16, 2019 		GNN

Tel-Aviv University & Allen Institute for AI

(Bogin et al., ACL '19) code 		40.7 39.4

72

Feb 25, 2019 		SASeq

Anonymous

 40.8 37.4

73

May 30, 2019 		GrammarSQL

Allen Institute for AI

(Lin et al., '19) 		34.8 33.8

74

Sep 1, 2019 		EditSQL

Yale University & Salesforce Research

(Zhang et al., EMNLP '19) code 		36.4 32.9

75

Dec 13, 2019 		GuideSQL

Anonymous

 36.8 31.5

76

Sep 20, 2018 		SyntaxSQLNet + augment

Yale University

(Yu et al., EMNLP '18) code 		24.8 27.2

77

April 18, 2019 		RCSQL

SAP Labs Korea

(Lee, EMNLP'19) 		28.5 24.3

78

Sep 20, 2018 		SyntaxSQLNet

Yale University

(Yu et al., EMNLP '18) code 		18.9 19.7

79

Sep 20, 2018 		SQLNet

Shanghai Jiao Tong University (modified by Yale)

(Xu et al., '18) code 		10.9 12.4

80

Sep 20, 2018 		TypeSQL

Yale University

(Yu et al., NAACL '18) code 		8.0 8.2

81

Sep 20, 2018 		Seq2Seq + attention

University of Edinburgh (modified by Yale)

(Dong and Lapata, ACL '16) code 		1.8 4.8

Other papers used Spider (evaluated on the dev but not test set):
  1. (Wang et al., KDD 2022), Alibaba DAMO
  2. (Ma et al., VLDB 2022), HKUST
  3. (Qin et al., COLING 2022), Alibaba DAMO
  4. (Usta et al., VLDB 2021), Bilkent University
  5. (Min et al., EMNLP 2019), Westlake University, Spider in Chinese
  6. (Yao et al., EMNLP 2019), OSU & Facebook AI Research
  7. (Shaw et al., ACL 2019), Google
  8. (Shin et al., NeurlPS 2019), UC Berkeley & MSR
  9. (Weir et al., SIGMOD 2019), Brown University & TU Darmstadt
  10. (Baik et al., ICDE 2019), U of Michigan & IBM