Modern SQL: While you were looking elsewhere

Markus Winand https://winand.at • https://modern-sql.com • https://use-the-index-luke.com

What is SQL anyway?

 Old 

While SQL has turned 50 this year, it is by no means the same as 50 years ago. The latest update was mid 2023.

Relational 

Depends on how you understand relational: It is not about atomicity. It is fine to have JSON, XML and even arrays in single fields. Yet, in the persistent model simpler types have their merits.

 Query Language 

It is rather a query and transformation language. A pure query language (XPath, CSS selectors) just picks items out of something. SQL can do so much more — it is actually Turning complete.

What is SQL anyway?

 Old 

While SQL has turned 50 this year, it is by no means the same as 50 years ago. The latest update was mid 2023.

Relational 

Depends on how you understand relational: It is not about atomicity. It is fine to have JSON, XML and even arrays in single fields. Yet, in the persistent model simpler types have their merits.

 Query Language 

It is rather a query and transformation language. A pure query language (XPath, CSS selectors) just picks items out of something. SQL can do so much more — it is actually Turning complete.

What is SQL anyway?

 Old 

While SQL has turned 50 this year, it is by no means the same as 50 years ago. The latest update was mid 2023.

Relational 

Depends on how you understand relational: It is not about atomicity. It is fine to have JSON, XML and even arrays in single fields. Yet, in the persistent model simpler types have their merits.

 Query Language 

It is rather a query and transformation language. A pure query language (XPath, CSS selectors) just picks items out of something. SQL can do so much more — it is actually Turning complete.

What is SQL anyway?

 Old 

While SQL has turned 50 this year, it is by no means the same as 50 years ago. The latest update was mid 2023.

Relational 

Depends on how you understand relational: It is not about atomicity. It is fine to have JSON, XML and even arrays in single fields. Yet, in the persistent model simpler types have their merits.

 Query Language 

It is rather a query and transformation language. A pure query language (XPath, CSS selectors) just picks items out of something. SQL can do so much more — it is actually Turning complete.

What is SQL anyway?

 Old 

While SQL has turned 50 this year, it is by no means the same as 50 years ago. The latest update was mid 2023.

Relational 

Depends on how you understand relational: It is not about atomicity. It is fine to have JSON, XML and even arrays in single fields. Yet, in the persistent model simpler types have their merits.

 Query Language 

It is rather a query and transformation language. A pure query language (XPath, CSS selectors) just picks items out of something. SQL can do so much more — it is actually Turning complete.

What is SQL anyway?

 Old 

While SQL has turned 50 this year, it is by no means the same as 50 years ago. The latest update was mid 2023.

Relational 

Depends on how you understand relational: It is not about atomicity. It is fine to have JSON, XML and even arrays in single fields. Yet, in the persistent model simpler types have their merits.

 Query Language 

It is rather a query and transformation language. A pure query language (XPath, CSS selectors) just picks items out of something. SQL can do so much more — it is actually Turning complete.

What is SQL anyway?

 Old 

While SQL has turned 50 this year, it is by no means the same as 50 years ago. The latest update was mid 2023.

Relational 

Depends on how you understand relational: It is not about atomicity. It is fine to have JSON, XML and even arrays in single fields. Yet, in the persistent model simpler types have their merits.

 Query Language 

It is rather a query and transformation language. A pure query language (XPath, CSS selectors) just picks items out of something. SQL can do so much more — it is actually Turning complete.

What is SQL anyway?

 Multi-paradigm 

Early SQL focused on relational concepts. In 1999, SQL broke free from the relational cage: Recursion, window-functions, composite types, XML/JSON, etc.

 Domain Specific 

While SQL is Turing complete, it is not a general purpose language. SQL complements other languages to manage [*] data more easily.

[*]: large amounts of, shared, persistent, etc.

 Extensible Industry Standard 

Since 1987, SQL is defined in international standards: ISO/IEC 9075-1 to -16. They define syntax and the semantics of statements using this syntax. They explicitly allow SQL implementations to accept non-standard syntax.

What is SQL anyway?

 Multi-paradigm 

Early SQL focused on relational concepts. In 1999, SQL broke free from the relational cage: Recursion, window-functions, composite types, XML/JSON, etc.

 Domain Specific 

While SQL is Turing complete, it is not a general purpose language. SQL complements other languages to manage [*] data more easily.

[*]: large amounts of, shared, persistent, etc.

 Extensible Industry Standard 

Since 1987, SQL is defined in international standards: ISO/IEC 9075-1 to -16. They define syntax and the semantics of statements using this syntax. They explicitly allow SQL implementations to accept non-standard syntax.

What is SQL anyway?

 Multi-paradigm 

Early SQL focused on relational concepts. In 1999, SQL broke free from the relational cage: Recursion, window-functions, composite types, XML/JSON, etc.

 Domain Specific 

While SQL is Turing complete, it is not a general purpose language. SQL complements other languages to manage [*] data more easily.

[*]: large amounts of, shared, persistent, etc.

 Extensible Industry Standard 

Since 1987, SQL is defined in international standards: ISO/IEC 9075-1 to -16. They define syntax and the semantics of statements using this syntax. They explicitly allow SQL implementations to accept non-standard syntax.

What is SQL anyway?

 Multi-paradigm 

Early SQL focused on relational concepts. In 1999, SQL broke free from the relational cage: Recursion, window-functions, composite types, XML/JSON, etc.

 Domain Specific 

While SQL is Turing complete, it is not a general purpose language. SQL complements other languages to manage [*] data more easily.

[*]: large amounts of, shared, persistent, etc.

 Extensible Industry Standard 

Since 1987, SQL is defined in international standards: ISO/IEC 9075-1 to -16. They define syntax and the semantics of statements using this syntax. They explicitly allow SQL implementations to accept non-standard syntax.

What is SQL anyway?

 Declarative 

SQL is generally referred to as being declarative… …but it also embraces ideas from the functional programming paradigm without enforcing its limitations. Functions may have side effects: NON DETERMINISTIC • MODIFIES SQL DATA

 Strictly Typed 

Although SQL is strictly typed[*] it has some flexible types:

CREATE TABLE t (
   a INTEGER ARRAY[1000], -- maximum length
   x XML,
   j JSON
)

[*] SQLite being the famous exception (but see STRICT tables).

What is SQL anyway?

 Declarative 

SQL is generally referred to as being declarative… …but it also embraces ideas from the functional programming paradigm without enforcing its limitations. Functions may have side effects: NON DETERMINISTIC • MODIFIES SQL DATA

 Strictly Typed 

Although SQL is strictly typed[*] it has some flexible types:

CREATE TABLE t (
   a INTEGER ARRAY[1000], -- maximum length
   x XML,
   j JSON
)

[*] SQLite being the famous exception (but see STRICT tables).

What is SQL anyway?

 Declarative 

SQL is generally referred to as being declarative… …but it also embraces ideas from the functional programming paradigm without enforcing its limitations. Functions may have side effects: NON DETERMINISTIC • MODIFIES SQL DATA

 Strictly Typed 

Although SQL is strictly typed[*] it has some flexible types:

CREATE TABLE t (
   a INTEGER ARRAY[1000], -- maximum length
   x XML,
   j JSON
)

[*] SQLite being the famous exception (but see STRICT tables).

What is SQL anyway?

 Declarative 

SQL is generally referred to as being declarative… …but it also embraces ideas from the functional programming paradigm without enforcing its limitations. Functions may have side effects: NON DETERMINISTIC • MODIFIES SQL DATA

 Strictly Typed 

Although SQL is strictly typed[*] it has some flexible types:

CREATE TABLE t (
   a INTEGER ARRAY[1000], -- maximum length
   x XML,
   j JSON
)

[*] SQLite being the famous exception (but see STRICT tables).

What is SQL anyway?

 Declarative 

SQL is generally referred to as being declarative… …but it also embraces ideas from the functional programming paradigm without enforcing its limitations. Functions may have side effects: NON DETERMINISTIC • MODIFIES SQL DATA

 Strictly Typed 

Although SQL is strictly typed[*] it has some flexible types:

CREATE TABLE t (
   a INTEGER ARRAY[1000], -- maximum length
   x XML,
   j JSON
)

[*] SQLite being the famous exception (but see STRICT tables).

What is SQL anyway?

 Declarative 

SQL is generally referred to as being declarative… …but it also embraces ideas from the functional programming paradigm without enforcing its limitations. Functions may have side effects: NON DETERMINISTIC • MODIFIES SQL DATA

 Strictly Typed 

Although SQL is strictly typed[*] it has some flexible types:

CREATE TABLE t (
   a INTEGER ARRAY[1000], -- maximum length
   x XML,
   j JSON
)

[*] SQLite being the famous exception (but see STRICT tables).

What is SQL anyway?

 Declarative 

SQL is generally referred to as being declarative… …but it also embraces ideas from the functional programming paradigm without enforcing its limitations. Functions may have side effects: NON DETERMINISTIC • MODIFIES SQL DATA

 Strictly Typed 

Although SQL is strictly typed[*] it has some flexible types:

CREATE TABLE t (
   a INTEGER ARRAY[1000], -- maximum length
   x XML,
   j JSON
)

[*] SQLite being the famous exception (but see STRICT tables).

What is SQL anyway?

What is SQL anyway?

After all,

What is SQL anyway?

After all,

SQL is very pragmatic

What is SQL anyway?

After all,

SQL is very pragmatic

While the relational model comes with rigid rules for doing it right, SQL offers many options to choose from and doesn’t judge on your choice. SQL broke free from the relational cage — in 1999.

Latest News Let’s start with the non-exciting stuff

Non-decimal Numbers: 0b101, 0o567, 0xBEEF
SELECT …
  FROM …
 WHERE id = 0b101010 -- binary literal 
Non-decimal Numbers: 0b101, 0o567, 0xBEEF
SELECT …
  FROM …
 WHERE id = 0b101010 -- binary literal 
SELECT …
  FROM …
 WHERE id = 0o52 -- octal literal      
Non-decimal Numbers: 0b101, 0o567, 0xBEEF
SELECT …
  FROM …
 WHERE id = 0b101010 -- binary literal 
SELECT …
  FROM …
 WHERE id = 0o52 -- octal literal      
SELECT …
  FROM …
 WHERE id = 0x2A -- hexadecimal literal
Non-decimal Numbers: 0b101, 0o567, 0xBEEF
SELECT …
  FROM …
 WHERE id = 0b101010 -- binary literal 
SELECT …
  FROM …
 WHERE id = 0o52 -- octal literal      
SELECT …
  FROM …
 WHERE id = 0x2A -- hexadecimal literal
SELECT …
  FROM …
 WHERE id = 42 -- decimal              
Non-decimal Numbers 0b101, 0o567, 0xBEEF: Availability in Various Dialects
BigQueryDb2 (LUW)MariaDBMySQLOracle DBPostgreSQLSQL ServerSQLite2005200720092011201320152017201920212023⚡ 3.8.11 - 3.50.0b⚡ 3.5.7 - 3.8.3a⚠ 2008R2 - 2022b✓ 16 - 17c⊘ 8.3 - 15⊘ 11gR1 - 23.9⚡ 5.0 - 9.3.0a⚡ 5.1 - 12.0.2a⊘ 9.7 - 12.1.2⚠ 2.0b
  1. Wrong results in cast
  2. Only hexadecimal, not binary nor octal
  3. Also with underscores (_) between the digits
Underscores in Numbers: 1_000_000
SELECT …
  FROM …
 WHERE id = 1_000_000
Underscores in Numbers: 1_000_000: Availability in Various Dialects
BigQueryDb2 (LUW)MariaDBMySQLOracle DBPostgreSQLSQL ServerSQLite2005200720092011201320152017201920212023⚡ 3.5.7 - 3.50.0a⊘ 2008R2 - 2022⚠ 16 - 17bc⊘ 8.3 - 15⊘ 11gR1 - 23.9⚡ 5.0 - 9.3.0a⚡ 5.1 - 12.0.2a⊘ 9.7 - 12.1.2⊘ 2.0
  1. Wrong results in cast
  2. Limited support in cast
  3. Also for binary, octal and hexadecimal notation
VARCHAR: Optional Maximum Length
CREATE TABLE … (
  x VARCHAR(/* what should I put here? */)       
)
VARCHAR: Optional Maximum Length
CREATE TABLE … (
  x VARCHAR(/* what should I put here? */)       
)
CREATE TABLE … (
  x VARCHAR -- Since SQL:2023, it's OK to skip it
)
VARCHAR without length: Availability in Various Dialects
BigQueryDb2 (LUW)MariaDBMySQLOracle DBPostgreSQLSQL ServerSQLite200820102012201420162018202020222024✓ 3.5.7 - 3.50.0⚠ 2008R2 - 2022ab✓ 8.3 - 17⊘ 11gR1 - 23.9⊘ 5.0 - 9.3.0⊘ 5.1 - 12.0.2⊘ 9.7 - 12.1.2
  1. Very low implementation-defined limit: 1
  2. Supports varchar(max) as reasonably long length limit
Null Treatment in UNIQUE Constraints
CREATE TABLE … (
  x BIGINT UNIQUE -- NULLs are allowed                          
)
INSERT INTO … (x) VALUES (NULL) -- First NULL, no doubt         
INSERT INTO … (x) VALUES (NULL) -- Should it work?
                                -- According to my understanding
                                -- of the standard before 2023
                                -- YES!
Null Treatment in UNIQUE Constraints
CREATE TABLE … (
  x BIGINT UNIQUE -- NULLs are allowed                          
)
INSERT INTO … (x) VALUES (NULL) -- First NULL, no doubt         
INSERT INTO … (x) VALUES (NULL) -- Should it work?
                                -- According to my understanding
                                -- of the standard before 2023
                                -- YES!
Null Treatment in UNIQUE Constraints
CREATE TABLE … (
  x BIGINT UNIQUE -- NULLs are allowed                          
)
INSERT INTO … (x) VALUES (NULL) -- First NULL, no doubt         
INSERT INTO … (x) VALUES (NULL) -- Should it work?
                                -- According to my understanding
                                -- of the standard before 2023
                                -- YES!
According to my Understanding Until SQL:2016
BigQueryDb2 (LUW)MariaDBMySQLOracle DBPostgreSQLSQL ServerSQLite2005200720092011201320152017201920212023✓ 3.5.7 - 3.50.0⊘ 2008R2 - 2022✓ 8.3 - 17⊘ 11gR1 - 23.9✓ 5.0 - 9.3.0✓ 5.1 - 12.0.2
Null Treatment in UNIQUE Constraints
CREATE TABLE … (
  x BIGINT UNIQUE NULLS DISTINCT    
)
INSERT INTO … (x) VALUES (NULL) -- first one works anyway           
INSERT INTO … (x) VALUES (NULL) -- Works if NULLS DISTINCT
                                -- Rejected if NULLS NOT DISTINCT
                                -- Default: “Implementation defined”
Null Treatment in UNIQUE Constraints
CREATE TABLE … (
  x BIGINT UNIQUE NULLS NOT DISTINCT
)
INSERT INTO … (x) VALUES (NULL) -- first one works anyway           
INSERT INTO … (x) VALUES (NULL) -- Works if NULLS DISTINCT
                                -- Rejected if NULLS NOT DISTINCT
                                -- Default: “Implementation defined”
Null Treatment in UNIQUE Constraints
CREATE TABLE … (
  x BIGINT UNIQUE                   
)
INSERT INTO … (x) VALUES (NULL) -- first one works anyway           
INSERT INTO … (x) VALUES (NULL) -- Works if NULLS DISTINCT
                                -- Rejected if NULLS NOT DISTINCT
                                -- Default: “Implementation defined”
NULLS [NOT] DISTINCT: Availability in Various Dialects
BigQueryDb2 (LUW)MariaDBMySQLOracle DBPostgreSQLSQL ServerSQLite20152017201920212023⊘ 3.5.7 - 3.50.0⊘ 2008R2 - 2022✓ 15 - 17⊘ 8.3 - 14⊘ 11gR1 - 23.9⊘ 5.0 - 9.3.0⊘ 5.1 - 12.0.2
NULLS FIRST|LAST: How to sort NULL values?
CREATE TABLE … (
  x BIGINT -- A NULL-able column
)
SELECT x
  FROM …
 ORDER BY x -- Where should the NULLs appear?
            -- It is “Implementation defined”
            -- Eighter before non-NULL values
            -- or after them, but not inbetween
NULLS FIRST|LAST: How to sort NULL values?
CREATE TABLE … (
  x BIGINT -- A NULL-able column
)
SELECT x
  FROM …
 ORDER BY x -- Where should the NULLs appear?
            -- It is “Implementation defined”
            -- Eighter before non-NULL values
            -- or after them, but not inbetween
NULLS FIRST|LAST: How to sort NULL values?
CREATE TABLE … (
  x BIGINT -- A NULL-able column
)
SELECT x
  FROM …
 ORDER BY x [ASC|DESC] NULLS FIRST
            -- Since 2003, the NULLS FIRST|LAST
            -- option is standardized

NULLS FIRST|LAST: Availability in Various Dialects
BigQueryDb2 (LUW)MariaDBMySQLOracle DBPostgreSQLSQL ServerSQLite200720092011201320152017201920212023✓ 3.30.0 - 3.50.0⊘ 3.5.7 - 3.29.0⊘ 2008R2 - 2022✓ 8.3 - 17✓ 11gR1 - 23.9⊘ 5.0 - 9.3.0⊘ 5.1 - 12.0.2✓ 11.1 - 12.1.2⊘ 9.7 - 10.5✓ 2.0⊘ 2.0
GREATEST / LEAST: Non-aggregate min/max
SELECT GREATEST(a, b, c)
     , LEAST(a, b, c)
  FROM …

The SQL functions greatest and least take one or more arguments and return the largest or smallest value if none of the arguments is null.

GREATEST / LEAST: Availability in Various Dialects
BigQueryDb2 (LUW)MariaDBMySQLOracle DBPostgreSQLSQL ServerSQLite2005200720092011201320152017201920212023⊘ 3.5.7 - 3.50.0b⚠ 2022a⊘ 2008R2 - 2019⚠ 8.4 - 17c⚠ 8.3a✓ 11gR1 - 23.9✓ 5.0 - 9.3.0✓ 5.1 - 12.0.2✓ 9.7 - 12.1.2✓ 2.0
  1. ⚡Non-conforming null handling
  2. Alternative: scalar min and max functions
  3. Non-conforming handling of null elements in row values
ANY_VALUE: When it does not matter
SELECT g
     , x -- selecting non-grouping column is not allowed
  FROM …
 GROUP BY g
ANY_VALUE: When it does not matter
SELECT g
     , x -- selecting non-grouping column is not allowed
  FROM …
 GROUP BY g
SELECT g
     , ANY_VALUE(x) -- just take any non-null value     
  FROM …
 GROUP BY g

You could always use min or max, but they might impose unnecessary overhead.

ANY_VALUE: Availability in Various Dialects
BigQueryDb2 (LUW)MariaDBMySQLOracle DBPostgreSQLSQL ServerSQLite201320152017201920212023⊘ 3.5.7 - 3.50.0⊘ 2008R2 - 2022✓ 16 - 17⊘ 8.3 - 15✓ 23.0 - 23.9d✓ 19c - 21ccd⊘ 11gR1 - 18c⚡ 5.7 - 9.3.0abcd⊘ 5.0 - 5.6⊘ 5.1 - 12.0.2⊘ 9.7 - 12.1.2⚡ 2.0a✓ 2.0a⚡ 2.0ac
  1. ⚡Might return null even if there are non-null values
  2. Not an aggregate function
  3. Without filter clause
  4. Not as window function (over clause)

JSON It’s not about persistence

JSON: It’s not about persistence
[ {"product": 42}, {"product": 123, "qty": 2} ]
INSERT INTO order_lines (product, qty) -- directly load into table
SELECT product, qty
  FROM JSON_TABLE(?,    -- bind parameter for the JSON document
                 '$[*]' -- SQL JSON/Path: one row per hit

         COLUMNS ()

       ) t
| product | qty |
+---------+-----+
|      42 |   1 |
|     123 |   2 | 
JSON: It’s not about persistence
[ {"product": 42}, {"product": 123, "qty": 2} ]
INSERT INTO order_lines (product, qty) -- directly load into table
SELECT product, qty
  FROM JSON_TABLE(?,    -- bind parameter for the JSON document
                 '$[*]' -- SQL JSON/Path: one row per hit

         COLUMNS ()

       ) t
| product | qty |
+---------+-----+
|      42 |   1 |
|     123 |   2 | 
JSON: It’s not about persistence
[ {"product": 42}, {"product": 123, "qty": 2} ]
INSERT INTO order_lines (product, qty) -- directly load into table
SELECT product, qty
  FROM JSON_TABLE(?,    -- bind parameter for the JSON document
                 '$[*]' -- SQL JSON/Path: one row per hit

         COLUMNS ()

       ) t
| product | qty |
+---------+-----+
|      42 |   1 |
|     123 |   2 | 
JSON: It’s not about persistence
[ {"product": 42}, {"product": 123, "qty": 2} ]
INSERT INTO order_lines (product, qty) -- directly load into table
SELECT product, qty
  FROM JSON_TABLE(?,    -- bind parameter for the JSON document
                 '$[*]' -- SQL JSON/Path: one row per hit

         COLUMNS ()

       ) t
| product | qty |
+---------+-----+
|      42 |   1 |
|     123 |   2 | 
JSON: It’s not about persistence
[ {"product": 42}, {"product": 123, "qty": 2} ]
INSERT INTO order_lines (product, qty) -- directly load into table
SELECT product, qty
  FROM JSON_TABLE(?,    -- bind parameter for the JSON document
                 '$[*]' -- SQL JSON/Path: one row per hit
         COLUMNS ( /* after specifying the rows to produce,
                      let's declare the columns */           )

       ) t
| product | qty |
+---------+-----+
|      42 |   1 |
|     123 |   2 | 
JSON: It’s not about persistence
[ {"product": 42}, {"product": 123, "qty": 2} ]
INSERT INTO order_lines (product, qty) -- directly load into table
SELECT product, qty
  FROM JSON_TABLE(?,    -- bind parameter for the JSON document
                 '$[*]' -- SQL JSON/Path: one row per hit
         COLUMNS (product INT PATH '$.product' ERROR ON EMPTY,
--[column name]---^^^^^^^                                    )

       ) t
| product | qty |
+---------+-----+
|      42 |   1 |
|     123 |   2 | 
JSON: It’s not about persistence
[ {"product": 42}, {"product": 123, "qty": 2} ]
INSERT INTO order_lines (product, qty) -- directly load into table
SELECT product, qty
  FROM JSON_TABLE(?,    -- bind parameter for the JSON document
                 '$[*]' -- SQL JSON/Path: one row per hit
         COLUMNS (product INT PATH '$.product' ERROR ON EMPTY,
----------[column type]---^^^                                )

       ) t
| product | qty |
+---------+-----+
|      42 |   1 |
|     123 |   2 | 
JSON: It’s not about persistence
[ {"product": 42}, {"product": 123, "qty": 2} ]
INSERT INTO order_lines (product, qty) -- directly load into table
SELECT product, qty
  FROM JSON_TABLE(?,    -- bind parameter for the JSON document
                 '$[*]' -- SQL JSON/Path: one row per hit
         COLUMNS (product INT PATH '$.product' ERROR ON EMPTY,
----------[Path: Which JSON item]---^^^^^^^^^                )

       ) t
| product | qty |
+---------+-----+
|      42 |   1 |
|     123 |   2 | 
JSON: It’s not about persistence
[ {"product": 42}, {"product": 123, "qty": 2} ]
INSERT INTO order_lines (product, qty) -- directly load into table
SELECT product, qty
  FROM JSON_TABLE(?,    -- bind parameter for the JSON document
                 '$[*]' -- SQL JSON/Path: one row per hit
         COLUMNS (product INT PATH '$.product' ERROR ON EMPTY,
----[What if the path doesn't hit anything?]---^^^^^^^^^^^^^^)

       ) t
| product | qty |
+---------+-----+
|      42 |   1 |
|     123 |   2 | 
JSON: It’s not about persistence
[ {"product": 42}, {"product": 123, "qty": 2} ]
INSERT INTO order_lines (product, qty) -- directly load into table
SELECT product, qty
  FROM JSON_TABLE(?,    -- bind parameter for the JSON document
                 '$[*]' -- SQL JSON/Path: one row per hit
         COLUMNS (product INT PATH '$.product' ERROR ON EMPTY,
                  qty     INT PATH '$.qty' DEFAULT 1 ON EMPTY)
-------------[If "qty" is missing, use 1]--^^^^^^^^^^^^^^^^^^
       ) t
| product | qty |
+---------+-----+
|      42 |   1 |
|     123 |   2 | 
JSON: It’s not about persistence
[ {"product": 42}, {"product": 123, "qty": 2} ]
INSERT INTO order_lines (product, qty) -- directly load into table
SELECT product, qty
  FROM JSON_TABLE(?,    -- bind parameter for the JSON document
                 '$[*]' -- SQL JSON/Path: one row per hit
         COLUMNS (product INT PATH '$.product' ERROR ON EMPTY,
                  qty     INT PATH '$.qty' DEFAULT 1 ON EMPTY)
         ERROR ON ERROR -- remaining errors abort
       ) t
| product | qty |
+---------+-----+
|      42 |   1 |
|     123 |   2 | 
JSON: It’s not about persistence
[ {"product": 42}, {"product": 123, "qty": 2} ]
INSERT INTO order_lines (product, qty) -- directly load into table
SELECT product, qty
  FROM JSON_TABLE(?,    -- bind parameter for the JSON document
                 '$[*]' -- SQL JSON/Path: one row per hit
         COLUMNS (product INT PATH '$.product' ERROR ON EMPTY,
                  qty     INT PATH '$.qty' DEFAULT 1 ON EMPTY)
         ERROR ON ERROR -- remaining errors abort
       ) t
| product | qty |
+---------+-----+
|      42 |   1 |
|     123 |   2 | 
json_table (or similar): Availability in Various Dialects
BigQueryDb2 (LUW)MariaDBMySQLOracle DBPostgreSQLSQL ServerSQLite201320152017201920212023⚠ 3.9.1 - 3.50.0b⊘ 3.5.7 - 3.8.11⚠ 2016 - 2022c⊘ 2008R2 - 2014✓ 17+ae⚠ 9.4 - 16a⊘ 8.3 - 9.3✓ 12cR1 - 23.9⊘ 11gR1 - 11gR2✓ 8.0.11 - 9.3.0⊘ 5.0 - 5.7✓ 10.6 - 12.0.2⊘ 5.1 - 10.5✓ 11.5.0 - 12.1.2⊘ 9.7 - 11.1⚠ 2.0d
  1. Alternative: jsonb_to_recordset
  2. Alternative: json_each and json_extract
  3. Alternative: openjson
  4. Alternative: json_query_array, unnest and json_value
  5. In 17beta1
JSON Duality Views: Oracle’s Automagic

SQL is an…

JSON Duality Views: Oracle’s Automagic

SQL is an…

Extensible Industry Standard

JSON Duality Views: Oracle’s Automagic

SQL is an…

Extensible Industry Standard

The Oracle Database can do the mapping automatically:

CREATE JSON DUALITY VIEW v AS 
SELECT JSON {'_id': {'cid': cid, 'ts': ts},
             'items': [ SELECT JSON {'product': product
                                    ,'qty'    : qty
                                    }
                          FROM order_items
                          WITH INSERT UPDATE DELETE
                         WHERE ol.cid=o.cid AND ol.ts=o.ts
                      ]
            }
  FROM orders WITH INSERT UPDATE DELETE
JSON Duality Views: Oracle’s Automagic
CREATE JSON DUALITY VIEW v AS 
SELECT JSON {'_id': {'cid': cid, 'ts': ts},
             'items': [ SELECT JSON {'product': product
                                    ,'qty'    : qty
                                    }
                          FROM order_items
                          WITH INSERT UPDATE DELETE
                         WHERE ol.cid=o.cid AND ol.ts=o.ts
                      ]
            }
  FROM orders WITH INSERT UPDATE DELETE
INSERT INTO v VALUES ('{"_id":{"cid": 1}
                       ,"items": [{"product": 42}
                                 ,{"product": 123, "qty":2}
                                 ]
                       }')
JSON Duality Views: Oracle’s Automagic
INSERT INTO v VALUES ('{"_id":{"cid": 1}
                       ,"items": [{"product": 42}
                                 ,{"product": 123, "qty":2}
                                 ]
                       }')
SELECT * FROM orders

   | cid | ts  |
   +-----+-----+
   |   1 | ... |
   |   1 | ... |
  SELECT * FROM order_items

| cid | ts  | product | qty |
+-----+-----+---------+-----+
|   1 | ... |      42 |   1 |
|   1 | ... |     123 |   2 |
JSON duality views: Availability in Various Dialects
BigQueryDb2 (LUW)MariaDBMySQLOracle DBPostgreSQLSQL ServerSQLite20152017201920212023⊘ 3.5.7 - 3.50.0⊘ 2008R2 - 2022⊘ 8.3 - 17⚠ 23.4 - 23.9a⊘ 11gR1 - 23.0⊘ 5.0 - 9.3.0⊘ 5.1 - 12.0.2⊘ 9.7 - 12.1.2⊘ 2.0
  1. Non-standard functionality
JSON: It’s not about persistence

A very common problem: IN lists of dynamic length

JSON: It’s not about persistence

A very common problem: IN lists of dynamic length

ORA-01795: maximum number of expressions in a list is 1000
JSON: It’s not about persistence

A very common problem: IN lists of dynamic length

SELECT tbl.i
  FROM tbl
 WHERE i IN (SELECT x        -- vvvvv--[in practice: bind param]
               FROM JSON_TABLE('[1,3]'
                              ,'$[*]'
                      COLUMNS (x INTEGER PATH '$')
                      ERROR ON ERROR
                    ) jt
            )
JSON: It’s not about persistence

A very common problem: IN lists of dynamic length

SELECT tbl.i
  FROM tbl
 WHERE i IN (SELECT x        -- vvvvv--[in practice: bind param]
               FROM JSON_TABLE('[1,3]'
                              ,'$[*]'
                      COLUMNS (x INTEGER PATH '$')
                      ERROR ON ERROR
                    ) jt
            )
PostgreSQL: SELECT i::INT FROM json_array_elements_text(?) t(i)
    SQLite: SELECT value  FROM json_each(?)
SQL Server: SELECT value  FROM openjson(?)
JSON: It’s not about persistence

Another common problem: Dynamic pivot

| year | revenue |
+------+---------+
| 2024 |   ...   |
| 2023 |   ...   |
| ...  |   ...   |
JSON: It’s not about persistence

Another common problem: Dynamic pivot

As SQL is statically typed: columns must be declared upfront.

| year | revenue |
+------+---------+
| 2024 |   ...   |
| 2023 |   ...   |
| ...  |   ...   |
JSON: It’s not about persistence

Another common problem: Dynamic pivot

As SQL is statically typed: columns must be declared upfront.

SELECT SUM(revenue) FILTER(WHERE year = 2024) AS revenue_2024
     , SUM(revenue) FILTER(WHERE year = 2023) AS revenue_2023
     , …
  FROM sales
 GROUP BY year
| year | revenue |
+------+---------+
| 2024 |   ...   |
| 2023 |   ...   |
| ...  |   ...   |
JSON: It’s not about persistence

Another common problem: Dynamic pivot

As SQL is statically typed: columns must be declared upfront.

SELECT SUM(revenue) FILTER(WHERE year = 2024) AS revenue_2024
     , SUM(revenue) FILTER(WHERE year = 2023) AS revenue_2023
     , …
  FROM sales
 GROUP BY year

But you can also put everything in a single column:

SELECT JSON_OBJECTAGG(year VALUE revenue)
  FROM (SELECT year, SUM(revenue) revenue
          FROM sales
         GROUP BY year
       ) t
| year | revenue |
+------+---------+
| 2024 |   ...   |
| 2023 |   ...   |
| ...  |   ...   |
JSON: It’s not about persistence

Another common problem: Dynamic pivot

As SQL is statically typed: columns must be declared upfront.

SELECT SUM(revenue) FILTER(WHERE year = 2024) AS revenue_2024
     , SUM(revenue) FILTER(WHERE year = 2023) AS revenue_2023
     , …
  FROM sales
 GROUP BY year

But you can also put everything in a single column:

SELECT JSON_OBJECTAGG(year VALUE revenue)
  FROM (SELECT year, SUM(revenue) revenue
          FROM sales
         GROUP BY year
       ) t
{"2024": ...      
,"2023": ...
...
}
JSON: It’s not about persistence

Another common problem: Dynamic pivot

As SQL is statically typed: columns must be declared upfront.

SELECT SUM(revenue) FILTER(WHERE year = 2024) AS revenue_2024
     , SUM(revenue) FILTER(WHERE year = 2023) AS revenue_2023
     , …
  FROM sales
 GROUP BY year

But you can also put everything in a single column:

SELECT JSON_OBJECTAGG(year VALUE revenue)
  FROM (SELECT year, SUM(revenue) revenue
          FROM sales
         GROUP BY year
       ) t
{"2024": ...      
,"2023": ...
...
}
json_objectagg (or similar): Availability in Various Dialects
BigQueryDb2 (LUW)MariaDBMySQLOracle DBPostgreSQLSQL ServerSQLite20152017201920212023⚠ 3.10.2 - 3.50.0a⊘ 3.5.7 - 3.9.1⊘ 2022b⊘ 2008R2 - 2019✓ 16 - 17⊘ 8.3 - 15✓ 12cR2 - 23.9⊘ 11gR1 - 12cR1✓ 8.0.11 - 9.3.0⊘ 5.0 - 5.7✓ 10.6 - 12.0.2⊘ 5.1 - 10.5⊘ 9.7 - 12.1.2⚠ 2.0c
  1. Alternative: json_group_object
  2. In preview
  3. Alternative: JSON_OBJECT and ARRAY_AGG
JSON: It’s not about persistence

What do these examples have in common? json_table • JSON duality views • IN lists • Dynamic pivot

JSON: It’s not about persistence

What do these examples have in common? json_table • JSON duality views • IN lists • Dynamic pivot

They don’t store JSON in the database They just use it as an interface

SQL’s JSON support in not so interesting because it allows us to store JSON in tables. It is so interesting to because it allows us to process JSON in SQL.

Recursion Non-trivial queries need logic

Recursion: Non-trivial queries need logic
WITH RECURSIVE prev (id, parent) AS (
   SELECT t.id, t.parent
     FROM t
    WHERE t.id = 42
UNION ALL
   SELECT t.id, t.parent
     FROM t
    WHERE t.parent = 42

       ON t.parent = prev.id

)
SELECT * FROM prev
Recursion: Non-trivial queries need logic
WITH RECURSIVE prev (id, parent) AS (
   SELECT t.id, t.parent
     FROM t
    WHERE t.id = 42
UNION ALL
   SELECT t.id, t.parent
     FROM t
    WHERE t.parent = 42

       ON t.parent = prev.id

)
SELECT * FROM prev
Recursion: Non-trivial queries need logic
WITH RECURSIVE prev (id, parent) AS (
   SELECT t.id, t.parent
     FROM t
    WHERE t.id = 42
UNION ALL
   SELECT t.id, t.parent
     FROM t
    WHERE t.parent = 42

       ON t.parent = prev.id

)
SELECT * FROM prev
42
Recursion: Non-trivial queries need logic
WITH RECURSIVE prev (id, parent) AS (
   SELECT t.id, t.parent
     FROM t
    WHERE t.id = 42
UNION ALL
   SELECT t.id, t.parent
     FROM t
    WHERE t.parent = 42

       ON t.parent = prev.id

)
SELECT * FROM prev
42
Recursion: Non-trivial queries need logic
WITH RECURSIVE prev (id, parent) AS (
   SELECT t.id, t.parent
     FROM t
    WHERE t.id = 42
UNION ALL
   SELECT t.id, t.parent
     FROM t
    WHERE t.parent = 42

       ON t.parent = prev.id

)
SELECT * FROM prev
42
Recursion: Non-trivial queries need logic
WITH RECURSIVE prev (id, parent) AS (
   SELECT t.id, t.parent
     FROM t
    WHERE t.id = 42
UNION ALL
   SELECT t.id, t.parent
     FROM t
    WHERE t.parent = 42

       ON t.parent = prev.id

)
SELECT * FROM prev
42
Recursion: Non-trivial queries need logic
WITH RECURSIVE prev (id, parent) AS (
   SELECT t.id, t.parent
     FROM t
    WHERE t.id = 42
UNION ALL
   SELECT t.id, t.parent
     FROM t
 /* WHERE t.parent = 42 */
     JOIN prev
       ON t.parent = prev.id

)
SELECT * FROM prev
42
Recursion: Non-trivial queries need logic
WITH RECURSIVE prev (id, parent) AS (
   SELECT t.id, t.parent
     FROM t
    WHERE t.id = 42
UNION ALL
   SELECT t.id, t.parent
     FROM t
 /* WHERE t.parent = 42 */
     JOIN prev
    /* ON t.parent = prev.id */
       ON t.id     = prev.parent
)
SELECT * FROM prev
42
WITH [RECURSIVE]: Availability in Various Dialects
BigQueryDb2 (LUW)MariaDBMySQLOracle DBPostgreSQLSQL ServerSQLite20092011201320152017201920212023✓ 3.8.3 - 3.50.0⊘ 3.5.7 - 3.7.11✓ 2008R2 - 2022✓ 8.4 - 17⊘ 8.3✓ 11gR2 - 23.9⊘ 11gR1✓ 8.0.11 - 9.3.0⊘ 5.0 - 5.7✓ 10.2 - 12.0.2⊘ 5.1 - 10.1✓ 9.7 - 12.1.2✓ 2.0

https://modern-sql.com/blog/2020-05/java-and-sql-stronger-together

Graphs Common Requirements Justify Dedicated Syntax

Graphs: Common Requirements Justify Dedicated Syntax
CREATE PROPERTY GRAPH grph
 VERTEX TABLES (
  t AS item KEY (id)
 )
 EDGE TABLES (
  t AS parent KEY (id)
       SOURCE KEY (id)     REFERENCES item (id)
  DESTINATION KEY (parent) REFERENCES item (id)
 )
SELECT *
  FROM GRAPH_TABLE (grph
        MATCH (i0) <-[]- {,10} (ix)   

        WHERE  i0.id = 42
        COLUMNS (ix.*) 
       ) gt
 ORDER BY id
Graphs: Common Requirements Justify Dedicated Syntax
CREATE PROPERTY GRAPH grph
 VERTEX TABLES (
  t AS item KEY (id)
 )
 EDGE TABLES (
  t AS parent KEY (id)
       SOURCE KEY (id)     REFERENCES item (id)
  DESTINATION KEY (parent) REFERENCES item (id)
 )
SELECT *
  FROM GRAPH_TABLE (grph
        MATCH (i0) <-[]- {,10} (ix)   

        WHERE  i0.id = 42
        COLUMNS (ix.*) 
       ) gt
 ORDER BY id
Graphs: Common Requirements Justify Dedicated Syntax
CREATE PROPERTY GRAPH grph
 VERTEX TABLES (
  t AS item KEY (id)
 )
 EDGE TABLES (
  t AS parent KEY (id)
       SOURCE KEY (id)     REFERENCES item (id)
  DESTINATION KEY (parent) REFERENCES item (id)
 )
SELECT *
  FROM GRAPH_TABLE (grph
     /* MATCH (i0) <-[]- {,10} (ix) */
        MATCH (i0)  -[]->{,10} (ix)
        WHERE  i0.id = 42
        COLUMNS (ix.*) 
       ) gt
 ORDER BY id
SQL/PGQ: Availability in Various Dialects
BigQueryDb2 (LUW)MariaDBMySQLOracle DBPostgreSQLSQL ServerSQLite20152017201920212023⊘ 3.5.7 - 3.50.0⚠ 2019 - 2022a⊘ 2008R2 - 2017⊘ 8.3 - 17✓ 23.4 - 23.9⊘ 11gR1 - 23.0⊘ 5.0 - 9.3.0⊘ 5.1 - 12.0.2⊘ 9.7 - 12.1.2⊘ 2.0
  1. Limited alternative: Graph processing with SQL Server and Azure SQL Database

A lot has happened since SQL-92

A lot has happened since SQL-92

SQL has evolved beyond the relational idea

A lot has happened since SQL-92

SQL has evolved beyond the relational idea

If you use SQL for CRUD operations only, you are doing it wrong

A lot has happened since SQL-92

SQL has evolved beyond the relational idea

If you use SQL for CRUD operations only, you are doing it wrong

https://modern-sql.com/ https://winand.at/