Categorical Databases

Transcription

1 Categorical Databases Patrick Schultz, David Spivak MIT Ryan Wisnesky Categorical Informatics and others October 2017

2 Introduction This talk describes a new algebraic (purely equational) way to formalize databases based on category theory. Category theory was designed to migrate theorems from one area of mathematics to another, but researchers at MIT developed a way to use it to migrate data from one schema to another. Research has culminated in an open-source prototype ETL and data integration tool, AQL (Algebraic Query Language), available at categoricaldata.net/aql.html. (These slides are also there.) Goal: Categorical databases needs you needs a community to grow. Please, talk to us and to each other and get involved. Outline: Review of basic category theory. Introduction to AQL. AQL demo. Optional interlude: additional AQL constructions. How AQL instances model the simply-typed λ-calculus. 2 / 35

3 Category Theory A category C consists of a set of objects, ObpCq forall X, Y P ObpCq, a set CpX, Y q of morphisms a.k.a arrows forall X P ObpCq, a morphism id P CpX, Xq forall X, Y, Z P ObpCq, a function : CpY, Zq ˆ CpX, Y q Ñ CpX, Zq s.t. f id f id f f pf gq h f pg hq The category Set has sets as objects and functions as arrows, and the category Haskell has types as objects and programs as arrows. A functor F : C Ñ D between categories C, D consists of a function ObpCq Ñ ObpDq forall X, Y P ObpCq, a function CpX, Y q Ñ DpF pxq, F py qq s.t. F pidq id F pf gq F pfq F pgq The functor P : Set Ñ Set takes each set to its power set, and the functor List : Haskell Ñ Haskell takes each type t to the type List t. 3 / 35

4 Schemas and Instances manager Emp works secretary Dept last first String rmanager.workss rworkss rsecretary.workss rs Emp mgr works first last q10 Al Akin x02 Bob Bo q10 Carl Cork Dept sec q CS x Math String Al Bob... 4 / 35

5 An AQL Schema: Code entities Emp Dept foreign keys manager : Emp -> Emp works : Emp -> Dept secretary : Dept -> Emp attributes first last : Emp -> string : Dept -> string path equations manager.works = works secretary.works = Department 5 / 35

6 Categorical Semantics of Schemas and Instances The meaning of a schema S is a category S. Ob(S) is the nodes of S. Forall nodes X, Y, SpX, Y q is the set of finite paths X Ñ Y, modulo the path equivalences in S. Path equivalence in S may not be decidable! ( the word problem ) A morphism of schemas (a schema mapping ) S Ñ T is a functor S Ñ T. It can be defined as an equation-preserving function: nodespsq Ñ nodespt q edgespsq Ñ pathspt q. An S-instance is a functor S Ñ Set. It can be defined as a set of tables, one per node in S and one column per edge in S, satisfying the path equivalences in S. A morphism of S-instances I Ñ J (a data mapping ) is a natural transformation I Ñ J. Instances on S and their mappings form a category, written S-inst. 6 / 35

7 Schema Mappings A schema mapping F : S Ñ T is an equation-preserving function: nodespsq Ñ nodespt q edgespsq Ñ pathspt q N1 String f N2 F ÝÝÝÑ String N salary Int age salary Int age F pintq Int F pstringq String F pn1q N F pn2q N F pq rs F pageq rages F psalaryq rsalarys F pfq rs 7 / 35

8 Functorial Data Migration A schema mapping F : S Ñ T induces three data migration functors: F : T -inst Ñ S-inst (like project) S F T I F piq : I F Set Π F : S-inst Ñ T -inst (right adjoint to F ; like J. S-instp F piq, Jq T -instpi, Π F pjqq Σ F : S-inst Ñ T -inst (left adjoint to F ; like outer union then J. S-instpJ, F piqq T -instpσ F pjq, Iq 8 / 35

9 (Project) N1 String N2 F ÝÝÝÑ String N salary Int age salary Int age N1 salary 1 Alice $100 2 Bob $250 3 Sue $300 N2 age F ÐÝÝ N salary age a Alice $ b Bob $ c Sue $ / 35

10 Π (Product) N1 String N2 F ÝÝÝÑ String N salary Int age salary Int age N1 salary 1 Alice $100 2 Bob $250 3 Sue $300 N2 age Π F ÝÝÑ N salary age a Alice $ b Alice $ c Alice $ d Bob $ e Bob $ f Bob $ g Sue $ h Sue $ i Sue $ / 35

11 Σ (Outer Union) N1 String N2 F ÝÝÝÑ String N salary Int age salary Int age N1 Name Salary 1 Alice $100 2 Bob $250 3 Sue $300 N2 Age Σ F ÝÝÑ N Name Salary Age a Alice $100 null 1 b Bob $250 null 2 c Sue $300 null 3 d null 4 null 5 20 e null 6 null 7 20 f null 8 null / 35

12 Unit of Σ F % F N1 Name Salary 1 Alice $100 2 Bob $250 3 Sue $300 N1 Name Salary a Alice $100 b Bob $250 c Sue $300 d null 4 null 5 e null 6 null 7 f null 8 null 9 đη N2 Age N2 Age a null 1 b null 2 c null 3 d 20 e 20 f 30 Σ F ÝÝÑ F Ö N Name Salary Age a Alice $100 null 1 b Bob $250 null 2 c Sue $300 null 3 d null 4 null 5 20 e null 6 null 7 20 f null 8 null / 35

13 A Foreign Key N1 salary String f Int N2 age F ÝÝÝÑ String N salary Int age N1 salary f 1 Alice $ Bob $ Sue $300 6 N2 age F ÐÝÝ Π F,Σ F ÝÝÝÝÑ N salary age a Alice $ b Bob $ c Sue $ / 35

14 Queries A query Q : S Ñ T is a schema X and mappings F : S Ñ X and G : T Ñ X. eval Q G Π F coeval Q F Π G These can be specified using comprehension notation similar to SQL. N1 String f N2 Q ÐÝ String N salary Int age salary Int age N1 -> select n1. as, n1.salary as salary from N as n1 N2 -> select n2.age as age from N as n2 f -> {n2 -> n1} 14 / 35

15 A Foreign Key N1 salary String f Int N2 age Q ÐÝ String N salary Int age N1 salary f 1 Alice $ Bob $ Sue $300 6 N2 age eval Q ÐÝÝÝÝ coeval Q ÝÝÝÝÝÑ N salary age a Alice $ b Bob $ c Sue $ / 35

16 AQL Demo AQL implements, Σ, Π, and more in software. catinf.com The AQL execution engine is an automated theorem prover. Value proposition: AQL catches mistakes at compile time that existing ETL / data integration tools catch at runtime if at all. Data import and export by JDBC-SQL and CSV. We are looking for collaborators for a real-world pilot project. 16 / 35

17 Interlude - Additional Constructions What is algebraic here? AQL vs SQL. Pivot. Non-equational data integrity constraints. Data integration via pushouts. AQL vs comprehension calculi. 17 / 35

18 Why Algebraic? A schema can be identified with an algebraic (equational) theory. Emp Dept String : Type first last : Emp Ñ String : Dept Ñ String works : Emp Ñ Dept mgr : Emp Ñ Emp secr : Dept Ñ : Emp. workspmanagerpeqq : Dept. workspsecretarypdqq d This perspective makes it easy to add functions such as ` : Int, Int Ñ Int to a schema. See Algebraic Databases. An S-instance can be identified with the initial algebra of an algebraic theory extending S : Emp q10 x02 : Dept mgrp101q 103 worksp101q q10... Treating instances as theories allows instances that are infinite or inconsistent (e.g., Alice=Bob). 18 / 35

19 AQL vs SQL Data migration triplets of the form Σ F Π G H can be expressed using relational algebra and keygen, provided: F is a discrete op-fibration (ensures union compatibility). G is surjective on attributes (ensures domain independence). All categories are finite (ensures computability). The difference-free fragment of relational algebra can be expressed using such triplets. See Relational Foundations. Such triplets can be written in foreign-key aware SQL-ish syntax. 19 / 35

20 Select-From-Where Syntax manager Emp works secretary Dept last first String Find the of every manager s department: AQL select e.manager.works. from Emp as e SQL select d. from Emp as e1, Emp as e2, Dept as d where e1.manager = e2. and e2.works = d. 20 / 35

21 Pivot (Instance ô Schema) CS q first works last Al Akin Math x02 works mgr works first mgr first last Bob last Carl Bo Cork mgr Emp mgr works first last q10 Al Akin x02 Bob Bo q10 Carl Cork Dept q10 CS x02 Math 21 / 35

22 Richer Constraints Not all data integrity constraints are equational (e.g., keys). A data mapping ϕ : A Ñ E defines a constraint: instance I satisfies ϕ if for every α : A Ñ I there exists an ɛ : E Ñ I s.t α ɛ ϕ. ϕ A E Most constraints used in practice can be captured the above way. E.g., α ɛ I is captured 1, d 2 : Dept. pd 1 q pd 2 q Ñ d 1 d 2 ApDeptq td 1, d 2 u EpDeptq tdu Apqpd 1 q Apqpd 2 q ϕpd 1 q ϕpd 2 q d See Database Queries and Constraints via Lifting Problems and Algebraic Model Management. 22 / 35

23 Pushouts A pushout of p, q is f, g s.t. for every f 1, g 1 there is a unique m s.t.: p q f g f 1 m g 1 The category of schemas has all pushouts. For every schema S, the category S-inst has all pushouts. Pushouts of schemas, instances, and Σ are used together to integrate data - see Algebraic Data Integration. 23 / 35

24 Using Pushouts for Data Integration Step 1: integrate schemas. Given input schemas S 1, S 2, an overlap schema S, and mappings F 1, F 2 : F S 1 F 1 Ð S Ñ 2 S2 we propose to use their pushout T as the integrated schema: G S 1 G 1 Ñ T Ð 2 S2 Step 2: integrate data. Given input S 1 -instance I 1, S 2 -instance I 2, overlap S-instance I and data mappings h 1 : Σ F1 piq Ñ I 1 and h 2 : Σ F2 piq Ñ I 2, we propose to use the pushout of: Σ G1 pi 1 q Σ G 1 ph 1 q Ð as the integrated T -instance. `Σ G1 F 1 piq Σ G2 F 2 piq Σ G 2 ph 2 q Ñ Σ G2 pi 2 q 24 / 35

25 Schema Integration Observation Observation g 1 Method Person f Ó g Type ÝÝÑ Person f Ó g 2 Type Observation Observation g 1 Method Person f h Gender g Type ÝÝÑ Person f h Gender g 2 Type 25 / 35

26 Data Integration Observation f g Person p Type BP Wt Ñ Gender F M Type BP Wt HR Observation f g o 5 Peter BP o 6 Paul HR o 7 Peter Wt Person h Paul M Peter M Ó Ó Method g2 m 1 BP m 2 BP m 3 Wt m 4 Wt Observation f g1 o 1 Pete m 1 o 2 Pete m 2 o 3 Jane m 3 o 4 Jane m 1 Type BP Wt Person Jane Pete Ñ Method g2 null 1 BP null 2 Wt null 3 HR m 1 BP m 2 BP m 3 Wt m 4 Wt Gender F M null 4 Type BP Wt HR Observation f g1 o 1 Peter m 1 o 2 Peter m 2 o 3 Jane m 3 o 4 Jane m 1 o 5 Peter null 1 o 6 Paul null 2 o 7 Peter null 3 Person h Jane null 4 Paul M Peter M 26 / 35

27 AQL vs LINQ Treating entity sets as types rather than terms makes AQL a conceptual dual to comprehension calculi (e.g., LINQ). See QINL: Query-Integrated Languages. LINQ enriches programs with (schemas, queries and instances). Collections are terms Employee: Set Int e: Employee is not a judgment. There is a term P: Int ˆ Set Int Ñ Bool. manager: Set pint ˆ Intq AQL enriches (schemas, queries and instances) with programs. Collections are types Employee: Type manager: Employee Ñ Employee e: Employee is a judgment. There is not a term P: Employee ˆ Type Ñ Bool. LINQ is more popular, but AQL s style is common in Coq, Agda, etc. 27 / 35

28 AQL is one level up from LINQ LINQ Schemas are collection types over a base type theory Set pint ˆ Stringq Instances are terms tp1, CSqu Y tp2, Mathqu Data migrations are functions π 1 : Set pint ˆ Stringq Ñ Set Int AQL Schemas are type theories over a base type theory Dept, : Dept Ñ String Instances are term models (initial algebras) of theories d 1, d 2 : Dept, pd 1 q CS, pd 2 q Math Data migrations are functors Dept : pdept, : Dept Ñ Stringq - inst Ñ pdeptq - inst 28 / 35

29 Part 2 For every schema S, S-inst models simply-typed λ-calculus (STLC). The STLC is the core of typed functional languages ML, Haskell, etc. We will use the internal language of a cartesian closed category, which is equivalent to the STLC. Lots of point-free functional programming ahead. The category of schemas and mappings is also cartesian closed - see talk at Boston Haskell. 29 / 35

30 Categorical Abstract Machine Language (CAML) Types t: t :: 1 t ˆ t t t Terms f, g: id t : t Ñ t pq t : t Ñ 1 π 1 s,t : s ˆ t Ñ s π 2 s,t : s ˆ t Ñ t eval s,t : t s ˆ s Ñ t Equations: f : s Ñ u g f : s Ñ t g : u Ñ t f : s ˆ u Ñ t λf : s Ñ t u f : s Ñ t g : s Ñ u pf, gq : s Ñ t ˆ u id f f f id f f pg hq pf gq h pq f pq π 1 pf, gq f π 2 pf, gq g pπ 1 f, π 2 fq f eval pλf π 1, π 2 q f λpeval pf π 1, π 2 qq f 30 / 35

31 Programming AQL in CAML For every schema S, the category S-inst is cartesian closed. Given a type t, you get an S-instance rts. Given a term f : t Ñ t 1, you get a data mapping rfs : rts Ñ rt 1 s. All equations obeyed. S-inst is further a topos (model of higher-order logic / set theory). We consider the following schema in the examples that follow: a f b 31 / 35

32 Programming AQL in CAML: Unit The unit instance 1 has one row per table: a x f x b x The data mapping pq t : t Ñ 1 sends every row in t to the only row in 1. For example, t a p r f q t b q t pq t ÝÝÑ a x f x b x 1 p, q, r, t pq t ÝÝÑ x 32 / 35

33 Programming AQL in CAML: Products Products s ˆ t are computed row-by-row, with evident projections π 1 : s ˆ t Ñ s and π 2 : s ˆ t Ñ t. For example: a f b 3 4 ˆ a a b f c c b c d a (1,a) (1,b) (2,a) (2,b ) f (3,c) (3,c) (3,c) (3,c) b (3,c) (3,d) (4,c) (4,d) Given data mappings f : s Ñ t and g : s Ñ u, how to define pf, gq : s Ñ t ˆ u is left to the reader. hint: try it on π 1 and π 2 and verify that pπ 1, π 2 q id. 33 / 35

34 Programming AQL in CAML: Exponentials Exponentials t s are given by finding all data mappings s Ñ t: a f b 3 4 Ñ a a b f c c b c d a f 1 ÞÑ a, 2 ÞÑ b, 3 ÞÑ c, 4 ÞÑ d 3 ÞÑ c, 4 ÞÑ d 1 ÞÑ b, 2 ÞÑ a, 3 ÞÑ c, 4 ÞÑ d 3 ÞÑ c, 4 ÞÑ d 1 ÞÑ a, 2 ÞÑ a, 3 ÞÑ c, 4 ÞÑ d 3 ÞÑ c, 4 ÞÑ d 1 ÞÑ b, 2 ÞÑ b, 3 ÞÑ c, 4 ÞÑ d 3 ÞÑ c, 4 ÞÑ d 1 ÞÑ a, 2 ÞÑ b, 3 ÞÑ d, 4 ÞÑ c 3 ÞÑ d, 4 ÞÑ c 1 ÞÑ b, 2 ÞÑ a, 3 ÞÑ d, 4 ÞÑ c 3 ÞÑ d, 4 ÞÑ c 1 ÞÑ a, 2 ÞÑ a, 3 ÞÑ d, 4 ÞÑ c 3 ÞÑ d, 4 ÞÑ c 1 ÞÑ b, 2 ÞÑ b, 3 ÞÑ d, 4 ÞÑ c 3 ÞÑ d, 4 ÞÑ c b 3 ÞÑ c, 4 ÞÑ c 3 ÞÑ c, 4 ÞÑ d 3 ÞÑ d, 4 ÞÑ c 3 ÞÑ d, 4 ÞÑ d Defining eval and λ are left to the reader. 34 / 35

35 Concusion We described a new algebraic approach to databases based on category theory. Schemas are categories, instances are set-valued functors. Three adjoint data migration functors, Σ,, Π manipulate data. Instances on a schema model the simply-typed λ-calculus. Our approach is implemented in AQL, an open-source project, available at catinf.com. Collaborators welcome! We are looking for a real-world pilot project. 35 / 35