SQLite.jl Documentation

• SQLite.jl Documentation
• High-level interface
• Types/Functions
• User Defined Functions

High-level interface

source |> SQLite.load!(db::SQLite.DB, tablename::String; temp::Bool=false, ifnotexists::Bool=false)
SQLite.load!(source, db, tablename; temp=false, ifnotexists=false)

Types/Functions

• SQLite.DB(file::AbstractString)

  SQLite.DB requires the file string argument as the name of either a pre-defined SQLite database to be opened, or if the file doesn't exist, a database will be created.

  The SQLite.DB object represents a single connection to an SQLite database. All other SQLite.jl functions take an SQLite.DB as the first argument as context.

  To create an in-memory temporary database, call SQLite.DB().

  The SQLite.DB will automatically closed/shutdown when it goes out of scope (i.e. the end of the Julia session, end of a function call wherein it was created, etc.)

• SQLite.Stmt(db::SQLite.DB, sql::String)

  Constructs and prepares (compiled by the SQLite library) an SQL statement in the context of the provided db. Note the SQL statement is not actually executed, but only compiled (mainly for usage where the same statement is repeated with different parameters bound as values. See bind! below).

  The SQLite.Stmt will automatically closed/shutdown when it goes out of scope (i.e. the end of the Julia session, end of a function call wherein it was created, etc.)

• SQLite.bind!(stmt::SQLite.Stmt,index,value)

  Used to bind values to parameter placeholders in an prepared SQLite.Stmt. From the SQLite documentation:

  Usually, though, it is not useful to evaluate exactly the same SQL statement more than once. More often, one wants to evaluate similar statements. For example, you might want to evaluate an INSERT statement multiple times though with different values to insert. To accommodate this kind of flexibility, SQLite allows SQL statements to contain parameters which are "bound" to values prior to being evaluated. These values can later be changed and the same prepared statement can be evaluated a second time using the new values.

  In SQLite, wherever it is valid to include a string literal, one can use a parameter in one of the following forms:

  ?

  ?NNN

  :AAA

  AAA

  @AAA

  In the examples above, NNN is an integer value and AAA is an identifier. A parameter initially has a value of NULL. Prior to calling sqlite3step() for the first time or immediately after sqlite3reset(), the application can invoke one of the sqlite3bind() interfaces to attach values to the parameters. Each call to sqlite3bind() overrides prior bindings on the same parameter.

• SQLite.execute!(stmt::SQLite.Stmt)

  SQLite.execute!(db::SQLite.DB, sql::String)

Used to execute a prepared SQLite.Stmt. The 2nd method is a convenience method to pass in an SQL statement as a string which gets prepared and executed in one call. This method does not check for or return any results, hence it is only useful for database manipulation methods (i.e. ALTER, CREATE, UPDATE, DROP). To return results, see SQLite.query below.

• SQLite.Query(db::SQLite.DB, sql::String, values=[])

  Constructs a SQLite.Query object by executing the SQL query sql against the sqlite database db and querying

the columns names and types of the result set, if any.

Will bind values to any parameters in sql. stricttypes=false will remove strict column typing in the result set, making each column effectively Vector{Any}; in sqlite, individual column values are only loosely associated with declared column types, and instead each carry their own type information. This can lead to type errors when trying to query columns when a single type is expected. nullable controls whether NULL (missing in Julia) values are expected in a column.

An SQLite.Query object will iterate NamedTuple rows by default, and also supports the Tables.jl interface for integrating with any other Tables.jl implementation. Due note however that iterating an sqlite result set is a forward-once-only operation. If you need to iterate over an SQLite.Query multiple times, but can't store the iterated NamedTuples, call SQLite.reset!(q::SQLite.Query) to re-execute the query and position the iterator back at the begining of the result set.

• SQLite.drop!(db::SQLite.DB,table::String;ifexists::Bool=false)

  SQLite.dropindex!(db::SQLite.DB,index::String;ifexists::Bool=false)

  These are pretty self-explanatory. They're really just a convenience methods to execute DROP TABLE/DROP INDEX commands, while also calling "VACUUM" to clean out freed memory from the database.

• SQLite.createindex!(db::DB,table::AbstractString,index::AbstractString,cols::AbstractString;unique::Bool=true,ifnotexists::Bool=false)

  Create a new index named index for table with the columns in cols, which should be a comma delimited list of column names. unique indicates whether the index will have unique values or not. ifnotexists will not throw an error if the index already exists.

• SQLite.removeduplicates!(db,table::AbstractString,cols::AbstractString)

  A convenience method for the common task of removing duplicate rows in a dataset according to some subset of columns that make up a "primary key".

• SQLite.tables(db::SQLite.DB)

  List the tables in an SQLite database db

• SQLite.columns(db::SQLite.DB,table::AbstractString)

  List the columns in an SQLite table

• SQLite.indices(db::SQLite.DB)

  List the indices that have been created in db

• SQLite.register(db::SQLite.DB, func::Function; nargs::Int=-1, name::AbstractString=string(func), isdeterm::Bool=true)

• SQLite.register(db::SQLite.DB, init, step::Function, final::Function=identity; nargs::Int=-1, name::AbstractString=string(final), isdeterm::Bool=true)

  Register a scalar (first method) or aggregate (second method) function with a SQLite.DB.

• @register db function

  Automatically define then register function with a SQLite.DB.

• sr"..."

  This string literal is used to escape all special characters in the string, useful for using regex in a query.

• SQLite.sqlreturn(contex, val)

  This function should never be called explicitly. Instead it is exported so that it can be overloaded when necessary, see below.

User Defined Functions

SQLite Regular Expressions

SQLite provides syntax for calling the regexp function from inside WHERE clauses. Unfortunately, however, SQLite does not provide a default implementation of the regexp function so SQLite.jl creates one automatically when you open a database. The function can be called in the following ways (examples using the Chinook Database)

julia> using SQLite

julia> db = SQLite.DB("Chinook_Sqlite.sqlite")

julia> # using SQLite's in-built syntax

julia> SQLite.Query(db, "SELECT FirstName, LastName FROM Employee WHERE LastName REGEXP 'e(?=a)'") |> DataFrame
1x2 ResultSet
| Row | "FirstName" | "LastName" |
|-----|-------------|------------|
| 1   | "Jane"      | "Peacock"  |

julia> # explicitly calling the regexp() function

julia> SQLite.Query(db, "SELECT * FROM Genre WHERE regexp('e[trs]', Name)") |> DataFrame
6x2 ResultSet
| Row | "GenreId" | "Name"               |
|-----|-----------|----------------------|
| 1   | 3         | "Metal"              |
| 2   | 4         | "Alternative & Punk" |
| 3   | 6         | "Blues"              |
| 4   | 13        | "Heavy Metal"        |
| 5   | 23        | "Alternative"        |
| 6   | 25        | "Opera"              |

julia> # you can even do strange things like this if you really want

julia> SQLite.Query(db, "SELECT * FROM Genre ORDER BY GenreId LIMIT 2") |> DataFrame
2x2 ResultSet
| Row | "GenreId" | "Name" |
|-----|-----------|--------|
| 1   | 1         | "Rock" |
| 2   | 2         | "Jazz" |

julia> SQLite.Query(db, "INSERT INTO Genre VALUES (regexp('^word', 'this is a string'), 'My Genre')") |> DataFrame
1x1 ResultSet
| Row | "Rows Affected" |
|-----|-----------------|
| 1   | 0               |

julia> SQLite.Query(db, "SELECT * FROM Genre ORDER BY GenreId LIMIT 2") |> DataFrame
2x2 ResultSet
| Row | "GenreId" | "Name"     |
|-----|-----------|------------|
| 1   | 0         | "My Genre" |
| 2   | 1         | "Rock"     |

Due to the heavy use of escape characters you may run into problems where julia parses out some backslashes in your query, for example "\y" simply becomes "y". For example the following two queries are identical

julia> SQLite.Query(db, "SELECT * FROM MediaType WHERE Name REGEXP '-\d'") |> DataFrame
1x1 ResultSet
| Row | "Rows Affected" |
|-----|-----------------|
| 1   | 0               |

julia> SQLite.Query(db, "SELECT * FROM MediaType WHERE Name REGEXP '-d'") |> DataFrame
1x1 ResultSet
| Row | "Rows Affected" |
|-----|-----------------|
| 1   | 0               |

This can be avoided in two ways. You can either escape each backslash yourself or you can use the sr"..." string literal that SQLite.jl exports. The previous query can then successfully be run like so

julia> # manually escaping backslashes

julia> SQLite.Query(db, "SELECT * FROM MediaType WHERE Name REGEXP '-\\d'") |> DataFrame
1x2 ResultSet
| Row | "MediaTypeId" | "Name"                        |
|-----|---------------|-------------------------------|
| 1   | 3             | "Protected MPEG-4 video file" |

julia> # using sr"..."

julia> SQLite.Query(db, sr"SELECT * FROM MediaType WHERE Name REGEXP '-\d'") |> DataFrame
1x2 ResultSet
| Row | "MediaTypeId" | "Name"                        |
|-----|---------------|-------------------------------|
| 1   | 3             | "Protected MPEG-4 video file" |

The sr"..." currently escapes all special characters in a string but it may be changed in the future to escape only characters which are part of a regex.

Custom Scalar Functions

SQLite.jl also provides a way that you can implement your own Scalar Functions. This is done using the register function and macro.

@register takes a SQLite.DB and a function. The function can be in block syntax

julia> @register db function add3(x)
       x + 3
       end

inline function syntax

julia> @register db mult3(x) = 3 * x

and previously defined functions

julia> @register db sin

The SQLite.register function takes optional arguments; nargs which defaults to -1, name which defaults to the name of the function, isdeterm which defaults to true. In practice these rarely need to be used.

The SQLite.register function uses the sqlreturn function to return your function's return value to SQLite. By default, sqlreturn maps the returned value to a native SQLite type or, failing that, serializes the julia value and stores it as a BLOB. To change this behaviour simply define a new method for sqlreturn which then calls a previously defined method for sqlreturn. Methods which map to native SQLite types are

sqlreturn(context, ::NullType)
sqlreturn(context, val::Int32)
sqlreturn(context, val::Int64)
sqlreturn(context, val::Float64)
sqlreturn(context, val::UTF16String)
sqlreturn(context, val::String)
sqlreturn(context, val::Any)

As an example, say you would like BigInts to be stored as TEXT rather than a BLOB. You would simply need to define the following method

sqlreturn(context, val::BigInt) = sqlreturn(context, string(val))

Another example is the sqlreturn used by the regexp function. For regexp to work correctly it must return it must return an Int (more specifically a 0 or 1) but ismatch (used by regexp) returns a Bool. For this reason the following method was defined

sqlreturn(context, val::Bool) = sqlreturn(context, int(val))

Any new method defined for sqlreturn must take two arguments and must pass the first argument straight through as the first argument.

Custom Aggregate Functions

Using the SQLite.register function, you can also define your own aggregate functions with largely the same semantics.

The SQLite.register function for aggregates must take a SQLite.DB, an initial value, a step function and a final function. The first argument to the step function will be the return value of the previous function (or the initial value if it is the first iteration). The final function must take a single argument which will be the return value of the last step function.

julia> dsum(prev, cur) = prev + cur

julia> dsum(prev) = 2 * prev

julia> SQLite.register(db, 0, dsum, dsum)

If no name is given the name of the first (step) function is used (in this case "dsum"). You can also use lambdas, the following does the same as the previous code snippet

julia> SQLite.register(db, 0, (p,c) -> p+c, p -> 2p, name="dsum")