The query optimizer is arguably the most important module in a DBMS. The job of the optimizer is to find the best way of executing a given query. The most common type of query optimization is cost-based query optimization, in which the optimizer tries to find the cheapest execution plan based on a pre-defined cost model. The cost model defines how to estimate the cost of each query operator based on its input and output cardinalities, as well as the available system resources (CPU, memory, etc.)

In cost-based query optimization, the optimizer estimates the cost of each query plan using information and statistics about the underlying data. These statistics exist in the database catalog. Based on these estimated costs, the optimizer then chooses the plan with the least estimated cost, and returns this plan to be used by the execution engine.

Cost-based optimizers work by breaking down a query into smaller logical expressions. A logical expression of size n is an expression involving n base tables. First the optimizer starts by estimating the cost of logical expressions of size 1. The cost of these expressions is usually estimated using the statistics available about the base tables. For each expression, the optimizer enumerates all possible execution plans that can be used to evaluate such expression, estimates the cost of each of these plans, then keeps only the cheapest plan for such expression*.

Next, the optimizer starts to work on expressions of size 2. The cost of these expressions is estimated based on the cost of the expressions of size 1. Again, for each expression, the optimizer enumerates all possible plans for that expression (using different join methods and join orders, etc.), estimates the cost of each plan, and keeps the cheapest plan. This continues for logical expressions of increasing sizes until the optimizer has the cheapest plan for the whole query.

* This is actually a simplification. In reality, the optimizer maintains several plans with different properties for each expression. For example: the cheapest plan overall, the cheapest plan that gives sorted output, the cheapest non-blocking plan (the one that gives the first output tuple as fast as possible), etc.

Stay tuned, and happy reading!!

Since materialized views are physically stored, they may become inconsistent with their original data sources (when the data sources change due to updates). For this reason, materialized views have to be periodically updated to account for changes in the base tables. This is not an issue with non-materialized views, since they are re-computed from the original data every time they are referenced.

Therefore, materialized views allow much more efficient access (since they do not have to be re-computed all the time), at the cost of some data being possibly out-of-date. They are most useful in data warehousing scenarios, where the base tables are not updated very often, and frequent queries of the actual base tables can be extremely expensive.

Materialized views can be used exactly in the same way as regular views. They can be used to simplify queries, and hide their details, or to store common computations that are referenced by multiple queries.

However, they have one additional use that is the quite the opposite of regular views. Recall that during query processing, queries that reference views are usually rewritten to reference the base tables. However, the query optimizer tries to reuse materialized views to evaluate queries. This is known as materialized view matching and utilization. Since the contents of these materialized views are already precomputed, using them for query evaluation can result in a significant performance improvement.

A view can be used in exactly the same way as a base table. Views can be used for the following reasons:

• Simplify queries by creating views to represnts parts of these queries
• Hiding the complexity of the data, by creating views that reflect on the parts of the data relevant to a particular user
• Providing extra security by giving users access only to the views, not to the base tables, thus restricting the data they can access
• Since the actual contents of the views are not physically stored, they do not require much space

Views can be seen in analogy to modules (or functions) in programming. They can be used to create abstraction. They can be nested (a query can reference a view, and a view can reference other views, etc.)

For example, consider the following two queries:

Both these queries contain the same sub-query. To simplify these queries, it is possible to define a view (v_cust_account) as follows:

CREATE VIEW v_cust_account AS
SELECT name, money_received, money_sent,
(money_received - money_sent) AS balance, address
FROM table_customers c, accounts_table a
WHERE a.customerid = c.customer_id


Now this view can be treated as a table containing the results of the subquery. Now the 2 initial queries can be simplified as follows:

Views are usually read-only, i.e. one cannot directly update the data in a view, but can only update the underlying tables. However, in some cases, a view can be updatable, but only if the database system is able to determine the reverse mapping from the view schema to the schema of the underlying base tables. In this case, INSERT, UPDATE, and DELETE operations can be performed. Read-only views do not support such operations because the DBMS is not able to map the changes to the underlying base tables.

During query processing, queries that reference views are usually rewritten to reference the base tables. For example, if a user issues Query 1a to the DBMS, the query processor will determine that the query references the view (v_cust_account). It will then proceed to rewrite this query to reference the base tables (thus it will look like the original query (Query 1).

Query processing usually consists of the following phases:

1. Parsing
   In this phase, the engine analyzes the text of the SQL query, to make sure that it is written correctly, and reports any syntax errors (misspellings, unidentified operators, etc.). If the query has no errors, it is translated into an internal representation that is to be used for later processing.
2. Semantic Checking
   Now that the text of the query has no errors, the semantics are checked. The engine makes sure that any referenced tables, columns, or views are valid. This phase is sometimes considered part of the parsing phase.
3. Query Rewrite
   First of all, note that usually the same query may have multiple representations in SQL. Complex queries often result in redundancy, especially with views. The Query rewriting phase rewrites a given SQL query into another query that is equivalent (produces the same result), but may be processed more efficiently.
4. Query Optimization
   This is probably the most important phase in query processing, in which the engine tries to figure out the best way to evaluate the query (the “how” part). The same query can usually be evaluated in multiple ways, depending on which tables are joined first, what join methods are used, whether the predicates are tested at the beginning or at the end, etc. Each such way is called a query execution plan (or just a query plan). For the same query, some plans are more efficient than others. The job of the query optimizer is:
   1. Determine all possible query plans for the query in hand
   2. Estimate the execution cost for each of these plans. This is achieved using the statistics available in the database catalog
   3. Pick the plan with the least estimated cost
5. Code Generation
   Once a query plan is chosen, the system translates this plan into executable machine code.
6. Plan Execution
   Finally the query is executed, and the results are returned.

The real advantage of a database system is not its ability to store the data in an organized format. If we have huge amounts of data that we only need to store, then we can just dump it into huge text files. Actually, we might as well just leave all the data in huge paper files tucked into an old dusty archive room. After all, the data will just be sitting there. But let’s say, for some reason, I want to check how many projects have been completed by the XYZ department in the year 2005, with only 5 employees or less working on each project. I think it will be hard to go through all those files to figure out the answer to that.

That’s where databases make our life easier. The real advantage of using a database is not just storing the data, but also being able to query the data…efficiently. Think of a database query as a question. The user asks the database a question, and the database returns the result. Database queries can be as simple or as complex as required.

In order to standardize the way databases are queries, SQL (Structured Query Language) was invented. SQL is a standard way of formulating a query that can be understood be any database system (except from some system-specific features that are not part of the standard SQL). SQL is considered a declarative language, not a procedural language. This means that the user only specifies WHAT they want, not HOW to get it. For example, the following simple SQL query asks the database to retrieve the names of all employees who work in the Sales department since before 2001:

SELECT name
FROM employee
WHERE dept_name = ‘Sales’
AND hire_year < 2001

SQL can also be used to insert or update the data in the database. We are not going to get into the details of SQL here, but a good introduction to SQL can be found here.

For example, the information about a database table would include the definition of the table (i.e. the names of the columns, the data types, the primary key, and any foreign keys in that table), as well as statistics on the data stored in that table. These statistics include the number of tuples in that table, the number of physical data pages that the table occupies on disk, the user who owns the table. It also includes statistics about each column in that table, for example: the highest and lowest values in the column, the number of distinct values in the column, and possibly the distribution of data values in the column (usually in the form of a histogram). In addition to statistics about single columns, sometimes statistics are also maintained about groups of columns. For example, similar to keeping the number of distinct values, the catalog might contain the number of distinct value combinations in a group of columns. This can be useful in query processing.

Note that the database catalog is itself a set of tables. The information in the catalog can usually be accessed in the same way as the information in any other table, but usually only for viewing, not for update. Updating the catalog manually can cause serious inconsistencies in the database which can cause the database system to be unusable. For example, imagine if a user can manually update the catalog and change the definition of a table. In this case, the table definition would not match the data already stored in it, thus the data will be lost. For this reason, manual updates of the catalog are not permitted, and must only be done through specific commands.

In databases, this “link” is represented by what we call a foreign key relationship. For example, consider the two tables shown. We have a customers table and an Orders table. Each table has its own primary key (which is called ID in both tables). So what is the link between those tables? Well, each order is made by one customer. A customer can make any number of orders. Notice the attribute Cust_ID in the Orders table. This attribute indicates the customer that made each particular order. So for example, order no. 2 is made by customer whose ID is “024″ (i.e., Bauer). The values in the Cust_ID column refer to the values in the ID column of the Customer table. The Cust_ID column is called a Foreign Key, since it refers to the key (primary key) of another (foreign) table.

So why do we need to do that? Why not just put the customer names in the Order table? The actual reason is called Normalization, which as a whole different topic that we might discuss later on. But for now, think of it as follows: Let’s say the company found a spelling mistake in Mr. Jack Bauer’s name (customer no. 024). If the customer name was placed with each single order, then the company would have to go and correct that spelling mistake in every order made my that customer. However, using foreign keys, the company will have to correct the mistake only in the Customers table, and the Orders will still be referring to the correct person.

This leads us to a closely-related concept, which is Referential Integrity. As we mentioned, the Cust_ID column (foreign key) references the ID column (primary key) of the Customer table. The referential integrity rule states that each value in the referring column (Cust_ID) must exist in the referenced column (ID). In other words, we cannot place an order with Cust_ID = 034 without having a customer in the Customers tables with ID = 034.

However, referential integrity might be violated for example, if we try to delete a row from the customer table (when there are orders made my that customer). Databases have multiple ways of ensuring that referential integrity still holds. One common way is to prevent (restrict) deletion in this case. For example, if a user tries to delete customer no. 024 from the Customer table, the database will not perform that action, but will give an error message instead. Another common way is to cascade the deletion process. So, in the above example, when the user tries to delete customer no. 024, the database will perform that action, but it will also delete all orders placed by that customer. So in the end, referential integrity still holds. The choice of which strategy to use can be defined by the user, and depends on the context of the data.

Columns (attributes) can be of different types. Possible types include text, numeric, date & time, binary objects, etc. For example, in Figure 1, the “Last name” column usually contains textual data while the “salary” column contains numeric data.

Figure 1 – Employee Table

Each table usually has one of its columns defined as the primary key (or simply the key) of the table. The primary key can be defined as follows: Given a particular value of that column, one can always identify a single row in that table. For example, assume that the table in figure 1 contains an additional column “ID”, and assume that each employee has a unique ID. Now if the user looks for any ID value, the system can find the single row corresponding to that ID (since no 2 employees can have the same ID). Any column that satisfies this condition is called a candidate key. One of these candidate keys is selected to be the primary key of the table. On he other hand, the column “Salary” is not a candidate key; given a salary value, one cannot identify a single employee with that salary. As a result, the primary key column cannot contain duplicate values. In some cases, however, the primary key is set as a group of columns, instead of a single column. Each of these columns might contain duplicates. However, each combination of values across the whole group of columns is unique. In this case, this primary key is called a compound key or a composite key.

So what is a database? It is a collection of related data. Databases have been around for so long, and in different forms. A paper record of information is a database. A text file with information in it can be considered as a database. However, the word Database has become more likely to refer to Relational Databases; a form of databases where data is arranged into tables, each of which consists of rows and columns. From now on, the term database will be used to mean a relational database, unless otherwise specified.

In a relational database, data is stored in tables (also known as relations). Each table usually represents an entity (an item, component, person, project, etc.). Each table has multiples columns (also called fields or attributes). These columns represent different properties of the entity represented by the table. For example, an employee table might have columns like name, hire date, salary, position, department, etc. Each row in that table represents one employee. Rows are also known as records or tuples (see Figure 1)

Figure 1 – Employee Table

Now, one cannot put all data in the same table. Why? Well, because things are different. They have different entity types. A student is not the same as an employee. It is true that they are both persons, but each of them has different properties that are of interest. An employee has a salary and a position, while a student has a major and a GPA score. So bottom line: different entities are stores in different tables, with each table containing the relevant columns (attributes) for that particular entity.

So why use databases? Why not just store everything in files? Actually, even databases store data in files on disk. However, to deal with that data, the user does not have to know the physical structure of these files (e.g. how the records are ordered in the file, the size of each record, etc.) All the user has to know is the logical representation of the data (i.e. the table names, what columns they have, etc.) This makes it possible to change the underlying file structure completely without having to get familiar with the new structure.

In order to hide the underlying details, the tables and their structure are encapsulated within a Database Management System (DBMS). A DBMS is the external shell that the user deals with when creating new tables, managing tables, dealing with the data in the tables, etc. There are many popular commercial DBMSs out there, such as Microsoft Access, SQL Server, DB2, Oracle, PostgreSQL, mySQL, etc.

DBMSs provide more flexibility when developing applications that deal with the data, since the applications can be made to interact only with the DBMS, without worrying about the underlying physical details of the file system on which the data is actually stored.

Since databases are my primary research area, I thought I might as well start blogging about them. I am not quite sure yet what I will be blogging about exactly, but it will be related to databases in one way or the other, so stay tuned.

I would like to thank my friend Brad Moyle, since he inspired me with the name of this blog. It was a casual conversation, when he asked “..and how are things going in Database land?”, so Brad, this is dedicated to you!