1242: Effective ClientDataSets and the BriefCase Model
Delphi Clinic C++Builder Gate Training & Consultancy Delphi Notes Weblog Dr.Bob's Webshop
Dr.Bob's Delphi Notes Dr.Bob's Delphi Clinics Dr.Bob's Delphi Courseware Manuals
 Dr.Bob Examines... #64
See Also: other Dr.Bob Examines columns or Delphi articles

This article is based on my ClientDataSet sessions at the Borland Conference (Sept 2004) in San Jose.

Effective ClientDataSets and the BriefCase Model
In this paper, we cover the ClientDataSet component in three situations: using the stand-alone MyBase format; using dbExpress; and finally, using a client-side "briefcase" for DataSnap multi-tier applications. I'll use Delphi 7 as well as Delphi 8 (for .NET) and Delphi 2005 to illustrate the use of ClientDataSet in VCL (for .NET) applications. However, similar techniques can be applied with Kylix (on Linux) and C++Builder.


1. Stand-alone ClientDataSet
The stand-alone ClientDataSet is the perfect solution for situations where standalone executables are required without the need to install additional database engines or drivers. Since a ClientDataSet loads all the data in memory it's very fast, but the size of the database tables is limited to the amount of available memory (and the load/save times also increase correspondingly). Furthermore, we cannot perform SQL queries, but filters and other operations are fine.

Why Local ClientDataSets?
The benefits of using ClientDataSet as local DataSet over, for example, the BDE are numerous. First of all, the ClientDataSet is contained within a single DLL called MIDAS.DLL. Just drop it inside the Windows\System directory (or leave it in the same directory as your Delphi executable) and you're in business. With Delphi 6, you can even add the MidasLib unit to your uses clause which embeds the entire MIDAS.DLL inside your executable (which will grow by about 200K), resulting in a true stand-alone zero-configuration executable! Compare this to installing the BDE on your client machine. And even it you decide to use ADO, which will probably be on your client's machine already, you need to make sure the actual database backend (Access, SQL Server) is also present and accounted for. In short, the MIDAS.DLL is probably one of the easiest to install database engines I've seen so far.
And the ClientDataSet component, implemented by this MIDAS.DLL is also one of the fastest DataSet implementation I've seen. Sorting, filtering, all with blazing speed (we'll see some examples of this next time, when we actually start using local ClientDataSet components in applications). How come it's so fast? Well, that's one of the potential downsides of the ClientDataSet. You see, everything is managed in memory. And every operation such as sorting, filtering or searching is also done in memory. Which explains the speed. But which also means that for a really large (amount of data in your) ClientDataSet, you also need a large amount of memory in your machine.

Feeding Local ClientDataSets
The easiest way to load a ClientDataSet at design-time is to right-click on the ClientDataSet component and select the "Assign Local Data" pop-up menu option. This will show a dialog that lists all available DataSets (Tables, Queries, etc) that are available on the form or data module itself. If you pick one, then all data from that DataSet will be assigned to the ClientDataSet. You can then remove the actual (source) DataSet from the form or data module, and be left with a stand-alone ClientDataSet only. Note that you still need to remove the DBTables unit if you used a BDE Table or Query (and want to make your new application BDE-less).
Apart from using the "Assign Local Data" option, a ClientDataSet can also load and store its information on disk. This is visible at design-time by the "Load From File" and "Save To File" pop-up menu options. The ClientDataSet component itself also contains these methods, as well as LoadFromStream and SaveToStream methods, which you can redirect to different kinds of streams (like a TMemoryStream, right before sending this stream over a socket connection, for example, but we'll look at that in more detail next time).
ClientDataSet can load and store two kinds of data formats. The first one is typically called "cds" format, and is the internal (and undocumented) binary format. Small, native and almost impossible to share (except with other ClientDataSet components from Delphi 5 and higher, C++Builder 5 and higher, and Kylix). The second data format that ClientDataSets support is XML. And we've all heard that XML means eXtended Markup Language, and is often used in the same sentence as "open", "cross-platform" and "portable". However, the XML format that is used by the ClientDataSet is still a propriety format defined by Borland, so it's not easy to let it be used by an ADO dataset, for example.

Using Local ClientDataSets
Once a local ClientDataSet is filled with data, we can navigate through it just like any other dataset. However, there is a difference, which can be both a benefit and a curse (if you're not aware of it, that is). The main difference between the TClientDataSet and the (BDE) TTable is the fact that the TClientDataSet doesn't automatically save its contents to disk. And even if it does, it only saves the changes, and doesn't really apply them to the dataset.
What does that mean, exactly, and how can we make best use of this functionality? First of all, let's take a look at the saving capabilities of the TClientDataSet. If the data inside the TClientDataSet is loaded by any other means than the FileName property, then obviously the TClientDataSet doesn't know where to save the data if it's modified (the original data is streamed in from the .DFM file). If the FileName property is used, then the TClientDataSet will save its contents back to the file when it's explicitly deactivated (or destroyed). However, in some situations it may be a good idea to explicitly call the SaveToFile method (just as you would then have to use LoadFromFile to load the new data back in again).
Now, if you use LoadFromFile at the start of your application, and SaveToFile at the end of it, then you may notice that the external file (with the TClientDataSet contents) grows. At a higher rate than you would expect from the few changes and possible appends that you make in the TClientDataSet. Something is going on inside this file, and if you've saved it in the XML format, then you can quickly find out what: all individual changes are saved, with both the "original" value of the record and the "changed" value of the record. This means that even for a few changes, the database quickly contains multiple versions of records with changes, which take up more room than the changes themselves (or the changes applied to the dataset). Why is this done? Basically, to enable the TClientDataSet to work in a multi-tier (and multi-user) environment that we'll examine later. In a multi-tier environment, the TClientDataSet needs to call the ApplyUpdates method, sending pending updates to the remote dataset tier. But this needs to include both the original version of the record(s) as well as the changes that are to be applied. If the original version of the record(s) doesn't match the actual version of the record(s), then the update may not be applied. This is a typical multi-user problem that had to be solved, to avoid database integrity problems. But when using TClientDataSet as a lean and mean stand-alone solution, it's all of a sudden not so lean and mean anymore.
It gets worse once you realise that the TClientDataSet contains the original value of all records as well as all changes, which means that when you load the external file, it has to re-apply all changes (which can take time as well, so it will load slower and slower over time, as more and more changes are applied).

MergeChangeLog
Fortunately, there is a special method called MergeChangeLog which does just as the name implies: it merges all changes in the TClientDataSet, resulting in a very small file again (with only the records and all changes applied to them). Obviously, this method should never be called in a multi-tier environment, since it would break all possible calls to ApplyUpdates that you want to make. But in a single-tier local environment, this method is just fine, as it shrinks down (and speeds up) the local representation of the TClientDataSet.
Before calling MergeChangeLog, it wouldn't hurt to check on the value of ChangeCount, which represents the number of changes that are currently available in the TClientDataSet. Only if ChangeCount is greater than zero, you should call MergeChangeLog (otherwise it's just a waste of time).
As last tip, you may want to consider the fact that a change log does include the ability to "undo" changes on a record by record basis using the UndoLastChange, CancelUpdates or RevertRecord methods. See the on-line help for more details. These methods can also be used in a multi-tier environment, of course (before you call ApplyUpdates).


2. ClientDataSet and dbExpress
When combined with dbExpress, the ClientDataSet component can be seen as the 'cache' (or briefcase) in which the unidirectional cursors can put their data, so users can view them in a multi-directional buffered (cached) set of records. This enables SQL capabilities, since the dbExpress layer (instead of the ClientDataSet) is now providing the data. We should now no longer rely on local storage, but call ApplyUpdates to send the changes to the actual database.

What is dbExpress?
dbExpress is a cross-platform, light-weight, fast and open database access architecture. A dbExpress driver must implement a number of interfaces to get metadata, execute SQL queries or a stored procedure, and returning a unidirectional cursor. We get back to this in a moment.
The Kylix Component Library is called CLX (Component Library for X-platform). And CLX is divided in four parts: BaseCLX, VisualCLX, NetCLX and DataCLX. A question that comes up often is where exactly dbExpress fits in. Obviously, dbExpress and DataCLX are connected. And in fact, that's exactly what's happening: dbExpress is the low-level database access driver, where DataCLX is a set of components that connect to these drivers. Not the visual data-aware components, mind you, these are part of VisualCLX, but the data access components that we can use to specifically work with the data (regardless of the differences in the underlying DBMS or flat file).

Custom dbExpress
And that's not all, because dbExpress was created as an Open Database Architecture, meaning that anyone can write a dbExpress compatible (or is that called compliant?) driver for use with Kylix and Delphi 6-7 or C++Builder 6. An article about the dbExpress internals by Ramesh Theivendran, the architect of dbExpress, was published on the Borland Community website. Although this was just a draft specification, it made it clear that anyone can write one.
As a practical example, Easysoft has developed a dbExpress Gateway for ODBC, which can be used to connect to UNIX ODBC, and - via their ODBC-ODBC Bridge - even to a remote Microsoft SQL Server, Access or other Windows ODBC driver.

Components
The dbExpress tab contains seven components: TSQLConnection, TSQLDataSet, TSQLQuery, TSQLStoredProcedure, TSQLTable, TSQLMonitor and TSQLClientDataSet.

dbExpress / DataCLX Components

TSQLConnection
The TSQLConnection component is literally the connection between the dbExpress drivers and the other DataCLX components. If you drop this component on a form or data module, you see only 12 properties. The one that's probably most used is the ConnectionName property, which can be assigned with one of the values from the drop-down combobox. On my Kylix installation, I have the choice of DB2Connection, IBLocal, MySQLConnection and OracleConnection. If you select the IBLocal value, then the DriverName property gets the value INTERBASE, the GetDriverFunc property gets the value getSQLDriverINTERBASE, the LibraryName property gets the value libsqlib.so.1 and the Vendorlib property gets the value libgds.so.0. All automatically, based on the value IBLocal for the ConnectionName.
You can open the Params string list editor to edit the values of the parameters. These are also automatically filled in, by the way, when you select a value for the ConnectionName property. If you do not want this to happen, for example when writing some non-visual code to access databases and you want to provide your own parameter values, then you can set the LoadParamsOnConnection to False.
If you right-click on the TSQLConnection component, you'll see the Connection Settings for the four different Connection Names.

Connections Editor

Once you have everything set right, you can set the Connected property to True (and either get an error message when the database cannot be found, or see the property get the value True indeed for success).

TSQLDataSet
Once you have a connected TSQLConnection component, you can use any of the other DataCLX components, such as the TSQLDataSet, which is the most "general" of these components. Always start by setting the SQLConnection property of this component to (one of) the available TSQLConnection component(s). The TSQLQuery, TSQLStoredProc and TSQLTable components can be seen as special instantions of the TSQLDataSet component. In fact, this reminds me a lot of ADOExpress, in which the TADODataSet component is the "mother" of the TADOQuery, TADODataSet and TADOTable components. And both the Delphi ADOExpress and dbGo for ADO, and the Delphi & Kylix dbExpress TxxxDataSet "core" components share the CommandType and CommandText properties, with which you can determine the sub-type of the component. If you set the value of the CommandType property to ctQuery, then the CommandText property is interpreted as SQL query. If you set the CommandType to ctStoredProc, then the CommandText specifies the name of the stored procedure. And finally, if you set CommandType to ctTable, then CommandText contains the name of the individual tables.
In our case, using the general TSQLDataSet component, we can set the CommandType to ctTable, and the CommandText to customer to select the customer table. If you set the Active property to True, you get live data at design-time, just as we've been used to with Delphi (and if you set the LoginPrompt property of the SQLConnection component to False, you don't even see the login dialog). Nothing special, nothing different. Yet.

Unidirectional!?
We can now move to the Data Controls tab of the component palette, and start using some of these to display the data we receive from the active TSQLDataSet component. Note that we cannot use all of these components right now without some special considerations. This is the place where the biggest difference between the BDE and the dbExpress architecture is present. TSQLDataSet (and the derived components TSQLQuery, TSQLStoredProc and TSQLTable) returns a unidirectional cursor. Meaning that you can move forward, but not backward. Which isn't useful when using in a TDBGrid (we can only see one record at a time!), and watch out when using a TDBNavigator as well, because clicking on the Back or First button will raise an exception!
Why a Unidirectional cursor? Well, the obvious answer is speed. The Borland Database Engine has never been our best friend (let's call it a good friend, or a friendly relative), but it has helped us with the small and simple database needs. Unfortunately, the BDE footprint and overhead hasn't been small. And BDE tables have never been known for their amazing speed. And that's an area where Borland wanted to show some real improvements. The new architecture called dbExpress is designed with this in mind. And hence unidirectional cursors as resultset, with no overhead for buffering data or managing metadata.
A unidirectional cursor is especially useful when you really only need to see the results once, or need to walk through your resultset from start to finish (again once), for example in a while not eof loop, processing the results of a query or stored procedure. Real-world situations where this is useful include reporting and web server applications that produce dynamic web pages as output.
But especially when combined with visual data-aware controls, you quickly realise that in a GUI driven environment, the user will want to go back one record. So you need to somehow cache these records in order to be able to show them in a grid and to browse back as well as forward. That's where the TClientDataSet comes in. It's entirely possible and sensible to use a TDataSetProvider (from the Data Access tab of the Component Palette) to hook up with the TSQLDataSet component, and then use a TClientDataSet to obtain its records from this TDataSetProvider. The result is a ClientDataSet that gets its records (once) from a unidirectional source: the SQLDataSet. The DataSetProvider is only use as a local transportation means.

TClientDataSet
The fact that the TClientDataSet component is available in Delphi as well as Kylix means a quick and easy way to migrate local database tables (such as, indeed, BDE tables in Paradox or dBASE format). This is the first way in which you can migrate from the BDE to dbExpress: migrating data. The second way is by migrating the application as well.
To continue now with the way to migrate data from the BDE to a native ClientDataSet format, consider the code below for a new utility called dbAlias that I've written, which will convert all tables from a given (command-line passed) alias to XML files:

  {$APPTYPE CONSOLE}
  program dbAlias;
  uses
    Classes, SysUtils, DB, DBTables, Provider, DBClient;
  var
    i: Integer;
    TableNames: TStringList;
    Table: TTable;
    DataSetProvider: TDataSetProvider;
    ClientDataSet: TClientDataSet;
  begin
    TableNames := TStringList.Create;
    with TSession.Create(nil) do
    try
      AutoSessionName := True;
      GetTableNames(ParamStr(1), '', True, False, TableNames);
    finally
      Free
    end {TSession};
    Table := TTable.Create(nil);
    DataSetProvider := TDataSetProvider.Create(nil);
    ClientDataSet := TClientDataSet.Create(nil);
    try
      Table.DatabaseName := ParamStr(1);
      for i:=0 to Pred(TableNames.Count) do
      begin
        writeln(Table.TableName);
        Table.TableName := TableNames[i];
        Table.Open;
        DataSetProvider.DataSet := Table;
        ClientDataSet.SetProvider(DataSetProvider);
        ClientDataSet.Open;
        ClientDataSet.SaveToFile(ChangeFileExt(Table.TableName,'.xml'));
        ClientDataSet.Close;
        Table.Close
      end
    finally
      Table.Free
    end
  end.
This is a quick-and-dirty way to convert your existing BDE aliases containing Paradox and dBASE files to XML, after which you can put these files on a Linux box (using FTP, a network connection or even a floppy disk) and load them in Kylix using a TClientDataSet component. Obviously, this code only compiles in Delphi, and not in Kylix.

dbExpress Updates
Delphi 6-7 and Kylix both use the new cross-platform dbExpress data access layer. But how do we ensure that data in our datasets is updated correctly? (and we don't miss any updates or other changes).
I will now show that using dbExpress we must adopt a new way of looking at data (and especially at saving data), because dbExpress provides read-only unidirectional datasets only (so we no longer have automatic posts or updates to our local tables).

Read-Only Uni-Directional
To illustrate this point (and show how we should proceed), let's use Delphi 6 or 7 to build a dbExpress application. First, do a File | New and select CLX Application (a cross-platform application, which will also compile on Linux using Kylix). Drop a TSQLConnection component on the CLX form, and set its ConnectionName to IBLocal. You may have to right-click on the SQLConnection component to start the Connections Editor in order to make sure the database is pointing to an actual physical InterBase gdb file. And of course the password for SYSDBA is masterkey (just in case there's someone in the community who doesn't know that, yet).
Next, drop a TSQLTable component and set its SQLConnection property to the SQLConnection1 component. Select one of the available tables in the TableName property. We now have a read-only unidirectional dataset (which supports moving one step ahead or all the way back to the beginning, but no other operations). This is nice when connecting to a DataSetTableProducer component (where we only need to walk through the resultset just once anyway), but not so useful in most other situations.

ClientDataSet
In order to display the information from the TSQLTable (or any dbExpress dataset), we need to cache it inside a TClientDataSet component, using a TDataSetProvider component as "connector". So, drop both a TDataSetProvider and a TClientDataSet component from the Data Access tab of the Component Palette. Assign the SQLTable component to the DataSet property of the DataSetProvider, and then assign the name of the DataSetProvider to the ProviderName property of the ClientDataSet.
As soon as you open the ClientDataSet (for example by setting the Active property to True), the content of the TSQLTable will be traversed (just once) and the records in the resultset will be provided to the ClientDataSet, which will cache them from that moment on. We can now use a DataSource and (for example) a DBGrid component to display the contents - provided by the ClientDataSet component.

Update
The update problem occurs when we run the resulting application, make some changes to some of the fields and records, and exit the application again. When we restart, we see the old values again! The ClientDataSet is great for caching, but did not update the actual database for us (automatically).
Since the TSQLTable component itself is read-only, and cannot Post (or even Insert/Edit) using the TSQLTable component, we should use the ClientDataSet (which can utilize the DataSetProvider to connect back to the original dbExpress database). The method that we should call is ApplyUpdates, so we can drop a TButton on the Form and set its Caption property to Apply Updates.
Inside the OnClick event handler for the Apply Updates button we should write one line of code:

  procedure TForm1.Button1Click(Sender: TObject);
  begin
    ClientDataSet1.ApplyUpdates(0)
  end;
This will send all pending updates (inside the ClientDataSet) back to the original dbExpress database - sending a so-called reconcile error back if an update error has occurred (such as a record which was already changed by another user), in which case we must respond to the OnReconcileError event of the ClientDataSet, see the DataSnap section for more details.

Auto Update
Although the presented solution will work, you cannot realistically expect your customers and endusers to (remember to) click on the Apply Updates button when they want to save their work. We can decide to make it an automatic (apply) update, by responding to the OnAfterPost event handler of the ClientDataSet component, and calling ApplyUpdates after each implicit or explicit Post event:

  procedure TForm1.ClientDataSet1AfterPost(DataSet: TDataSet);
  begin
    (DataSet as TClientDataSet).ApplyUpdates(0)
  end;
Although this may feel good already, there are still situations we need to consider: for example deleting records. There is no Post event after you delete a record, so the OnAfterDelete event handler should also be implemented, calling the same ApplyUpdates method.
And finally, when you close your application just after your last change but before the post (for example if you changed an editfield, but didn't actually posted the change, yet), then you may want to post this change. This means that you need to call the ClientDataSet.ApplyUpdates(0) in the OnDestroy or OnClose event hander of your data module, form or frame (or whatever container you are using for your ClientDataSet).

3. ClientDataSet and DataSnap
The third solution (where we spend most of our time) positions the ClientDataSet as client-side 'briefcase' in a DataSnap multi-tier application. This means that we can disconnect the client application, storing the data locally in a physical briefcase (in the MyBase-format). We can always reload the data from the briefcase and continue to work locally. Until the time when we're back and re-connected to the DataSnap server, in which case we can send our changes and updates back to the server (called applying updates).
Note that we do not focus on the server-side issues of DataSnap, nor on the DataSnap communication protocols, but on the ClientDataSet and its abilities to connect to local or remote Providers and apply the updates to the remote middle-ware server.
Finally, applied updates can result in reconcile errors (when one or more users have already changed fields or records that conflict with our intended update), and we see how we can detect and respond on these errors (using the standard reconcile error dialog provided by Borland as well as a manual implementation).

Multitier Database Architecture
Using a multitier database architecture, you can partition applications in a way so you can access data on a (database) server without having a full set of database tools on your local machine. It also allows you to centralize business rules and processes and distribute the processing load throughout the network. DataSnap supports a three-tier technology, which in its classic form consists of the following:

  • A database server on one (server) machine
  • An application server on a second (middle-tier) machine
  • A thin client on a third (client) machine
The server would be a tool such as InterBase, Oracle, MS SQL server, and so on. The application server and the thin client would be built in Delphi (or Kylix or C++Builder). The application server would contain the business rules and the tools for manipulating the data. The client would do nothing more than present the data to the user for viewing and editing.

What Is DataSnap?
DataSnap is based on technology that allows you to package datasets and send them across the network as parameters to remote method calls. It includes technology for converting a dataset into a Variant or XML package on the server side, and then unbundling the dataset on the client and displaying it to the user in a grid via the aid of the TClientDataSet or TInetXPageProducer components.
Seen from a slightly different angle, DataSnap is a technology for moving a dataset descendant from a TTable or TQuery object on a server to a TClientDataSet object on a client. TClientDataSet looks, acts, and feels exactly like a TTable or TQuery component, except that it doesn't have to be attached to the BDE or any other database driver for that matter - apart from the DataSnap middle-ware DLL itself (which is still called MIDAS.DLL by the way). In this particular case, the TClientDataSet gets its data from unpacking the variant that it retrieves from the server.
DataSnap allows you to use all the standard Delphi components including database tools in your client-side applications, but the client side is a true thin-client: it does not have to include or link any database drivers apart from the MIDAS.DLL itself (you can even embed the MIDAS.DLL in the client executable).

DataSnap Clients and Servers
So far the theory. The best way to understand what DataSnap is and how DataSnap works is to actually build a DataSnap application, consisting of a DataSnap Client and DataSnap Server. Usually, I start with the DataSnap Server, to encapsulate and export the datasets. Having a server, the next step is to build a DataSnap Client that connects to this server and displays the data in some way.

DataSnap Server
To build your first DataSnap Server you need to start with a new empty application with File | New | Application. The fact that the mainform of this application is shown actually ensures that the DataSnap Server will remain loaded (the message loop of the main form keeps the DataSnap Server alive). The Caption property of the main form is set to "Dr.Bob's Delphi Clinic". However, to be able to easily identify the DataSnap Server, I always drop a TLabel on the mainform, set its Font property to something that's big and readable (like Comic Sans MS size 24) and set the Caption property of the TLabel component to the name of the DataSnap Server ("My First DataSnap Server" in this case).
To turn a regular application into a middle-ware database server, you have to add a Remote Data Module to it. This special data module can be found on the Multitier tab of the Object Repository, so do File | New | Other and go to the Multitier tab, which shows several CORBA Wizards, a Remote Data Module and a Transactional Data Module. The latter can be used with MTS (Microsoft Transaction Server) before Windows 2000 or COM+ in Windows 2000 and later, and won't be covered here. It's the "normal" Remote Data Module that you need to select to create your first Simple DataSnap Server. When you select the Remote Data Module icon and click on the OK-button, the New Remote Data Module Object Wizard is started.
There are a few options you must specify. First of all, the CoClass Name, which is the name of the internal class. This must be a name that you can remember later, so I'll recommend that you use SimpleRemoteDataModule at this time. Leave Instancing set to Multiple Instance, so your DataSnap Server can contain multiple instances of the Remote Data Module. Now you can press OK to generate the Remote Data Module.

DataSnap Remote Data Module
The result is a remote data module which looks very much like a regular data module. Visually, there's no difference, and if you plan to use the BDE, then you can begin by treating it like a regular data module by dropping a TSession component and setting the AutoSessionName property to true (remember that you need to do this when using the Apartment Threading Model).
Once you have a TSession component, you can add other components. For example, you can drop a TTable component and set its name to TableCustomer. Set its DatabaseName property to DBDEMOS and open the drop-down combobox for the TableName property to select the customer.db table.
So far for the regular data module. Now it's time to look at the remote aspects of this (remote) data module. Go to the Data Access tab of the Component Palette. Here you'll find a TDataSetProvider component. This component is the key to exporting datasets from a remote data module to the outside world (more specifically: to DataSnap client applications). Drop the TDataSetProvider component on the Remote Data Module, and assign its DataSet property to TableCustomer. This means that the TDataSetProvider will "provide" or export the TableCustomer to a DataSnap client application that connects to it (one that you will build in the following section).
An important property of TDataSetProvider is the Exported property, which is set to true to indicate that the TableCustomer is exported. You can set this property to false to "hide" the fact that TableCustomer is exported from the Remote Data Module, so clients cannot connect to it. This can be useful for example in a 24x7 running server where you need to make a backup of certain tables, and must ensure that nobody is working on them during the backup. With the Exported property set to false, nobody can make a connection to them anymore (until you set it to true again, of course).

DataSnap Server Compilation
Basically, this is all that it takes to create a first Simple DataSnap Server. The only things that's left for you is to save the project. I've put the main form in file ServerMainForm.pas, the Remote Data Module will be placed in file RDataMod.pas, and I've put the project itself in SimpleDataSnapServer.dpr. After the project is saved you need to compile and run it. Running the DataSnap Server - which shows only the main form, of course - will register it (inside the Windows Registry), so any DataSnap Client can find it in order to connect to it. If you ever want to move the DataSnap Server to another directory (on the same machine), you only need to move it and immediately run it again, so it re-registers itself for that new location. This is a very convenient way of managing DataSnap Server applications.
So far, you haven't written a single line of Object Pascal code for the Simple DataSnap Server. Let's see what it takes to write a DataSnap Client to connect to it.

DataSnap Client
There are a number of different DataSnap Clients that you can develop. Regular Windows (GUI) applications, ActiveForms and even Web Server applications (using Web Broker or InternetExpress). In fact, just about everything can act as a DataSnap Client, as you'll see in a moment. For now, you'll create a simple regular Windows application that will act as the first Simple DataSnap Client to connect to the Simple DataSnap Server of the previous section. At this stage, you should not be trying to run the client and the server on separate machines. Instead, get everything up and running on one machine, and then later you can distribute the application on the network.
Do a File | New | Application to start a new Delphi Application. At this time, you may decide to add a data module to it (using File | New and selecting a Data Module from the Object Repository. In order to avoid unnecessary screenshots in this paper, I've skipped the data module, and use the main form to drop my non-visual (DataSnap) components as well as my normal visual components.
Before anything else, your DataSnap Client must make a connection with the DataSnap Server application. This connection can be made using a number of different protocols, such as (D)COM, TCP/IP (sockets) and HTTP. The components that implement these connection protocols are respectively TDCOMConnection, TSocketConnection, TWebConnection and TCorbaConnection on the DataSnap tab, and the TSoapConnection component on the WebServices tab. For the first Simple DataSnap Client, you'll use the TDCOMConnection component, so drop one from the DataSnap tab onto the main form of your DataSnap Client.
The TDCOMConnection component has a property called ServerName which holds the name of the DataSnap Server you want to connect to. In fact, if you open the drop-down combobox for the RemoteServer property in the Object Inspector, you'll see a list of all registered DataSnap servers on your local machine. In your case, this list might only include one item (namely the SimpleDataSnapServer.SimpleRemoteDataModule), but all MIDAS 3 and DataSnap Servers that are registered will end up in this list eventually. The names in this list all consist of two parts: the part before the dot denotes the application name, the part after the dot denotes the Remote Data Module name. So, in the current case, you select the SimpleRemoteDataModule of the SimpleDataSnapServer application. Once you've selected this RemoteServer, you'll notice that the ServerGUID property of the TDCOMConnection component also gets a value, as found in the registry. Developers with a real good memory are free to type in the ServerGUID property here in order to automatically get the corresponding RemoteServer name. The fun really starts when you double-click on the Connected property of the TDCOMConnection component, which will toggle this property value from false to true. To actually make the connection, the DataSnap Server must be executed, which results in the pop-up of the main form of the Simple DataSnap Server that you created in last section.

ClientDataSets
Double-click again on the Connected property of the TDCOMConnection component to close down the DataSnap Server. Now that you've seen you can connect to it, it's time to import some of the datasets that are exported by (the TDataSetProvider component on) the remote data module. Drop a TClientDataSet component from the Data Access tab on the main form, and connect its RemoteServer property to the TDCOMConnection component. The TClientDataSet component will obtain its data from the DataSnap Server. You now need to specify which provider to use - in other words, from which TDataSetProvider you want to import the dataset into the TClientDataSet component. This can be done with the ProviderName property of the TClientDataSet component. Just open the drop-down combobox and you'll see a list of all available provider names (i.e. all TDataSetProvider components on the Remote Data Module that have their Exported property set to true at this time). In this case, there is only one: the only TDataSetProvider component that you used on the Simple DataSnap Server of last section, so just select that one.
Before you picked a value for the ProviderName property, you closed down the Simple DataSnap Server. However, when you opened up the drop-down combobox to list all available TDataSetProvider components on the Remote Data Module that currently have their Exported property set to true, there is only one way (for Delphi and the Object Inspector) to know exactly which of these providers are available - by asking the Simple DataSnap Server (more specifically, by actively looking at the Remote Data Module and finding out which of the available TDataSetProvider components have their Exported property set to true. And since the Simple DataSnap Server was down, it has to be started again in order to present this list to you in the Object Inspector. As a result, the moment you drop-down the combobox of the ProviderName property, the Simple DataSnap Server will be started again.
Once you've selected the RemoteServer and ProviderName, it's time to open (or activate) the TClientDataSet. You can do this by setting the Active property of the TClientDataSet component to true. At that time, the Simple DataSnap Server is feeding data from the TableCustomer table via the TDataSetProvider component and a COM connection to the TDCOMConnection component which routes it to the TClientDataSet component on your Simple DataSnap Client. Ready to be used...
And now, you can drop a TDataSource component and move to the Data Controls tab of the Component Palette and drop one or more data-aware controls. To keep the example simple, just drop a TDBGrid component. Connect the DataSet property of the TDataSource component to the TClientDataSet, and connect the DataSource property of the TDBGrid component to the TDataSource. Since the TClientDataSet component was just activated, you should now see "live" data at design time, provided by the Simple DataSnap Server.
You are almost ready. First give the Caption property of the main form a useful name (like "Simple DataSnap Client"). Then save your work. Put the main form in file ClientMainForm.pas and call the project SimpleDataSnapClient. Then, you're ready to compile and run the Simple DataSnap Client. Again, you haven't written a single line of Object Pascal code (but rest assured, that will change soon enough in the upcoming sections).

BriefCase Model
When you run the Simple DataSnap Client, you see the entire CustomerTable data inside the grid. You can browse through it, change field values, even enter new records or delete records. However, once you close the application, all changes are gone, and you're back at the original dataset inside the Delphi IDE again. No matter how hard you try, the changes that you make to the visual data seem to affect the data inside the (local) TClientDataSet only, and not the (remote) actual TableCustomer.
What you experience here is actually a feature of the so-called briefcase model. Using this briefcase model, you can disconnect the client from the network and still access the data. It works as follows:

  • Save a remote dataset to disk, shut down your machine, and disconnect from the network. You can then boot up again and edit your (local) data without connecting to the network.
  • When you get back to the network, you can reconnect and update the database. A special mechanism is provided for being notified about database errors and resolving any conflicts that might occur. For instance, if two people edited the same record, then you will be notified of the fact and given options to resolve it.
The point is that you don't have to actually be able to reach the server at all times to be able to work with your data. This capability is ideal for laptop users or for sites where you want to keep database traffic to a minimum.
Now then, you've already experienced that (apparently) your Simple DataSnap Client works on the local data inside your TClientDataSet component only. It appears you can even save the data to a local file, and load it back in again. To save the current content of a TClientDataSet, drop a TButton on the main form, set the Name property to ButtonSave, set the Caption to Save, and write the following code for the OnClick event handler:
  procedure TClientForm.ButtonSaveClick(Sender: TObject);
  begin
    ClientDataSet1.SaveToFile('customer.cds')
  end;
This saves all records from the TClientDataSet in a file called customer.cds in the current directory. By the way, cds stands for ClientDataSet, but you can use your own file and extension names, of course. Note the dfBinary flag which is passed as second argument to the SaveToFile method of the TClientDataSet. This value indicates that I wish to save the data in binary - Borland propriety - format. Alternately, I could specify to save the data in XML format, passing the dfXML value. An XML file will be much larger (14K vs. 7K for the entire TableCustomer data), but has the advantage that it can potentially be used by other applications as well. I'll just stick to the smaller (and more efficient) binary format. Similarly, to implement the functionality that you can load the customer.cds file again into your TClientDataSet component, you need to drop another TButton component, set its Name property to ButtonLoad, set the Caption to Load, and write the following Object Pascal code for the OnClick event handler:
  procedure TClientForm.ButtonLoadClick(Sender: TObject);
  begin
    ClientDataSet1.LoadFromFile('customer.cds')
  end;
Note that the LoadFromFile method of the TClientDataSet component does not need a second argument; it's obviously smart enough to determine whether it's reading a binary or an XML file. And while the binary file can probably only be generated by another TClientDataSet component, the XML file could actually have been produced by an entirely other application.
Armed with these two buttons, you can now (locally) save the changes to your data, and even reload those changes once you stop and start the Simple DataSnap Client application again. In order to control the fact whether or not the TClientDataSet component is "live" connected to the DataSnap Server, you can drop a third TButton component on the form which toggles the Active property of the TClientDataSet component. Set the Name property of this TButton to ButtonConnect, and give the Caption property the value Connect. Now, write the following code for the OnClick event handler:
  procedure TClientForm.ButtonConnectClick(Sender: TObject);
  begin
    if ClientDataSet1.Active then // close and disconnect
    begin
      ClientDataSet1.Close;
      DCOMConnection1.Close;
    end
    else // open (will automatically connect)
    begin
  //  DCOMConnection1.Open;
      ClientDataSet1.Open;
    end
  end;
Note that in order to close the connection you actually have to Close the TClientDataSet component and Close the TDCOMConnection as well, while in order to open the connection you only need to Open the TClientDataSet component, which will implicitly Open the TDCOMConnection as well.
Finally, there's one more thing you really need to do: make sure the TDCOMConnection and TClientDataSet components are not connected to the SimpleDataSnapServer at design-time. Otherwise, whenever you re-open your SimpleDataSnapClient project in the Delphi IDE again, it will need to make a connection to the SimpleDataSnapServer - loading that DataSnap Server. And when - for one reason or another - the SimpleDataSnapServer is not found on your machine, you will have a hard time loading the SimpleDataSnapClient project. So I always make sure they are not connected at design-time. In order to do so, you have to assign false to the Connected property of the TDCOMConnection component (which will unload the main form of the SimpleDataSnapServer) and false to the Active property of the TClientDataSet component (which results in the fact that you won't see any data at design-time anymore).
Let me take a moment to discuss the whole process of clients timing out when they can't talk to their server. If you try to talk to DCOM server but can't reach it, the system will not immediately give up the search. Instead, it can keep trying for a set period of time that rarely exceeds two minutes. During those two minutes, however, the application will be busy and will appear to be locked up. If the application is loaded into the IDE, then all of Delphi will appear to be locked up. You can have this problem when you do nothing more than attempt to set the Connected property of the TDCOMConnection component to true.
Now, when you recompile and run your SimpleDataSnapClient, it will show up with no data inside the TDBGrid component. And this is the time where you can click on the Connect button in order to connect to the SimpleDataSnapServer and obtain all records (from the database server). However, there are times (for example when you are "on the road" or simply not connected to the machine that runs the SimpleDataSnapServer), when you cannot connect to the SimpleDataSnapServer. In those cases, you can click on the Load button instead, and work on the local copy of the records. Note that this local copy is the one that you last saved, and is only updated when you click on the Save button to write the entire contents of the TClientDataSet component to disk.

ApplyUpdates
It's nice to be able to Connect or Load up a local dataset and Save it to disk again. But how do you ever apply your updates to the actual (remote) database again? This can be done using the ApplyUpdates method of the TClientDataSet component.
Drop a fourth button on the SimpleDataSnapClient main form, set its Name property to ButtonApplyUpdates and the Caption property to "Apply Updates". Like the Save button, this button should only be enabled when the TClientDataSet component actually contains some data (I leave that code as an exercise for the readers - contact me if you're having problems with it).
The OnClick event handler of the Apply button should get the following simple line of code:

  procedure TClientForm.ButtonApplyClick(Sender: TObject);
  begin
    if ClientDataSet1.ChangeCount > 0 then
      ClientDataSet1.ApplyUpdates(0)
  end;
The ApplyUpdates method of the TClientDataSet component has one argument: the maximum number of errors that it will "allow" before stopping with applying (more) updates. With a single SimpleDataSnapClient connected to the SimpleDataSnapServer, you will never encounter any problems, so feel free to run your SimpleDataSnapClient now. Click on the Connect button to connect to (and load) the SimpleDataSnapServer, and use the Save and Load buttons to store and read the contents of the TClientDataSet component to and from disk. You can even remove your machine from the network and work on your local data for a significant amount of time, which is exactly the idea behind the briefcase model (your laptop being the briefcase). Any changes you will make to your local copy will remain visible, and you can apply the changes back to the remote database with a click on the Apply Updates button - once you've reconnected to the network with the SimpleDataSnapServer.

Error Handling
So what if two clients, both using the BriefCase Model, connect to the Simple DataSnap Server, obtain the entire TableCustomer and both make some changes to the first record. According to what you've build so far, both clients could then send the updated record back to the DataSnap Server using the ApplyUpdates method of their TClientDataSet component. If both pass zero as value for the "MaxErrors" argument of ApplyUpdates, then the second one to attempt the update will be stopped. The second client could pass a numerical value bigger than zero to indicate a fixed number of errors/conflicts that are allowed to occur before the update is stopped. However, even if the second client passed -1 as argument (to indicate that it should continue updating no matter how many errors occur), it will never update the records that have been changed by the previous client. In other words: you need to perform some reconcile actions to handle updates on already-updated records and fields.
Fortunately, Delphi contains a very useful dialog especially written for this purpose. And whenever you need to do some error reconciliation, you should consider adding this dialog to your DataSnap Client application (or write one yourself, but at least do something about it). To use the one available in Delphi, just do File | New | Other, go to the Dialogs tab of the Object Repository and select the Reconcile Error Dialog icon. Once you select this icon and click on OK, a new unit is added to your SimpleDataSnapClient project. This unit contains the definition and implementation of the Update Error dialog that can be used to resolve database update errors.
Once this unit is added to your SimpleDataSnapProject, there is something very important you have to check. First save your work (put the new unit in file ErrorDialog.pas). Now, unless you've already unchecked the option to "Auto create forms" inside the Designer tab of the Tools | Environment Options dialog, you need to make sure that the TReconcileErrorForm is not one of the forms that are autocreated by your application - see the Forms tab of the Project | Options dialog. In the Forms tab of the Project | Options dialog, you'll find a list of Auto-create forms and a list of Available forms. Just check to make sure the ReconcileErrorForm is not on the list of Auto-create forms. An instance of the ReconcileErrorForm will be created dynamically, on-the-fly, when it is needed. So when or how do you use this special ReconcileErrorForm? Well, it's actually very simple. For every record for which the update did not succeed (for whatever reason), the OnReconcileError event handler of the TClientDataSet component is called. This event handler of TClientDataSet is defined as follows:

  procedure TClientForm.ClientDataSet1ReconcileError(DataSet: TClientDataSet;
    E: EReconcileError; UpdateKind: TUpdateKind;
    var Action: TReconcileAction);
  begin

  end;
This is an event handler with four arguments: first of all the TClientDataSet component that raised the error, second a specific ReconcileError that contains a message about the cause of the error condition, third the UpdateKind (insert, delete or modify) that generated the error and finally as fourth argument the Action that you feel should be taken.
As Action, you can return the following possible enum values (the order is based upon their actual enum values):
  • raSkip - do not update this record, but leave the unapplied changes in the change log. Ready to try again next time.
  • raAbort - abort the entire reconcile handling; no more records will be passed to the OnReconcileError event handler.
  • raMerge - merge the updated record with the current record in the (remote) database, only changing (remote) field values if they changed on your side.
  • raCorrect - replace the updated record with a corrected value of the record that you made in the event handler (or inside the ReconcileErrorDialog. This is the option in which user intervention (i.e. typing) is required.
  • raCancel - undo all changes inside this record, turning it back into the original (local) record you had.
  • raRefresh - undo all changes inside this record, but reloading the record values from the current (remote) database (and not from the original local record you had).
The good thing about the ReconcileErrorForm is that you don't really need to concern yourself with all this. You only need to do two things. First, you need to include the ErrorDialog unit inside the SimpleDataSnapClient main form definition. Click on the ClientMainForm and do File | Use Unit to get the Use Unit dialog. With the ClientMainForm as your current unit, the Use Unit dialog will list the only other available unit, which is the ErrorDialog. Just select it and click on OK. The second thing you need to do is to write one line of code in the OnReconcileError event handler in order to call the HandleReconcileError function from the ErrorDialog unit (that you just added to your ClientMainForm import list). The HandleReconcileError function has the same four arguments as the OnReconcileError event handler (not a real coincidence, of course), so it's a matter of passing arguments from one to another, nothing more and nothing less. So, the OnReconcileError event handler of the TClientDataSet component can be coded as follows:
  procedure TClientForm.ClientDataSet1ReconcileError(DataSet: TClientDataSet;
    E: EReconcileError; UpdateKind: TUpdateKind;
    var Action: TReconcileAction);
  begin
    Action := HandleReconcileError(DataSet, UpdateKind, E)
  end;

Demonstrating Reconcile Errors
The big question now is: how does it all work in practice? In order to test it, you obviously need two (or more) SimpleDataSnapClient applications running simultaneously. For a complete test using the current SimpleDataSnapClient and SimpleDataSnapServer applications, you need to perform the following steps:

  • Start the first SimpleDataSnapClient, and click on the Connect button (the SimpleDataSnapServer will now be loaded as well).
  • Start the second SimpleDataSnapClient and click on the Connect button. Data will be obtained from the same SimpleDataSnapServer that's already running.
  • Using the first SimpleDataSnapClient, change the field "Company" for the first record.
  • Using the second SimpleDataSnapClient, also change the field "Company" for the first record (make sure you don't change it to the same value as you did in the previous step using the first SimpleDataSnapClient).
  • Click on the "Apply Updates" button of the first SimpleDataSnapClient. All updates will be applied without any problems.
  • Click on the "Apply Updates" button of the second SimpleDataSnapClient. This time, one or more errors will occur, because the first record has its "Company" field value changed (by the first SimpleDataSnapClient), and for this and possibly more conflicting records, the OnReconcileError event handler is called.
  • Inside the Update Error dialog, you can now experiment with the reconcile Actions (skip, abort, merge, correct, cancel and refresh) to get a good feeling of what they do. Pay special attention to the differences between Skip and Cancel, and the differences between Correct, Refresh and Merge.
Skip moves on to the next record, skipping the requested update (for the time being). The unapplied change will remain in the change log. Cancel also skips the requested update, but it cancels all further updates (in the same update packet). The current update request is skipped in both cases, but Skip continues with other update requests, and Cancel cancels the entire ApplyUpdate request.
Refresh just forgets all updates you made to the record and refreshes the record with the current value from the server database. Merge tries to merge the update record with the record on the server, placing your changes inside the server record. Refresh and Merge will not process the change request any further, so the records are synchronized after Refresh and Merge (while the change request can still be redone after a Skip or Cancel).
Correct, the most powerful option, actually gives you the option of customizing the update record inside the event handler. For this you need to write some code or enter the values in the dialog yourself.

Summary
In this paper, I have described you how TClientDataSet can be used as stand-alone (in-memory) database table, as well as the use incombination with dbExpress (for 2-tier) and DataSnap (for 3-tier) as database cache, providing the so-called Briefcase Model. With the frozen BDE, the importance of the ClientDataSet component continues to grow in Delphi, Kylix and C++Builder applications!

For more information, see Dr.Bob Examines #63 on C++Builder Data Access Technologies and #58 on Delphi Database Development, as well as my BorCon 2004 paper on Data Access Techniques with ClientDataSets and the BorCon 2003 paper on Introduction to ClientDataSet and dbExpress.


This webpage © 2005-2010 by Bob Swart (aka Dr.Bob - www.drbob42.com). All Rights Reserved.