Delta Synchronization

Synchronizing client and server changes.

Introduction

A mobile app created on the Appery.io platform can work even when there is no network connection. The platform provides several ways to work in an offline mode which are introduced in this document.

📘

Offline Support

An app can work in an offline mode only with the API Express generated services.

API Express offers two types of generated services:

  1. Generated services.
  2. 2-way sync generated services.

Generated Services

A generated service can be created from a relational database table or from the Appery.io database collection. The only requirement is that the collection/table has only one primary key.

When a mobile app is online, a generated service caches the response of find and read operations into the local cache.

When a mobile app goes into the offline mode, a generated service gets data from local cache.

All data modification (CREATE, UPDATE, DELETE) operations in the offline mode are stored in the Client History and are executed after switching back to the online mode.

The main disadvantage of a generated service is controlling relevant data in the local cache and network overhead.

2-way Sync Generated Service

A 2-way sync generated service can be created from a relational database table or from the Appery.io database. This service has the following requirements:

  1. Has only one primary key.
  2. There is a column with the last modification date value filled by the database.
  3. There is a column that indicates that the record was deleted from the database.
  4. Genererated services are marked in API Express as 2-way sync.

A difference between a generated service and 2-way sync generated service is the presence of a FETCH operation.

AppClient keeps the last synchronization date information and based on this information fetch operation returns only a list of changed records from the last synchronization date.

The synchronization process works equally for a generated service and for a 2-way sync generated service.

When an app goes from the offline to the online mode, the synchronization process starts.

AppClient synchronizes operations from Client History one by one, in a not transactional mode. For example, if 10 data modification operations were made in the offline mode and 5 of them were performed successfully but the 6th operation failed, then AppClient stops the synchronization process and there is no option to automatically roll back the first 5 operations.

Example

To understand this process better let's see a simple example.

There is a collection called Label in the Appery.io database with only one custom field: name (type String).

A 2-way sync generated service was imported into the app via API Express Extension with the name of label.

This is the status of the client/server as you start going through this scenario:

  • Collection on the server is empty.
  • Client Cache is empty.
  • Client History is empty.
  • Last synchronization date variable (LastSyncDate) is not initialized.
  • AppClient is in the online mode.

The following shows the current client/server status:

Client History Client (-) ONLINE Server
operation _id old new
_id name _updatedAt
_id name _updatedAt deleted

At this point, AppClient (app) creates the first label (name: item1) at 9:00. You can see the record added on the server:

Client History Client (-) ONLINE Server
operation _id old new
_id name _updatedAt
_id name _updatedAt deleted
1 item1 9:00 false

AppClient invokes the FETCH operation (LastSyncDate is undefined).

API Express returns all records. In our example, it is only one record: item1.

Client History Client (9:00) ONLINE Server
operation _id old new
_id name _updatedAt
1 item1 9:00
_id name _updatedAt deleted
1 item1 9:00 false

At this point, the data on the client and server are the same. The LastSyncDate is 9:00.

Next, item1 was updated to item1_1. The update happened not from our mobile app but from an external system such as an external API or a web application. This update happened at 9:10 as shown below:

Client History Client (9:00) ONLINE Server
operation _id old new
_id name _updatedAt
1 item1 9:00
_id name _updatedAt deleted
1 item1_1 9:10 false

AppClient invokes the FETCH operation (LastSyncDate is 9:00).

API Express returns all records where the value of _updatedAt field is later than 9:00.
In our case, API Express returns item1_1. This step is shown below:

Client History Client (9:10) ONLINE Server
operation _id old new
_id name _updatedAt
1 item1_1 9:10
_id name _updatedAt deleted
1 item1_1 9:10 false

The data on the client and the server is the same. LastSyncDate is 9:10.

item2 on the server was created again from an external application such as an API or a web application at 9:20 as shown below:

Client History Client (9:10) ONLINE Server
operation _id old new
_id name _updatedAt
1 item1_1 9:10
_id name _updatedAt deleted
1 item1_1 9:10 false
2 item2 9:20 false

At this point, AppClient invokes the FETCH operation (LastSyncTime is 9:10).

The FETCH operation returns item2 to the client.

Client History Client (9:20) ONLINE Server
operation _id old new
_id name _updatedAt
1 item1_1 9:10
2 item2 9:20
_id name _updatedAt deleted
1 item1_1 9:10 false
2 item2 9:20 false

Data on the client and server is the same. LastSyncDate is to 9:20.

AppClient (app) deletes item1_1 at 9:30. AppClient invokes a delete REST API call and passes item1_1 and value of the _updatedAt field.

At this point, API Express verifies that the value of _updatedAt field of item1_1 from the mobile app matches with the server value and marks the column as deleted (set the value of deleted field to true).

If the server returns a successful response, AppClient removes item1_1 from local cache.

Client History Client (9:20) ONLINE Server
operation _id old new
_id name _updatedAt
1 item1_1 9:10
2 item2 9:20
_id name _updatedAt deleted
1 item1_1 9:30 true
2 item2 9:20 false

Data on the client and the server side is now different.

Database on the server contains information about the deleted item when AppClient doesn't. LastSyncDate is 9:20.
AppClient invokes the FETCH operation. LastSyncDate is 9:20.

API Express returns item1_1 and flags that record as deleted.

AppClient can't find item1_1 in local cache and considers this situation as correct.

Client History Client (9:30) ONLINE Server
operation _id old new
_id name _updatedAt
2 item2 9:20
_id name _updatedAt deleted
1 item1_1 9:30 true
2 item2 9:20 false

LastSyncDate is 9:30.

AppClient now goes into the offline mode.

Client History Client (9:30) OFFLINE Server
operation _id old new
_id name _updatedAt
2 item2 9:20
_id name _updatedAt deleted
1 item1_1 9:30 true
2 item2 9:20 false

AppClient invokes the FIND operation and gets the response from local cache (in our case item2).

AppClient updates item2 to item2_1.

Client History Client (9:30) OFFLINE Server
operation _id old new
UPDATE 2 item2 item2_1
_id name _updatedAt
2 item2_1 9:20
_id name _updatedAt deleted
1 item1_1 9:30 true
2 item2 9:20 false

The UPDATE operation was recorded into the Client History with new and old values (old value is required for the REVERT operation and will be explained later).

AppClient creates item3.

Client History Client (9:30) OFFLINE Server
operation _id old new
UPDATE 2 item2 item2_1
CREATE t_3 - item3
_id name _updatedAt
2 item2_1 9:20
t_3 item3 -
_id name _updatedAt deleted
1 item1_1 9:30 true
2 item2 9:20 false

item3 was created with a temporary _id (t_3).

The CREATE operation was recorded into Client History.

AppClient deletes item2_1.

Client History Client (9:30) OFFLINE Server
operation _id old new
UPDATE 2 item2 item2_1
CREATE t_3 - item3
DELETE 2 item2_1 -
_id name _updatedAt
t_3 item3 -
_id name _updatedAt deleted
1 item1_1 9:30 true
2 item2 9:20 false

item4 on the server was created from an external source (for example, external API) at 9:40.

Client History Client (9:30) OFFLINE Server
operation _id old new
UPDATE 2 item2 item2_1
CREATE t_3 - item3
DELETE 2 item2_1 -
_id name _updatedAt
t_3 item3 -
_id name _updatedAt deleted
1 item1_1 9:30 true
2 item2 9:20 false
3 item4 9:40 false

AppClient goes into the online mode from the offline mode.

Since Client History contains operations, the synchronization process is started.

Client History Client (9:30) SYNC Server
operation _id old new
UPDATE 2 item2 item2_1
CREATE t_3 - item3
DELETE 2 item2_1 -
_id name _updatedAt
t_3 item3 -
_id name _updatedAt deleted
1 item1_1 9:30 true
2 item2 9:20 false
3 item4 9:40 false

AppClient performs all operations one by one and starts from updating item2 despite that item2 was eventually deleted.

Let's suppose that the synchronization process begins at 9:50 and every operation is performed for 1 minute.

AppClient invokes the first operation UPDATE and passes item2_1 to the server.

Client History Client (9:30) SYNC Server
operation _id old new
CREATE t_3 - item3
DELETE 2 item2_1 -
_id name _updatedAt
t_3 item3 -
_id name _updatedAt deleted
1 item1_1 9:30 true
2 item2_1 9:50 false
3 item4 9:40 false

item2 was updated on the server (field _updatedAt was also updated).

After UPDATE is successfully performed, it is deleted from Client History.

AppClient invokes the second operation, CREATE and passes item3 to the server.

Client History Client (9:30) SYNC Server
operation _id old new
DELETE 2 item2_1 -
_id name _updatedAt
4 item3 9:51
_id name _updatedAt deleted
1 item1_1 9:30 true
2 item2_1 9:50 false
3 item4 9:40 false
4 item3 9:51 false

item3 is created on the server with _id = 4 and _updatedAt = 9:51.
AppClient updates _id and _updatedAt of item3 in local cache.

AppClient invokes last operation DELETE and passes item2_1 to the server.

Client History Client (9:30) SYNC Server
operation _id old new
_id name _updatedAt
4 item3 9:51
_id name _updatedAt deleted
1 item1_1 9:30 true
2 item2_1 9:52 true
3 item4 9:40 false
4 item3 9:51 false

item2_1 is marked as deleted and _updatedAt equals 9:52.

Since all operations were performed without errors AppClient goes into the online mode.

AppClient invokes the FETCH method and passes LastSyncDate = 9:30.

The server returns records item2_1 (deleted), item4 and item3.

Client History Client (9:52) ONLINE Server
operation _id old new
_id name _updatedAt
4 item3 9:51
3 item4 9:40
_id name _updatedAt deleted
1 item1_1 9:30 true
2 item2_1 9:52 true
3 item4 9:40 false
4 item3 9:51 false

LastSyncDate is 9:52.