Introduction ------------ SyncEvolution synchronizes Evolution's contact, calender and task items via SyncML. The items are exchanged in the vCard 2.1 or 3.0 format and iCalender 2.0 format via the Funambol C++ client API library, which should make SyncEvolution compatible with the majority of SyncML servers. SyncEvolution does not synchronize with another SyncML capable device or another computer directly. A SyncML server that that device and SyncEvolution can talk to is needed. There are several options for that: - using a web service like ScheduleWorld (sync.scheduleworld.com) which stores the data to be synchronized on a ScheduleWorld server and provides access to it via SyncML - installing a SyncML server like the free one from Funambol (www.funambol.com) on one's own server - installing a SyncML server on the desktop The recommended solution is ScheduleWorld because it is easier than setting up a server and provides better support for vCard and iCalendar data than the stock Funambol server installation. Setting up a server on the desktop has the additional problem that not all mobile devices can communicate with the desktop via HTTP. All SyncML synchronization modes are supported by SyncEvolution: - exchanging just the changes between client and server ("two-way") - sending just the changes in one direction ("one-way-from-client/server") - replacing all items with the ones stored in the peer ("refresh-from-client/server") - a full synchronization where all items are sent to the server and the server then decides which items need to be deleted, added or updated on the client ("slow") The remainder of this document assumes that either Funambol's server bundle was installed using the default configuration or ScheduleWorld is used: because ScheduleWorld is based on the Funambol server, configuration and usage are often similar. With a server that fully supports SyncML and vCard/iCalender the following works: - copy a complete database to the server and restore it from the server later - delete or modify an item locally, then make the same change on the server - delete, modify or add items on the server (by synchronizing with another client or using a web interface), then apply the same change locally - conflict resolution (where two clients modify the same item, then sync with the server) is handled by the server, but SyncEvolution has support which ensures that no data is lost by creating duplicates (see "Conflict Resolution" below) For conflict resolution and synchronization between clients which support different attributes of items the server needs an understanding of the format of items. The Funambol server supports that for contacts, but not yet for the calendar events and tasks that SyncEvolution sends; see "Configuration with Funambol" below for more information. ScheduleWorld also works with SyncEvolution for calendars plus tasks. Installation ------------ To install SyncEvolution, just unpack an archive with a precompiled binary for your platform in a directory of your choice. Then create a configuration in $HOME/.sync4j/evolution as described below under "Configuration". No special environment variables are needed, although one might want to add the directory with contains the "syncevolution" binary to the shell's PATH variable. When a binary packages is not available for the target system and/or is not up-to-date, compiling from source can also be used to produce a binary. See below in "Compiling from Source" for details. You also need a working SyncML server. If you do not have an account and/or do not want to get an account on an existing one like ScheduleWorld, installing the Funambol server bundle is very easy: http://www.funambol.com/opensource/downloads.html For Funambol Sync4j V2.3, an additional patch is recommended to preserve line breaks of items on the server: http://forge.objectweb.org/tracker/index.php?func=detail&aid=304718&group_id=96&atid=300096 Although all of the features are covered by unit testing and have been verified to work, it is highly recommended that you make a backup of your $HOME/.evolution/addressbook $HOME/.evolution/calender $HOME/.evolution/tasks directories before running SyncEvolution for the first time. In older Evolution versions the same data is found in $HOME/evolution. Usage ----- Currently SyncEvolution comes as a simple command line tool which is configured via files. A graphical interface via an Evolution plugin would also be possible, but is not implemented yet. As command line parameters SyncEvolution only supports one option which specifies the configuration file that drives the synchronization run and several options which choose the data sources in that configuration: syncevolution [] [ ...] The string is used to find the configuration which determines how synchronization is going to proceed. Selection of sources of Evolution data which are to be synchronized with that server is done via configuration files. Each source corresponds to one Evolution address book, calendar or task list. If no arguments are given, SyncEvolution lists all available Evolution data sources. For each source one can set a different synchronization mode. It is possible to configure sources without activating their synchronization: if the synchronization mode of a source is set to "none", it will be ignore. Explicitely listing one or more source names after the server chooses just those sources for synchronization with their configured sync mode. If such a selected source is currently set to "none", it will be synchronized in "two-way" mode once. The setting in the configuration file is not changed. Progress and error messages are written into a log file that is preserved for each synchronization run. Details about that is found in the "Automatic Backups and Logging" section below. Immediately before quitting SyncEvolution will show all errors or warnings encountered and print a summary of how the databases were modified. This is done with the "synccompare" utility script described in the "Exchanging Data" section. In case of a severe error the synchronization run is aborted prematurely and SyncEvolution will return a non-zero value. Recovery from failed synchronization is done by forcing a full synchronization during the next run, i.e. by sending all items and letting the SyncML server compare against the ones it already knows. This is avoided whenever possible because matching items during a slow synchronization can lead to duplicate entries. After a successful synchronization the server's configuration file is updated so that the next run can be done incrementally. If the configuration file has to be recreated e.g. because it was lost, the next run recovers from that by doing a full synchronization. The risk associated with this is that the server might not recognize items that it already has stored previously which then would lead to duplication of items. Getting Started with ScheduleWorld ---------------------------------- - get a ScheduleWorld account - create a configuration: mkdir -p ~/.sync4j/evolution cp -r /scheduleworld ~/.sync4j/evolution/ - edit ~/.sync4j/scheduleworld/evolution/spds/syncml/config.txt, set "username" and "password" (beware, avoid extra spaces at the line ends) and choose a unique string for "deviceId"; optional: create a directory where SyncEvolution can store backup files and a log during each synchronization, see documentation of "logdir/maxlogdirs" for details - edit ~/.sync4j/scheduleworld/evolution/spds/sources/*/config.txt, set "evolutionsource" to the name or uri of your Evolution database ("syncevolution" without parameter prints both for each available database) - synchronize with "syncevolution scheduleworld" Configuration ------------- The configuration file of a certain is stored in $HOME/.sync4j/evolution//spds/syncml/config.txt The format is a simple list of = pairs with one pair per line. Leading spaces and space around the equals character are skipped. then runs until the the end of the line. In other words, it cannot start with spaces nor contain line breaks BUT spaces at the end of the line are part of the value! Be careful not to add extraneous spaces. Do not put quotation marks around , they would be treated as part of the value itself. Lines starting with a hash (#) after optional leading spaces are treated as comments and skipped. Each data source is configured in $HOME/.sync4j/evolution//spds/sources//config.txt See "scheduleworld/spds/syncml/config.txt" for options in the server configuration and "scheduleworld/spds/sources/addressbook/config.txt" for options in the data source configuration. In packages of SyncEvolution these files might be installed in [/usr/local]/shared/doc/syncevolution. Another example configuration "funambol" is provided for use with a local Funambol installation. Normally at least the following configuration options need to be adapted: spds/syncml syncURL deviceId username password spds/sources uri evolutionsource Each data source corresponds to one database at the SyncML server. The Evolution data source is determined by the type of data given in "type" and uniquely identified with the "evolutionsource" property. To get a list of available data sources, run SyncEvolution with no arguments. "evolutionsource" can be set to either the name or URL of a data source that SyncEvolution prints then. The "uri" property is used to identify with which database on the SyncML server the Evolution data is to be synchronized. Each server usually documents what needs to be configured here. The provided configurations for ScheduleWorld and Funambol on the local host already have this set correctly. One can synchronize with multiple server databases in one run, but the same server database can only be accessed once. To synchronize the same server database with multiple local Evolution data sources, one has to setup two independent configurations with different "deviceId" settings and synchronize them separately. To create such a setup simply copy the whole configuration tree of the server, e.g.: cp -r ~/.sync4j/evolution/localhost ~/.sync4j/evolution/localhost_copy and then edit ~/.sync4j/evolution/localhost_copy/spds/syncml/config.txt to update the "deviceId" and the sources/*/config.txt files to update the "evolutionsource". If the Evolution data source requires authentication, the "evolutionuser" and "evolutionpassword" are used as credentials. In this case the directory that contains the source's config.txt should only be accessible by the user. Usually these fields can be left empty. *** *** Warning: setting evolutionuser/password in cases where it is not *** needed as with local calendars and addressbooks can cause *** the Evolution backend to hang. *** Automatic Backups and Logging ----------------------------- To support recovery from a synchronization which damaged the local data or modified it in an unexpected way, SyncEvolution can create the following files during a synchronization: - a dump of the data in a format which can be imported back into Evolution, e.g. .vcf for address books - a full log file with debug information - a dump of the data after the synchronization for automatic comparison of the before/after state with "synccompare" If the source configuration option "logdir" is set, then a new directory will be created for each synchronization in that directory, using the format SyncEvolution----
--[-] with the various fields filled in with the time when the synchronization started. The sequence suffix will only be used when necessary to make the name unique. By default, SyncEvolution will never delete any data in that log directory unless explicitly asked to keep only a limited number of previous log directories. This is done by setting the "maxlogdirs" limit to something different than the empty string and 0. If a limit is set, then SyncEvolution will only keep that many log directories and start removing the oldest ones when it reaches the limit. This cleanup is only done after a successful synchronization and is limited to directories starting with the SyncEvolution- prefix, so it is safe to put other files or directories into the configured log directory. If that option is not set (as in the example configurations), then the directory will be created as $TMPDIR/SyncEvolution-- with access allowed for the user only. Files from a previous synchronization will be overwritten. This is a lot less useful because the data will usually be lost during the next reboot and each synchronization run overwrites the data of the previous one. To avoid writing any additional log file or database dumps during a synchronization the "logdir" can be set to "none". To reduce the verbosity of the log set "logLevel". If not set or 0, then the verbosity is set to 3 = DEBUG when writing to a log file and 2 = INFO when writing to the console directly. Configuration with ScheduleWorld -------------------------------- It is recommended to sync against the new vCard 3.0 URI (card3) and iCalendar 2.0 URIs (cal2, task2), using the "text/vcard", "text/calendar" and "text/x-todo" type setting respectively. These are the native formats of Evolution and a lot of effort went into ensuring that they store as much Evolution data as possible. The "note" URI and "text/x-journal" type can be used to synchronize memos. SyncEvolution is primarily tested against ScheduleWorld. The "scheduleworld" example configuration is ready to be used with these URIs, one only has to fill in the real username and password. Configuration with Funambol --------------------------- A default Funambol installation already contains databases which SyncEvolution can synchronize with Evolution address books and calendars. They are adressed in a source config with uri = card for contacts and uri = cal for calendars. Tasks (aka todos) are not supported directly. WARNING: the current Sync4j 2.3 and Funambol 3.0 cannot parse calendar entries as sent by SyncEvolution. Until these issues are resolved in the server, one has to fall back to a setup where the server does not have to convert calendar events. This is similar to the configuration for tasks described in the next paragraph. Although Evolution tasks are sent and received as special calendar events, the server cannot parse them. Therefore it is necessary to store each task as files on the server without having the server parse them. One can use the existing uri = snote file URI, but if that URI is also used for real notes or files, then one has to setup what is called FileSyncSource. Just as the client library, the Funambol server also has "sync sources" of data. Adding a new "todo" sync source is best done via the graphical admin tool: - connect to the server - unfold the tree and select "Modules/pdi/FileSystem SyncSource" - enter the following values: Source URI: todo Name: todo Type: text/x-todo Source Directory: /db/todo Supported types: text/x-todo Supported versions: text/x-todo Encoded: deselected MultiUser: deselected if all users access the same database, selected if every user gets his own database - add the new sync source - create the /db/todo directory The drawback of the file based approach is that the server always keeps tasks exactly as the clients send them: imagine that SyncEvolution stores a task on the server and then a phone fetches that task from the server, possibly throwing away attributes that it does not support. When now the task is modified on the phone and sent back to the server, only the attributes supported by the phone remain on the server and thus after another synchronization with SyncEvolution also in Evolution. For calendar entries and contacts the Funambol server avoids this kind of data loss by updating its copy of an item instead of replacing it wholesale. Exchanging Data --------------- SyncEvolution transmits address book entries as vCard 2.1 or 3.0 depending on the type chosen in the configuration. Evolution uses 3.0 internally, so SyncEvolution converts between the two formats as needed. Calendar items and tasks have to be sent and received in iCalendar 2.0 format. Only the UTF-8 character set is supported. How the server stores the items depends on its implementation and configuration. In the default Funambol server installation, contacts and calendar items are converted into an internal format, but at least for contacts it preserves most of the properties used by Evolution whereas iCalendar 2.0 items are not parsed properly at all at the moment. ScheduleWorld uses the same format as Evolution for calendars and tasks and thus requires no conversion. Contacts are stored in an LDAP server using the same schema also supported by Evolution. To check which data is preserved, one can use this procedure (described for contacts, but works the same way for calendars and tasks): 1. synchronize the address book with the server 2. create an new address book in Evolution and view it in Evolution once (the second step is necessary in at least Evolution 2.0.4 to make the new address book usable in SyncEvolution) 3. add a configuration for that second address book and the same URI on the SyncML server 4. synchronize again, this time using the other data source The default example configurations already reference Evolution databases under the fictional names "SyncEvolution test #1". After copying them into .sync4j/evolution under two different names, changing "#1" into "#2" and the deviceId in the second copy and creating the "SyncEvolution test #1/2" databases, "syncevolution" can be invoked first for one configuration, then the other to test how data is copied via the server. Now one can either compare the address books in Evolution or do that automatically, described here for contacts: - save the complete address books: mark all entries, save as vCard - invoke synccompare with two file names as arguments and it will normalize and compare them automatically Normalizing is necessary because the order of cards and their properties as well as other minor formatting aspects may be different. The output comes from a side-by-side comparison, but is augmented by the script so that the context of each change is always the complete item that was modified. Lines or items following a ">" on the right side were added, those on the left side followed by a "<" were removed, and those with a "!" between text on the left and right side were modified. The automatic unit testing (see HACKING) contains a "testItems" test which verifies the copying of special entries using the same method. Modifying one of the address books or even both at the same time and then synchronizing back and forth can be used to verify that SyncEvolution works as expected. If you do not trust SyncEvolution or the server, then it is prudent to run these checks with a copy of the original address book. Make a backup of the .evolution/addressbook directory. Conflict Resolution ------------------- If two clients make changes to the same item, the first one to synchronize will copy its changes to the server. The second one then runs into a conflict when it tries to push its own changes into the server. The SyncML server now has to decide how to proceed. If the server decides to continue with its own copy and asks to overwrite the locally modified copy (the default with Sync4j), SyncEvolution will make a local copy first. This leads to duplicates which have to be merged manually on the client side where the conflict occurred. Currently there is no support for that inside SyncEvolution: there is only an ERROR entry in the log and the summary will show the duplicated items. Merging items on the server is difficult because the SyncML protocol does not specify which parts of a conflicting item were updated. In general a server can only make more or less educated guesses which might lead to data loss. It is better to avoid this situation in the first place by synchronizing before making changes. Tracking Changes inside Evolution --------------------------------- The SyncML protocol requires that a client knows which items have been added, modified and deleted since the last sync. This is supported by the Evolution data server, albeit in a limited way and some backends might not implement it: the same function lists changes and also moves the so called "change marker" forward. Therefore asking for changes twice in a row will only list changes the first time and not report the same changes a second time. SyncEvolution delays asking for changes as long as possible and only does it when synchronization has really started. Then if synchronization completed and items where added, modified or deleted on behalf of the server, the change marker is moved forward. If synchronization fails for some or all items, then SyncEvolution cannot mark individual items for retransmission during the next sync and forces the next sync to execute in slow mode. The change marker that SyncEvolution uses is a string which is composed as "SyncEvolution:/" where comes from the server config file and from the source config file. This implies that changes are tracked separately for each server and source configuration that Evolution might be synchronized with. Known Problems -------------- The full list and up-to-date status of known problems is maintained at: http://www.estamos.de/projects/SyncML/KnownIssues.html Support ------- The home page for SyncEvolution is located at: http://www.estamos.de/projects/SyncML/ If you would like to ask questions, please use the Funambol Sync4j users mailing list. You can subscribe at http://sourceforge.net/mail/?group_id=30236 or mail sync4j-users@lists.sourceforge.net directly. If you mail the list without subscribing, please ask to get replies to you directly, as some people might reply only to the list otherwise. However, before asking a question make sure that it has not been answered already (archives are linked to from the list page) and is not covered in this document. There is no FAQ yet. If you run into any kind of issue during synchronization with SyncEvolution, please try to determine as good as you can whether it is caused by SyncEvolution, the Funambol C++ client library or the server that you talk to. Issues in the server should be reported using the issue tracker for it. For Funambol, it is located at http://forge.objectweb.org/tracker/?group_id=96 There you can also report issues with the client library. Because binary releases of SyncEvolution will contain that library and have to be updated, please also create another issue in the SyncEvolution tracker linking to the objectweb issue. If unsure, just open a ticket in the SyncEvolution tracker and it will be forwarded appropriately after an initial analysis. That tracker is hosted at SourceForge and can also be used for feature requests: http://sourceforge.net/tracker/?group_id=146288&atid=764733 Be sure to always include the following information: - version of SyncEvolution - version of Evolution - server and its version - Linux distribution - client log if appropriate, server log if appropriate and available - a description of what you do, what you expect to see, what you get instead *Warning*: logs may expose user/password if the basic authentication mechanism is used. Before posting such logs search for syncml:auth-basic and overwrite the directly afterwards. It might look like this and occur multiple times in the log: MGKlldQ2MZnJha3RhbA== Compiling from Source --------------------- To compile the code the 3.x version of the Funambol C++ client library is needed. A compatible snapshot of it is included in SyncEvolution source packages and will be used automatically. Instructions for working with CVS sources directly are contained in the HACKING document. Also needed are the Evolution and libcurl development files. On Debian 3.1 (Sarge) you can install them with: apt-get install libcurl3-dev evolution-data-server-dev libdb3-dev and optionally for the test suite: apt-get install libcppunit-dev The build system is the normal autotools system. See INSTALL for general instructions how to use that and "./configure --help" for SyncEvolution specific options. Author ------ Patrick Ohly patrick.ohly@gmx.de http://www.estamos.de/