Introduction ------------ SyncEvolution synchronizes personal information management (PIM) data like contacts, calendars, tasks and memos via the SyncML information synchronization standard. It supports all of these for GNOME's Evolution and contacts for the system address book of the Nokia Internet Tablets, Mac OS X and (at one point, but not anymore) the iPhone. The command-line tool 'syncevolution' (compiled separately for each of these platforms) executes the synchronization. On platforms with GTK, the 'sync-ui' provides a graphical user interface. The project 'Genesis' (available separately) implements a graphical frontend that sits in the system tray. The items are exchanged in the vCard 2.1/3.0, iCalender 2.0/vCalendar 1.0 and textual format via the open source Synthesis SyncML engine, which makes SyncEvolution compatible with the majority of SyncML servers. Full, one-way and incremental synchronization of items are supported. SyncEvolution does not synchronize with another SyncML capable device or another computer directly. A SyncML server that that device and SyncEvolution can talk to is needed. There are several options for that: - using a web service like ScheduleWorld or myFUNAMBOL which store the data to be synchronized on a server and provide access to it via SyncML - installing a SyncML server like the free one from Funambol on one's own server - installing a SyncML server on the desktop The recommended solution is ScheduleWorld because it is easier than setting up a server and provides better support for vCard and iCalendar data than the stock Funambol server installation. Setting up a server on the desktop has the additional problem that not all mobile devices can communicate with the desktop via HTTP. All SyncML synchronization modes are supported by SyncEvolution: - exchanging just the changes between client and server ("two-way") - sending just the changes in one direction ("one-way-from-client/server") - replacing all items with the ones stored in the peer ("refresh-from-client/server") - a full synchronization where all items are sent to the server and the server then decides which items need to be deleted, added or updated on the client ("slow") The remainder of this document assumes that either Funambol's myFUNAMBOL service or ScheduleWorld are used: because ScheduleWorld is based on the Funambol server, configuration and usage are often similar. With a server that fully supports SyncML and vCard/iCalender the following works: - copy a complete database to the server and restore it from the server later - delete or modify an item locally, then make the same change on the server - delete, modify or add items on the server (by synchronizing with another client or using a web interface), then apply the same change locally - conflict resolution (where two clients modify the same item, then sync with the server) is handled by the server, but SyncEvolution has support which ensures that no data is lost by creating duplicates (see "Conflict Resolution" below) For conflict resolution and synchronization between clients which support different attributes of items the server needs an understanding of the format of items. The Funambol server supports that for contacts, but not yet for the calendar events and tasks that SyncEvolution sends; see "Configuration with Funambol" below for more information. ScheduleWorld also works with SyncEvolution for calendars plus tasks. Installation ------------ To install SyncEvolution, just unpack an archive with a precompiled binary for your platform in a directory of your choice or install one of the packages. Then create a configuration as described below under "Configuration". No special environment variables are needed, although one might want to add the directory which contains the "syncevolution" binary to the shell's PATH variable. When a binary packages is not available for the target system and/or is not up-to-date, compiling from source can also be used to produce a binary. See below in "Compiling from Source" for details. Although all of the features are covered by unit testing and have been verified to work, it is highly recommended that you make a backup of your $HOME/.evolution/addressbook $HOME/.evolution/calendar $HOME/.evolution/tasks $HOME/.evolution/memos directories before running SyncEvolution for the first time with Evolution. In older Evolution versions the same data is found in $HOME/evolution. Configuration templates for SyncML servers are searched in the following order: - "default/syncevolution" inside --sysconfdir DIR, usually /etc/default/syncevolution (when packaged for a distribution) or /usr/local/etc/default/syncevolution (when compiling from source) - built-in templates The properties defined in a template override the default properties, so usually only those properties which specifically need to be set for a certain server should be listed in its template config. Usage ----- Currently SyncEvolution comes as a simple command line tool which is configured via files. The general synopsis of the command line parameters is: syncevolution [] [] [ ...] The and the strings are used to find the configuration files which determine how synchronization is going to proceed. Each source corresponds to one local address book, calendar, task list or set of memos and the corresponding database on the server. Depending on which parameters are given, different operations are executed. syncevolution If no arguments are given, then SyncEvolution will list all available data sources regardless whether there is a configuration file for them or not. The output includes the identifiers which can then be used to select those sources in a configuration file. For each source one can set a different synchronization mode in its configuration file. syncevolution Without the optional list of sources all sources which are enabled in their configuration file are synchronized. syncevolution ... Otherwise only the ones mentioned on the command line are active. It is possible to configure sources without activating their synchronization: if the synchronization mode of a source is set to "none", the source will be ignored. Explicitly listing such a source will synchronize it in "two-way" mode once. Progress and error messages are written into a log file that is preserved for each synchronization run. Details about that is found in the "Automatic Backups and Logging" section below. All errors and warnings are printed directly to the console in addition to writing them into the log file. Before quitting SyncEvolution will print a summary of how the local data was modified. This is done with the "synccompare" utility script described in the "Exchanging Data" section. When the "logdir" option is enabled (since v0.9 done by default for new configurations), then the same comparison is also done before the synchronization starts. In case of a severe error the synchronization run is aborted prematurely and SyncEvolution will return a non-zero value. Recovery from failed synchronization is done by forcing a full synchronization during the next run, i.e. by sending all items and letting the SyncML server compare against the ones it already knows. This is avoided whenever possible because matching items during a slow synchronization can lead to duplicate entries. After a successful synchronization the server's configuration file is updated so that the next run can be done incrementally. If the configuration file has to be recreated e.g. because it was lost, the next run recovers from that by doing a full synchronization. The risk associated with this is that the server might not recognize items that it already has stored previously which then would lead to duplication of items. syncevolution --configure [ ...] Options in the configuration can be modified via the command line. Source properties are changed for all sources unless sources are listed explicitly. Some source properties have to be different for each source, in which case syncevolution must be called multiple times with one source listed in each invocation. syncevolution --remove Deletes the configuration of the selected server. Note that there is no confirmation question. Neither local data referenced by the configuration nor the content of log dirs are deleted. syncevolution --run [ ...] Options can also be overridden for just the current run, without changing the configuration. In order to prevent accidentally running a sync session when a configuration change was intended, either --configure or --run must be given explicitly if options are specified on the command line. syncevolution --status [ ...] Prints what changes were made locally since the last synchronization. Depends on access to database dumps from the last run, so using the "logdir" option is recommended. syncevolution --print-servers syncevolution --print-config [--quiet] [sync| ...] syncevolution --print-sessions [--quiet] These commands print information about existing configurations. When printing a configuration a short version without comments can be selected with --quiet. When sources are listed, only their configuration is shown. "Sync" instead or in combination with sources lists only the main server configuration. With --print-session information about previous synchronization sessions for the selected server are printed. This depends on the "logdir" option. The information includes the log directory name (useful for --restore) and the synchronization report. In combination with --quiet, only the paths are listed. syncevolution --restore --before|--after [--dry-run] ... This restores local data from the backups made before or after a synchronization session. The --print-sessions command can be used to find these backups. The source(s) have to be listed explicitly. There is intentionally no default, because as with --remove there is no confirmation question. With --dry-run, the restore is only simulated. The session directory has to be specified explicitly with its path name (absolute or relative to current directory). It does not have to be one of the currently active log directories, as long as it contains the right database dumps for the selected sources. A restore tries to minimize the number of item changes (see section "Item Changes and Data Changes"). This means that items that are identical before and after the change will not be transmitted anew to the server during the next synchronization. If the server somehow needs to get a clean copy of all items on the client then, use "--sync refresh-from-client" in the next run. Here is a full description of all that can be put in front of the server name. Whenever an option accepts multiple values, a question mark can be used to get the corresponding help text and/or a list of valid values. --sync|-s --sync|-s ? Temporarily synchronize the active sources in that mode. Useful for a "refresh-from-server" or "refresh-from-client" sync which clears all data at one end and copies all items from the other. --print-servers Prints the names of all configured servers to stdout. --print-config|-p Prints the complete configuration for the selected server to stdout, including up-to-date comments for all properties. The format is the normal .ini format with source configurations in different sections introduced with [] lines. Can be combined with --sync-property and --source-property to modify the configuration on-the-fly. When one or more sources are listed after the name on the command line, then only the configs of those sources are printed. "Sync" selects the main configuration instead of source configurations. Using --quiet suppresses the comments for each property. When setting a --template, then the reference configuration for that server is printed instead of an existing configuration. --configure|-c Modify the configuration files for the selected server. If no such configuration exists, then a new one is created using one of the template configurations (see --template option). When creating a new configuration only the active sources will be set to active in the new configuration, i.e. "syncevolution -c scheduleworld addressbook" followed by "syncevolution scheduleworld" will only synchronize the address book. The other sources are created in a disabled state. When modifying an existing configuration and sources are specified, then the source properties of only those sources are modified. --run|-r To prevent accidental sync runs when a configuration change was intended, but the "--configure" option was not used, "--run" must be specified explicitly when sync or source properties are selected on the command line and they are meant to be used during a sync session triggered by the invocation. --migrate In SyncEvolution <= 0.7 a different layout of configuration files was used. Using --migrate will automatically migrate to the new layout and rename the old directory $HOME/.sync4j/evolution/ into $HOME/.sync4j/evolution/.old to prevent accidental use of the old configuration. WARNING: old SyncEvolution releases cannot use the new configuration! The switch can also be used to migrate a configuration in the current configuration directory: this preserves all property values, discards obsolete properties and sets all comments exactly as if the configuration had been created from scratch. WARNING: custom comments in the configuration are not preserved. --migrate implies --configure and can be combined with modifying properties. --sync-property|-y = --sync-property|-y ? --sync-property|-y =? Overrides a configuration property in the /config.ini file for the current synchronization run or permanently when --configure is used to update the configuration. Can be used multiple times. Specifying an unused property will trigger an error message. --source-property|-z = --source-property|-z ? --source-property|-z =? Same as --sync-property, but applies to the configuration of all active sources. "--sync " is a shortcut for "--source-property sync=". When combined with "--configure", the configuration of all sources is modified. Properties cannot be specified differently for different sources, so if you want to change a source property of just one specific sync source, then use "--configure --source-property ... ". --template|-l |default|? Can be used to select from one of the built-in default configurations for known SyncML servers. Defaults to the name, so --template only has to be specified when creating multiple different configurations for the same server. "default" is an alias for "scheduleworld" and can be used as the starting point for servers which do not have a built-in configuration. Each template contains a pseudo-random device ID. Therefore setting the "deviceId" sync property is only necessary when manually recreating a configuration or when a more descriptive name is desired. --status|-t The changes made to local data since the last synchronization are shown without starting a new one. This can be used to see in advance whether the local data needs to be synchronized with the server. --quiet|-q Suppresses most of the normal output during a synchronization. The log file still contains all the information. --keyring|-k Save or retrieve passwords from the GNOME keyring when modifying the configuration or running a synchronization. Note that using this option applies to *all* passwords in a configuration, so setting a single password as follows moves the other passwords into the keyring, if they were not stored there already: --keyring --configure --sync-property proxyPassword=foo When passwords were stored in the keyring, their value is set to "-" in the configuration. This means that when running a synchronization without the --keyring argument, the password has to be entered interactively. The --print-config output always shows "-" instead of retrieving the password from the keyring. --help|-h Prints usage information. --version Prints the SyncEvolution version. Use Cases --------- Migrate a configuration from the <= 0.7 format to the current one and/or updates the configuration so that it looks like configurations created anew with the current syncevolution: $ syncevolution --migrate scheduleworld Deactivate all sources: $ syncevolution --configure \ --source-property sync=none \ scheduleworld Activate address book synchronization again, using the --sync shortcut: $ syncevolution --configure \ --sync two-way \ scheduleworld addressbook Change the password for a configuration: $ syncevolution --configure \ --sync-property password=foo \ scheduleworld Set up another configuration for scheduleworld under a different name: $ syncevolution --configure \ --sync-property username=joe \ --sync-property password=foo \ --template scheduleworld \ scheduleworld_joe Configuration ------------- The configuration file of a certain is stored in $XDG_CONFIG_HOME/syncevolution//config.ini with " $HOME/.config" as fallback if XDG_CONFIG_HOME is not set. Each data source is configured in $XDG_CONFIG_HOME/syncevolution//sources//config.ini The configuration of older SyncEvolution releases in the following directory is also still supported: $HOME/.sync4j/evolution//spds/syncml/config.txt The format is a simple list of = pairs with one pair per line. Leading spaces and space around the equals character and at the end of the line are skipped. In other words, values can neither start or end with spaces nor contain line breaks. Do not put quotation marks around , they would be treated as part of the value itself. Lines starting with a hash (#) after optional leading spaces are treated as comments and skipped. All supported properties are listed in the template configurations. Those which have reasonable defaults do not have to be set and are commented out in the template configurations. Normally at least the following configuration options need to be adapted: config.ini syncURL username password sources/*/config.ini uri type See "syncevolution --sync-property ?" for options in the server configuration and "syncevolution --source-property ?" for options in each data source configuration. Each data source corresponds to one database at the SyncML server. The local data source is determined by the type of data given in "type" and uniquely identified with the "evolutionsource" property. To get a list of available data sources, run SyncEvolution with no arguments. "evolutionsource" can be set to either the name or URL of a data source that SyncEvolution prints then. The "uri" property is used to identify with which database on the SyncML server the local data is to be synchronized. Each server usually documents what needs to be configured here. The template configurations already have this set correctly. One can synchronize with multiple server databases in one run, but the same server database can only be accessed once. To synchronize the same server database with multiple local data sources, one has to setup two independent configurations with different "deviceId" settings and synchronize them separately. To create such a setup simply copy the whole configuration tree of the server, e.g.: cp -r ~/.config/syncevolution/localhost ~/.config/syncevolution/localhost_copy and then edit ~/.config/syncevolution/localhost_copy/config.ini to update the "deviceId" and the sources/*/config.ini files to update the "evolutionsource". If an Evolution data source requires authentication, the "evolutionuser" and "evolutionpassword" are used as credentials. In this case the directory that contains the source's config.ini should only be accessible by the user. Usually these fields can be left empty. Specifying a single hyphen ("-") as value for a password instructs SyncEvolution to ask for the password at runtime. The value "${}" takes the password from the named environment variable. Entering the password manually is the most secure method. *** *** Warning: setting evolutionuser/password in cases where it is not *** needed as with local calendars and addressbooks can cause *** the Evolution backend to hang. *** SSL encryption of the HTTP connection is supported if the underlying platform supports it. In order to enable it, use a syncURL with https instead of http prefix. Not all servers support this, though. In the default configuration, servers must present a trusted certificate where the host name of the certificate matches the host name of the URL. Configuration settings can be used to relax this checking, but this makes the connection less secure and is not recommended. If you get errors about a missing certificate file under /etc/ssl/certs, then check whether the system packages which provide that file are installed. On Debian/Ubuntu the package is called "ca-certificates". Alternatively it is possible to specify a different location of a custom certificate file in the configuration. Automatic Backups and Logging ----------------------------- To support recovery from a synchronization which damaged the local data or modified it in an unexpected way, SyncEvolution can create the following files during a synchronization: - a dump of the data in a format which can be imported back into Evolution, e.g. .vcf for address books - a full log file with debug information - a dump of the data after the synchronization for automatic comparison of the before/after state with "synccompare" If the server configuration option "logdir" is set, then a new directory will be created for each synchronization in that directory, using the format ---
--[-] with the various fields filled in with the time when the synchronization started. The sequence suffix will only be used when necessary to make the name unique. By default, SyncEvolution will never delete any data in that log directory unless explicitly asked to keep only a limited number of previous log directories. This is done by setting the "maxlogdirs" limit to something different than the empty string and 0. If a limit is set, then SyncEvolution will only keep that many log directories and start removing the oldest ones when it reaches the limit. This cleanup is only done after a successful synchronization and is limited to directories that match the pattern for SyncEvolution log directory names, so it is safe to put other files or directories into the configured log directory. To avoid writing any additional log file or database dumps during a synchronization the "logdir" can be set to "none". To reduce the verbosity of the log, set "loglevel". If not set or 0, then the verbosity is set to 3 = DEBUG when writing to a log file and 2 = INFO when writing to the console directly. Configuration with ScheduleWorld -------------------------------- It is recommended to sync against the new vCard 3.0 URI (card3) and iCalendar 2.0 URIs (cal2, task2), using the "text/vcard", "text/calendar" and "text/x-todo" type setting respectively. These are the native formats of Evolution and a lot of effort went into ensuring that they store as much Evolution data as possible. The "note" URI and "text/x-journal" type can be used to synchronize memos. SyncEvolution is primarily tested against ScheduleWorld. The "scheduleworld" configuration template is ready to be used with these URIs, one only has to fill in the real username and password. Configuration with Funambol --------------------------- A default Funambol installation already contains databases which SyncEvolution can synchronize with Evolution address books and calendars. They are adressed in a source config with uri = card for contacts and uri = cal for calendars. Tasks (aka todos) are not supported directly. At some point the Funambol server started supporting iCalendar 2.0 for calendars. This is the more capable (and recommended!) format, but the server does not suggest it as "preferred". Therefore the client's configuration must use the exclamation qualifier to pick the 2.0 format (part of the current template): type = calendar:text/calendar! Exchanging Data --------------- SyncEvolution transmits address book entries as vCard 2.1 or 3.0 depending on the type chosen in the configuration. Evolution uses 3.0 internally, so SyncEvolution converts between the two formats as needed. Calendar items and tasks can be sent and received in iCalendar 2.0 as well as vCalendar 1.0, but vCalendar 1.0 should be avoided if possible because it cannot represent all data that Evolution stores. How the server stores the items depends on its implementation and configuration. In the default Funambol server installation, contacts and calendar items are converted into an internal format, but at least for contacts it preserves most of the properties used by Evolution whereas iCalendar 2.0 items are not preserved properly in Funambol 6.0. ScheduleWorld uses the same format as Evolution for calendars and tasks and thus requires no conversion. To check which data is preserved, one can use this procedure (described for contacts, but works the same way for calendars and tasks): 1. synchronize the address book with the server 2. create a new address book in Evolution and view it in Evolution once (the second step is necessary in at least Evolution 2.0.4 to make the new address book usable in SyncEvolution) 3. add a configuration for that second address book and the same URI on the SyncML server 4. synchronize again, this time using the other data source Now one can either compare the address books in Evolution or do that automatically, described here for contacts: - save the complete address books: mark all entries, save as vCard - invoke synccompare with two file names as arguments and it will normalize and compare them automatically Normalizing is necessary because the order of cards and their properties as well as other minor formatting aspects may be different. The output comes from a side-by-side comparison, but is augmented by the script so that the context of each change is always the complete item that was modified. Lines or items following a ">" on the right side were added, those on the left side followed by a "<" were removed, and those with a "|" between text on the left and right side were modified. The automatic unit testing (see HACKING) contains a "testItems" test which verifies the copying of special entries using the same method. Modifying one of the address books or even both at the same time and then synchronizing back and forth can be used to verify that SyncEvolution works as expected. If you do not trust SyncEvolution or the server, then it is prudent to run these checks with a copy of the original address book. Make a backup of the .evolution/addressbook directory. Item Changes and Data Changes ----------------------------- SyncML clients and servers consider each entry in a database as one item. Items can be added, removed or updated. This is the item change information that client and server exchange during a normal, incremental synchronization. If an item is saved, removed locally, and reimported, then this is usually reported to a peer as "one item removed, one added" because the information available to SyncEvolution is not sufficient to determine that this is in fact the same item. One exception are iCalendar 2.0 items with their globally unique ID: the modification above will be reported to the server as "one item updated". That is better, but still not quite correct because the content of the item has not changed, only the meta information about it which is used to detect changes. This cannot be avoided without creating additional overhead for normal synchronizations. SyncEvolution reports "item changes" (the number of added, removed and updated items) as well as data changes. These data changes are calculated by comparing database dumps. Because this data comparison ignores information about which data belongs to which item, it is able to detect that re-adding an item that was removed earlier does not change the data, in contrast to the item changes. On the other hand, removing one item and adding a different one may look like updating just one item, which is not quite correct. Conflict Resolution ------------------- If two clients make changes to the same item, the first one to synchronize will copy its changes to the server. The second one then runs into a conflict when it tries to push its own changes into the server. The SyncML server now has to decide how to proceed. Some decide to preserve both conflicting items, leading to duplicates which have to be merged manually later. Other servers merge automatically and throw away conflicting data based on heuristics; this may remove valid data. The client has no control over the method chosen by the server. Merging items on the server is difficult because the SyncML protocol does not specify which parts of a conflicting item were updated. In general a server can only make more or less educated guesses which might lead to data loss. It is better to avoid this situation in the first place by synchronizing before making changes. Known Problems + Support ------------------------ Please visit http://syncevolution.org/ for up-to-date information about known problems and links to places where further help can be found. Compiling from Source --------------------- To compile the code the source or an installation of the Synthesis SyncML engine is needed. A compatible snapshot of it is included in SyncEvolution source packages and will be used automatically. Instructions for working with upstream Synthesis sources directly are contained in the HACKING document. Also needed are the Evolution and Boost (>= 1.35) development files. For HTTP, either Curl or libsoup can be used. On Debian based systems the required packages can be installed with apt-get install evolution-data-server-dev \ libecal1.2-dev libebook1.2-dev \ libsoup2.4-dev \ libboost-dev libboost-dev >= 1.34, available as libboost1.35-dev backport for Debian Etch. Necessary on some distros due to bad dependencies (not needed by SyncEvolution itself): apt-get install libdb3-dev Optional (enables reading proxy settings from GNOME preferences): apt-get install libsoup-gnome2.4-dev See also the libsynthesis README file for other required packages. The test framework also requires CPPUnit: apt-get install libcppunit-dev For the GUI and its D-Bus based service backend: apt-get install libdbus-glib-1-dev \ xsltproc \ libglib2.0-dev \ libgtk2.0-dev libglade2-dev \ libgnome-keyring-dev \ libgconf2-dev libgnomevfs2-dev Optional packages for GUI: apt-get install libunique-dev libunique = ensure that GTK GUI only runs once per user The build system is the normal autotools system. See INSTALL for general instructions how to use that and "./configure --help" for SyncEvolution specific options. Note that compiling without the Evolution development files is possible. But because this is usually not what people want, the configure script needs explicit --disable-ecal --disable-ebook parameters, otherwise it will refuse to compile without Evolution support. When compiling from a git checkout, remember to run "./autogen.sh". It depends on: apt-get install libtool intltool automake Supporting SyncEvolution ------------------------ SyncEvolution is free software: available free of charge and you have the freedom of modifying and distributing it. If you are a software developer, the best way to support SyncEvolution is to port it to other backends and systems. Get in touch if you want to hear more about this. If you are a (hopefully happy) user of SyncEvolution, then you can make your appreciation or suggestions for improvements known in several ways. Although SyncEvolution is free, this does not mean that its development did not cost much effort - quite the opposite, a lot of time went into it. - Send a postcard to the author (see main page). - Leave comments on the author's blog (http://www.estamos.de/blog). - Donations are not necessary and cannot be accepted either. Author ------ Patrick Ohly patrick.ohly@gmx.de http://www.estamos.de/