diff options
-rw-r--r-- | NEWS | 843 | ||||
-rw-r--r-- | configure.ac | 4 | ||||
-rw-r--r-- | src/backends/activesync/ActiveSyncSource.cpp | 11 | ||||
-rw-r--r-- | src/backends/activesync/ActiveSyncSource.h | 5 | ||||
-rw-r--r-- | src/backends/kde/KDEPlatform.cpp | 19 | ||||
-rw-r--r-- | src/dbus/server/main.cpp | 5 | ||||
-rw-r--r-- | src/gdbus/gdbus-cxx-bridge.h | 15 | ||||
-rw-r--r-- | src/gdbusxx/gdbus-cxx-bridge.cpp | 95 | ||||
-rw-r--r-- | src/gdbusxx/gdbus-cxx-bridge.h | 19 | ||||
-rw-r--r-- | src/syncevo/configs/datatypes/02vcard-types.xml | 2 | ||||
-rw-r--r-- | src/syncevo/configs/scripting/04vcard-photo-value.xml | 9 | ||||
-rw-r--r-- | test/ClientTest.cpp | 24 | ||||
-rwxr-xr-x | test/runtests.py | 37 | ||||
-rw-r--r-- | test/synccompare.pl | 9 | ||||
-rwxr-xr-x | test/syncevo-http-server.py | 7 | ||||
-rw-r--r-- | test/sys.supp | 72 | ||||
-rwxr-xr-x | test/test-dbus.py | 71 | ||||
-rw-r--r-- | test/testcases/eds_contact.vcf.exchange.tem.patch | 19 |
18 files changed, 1195 insertions, 71 deletions
@@ -1,3 +1,846 @@ +SyncEvolution 1.2.2 -> 1.3, 10.09.2012 +====================================== + +After almost three months of public beta testing the next major +version of SyncEvolution is ready for release. The pre-releases did +have the desired effect of flushing out bugs not found by nightly +testing alone. Thanks everyone for packaging, downloading and testing +them! + +New features are KDE/Akonadi and ActiveSync support, not only in the +source code but also in syncevolution.org binaries. ActiveSync is the +recommended way of synchronizing contacts with Google: +https://syncevolution.org/wiki/google-contacts-activesync + +The D-Bus server and local sync were rewritten considerably, to make +the code cleaner and more robust. The CalDAV backend now also supports +tasks and memos. CalDAV and CardDAV can be used in combination with a +SyncML peer ("bridging SyncML and WebDAV"), thus allowing a device +which only supports SyncML to talk to a WebDAV service without any +intermediate storage. + +1.3 contains bug fixes that were not backported to 1.2.x, so upgrading +is recommended. For example, SyncEvolution 1.3 is required for +Evolution 3.4, otherwise photos are not exported properly. Support for +Evolution >= 3.6 is in the source code, but not in syncevolution.org +binaries. Further workarounds for recent changes in Google CalDAV and +Funambol One Media were added. + + +Details: + +* ActiveSync: updated to work with latest activesyncd and Google, package binaries + + Syncing Google contacts was added to the nightly testing. Syncing + contacts and events with Exchange 2012 was already working. Setup + instructions and known issues are described here: + https://syncevolution.org/wiki/google-contacts-activesync + +* phone sync: delete<->delete conflict + phone calendar+todo sync (BMC #23744) + + When deleting an item on phone and locally, the next sync failed with + ERROR messages about "object not found". This has several reasons: + - libsynthesis super data store attempts to read items + which may or may not exist (triggers ERROR message) + - it checks for 404 but Evolution backends only return a generic + database error (causes sync to fail) + +* phone sync: get phone vendor and model from Device ID profile (BMC #736) + + In the past we have relied on the user-modifiable device name to be + the fingerprint for matching a phone to a template which is unreliable. + + This release changes this in the cases where the phone supports the + Device ID profile (DIP). If support for DIP is detected, then we + extract the vendor and product ids and attempt to associate them + with a product and vendor name by using a newly added lookup table. + + This lookup table has to be maintained manually and depends on + contributions by users to cover more devices. See + http://blixtra.org/blog/2011/09/22/syncevolution-needs-you-or-at-least-your-bluetooth-phones/ + +* vCalendar 1.0: fixed recurring all-day event support + + vCalendar 1.0 cannot represent all-day events. The workarounds for + mapping iCalendar 2.0 all-day events into vCalendar 1.0 was + incomplete, leading to effects like shifting EXDATEs and end + times. + +* Funambol: ignore UID + + Funambol's OneMedia sends UID, but not RECURRENCE-ID. That becomes a + problem when multiple events of the same event series are added to a + backend which follows the iCalendar 2.0 standard (CalDAV, EDS, KDE), + because these events all look like the master event, and there can be + only one of those. + + SyncEvolution now strips the UID from all events coming from any + Funambol server (regardless of the version). If a future Funambol + server release adds support for both UID and RECURRENCE-ID, then + SyncEvolution will have to be updated to take advantage of the + improved server. Because the RECURRENCE-ID is also getting + stripped (despite not being set at the moment), SyncEvolution should + continue to work as it does now even if the server changes. + + It would have been nice to limit this workaround to affected Funambol + server versions, but an inquiry on the Funambol mailing list didn't + get a reply, therefore SyncEvolution is playing it safe and assumes + that all future Funambol releases will have the same problem. + +* Funambol: work around PHOTO TYPE=image/jpeg + + A combination of Funambol Android and Funambol server recently led to + the Funambol server sending PHOTO data with TYPE=image/jpeg. This is + invalid and caused EDS to reject the photo (Vladimir Elisseev, + "[SyncEvolution] issues with syncing photos"). + + Work around the problem by only keeping the part of the type after the + last slash, if there is any. For image/jpeg and similar types that + leads to the desired value and does not affect valid values, because + those do not contain a slash + (http://www.iana.org/assignments/media-types/image/index.html). + +* Funambol: avoid slow syncs in refresh from server + + libsynthesis has traditionally implemented "refresh-from-server" as + "delete local data" plus "slow" sync. This is more compatible, because + some servers (like Google) do not support "refresh-from-server". + + But it has the downside that the server cannot know that the client + won't send any data, and Funambol's OneMedia now only allows one slow + sync before blocking the next one for a certain period of time. This is + done to prevent excessive resource usage by badly behaving clients. + + To accomodate both kinds of servers, the new "enableRefreshSync" + sync property can be set set to explicitly allow the usage of + the "refresh-from-server" sync mode. It's off by default. The Funambol + template has it turned on, existing configs must be updated manually + (see upgrading comments below). + +* Mobical (aka Everdroid): stopped testing memo syncing + + Memos used to work, but now only trigger an unspecific 400 error + on the server side. + +* GTK-UI: accept service config with a username again (BMC#23106) + + Suppressing configs with empty username had undesired side effects: + modifying configs for direct syncing with a device incorrectly + triggered the same error message, without any means of entering + a username. The faulty check was removed without replacement. + +* GTK-UI: added GTK 3 version of UI + + When GTK 3 is found during compilation, a GTK 3 version of the + UI is built. The source code of both is different to avoid + excessive use of ifdefs. At the moment, both versions offer + the same features. In the long run, the GTK 3 version will + replace the GTK 2 version. + +* command line: added refresh/one-way-from-local/remote (BMC #23537) + + The -from-client/server sync modes are confusing because the direction + of the data exchange depends on which side acts as SyncML server or + client. + + This release introduces new modes which use -from-local/remote + instead. The statistics and messages also use these variants + now. The old modes are still understood, but are declared as "not + recommended" in the documentation. + +* command line: config and source names are optional (BMC #23783) + + The need to add "foo" and "bar" pseudo config and source names to the + command line even when all parameters for the operation where + explicitly specified on the command line was confusing. + + Now it is possible to invoke item operations without the config and + source name. Names which refer to non-existent configs are still + accepted, as in previous releases. Typos are handled better by + producing a detailed error report which includes (as applicable): + - config doesn't exist + - source doesn't exist or not selected + - backend property not set + + Because luids used to be positional arguments after <config> and + <source>, a new --luids keyword is necessary to indicate that the + ensuing parameters are luids and not <config> and <source>. + +* command line: introduced --print-databases, supported for CalDAV/CardDAV + + Listing databases is now a dedicated operation, instead of being done + whenever syncevolution was invoked without parameters. + + Advantages: + - can be combined with property assignments for backends + which do not work without that additional information, for example + CalDAV/CardDAV: + syncevolution --print-databases \ + backend=[caldav|carddav] \ + syncURL=... \ + username=... \ + password=... + - can be done for configured sources + +* command line: use both stdout and stderr + + Traditionally, the "syncevolution" command line tool mixed its + INFO/ERROR/DEBUG messages into the normal stdout. This has the major + drawback that error messages get lost during operations like + syncevolution --export - @default addressbook | grep "John Doe" + + Now anything which is not the expected result of the operation is + always sent to stderr. Obviously this includes ERROR messages. INFO + and DEBUG are harder to decide. Because they usually convey meta + information about the running operation, they are also sent to + stderr. The output of running a sync goes to both stdout (summary) + and stderr (progress). + +* command line: allow setting empty properties + + Due to the way how properties were handled internally, it wasn't + possible to explicitly set a property to its default value. Instead + the property was unset. For example, explicitly setting database= was + not possible. + + This is necessary for client-test and ActiveSync, because client-test + needs to know that the testing is expected to run with the default + databases (something which normally is avoided by overwriting empty + database properties). + + Now the "is set" state is tracked explicitly in the config storage and + command line property APIs. Unsetting a property via the command line + could be implemented with an explicit command line option, but is not + supported at the moment. + +* command line: fixed --export <file name> + + When exporting items into a file, the delimiter between items + was missing. + +* command line + local sync: fixed erroneous "Comparison impossible" output. + + "Comparison impossible" was incorrectly printed after a successful + comparison on the target side of local sync. + +* local sync: fix timeout with local sync with libdbus + + When using libdbus instead of GIO D-Bus (as done by syncevolution.org + binaries and SyncEvolution on Maemo), local sync may have aborted + after 25 seconds when syncing many items with a D-Bus timeout error: + + [ERROR] sending message to child failed: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible ca + + Reported by Toke Høiland-Jørgensen for Harmattan. Somehow not encountered + elsewhere. + +* synccompare: shorter data dump of PHOTO + + A full comparison of the base64 PHOTO data can be very long. + Now some key characteristics of the PHOTO data (number of + characters in base64 encoding, number of bytes in decoded + data, md5sum of decoded data) are printed instead. + + That way, unintended changes of the data (different encoding, + different content) should still be found while testing and + added/removed photos are nicely visible in synccompare diffs. + +* synccompare: fixed output for byte-identical duplicates + + If database dumps contained byte-identical duplicates, they + were treated as a single item on the left side of a comparison. + This caused erroneous "added" entries on the right side. + +* secure password storage: usage of GNOME Keyring vs. KDE KWallet configurable + + Automatically detecting KDE users is not possible at the + moment. Instead KDE users have to manually set the new "keyring" + global config property to "KDE" (case insensitive) if the + SyncEvolution installation supports both, because GNOME Keyring is the + default to avoid surprises for traditional users. If only KWallet + support is enabled, then this is not necessary. + + "GNOME" and "true/false/1/0/yes/no" can also be set. This has the + advantage that keyring usage can be enabled permanently for the + command line in --daemon=no mode; normally keyrings are not used in + that mode because accessing them can bring up UI dialogs. + + It also becomes possible to disable keyring usage in syncevo-dbus-server, + something which couldn't be done before. + + The --keyring command line option is still supported, as an alias for + "[--sync-property] keyring=<value>". The default value for --keyring + is true, to match the traditional behavior. In contrast to other sync + properties, setting "keyring" does not require an explicit --run + parameter. Again this is done to mirror traditional usage. + +* config: improved 'maxlogdirs' documentation + + The old explanation made it sound like nothing would get deleted by + default ("If set, ..."). That's not correct, by default only 10 + sessions are kept. + Also explain the behavior of deleting intermediate sessions first. + +* Evolution: always create databases (PTCOM-113) + + Always try to create address book or calendar database, because even + if there is a source there's no guarantee that the actual database + was created already; the original logic for only setting this when + explicitly requesting a new database therefore failed in some cases. + + This problem affected users who had never created anything locally + and wanted to use SyncEvolution to migrate their data. Now that + works without having to create dummy entries first. + +* Evolution contacts: changed default sync format to vCard 3.0 + + vCard 3.0 is the better default because it has saner encoding + rules and defines more properties, thus avoiding the need for + non-standard extensions. However, Mobical has problems with + the new default. See upgrade instructions below. + +* Evolution: added support for EDS 3.5.x + + When compiled against EDS 3.5.x or later, SyncEvolution now uses + the backend code originally written for the EClient API introduced + in EDS 3.2. That code was changed so that it works with the new + include file rules and ESourceRegistry in EDS 3.5.x. Support + for using the EClient API with EDS 3.4 was removed because maintaining + three different flavors of the EDS backend code would be too much + work and not gain much (just the possibility to test the EDSClient + code with 3.4). + + At the moment, this is a compile time choice made automatically + by configure. syncevolution.org binaries are compiled against + an older EDS and thus do not work with EDS 3.5.x or later. + + EDS 3.5.x handles authentication itself, using a standard system + prompt if necessary. SyncEvolution can no longer provide the password, + and thus the "databaseUser/Password" options have no effect when using + EDS 3.5.x. + +* D-Bus server: fixed HTTP presence for recent libdbus + + Testing with libdbus 1.6.0 on Debian Testing failed because the lib + changed some behavior: instead of looking up the owner of a certain + bus name immediately, it now does that when invoking a + method. Therefore the check for "have connection" in SyncEvolution + was too simplistic and missed the fact that both were not usable, + causing the server to assume that HTTP was down while in reality it + should have assumed it to be up. This prevented auto-syncing and + manually clicking "Sync" in the GTK UI. + +* D-Bus server: made notification verbosity configurable with "notifyLevel" + + The new "notifyLevel" per-peer configuration option allows users to + control how many desktop notifications the D-Bus server produces while + executing an automatic sync: + + 0 - suppress all notifications + 1 - show only errors + 2 - show information about changes and errors (in practice currently the same as level 3) + 3 - show all notifications, including starting a sync (default) + +* WebDAV: fixed data corruption issue when uploading item with long UID + + In some cases data with a very long UID wasn't handled correctly, + causing the out-going data to be malformed and probably causing a + rejection by the server. The root cause is incorrect string handling. + In releases before 1.2.99.1, only the --import operation of contacts + into CardDAV were affected. In 1.2.99.1, the same code also got used + for calendar items and then could also affect syncing. + +* CalDAV: updated Google workarounds + + Google started sending empty items (VCALENDAR with no VEVENT inside) + which cannot be removed. SyncEvolution 1.3 ignores such items. + + The workaround for a 404 from Google Calendar for a GET (sending a + REPORT request matching the item's UID) was broken: first, processing + the result ended up calling the unset responseEnd boost function + pointer, which caused the request to fail. Second, getting multiple + items wasn't handled (data from all items concatenated together was + used). + + That can happen in the somewhat unlike case that some items have a UID + which is a complete superset of the requested UID - not realistic in + real life, but happens during testing. + +* Google Calendar: updated URL redirect handling + + Google Calendar sometimes returns redirect requests to human-readable + web sites (an "unavailable" page, a login form). This is of course + bogus when the client is an automated CalDAV client. + + The "unavailable.html" case was already handled. Made it a bit more + flexible to also catch possible variations of it (additional + parameters, https instead of http). + + Added the https://accounts.google.com/ServiceLogin case. Not sure + whether retrying will help in that case, but there's not much else + that SyncEvolution can do. + +* WebDAV: bridge with SyncML + + Now a peer accessed via SyncML can read/write data stored in a + CalDAV/CardDAV server directly. This can be used to connect a device + which only supports SyncML to a CalDAV/CardDAV server, or sync data + between a SyncML server and a CalDAV/CardDAV server. See "CalDAV and + CardDAV" in the README for details. + +* WebDAV: improved --configure + + Added INFO output about checking sources. This helps with WebDAV when + the server cannot be contacted (dead, misconfigured) because otherwise + there would be no indication at all why the --configure operation + seems to hang. + + Here is some example output, including aborting: + $ syncevolution --configure --template webdav \ + syncURL=http://192.168.1.100:9000/ \ + username=foo password=bar retryDuration=2s \ + target-config@webdav-temp + [INFO] creating configuration target-config@webdav-temp + [INFO] addressbook: looking for databases... + [INFO] addressbook: no database to synchronize + [INFO] calendar: looking for databases... + [INFO] calendar: no database to synchronize + [INFO] memo: looking for databases... + [INFO] memo: no database to synchronize + [INFO] todo: looking for databases... + [INFO] todo: no database to synchronize + + It timed out fairly quickly here because of the retryDuration=2s. That + also gets placed in the resulting config, which is probably not desired. + + Aborting the operation is now supported: + + $ syncevolution --configure \ + --template webdav \ + syncURL=http://192.168.1.100:9000/ \ + username=foo password=bar \ + target-config@webdav-temp + [INFO] creating configuration target-config@webdav-temp + [INFO] addressbook: looking for databases... + ^C[INFO] Asking to suspend... + [INFO] Press CTRL-C again quickly (within 2s) to stop immediately (can cause problems in the future!) + ^C[INFO] Aborting immediately ... + [ERROR] error code from SyncEvolution aborted on behalf of user (local, status 20017): aborting as requested by user + + It would be good to make the CTRL-C handling code aware that it can + abort immediately instead of doing the intermediate "asking to suspend" + step, which only makes sense for sync sessions. + +* WebDAV: support tasks and memos (BMC #24893) + + The new backend property values "CalDAVTodo" and "CalDAVJournal" + select tasks resp. memos stored in a CalDAV collection. "CalDAV" + continues to select events. + + Events, tasks and journals can be mixed in the same resource (= + URL). However, this is less efficient than storing them separately. + + A good CalDAV server allows filtering items by type, and SyncEvolution + uses that. However, it was found that Radicale 0.7 ignores this + filtering, which could have led to data loss (SyncEvolution asks for + all VTODOs in preparation for a "delete all items" operation in a + "CalDAVTodo" source, gets also VJOURNALs, then deletes them). + + Therefore SyncEvolution plays it safe and downloads the VTODO and + VJOURNAL data to double-check that it is working on the right items. + This causes additional traffic for well-behaving servers; currently + it cannot be turned off. + + Tasks are exchanged as vCalendar 1.0 or iCalendar 2.0 VJOURNAL. + Memos are exchanged as VTODO or plain text. The logic for storing + incoming plain text is slightly different compared to the way how + the EDS memo backend did it: instead of copying the first line + from the text into the summary, it is now moved. In other words, + the first line gets stripped. The change is primarily technically + motivated; both approaches have pros and cons. + +* WebDAV: improved Radicale support + + Radicale > 0.7 will return status 200 for delete requests; + is now treated like 204 by SyncEvolution. 412 'Preconditiona Failed' + when asking to delete an already removed item is treated like + the more common 404 'not found'. Same with 410 'gone' instead + of 404 when trying to read a non-existent item. + +* CalDAV/CardDAV sync: improved target side output + + Added a "target side of local sync ready" INFO message to introduce + the output which has the target context in the [INFO] tag. The sync report + from the target side now has the target context embedded in brackets + after the "Changes applied during synchronization" header, to avoid + ambiguities. + + Sometimes the backend has to resend requests because of temporary + issues. If the problem turned out to be permanent, there was a long + period of time, retryDuration=5 minutes to be precice, in which no + visible progress happened. + + Now SyncEvolution's WebDAV backend will print a message like this + before going to sleep until it is time to retry: + + [INFO @googlecalendar] operation temporarily (?) failed, going to retry in 5.0s before giving up in 18.4s: PROPFIND: Neon error code 1: 401 Unauthorized + + The uncertainty comes from several factors. In this example, the 401 + might indicate a permanent problem (wrong credentials), or it could be + Google reporting a temporary authorization problem which is (probably) + meant to slow down the client while it asks the user to re-enter the + password. SyncEvolution only asks for passwords once, so it tries + again with the same password if it was successful with it in the + past. Otherwise it gives up immediately. + + Another dubious example are name server lookup errors. They can be + permanent (wrong host name) or temporary (name server + down). SyncEvolution errs on the side of retrying, to avoid + interrupting an operation which still has a chance to continue. + + Output from the target side of a local sync was passed through stderr + redirection as chunks of text to the frontends. This had several + drawbacks: + - forwarding only happened when the local sync parent was processing + the output redirection, which (due to limitations of the implementation) + only happens when it needs to print something itself + - debug messages were not forwarded + - message boundaries might have been lost + + In particular the new INFO messages are relevant while the sync runs + and need to be shown immediately. + +* WebDAV: --status for WebDAV source aborted + + The command line --status operation did not complete when applied to a + CalDAV/CardDAV source. Instead it aborted because the operation took a + code path where the backend was not fully initialized. + +* file backend: more flexible sync support for memos + + The databaseFormat=text/calendar for memos did not support + synchronizing as plain text. When using the new + databaseFormat=text/calendar+plain, vCalendar/iCalendar/plain text + are all valid sync formats; the storage is iCalendar 2.0 + VJOURNAL in all cases. + +* WebDAV: avoid potential crash during database detection + + When a server responds to a PROPFIND for a path with results for some + other path, then SyncEvolution crashed during the search for the + default calendar or address book because of a bug in the code which + was meant to handle that kind of response. Apparently Yahoo Calendar + did that. Now seen again in combination with Radicale 0.6.4. + + In general, the code was made more robust to cope with bugs in + Radicale 0.6.4. Later Radicale versions fixed these issues and also + worked with SyncEvolution 1.2.2 without client-side workarounds. + +* WebDAV: better path normalization + + "syncURL" and "database" properties had to end in a trailing slash, + otherwise items were not found (404 errors). Now the necessary slash + is added automatically. + +* Curl transport: support SSLServerCertificates=<path> + + When the setting refers to a directory, then CURLOPT_CAINFO doesn't + work (must be a file). Check this and use CURLOPT_CAPATH instead. + + Caveat: there are some comments in the API documentation about "NSS + enabled libcurl" which supports a directory in + CURLOPT_CAINFO. Hopefully providing an explicit path in CURLOPT_CAPATH + also works in that configuration. + +* code cleanup + rewrite: syncing done in separate process + + syncevo-dbus-server now runs syncing in a separate process. Local + sync also uses a second helper process. This makes the D-Bus server + more responsive via D-Bus (no more blocking operations) and + minimizes the effect of bugs in code involved with syncing + (backends, system libraries, etc.). + + In the long term this restructuring will also allow more advanced + features, like monitoring local or remote storage for changes. + +* SyncEvolution <-> SyncEvolution sync: multiple cycles per session + + SyncML only allows one send/receive cycle per session. There are cases + (for example, client side merges data that a dumber server failed to + match correctly) where client and server are still out of sync at + the end of a cycle. When SyncEvolution syncs with another SyncEvolution + instance (locally or remotely), both sides detect that the peer + can continue syncing in the same session and start over automatically + when needed. Previously the user had to start another sync session manually. + + To the user this is shown as "number of cycles" in a sync session + in the sync report. "Restart" is the process of entering a new cycle. + + The cycles are also visible in the command line output as multiple + INFO lines: + + [INFO] eds_contact: starting first time sync from client (peer is server) + [INFO] creating complete data backup of source eds_contact before sync (enabled with dumpData and needed for prin + Local data changes to be applied during synchronization: + *** eds_contact *** + no changes + + [INFO] eds_contact: sent 1/1 + [INFO] eds_contact: started + [INFO] eds_contact: first time sync done successfully + [INFO] eds_contact: starting normal sync from client (peer is server) <=== + [INFO] eds_contact: started <=== + [INFO] eds_contact: normal sync done successfully <=== + [INFO] creating complete data backup after sync (enabled with dumpData and needed for printChanges) + + Synchronization successful. + + Changes applied during synchronization: + +---------------|-----------------------|-----------------------|-CON-+ + | | LOCAL | REMOTE | FLI | + | Source | NEW | MOD | DEL | ERR | NEW | MOD | DEL | ERR | CTS | + +---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+ + | eds_contact | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | + | refresh-from-local, 2 cycles, 0 KB sent by client, 0 KB received | + ^^^^^^^^ + | item(s) in database backup: 1 before sync, 1 after it | + +---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+ + | start Tue Feb 7 17:07:49 2012, duration 0:03min | + | synchronization completed successfully | + +---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+ + +* SyncEvolution <-> SyncEvolution sync: negotiate UID support via SyncCap (BMC #22783) + + The semantic of UID/RECURRENCE-ID in calendar data is now tracked + per data store involved in a sync. If full iCalendar 2.0 semantic + (= IDs are globally unique) is guaranteed, then pairs are found + based on these IDs. Otherwise pairs must be found by looking at + item attributes. + + Previously a hack was used to detect this kind of support (any kind + of SyncEvolution instance was assumed to support it, although some + backends do not). + +* engine: add DTSTAMP+LAST-MODIFIED before writing calendar items + + When writing calendar items into a backend storage as iCalendar 2.0 or + vCalendar 1.0, they should have DTSTAMP and LAST-MODIFIED values. DTSTAMP + is expected by some CalDAV servers (like Apple). LAST-MODIFIED is usually + added by the storage, but not always. + + In the text/plain -> syncevolution -> text/calendar -> Radicale -> EDS + -> syncevolution chain the LAST-MODIFIED was not added by Radicale, which caused + problems for change tracking in an EDS-based SyncEvolution. + + Also necessary when importing from a phone using vCalendar without + DTSTAMP directly into CalDAV. + +* autotools: ensure that link lines are complete + + As mentioned by Tino Keitel on the mailing list, some libs and + executables were only implicitly linked against libraries that they + called directly. This happened to work by chance because these libraries + ended up in the running executable anyway, due to indirect loading. + Now there is a "make installcheck" test for this kind of defect + and the makefiles were updated to avoid it. + + One exception is libsmltk, which depends on the caller providing + SySync logging support. + +* syncevolution.org packages: fixed D-Bus server autostart in .deb and .rpm packages + + syncevo-dbus-server wasn't started automatically as part of a user + session because /etc/xdg/autostart/syncevo-dbus-server.desktop wasn't + included in the packages. This broke auto syncing after a session + restart (required manually starting SyncEvolution). + +* syncevolution.org packages: support KDE + + The traditional "syncevolution-evolution" package was + replaced with "syncevolution-bundle". A meta "syncevolution-evolution" + package depends on it, to support seamless updates for users who have + "syncevolution-evolution" installed. + + Binary dependencies of the main .deb are ignored for backends + because loading them is optional. The new "syncevolution-kde" + package has the right dependencies for KDE/Akonadi, while + "syncevolution-evolution" mostly just lists standard libs + if the "EDS compatibility" mode is used, where libebook/libecal + are loaded dynamically. + + Platform specific code (GNOME keyring, KDE wallet) was moved into + loadable, optional modules, to allow installation of the SyncEvolution + bundle without forcing the installation of unused system components. + +* D-Bus: use GIO D-Bus instead of libdbus if available + + When compiling from source, the more modern GIO D-Bus is used instead + of libdbus if available and recent enough (>= 2.30). syncevolution.org + binaries still use libdbus, to stay compatible with older Linux + distros. + +* several minor bug fixes + + syncevo-dbus-server now runs under valgrind in the nightly testing, + plus several more test scenarios were added. This helped to find + and fix various minor memory handling issues. + +* developers: backend API changes + + beginSync/endSync() (aka m_startDataRead/m_endDataWrite) may now be + called multiple times per SyncSource instance life cycle. SyncSources + derived from TrackingSyncSource should work without changes. Use the + Client::Source::*::testChangesMultiCycles test to check whether your + backend supports this correctly. + + Reading and deleting must throw a 404 status exception when an item + is not found. The Client::Source::*::*404 tests cover this. + + The special semantic of the former RegisterSyncSource::InactiveSource + (invalid pointer of value 1) caused bugs, like using it in + --print-databases (=> segfault) or not being able to store the result + of a createSource() directly in a smart pointer (=> potential leak in + SyncSource::createSource()). + + Obviously a bad idea to start with. Replaced with a + RegisterSyncSource::InactiveSource() method which creates a real, + inactive SyncSource instance which can and must be deleted by the + caller. + + This is a SyncSource API change for backend developers. Instead of + RegisterSyncSource::InactiveSource, return + RegisterSyncSource::InactiveSource(). Comparisons against + RegisterSyncSource::InactiveSource needs to be replaced with a call + to the new SyncSource::isInactive(). + + Long-running backend calls are encouraged to check for events on the + main glib context (either in a loop or with + g_main_context_iteration(NULL)) and abort when + SuspendFlags::getSuspendFlags().getState() returns + SuspendFlags::ABORT. + + Implementing the improved local sync output required extending the + D-Bus API. The Server.LogOutput signal now has an additional + "process name" parameter. Normally it is empty. For messages + originating from the target side, it carries that extra target + context string. + + This D-Bus API change is backward compatible. Older clients can still + subscribe to and decode the LogOutput messages, they'll simply ignore + the extra parameter. Newer clients expecting that extra parameter + won't work with an older D-Bus daemon: they'll fail to decode the + D-Bus message. + +* packagers: + + libgdbussyncevo is now installed as a normal library in /usr/lib, + even though SyncEvolution is the only user. + + pcrecpp is now a new hard dependency. + + +Upgrading from release 1.2.x: + +The sync format of existing configurations for Mobical (aka Everdroid) +must be updated manually, because the server has encoding problems when +using vCard 3.0 (now the default for Evolution contacts): + syncevolution --configure \ + syncFormat=text/x-vcard \ + mobical addressbook + +The Funambol template explicitly enables usage of the +"refresh-from-server" sync mode to avoid getting throttled with 417 +'retry later' errors. The same must be added to existing configs +manually: + syncevolution --configure \ + enableRefreshSync=TRUE \ + funambol + +Upgrading from releases before 1.2: + +Old configurations can still be read. But writing, as it happens +during a sync, must migrate the configuration first. Releases >= 1.2 +automatically migrates configurations. The old configurations +will still be available (see "syncevolution --print-configs") but must +be renamed manually to use them again under their original names with +older SyncEvolution releases. + + +SyncEvolution 1.2.99.3 -> 1.3, 10.09.2012 +========================================= + +Final SyncEvolution 1.3 release. The pre-releases did have the desired +effect of flushing out bugs not found by nightly testing alone. Thanks +everyone for packaging, downloading and testing them! Time to get it +out officially as the next stable release. + +* D-Bus server + GIO D-Bus: shutdown fix + + When compiled against GIO D-Bus (not the case in syncevolution.org + binaries), the syncevo-dbus-server occasionally shut down before + sending out all pending D-Bus messages. Showed up only in nightly + testing. + +* D-Bus server + GIO D-Bus: fix auto-activation (Debian bug #599247) + + When syncevo-dbus-server was started on demand by the D-Bus daemon, + then it registered itself with the daemon before it was ready to + serve requests. Only happened in combination with GIO D-Bus and + thus was not a problem before 1.2.99.x. + + One user-visible effect was that the GTK UI did not select the default + service when it was started for the first time, because it could not + retrieve that information from syncevo-dbus-server. + +* local sync: fix timeout with local sync with libdbus + + When using libdbus instead of GIO D-Bus (as done by syncevolution.org + binaries and SyncEvolution on Maemo), local sync may have aborted + after 25 seconds when syncing many items with a D-Bus timeout error: + + [ERROR] sending message to child failed: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible ca + + Reported by Toke Høiland-Jørgensen for Harmattan. Somehow not encountered + elsewhere. + +* KDE: check for D-Bus to avoid crash in KApplication (BMC #25596) + + Some unnamed version of KDE crashes in KApplication when invoked + without a D-Bus session. The reporter ran into this when compiling + from source, because the SyncEvolution binary is invoked as part of + the build process, which ran outside of a D-Bus session. + + Avoid the crash by checking for a D-Bus session bus before instantiating + KApplication. Instantiating KApplication was added for KWallet support. + Without D-Bus, KWallet does not work either, therefore throw an explicit + error when the lack of D-Bus is detected. + +* Funambol: work around PHOTO TYPE=image/jpeg + + A combination of Funambol Android and Funambol server recently led to + the Funambol server sending PHOTO data with TYPE=image/jpeg. This is + invalid and caused EDS to reject the photo (Vladimir Elisseev, + "[SyncEvolution] issues with syncing photos"). + + Work around the problem by only keeping the part of the type after the + last slash, if there is any. For image/jpeg and similar types that + leads to the desired value and does not affect valid values, because + those do not contain a slash + (http://www.iana.org/assignments/media-types/image/index.html). + +* syncevo-http-server: fixed printing of server debug output + + Python failed to call logSyncEvoOutput() after adding the additional + 'process' parameter to LogOutput because it extracts all four + parameters and then cannot pass them to logSyncEvoOutput(). + + Now logSyncEvoOutput() uses the new process information to instantiate + a logger with the right prefix, using 'sync' as fallback for messages + without that information (as before). + +* Some minor code and test cleanup. + + SyncEvolution 1.2.99.3 -> 1.2.99.4, 07.08.2012 ============================================== diff --git a/configure.ac b/configure.ac index 8043aa38..79fc4f52 100644 --- a/configure.ac +++ b/configure.ac @@ -8,7 +8,7 @@ dnl Invoke autogen.sh to produce a configure script. # # Starting with the 1.1 release cycle, the rpm-style # .99 pseudo-version number is used to mark a pre-release. -AC_INIT([syncevolution], [m4_esyscmd([build/gen-git-version.sh 1.2.99.4])]) +AC_INIT([syncevolution], [m4_esyscmd([build/gen-git-version.sh 1.3])]) # STABLE_VERSION=1.0.1+ AC_SUBST(STABLE_VERSION) @@ -25,7 +25,7 @@ SE_CHECK_FOR_STABLE_RELEASE # Minimum version of libsynthesis as defined in its # configure script and thus .pc files: -define([SYNTHESIS_MIN_VERSION], [3.4.0.16.7]) +define([SYNTHESIS_MIN_VERSION], [3.4.0.16.8]) # Line above is patched by gen-autotools.sh. Handle # both "yes" and "no". diff --git a/src/backends/activesync/ActiveSyncSource.cpp b/src/backends/activesync/ActiveSyncSource.cpp index fe320552..0c50a1a8 100644 --- a/src/backends/activesync/ActiveSyncSource.cpp +++ b/src/backends/activesync/ActiveSyncSource.cpp @@ -68,11 +68,11 @@ void ActiveSyncSource::open() username.c_str(), m_context->getConfigName().c_str()); - m_account = username.c_str(); + m_account = username; m_folder = getDatabaseID(); // create handler - m_handler.set(eas_sync_handler_new(m_account), "EAS handler"); + m_handler.set(eas_sync_handler_new(m_account.c_str()), "EAS handler"); } void ActiveSyncSource::close() @@ -98,7 +98,10 @@ void ActiveSyncSource::beginSync(const std::string &lastToken, const std::string SE_LOG_DEBUG(this, NULL, "sync key empty, starting slow sync"); m_ids->clear(); } else { - SE_LOG_DEBUG(this, NULL, "sync key %s, starting incremental sync", lastToken.c_str()); + SE_LOG_DEBUG(this, NULL, "sync key %s for account '%s' folder '%s', starting incremental sync", + lastToken.c_str(), + m_account.c_str(), + m_folder.c_str()); } gboolean moreAvailable = TRUE; @@ -219,7 +222,7 @@ void ActiveSyncSource::beginSync(const std::string &lastToken, const std::string } if (slowSync) { - // tell engine that we need a slow sync + // tell engine that we need a slow sync, if it didn't know already SE_THROW_EXCEPTION_STATUS(StatusException, "ActiveSync error: Invalid synchronization key", STATUS_SLOW_SYNC_508); diff --git a/src/backends/activesync/ActiveSyncSource.h b/src/backends/activesync/ActiveSyncSource.h index 28f61eac..ffeed231 100644 --- a/src/backends/activesync/ActiveSyncSource.h +++ b/src/backends/activesync/ActiveSyncSource.h @@ -135,8 +135,7 @@ class ActiveSyncSource : // also for other keys if the need ever arises). m_itemNode(new PrefixConfigNode("item-", boost::shared_ptr<ConfigNode>(new SafeConfigNode(params.m_nodes.getTrackingNode())))), - m_context(params.m_context), - m_account(0) + m_context(params.m_context) { if (!m_context) { m_context.reset(new SyncConfig()); @@ -188,7 +187,7 @@ class ActiveSyncSource : boost::shared_ptr<SyncConfig> m_context; /** account ID for libeas, must be set in "username" config property */ - const char* m_account; + std::string m_account; /** folder ID for libeas, optionally set in "database" config property */ std::string m_folder; diff --git a/src/backends/kde/KDEPlatform.cpp b/src/backends/kde/KDEPlatform.cpp index 1016dbfd..1b98ccdc 100644 --- a/src/backends/kde/KDEPlatform.cpp +++ b/src/backends/kde/KDEPlatform.cpp @@ -45,8 +45,22 @@ #include <syncevo/declarations.h> SE_BEGIN_CXX +// TODO: this check should be global +static bool HaveDBus; + void KDEInitMainSlot(const char *appname) { + // Very simple check. API doesn't say whether asking + // for the bus connection will connect immediately. + QDBusConnection dbus = QDBusConnection::sessionBus(); + HaveDBus = dbus.isConnected(); + + if (!HaveDBus) { + // KApplication has been seen to crash without D-Bus (BMC #25596). + // Bail out here if we don't have D-Bus. + return; + } + //QCoreApplication *app; int argc = 1; static char *argv[] = { const_cast<char *>(appname), NULL }; @@ -116,6 +130,11 @@ static bool UseKWallet(const InitStateTri &keyring, return false; } + // User wants KWallet, but is it usable? + if (!HaveDBus) { + SE_THROW("KDE KWallet requested, but it is not usable (running outside of a D-Bus session)"); + } + // Use KWallet. return true; } diff --git a/src/dbus/server/main.cpp b/src/dbus/server/main.cpp index 965821b2..6fceb013 100644 --- a/src/dbus/server/main.cpp +++ b/src/dbus/server/main.cpp @@ -150,12 +150,15 @@ int main(int argc, char **argv, char **envp) unsetenv("G_DBUS_DEBUG"); } + dbus_bus_connection_undelay(conn); server->run(); SE_LOG_DEBUG(NULL, NULL, "cleaning up"); server.reset(); - conn.reset(); obj.reset(); guard.reset(); + SE_LOG_DEBUG(NULL, NULL, "flushing D-Bus connection"); + conn.flush(); + conn.reset(); SE_LOG_INFO(NULL, NULL, "terminating"); return 0; } catch ( const std::exception &ex ) { diff --git a/src/gdbus/gdbus-cxx-bridge.h b/src/gdbus/gdbus-cxx-bridge.h index b8426cfa..92213903 100644 --- a/src/gdbus/gdbus-cxx-bridge.h +++ b/src/gdbus/gdbus-cxx-bridge.h @@ -60,6 +60,14 @@ #include "gdbus.h" #include "gdbus-cxx.h" +// Not defined by 1.4.x in Maemo Harmattan; INT_MAX has the same +// value and effect there. In older libdbus, it is the same as +// a very long timeout (2147483s), which is good enough. +#include <stdint.h> +#ifndef DBUS_TIMEOUT_INFINITE +# define DBUS_TIMEOUT_INFINITE INT_MAX +#endif + #include <map> #include <vector> #include <utility> @@ -120,6 +128,9 @@ class DBusConnectionPtr : public boost::intrusive_ptr<DBusConnection> return conn; } + /** empty stub: flushing only necessary with GIO D-Bus */ + void flush() {} + /** GDBus GIO specific: disconnect callback */ typedef boost::function<void ()> Disconnect_t; void setDisconnect(const Disconnect_t &func); @@ -4213,7 +4224,7 @@ protected: void send(DBusMessagePtr &msg, const Callback_t &callback) { DBusPendingCall *call; - if (!dbus_connection_send_with_reply(m_conn.get(), msg.get(), &call, -1)) { + if (!dbus_connection_send_with_reply(m_conn.get(), msg.get(), &call, DBUS_TIMEOUT_INFINITE)) { throw std::runtime_error("dbus_connection_send failed"); } else if (call == NULL) { throw std::runtime_error("received pending call is NULL"); @@ -4233,7 +4244,7 @@ protected: DBusErrorCXX error; // Constructor steals reference, reset() doesn't! // Therefore use constructor+copy instead of reset(). - DBusMessagePtr reply = DBusMessagePtr(dbus_connection_send_with_reply_and_block(m_conn.get(), msg.get(), -1, &error)); + DBusMessagePtr reply = DBusMessagePtr(dbus_connection_send_with_reply_and_block(m_conn.get(), msg.get(), DBUS_TIMEOUT_INFINITE, &error)); if (!reply) { error.throwFailure(m_method); } diff --git a/src/gdbusxx/gdbus-cxx-bridge.cpp b/src/gdbusxx/gdbus-cxx-bridge.cpp index 62c0fc94..7056a2ed 100644 --- a/src/gdbusxx/gdbus-cxx-bridge.cpp +++ b/src/gdbusxx/gdbus-cxx-bridge.cpp @@ -33,12 +33,63 @@ namespace GDBusCXX { MethodHandler::MethodMap MethodHandler::m_methodMap; boost::function<void (void)> MethodHandler::m_callback; -static void GDBusNameLost(GDBusConnection *connection, - const gchar *name, - gpointer user_data) +// It should be okay to use global variables here because they are +// only used inside the main thread while it waits in undelay() for a +// positive or negative result to g_bus_own_name_on_connection(). +// Once acquired, the name can only get lost again when the +// D-Bus daemon dies (no name owership alloed), in which case the +// process dies anyway. +static bool nameError; +static bool nameAcquired; +static void BusNameAcquired(GDBusConnection *connection, + const gchar *name, + gpointer userData) throw () { - g_critical("lost D-Bus connection or failed to obtain %s D-Bus name, quitting", name); - exit(1); + try { + g_debug("got D-Bus name %s", name); + nameAcquired = true; + } catch (...) { + nameError = true; + } +} + +static void BusNameLost(GDBusConnection *connection, + const gchar *name, + gpointer userData) throw () +{ + try { + g_debug("lost %s %s", + connection ? "D-Bus connection for name" : + "D-Bus name", + name); + } catch (...) { + } + nameError = true; +} + +void DBusConnectionPtr::undelay() const +{ + if (!m_name.empty()) { + g_debug("starting to acquire D-Bus name %s", m_name.c_str()); + nameAcquired = false; + nameError = false; + char *copy = g_strdup(m_name.c_str()); + g_bus_own_name_on_connection(get(), + copy, + G_BUS_NAME_OWNER_FLAGS_NONE, + BusNameAcquired, + BusNameLost, + copy, + g_free); + while (!nameAcquired && !nameError) { + g_main_context_iteration(NULL, true); + } + g_debug("done with acquisition of %s", m_name.c_str()); + if (nameError) { + throw std::runtime_error("could not obtain D-Bus name - already running?"); + } + } + g_dbus_connection_start_message_processing(get()); } DBusConnectionPtr dbus_get_bus_connection(const char *busType, @@ -48,10 +99,13 @@ DBusConnectionPtr dbus_get_bus_connection(const char *busType, { DBusConnectionPtr conn; GError* error = NULL; + GBusType type = + boost::iequals(busType, "SESSION") ? + G_BUS_TYPE_SESSION : + G_BUS_TYPE_SYSTEM; - if(unshared) { - char *address = g_dbus_address_get_for_bus_sync(boost::iequals(busType, "SESSION") ? - G_BUS_TYPE_SESSION : G_BUS_TYPE_SYSTEM, + if (unshared) { + char *address = g_dbus_address_get_for_bus_sync(type, NULL, &error); if(address == NULL) { if (err) { @@ -76,9 +130,7 @@ DBusConnectionPtr dbus_get_bus_connection(const char *busType, } } else { // This returns a singleton, shared connection object. - conn = DBusConnectionPtr(g_bus_get_sync(boost::iequals(busType, "SESSION") ? - G_BUS_TYPE_SESSION : - G_BUS_TYPE_SYSTEM, + conn = DBusConnectionPtr(g_bus_get_sync(type, NULL, &error), false); if(conn == NULL) { @@ -89,11 +141,11 @@ DBusConnectionPtr dbus_get_bus_connection(const char *busType, } } - if(name) { - // Copy name, to ensure that it remains available. - char *copy = g_strdup(name); - g_bus_own_name_on_connection(conn.get(), copy, G_BUS_NAME_OWNER_FLAGS_NONE, - NULL, GDBusNameLost, copy, g_free); + if (name) { + // Request name later in undelay(), after the caller + // had a chance to add objects. + conn.addName(name); + // Acting as client, need to stop when D-Bus daemon dies. g_dbus_connection_set_exit_on_close(conn.get(), TRUE); } @@ -124,11 +176,6 @@ DBusConnectionPtr dbus_get_bus_connection(const std::string &address, return conn; } -void dbus_bus_connection_undelay(const DBusConnectionPtr &conn) -{ - g_dbus_connection_start_message_processing(conn.get()); -} - static void ConnectionLost(GDBusConnection *connection, gboolean remotePeerVanished, GError *error, @@ -145,6 +192,12 @@ static void DestroyDisconnect(gpointer data, delete cb; } +void DBusConnectionPtr::flush() +{ + // ignore errors + g_dbus_connection_flush_sync(get(), NULL, NULL); +} + void DBusConnectionPtr::setDisconnect(const Disconnect_t &func) { g_signal_connect_closure(get(), diff --git a/src/gdbusxx/gdbus-cxx-bridge.h b/src/gdbusxx/gdbus-cxx-bridge.h index e45b1cd2..18bf0168 100644 --- a/src/gdbusxx/gdbus-cxx-bridge.h +++ b/src/gdbusxx/gdbus-cxx-bridge.h @@ -124,6 +124,14 @@ inline void throwFailure(const std::string &object, class DBusConnectionPtr : public boost::intrusive_ptr<GDBusConnection> { + /** + * Bus name of client, as passed to dbus_get_bus_connection(). + * The name will be requested in dbus_bus_connection_undelay() = + * undelay(), to give the caller a chance to register objects on + * the new connection. + */ + std::string m_name; + public: DBusConnectionPtr() {} // connections are typically created once, so increment the ref counter by default @@ -138,9 +146,18 @@ class DBusConnectionPtr : public boost::intrusive_ptr<GDBusConnection> return conn; } + /** + * Ensure that all IO is sent out of the process. + * Blocks. Only use it right before shutting down. + */ + void flush(); + typedef boost::function<void ()> Disconnect_t; void setDisconnect(const Disconnect_t &func); // #define GDBUS_CXX_HAVE_DISCONNECT 1 + + void undelay() const; + void addName(const std::string &name) { m_name = name; } }; class DBusMessagePtr : public boost::intrusive_ptr<GDBusMessage> @@ -224,7 +241,7 @@ DBusConnectionPtr dbus_get_bus_connection(const std::string &address, DBusErrorCXX *err, bool delayed = false); -void dbus_bus_connection_undelay(const DBusConnectionPtr &conn); +inline void dbus_bus_connection_undelay(const DBusConnectionPtr &conn) { conn.undelay(); } /** * Wrapper around DBusServer. Does intentionally not expose diff --git a/src/syncevo/configs/datatypes/02vcard-types.xml b/src/syncevo/configs/datatypes/02vcard-types.xml index 226af996..41c21725 100644 --- a/src/syncevo/configs/datatypes/02vcard-types.xml +++ b/src/syncevo/configs/datatypes/02vcard-types.xml @@ -7,6 +7,7 @@ <incomingscript><![CDATA[ $VCARD_INCOMING_NAMECHANGE_SCRIPT $VCARD_INCOMING_PHOTO_VALUE_SCRIPT + $VCARD_INCOMING_PHOTO_TYPE_SCRIPT ]]></incomingscript> <outgoingscript><![CDATA[ $VCARD_OUTGOING_ADDREV_SCRIPT @@ -22,6 +23,7 @@ <incomingscript><![CDATA[ $VCARD_INCOMING_NAMECHANGE_SCRIPT $VCARD_INCOMING_PHOTO_VALUE_SCRIPT + $VCARD_INCOMING_PHOTO_TYPE_SCRIPT ]]></incomingscript> <outgoingscript><![CDATA[ $VCARD_OUTGOING_ADDREV_SCRIPT diff --git a/src/syncevo/configs/scripting/04vcard-photo-value.xml b/src/syncevo/configs/scripting/04vcard-photo-value.xml index 7981aa31..d3a40643 100644 --- a/src/syncevo/configs/scripting/04vcard-photo-value.xml +++ b/src/syncevo/configs/scripting/04vcard-photo-value.xml @@ -10,6 +10,15 @@ } ]]></macro> + <macro name="VCARD_INCOMING_PHOTO_TYPE_SCRIPT"><![CDATA[ + // Fix PHOTO TYPE=image/jpeg (sent by Funambol). + INTEGER pos; + pos = RFIND(PHOTO_TYPE, "/"); + if (pos != UNASSIGNED) { + PHOTO_TYPE = SUBSTR(PHOTO_TYPE, pos + 1); + } + ]]></macro> + <macro name="VCARD_OUTGOING_PHOTO_VALUE_SCRIPT"><![CDATA[ // Ensure that PHOTO_VALUE == "binary" is not sent (it's the default). if (PHOTO_VALUE == "binary") { diff --git a/test/ClientTest.cpp b/test/ClientTest.cpp index 26c18cc1..f1f8b59f 100644 --- a/test/ClientTest.cpp +++ b/test/ClientTest.cpp @@ -4695,14 +4695,8 @@ bool addBothSidesAddStatsBroken = false; // duplicates itself; the client needs to do that bool addBothSidesServerIsDumb = getenv("CLIENT_TEST_ADD_BOTH_SIDES_SERVER_IS_DUMB") != NULL; -void SyncTests::testAddBothSides() +static void testAddBothSidesFixUpdateItem(std::string &updateItem) { - CT_ASSERT_NO_THROW(deleteAll()); - accessClientB->deleteAll(); - - std::string insertItem = sources[0].second->config.m_insertItem; - std::string updateItem = sources[0].second->config.m_updateItem; - if (addBothSidesNoMergeLines) { // VEVENT boost::replace_all(updateItem, "LOCATION:big meeting room", "LOCATION:my office"); @@ -4712,6 +4706,16 @@ void SyncTests::testAddBothSides() // VTODO boost::replace_all(updateItem, "DESCRIPTION:to be done", "DESCRIPTION:to be done<<REVISION>>"); } +} + +void SyncTests::testAddBothSides() +{ + CT_ASSERT_NO_THROW(deleteAll()); + accessClientB->deleteAll(); + + std::string insertItem = sources[0].second->config.m_insertItem; + std::string updateItem = sources[0].second->config.m_updateItem; + testAddBothSidesFixUpdateItem(updateItem); CT_ASSERT_NO_THROW(sources[0].second->insert(sources[0].second->createSourceA, insertItem)); @@ -4809,11 +4813,7 @@ void SyncTests::testAddBothSidesRefresh() std::string insertItem = sources[0].second->config.m_insertItem; std::string updateItem = sources[0].second->config.m_updateItem; - - if (addBothSidesNoMergeLines) { - boost::replace_all(updateItem, "LOCATION:big meeting room", "LOCATION:my office"); - boost::replace_all(updateItem, "DESCRIPTION:nice to see you", "DESCRIPTION:let's talk<<REVISION>>"); - } + testAddBothSidesFixUpdateItem(updateItem); // insert initial item data on B CT_ASSERT_NO_THROW(accessClientB->sources[0].second->insert(accessClientB->sources[0].second->createSourceA, diff --git a/test/runtests.py b/test/runtests.py index 4d424570..bf9e8b1c 100755 --- a/test/runtests.py +++ b/test/runtests.py @@ -1146,7 +1146,9 @@ test.alarmSeconds = 2400 context.add(test) class ActiveSyncTest(SyncEvolutionTest): - def __init__(self, name, sources = [ "eas_event", "eas_contact", "eds_event", "eds_contact" ], env = ""): + def __init__(self, name, sources = [ "eas_event", "eas_contact", "eds_event", "eds_contact" ], + env = "", + knownFailures = []): tests = [] if "eds_event" in sources: tests.append("Client::Sync::eds_event") @@ -1167,6 +1169,29 @@ class ActiveSyncTest(SyncEvolutionTest): "EAS_SOUP_LOGGER=1 " "EAS_DEBUG=5 " "EAS_DEBUG_DETACHED_RECURRENCES=1 " + + "CLIENT_TEST_FAILURES=" + + ",".join(knownFailures + + # time zone mismatch between client and server, + # still need to investigate + [ ".*::LinkedItemsWeekly::testSubsetStart11Skip[0-3]", + ".*::LinkedItemsWeekly::testSubsetStart22Skip[1-3]", + ".*::LinkedItemsWeekly::testSubsetStart33Skip[1-3]", + ".*::LinkedItemsWeekly::testSubsetStart44.*" ] + + # The disables the synccompare simplifications for + # BDAY and friends, and therefore fails. + [ ".*::testExtensions" ] + ) + + " " + + "CLIENT_TEST_SKIP=" + # See "[SyncEvolution] one-way sync + sync tokens not updated": + # one-way sync keeps using old (and obsolete) sync keys, + # thus running into unexpected slow syncs with ActiveSync. + "Client::Sync::.*::testOneWayFromClient," + "Client::Sync::.*::testOneWayFromLocal," + " " + "CLIENT_TEST_LOG=activesyncd.log " , testPrefix=" ".join(("env EAS_DEBUG_FILE=activesyncd.log", @@ -1205,7 +1230,15 @@ context.add(test) test = ActiveSyncTest("googleeas", ["eds_contact", "eas_contact"], - "CLIENT_TEST_DELAY=10 CLIENT_TEST_SOURCE_DELAY=10 ") + env="CLIENT_TEST_DELAY=10 CLIENT_TEST_SOURCE_DELAY=10 ", + knownFailures=[ + # Google does not support the Fetch operation, leading + # to an unhandled generic error. + ".*::testReadItem404", + # Remove of PHOTO not supported by Google (?), + # works with Exchange. + "Client::Source::eas_contact::testRemoveProperties", + ]) context.add(test) syncevoPrefix=" ".join([os.path.join(sync.basedir, "test", "wrappercheck.sh")] + diff --git a/test/synccompare.pl b/test/synccompare.pl index 6c4cf4d3..f66937a8 100644 --- a/test/synccompare.pl +++ b/test/synccompare.pl @@ -658,8 +658,13 @@ sub NormalizeItem { } if ($googleeas) { - # unsupported properties - s/^(FN|X-EVOLUTION-FILE-AS|CATEGORIES)(;[^:;\n]*)*:.*\r?\n?//gm; + # properties not supported by Google + s/^(X-EVOLUTION-FILE-AS|CATEGORIES)(;[^:;\n]*)*:.*\r?\n?//gm; + } + + if ($googleeas || $exchange) { + # properties not supported by ActiveSync + s/^(FN)(;[^:;\n]*)*:.*\r?\n?//gm; } if ($googleeas || $exchange) { diff --git a/test/syncevo-http-server.py b/test/syncevo-http-server.py index 8a0b6240..bf95a547 100755 --- a/test/syncevo-http-server.py +++ b/test/syncevo-http-server.py @@ -32,9 +32,6 @@ from OpenSSL import SSL # for output from this script itself logger = logging.getLogger("syncevo-http") -# for output from core SyncEvolution -loggerCore = logging.getLogger("sync") - class ChainedOpenSSLContextFactory(ssl.DefaultOpenSSLContextFactory): def __init__(self, privateKeyFileName, certificateChainFileName, sslmethod = SSL.SSLv3_METHOD): @@ -372,8 +369,8 @@ evo2python = { "WARNING": logging.WARNING } -def logSyncEvoOutput(path, level, output): - loggerCore.log(evo2python.get(level, logging.ERROR), "%s: %s", path, output) +def logSyncEvoOutput(path, level, output, component): + logging.getLogger(component or 'sync').log(evo2python.get(level, logging.ERROR), "%s: %s", path, output) usage = """usage: %prog [options] http://localhost:<port>/<path> diff --git a/test/sys.supp b/test/sys.supp index c9c00e32..a8bdbaae 100644 --- a/test/sys.supp +++ b/test/sys.supp @@ -29,9 +29,7 @@ obj:*/libdb-5.1.so fun:__log_put_record fun:__ham_add_el - obj:*/libdb-5.1.so - fun:__dbc_iput - fun:__db_put + ... fun:__db_put_pp } @@ -826,6 +824,34 @@ fun:*GNOME*Password* } +# ==5221== 280 (24 direct, 256 indirect) bytes in 1 blocks are definitely lost in loss record 2,234 of 2,429 +# ==5221== at 0x4C28BED: malloc (vg_replace_malloc.c:263) +# ==5221== by 0xB37B960: ??? (in /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0) +# ==5221== by 0xB37C888: ??? (in /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0) +# ==5221== by 0xB37CBCE: ??? (in /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0) +# ==5221== by 0xB3C5A09: ??? (in /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0) +# ==5221== by 0x7EE7AFA: ??? (in /usr/lib/x86_64-linux-gnu/libgnome-keyring.so.0.2.0) +# ==5221== by 0x7EDC0AB: ??? (in /usr/lib/x86_64-linux-gnu/libgnome-keyring.so.0.2.0) +# ==5221== by 0x7EDC2B1: ??? (in /usr/lib/x86_64-linux-gnu/libgnome-keyring.so.0.2.0) +# ==5221== by 0x7EDB743: ??? (in /usr/lib/x86_64-linux-gnu/libgnome-keyring.so.0.2.0) +# ==5221== by 0x5063089: ??? (in /lib/x86_64-linux-gnu/libdbus-1.so.3.7.1) +# ==5221== by 0x5066152: dbus_connection_dispatch (in /lib/x86_64-linux-gnu/libdbus-1.so.3.7.1) +# ==5221== by 0x7EE6E44: ??? (in /usr/lib/x86_64-linux-gnu/libgnome-keyring.so.0.2.0) +# ==5221== by 0x5536204: g_main_context_dispatch (gmain.c:2539) +# ==5221== by 0x5536537: g_main_context_iterate.isra.23 (gmain.c:3146) +# ==5221== by 0x5536931: g_main_loop_run (gmain.c:3340) +# ==5221== by 0x4042F6: main (activesyncd-server.c:300) +# ==5221== +{ + gcrypt + activesyncd + Memcheck:Leak + fun:malloc + obj:*libgcrypt.so* + ... + obj:*libgnome-keyring.so* +} + + # ==10804== 60 (16 direct, 44 indirect) bytes in 1 blocks are definitely lost in loss record 1,121 of 2,014 # ==10804== at 0x4C27673: malloc (vg_replace_malloc.c:263) # ==10804== by 0x7FABC02: g_malloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3000.2) @@ -884,6 +910,42 @@ fun:start_thread } +# ==12501== 168 (64 direct, 104 indirect) bytes in 1 blocks are definitely lost in loss record 1,124 of 1,317 +# ==12501== at 0x4C272B8: calloc (vg_replace_malloc.c:566) +# ==12501== by 0x9736E38: g_malloc0 (gmem.c:189) +# ==12501== by 0x9211965: _g_socket_read_with_control_messages (gdbusprivate.c:182) +# ==12501== by 0x9211B54: _g_dbus_worker_do_read_unlocked (gdbusprivate.c:860) +# ==12501== by 0x9213AA1: _g_dbus_worker_do_read_cb (gdbusprivate.c:809) +# ==12501== by 0x91B53F6: g_simple_async_result_complete (gsimpleasyncresult.c:767) +# ==12501== by 0x91B54F8: complete_in_idle_cb (gsimpleasyncresult.c:779) +# ==12501== by 0x9731204: g_main_context_dispatch (gmain.c:2539) +# ==12501== by 0x9731537: g_main_context_iterate.isra.23 (gmain.c:3146) +# ==12501== by 0x9731931: g_main_loop_run (gmain.c:3340) +# ==12501== by 0x92117F5: gdbus_shared_thread_func (gdbusprivate.c:277) +# ==12501== by 0x9753DF4: g_thread_proxy (gthread.c:801) +# ==12501== by 0x7C08B4F: start_thread (pthread_create.c:304) +# ==12501== by 0xA2556DC: clone (clone.S:112) +# +# Also found elsewhere: +# https://mail.gnome.org/archives/commits-list/2011-November/msg05945.html +# +# Seen in syncevo-local-sync during TestLocalSync.testPassswordRequestTimeout +# and some other, non-D-Bus tests. Perhaps we fail to handle a message? +{ + GIO D-Bus leak in syncevo-local-sync + Memcheck:Leak + fun:calloc + fun:g_malloc0 + fun:_g_socket_read_with_control_messages + fun:_g_dbus_worker_do_read_unlocked + fun:_g_dbus_worker_do_read_cb + fun:g_simple_async_result_complete + fun:complete_in_idle_cb + ... + fun:g_thread_proxy + fun:start_thread +} + # ==24097== 596 (192 direct, 404 indirect) bytes in 1 blocks are definitely lost in loss record 1,605 of 1,743 # ==24097== at 0x4C260C6: calloc (vg_replace_malloc.c:566) # ==24097== by 0x8064763: ??? (in /lib/x86_64-linux-gnu/libdbus-1.so.3.5.8) @@ -1297,9 +1359,7 @@ Memcheck:Jump obj:* ... - fun:_dl_* - ... - fun:openaux + fun:_dl_start } # ==19740== 72 bytes in 1 blocks are possibly lost in loss record 1,226 of 1,985 diff --git a/test/test-dbus.py b/test/test-dbus.py index c38bf185..8578b26a 100755 --- a/test/test-dbus.py +++ b/test/test-dbus.py @@ -63,6 +63,18 @@ configName = "dbus_unittest" def usingValgrind(): return 'valgrind' in os.environ.get("TEST_DBUS_PREFIX", "") +def which(program): + '''find absolute path to program (simple file name, no path) in PATH env variable''' + def isExe(fpath): + return os.path.isfile(fpath) and os.access(fpath, os.X_OK) + + for path in os.environ["PATH"].split(os.pathsep): + exeFile = os.path.join(path, program) + if isExe(exeFile): + return os.path.abspath(exeFile) + + return None + def GrepNotifications(dbuslog): '''finds all Notify calls and returns their parameters as list of line lists''' return re.findall(r'^method call .* dest=.* .*interface=org.freedesktop.Notifications; member=Notify\n((?:^ .*\n)*)', @@ -1250,6 +1262,56 @@ class TestDBusServer(DBusUtil, unittest.TestCase): else: self.fail("no exception thrown") +class TestDBusServerStart(DBusUtil, unittest.TestCase): + def testAutoActivation(self): + '''TestDBusServerStart.testAutoActivation - check that activating syncevo-dbus-server via D-Bus daemon works''' + # Create a D-Bus service file for the syncevo-dbus-server that we + # are testing (= the one in PATH). + shutil.rmtree(xdg_root, True) + dirname = os.path.join(xdg_root, "share", "dbus-1", "services") + os.makedirs(dirname) + service = open(os.path.join(dirname, "org.syncevolution.service"), "w") + service.write('''[D-BUS Service] +Name=org.syncevolution +Exec=%s +''' % which('syncevo-dbus-server')) + service.close() + + # Now run a private D-Bus session in which dbus-send activates + # that syncevo-dbus-server. Uses a dbus-session.sh from the + # same dir as test-dbus.py itself. + env = copy.deepcopy(os.environ) + env['XDG_DATA_DIRS'] = os.path.abspath(os.path.join(xdg_root, "share")) + + # First run something which just starts the daemon. + dbus = subprocess.Popen((os.path.join(os.path.dirname(sys.argv[0]), 'dbus-session.sh'), + 'dbus-send', + '--print-reply', + '--dest=org.syncevolution', + '/', + 'org.freedesktop.DBus.Introspectable.Introspect'), + env=env, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT) + (out, err) = dbus.communicate() + self.assertEqual(0, dbus.returncode, + msg='introspection of syncevo-dbus-server failed:\n' + out) + + # Now try some real command. + dbus = subprocess.Popen((os.path.join(os.path.dirname(sys.argv[0]), 'dbus-session.sh'), + 'dbus-send', + '--print-reply', + '--dest=org.syncevolution', + '/org/syncevolution/Server', + 'org.syncevolution.Server.GetVersions'), + env=env, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT) + (out, err) = dbus.communicate() + self.assertEqual(0, dbus.returncode, + msg='GetVersions of syncevo-dbus-server failed:\n' + out) + + class TestDBusServerTerm(DBusUtil, unittest.TestCase): def setUp(self): self.setUpServer() @@ -1257,6 +1319,15 @@ class TestDBusServerTerm(DBusUtil, unittest.TestCase): def run(self, result): self.runTest(result, True, ["-d", "10"]) + def testSingleton(self): + """TestDBusServerTerm.testSingleton - a second instance of syncevo-dbus-server must terminate right away""" + dbus = subprocess.Popen([ 'syncevo-dbus-server' ], + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT) + (out, err) = dbus.communicate() + if not re.search(r'''\[ERROR.*already running\?\n''', out): + self.fail('second dbus server did not recognize already running server:\n%s' % out) + @timeout(100) def testNoTerm(self): """TestDBusServerTerm.testNoTerm - D-Bus server must stay around during calls""" diff --git a/test/testcases/eds_contact.vcf.exchange.tem.patch b/test/testcases/eds_contact.vcf.exchange.tem.patch index 4ebe1ab4..a3b55395 100644 --- a/test/testcases/eds_contact.vcf.exchange.tem.patch +++ b/test/testcases/eds_contact.vcf.exchange.tem.patch @@ -111,7 +111,7 @@ LABEL;TYPE=HOME:Address Line 1\nAddress Line 2\nAddress Line 3
UID:pas-id-43C15DFB000001AB
END:VCARD
-@@ -167,48 +156,34 @@ +@@ -167,48 +156,31 @@ X-EVOLUTION-ANNIVERSARY:2006-01-09
X-EVOLUTION-SPOUSE:Joan Doe
NOTE:This is a test case which uses almost all Evolution fields.
@@ -135,9 +135,8 @@ LABEL;TYPE=HOME:Test Drive 1\nTest Village\, Lower Test County\n12345\nTest
Box #1\nTestovia
-ADR:Test Box #3;;Test Drive 3;Test Megacity;Test County;12347;New Testonia
-+ADR;TYPE=OTHER:;;Test Drive 3;Test Megacity;Test County;12347;New Testonia
- LABEL;TYPE=OTHER:Test Drive 3\nTest Megacity\, Test County\n12347\nTest Box
- #3\nNew Testonia
+-LABEL;TYPE=OTHER:Test Drive 3\nTest Megacity\, Test County\n12347\nTest Box
+- #3\nNew Testonia
UID:pas-id-43C0ED3900000001
-EMAIL;TYPE=WORK;X-EVOLUTION-UI-SLOT=1:john.doe@work.com
-EMAIL;TYPE=HOME;X-EVOLUTION-UI-SLOT=2:john.doe@home.priv
@@ -170,13 +169,13 @@ +TEL;TYPE=HOME;TYPE=FAX:homefax 5
+TEL;TYPE=PAGER:pager 6
+TEL;TYPE=CAR:car 7
-+TEL;TYPE=RADIO:radio 8
++TEL;TYPE="X-EVOLUTION-RADIO":radio 8
+TEL;TYPE=work:business 9
+TEL;HOME:home 10
END:VCARD
BEGIN:VCARD
-@@ -221,8 +196,7 @@ +@@ -221,8 +193,7 @@ NICKNAME:user5
X-EVOLUTION-SPOUSE:
NOTE:image in JPG format
@@ -186,7 +185,7 @@ X-EVOLUTION-FILE-AS:JPG
X-EVOLUTION-BLOG-URL:
CALURI:
-@@ -257,8 +231,7 @@ +@@ -257,8 +228,7 @@ NICKNAME:user4
X-EVOLUTION-SPOUSE:
NOTE:image in PNG format
@@ -196,7 +195,7 @@ X-EVOLUTION-FILE-AS:PNG
X-EVOLUTION-BLOG-URL:
CALURI:
-@@ -288,8 +261,7 @@ +@@ -288,8 +258,7 @@ NICKNAME:user6
X-EVOLUTION-SPOUSE:
NOTE:The first name is "First \; special \;".
@@ -206,7 +205,7 @@ X-EVOLUTION-FILE-AS:Last\, First \; special \;
X-EVOLUTION-BLOG-URL:
CALURI:
-@@ -309,8 +281,7 @@ +@@ -309,8 +278,7 @@ NICKNAME:user3
X-EVOLUTION-SPOUSE:
NOTE:image in GIF format
@@ -216,7 +215,7 @@ X-EVOLUTION-FILE-AS:GIF
X-EVOLUTION-BLOG-URL:
CALURI:
-@@ -429,7 +400,6 @@ +@@ -429,7 +397,6 @@ & < & < & < & < & < & < & < & < & < & < & < & < & < & < & < & < & < & <
& < & < & < & < & < & < & < & < & < & < & < & < & < & < & < & < & < & <
& < & < & < & < & < & < & < & < & < & < & < & < & < & < & < & < & < & <
|