Overview ======== The code in this directory only gets compiled when SyncEvolution is configured with --enable-dbus-service-pim. It uses libfolks and the PBAP backend to implement a unified address book. This additional functionality is provided via the org._01.pim D-Bus API. That API is independent of libfolks and SyncEvolution. That it is implemented inside syncevo-dbus-server merely simplifies the implementation, by reusing some code provided by SyncEvolution and the core Server class: * C++ D-Bus bindings * logging * D-Bus server life cycle control: delay shut down while clients make calls, inhibit shutdown while clients are registered, restart when files change on disk during system update, ... * direct access to SyncEvolution instead having to go through the http://api.syncevolution.org D-Bus API Compilation =========== Use --enable-dbus-service-pim --enable-pbap --enable-ebook. --enable-ebook is already the default at the moment. It must not be disabled. The PBAP backend can be disabled. However, then fetching data from phones via PBAP obviously does not work. More information ================ Public discussion started here, comments can be sent via the gmane "reply" feature: http://comments.gmane.org/gmane.comp.mobile.syncevolution/4009 Issues specific to PIM Manager can be found in this graph: https://bugs.freedesktop.org/showdependencygraph.cgi?id=55916&display=web&rankdir=TB Dependencies ============ A fairly recent libfolks > 0.7.4 is required. The 837a88 commit is required and several other pending changes are recommended: https://bugzilla.gnome.org/show_bug.cgi?id=686693 writing birthday lacks conversion from UTC https://bugzilla.gnome.org/show_bug.cgi?id=685401 linking by email https://bugzilla.gnome.org/show_bug.cgi?id=686695 support nickname in add_persona_from_details Other requirements: * Evolution Data Server >= 3.6 * boost::locale and libphonenumber (when using the default sorting and searching) * glib >= 2.30 * Python >= 2.7 (only for testing) Known issues ============ None at the moment. Extending the implementation ============================ Sorting and searching can be replaced with a different implementation at compile time via --enable-dbus-service-pim[=]: it uses the file src/dbus/server/pim/locale-factory-.cpp to implement sorting and searching. Any additional library dependencies for that file need to be added to the main configure.ac. The default implementation is based on boost::locale and libphonenumber and is described below. Sorting ======= The sort order can be "first/last", "last/first", "full name". "first/last" sorts based on the first name stored in the "name" property, with the last name used to break ties between equal first names. "last/first" reverts that comparison. "full name" sorts based on the full name chosen for the contact if there is such a string, otherwise it uses the concatenation of the individual name componts without prefix (= "[] [] [] [])" as fallback. In other words, it sorts correctly when either all contacts have a full name explicitly set or the full names that were set following the same pattern as the fallback. Sorting is case-insensitive. The default is "last/first" if not set earlier. Searching ========= Supported searches: [ ] - An empty list matches all contacts. [ ['phone', ''] ] - Look up a valid phone number (= "caller ID"). The country code for the current locale is added if no country code was given in the number. Phone numbers in the unified address book must start with the resulting full number, after being normalized the same way. In other words: - Formatting does not matter. - Alpha characters are aliases for numbers on the keypad and match their corresponding number. - Additional digits in the address book are ignored, only the prefix must match (extensions may or may not be included in ). - Phone numbers in the address book which cannot be normalized cannot be matched. [ ['any-contains', '', ] ] - Sub-string search for in the following contact values: first, middle or last name, formatted name, nick name, phone number (sub-string search which ignores formatting and treats alpha characters as aliases, in contrast to prefix match in 'phone') or email address. Optional flags include: 'case-insensitive' (the default), 'case-sensitive'. Using a list will allow extending the search capabilities later, for example by allowing multiple terms which all must match and/or adding recursive queries like this: ['and', ['or', ['any-contains', 'joe'], ['any-contains', 'joan']], ['phone', '1234'] ] Lookup and search are different: the former is based on a valid number, the later on user input. A 'phone' lookup can compare normalized numbers including the country code, to ensure that the lookup is exact and does not mismatch numbers from different countries. Heuristics like suffix matching do not do this correctly in all cases. An 'any-contains' search is based on user input, which might contain just some digits in the middle of the phone number. The search ignores formatting in both input and address book. In both cases, alpha characters are treated as aliases for their corresponding digit on a keypad when matching phone numbers. In addition, an empty string as sort order picks a simple, ASCII-based "last/first" sorting. This is used for testing. A 'limit' search term with a number as parameter (formatted as string) can be added to a 'phone' or 'any-contains' search term to truncate the search results after a certain number of contacts. Example: Search([['any-contains', 'Joe'], ['limit', '10']]) => return the first 10 Joes. As with any other search, the resulting view will be updated if contact data changes. The limit must not be changed in a RefineSearch(). A 'limit' term may (but doesn't have to) be given. If it is given, its value must match the value set when creating the search. This limitation simplifies the implementation and its testing. The limitation could be removed if there is sufficient demand. Phone number lookup =================== A "phone" search must return results quickly (<30ms with 10000 contacts) under all circumstances, including the period of time where the unified address book is still getting assembled in memory. To achieve this, SyncEvolution searches directly in the active address books and presents these results until the ones from the unified address book are ready. A quiescence signal will be sent when: 1. results from EDS are complete before the ones from the unified address book 2. results from the unified address book are complete. The first signal will be skipped and the EDS results discarded if EDS turns out to be slower than the unified address book. Results from different EDS address books are not unified, for the sake of simplicity. They get sorted according to the sort order that was active when starting the search. Changing the sort order while the search runs will only affect the final results from the unified address book. Refining such a search is not supported because refining a phone number lookup is not useful. Peers ===== The following keys are supported for the configuration of a peer: - "protocol" - defines how to access the address book. "PBAB" (phone book access protocol) and "file" (read vCard files from directory) are implemented. "SyncML" and "CardDAV" could be added easily. - "transport" - defines how to establish the connection. The only supported value is "Bluetooth" (for protocol=PBAP or SyncML), which is also the default if not given explicitly. - "address" - the Bluetooth MAC address in the aa:bb:cc:dd:ee:ff format (for transport=Bluetooth). - "database" - empty or unset for the internal address book (protocol=PBAP), or the path to the directory (protocol=file). - "logdir" - a directory in which directories are created with debug information about sync session. - "maxsessions" - number of sessions that are allowed to exist after a sync (>= 0): 0 is special and means unlimited, 1 for just the latest, etc.; old sessions are pruned heuristically (for example, keep sessions where something changed instead of some where nothing changed), so there is no hard guarantee that the last n sessions are present. Not supported via the API at the moment: - selecting a specific phone address book - selecting which vCard properties get cached Syncing ======= SetSync() in SyncEvolution will return a dict with all of the following entries set: "modified": boolean - data was modified "added" : integer - number of new contacts "updated" : integer - number of updated contacts "removed" : integer - number of deleted contacts In other words, the caller can reliably detect when nothing changed, but when contacts were modified or added, it needs to read them to determine which kind of properties were modified or added. The SyncProgress is triggered by SyncEvolution with three different keys (in this order, with "modified" occuring zero or more times): "started" "modified"* "done" "started" and "done" send an empty data dictionary. "modified" sends the same dictionary as the one returned by SyncPeer(), if contact data was modified. So by definition, "modified" will be True in the dictionary, but is included anyway for the sake of consistency. Contact Data ============ A contact dictionary has the following key/value pairs. More properties may be added later. contact dictionary: "id" - string, see API description "source" - string, see API description "full-name" - string, formatted by the user or automatically (vCard FN) "nickname" - string "structured-name" - name dictionary (vCard N) "photo" - string, the URL (usually of a local file; EDS strips all inline photo data in vCards and puts them into local files, leading to URLs like: file:///home/user/.local/share/evolution/addressbook/system/photos/pas_id_5012983E0000065A_photo-file0.image%2Fjpeg) "birthday" - birthday tuple "emails" - value list with strings as value "phones" - value list with strings as value "urls" - value list with strings as value "notes" - list of strings; in practice only one entry is supported "addresses" - value list with an address dictionary as value "roles" - list of role dictionaries "groups" - list strings representing the names of groups the contact belongs to (CATEGORIES in vCard) "location" - a pair of doubles, representing latitude + longitude (in this order, see GEO in vCard); typically based on WGS84 name dictionary: "family" - string, the last name "given" - string, the fist name "additional" - string, middle names "prefixes" - string, name prefix like "Mr." "suffixes" - string, name suffix like "Sr." birthday tuple: (yy, mm, dd) - integer values for year, month, and day role dictionary: "organisation" - main organization ("Foo ACME") "title" - title inside that organization ("vice president") "role" - role as part of that origanization ("adviser") value list: [ (value, [parameter, ...]), (value, ...) ] value - depends on the property parameter - a string, with values again depending on the property; a value may have zero to n different parameters phone parameters: "voice" "fax" "car" "cell" "pager" "pref" "home" "work" "other" (might not be set explicitly) address parameters: "home" "work" "other" (might not be set explicitly) url parameters: "x-home-page" "x-blog" "x-free-busy" - public calendar free/busy URL "x-video" - video chat URL address dictionary: "po-box" - string, post office box "extension" - string, address extension "street" - string, street name "locality" - string, city name "region" - string, area name "postal-code" - string "country" - string Note: all of the strings and values above are defined by SyncEvolution in individual-traits.cpp, except for parameter strings. Those come directly from folks, more specifically AbstractFieldDetails: http://telepathy.freedesktop.org/doc/folks/vala/Folks.AbstractFieldDetails.html Here is an example contact dictionary, using Python syntax: { 'full-name': 'John Doe', 'nickname': 'Johnny', 'groups': ['friends', 'soccer'], 'location': (30.12, -130.34), 'structured-name': {'given': 'John', 'family': 'Doe'}, 'birthday': (2006, 1, 8), 'photo': 'file:///home/user/file.png', 'roles': [ { 'organisation': 'Test Inc.', 'role': 'professional test case', 'title': 'Senior Tester', }, ], 'source': [ ('test-dbus-foo', luids[0]) ], 'id': 'xyz', 'notes': [ 'This is a test case which uses almost all Evolution fields.', ], 'emails': [ ('john.doe@home.priv', ['home']), ('john.doe@other.world', ['other']), ('john.doe@work.com', ['work']), ('john.doe@yet.another.world', ['other']), ], 'phones': [ ('business 1', ['voice', 'work']), ('businessfax 4', ['fax', 'work']), ('car 7', ['car']), ('home 2', ['home', 'voice']), ('homefax 5', ['fax', 'home']), ('mobile 3', ['cell']), ('pager 6', ['pager']), ('primary 8', ['pref']), ], 'addresses': [ ({'country': 'New Testonia', 'locality': 'Test Megacity', 'po-box': 'Test Box #3', 'postal-code': '12347', 'region': 'Test County', 'street': 'Test Drive 3'}, []), ({'country': 'Old Testovia', 'locality': 'Test Town', 'po-box': 'Test Box #2', 'postal-code': '12346', 'region': 'Upper Test County', 'street': 'Test Drive 2'}, ['work']), ({'country': 'Testovia', 'locality': 'Test Village', 'po-box': 'Test Box #1', 'postal-code': '12345', 'region': 'Lower Test County', 'street': 'Test Drive 1'}, ['home']), ], 'urls': [ ('chat', ['x-video']), ('free/busy', ['x-free-busy']), ('http://john.doe.com', ['x-home-page']), ('web log', ['x-blog']), ], } Configuration and data handling =============================== The configuration of peers is mapped to normal SyncEvolution configurations. Every peer gets its own context, which contains both the local source definition for the peer and sync plus target configs. The special name space prefix "pim-manager-" used to identify them and avoid potential conflicts with normal SyncEvolution configurations. Look in ~/.config/syncevolution/pim-manager-* to find the configuration files. Local EDS databases are created with a fixed UID and name that also have that prefix. One could go one step further than it is currently done and set these databases to "disabled" in the ESourceRegistry, which would hide them in Evolution. The SyncEvolution command line can be used to view or manipulate these databases: https://syncevolution.org/wiki/item-operations The sort order and set of active address books are stored persistently in ~/.config/syncevolution/pim-manager.ini as: - "sort" = same value as in the API - "active" = space, comma or tab separated EDS source UUIDs Usage ===== The PIM manager is part of the syncevo-dbus-server. It can be started automatically via D-Bus or explicitly. When started automatically, it logs error messages to syslog. The PIM manager starts as idle and needs to be started via the D-Bus Start() API before it begins assembling the unified address book. SyncEvolution installs an XDG autostart file (/etc/xdg/autostart/syncevo-dbus-server.desktop) which, on systems supporting that mechanism, ensures that syncevo-dbus-server is started once. This is used for automatic syncing, which is not active when using only the PIM Manager. Therefore syncevo-dbus-server will terminate again after some idle period. It's recommended to disable this mechanism or patch syncevo-dbus-server.desktop to start the server with different options. In particular the "--start-pim" option may be useful to start up the UI and the server in parallel. Here is a list of supported options: Help Options: -h, --help Show help options Application Options: -d, --duration=seconds/'unlimited' Shut down automatically when idle for this duration (default 300 seconds) -v, --verbosity=level Choose amount of output, 0 = no output, 1 = errors, 2 = info, 3 = debug; default is 1. -o, --stdout Enable printing to stdout (result of operations) and stderr (errors/info/debug). -s, --no-syslog Disable printing to syslog. -p, --start-pim Activate the PIM Manager (= unified address book) immediately. Limitations =========== There are no hard-coded limits. In practice the number of contacts, parallel searches, peers, etc. is limited by CPU performance, amount of RAM and disk space. Functionality ============= Caching ------- Caching of contacts is done roughly like this: 1. The PBAP backend reads all contact data. See src/backends/pbap. 2. As part of a local sync (see main SyncEvolution README), a local-cache-slow sync is done. In that mode all local data is compared against the incoming data. Matches are found according to the properties that are defined as compare="always" or compare="slowsync" in src/syncevo/configs/datatypes/00vcard-fieldlist.xml. In the default file, these are the first, middle and last name and the organization. 3. Local entries which have a match are updated with data from that match, unmatched local entries are removed and unmatched incoming ones are created. This means that changing "less important" properties will be turned into local updates. This has the desirable effect of not changing the local ID of the existing contact, which might become important when attaching local meta data to that contact via its local ID (not supported at the moment; folks itself has a favorite flag which is stored like that). It also reduces disk IO. If matching fails, a contact will get deleted and recreated. The end result in the unified address book is still the same, because folks does not rely on the ID for linking. Supported fields ---------------- The PBAP backend always downloads the entire vCard. It supports restricting the vCard to specific properties, see src/backend/pbap/README. However, this is not used by the PIM Manager. In particular, PHOTO data is always included in the download right away. A mode where caching only works without PHOTO in the initial pass and then adds that in a second step could be added. Internally a vCard is represented as a Synthesis field list (again, see 00vcard-fieldlist.xml). X- extensions in the incoming vCard are preserved. Unified address book -------------------- The unified address book is assembled by folks based on properties that it considers "linkable". Linkable properties must be unique for a person. A phone number is not linkable, because different persons living at the same place might share a phone. For EDS, the linkable properties are defined in folk's backends/eds/lib/edsf-persona.vala and currently are: - instant messaging handles - email addresses - local IDs (not used in this context) - web service addresses (not used in this context) folks supports heuristics for identifying persons which are likely to be the same. This is not used by the PIM Manager. If it was, then links would have to be established based on the local IDs and keeping those stable became more important. Localization ------------ The usual environment variables are used to determine the locale: LC_CTYPE, LC_ALL, and LANG in that order (i.e. LC_CTYPE first and LANG last). There is no support for changing that once syncevo-dbus-server runs. boost::locale is used for collation while sorting contacts. The secondary collation level is used, which means that case, punctuation are ignored while accents are relevant. This is hard-coded in locale-factory-boost.cpp as DEFAULT_COLLATION_LEVEL. Searching is either done with a strict text comparison (case sensitive) or after using case folding (case insensitive, see http://www.boost.org/doc/libs/1_51_0/libs/locale/doc/html/glossary.html#term_case_folding). A 'phone' lookup uses libphonenumber to compare the free-form text field for telephone numbers against a caller ID. The text field must be in a format that libphonenumber understands, otherwise it will be ignored. A country code is optional. If missing, the country code of the current locale will be added before the comparison. Testing ======= Tests are mostly written in Python. See test/test-dbus.py (base classes and testing of the normal SyncEvolution D-Bus API) and src/dbus/server/pim/testpim.py. These operations manipulate the system address book. To avoid unexpected data loss when a developer runs the script in his normal environment, check that testpim.py gets run like this: XDG_CONFIG_HOME=`pwd`/temp-testpim/config \ XDG_DATA_HOME=`pwd`/temp-testpim/local/cache \ PATH=/bin:/libexec:$PATH \ /test/dbus-session.sh \ /src/dbus/server/pim/testpim.py This will use temp-testpim in addition to temp-testdbus in the current directory for temporary files. If TEST_DBUS_PBAP_PHONE is set to the Bluetooth MAC address (like A0:4E:04:1E:AD:30) of a phone, PBAP syncing with that phone is tested. The phone must be paired, connected and support SyncML in addition to PBAP. The test hard-codes Nokia SyncML settings to keep it simpe and because Nokia phones are most likely to be used.