Beta version of Android app to connect as a device to calibre using wifi

Latest version: V1.2 14/August 07:00 BST

Latest changes to this post:
- 12/Aug 14:00 BST: updated connection FAQ entry and added debug log FAQ entry.
- 12/Aug 21:00 BST: updated problems section to remove problem with metadata refreshing after sending books, and to add section about content server IP address.
- 13/Aug 16:00 BST: updated problems section to remove portrait/landscape problem and to update the calibre server port problem description.
- 13/Aug 23:00 BST: added "What does this app really do" FAQ entry
- 14/Aug 07:00 BST: added "Kindle for Android" FAQ entry

==============================================

This thread discusses the beta release of the app discussed in this thread.

The app, callled Calibre Companion, is available on Google Play. It runs on Android Gingerbread (2.3) and later.

For various reasons we are choosing to charge for the beta product. The price in the US is $2.99, in the UK £1.99, and in euroland €2.49. The benefit to us is that we can easily and rapidly announce and distribute those updates over Google Play, and use Play's licensing. We won't be offended if you choose not to buy the product to be a tester. In addition, if before we go into "production" (in a week or two) you decide that the app doesn't work, or even that it doesn't work for you, we will refund your money. Steven will handle these requests and all others related to Google Play.

Please make comments or report bugs using the devices MobileRead forum (this forum) (a thread per comment would be good), via private messages to either of us (chaley or steve_haley), or via email to “calibre_companion [at] MultiPie dot co dot uk”. Be sure to type “MultiPie” (many pies) and not “multiple”. :)

Full disclosure: Kovid has started work in calibre to support MTP connection to devices. He isn't yet able to give a delivery date, but he is making progress. Although our app does more than calibre will do over an MTP connection, you might prefer to wait for whatever he produces. If so, again, we won't be offended.

Regarding problems in the beta version:

1) If you enter the calibre content server IP address and port into the app and then try to connect, the app will appear to hang. After several seconds the app will pop up a dialog asking if the right IP address & port were entered. See FAQ #1.

2) There is a bug in MacOS that causes intermittent problems with connections and sending books. Although the problem is in MacOS itself, a workaround for the problem will be in the next calibre release.

Thank you for your interest and support.

FAQ:

1. The app won’t connect to calibre!
   
   Spoiler:

   1. First, ensure that you have started the wireless device driver in calibre. Click on Connect/Share, then “Start wireless device connection”.
      Attachment 90526
   2. If the above isn't enough to get you going, then it is possible that your wireless router has disabled multicast or that your firewall does not handle dynamic port assignment. See the next item if your firewall is configured port-by-port. Regarding multicast: this feature is sometimes called multicast-DNS, zero-config, or bonjour. Some Cisco routers have this “feature”. It is impractical for us to provide detailed instructions for all the routers out there. Check the manual for your router to see how to enable multicast.
   3. If you can’t enable multicast, if your device and the computer running calibre are not on the same local network, or if you are running a firewall that does not handle dynamic port assignment, then you must provide calibre’s IP address to your phone.
      
      First, in calibre, go to the device customization panel for the Smartdevice driver (Preferences -> plugins -> Device interface plugins -> Smartdevice app interface). Check the box "Use a fixed network port". Change the default port number (9090) if you wish. You must not enter the same port used calibre's content server. Make a note of the port number that you use.
      Attachment 90593
      Next, start the wireless driver in calibre. You will then be able to see calibre’s IP address, as shown in the following:
      Attachment 90527
      Note the IP address.
      
      Now, enter the IP address on your phone. Click on the “Calibre’s IP address” menu item.
      Attachment 90528
      You will see the following dialog.
      Attachment 90529
      Enter the IP address you noted. Enter the port number you noted in device customization (above) in the port box. If your machine has a domain name, you can (at least in theory) enter that name instead of the IP address. You must always enter both an IP address/name and a port number.
      
      Last: be sure that the firewall you are running allows incoming connections to the chosen port number.
2. I have books on my device that don’t show up in calibre. How do I fix this?
   
   Spoiler:

   For the moment the app does not scan your device for books. The only way calibre knows about a book is if calibre sent it to the device. For several reasons related to how calibre gets metadata from “unknown” books, scanning would be very slow.
   
   We are considering adding a feature to the app that permits adding books on the device to the app’s database. The user would provide the correct title and author when the book is added, eliminating the need for calibre to scan the book’s internal metadata (the part that is so slow). We might be able to get the author/title information from the book automatically for some formats, in particular epub, but as book-contained metadata is often wrong we would still provide a way for the user to change this information.
   
   This feature will interact with extension (format) routing, so it won't happen before extension routing is implemented.
   
3. How do I get a debug log to submit with a problem report?
   
   Spoiler:

   You get a debug log by running calibre in debug mode. Start calibre, then click the small down arrow next to preferences as shown in this screenshot.
   Attachment 90594
   Click on "Start calibre in debug mode". Let calibre restart and answer OK to the dialog that confirms that calibre is in debug mode. Attempt to connect to calibre. Get to where you are having the problem, then close calibre. You will see a debug log in a new window. An example is
   

   Spoiler:

   SMART_DEV ( 0.00: 0.002) startup_on_demand try port 18700
   SMART_DEV ( 0.89: 0.803) startup_on_demand listening on port 18700
   Started up in 37.73 seconds with 66 books
   SMART_DEV ( 95.11: 12.261) reset
   SMART_DEV ( 95.12: 0.000) set_progress_reporter
   SMART_DEV ( 95.14: 0.000) open
   SMART_DEV ( 95.17: 0.016) _call_client GET_INITIALIZATION_INFO
   SMART_DEV ( 95.90: 0.718) open selected formats [u'epub', u'mobi']
   SMART_DEV ( 95.92: 0.000) card_prefix
   SMART_DEV ( 96.17: 0.218) set_progress_reporter
   Job: 1 Get device information started
   SMART_DEV ( 96.20: 0.000) get_device_information
   SMART_DEV ( 96.25: 0.016) _call_client GET_DEVICE_INFORMATION
   SMART_DEV ( 96.56: 0.281) _call_client SET_CALIBRE_DEVICE_INFO
   SMART_DEV ( 96.82: 0.249) card_prefix
   SMART_DEV ( 96.84: 0.000) free_space
   SMART_DEV ( 96.85: 0.000) _call_client FREE_SPACE
   DeviceJob: 1 Get device information done, calling callback
   DeviceJob: 1 Get device information callback returned
   SMART_DEV ( 97.18: 0.312) set_progress_reporter
   Job: 2 Get list of books on device started
   SMART_DEV ( 97.20: 0.000) books None
   SMART_DEV ( 97.21: 0.000) _call_client GET_BOOK_COUNT
   Job: 1 Get device information finished
   No details available.
   SMART_DEV ( 104.11: 6.879) books carda
   SMART_DEV ( 104.14: 0.000) books cardb
   DeviceJob: 2 Get list of books on device done, calling callback
   SMART_DEV ( 107.98: 3.822) set_plugboards
   READING LIST: Metadata available: {u'main': {u'device_store_uuid': u'efc3cde6-794f-4d23-bf8d-c7f95613e2b3', u'prefix': u'', u'device_name': u'SmartDevice', u'calibre_version': u'0.8.64', u'location_code': u'main', u'last_library_uuid': u'62e15c2b-a8ca-40fd-a784-05773082742a', u'date_last_connected': u'2012-08-12T12:17:32.261000+00:00'}}
   READING LIST: Sync Now - force_sync= False
   DeviceJob: 2 Get list of books on device callback returned
   Job: 2 Get list of books on device finished
   No details available.
   SMART_DEV ( 108.18: 0.187) set_progress_reporter
   Job: 3 Send metadata to device started

   
   
4. What does this really app do?
   
   Spoiler:

   Calibre currently supports devices in two ways. The first is by connecting the device to the computer with a USB cable. When a device is connected this way, calibre is in charge. One uses calibre to send books to the device, delete books from the device, etc. The "on device" checkmarks show in the library when using this method.
   
   The second way of connecting is by using calibre#s content server. When connecting this way, the device is in charge. The user taps/types on the device to navigate through calibre's web pages, getting books etc. There is no "on device" indication, either in calibre or on the device.
   
   One problem becoming more and more common with the first method, and a major driver for this app, is that newer Android devices cannot be connected to calibre by a cable. These newer versions of Android (versions 3 and up, Honeycomb, Ice Cream Sandwich, and Jellybean) do not support being connected as a "disk drive", something calibre requires to be able to "see" the device. The app we are developing gets around this problem by connecting to calibre wirelessly (wifi, no cable).
   
   The beta version will support:

   • Full support of calibre's device interface, including add and delete books, "on device", plugin notification, and automatic conversion.
   • Book display on the device grouped by title, author, tags, series, and publisher with first-letter navigation.
   • Book display grouped by custom fields.
   • Support for calibre's automatic metadata management facility. The app displays metadata supplied by calibre, not the metadata contained in the book's file.
   • One destination folder. Book format routing will come ASAP.
   • Launching of reader apps if they advertize their capabilities. It works with FBReader, Cool Reader, and Adobe reader. Kindle for Android does not play nice, so files to be read with it must be placed in the "kindle" folder. I haven't tried other reader apps.
   • Collections (similar to Sony collections, Nook shelves & Kindle collections). The collections will appear in the same way as grouping by tags/series etc. The collections will be managed by calibre similarly to calibre's collection management for Sony devices.
   • Simplified driver control in calibre. The driver can be started and stopped using the Connect/Share menu.
   • Optional password-protected access and rogue client flood-connection detection.
   • Manual IP setup in case the network/wifi router does not support finding calibre automatically or if the device and calibre are not on the same local network.
   • Support for Android 2.3 and up.

   What won't be in the initial beta:

   • Multiple folder format routing. As noted above, this is a priority.
   • Detailed/complete metadata display (book details).
   • The content server interface.
   • Some menus, such as Setup, won't be as pretty as we would like.
5. I am a Sony user and I want my collections!
   Spoiler:

   
   First, check if the grouping and sorting features available in the app don’t already provide an alternate way to get the same information. If they do not, then you can enable collections for the Smartdevice driver in the same way you did for the Sony driver. Go to the device preferences (in calibre, go to preferences -> plugins -> device plugins -> Smartdevice App Interface), and enter the lookup names for the columns for which you want collections. The Sony collections tweaks work here as well. Reconnect the device and your collections will appear under the group menu item “User Categories”.
   Attachment 90530
   
6. Why does the sort menu disappear when I select User Categories (collections)?
   
   Spoiler:

   Collections are sorted by calibre, and the app must respect that order. Although it would be possible to add yet another sort type to the list, we thought that the need would be too rare to compensate for the extra confusion about what it means.
   
7. How do I add my custom columns to the Group menu?
   
   Spoiler:

   Add the column’s lookup key (the word beginning with #, as in #genre) to the app’s list of custom columns to show. To do this, click the menu button on your device (usually three dots) and select Custom Categories.
   Attachment 90531
   Enter the names in the text box shown in the resulting dialog.
   Attachment 90532
   Note: it is not an error if a lookup name does not actually exist in any books on your device. This can happen if you connect to multiple libraries that have differing custom column setups.
8. How do I control which book extensions (“formats” in calibre-speak) are transferred when I use calibre’s send-to-device command?
   
   Spoiler:

   Unlike devices calibre supports by cable, you control in the app which formats are acceptable along with their priority on the device. This helps if you have multiple devices for which you want to use a different list or different priority.
   
   To set up the extensions, click the menu button on your device (usually three dots) and select Select extensions.
   Attachment 90533
   Next, scroll through the long list of extensions and check the ones you want calibre to be able to send to your device. Don’t try to order them at this point, it is easier to do in a second pass. In the screenshot below, epub and mobi were checked when the dialog opened, and I checked cbz.
   Attachment 90534
   After you have checked all the extensions you want to use, it is time to set their priority. This is easiest to do if you close and reopen the dialog, because all the checked extensions will be at the top. So press OK, then reopen the dialog. Now use the arrows to move the selected extensions up and down. The higher an extension, the higher its priority.
   Attachment 90535
9. When I first look at the extensions dialog, it contains only epub and mobi. Why?
   
   Spoiler:

   The app gets the complete list of extensions from calibre the first time it connects. If you want to add extensions, connect once to calibre then disconnect. The list of formats will be available.
   
10. Sending books seems slow
    
    Spoiler:

    We agree. There is some interaction between wifi and the protocol we have chosen to send the books. We have some ideas for increasing the performance, and will look at it when time permits.
    
11. What does the Delete Passwords menu item do?
    
    Spoiler:

    You can tell calibre to require a password from the device when the device connects. If you do this, then the app will ask you for that password and remember it. Selecting Delete passwords will force the app to forget all remembered passwords.
    
12. I see a scary warning message when I try to change the storage folder. What gives?
    
    Spoiler:

    For various reasons our app needs to reinitialize its database when you change the storage folder, removing all the metadata for the books you have sent. You need to send the books again for the app to have the metadata.
    
    Note that the books themselves are not deleted from the old folder. If for some reason the books are no longer in calibre, you can copy them from your device and re-add them to calibre. Once you are happy with the new folder you chose, you should delete the old copies of the books, and possibly the containing folder as well.
    
13. How do I get Kindle for Android to open my books?
    
    Spoiler:

    Kindle for Android is one of the very few reader apps that does not publish to Android the kinds of books that it can open. Because of this, Calibre Companion cannot automatically pass books to it.
    
    You can resolve this problem one of two ways:

    1. Use a different reader app that can open .mobi files. There are several. This will work only if your books are not infested with DRM.
    2. Set Calibre Companion's storage folder to Kindle for Amazon's books folder. Its name is "kindle". Its location depends on the version of Android you are running.
       Important: changing the storage folder will erase Calibre Companion's books database. You will need to send the books again. The books themselves are not deleted from the old location. You should delete those books from your phone once you are sure you have no further use for them.
       
       You change the storage folder using this menu.
       Attachment 90737
       You will see the following warning dialog telling you that continuing will remove the books from the app (erase its database).
       Attachment 90738
       Press the "Change" button if you wish to continue. You will see the following dialog.
       Attachment 90740
       Scroll the list of folders until you see the "kindle" folder. Tap that folder to enter into it, then tap the "This folder" button.
       
       After having changed the storage folder, to read the books you must manually start Kindle for Android. K4A will import the books you have sent to Calibre Companion so you can read them.

    This process will change somewhat once we have implemented book "format" routing. This feature will let you select a different storage folder per book type, which will help when you are using multiple reader apps.
    
14. Why is there cat hair in some of the dialogs?
    
    Spoiler:

    Charles’ cat Lizzie insists on sitting on his keyboard. Some fur she sheds is molecularly transmitted through the keys into the app’s code.
    Attachment 90536
    

Attached Thumbnails