In Portal we use (abuse) blurhashes extensively on the client. We Use them for many UI flourishes, including:
Placeholders whilst full-res images are loading
transitions between images and solid backgrounds
shadow/glow colour tinting on hover
We’ve found them to be a great tool to improve the App experience but we’ve never been 100% happy with the appearance of the hashes for some of our images, we often find they have large dark areas or undesirable artifacts. We’d been discussing finding a better solution internally for a while so this week I was excited to stumble accross the thumbhash alogithm.
We use Elixir on the backend so I got to work using Rustler to bridge from Elixir land to the Rust implmentation. It was also a great excuse to play with Vix / libvips too. The code is open source and available on GitHub but not it’s not in the hex package index yet.
You can instead add the thumb_hash package like this:
def deps do
[
{:thumb_hash, gitub: "portal-labs/thumb_hash"}
]
end
We are running it in beta currently but performance seems to be pretty good so far. 🤞
Next up we had to add support for rendering them on the client, this was super easy as there is already a Swift implmentation of the algorithm available that supports both AppKit and UIKit. After a little wrapping and refactoring we were all switched over.
Since making the switch from Blurhashes to Thumbhashes the difference is quite noticable, gradients are smoother, loading states are clearer and the harsh dark blobs are much reduced. So far we are very happy with this change.
I’ve been trying to hunt down a bug with audio playback stuttering on iOS 16.3, after 3 days I think I may have tracked it down to App Intents of all things.
Portal 3.7 switched over to the new App Intents framework when running on iOS 16. This allows for pre-prepared Siri Shortcuts which are a really nice addition. In order for Siri to create these Shortcuts and keep them in sync you need to call updateAppShortcutParameters whenever your list of AppEntity records changes.
We call updateAppShortcutParameters whenever a customer adds or removes a portal. Unfortunately the Apple docs contain no info at all about this functions behaviour so who knows what is supposed to happen. But from what I can see the call returns immediately to the callee, then, still inside your App process, it loops through and injests all your App Entities. This includes re-cloning the images provided for every AppEntity you have. If you have a collection of 20 Entities all with images this process seems to take as long as 20 seconds to complete in my testing.
So now we have an issue where there is a bunch of background work going on in our process that can run for a while and we have no idea when it’s complete. The side-effect of this is that in certain situations we end up calling updateAppShortcutParameters too regularly and these background tasks compound causing jitter elsewhere in the App.
I don’thave a great solution to this but currently we have settled on limiting calls to updateAppShortcutParameters to only cases where we know data has 100% changed and throttling calls to no more than once a minute to try and avoid tasks stacking up. Ideally we’d love it if Apple could make updateAppShortcutParameters an async call that could block until completion, that way we could manage scheduling and queuing ourselves.
The fix should be available in Portal 3.7.4 and I’m really hoping this fixes the audio issues for those that have reported it.
This week I’ve been looking at adding Apple Music integration into Portal. MusicKit is pretty awesome but whilst it’s available on macOS for exploring the library content the music playback APIs are not (not yet at least crosses fingers).
After some investigations I found that there exists a MusicKit JS it’s currently listed as in beta but seems quite complete on the playback side. So the plan was hatched, could we lean on the JS API for music playback when running on Mac.
The process looks something like this:
call MusicAuthorization.request() from Swift to request access.
call DefaultMusicTokenProvider().developerToken() from Swift to get an API token
call DefaultMusicTokenProvider().userToken(for: developerToken, options: [.ignoreCache]) from Swift to get an API token as the signed in user, relying on their Mac for auth.
create a WKWebView(frame: CGRect.zero) to act as a headless browser.
Inject a page into the webview that loads musickit.js and the above tokens.
Configure the player in JS land, avoiding calling music.authorize() as the docs suggest and instead using the injected credentials. Something like the below.
document.addEventListener('musickitloaded', function() {
MusicKit.configure({
developerToken,
app: {
name: 'Portal',
build: '1.0'
}
});
let music = MusicKit.getInstance();
music.musicUserToken = userToken;
window.music = music
});
With this all in place we can now control playback from the Swift side via, e.g. webView.evaluateJavaScript("window.music.pause()")
So far this has actually worked a lot better than I was expecting and might actually be workable as a solution until such time as Apple ship MPMusicPlayerController support on macOS.
Something fishy is going on with AirPod Pro and AirPod Max, we have recently been getting a bunch of reports from customers of “Headphones Not Supported” messages showing up in our App. On closer investigation, this message shows up when your headphones don’t send motion data (admittedly the messaging on our part could be clearer here).
After a lot of trial and error I managed to get my AirPod Max into this state where they no longer report any motion data to the connected device. This is not Portal specific as it affects Apple Music, Apple TV etc. The only way I have been able to fix it is by returning the AirPods to their case for 10 seconds or so and then reconnecting. I’m assuming this somehow resets the sate?
We are releasing and update this week to improve the messaging in App and help people get back on track but it’s not ideal.
After a lot of team discussions pre Christmas we made the decision that Portal will need some form of sync and accounts functionality and that we cannot rely on iCloud. The reasons for this are interesting in themselves but for right now I am focussing on our options for client data syncing.
The Requirements
Customer Opt-in: Accounts must be optional and clients should be able to store all data locally without the need for an account.
Offline First: Everything should be fully functional offline for extended periods.
Reliable: When online and signed in clients should sync up to a consistent and understandable state.
I am genuinely surprised how little discussion there is online for App data syncing outside iCloud. Most of the public discussion focusses around:
Firebase: We moved away from Firebase services a few years ago and have no plans to go back, mainly based on privacy concerns.
Roam (MongoDB): We’d rather avoid using a 3rd party hosted service, their granular pricing makes it hard to comprehend how much we would be paying but I think it would be a sizeable bill.
Replicache: JS only on the client side, doesn’t have a good story for initial local usage without an account and the pricing model would be unsustainable for us.
Because of this I think we need to look into options for handling sync ourselves. We maintain our own backend services for our library content, adding sync might be a natural next step.