They're paving the way for their revolutionary consumer AR eyewear (which may be delayed now apparently). Although maybe we'll see tech from some of their recent AR acquisitions (PrimeSense/Metaio) integrated into the upcoming iPhone, possibly in the form of an advanced depth/surface mapping sensor of sorts.
What's fascinating is Apple has a large team focusing on HCI & AR - what "pinch-to-zoom" and multitouch did for smartphones is what Apple will need to successfully pull off the "big bang" for consumer AR.
Rough bill of materials of Project Mirrorshades - Apple Iris. Note Iris is Siri backwards.
Kopin NED Acetate frame Polarisized or prescription lens with Zeiss smart optics Bone conduction modules Microphones (noise cancellation) Light sensor Accelerometer for step tracking and head movement app navigation Magnetometer for navigation Capacitive Pavel Ceramic battery Apple chipset Charging circuit BL5 Induction module
Crystal, champagne and black.
We only saw transparent reader lens but I understand they can procure polarised or prescription lenses
Cellulose acetate Injection mold frames into an aluminum mold. Colors are added and tumbled for finish.
P3 frame design
Two sizes, men's and woman's
Usually 4x the BOM cost; it would indicate around $600 mark.
Crystal is just clear. But clear without artefacts is difficult.
I've heard about this being real from someone who works in a very very large corp. They are working with Apple building the first apps for the device. He said it was not much better than Hololens at the moment.
Many were speculating that these glasses would be Apple's first major new product release at their new headquarters/"Spaceship", coinciding with the 10th anniversary iPhone launch. Robert Scoble has been mentioning the Zeiss partnership since CES, and recently someone from Foxconn/Hon Hai had allegedly leaked some details about a Google-Glass-like device with a near eye display in the bill of materials. Supposedly they won't be ready this year due to unique design challenges, but who knows for certain...
I work closely with people in the AR industry and everyone knows Apple has been investing heavily in developing this wearable device. Augmented reality is the next major fundamental platform shift, and Apple is hoping it will play the role the iPhone did as the catalyst for the smartphone/mobile era - but this time for consumer AR.
What Apple announced today is simply the beginning - they are providing tools so they can establish the foundation for "the largest AR platform in the world" (as mentioned at WWDC today).
Side note: There's a video from the 90's or late 80's showing an Apple concept for augmented reality glasses - it features a woman translating French or something while reading a restaurant menu. I can't seem to find it but it was on YouTube at some point...
I can't even imagine what the AR device should look like so that it would be "cool" to wear casually. Should it try to look like normal eyewear or be distinctly different? How to cope with indoor vs outdoor use (regular vs sunglasses)? How to deal with the issues Google hit (taking pictures, others feeling uncertain if you are with them or reading emails).
This also creates an opportunity for Apple. It's not just about the technology. You need to really figure out what customers will love and make the right decisions on what to include and what to leave out.
Precisely - it's a massive challenge they're undertaking. I think they'll omit cameras on such a device for this reason.
Smart glasses have never been "cool", but keep in mind that prior to the iPhone smartphones were generally uncommon for average users - unless you were a geek (I used to use a Palm Treo/PDAs in HS).
I think what they're aiming for will be something like the AirPods but intended to be worn all the time like Apple Watch. If AirPods were slightly smaller and had all-day battery life or wireless charging they could stay in your ears all day as a "wearable", like how many wear hearing aids today. AR Glasses would have to be lightweight and as un-intrusive as possible - they'd have to appeal not only to people who wear eyewear regularly but also to users who are not accustomed to doing so.
Perhaps they'll incorporate a retinal projector like Magic Leap or some other novel tech...
> I think they'll omit cameras on such a device for this reason
How would they track objects in 3D without a camera? Accelerometers are not accurate enough to to overlay-type augmentation they demoed. Unless you mean that the user won't be able to record pictures/videos so they won't call it a camera.
I mean that they would disallow saving photos/videos for privacy reasons, and could stand behind their hardware being secure enough that people don't have to worry if they're still being secretly filmed by hacked iRis's or whatever.
I must admit I was turned off by Google Glass as a consumer product because I don't want to be "that guy" suspected of recording what really should be off-the-record.
AR devices rely on cameras for surface mapping - which could be accomplished with some sort of depth sensor (using infrared like the Kinect or otherwise).
Not very much related, but you can also go wrong with IR. Something like 20 years ago Sony shipped night vision mode on camcorders. Turned out the functionality also allowed to see through clothes [1] which created a controversy.
Apple did definitely make the smartphone cool, but I'm dubious that they've made smartwatches cool. They're still very niche. I can see these glasses being the same, if not more so.
I don't think it's a niche. I see a lot of people wearing them. It's not a success on the level of the iPhone, but nothing is. If Apple Watch was a separate company it would be listed on the fortune 500 (according to Tim Cook on last earnings call)
The "Foxconn Insider" on Reddit claimed the original designs were using the P3 style frames[0] (in colors Clear, Champagne, and Black). In my humble opinion, this is a savvy choice – while maybe not as "iconic" as your average Wayfarer-style glasses, the P3 style speaks to many demographics. They are quite in style these days.
The same leaker said there were also plans to release new styles seasonally (presumably in order to stay in style) – which begs the question: would the projection apparatus be modular?
FWIW the "leaker" claimed the project was postponed indefinitely due to battery manufacturing feasibilities.
The frame they're using is very similar in shape to the Snap spectacles. I expect them to have a more classic design treatment than Snap's bold colours. The frame design has little accents at the upper-left most and upper-right most aspect of the frame, as you look at them (where the arms connect) which is one of the very few ways you can imagine integrating the two cameras which will be required for depth perception without bulkily clipping them on.
Studied the market for AR development solutions recently. A few thoughts:
- Probably using a lot of Metaio tech, a leading AR dev solutions provider Apple acquired/shuttered in May 2015
- Unfortunate naming. Sounds extremely similar to ARToolkit [1], the leading open source/free AR dev solution.
- Markerless is the future of AR, so good on Apple for getting this markerless tech out to compete with Google Tango and Microsoft Hololens. The status quo is currently marker-based recognition using "tags" or QR codes. It's also much harder than marker-based, or "on body" (Snapchat, Facebook AR Studio) recognition. I'd say Google Tango has some of the most impressive SDKs out there for tracking, area learning, and depth perception, but it does require specialized smartphone hardware from Lenovo or Asus, which significantly limits its utility and mkt penetration at this point. Google was just too early with Glass and got burned, but it needs a stronger hardware platform, which is something that Apple can deliver.
- The Unreal-engine AR video demo was cool and the graphics seemed decent for real-time rendering on an iPad, but the real future of AR applications (read: $$$) will come from industrial applications on wearable devices or head mounted displays such as the Hololens, Vuzix, Daqri, or Epson Moverio devices. Examples of industrial/enterprise AR applications: Remote help, complex assembly, pick & pack, line monitoring, materials handling, systems training, etc. This is the market Microsoft and PTC (Vuforia, ThingWorx) are targeting, not the consumer gaming/advertising markets, which use more "basic" forms of AR. This is not to say Pokemon Go/Snapchat do not generate a lot of revenue, but it's very debatable if they can be considered true AR applications.
- Other vendors in the AR development solutions space include: Aurasma (HPE), Blippar, Catchoom, EON Reality, Kudan, Pikkart, Wikitude, among others.
- In conclusion, Apple is currently stuck between a rock and a hard place. It's somewhat late to market with ARKit, will have to compete with Facebook, Google, and Snapchat for consumer AR-oriented developer mindshare, and will have no play at the industrial/enterprise market if it limits its tools to development for the iOS family.
- The best way for Apple to alleviate this jam is to release its own AR glasses/headset, which it is widely rumored to be developing.
> In conclusion, Apple is currently stuck between a rock and a hard place. It's somewhat late to market with ARKit, will have to compete with Facebook, Google, and Snapchat for consumer AR-oriented developer mindshare, and will have no play at the industrial/enterprise market if it limits its tools to development for the iOS family.
See also portable music, phones, tablets …
It's Apple's SOP, and if they've shown anything in the last 10-15 years, entering markets late is not a barrier to them competing
> In conclusion, Apple is . . . somewhat late to market . . . The best way for Apple to alleviate this jam is to release its own AR glasses/headset, which it is widely rumored to be developing.
I'm not sure this line of reasoning holds up. You mention 3 markets:
1. consumer
2. enterprise
3. the "market" for developer mindshare
Apple doesn't especially care about being early or late to any of these markets. Let's take them one by one.
1. There is currently no real consumer market for AR, so Apple is not late. And I don't see any reason why Apple would not be content to wait and see if one develops. It's not clear how releasing an improved version of Google Glass saves Apple from any kind of "jam". People are not clamoring for this type of product, because its value is highly unclear.
2. There is currently no real enterprise market for AR, so Apple is not late. Apple has no history of making industrial equipment. Are you saying Apple is about to take the unusual step of starting a whole new sales channel, targeting a whole new class of buyers, to try to get first-mover advantage in a new and unproven market helping warehouse workers do line monitoring and "remote help"? It seems unlikely.
3. Apple has a huge, strong developer base it is in no danger of losing. Again, it's not clear how offering an AR SDK gets them out of a jam.
If I had to guess as to why they are playing in this area, it may be to help seed the ground with more iOS devs familiar with this tech, so that in the event that the market does develop, they are in a good position to harvest ideas, talent, and apps.
1. Consumer AR market - mobile phone AR-enabled applications - education/gaming/entertainment/enhanced retail. Niantic, Blippar, Catchoom, Snapchat would all beg to differ that this market does not exist.
2. Enterprise/industrial AR market - wearable AR-enabled applications - I've already listed them. This market is very small currently...but it exists. I've talked to engineers and AR tool vendors in this market, creating applications for Vuzix, Epson devices. My argument is that this portion of the market will see a lot of growth over the next 5 years. Hard to explain Microsoft's play with Hololens, or PTC's ThingWorx Studio tracking tech away.
3. They are in a jam with AR application developers specifically, not all developers as a whole. You're right they have a large, strong mobile developer base, but AR developers will move away from mobile/tablet-focused platforms (iOS) to Android/Windows in order to develop for AR wearables.
They're playing in AR because they're looking for the new iPhone...a new form factor to deliver new applications and jump start sales. AR glasses could be that form factor if they can execute like Jobs did with the smartphone.
Apple is doing very well for these applications, so again they're not behind.
> Enterprise/industrial...This market is very small currently
Exactly.
> AR developers will move away from mobile/tablet-focused platforms (iOS) to Android/Windows in order to develop for AR wearables.
Why are you so sure they will leave mobile? Wearables have three benefits: 1) hands-free, 2) immersive, 3) Stereo vision. None of those features have shown to be required for AR killer apps. So far the leading apps (Pokemon Go, SnapChat, etc) have not needed them. Do you have some reason to speculate that those specific benefits 1-3 will grow in importance?
Hands-on/non-immersive is actually a feature in some ways, it allows the user more control over their degree of immersion. And head mounted overlay AR requires a massive increase in processing power in order to get acceptable latency. Latency doesn't matter as much for handheld passthrough AR. Lots of tradeoffs there, it's not clear to me HMDs are a win for consumer applications. Industrial; sure.
We've seen some of the stuff Microsoft talks about with hololens. The idea that it could be useful to some people in industrial/business settings.
What if instead of needing a
Hololens dev kit for $3k or waiting for a production version that will probably be at least $1k… you could get your application to work with the $350 iPad when the new OS comes out in September?
Sure Hololens Will be better at some things, but that's a hell of a price difference and it will be available retail soon. By the time your app is ready for your workers to use iOS 11 will probably be out.
Almost certainly - Metaio presented realtime lighting estimation at InsideAR (their dev conference) in late 2014. It's also been shown by Hans Kaufmann's research at TU Wien in Austria [1] although this has tighter links with other vendors than Metaio.
| Other vendors in the AR development solutions space include
I've had the pleasure of dealing with most of these folks within the last few years. They all have their own specialisms around certain tracking methodologies, pricing models and other features. Evaluate them with your own purposes in mind, and choose the right tool for the job.
| will have no play at the industrial/enterprise market if it limits its tools to development for the iOS family
In this market, Vuforia really does have it fairly cornered - it seems to be a chunk of what PTC are chasing, although they're far from turning a blind eye to B2C.
For AR, to win, you do not have to be the first one to market to win, its who can make the best platform for developers will win. As an AR developer, previous platforms like Vuforia, ARToolkit, and even Tango are either limiting or very hard to work with.
What? Vuforia is extremely easy to work with. Are you sure your difficulty issues weren't just Unity being a garbage heap of bad UI and poorly documented processes?
Lol, I mean building a native ios swift app with vuforia was a pain in the butt, since they have NO documentation on their code or any tutorials. Github link here: https://github.com/interactivetech/VuforiaScenekit
I have been avoiding building AR for Unity for the reasons you described xD!
If you can get someone to walk you through it a few times, point out the major parts, it's the easiest way I know of to get 3D apps up and running. And it's in very high demand.
The only problem is that I very much believe in the web application model as a means of reaching a wide audience with the fewest resources, but Unity's WebGL export is functionally useless.
That would of been great to have mentorship, but I ended up doing it the harder way, Struggling and reading code like a book. I learned a ton though, and its taken a while.I hope to soon provide people with tutorial/walkthrough with 3D apps.
You should take a look at AR.js, its a cool project that will hopefully provide that wider accessibility. The challenge with Apple's ARKit is that its AR technology( is much more challenging than Vuforia, and tricker to implement.
Currently locking the AR to the real world requires a QR code-like mark but they are demoing AR with no mark. Not having a mark requires 3D data of something in the field of view:
Thingworx 8.0 will release, and Thingworx Studio will be upgraded. The new version of Thingworx studio can recognize the Thing Object by Thing shape just like demo video, not only by QR Code-like mark.
As you say, the Wingnut demo had nice content, but the whole experience looked unbelievably bad. Like, "original Wii ads with people jumping behind their couches" bad. If they showed it as a tech demo for integrating VFX in real time preview when shooting a movie I would have bought it 100%.
The documentation says that it can do rough measurements and automatically scales things correctly. They pointed that out in the demo where the coffee cup was roughly the right size for a coffee cup instead of 5 feet tall or something like that.
There's also Sturfee which focuses on outdoor spaces which is a very hard problem for simple AR toolkits like this. Depth is hard and so most of these players cannot do outdoor AR.
sturfee.com
DISCLAIMER: I work at Sturfee and it's really great.
I thought we might be seeing an AR app built into the Camera or Map app.
Nope just a kit for developers to build games only or does the ARkit allow access to the camera app? I'm dying to see innovation from Apple like in the camera app when you open it there's an option to view how things look in the past(100 years ago what did this spot in New York City look like.. where you can take a pic of yourself in the past).
I'm sure there's oodles that can be done with an AR Camera.
Apple and the others to me seem to be slow to innovate.
What will happen is a developer will build the app I mentioned above (could exist already) and then Apple will later build it into the camera app. I've never seen an app maker build functionality into any of the iPhone built in apps.
I don't think Apple offers support to extend their camera app (there is support for extending the photos app, however).
What you _can_ do, (and it's quite easy), is to create your own app, capturing video from the camera. Then you can overlay any AR content you wish onto it.
Even ignoring that you'll need multiple photos to do the 3D reconstruction you need for augmented reality, that 100 year old photos may be lacking in resolution, and giving a few decades of leeway either way for that "100 years ago" I doubt even Google has enough data to have a reasonable chance of having a photo for random locations you might point your camera at.
Apple's Extensions operate over IPC. Trying to add overlays onto a live camera over IPC would probably introduce unacceptable lag. Especially if you also need to handle touch events too. And of course you wouldn't be able to actually do much besides have an overlay, e.g. transitioning to other screens, showing other UI controls etc. Plus this would significantly complicate the already-complicated Camera app interface.
And what would be the point? You can just launch a 3rd party AR app and it will use the camera APIs to show you the live camera feed, exactly like the built-in Camera app does.
As a developer, this was the most exciting thing for me from today's WWDC opening speech. Can't wait to try this out myself. The demo they did with the AR game was really neat.
I think there must have always been some way to do AR (Pokemon GO and other games) but I believe it always required a team to accomplish this and possibly work with engineers at Apple.
As an independent developer, I hope the AR kit will make it much easier for me to create something fun and possibly integrate with Siri and other kits. This is what I'm hoping for :D
I know VR is hot right now, but I've always had the thought AR > VR.
Wasn't Pokemon GO more like pseudo-AR? Like it just superimposed an animation with an alpha channel shadow but which otherwise had nothing to do with whatever the camera was looking at?
Yes. It's AR in that they overlaid something over the real world.
That's why the little bit during the demo was interesting. They showed Pokémon go using AR but it was actually doing it correctly, where the Pokémon stayed in position as they moved the camera around.
I was wondering what Pokémon Go would do on android. Will this feature be iOS only or will they implement their own version on android?
Yeah basically until now if you wanted to do AR you would need to role your own or use a third party library. Having ARKit as part of iOS will make building an AR app much easier.
I find it funny that Apple is just now releasing an AR SDK as when the iPhone first came out there were a lot of AR apps available. Most of them were pretty boring, just games like shooting spaceships or POI explorers, but they didn't look that much more primitive than the apps demoed today. I wonder why they didn't jump on AR back then.
I remember playing with those on my 3G or my 4. It was pretty impressive what people were able to do with those little phones.
I'm sure part of it is just the ability to do useful things, part of it may simply be that they have such had room on the chips that they can do it without draining battery life horribly. I remember those old apps clearly used a lot of processing power.
How does it compare to Google Tango? Tango needs special hardware, but ARKit is not. Even if it required, there would be a definitely device (iPhone or iPad). Traction wise Apple could win, but what about performance and accuracy wise?
For now it's more like how Snapchat's latest world filters work (try them out). Basic environment tracking, good enough for a smooth experience, considering you keep your camera steady.
Tango takes the tracking to the next level. Maybe future iPhones will have that kind of IMU and other sensors, allowing Apple to expand their AR API.
HealthKit is still going. They announced some big wins with HealthKit on the Apple Watch but they weren't included in the keynote. More than likely, there wasn't enough time considering the keynote without any HealthKit announcements was already longer than normal.
Thanks. Last month some medtech guy reported about different efforts and said HealthKit vanished. The fact Apple didn't mention made me think he was right. Now I'll wait for more.
Does anyone have resources for "State of the art" algorithms and research that is used to do the sensor fusion (camera frames + motion sensor data)? I found this but not sure its the most up to date. I'm also looking for more detail in terms of code:
(the last one being released just a few days ago...)
Of course there are a lot more resources out there and a lot of the cutting edge research is involved with deep learning techniques (starting with e.g. https://github.com/alexgkendall/caffe-posenet).
You may also want to watch out for this conference http://cvpr2017.thecvf.com/ scheduled for the end of this month and is usually a good source of cutting edge research...
I had a look at the documentation at [1], which says "The AVCaptureDepthDataOutput class captures and delivers depth data in a stream (similar to how the AVCaptureVideoDataOutput delivers video data)."
So you can now access a stream of depth data. I'd put money on ARToolkit using this stream when it's available. For now, it would only be from the iPhone 7+.
AR kit isn't limited to the 7+ ( they demoed it on an iPad) space so clearly that's not REQUIRED. I wonder if they're using the second camera for depth on the 7+ when it's available.
Perhaps that data isn't granular enough (or fast enough) to be useful when doing AR compared to the normal camera picture plus the motion sensor data.
Magic Leap is conspicuously but rightly missing from your list, because they're a fraud that's nowhere near releasing a real product, and whose bullshit is besmirching and dragging down the reputation of AR.
Google has at least three AR products: Glass is still in business for enterprise, Tango tablets and phones are finally out, and future versions of Daydream are supposed to be capable of inside-out tracking via an AR tracking system.
Microsoft has the HoloLens, which technology they have also cannibalized for a series of headsets they are dubbing "Mixed Reality" for the depth camera pass-through they are performing.
HTC is supposedly developing their own AR/inside-out system.
Leap Motion, Intel, Qualcomm, Meta, and Magic Leap are still considered in the running, with varying levels of believability.
And then there are a slew of different companies that are doing AR solutions with either the standard webcam (e.g. Facebook, Snap, Vuforia, Wikitude) or their own, standalone camera (e.g. Zed, Occipital).
I don't count Google. Google is an unreliable product company. They kill half their products and the other half are absolute tire fires. You may like android, but it's ugly AF compared to iOS which is far from perfect. Gmail needs some major work on the interface. Speaking of interfaces have you seen their tv app? Kill me now.
I have no love for Google. I certainly don't trust them enough to purchase a product they make for fear of them killing it. Plus they mine everything they can get access to for who knows what ends. "So for those reasons, I'm out".
MS.... Meh. No one has come to expect great things from them.
What's fascinating is Apple has a large team focusing on HCI & AR - what "pinch-to-zoom" and multitouch did for smartphones is what Apple will need to successfully pull off the "big bang" for consumer AR.