Leading Effort to Hack PSVR 2 into Playing PC VR Games Unlocks 6DOF Cameras, Begins Work on Controllers

TL;DR: Very interesting progress, thanks to Sony not even trying to lock it down, the PSVR 2 being rather standard compliant and doing a lot of tracking onboard. This should make it possible to use the PSVR 2 as a 6DoF HMD similar to a Quest 2 or Rift, but the features that make the PSVR 2 special like haptics, HDR and eye tracking are still far off and will be much harder to utilize. And ETFR with huge performance gains is still very unlikely to ever become usable despite the sensor data being readable, as the matching tracking software on the PC side is missing.

Not to diminish iVRy’s efforts and progress, but they aren’t “hacking” anything. They are finding out how the PSVR 2 talks to the P5 by listening to the data steams and sending test data to the HMD, basically re-engineering the protocols. So far there is no need to hack anything, because Sony didn’t encrypt anything, they just didn’t publish the documentation.

To quote myself from the discussion in January, when iVRy still dampened hope the PSVR 2 could ever be made to work with a PC:

In the best case scenario where the basic tracking is done on a SoC on the PSVR 2, sending back just the readable results via USB-2, and Sony hasn’t scrambled/encrypted the protocol, this could make it rather easy to utilize at least the basic 6DoF functionality.

That’s pretty much what turned out to be true, with the main change being that the PSVR 2 uses USB-C VirtualLink Alt mode instead of USB-C DisplayPort Alt mode, adding some power and allowing to send back data at USB3 instead of USB 2 speed, and requiring extra hardware adapters to connect it to a PC. Basically the PSVR 2 does a lot of the tracking onboard and then sends the positional/rotational data back to the PS5 for the next frame to be rendered. iVRy now managed to find and interpret the tracking data, so it can be send to e.g. an OpenXR stack.

Analyzing protocols is a PITA and at the high speeds of USB-C requires expensive equipment, but the result is still using what was already openly there, not getting/hacking it to do something new. So far pretty much everything speculated regarding PSVR 2 standard compliance, protocols used and Sony not bothering to encrypt it or locking away the sensors turned out to be true, it was just hard to verify and understand how exactly it works.

We are still seeing pretty much the best case, and the hard part has yet to come. Quoting myself again:

But even if Sony encrypts nothing, you can access all the sensors over USB and Sony even publishes the full specs and all the needed documentation free for everyone, you still don’t have anything that could do inside-out, controller, hand or eye tracking on a PC. It would be like a HTC Vive Pro Eye without the Tobii eye tracking software and without the HTC software stack that makes it OpenVR/OpenXR compliant. Some kind of controller tracking shouldn’t be too hard to implement for a third party, but WMR has taught us that some tracking doesn’t equal usable tracking, and everything else will be much more complicated. […] Your chances to find any type of usable eye tracking that allows to use a HMD like the PSVR 2 as an OpenXR device, or something that supports the haptics are pretty much zero, because this stuff is really hard.

The (somewhat expected) good news is that the PSVR 2 SoC already does the inside-out tracking for the HMD. Expected, because this is done even on the Quest 1 by the DSP section of the SoC that takes up ~10% of the silicon, to unburden CPU/GPU and make them fully available for OS and apps. I’d fully expect the PSVR 2 SoC to also do all of the controller tracking they are currently trying to figure out, because doing this lightweight tasks on the HMD make a lot more sense then sending back multiple video streams to the PS5 for analysis. Once this is figured out, the PSVR 2 can be used as a 2000*2040/eye 6DoF HMD similar to a Rift or Quest connected to a PC, which will be a big step.

Still lacking would be a way to control the haptics, and if this is figured out, a way to make them usable in existing games/apps. Given that nothing so far has been actively hidden, I’d expect these to be readable/writable too, with the main problem to identify the right parts and then translate it. The PSVR Sense controllers use LRAs/VCAs, basically low frequency speakers that move a mass instead of a membrane in one dimension, while current controllers use ERMs, small motors rotating a mass in two dimensions. LRAs expect a signal similar to a sound wave, you’d have to create a translation layer, and no apps would use the haptics in the HMD itself because no other HMDs have offered these.

The biggest problem is what people want most: eye tracking and foveated rendering. From all we know this is very hard and compute intensive, so it is very likely that the PSVR 2 itself at most calculates pupil position and defers the motion estimation needed for ETFR to the PS5. This would have to be completely reimplemented. And considering that Sony gave up on their inhouse development and instead partnered with Tobii for integration and licensing the eye tracking software they have been working on for 20 years, this won’t be easy. If all you want is the gaze direction, just accessing the eye tracking cameras and some minor image processing will get you there. You can add that level of eye tracking with the open source EyeTrackVR and about USD 25 in cheap SoCs and cameras from AliExpress, allowing your VRChat avatar to look someone in the eye.

Again, it is great that iVRy stuck to the project, did all the work, and (thanks to Sony not bothering with locking anything) is already very far on the way to make the PSVR 2 a usable 6DoF PCVR HMD, at roughly 167% the price of a Quest 2 with a slightly higher resolution. That may make it attractive to those that don’t want to buy two HMDs or already have a PSVR 2, but it still would make little sense to buy a PSVR 2 just for PCVR. And it will be very hard to get there, with there simply being no software that currently could utilize haptics or eye tracking. And even functionality like HDR that would work out of the box due to the HMD just identifying itself as a display using DP1.4 with DSC to squeeze high pixel and color counts through USB-C will remain unused, unless someone hacks pseudo-HDR into ReShade or other HMDs and software stacks will start to support it too. ETFR and its matching performance gains will be almost impossible to get due to lack of freely available tracking/motion estimation software, even if the feature would be basically supported by OpenXR.

Remember that the PSVR 1 support by iVRy also basically only allowed using the easily accessible 3DoF tracking data from the IMU, everything else like controller or 6DoF tracking had to be reimplemented, causing the setup to be, lets say, challenging, and till today only allowing for sort of 5DoF, tracking up and down and sideways head movements, but not forward and backwards. And even that took a very long time to implement. The work iVRy is doing is great and important, and the fact that they got access to the raw stereo camera feed is tempting even me go get a PSVR 2, as this is prohibited on pretty much all other HMDs for data privacy reasons, making AR experiments impossible. But do not expect the PSVR 2 to (ever) provide all the advanced features it offers when connected to a PS5 to PCVR users too, or for it to ever become plug’n’play without active support from Sony.

But why would anyone want this? I bought the PSVR2 for the games. There are already so many choices for hardware on PC, ranging from the smallest (form factor) to the widest (FOV) to the highest (resolution). The most interesting features of PSVR2 (haptics and HDR) are not going to work without game support anyway. If you ask me, the best all-around PCVR headset is the Quest Pro.

Not bad for a team that at the start of the project said they would have needed years to make it work…

Totally a waste of time but they are definitely cashing in on the supposed attempt. That Patreon must be raking in the money.

Multiple headsets big and small, expensive and cheap on PC already. But some gamers want Sony’s headset because it’s the only high end headset PC gamers don’t have access to. Pure entitlement. Want to use it, buy a PS5.

Buying the device to use it on PC doesn’t help Sony at all. Only companies like Valve. If gamers don’t buy VR games on PS5, Sony won’t be making money and get a return on their investment on the headset. No profit means no higher quality games created by them or 3rd parties. If it’s successful, some of these non exclusive games will get ported cross platform. No PS VR 2, no higher fidelity games for PC. You’ll be killing the very platform you’re trying to support when it’s already supported with dozens of VR headsets. Including wireless. And killing console VR in the process.

Then, PC gamers will be stuck back to the same Quest ports as hand me downs. Where the games take no advantage of PC power. And still waiting for modders like pray dog to push Unreal Engine support.

Use the device for what it was intended for. Sony spent the money making an affordable headset for console. But PC gamers want to ruin that by using it on PC instead. Is the Index, Pimax, Reverb, Quest and Pico not good enough?