Some of you guys might remember that, a long time ago, there was a driver for other os that allowed 3D stuff thanks to a nifty trick that gave access to the RSX in OtherOS. Since there are people working on ps3 emulators, and the RSX docs are very scarce, would there be a way to access the RSX extra memory? it is blocked by hypervisor, but i dont know in what way or how…
For those clueless, theyre the last 4MB of RSX memory in MMIO
“The hack consists of asking the Hypervisor to return without waiting for a blit to end. After the Hypervisor returns there is a small length of time during which the FIFO or FIFO registers can be modified before the GPU has finished reading the command. This will occur when a large blit is decomposed into many smaller 1024×1024 blits by the Hypervisor. The last operation pushed to the FIFO by the Hypervisor is a wait for the GPU engine to go idle. By skipping this operation, it is possible to enqueue more commands to the FIFO for the GPU to execute. So the hack consists in either patching the last operation with a NOP, or changing the FIFO write pointer to stop earlier.”
“Once arbitrary commands can be sent to the FIFO using the FIFO workaround, is it possible to enqueue a blit command from video RAM to video RAM. This command can be used to copy the end of video memory to lower regions of the video memory where it can be accessed directly from the Cell (direct access to memory above 254MB from the Cell is blocked by the Hypervisor). As this memory contains the RAMIN, RAMHT, RAMFC and GRAPH information, it can be analyzed to observe what the GPU-related Hypervisor calls do. Changing the direction of the blit (low video RAM to end of video RAM) allows a program outside of the Hypervisor to write to this reserved region (although the destination address must be 256-byte aligned due to restriction in the GPU blitter). To achieve a single unaligned write it is possible to read a large block, modify the desired value, and then write the entire block back.
However, the first parameter to the lv1_gpu_memory_allocate call sets a limit to the video RAM zone which can be DMA’d from/to. Setting this value to zero equals to ‘no limit’. This is probably due to a missing check in the Hypervisor, as the DMA objects size is specified as using a limit , equal to the size minus 1. Thus a size of zero is actually interpreted as a limit of 2^32-1 bytes (~4GB) by the GPU.
At first glance this would appear to be a security hole. But in actuality it is a feature Sony knows about and has used in official code: size = 0 is the default value in the Linux frame buffer driver, and were this issue properly resolved (0 = no RAM, no access) the official driver would be unable to work. ”