Ray Picking in OPhone 3D Development - Technical Article - OPhone SDN [OPhone Developer Network].

In the previous two articles introducing OPhone 3D development, we focused on the powerful 3D rendering API in the OPhone platform. In this article, we will demonstrate how to handle the interaction between 2D screen space and the 3D world in OPhone by writing a small program that picks 3D spatial primitives through touch on the screen, along with an introduction to bounding volumes, geometric intersection detection, and other related knowledge. The final effect of the program is shown in Figure 1, where the selected triangle is rendered in red translucent, and the deep red dot indicates the current screen touch position:

Figure 1 Effect of Triangle Picking Program

**What is Picking**#

In PC 3D games, we usually need to click on a model on the screen using the mouse. The screen where the mouse is located is a 2D plane space, while the game world is a standard three-dimensional space. This operation of selecting a primitive in 3D space based on 2D screen coordinates is called picking. On mobile platforms, touching the screen replaces mouse clicks, and triangles are the most basic rendering primitives. Therefore, how to accurately select a triangle of a model through touch on the screen is the topic discussed in this article.

**Picking Ray**

In the picking operation, the first step is to obtain the picking ray in 3D space based on the screen touch coordinates. The mathematical derivation of calculating the picking ray will not be elaborated here; readers can refer to relevant materials. Below, we will directly introduce how to obtain the picking ray.

Figure 2 Picking Principle Diagram

As shown in Figure 2, z = 0 is the near clipping plane of the view frustum, and z = 1 is the far clipping plane. Our picking ray is composed of the position P0 on the near clipping plane where the touch occurs and the position P1 on the far clipping plane, where P0 is the origin of the ray, and the ray is emitted from P0 towards P1. We will start from P0 and follow this picking ray to find the nearest intersecting triangle to P0 and render it on the screen. Therefore, our problem becomes how to obtain the 3D coordinates of P0 and P1.

Before obtaining P0 and P1, we first need to get the coordinates of the screen touch event. As mentioned in the previous two articles, after overriding the onTouchEvent() method of GLSurfaceView, we can listen for touch events and obtain the screen coordinates (ScreenX, ScreenY) where the event occurs. It is important to note that since the origin of the screen space is at the top left corner, while in OpenGL, the origin of the viewport coordinate system is at the bottom left corner, we need an additional operation to convert the screen coordinates to OpenGL viewport coordinates:

OpenGLX = ScreenX;

OpenGLY = ViewportHeight – ScreenY;

Where ViewportHeight refers to the height of the viewport.

After obtaining the viewport coordinates in OpenGL, the OPhone platform provides an auxiliary function GLU.gluUnproject() to convert 2D viewport coordinates into 3D space coordinates. The detailed parameters of this function are as follows:

public static int gluUnProject (float winX, float winY, float winZ, float[] model, int modelOffset, float[] project, int projectOffset, int[] view, int viewOffset, float[] obj, int objOffset)

As can be seen, it only requires passing in the viewport coordinates (winX, winY, winZ), as well as the current model view matrix model and projection matrix project, to obtain the coordinates in 3D space, and the calculation results will be stored in the obj array.

Where winX is ScreenX, winY is OpenGLY, and for winZ, when calculating P0, winZ is set to 0, and when calculating P1, winZ is set to 1. The 3D coordinates of P0 and P1 will be stored in the obj array.

After obtaining P0 and P1, we need to convert these two points into a picking ray. P0 serves as the origin of the ray, and the direction of the ray is obtained from the vector P1-P0. The relevant code is as follows:

- /**
- * Render the selected triangle
- * @param gl
- */
- private void drawPickedTriangle(GL10 gl) {
- if(!AppConfig.gbTrianglePicked) {
- return;
- }
- }

After going through the above operations, if we have obtained the triangle that intersects with the picking determination, we need to render it. In this example, we will render the returned triangle with a solid color filled in red translucent. It is important to note that the triangle data we obtained is also in the model coordinate system, and it needs to be transformed to the world coordinate system through the same model transformation as the model to ensure that the triangle's position matches that of the model. The relevant code is as follows:

- /**
- * Render the picked triangle
- * @param gl
- */
- private void drawPickedTriangle(GL10 gl) {
- if(!AppConfig.gbTrianglePicked) {
- return;
- }
- }

**Advanced Topics**

In the above discussion, the most time-consuming operation is undoubtedly the intersection detection between the ray and all triangles of the model. The models in this example have relatively few faces, so the cost is acceptable. However, if a model has thousands of faces, traversing and detecting will greatly consume CPU resources, especially in real-time operations, leading to extremely slow application responses, which is obviously unacceptable. A common solution to this problem is to establish a hierarchical bounding volume tree, such as Sphere-tree, AABB-tree, OBB-tree, kD-tree, etc. Due to the complexity of constructing and using hierarchical bounding volume trees, we will not discuss it further here. Readers can refer to relevant materials or discuss with me via email (xueyong@live.com).

**Conclusion**

During the development process, the interaction between the application and the user is a very important part, and in 3D, this interaction is relatively complex. This article introduces how to handle the interaction between 2D and 3D in the OPhone platform by analyzing and solving the problem of picking 3D spatial primitives on a mobile touch screen, as well as related knowledge such as bounding volumes, rays, and geometric intersection detection.

**Author Introduction**

Xue Yong, focused on the development of 3D applications on mobile platforms, familiar with various mobile 3D development platforms such as M3G, JSR 239, OpenGL ES (OPhone & iPhone), etc. Currently developing a complete set of 3D engine independently, including PC-side scene/model/animation/UI editor, 3ds max export plugin, aimed at Java and C++ clients. At the same time, he is working on a 3D shooting game, which will be released on multiple platforms including OPhone and iPhone.