Square Aspect ratio or Can we get the source code to make changes ourselves?

[pili-engineering]

Square Aspect ratio or Can we get the source code to make changes ourselves?

[jpxiong]

1:1 aspect ratio maybe stretch or compress the camera preview. Is the following snippet what you expected:

    <com.pili.pldroid.streaming.widget.AspectFrameLayout
        android:id="@+id/cameraPreview_afl"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_centerInParent="true" >

        <android.opengl.GLSurfaceView
            android:id="@+id/cameraPreview_surfaceView"
            android:layout_width="200dp"
            android:layout_height="200dp"
            android:layout_gravity="center" />
    </com.pili.pldroid.streaming.widget.AspectFrameLayout>

[gouravd]

The scaling and the cropping has to be handled by your code (may be the
same way you handle 4:3 ratio) else the output will be stretched or
compressed.

I am currently using open source Kickflip sdk and I modified certain part
of their code via matrix transformation for proper square aspect ratio.
On Aug 23, 2015 7:45 AM, "jpxiong" [email protected] wrote:

1:1 aspect ratio maybe stretch or compress the camera preview. Is the
following snippet what you expected:

<com.pili.pldroid.streaming.widget.AspectFrameLayout
    android:id="@+id/cameraPreview_afl"
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:layout_centerInParent="true" >

    <android.opengl.GLSurfaceView
        android:id="@+id/cameraPreview_surfaceView"
        android:layout_width="200dp"
        android:layout_height="200dp"
        android:layout_gravity="center" />
</com.pili.pldroid.streaming.widget.AspectFrameLayout>

—
Reply to this email directly or view it on GitHub
#16 (comment)
.

[jpxiong]

OK. You just need the 1:1 (Square) aspect ratio for the layout , but not the GLSurfaceView. Right ?

[gouravd]

My requirement is that the output video should be Square format and the preview should be square as well. So I guess it has to be implemented in your code to properly handle 1:1 aspect and exposed via your API, something like CameraStreamingSetting.PREVIEW_SIZE_RATIO.RATIO_SQUARE

In Kickflip, they have a FullScreenRect.java (https://github.com/Kickflip/kickflip-android-sdk/blob/master/sdk/src/main/java/io/kickflip/sdk/av/FullFrameRect.java) which accepts a TextTure2DProgram as constructor argument. They have a function called adjustForVerticalVideo() which is called first time camera and preview is ready and called subsequently to adjust ratios when the camera is rotated (from portrait to landscape etc..)

I modified that part of code as follows, for proper scaling into square aspect ratio. The SurfaceVIew in my code resides in a square layout and hence the preview and output are both square.

Matrix.setIdentityM(IDENTITY_MATRIX, 0);
float scaleval = isWideScreen ? 1.78f : 1.33f; //if(16:9) then 1.78 scale factor, else if 4:3 then 1.33 scale factor
switch (orientation) {
case VERTICAL:
Matrix.scaleM(IDENTITY_MATRIX, 0, 1f, 1f, 1f);
case LANDSCAPE:
Matrix.scaleM(IDENTITY_MATRIX, 0, 1f, scaleval, 1f);
break;
case UPSIDEDOWN_VERTICAL:
Matrix.scaleM(IDENTITY_MATRIX, 0, 1f, scaleval, 1f);
break;
case UPSIDEDOWN_LANDSCAPE:
Matrix.scaleM(IDENTITY_MATRIX, 0, 1f, scaleval, 1f);
break;
}

[jpxiong]

Thanks for your sharing. It's very helpful to me.

[jpxiong]

It seems adjustForVerticalVideo can't work properly .

After i invoked the adjustForVerticalVideo, the preview didn't any change. What did i miss?

public void adjustForVerticalVideo(SCREEN_ROTATION orientation, boolean isWideScreen) {
        synchronized (mDrawLock) {
            Log.i("FullFrameRect", "adjustForVerticalVideo orientation:" + orientation + ",isWideScreen:" + isWideScreen);
            mCorrectVerticalVideo = true;
            requestedOrientation = orientation;
            Matrix.setIdentityM(IDENTITY_MATRIX, 0);
            float scaleval = isWideScreen ? 1.78f : 1.33f; //if(16:9) then 1.78 scale factor, else if 4:3 then 1.33 scale factor
            switch (orientation) {
                case VERTICAL:
                    Matrix.scaleM(IDENTITY_MATRIX, 0, 1f, 1f, 1f);
                case LANDSCAPE:
                    Matrix.scaleM(IDENTITY_MATRIX, 0, 1f, scaleval, 1f);
                    break;
                case UPSIDEDOWN_VERTICAL:
                    Matrix.scaleM(IDENTITY_MATRIX, 0, 1f, scaleval, 1f);
                    break;
                case UPSIDEDOWN_LANDSCAPE:
                    Matrix.scaleM(IDENTITY_MATRIX, 0, 1f, scaleval, 1f);
                    break;
            }
        }
    }

[gouravd]

I remember that I had to change a lot of other stuff as well along with this piece of code. I had to change the section which calls adjustForverticalVideo and the section that handles phone rotation among other things..I also had to comment out a portion of code in the same file fullframerect.java, which

P.S: I am still trying to figure out how to go to a specific time in my commit to retrieve the entire source code tree and then I would be able to tell you what changes I did exactly. Source code is hosted in Microsoft and I guess there is no direct way to browse the entire tree at certain commit. I guess I would have to clone a specific commit to my machine.

[gouravd]

This is how my fullframerect.java looks. Please note that //Matrix.scaleM(texMatrix, 0, 0.316f, 1.0f, 1f); is commented out in drawFrame()

public class FullFrameRect {
public static enum SCREEN_ROTATION {LANDSCAPE, VERTICAL, UPSIDEDOWN_LANDSCAPE, UPSIDEDOWN_VERTICAL}

private final Drawable2d mRectDrawable = new Drawable2d(Drawable2d.Prefab.FULL_RECTANGLE);
private Texture2dProgram mProgram;
private final Object mDrawLock = new Object();

private static final int SIZEOF_FLOAT = 4;

private float[] IDENTITY_MATRIX = new float[16];

private static final float TEX_COORDS[] = {
        0.0f, 0.0f,     // 0 bottom left
        1.0f, 0.0f,     // 1 bottom right
        0.0f, 1.0f,     // 2 top left
        1.0f, 1.0f      // 3 top right
};
private static final FloatBuffer TEX_COORDS_BUF = GlUtil.createFloatBuffer(TEX_COORDS);
private static final int TEX_COORDS_STRIDE = 2 * SIZEOF_FLOAT;

private boolean mCorrectHorizontalVideo = false;
private boolean mScaleToFit;
private SCREEN_ROTATION requestedOrientation = SCREEN_ROTATION.VERTICAL;


/**
 * Prepares the object.
 *
 * @param program The program to use.  FullFrameRect takes ownership, and will release
 *                the program when no longer needed.
 */
public FullFrameRect(Texture2dProgram program) {
    mProgram = program;
    Matrix.setIdentityM(IDENTITY_MATRIX, 0);
}

/**
 * Adjust the MVP Matrix to rotate and crop the texture
 * to make Horizontal video appear upright
 */
public void fixAspectRatio(SCREEN_ROTATION orientation, boolean scaleToFit, boolean isWideScreen) {
    synchronized (mDrawLock) {
        mCorrectHorizontalVideo = true;
        mScaleToFit = scaleToFit;
        requestedOrientation = orientation;
        Matrix.setIdentityM(IDENTITY_MATRIX, 0);
        float scaleval = isWideScreen ? 1.78f : 1.33f;
        switch (orientation) {
            case VERTICAL:
                if(scaleToFit)
                {
                    //Matrix.rotateM(IDENTITY_MATRIX, 0, -90, 0f, 0f, 1f);
                    Matrix.scaleM(IDENTITY_MATRIX, 0, 1f, 1f, 1f);
                }
                else {
                    Matrix.scaleM(IDENTITY_MATRIX, 0, 1f, 1f, 1f);
                }
            case LANDSCAPE:
                if (scaleToFit) {
                    //Matrix.rotateM(IDENTITY_MATRIX, 0, 90, 0f, 0f, 1f);
                    Matrix.scaleM(IDENTITY_MATRIX, 0, 1f, scaleval, 1f);
                } else {
                    Matrix.scaleM(IDENTITY_MATRIX, 0, 1f, scaleval, 1f);
                }
                break;
            case UPSIDEDOWN_VERTICAL:
                if(scaleToFit)
                {
                    //Matrix.rotateM(IDENTITY_MATRIX, 0, -90, 0f, 0f, 1f);
                    Matrix.scaleM(IDENTITY_MATRIX, 0, 1f, scaleval, 1f);
                }
                else {
                    Matrix.scaleM(IDENTITY_MATRIX, 0, 1f, scaleval, 1f);
                }
                break;
            case UPSIDEDOWN_LANDSCAPE:
                if (scaleToFit) {
                    //Matrix.rotateM(IDENTITY_MATRIX, 0, -90, 0f, 0f, 1f);
                    Matrix.scaleM(IDENTITY_MATRIX, 0, 1f, scaleval, 1f);
                } else {
                    Matrix.scaleM(IDENTITY_MATRIX, 0, 1f, scaleval, 1f);
                }
                break;
        }
    }
}

/**
 * Releases resources.
 */
public void release() {
    if (mProgram != null) {
        mProgram.release();
        mProgram = null;
    }
}

/**
 * Returns the program currently in use.
 */
public Texture2dProgram getProgram() {
    return mProgram;
}

/**
 * Changes the program.  The previous program will be released.
 */
public void changeProgram(Texture2dProgram program) {
    mProgram.release();
    mProgram = program;
}

/**
 * Creates a texture object suitable for use with drawFrame().
 */
public int createTextureObject() {
    return mProgram.createTextureObject();
}

/**
 * Draws a viewport-filling rect, texturing it with the specified texture object.
 */
public void drawFrame(int textureId, float[] texMatrix) {
    // Use the identity matrix for MVP so our 2x2 FULL_RECTANGLE covers the viewport.
    synchronized (mDrawLock) {
        if (mCorrectHorizontalVideo && !mScaleToFit && (requestedOrientation == SCREEN_ROTATION.LANDSCAPE || requestedOrientation == SCREEN_ROTATION.UPSIDEDOWN_LANDSCAPE)) {
            //Matrix.scaleM(texMatrix, 0, 0.316f, 1.0f, 1f);
        }
        mProgram.draw(IDENTITY_MATRIX, mRectDrawable.getVertexArray(), 0,
                mRectDrawable.getVertexCount(), mRectDrawable.getCoordsPerVertex(),
                mRectDrawable.getVertexStride(),
                texMatrix, TEX_COORDS_BUF, textureId, TEX_COORDS_STRIDE);
    }
}

/**
 * Pass touch event down to the
 * texture's shader program
 *
 * @param ev
 */
public void handleTouchEvent(MotionEvent ev) {
    mProgram.handleTouchEvent(ev);
}

}

[gouravd]

This is how my SensonChangedListener looks like (BroadcastActivity.java). mBroadcaster.fixAspectRatio calls fixAspectRatio in CameraEncoder.

private SensorEventListener mOrientationListener = new SensorEventListener() {
final int SENSOR_CONFIRMATION_THRESHOLD = 5;
int[] confirmations = new int[2];
int orientation = -1;

    @Override
    public void onSensorChanged(SensorEvent event) {
        if (activity != null && activity.findViewById(R.id.rotateDeviceHint) != null) {
            //Log.i(TAG, "Sensor " + event.values[1]);
            if (event.values[1] > 10 || event.values[1] < -10) {
                // Sensor noise. Ignore.
            } else if (event.values[1] < 5.5 && event.values[1] > -5.5) {
                // Landscape
                if (orientation != 1 && readingConfirmed(1)) {
                    if (mBroadcaster.getSessionConfig().getFixAspectRatio()) {
                        if (event.values[0] > 0) {
                            mBroadcaster.fixAspectRatio(FullFrameRect.SCREEN_ROTATION.LANDSCAPE);
                        } else {
                            mBroadcaster.fixAspectRatio(FullFrameRect.SCREEN_ROTATION.UPSIDEDOWN_LANDSCAPE);
                        }
                    } else {
                        activity.findViewById(R.id.rotateDeviceHint).setVisibility(View.VISIBLE);
                    }
                    orientation = 1;
                }
            } else if (event.values[1] > 7.5 || event.values[1] < -7.5) {
                // Portrait
                if (orientation != 0 && readingConfirmed(0)) {
                    if (mBroadcaster.getSessionConfig().getFixAspectRatio()) {
                        if (event.values[1] > 0) {
                            mBroadcaster.fixAspectRatio(FullFrameRect.SCREEN_ROTATION.VERTICAL);
                        } else {
                            mBroadcaster.fixAspectRatio(FullFrameRect.SCREEN_ROTATION.UPSIDEDOWN_VERTICAL);
                        }
                    } else {
                        activity.findViewById(R.id.rotateDeviceHint).setVisibility(View.GONE);
                    }
                    orientation = 0;
                }
            }
        }
    }

    /**
     * Determine if a sensor reading is trustworthy
     * based on a series of consistent readings
     */
    private boolean readingConfirmed(int orientation) {
        confirmations[orientation]++;
        confirmations[orientation == 0 ? 1 : 0] = 0;
        return confirmations[orientation] > SENSOR_CONFIRMATION_THRESHOLD;
    }

    @Override
    public void onAccuracyChanged(Sensor sensor, int accuracy) {
    }
};

[gouravd]

This is how my fixAspectRatio() in CameraEncoder.java looks like mFullScreen.fixAspectRatio calls AspectRatio of fullframerect.java and mDisplayRenderer.fixAspectRatio calls fixAspectRatio of CameraSurfaceRenderer.java

public void fixAspectRatio(FullFrameRect.SCREEN_ROTATION orientation) {
if (mFullScreen != null) mFullScreen.fixAspectRatio(orientation, true, isWideScreen());
mDisplayRenderer.fixAspectRatio(orientation);
}

private boolean isWideScreen()
{
if(mCamera == null) return true;
return mCamera.getParameters().getPreviewSize().height * 16 == mCamera.getParameters().getPreviewSize().width * 9;
}

[gouravd]

This is how my fixAspectRatio in CameraSurfaceRenderer.java looks like

public void fixAspectRatio(FullFrameRect.SCREEN_ROTATION isVertical) {
if (mFullScreenCamera != null) mFullScreenCamera.fixAspectRatio(isVertical, false, isWideScreen(mCameraEncoder));
}

[gouravd]

This is how my GLCameraView looks like

public class GLCameraView extends GLSurfaceView {
private static final String TAG = "GLCameraView";

private static final double ASPECT_RATIO = 4.0 / 4.0;

protected ScaleGestureDetector mScaleGestureDetector;
private Camera mCamera;
private int mMaxZoom;

public GLCameraView(Context context) {
    super(context);
    init(context);
}

public GLCameraView(Context context, AttributeSet attrs) {
    super(context, attrs);
    init(context);
}

private void init(Context context){
    mMaxZoom = 0;

}

public void setCamera(Camera camera){
    mCamera = camera;
    mCamera.setDisplayOrientation(90);
    Camera.Parameters camParams = mCamera.getParameters();
    if(camParams.isZoomSupported()){
        mMaxZoom = camParams.getMaxZoom();
        mScaleGestureDetector = new ScaleGestureDetector(getContext(), mScaleListener);
    }
}

public void releaseCamera(){
    mCamera = null;
    mScaleGestureDetector = null;
}

private ScaleGestureDetector.SimpleOnScaleGestureListener mScaleListener = new ScaleGestureDetector.SimpleOnScaleGestureListener(){

    int mZoomWhenScaleBegan = 0;
    int mCurrentZoom = 0;

    @Override
    public boolean onScale(ScaleGestureDetector detector) {
        if(mCamera != null){
            Camera.Parameters params = mCamera.getParameters();
            mCurrentZoom = (int) (mZoomWhenScaleBegan + (mMaxZoom * (detector.getScaleFactor() - 1)));
            mCurrentZoom = Math.min(mCurrentZoom, mMaxZoom);
            mCurrentZoom = Math.max(0, mCurrentZoom);
            params.setZoom(mCurrentZoom);
            mCamera.setParameters(params);
        }

        return false;
    }

    @Override
    public boolean onScaleBegin(ScaleGestureDetector detector) {
        mZoomWhenScaleBegan =  mCamera.getParameters().getZoom();
        return true;
    }

    @Override
    public void onScaleEnd(ScaleGestureDetector detector) {
    }
};

@Override
public boolean onTouchEvent(MotionEvent ev) {
    if(mScaleGestureDetector != null){
        if(!mScaleGestureDetector.onTouchEvent(ev)){
            // No scale gesture detected

        }
    }
    return true;
}

/**
 * Measure the view and its content to determine the measured width and the
 * measured height
 */
@Override
protected void onMeasure(int widthMeasureSpec, int heightMeasureSpec) {
    int height = MeasureSpec.getSize(heightMeasureSpec);
    int width = MeasureSpec.getSize(widthMeasureSpec);

    if (width > height * ASPECT_RATIO) {
        width = (int) (height * ASPECT_RATIO + 0.5);
    }
    else {
        height = (int) (width / ASPECT_RATIO + 0.5);
    }

    setMeasuredDimension(width, height);
}

/*@Override
protected void onMeasure(int widthMeasureSpec, int heightMeasureSpec) {
    int width = View.MeasureSpec.getSize( widthMeasureSpec );
    int heightMode = View.MeasureSpec.getMode(heightMeasureSpec);

    int height = (width * 9)/16;
    int newheightMeasureSpec = View.MeasureSpec.makeMeasureSpec( height, heightMode  );

    super.onMeasure(widthMeasureSpec, newheightMeasureSpec);
}*/

}