[pjsip] Android camera rotation

Сергей Митрофанов goretz.m at gmail.com
Tue Apr 5 08:14:19 EDT 2016


Hi Ming

I know that old ticket, and I have some info from my experience for you
here.
I'm developing a free video chat using your superb library. Really, it is
awesome in comparison to undocumented linphone or non-sip solutions (laggy
red5, proprietary adobe air or deprecated vitamio).
But pjsip for sure has an issue with the camera orientation at least in
android platforms.
First of all we have a lot of devices to test on, so I'm talking with the
real world results in my hands now.

For example if I use your sample app as is with the most recent svn version
of pjsip on major variety of our devices the front camera preview and
stream appears upside down in any orientation, while the back camera shows
normal.

This is because for the most devices initial rotation value from android
camera API is 90 for both front and back cameras. While other devices
(tablets, TV - from 20% to 30%) can have other values: 0 or 90 or 270 for
any of cameras.

As it is obvious from pjsip code, you don't use at all the actual camera
rotation from android hardware.camera API.
Looks like you suggested "all back cameras are always rotated for 90 and
all front cameras are always rotated for 270". And that why you map android
device orientation with 90 offset to the library orientation in the test
app.
But in fact this simple mapping is not always true and device can also have
external plugged in cameras (in case of TVs).

So to make it work correctly I have to make strange crutches (And I done it
before) to figure out the real initial rotation of device and fix the
orientation before passing it to the library. And since there is no
straight mapping between android cameras IDs and pjsip capture devices IDs
it becomes very ugly and unreliable.
Speaking specifically that how I had solve this issue before to make it
work as intended on all devices:
1. Get the list of android cameras from hardware.camera API
2. Get rotation for front and back cameras (only 2 values, just for front
and back, since we have no straight mapping to pjsip devices here)
3. Tweak rotation values: rotationBack = 90 - cameraInfo.orientation and
rotationFront = 270 - cameraInfo.orientation
4. Get list of the pjsip devices
5. Create an Array of POJOs to keep the rotation values for pjsip devices
6. Assign the rotationBack to the pjsip devices that contain in their name
the word "back"
7. Assign the rotationFront to the pjsip devices that contain in their name
the word "front"
8. Remove non-camera devices (driver is not "Android" or direction is not
"PJMEDIA_DIR_ENCODING")
And we have a list of pjsip camera IDs with their rotations values packed
in POJOs
Then at the time of rotation those one used like this:
1. figure out the actualRotation = screenOrientation - cameraRotation (from
camera POJO - either rotationBack or rotationFront)
2. Normalize the result value, so it will be between 0 and 360 (+ or - 360
to the value)
3. Make the same mapping as in your sample application (0 = 270DEG, 90 =
NATUAL etc.)
As you see it is a long and unstable way to achieve this.
After tweaking the android_dev.c I now can use the camera rotation
functionality without any extra crutches and wheels. And it works on any of
android devices that is out there now.

The main issue in the current pjsip work with cameras - it does not respect
the real initial rotation of camera devices.
If this is not a problem for other platforms it doesn't mean that there is
no problem at all.
And if you want to keep behavior the same with other platforms to the
prejudice of library usability on other platforms, then it appears sad for
me...

Anyway I just pointed out the big annoying issue with current camera
rotation calculations scheme on Android devices and I don't force you to
use this solution for the main branch.
Maybe later you will find a nice solution that will work as simple as with
my patch (without any excessive code-magic) across all platforms supported
by your team. And I will be very glad when you will make it and will make a
standing ovation for you guys!
But until this happen I will use this patch.
Maybe it will be useful for someone else, that why I publish it here...

Best Regards,
Sergey

вт, 5 апр. 2016 г. в 13:06, Ming <ming at teluu.com>:

> Hi Сергей,
>
> Unfortunately this will make the behavior different with other
> platforms. And FYI, we just changed the spec of pjmedia_orient in
> ticket #1880 (https://trac.pjsip.org/repos/ticket/1880) -->
>
> https://trac.pjsip.org/repos/browser/pjproject/trunk/pjmedia/include/pjmedia/types.h#L197
>
> Regards,
> Ming
>
> On Mon, Apr 4, 2016 at 7:42 PM, Сергей Митрофанов <goretz.m at gmail.com>
> wrote:
> > Yesterday I have published a patch, but today rechecked and it produces
> > incorrect rotations for back camera.
> > Here is the correct one patch.
> > It works without any additional rotation offset calculations in app and
> with
> > it pjsip rotations are mapped to android screen orientations directly.
> >
> > Was in test app:
> >
> > wm = (WindowManager)this.getSystemService(Context.WINDOW_SERVICE);
> > display = wm.getDefaultDisplay();
> > rotation = display.getRotation();
> > Log.d("CallActivity", "Device orientation changed: " + rotation);
> > switch (rotation) {
> >     case Surface.ROTATION_0:   // Portrait
> >         orient = pjmedia_orient.PJMEDIA_ORIENT_ROTATE_270DEG;
> >         break;
> >     case Surface.ROTATION_90:  // Landscape, home button on the right
> >         orient = pjmedia_orient.PJMEDIA_ORIENT_NATURAL;
> >         break;
> >     case Surface.ROTATION_180:
> >         orient = pjmedia_orient.PJMEDIA_ORIENT_ROTATE_90DEG;
> >         break;
> >     case Surface.ROTATION_270: // Landscape, home button on the left
> >         orient = pjmedia_orient.PJMEDIA_ORIENT_ROTATE_180DEG;
> >         break;
> >     default:
> >         orient = pjmedia_orient.PJMEDIA_ORIENT_UNKNOWN;
> > }
> >
> > With this patch:
> >
> > wm = (WindowManager)this.getSystemService(Context.WINDOW_SERVICE);
> > display = wm.getDefaultDisplay();
> > rotation = display.getRotation();
> > Log.d("CallActivity", "Device orientation changed: " + rotation);
> > switch (rotation) {
> >     case Surface.ROTATION_0:   // Portrait
> >         orient = pjmedia_orient.PJMEDIA_ORIENT_NATURAL;
> >         break;
> >     case Surface.ROTATION_90:  // Landscape, home button on the right
> >         orient = pjmedia_orient.PJMEDIA_ORIENT_ROTATE_90DEG;
> >         break;
> >     case Surface.ROTATION_180:
> >         orient = pjmedia_orient.PJMEDIA_ORIENT_ROTATE_180DEG;
> >         break;
> >     case Surface.ROTATION_270: // Landscape, home button on the left
> >         orient = pjmedia_orient.PJMEDIA_ORIENT_ROTATE_270DEG;
> >         break;
> >     default:
> >         orient = pjmedia_orient.PJMEDIA_ORIENT_UNKNOWN;
> > }
> >
> > And this works correct for both facing and back cameras without any
> > additional work in application.
> >
> >
> > вс, 3 апр. 2016 г. в 17:25, Сергей Митрофанов <goretz.m at gmail.com>:
> >>
> >> Hi!
> >>
> >> I have found that pjsip does not respect initial camera rotation in
> >> android devices.
> >> Sure the android_dev.c itself extracts the rotation value from android
> >> camera devices,
> >> but for now this value is actually not used, resulting wrong capture
> video
> >> orientation.
> >> I made a simple patch (a little bit raw code style) that resolves this
> >> bug.
> >> It takes the camera rotation value in the game and make PJSUA2 easier to
> >> use - developer need no more think about camera initial rotation and
> tweak
> >> the rotation value before giving it to PJSUA2 itself.
> >>
> >> Here is the diff patch for the changes:
> >>
> >> From 1ea38ce9103f6fadd9812f635c83442994fd37ea Mon Sep 17 00:00:00 2001
> >>
> >> From: Sergey Mitrofanov <GOretZ.M at gmail.com>
> >>
> >> Date: Sun, 3 Apr 2016 17:21:48 +0300
> >>
> >> Subject: [PATCH] Fixing to respect initial camera rotation in android
> >> devices.
> >>
> >>
> >> ---
> >>
> >>  pjmedia/src/pjmedia-videodev/android_dev.c | 36
> >> ++++++++++++++++++++++++++++++
> >>
> >>  1 file changed, 36 insertions(+)
> >>
> >>
> >> diff --git a/pjmedia/src/pjmedia-videodev/android_dev.c
> >> b/pjmedia/src/pjmedia-videodev/android_dev.c
> >>
> >> index 1da7261..fd6e61d 100644
> >>
> >> --- a/pjmedia/src/pjmedia-videodev/android_dev.c
> >>
> >> +++ b/pjmedia/src/pjmedia-videodev/android_dev.c
> >>
> >> @@ -72,6 +72,7 @@ typedef struct and_dev_info
> >>
> >>      pjmedia_vid_dev_info info; /**< Base info         */
> >>
> >>      unsigned dev_idx; /**< Original dev ID   */
> >>
> >>      pj_bool_t facing; /**< Front/back camera?*/
> >>
> >> +    unsigned rotation; /**< initial camera rotation*/
> >>
> >>      unsigned sup_size_cnt; /**< # of supp'd size  */
> >>
> >>      pjmedia_rect_size *sup_size; /**< Supported size    */
> >>
> >>      unsigned sup_fps_cnt; /**< # of supp'd FPS   */
> >>
> >> @@ -532,6 +533,8 @@ static pj_status_t
> >> and_factory_refresh(pjmedia_vid_dev_factory *ff)
> >>
> >>   } else {
> >>
> >>       pj_ansi_strncpy(vdi->name, "Front camera", sizeof(vdi->name));
> >>
> >>   }
> >>
> >> + adi->rotation = (*jni_env)->GetIntField(jni_env, jdev_info,
> >>
> >> + jobjs.cam_info.f_orient);
> >>
> >>
> >>
> >>   /* Get supported sizes */
> >>
> >>   jtmp = (*jni_env)->GetObjectField(jni_env, jdev_info,
> >>
> >> @@ -1001,6 +1004,39 @@ static pj_status_t
> >> and_stream_set_cap(pjmedia_vid_dev_stream *s,
> >>
> >>   else if (eff_ori == PJMEDIA_ORIENT_ROTATE_270DEG)
> >>
> >>       eff_ori = PJMEDIA_ORIENT_ROTATE_90DEG;
> >>
> >>       }
> >>
> >> +     /* Normalize the orientation for rotated camera */
> >>
> >> +     switch(adi->rotation){
> >>
> >> + case 90:
> >>
> >> + if (eff_ori == PJMEDIA_ORIENT_ROTATE_90DEG)
> >>
> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_180DEG;
> >>
> >> + else if (eff_ori == PJMEDIA_ORIENT_ROTATE_180DEG)
> >>
> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_270DEG;
> >>
> >> + else if (eff_ori == PJMEDIA_ORIENT_ROTATE_270DEG)
> >>
> >> +     eff_ori = PJMEDIA_ORIENT_NATURAL;
> >>
> >> + else if (eff_ori == PJMEDIA_ORIENT_NATURAL)
> >>
> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_90DEG;
> >>
> >> + break;
> >>
> >> + case 180:
> >>
> >> + if (eff_ori == PJMEDIA_ORIENT_ROTATE_90DEG)
> >>
> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_270DEG;
> >>
> >> + else if (eff_ori == PJMEDIA_ORIENT_ROTATE_180DEG)
> >>
> >> +     eff_ori = PJMEDIA_ORIENT_NATURAL;
> >>
> >> + else if (eff_ori == PJMEDIA_ORIENT_ROTATE_270DEG)
> >>
> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_90DEG;
> >>
> >> + else if (eff_ori == PJMEDIA_ORIENT_NATURAL)
> >>
> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_180DEG;
> >>
> >> + break;
> >>
> >> + case 270:
> >>
> >> + if (eff_ori == PJMEDIA_ORIENT_ROTATE_90DEG)
> >>
> >> +     eff_ori = PJMEDIA_ORIENT_NATURAL;
> >>
> >> + else if (eff_ori == PJMEDIA_ORIENT_ROTATE_180DEG)
> >>
> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_90DEG;
> >>
> >> + else if (eff_ori == PJMEDIA_ORIENT_ROTATE_270DEG)
> >>
> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_180DEG;
> >>
> >> + else if (eff_ori == PJMEDIA_ORIENT_NATURAL)
> >>
> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_270DEG;
> >>
> >> + break;
> >>
> >> +     }
> >>
> >>       pjmedia_vid_dev_conv_set_rotation(&strm->conv, eff_ori);
> >>
> >>
> >>
> >>       PJ_LOG(4, (THIS_FILE, "Video capture orientation set to %d",
> >>
> >> --
> >>
> >> 2.6.4 (Apple Git-63)
> >>
> >>
> >
> > _______________________________________________
> > Visit our blog: http://blog.pjsip.org
> >
> > pjsip mailing list
> > pjsip at lists.pjsip.org
> > http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
> >
>
> _______________________________________________
> Visit our blog: http://blog.pjsip.org
>
> pjsip mailing list
> pjsip at lists.pjsip.org
> http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.pjsip.org/pipermail/pjsip_lists.pjsip.org/attachments/20160405/373c8142/attachment-0002.html>


More information about the pjsip mailing list