[pjsip] Android camera rotation

Сергей Митрофанов goretz.m at gmail.com
Wed Apr 6 00:14:06 EDT 2016


Hi Ming,

You are welcome.
I agree, you have to do more testings by yourself to see what is going on.
Btw, I suppose I should point out my building system details.
So I build the pjsip in MacBook Pro (Retina 13", early 2015)
With OS X El Capitan 10.11.4 (15E65) on board.
I also use libyuv-android and openh264-1.0.0 according to the video guide
tickets
Also I use the most recent opus snapshot and stripped version of webrtc EC
(from ticket about EC).
I'm building for armeabi-v7a only for now
And this is my configure options

TARGET_ABI=armeabi-v7a APP_PLATFORM=android-16 ./configure-android
--use-ndk-cflags --disable-speex-aec --disable-g711-codec
--disable-l16-codec --disable-gsm-codec --disable-g722-codec
--disable-g7221-codec --disable-speex-codec --disable-ilbc-codec
--disable-ffmpeg --disable-opencore-amr --disable-silk
--with-openh264=/Users/sergej/Projects/openh264-1.0.0/android
--with-libyuv=/Users/sergej/Projects/libyuv-android/jni
--with-opus=/Users/sergej/Projects/opus/build --disable-webrtc
Ask freely if you need some other details about my building system.

Regards,
Sergey

ср, 6 апр. 2016 г. в 6:04, Ming <ming at teluu.com>:

> Hi Sergey,
>
> Thanks for your valuable insight and feedback.
> Let us do more testing first with this.
>
> Regards,
> Ming
>
> On Tue, Apr 5, 2016 at 8:14 PM, Сергей Митрофанов <goretz.m at gmail.com>
> wrote:
> > Hi Ming
> >
> > I know that old ticket, and I have some info from my experience for you
> > here.
> > I'm developing a free video chat using your superb library. Really, it is
> > awesome in comparison to undocumented linphone or non-sip solutions
> (laggy
> > red5, proprietary adobe air or deprecated vitamio).
> > But pjsip for sure has an issue with the camera orientation at least in
> > android platforms.
> > First of all we have a lot of devices to test on, so I'm talking with the
> > real world results in my hands now.
> >
> > For example if I use your sample app as is with the most recent svn
> version
> > of pjsip on major variety of our devices the front camera preview and
> stream
> > appears upside down in any orientation, while the back camera shows
> normal.
> >
> > This is because for the most devices initial rotation value from android
> > camera API is 90 for both front and back cameras. While other devices
> > (tablets, TV - from 20% to 30%) can have other values: 0 or 90 or 270 for
> > any of cameras.
> >
> > As it is obvious from pjsip code, you don't use at all the actual camera
> > rotation from android hardware.camera API.
> > Looks like you suggested "all back cameras are always rotated for 90 and
> all
> > front cameras are always rotated for 270". And that why you map android
> > device orientation with 90 offset to the library orientation in the test
> > app.
> > But in fact this simple mapping is not always true and device can also
> have
> > external plugged in cameras (in case of TVs).
> >
> > So to make it work correctly I have to make strange crutches (And I done
> it
> > before) to figure out the real initial rotation of device and fix the
> > orientation before passing it to the library. And since there is no
> straight
> > mapping between android cameras IDs and pjsip capture devices IDs it
> becomes
> > very ugly and unreliable.
> > Speaking specifically that how I had solve this issue before to make it
> work
> > as intended on all devices:
> > 1. Get the list of android cameras from hardware.camera API
> > 2. Get rotation for front and back cameras (only 2 values, just for front
> > and back, since we have no straight mapping to pjsip devices here)
> > 3. Tweak rotation values: rotationBack = 90 - cameraInfo.orientation and
> > rotationFront = 270 - cameraInfo.orientation
> > 4. Get list of the pjsip devices
> > 5. Create an Array of POJOs to keep the rotation values for pjsip devices
> > 6. Assign the rotationBack to the pjsip devices that contain in their
> name
> > the word "back"
> > 7. Assign the rotationFront to the pjsip devices that contain in their
> name
> > the word "front"
> > 8. Remove non-camera devices (driver is not "Android" or direction is not
> > "PJMEDIA_DIR_ENCODING")
> > And we have a list of pjsip camera IDs with their rotations values
> packed in
> > POJOs
> > Then at the time of rotation those one used like this:
> > 1. figure out the actualRotation = screenOrientation - cameraRotation
> (from
> > camera POJO - either rotationBack or rotationFront)
> > 2. Normalize the result value, so it will be between 0 and 360 (+ or -
> 360
> > to the value)
> > 3. Make the same mapping as in your sample application (0 = 270DEG, 90 =
> > NATUAL etc.)
> > As you see it is a long and unstable way to achieve this.
> > After tweaking the android_dev.c I now can use the camera rotation
> > functionality without any extra crutches and wheels. And it works on any
> of
> > android devices that is out there now.
> >
> > The main issue in the current pjsip work with cameras - it does not
> respect
> > the real initial rotation of camera devices.
> > If this is not a problem for other platforms it doesn't mean that there
> is
> > no problem at all.
> > And if you want to keep behavior the same with other platforms to the
> > prejudice of library usability on other platforms, then it appears sad
> for
> > me...
> >
> > Anyway I just pointed out the big annoying issue with current camera
> > rotation calculations scheme on Android devices and I don't force you to
> use
> > this solution for the main branch.
> > Maybe later you will find a nice solution that will work as simple as
> with
> > my patch (without any excessive code-magic) across all platforms
> supported
> > by your team. And I will be very glad when you will make it and will
> make a
> > standing ovation for you guys!
> > But until this happen I will use this patch.
> > Maybe it will be useful for someone else, that why I publish it here...
> >
> > Best Regards,
> > Sergey
> >
> > вт, 5 апр. 2016 г. в 13:06, Ming <ming at teluu.com>:
> >>
> >> Hi Сергей,
> >>
> >> Unfortunately this will make the behavior different with other
> >> platforms. And FYI, we just changed the spec of pjmedia_orient in
> >> ticket #1880 (https://trac.pjsip.org/repos/ticket/1880) -->
> >>
> >>
> https://trac.pjsip.org/repos/browser/pjproject/trunk/pjmedia/include/pjmedia/types.h#L197
> >>
> >> Regards,
> >> Ming
> >>
> >> On Mon, Apr 4, 2016 at 7:42 PM, Сергей Митрофанов <goretz.m at gmail.com>
> >> wrote:
> >> > Yesterday I have published a patch, but today rechecked and it
> produces
> >> > incorrect rotations for back camera.
> >> > Here is the correct one patch.
> >> > It works without any additional rotation offset calculations in app
> and
> >> > with
> >> > it pjsip rotations are mapped to android screen orientations directly.
> >> >
> >> > Was in test app:
> >> >
> >> > wm = (WindowManager)this.getSystemService(Context.WINDOW_SERVICE);
> >> > display = wm.getDefaultDisplay();
> >> > rotation = display.getRotation();
> >> > Log.d("CallActivity", "Device orientation changed: " + rotation);
> >> > switch (rotation) {
> >> >     case Surface.ROTATION_0:   // Portrait
> >> >         orient = pjmedia_orient.PJMEDIA_ORIENT_ROTATE_270DEG;
> >> >         break;
> >> >     case Surface.ROTATION_90:  // Landscape, home button on the right
> >> >         orient = pjmedia_orient.PJMEDIA_ORIENT_NATURAL;
> >> >         break;
> >> >     case Surface.ROTATION_180:
> >> >         orient = pjmedia_orient.PJMEDIA_ORIENT_ROTATE_90DEG;
> >> >         break;
> >> >     case Surface.ROTATION_270: // Landscape, home button on the left
> >> >         orient = pjmedia_orient.PJMEDIA_ORIENT_ROTATE_180DEG;
> >> >         break;
> >> >     default:
> >> >         orient = pjmedia_orient.PJMEDIA_ORIENT_UNKNOWN;
> >> > }
> >> >
> >> > With this patch:
> >> >
> >> > wm = (WindowManager)this.getSystemService(Context.WINDOW_SERVICE);
> >> > display = wm.getDefaultDisplay();
> >> > rotation = display.getRotation();
> >> > Log.d("CallActivity", "Device orientation changed: " + rotation);
> >> > switch (rotation) {
> >> >     case Surface.ROTATION_0:   // Portrait
> >> >         orient = pjmedia_orient.PJMEDIA_ORIENT_NATURAL;
> >> >         break;
> >> >     case Surface.ROTATION_90:  // Landscape, home button on the right
> >> >         orient = pjmedia_orient.PJMEDIA_ORIENT_ROTATE_90DEG;
> >> >         break;
> >> >     case Surface.ROTATION_180:
> >> >         orient = pjmedia_orient.PJMEDIA_ORIENT_ROTATE_180DEG;
> >> >         break;
> >> >     case Surface.ROTATION_270: // Landscape, home button on the left
> >> >         orient = pjmedia_orient.PJMEDIA_ORIENT_ROTATE_270DEG;
> >> >         break;
> >> >     default:
> >> >         orient = pjmedia_orient.PJMEDIA_ORIENT_UNKNOWN;
> >> > }
> >> >
> >> > And this works correct for both facing and back cameras without any
> >> > additional work in application.
> >> >
> >> >
> >> > вс, 3 апр. 2016 г. в 17:25, Сергей Митрофанов <goretz.m at gmail.com>:
> >> >>
> >> >> Hi!
> >> >>
> >> >> I have found that pjsip does not respect initial camera rotation in
> >> >> android devices.
> >> >> Sure the android_dev.c itself extracts the rotation value from
> android
> >> >> camera devices,
> >> >> but for now this value is actually not used, resulting wrong capture
> >> >> video
> >> >> orientation.
> >> >> I made a simple patch (a little bit raw code style) that resolves
> this
> >> >> bug.
> >> >> It takes the camera rotation value in the game and make PJSUA2 easier
> >> >> to
> >> >> use - developer need no more think about camera initial rotation and
> >> >> tweak
> >> >> the rotation value before giving it to PJSUA2 itself.
> >> >>
> >> >> Here is the diff patch for the changes:
> >> >>
> >> >> From 1ea38ce9103f6fadd9812f635c83442994fd37ea Mon Sep 17 00:00:00
> 2001
> >> >>
> >> >> From: Sergey Mitrofanov <GOretZ.M at gmail.com>
> >> >>
> >> >> Date: Sun, 3 Apr 2016 17:21:48 +0300
> >> >>
> >> >> Subject: [PATCH] Fixing to respect initial camera rotation in android
> >> >> devices.
> >> >>
> >> >>
> >> >> ---
> >> >>
> >> >>  pjmedia/src/pjmedia-videodev/android_dev.c | 36
> >> >> ++++++++++++++++++++++++++++++
> >> >>
> >> >>  1 file changed, 36 insertions(+)
> >> >>
> >> >>
> >> >> diff --git a/pjmedia/src/pjmedia-videodev/android_dev.c
> >> >> b/pjmedia/src/pjmedia-videodev/android_dev.c
> >> >>
> >> >> index 1da7261..fd6e61d 100644
> >> >>
> >> >> --- a/pjmedia/src/pjmedia-videodev/android_dev.c
> >> >>
> >> >> +++ b/pjmedia/src/pjmedia-videodev/android_dev.c
> >> >>
> >> >> @@ -72,6 +72,7 @@ typedef struct and_dev_info
> >> >>
> >> >>      pjmedia_vid_dev_info info; /**< Base info         */
> >> >>
> >> >>      unsigned dev_idx; /**< Original dev ID   */
> >> >>
> >> >>      pj_bool_t facing; /**< Front/back camera?*/
> >> >>
> >> >> +    unsigned rotation; /**< initial camera rotation*/
> >> >>
> >> >>      unsigned sup_size_cnt; /**< # of supp'd size  */
> >> >>
> >> >>      pjmedia_rect_size *sup_size; /**< Supported size    */
> >> >>
> >> >>      unsigned sup_fps_cnt; /**< # of supp'd FPS   */
> >> >>
> >> >> @@ -532,6 +533,8 @@ static pj_status_t
> >> >> and_factory_refresh(pjmedia_vid_dev_factory *ff)
> >> >>
> >> >>   } else {
> >> >>
> >> >>       pj_ansi_strncpy(vdi->name, "Front camera", sizeof(vdi->name));
> >> >>
> >> >>   }
> >> >>
> >> >> + adi->rotation = (*jni_env)->GetIntField(jni_env, jdev_info,
> >> >>
> >> >> + jobjs.cam_info.f_orient);
> >> >>
> >> >>
> >> >>
> >> >>   /* Get supported sizes */
> >> >>
> >> >>   jtmp = (*jni_env)->GetObjectField(jni_env, jdev_info,
> >> >>
> >> >> @@ -1001,6 +1004,39 @@ static pj_status_t
> >> >> and_stream_set_cap(pjmedia_vid_dev_stream *s,
> >> >>
> >> >>   else if (eff_ori == PJMEDIA_ORIENT_ROTATE_270DEG)
> >> >>
> >> >>       eff_ori = PJMEDIA_ORIENT_ROTATE_90DEG;
> >> >>
> >> >>       }
> >> >>
> >> >> +     /* Normalize the orientation for rotated camera */
> >> >>
> >> >> +     switch(adi->rotation){
> >> >>
> >> >> + case 90:
> >> >>
> >> >> + if (eff_ori == PJMEDIA_ORIENT_ROTATE_90DEG)
> >> >>
> >> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_180DEG;
> >> >>
> >> >> + else if (eff_ori == PJMEDIA_ORIENT_ROTATE_180DEG)
> >> >>
> >> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_270DEG;
> >> >>
> >> >> + else if (eff_ori == PJMEDIA_ORIENT_ROTATE_270DEG)
> >> >>
> >> >> +     eff_ori = PJMEDIA_ORIENT_NATURAL;
> >> >>
> >> >> + else if (eff_ori == PJMEDIA_ORIENT_NATURAL)
> >> >>
> >> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_90DEG;
> >> >>
> >> >> + break;
> >> >>
> >> >> + case 180:
> >> >>
> >> >> + if (eff_ori == PJMEDIA_ORIENT_ROTATE_90DEG)
> >> >>
> >> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_270DEG;
> >> >>
> >> >> + else if (eff_ori == PJMEDIA_ORIENT_ROTATE_180DEG)
> >> >>
> >> >> +     eff_ori = PJMEDIA_ORIENT_NATURAL;
> >> >>
> >> >> + else if (eff_ori == PJMEDIA_ORIENT_ROTATE_270DEG)
> >> >>
> >> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_90DEG;
> >> >>
> >> >> + else if (eff_ori == PJMEDIA_ORIENT_NATURAL)
> >> >>
> >> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_180DEG;
> >> >>
> >> >> + break;
> >> >>
> >> >> + case 270:
> >> >>
> >> >> + if (eff_ori == PJMEDIA_ORIENT_ROTATE_90DEG)
> >> >>
> >> >> +     eff_ori = PJMEDIA_ORIENT_NATURAL;
> >> >>
> >> >> + else if (eff_ori == PJMEDIA_ORIENT_ROTATE_180DEG)
> >> >>
> >> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_90DEG;
> >> >>
> >> >> + else if (eff_ori == PJMEDIA_ORIENT_ROTATE_270DEG)
> >> >>
> >> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_180DEG;
> >> >>
> >> >> + else if (eff_ori == PJMEDIA_ORIENT_NATURAL)
> >> >>
> >> >> +     eff_ori = PJMEDIA_ORIENT_ROTATE_270DEG;
> >> >>
> >> >> + break;
> >> >>
> >> >> +     }
> >> >>
> >> >>       pjmedia_vid_dev_conv_set_rotation(&strm->conv, eff_ori);
> >> >>
> >> >>
> >> >>
> >> >>       PJ_LOG(4, (THIS_FILE, "Video capture orientation set to %d",
> >> >>
> >> >> --
> >> >>
> >> >> 2.6.4 (Apple Git-63)
> >> >>
> >> >>
> >> >
> >> > _______________________________________________
> >> > Visit our blog: http://blog.pjsip.org
> >> >
> >> > pjsip mailing list
> >> > pjsip at lists.pjsip.org
> >> > http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
> >> >
> >>
> >> _______________________________________________
> >> Visit our blog: http://blog.pjsip.org
> >>
> >> pjsip mailing list
> >> pjsip at lists.pjsip.org
> >> http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
> >
> >
> > _______________________________________________
> > Visit our blog: http://blog.pjsip.org
> >
> > pjsip mailing list
> > pjsip at lists.pjsip.org
> > http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
> >
>
> _______________________________________________
> Visit our blog: http://blog.pjsip.org
>
> pjsip mailing list
> pjsip at lists.pjsip.org
> http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.pjsip.org/pipermail/pjsip_lists.pjsip.org/attachments/20160406/638d4592/attachment-0002.html>


More information about the pjsip mailing list