2012/08/15

How to do real time image processing in Android using OpenCV?

This article teaches how to pass camera preview frame to android-opencv on the fly.

The Android camera have preview data callback function. I can get the data from the function convert to the OpenCV Mat data form.Using Android JNI IF pass the frame data to the OpenCV, let the image processing do in the native OpenCV library.

Step1:I need a camera preview class to handle the Android camera device.The class i created from the article How to use camera in Android?.In the article shows the way to take picture and save to the file.But in this article i only need the frame data in the camera preview period.
In the new CameraPreview.java file, i removed unused callback functions. Leave the only callback function onPreviewFrame(), and create a native function interface.
And create a Runnable() object to do the image processing.

/*
*  CameraPreview.java
*/
public class CameraPreview implements SurfaceHolder.Callback, Camera.PreviewCallback
{
  private Camera mCamera = null;
  private ImageView MyCameraPreview = null;
  private Bitmap bitmap = null;
  private int[] pixels = null;
  private byte[] FrameData = null;
  private int imageFormat;
  private int PreviewSizeWidth;
  private int PreviewSizeHeight;
  private boolean bProcessing = false;
 
  Handler mHandler = new Handler(Looper.getMainLooper());
  
  public CameraPreview(int PreviewlayoutWidth, int PreviewlayoutHeight,
     ImageView CameraPreview)
  {
    PreviewSizeWidth = PreviewlayoutWidth;
    PreviewSizeHeight = PreviewlayoutHeight;
    MyCameraPreview = CameraPreview;
    bitmap = Bitmap.createBitmap(PreviewSizeWidth, PreviewSizeHeight, Bitmap.Config.ARGB_8888);
    pixels = new int[PreviewSizeWidth * PreviewSizeHeight];
  }

  @Override
  public void onPreviewFrame(byte[] arg0, Camera arg1) 
  {
    // At preview mode, the frame data will push to here.
    if (imageFormat == ImageFormat.NV21)
    {
      //We only accept the NV21(YUV420) format.
      if ( !bProcessing )
      {
        FrameData = arg0;
        mHandler.post(DoImageProcessing);
      }
    }
  }
 
  public void onPause()
  {
    mCamera.stopPreview();
  }

  @Override
  public void surfaceChanged(SurfaceHolder arg0, int arg1, int arg2, int arg3) 
  {
    Parameters parameters;
  
    parameters = mCamera.getParameters();
    // Set the camera preview size
    parameters.setPreviewSize(PreviewSizeWidth, PreviewSizeHeight);
  
    imageFormat = parameters.getPreviewFormat();
  
    mCamera.setParameters(parameters);
  
    mCamera.startPreview();
  }

  @Override
  public void surfaceCreated(SurfaceHolder arg0) 
  {
    mCamera = Camera.open();
    try
    {
      // If did not set the SurfaceHolder, the preview area will be black.
      mCamera.setPreviewDisplay(arg0);
      mCamera.setPreviewCallback(this);
    } 
    catch (IOException e)
    {
      mCamera.release();
      mCamera = null;
    }
  }

  @Override
  public void surfaceDestroyed(SurfaceHolder arg0) 
  {
     mCamera.setPreviewCallback(null);
  mCamera.stopPreview();
  mCamera.release();
  mCamera = null;
  }

  //
  // Native JNI 
  //
  public native boolean ImageProcessing(int width, int height, 
      byte[] NV21FrameData, int [] pixels);
  static 
  {
     System.loadLibrary("ImageProcessing");
  }
    
  private Runnable DoImageProcessing = new Runnable() 
  {
    public void run() 
    {
      Log.i("MyRealTimeImageProcessing", "DoImageProcessing():");
      bProcessing = true;
      ImageProcessing(PreviewSizeWidth, PreviewSizeHeight, FrameData, pixels);
      
      bitmap.setPixels(pixels, 0, PreviewSizeWidth, 0, 0, PreviewSizeWidth, PreviewSizeHeight);
      MyCameraPreview.setImageBitmap(bitmap);
      bProcessing = false;
    }
  };
}
Step2:Create a JNI cpp file to do the image processing.
/*
*  ImageProcessing.cpp
*/
#include <jni.h>

#include <opencv2/core/core.hpp>
#include <opencv2/imgproc/imgproc_c.h>

using namespace std;
using namespace cv;

Mat * mCanny = NULL;

extern "C"
jboolean
Java_my_project_MyRealTimeImageProcessing_CameraPreview_ImageProcessing(
    JNIEnv* env, jobject thiz,
    jint width, jint height,
    jbyteArray NV21FrameData, jintArray outPixels)
{
  jbyte * pNV21FrameData = env->GetByteArrayElements(NV21FrameData, 0);
  jint * poutPixels = env->GetIntArrayElements(outPixels, 0);

  if ( mCanny == NULL )
  {
    mCanny = new Mat(height, width, CV_8UC1);
  }

  Mat mGray(height, width, CV_8UC1, (unsigned char *)pNV21FrameData);
  Mat mResult(height, width, CV_8UC4, (unsigned char *)poutPixels);
  IplImage srcImg = mGray;
  IplImage CannyImg = *mCanny;
  IplImage ResultImg = mResult;

  cvCanny(&srcImg, &CannyImg, 80, 100, 3);
  cvCvtColor(&CannyImg, &ResultImg, CV_GRAY2BGRA);

  env->ReleaseByteArrayElements(NV21FrameData, pNV21FrameData, 0);
  env->ReleaseIntArrayElements(outPixels, poutPixels, 0);
  return true;
}
Step3:Create a instance in the main class.
public class MyRealTimeImageProcessing extends Activity 
{
  private CameraPreview camPreview;
  private ImageView MyCameraPreview = null;
  private FrameLayout mainLayout;
  private int PreviewSizeWidth = 640;
  private int PreviewSizeHeight= 480;
  
  @Override
  public void onCreate(Bundle savedInstanceState) 
  {
    super.onCreate(savedInstanceState);
    //Set this APK Full screen
    getWindow().setFlags(WindowManager.LayoutParams.FLAG_FULLSCREEN,  
              WindowManager.LayoutParams.FLAG_FULLSCREEN);
    //Set this APK no title
    requestWindowFeature(Window.FEATURE_NO_TITLE);  
    setContentView(R.layout.main);
        
    //
    // Create my camera preview 
    //
    MyCameraPreview = new ImageView(this);

    SurfaceView camView = new SurfaceView(this);
    SurfaceHolder camHolder = camView.getHolder();
    camPreview = new CameraPreview(PreviewSizeWidth, PreviewSizeHeight, MyCameraPreview);
        
    camHolder.addCallback(camPreview);
    camHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
        
        
    mainLayout = (FrameLayout) findViewById(R.id.frameLayout1);
    mainLayout.addView(camView, new LayoutParams(PreviewSizeWidth, PreviewSizeHeight));
    mainLayout.addView(MyCameraPreview, new LayoutParams(PreviewSizeWidth, PreviewSizeHeight));
  }
  protected void onPause()
  {
    if ( camPreview != null)
      camPreview.onPause();
    super.onPause();
 }
}
Step4:The preview will be the canny image process result.

89 comments:

  1. can you upload the project as .zip?

    ReplyDelete
    Replies
    1. Hi rrogow2:

      I submit the project to GitHub.
      https://github.com/ikkiChung/MyRealTimeImageProcessing

      Delete

  2. Thank you for the info. It sounds pretty user friendly. I guess I’ll pick one up for fun. thank u
    Image Processing India

    ReplyDelete
  3. Hi, thanks for this article. It seems to be the best way to do real time image processing with a smooth preview.

    But I'm encountering problems to call c++ classes from my JNI function.
    I have some c++ classes that I use on iOS and I would like to use the same on Android. I tried to call them with the opencv camera preview to get the frames (CvCameraViewFrame). My c++ processing works well like this but this is very slower than the Android camera preview.

    So, your solution seems to be the best. But I don't manage to call my c++ functions. My app quits without any error message.

    Here is my code :

    [code]
    JNIEXPORT void JNICALL Java_my_company_scanner_Scanner_getZoneRect(JNIEnv *env, jobject obj, jint width, jint height, jbyteArray NV21FrameData, jintArray outPixels) {
    jbyte * pNV21FrameData = env->GetByteArrayElements(NV21FrameData, 0);
    jint * poutPixels = env->GetIntArrayElements(outPixels, 0);

    Mat mRgb(height, width, CV_8UC4, (unsigned char *)pNV21FrameData);
    Mat mResult(height, width, CV_8UC4, (unsigned char *)outPixels);

    __android_log_print(ANDROID_LOG_ERROR, "JNIWrapper", "image size --> %i, %i", mRgb.cols, mRgb.rows);

    Rect rect = scanner->getZoneRect(mRgb);
    __android_log_print(ANDROID_LOG_ERROR, "JNIWrapper", "%i, %i, %i, %i", rect.x, rect.y, rect.width, rect.height);

    mResult = scanner->drawIndicator(mRgb, rect);

    env->ReleaseByteArrayElements(NV21FrameData, pNV21FrameData, 0);
    env->ReleaseIntArrayElements(outPixels, poutPixels, 0);
    }
    [/code]

    "scanner" is initialized in an other JNI method.
    If I comment the two lines with "getZoneRect()" and "drawIndicator()", my app works well and I can see the preview. Otherwise, I can see my log print once or two times and it exits without error message. I think this is caused by a memory leak but I don't know how to fix it.

    Have you any idea to help me ?

    Thanks !

    ReplyDelete
    Replies
    1. PS : I don't know how to display code in my posts. Could you tell me the tag to use ? Thanks

      Delete
    2. I think the problem is the
      Mat mRgb(height, width, CV_8UC4, (unsigned char *)pNV21FrameData);
      Input data is YUV420(8UC1) format not RGBA(8UC4) data.
      YUV420 total data bytes are Width * Height * 1.5. if you treat it as RGBA Width * Height * 4, it should have memory leak.

      Delete
    3. Ok thanks for your answer, I have fixed this problem.

      I noted that you display the processed image in an ImageView, above the camera preview.
      But the ImageView refresh causes lag.
      I had to rotate it before processing, because I changed the display orientation of the camera to portrait.
      So I would like to process my rotated image and display the real camera preview (which is smoother).

      But the image displayed in the ImageView is not really at the same "place".
      Here is a screenshot : https://dl.dropboxusercontent.com/u/4662606/IMG_9282.JPG

      You can see the ImageView above the camera preview.

      Do you know why the preview and the ImageView don't display the same image ?

      Thanks for your help !

      Delete
    4. I don't know how do you rotate the image and camera preview. Maybe the camera preview just keep in the landscape , not portrait.

      Delete
  4. This comment has been removed by the author.

    ReplyDelete
  5. Anonymous2/6/13 00:00

    HI, Thank you for this great tutorial, I need to ask you something. I have an image(lottery). and I need to capture some information within that image (Barcode, lottery numbers, date, draw number). How can I capture these four elements? I am new to android and have been searching about this for days. Still unable to find a proper answer. Please reply me to tharagal@rocketmail.com / tharakanirmana86@gmail.com, would be a great help!!! Thank you...

    ReplyDelete
    Replies
    1. I believe you should separate it to two different tech. One is barcode scanner, and the other is OCR.
      The lottery has fixed area for barcode and other numbers.
      I think the first step is to find the barcode position, and identified barcode rectangle size.
      So you can use OCR to recognize the fixed relative position numbers and date.

      Delete
  6. Hi,

    I'm trying to run the example code you've submitted to GitHub. First, I try to build JNI part using ndk-build command in Cygwin. I get the following error messages:

    $ ndk-build
    SharedLibrary : libImageProcessing.so
    C:/android/android-ndk-r8e/toolchains/arm-linux-androideabi-4.6/prebuilt/windows/bin/../lib/gcc/arm-linux-androideabi/4.6/../../../../arm-linux-androideabi/bin/ld.exe: ./obj/local/armeabi-v7a/objs/ImageProcessing/ImageProcessing.o: in function cv::Mat::create(int, int, int):C:\opencv4android_v2.4.6\sdk\native\jni\include/opencv2/core/mat.hpp:353: error: undefined reference to 'cv::Mat::create(int, int const*, int)'
    C:/android/android-ndk-r8e/toolchains/arm-linux-androideabi-4.6/prebuilt/windows/bin/../lib/gcc/arm-linux-androideabi/4.6/../../../../arm-linux-androideabi/bin/ld.exe: ./obj/local/armeabi-v7a/objs/ImageProcessing/ImageProcessing.o: in function cv::Mat::release():C:\opencv4android_v2.4.6\sdk\native\jni\include/opencv2/core/mat.hpp:367: error: undefined reference to 'cv::Mat::deallocate()'
    C:/android/android-ndk-r8e/toolchains/arm-linux-androideabi-4.6/prebuilt/windows/bin/../lib/gcc/arm-linux-androideabi/4.6/../../../../arm-linux-androideabi/bin/ld.exe: ./obj/local/armeabi-v7a/objs/ImageProcessing/ImageProcessing.o: in function cv::Mat::~Mat():C:\opencv4android_v2.4.6\sdk\native\jni\include/opencv2/core/mat.hpp:278: error: undefined reference to 'cv::fastFree(void*)'
    C:/android/android-ndk-r8e/toolchains/arm-linux-androideabi-4.6/prebuilt/windows/bin/../lib/gcc/arm-linux-androideabi/4.6/../../../../arm-linux-androideabi/bin/ld.exe: ./obj/local/armeabi-v7a/objs/ImageProcessing/ImageProcessing.o: in function Java_my_project_MyRealTimeImageProcessing_CameraPreview_ImageProcessing:jni/ImageProcessing.cpp:31: error: undefined reference to 'cv::Mat::operator _IplImage() const'
    C:/android/android-ndk-r8e/toolchains/arm-linux-androideabi-4.6/prebuilt/windows/bin/../lib/gcc/arm-linux-androideabi/4.6/../../../../arm-linux-androideabi/bin/ld.exe: ./obj/local/armeabi-v7a/objs/ImageProcessing/ImageProcessing.o: in function Java_my_project_MyRealTimeImageProcessing_CameraPreview_ImageProcessing:jni/ImageProcessing.cpp:32: error: undefined reference to 'cv::Mat::operator _IplImage() const'
    C:/android/android-ndk-r8e/toolchains/arm-linux-androideabi-4.6/prebuilt/windows/bin/../lib/gcc/arm-linux-androideabi/4.6/../../../../arm-linux-androideabi/bin/ld.exe: ./obj/local/armeabi-v7a/objs/ImageProcessing/ImageProcessing.o: in function Java_my_project_MyRealTimeImageProcessing_CameraPreview_ImageProcessing:jni/ImageProcessing.cpp:33: error: undefined reference to 'cv::Mat::operator _IplImage() const'
    C:/android/android-ndk-r8e/toolchains/arm-linux-androideabi-4.6/prebuilt/windows/bin/../lib/gcc/arm-linux-androideabi/4.6/../../../../arm-linux-androideabi/bin/ld.exe: ./obj/local/armeabi-v7a/objs/ImageProcessing/ImageProcessing.o: in function Java_my_project_MyRealTimeImageProcessing_CameraPreview_ImageProcessing:jni/ImageProcessing.cpp:35: error: undefined reference to 'cvCanny'
    C:/android/android-ndk-r8e/toolchains/arm-linux-androideabi-4.6/prebuilt/windows/bin/../lib/gcc/arm-linux-androideabi/4.6/../../../../arm-linux-androideabi/bin/ld.exe: ./obj/local/armeabi-v7a/objs/ImageProcessing/ImageProcessing.o: in function Java_my_project_MyRealTimeImageProcessing_CameraPreview_ImageProcessing:jni/ImageProcessing.cpp:36: error: undefined reference to 'cvCvtColor'
    collect2: ld returned 1 exit status
    /cygdrive/c/android/android-ndk-r8e/build/core/build-binary.mk:450: recipe for target `obj/local/armeabi-v7a/libImageProcessing.so' failed
    make: *** [obj/local/armeabi-v7a/libImageProcessing.so] Error 1

    Apparently some includes are missing, but I have no idea where to find them. I use OpenCV4Android version 2.4.6 and Android NDK r8e. Have you got any idea how to fix it?

    ReplyDelete
    Replies
    1. I am not using OpenCV4Android, this is a new version.
      Maybe it use different interface.

      Delete
  7. Is there no other way to use opencv without jni or maybe write all functions in java?

    ReplyDelete
    Replies
    1. There are some people working on this.
      https://code.google.com/p/javacv/

      Delete
  8. 09-17 13:13:43.783: E/Trace(22413): error opening trace file: No such file or directory (2)
    09-17 13:13:43.783: D/ActivityThread(22413): setTargetHeapUtilization:0.25
    09-17 13:13:43.783: D/ActivityThread(22413): setTargetHeapIdealFree:8388608
    09-17 13:13:43.793: D/ActivityThread(22413): setTargetHeapConcurrentStart:2097152
    09-17 13:13:43.843: W/dalvikvm(22413): Exception Ljava/lang/UnsatisfiedLinkError; thrown while initializing Lmy/project/MyRealTimeImageProcessing/CameraPreview;
    09-17 13:13:43.843: D/AndroidRuntime(22413): Shutting down VM
    09-17 13:13:43.843: W/dalvikvm(22413): threadid=1: thread exiting with uncaught exception (group=0x40c11438)
    09-17 13:13:43.843: E/AndroidRuntime(22413): FATAL EXCEPTION: main
    09-17 13:13:43.843: E/AndroidRuntime(22413): java.lang.ExceptionInInitializerError
    09-17 13:13:43.843: E/AndroidRuntime(22413): at my.project.MyRealTimeImageProcessing.MyRealTimeImageProcessing.onCreate(MyRealTimeImageProcessing.java:39)
    09-17 13:13:43.843: E/AndroidRuntime(22413): at android.app.Activity.performCreate(Activity.java:5058)
    09-17 13:13:43.843: E/AndroidRuntime(22413): at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1079)
    09-17 13:13:43.843: E/AndroidRuntime(22413): at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2064)
    09-17 13:13:43.843: E/AndroidRuntime(22413): at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2132)
    09-17 13:13:43.843: E/AndroidRuntime(22413): at android.app.ActivityThread.access$600(ActivityThread.java:139)
    09-17 13:13:43.843: E/AndroidRuntime(22413): at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1231)
    09-17 13:13:43.843: E/AndroidRuntime(22413): at android.os.Handler.dispatchMessage(Handler.java:99)
    09-17 13:13:43.843: E/AndroidRuntime(22413): at android.os.Looper.loop(Looper.java:137)
    09-17 13:13:43.843: E/AndroidRuntime(22413): at android.app.ActivityThread.main(ActivityThread.java:5021)
    09-17 13:13:43.843: E/AndroidRuntime(22413): at java.lang.reflect.Method.invokeNative(Native Method)
    09-17 13:13:43.843: E/AndroidRuntime(22413): at java.lang.reflect.Method.invoke(Method.java:511)
    09-17 13:13:43.843: E/AndroidRuntime(22413): at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:789)
    09-17 13:13:43.843: E/AndroidRuntime(22413): at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:556)
    09-17 13:13:43.843: E/AndroidRuntime(22413): at dalvik.system.NativeStart.main(Native Method)
    09-17 13:13:43.843: E/AndroidRuntime(22413): Caused by: java.lang.UnsatisfiedLinkError: Couldn't load ImageProcessing: findLibrary returned null
    09-17 13:13:43.843: E/AndroidRuntime(22413): at java.lang.Runtime.loadLibrary(Runtime.java:365)
    09-17 13:13:43.843: E/AndroidRuntime(22413): at java.lang.System.loadLibrary(System.java:535)
    09-17 13:13:43.843: E/AndroidRuntime(22413): at my.project.MyRealTimeImageProcessing.CameraPreview.(CameraPreview.java:112)
    09-17 13:13:43.843: E/AndroidRuntime(22413): ... 15 more

    ReplyDelete
  9. hi, this is quite a good job you have done here;
    wanted to inquire whether there is away i can manage to take a picture with an android phone then save it as a binary image. kindly assist.

    ReplyDelete
    Replies
    1. This article may help you.
      http://ibuzzlog.blogspot.tw/2012/08/how-to-use-camera-in-android.html

      Delete
  10. Hi,

    I would like to clean the background of the captured image. Can you please throw some light, how can i achieve that. any library that supports it.

    ReplyDelete
    Replies
    1. Yes, OpenCV can do that!!!
      Keyword: grabcut

      Delete
    2. This comment has been removed by the author.

      Delete
    3. Thank for the help ikki... i was going through
      http://docs.opencv.org/trunk/doc/py_tutorials/py_imgproc/py_grabcut/py_grabcut.html

      but still having some issues...Can you suggest any sample app or program, or point to get started

      Delete
    4. maybe you can try his.
      http://hiankun.blogspot.tw/2013/08/try-grabcut-using-opencv.html

      Delete
  11. Is this code working? I've cloned the code and can't execute.

    ReplyDelete
    Replies
    1. Yes, it works. What is your error messages?

      Delete
  12. Hello.
    This article is very useful. And the most useful part is the source code.
    Can you add a list of all dependencies (what version of opencv it uses, etc.)?
    Thank you anyway.

    ReplyDelete
  13. Anonymous18/4/14 16:34

    Hi, This article is really good and understandable. Please help me about this.
    http://stackoverflow.com/questions/23148707/motion-analysis-and-object-tracking-in-opencv-for-android-to-move-ip-cam
    Please refer this link. How can i start this using opencv please let me know.
    Thank you.

    ReplyDelete
  14. Android NDK: WARNING: APP_PLATFORM android-19 is larger than android:minSdkVersion 8 in ./AndroidManifest.xml
    [armeabi] Compile++ thumb: ImageProcessing <= ImageProcessing.cpp
    jni/ImageProcessing.cpp:3:33: fatal error: opencv2/core/core.hpp: No such file or directory
    compilation terminated.
    make.exe: *** [obj/local/armeabi/objs/ImageProcessing/ImageProcessing.o] Error 1


    i Got this above error when i compile the code.Is it possible to get some answer
    please help me....

    ReplyDelete
  15. Hello there,

    First of all I must say thanks for this tutorial. Really great one.

    However I'm facing some issues I hope you can help me with. I'm converting your code to C# version ( xamarin for android ). After some work done code is buildable, and app installs on the phone. But somehow image view is not displayed at all.

    I must point out that I'm not calling JNI but simply filling int array with values depending on respective value in byte array. Which ever logic that I can use, it still should produce some result, right? For simplicity, let's say that int array contains only 0 and 1 values.

    Code is running well, no app crashes reported, every line is called, but image view is not displayed. On the other hand surface view properly displays stream from camera.

    I'd gladly share code if it'll help in solving this issue, but I hope that you can give me some hint on what I might be doing wrong, even without looking at code.

    Thanks in advance.

    ReplyDelete
    Replies
    1. I think you can try put all 0 to pixels array. Camera preview should be black.
      If not, maybe some thing different in C# Android develop.

      bitmap.setPixels(pixels, 0, PreviewSizeWidth, 0, 0, PreviewSizeWidth, PreviewSizeHeight);
      MyCameraPreview.setImageBitmap(bitmap);

      Delete
  16. Hello i'm having an issue with analyzing the frame that comes from the camera , well i've developped my native part that analyze the frame and get you the match but my real problem is how to get the frame to this function because i can not pass an image as a parameter to the native part

    ReplyDelete
    Replies
    1. In this project the frame will be put on the global variable FrameData.

      Delete
  17. This comment has been removed by the author.

    ReplyDelete
  18. hello there.
    your above code is working well.
    have you receive the color image for NDK?!

    like this:
    Mat mGray(height, width, CV_8UC4, (unsigned char *)pNV21FrameData);

    when i try to receive the color image from android to NDK file.
    the image is crapped and not good.
    please, advise me. thank you.

    ReplyDelete
    Replies
    1. The format of the input image pNV21FrameData is YUV420.
      Maybe you can try.
      Mat yuvImg(height * 2 / 3, width, CV_8UC1, (unsigned char *)pNV21FrameData);
      Mat rgbImg;
      cvCvtColor(&yuvImg, &rgbImg, CV_YUV2BGR_I420);

      Delete
    2. ok. thank you so much for your reply.
      i have the new question for your code.
      the output of NDK code is boolean.
      but actually we need the output image data such as RGB ro NV21 data.
      My question is how to receive the image data from NDK code to java.
      the image is displayed by java code.
      thank you~~

      Delete
    3. For more good advice, i am explaining my project.
      i want to real time image processing with color image.
      color image-> image processing-> color image.
      for these work, NDK receive and give the color image.
      the gray image is working well(your code)
      but color image is not good. would you help?
      thank you.

      Delete
    4. The output image is stored in outPixels. You can put your color image in this array.

      Delete
  19. Hello,

    really great work there. can we compare the preview image data with the image in gallery?

    ReplyDelete
    Replies
    1. what's the image in gallery you want to compare? This demo shows a realtime preview image.

      Delete
  20. I am trying to do real time image processing using your code, to increase the brightness of the image from camera in real time.But my app is crashing when m trying to run it.Please help me as its very urgent for my project.

    ReplyDelete
  21. It will be really helpful if you look into it.

    ReplyDelete
  22. Even when i am trying to run your exact code in a new android application its stopping.The application shows the image (unprocessed) for 1 sec and displays the message "the application has unfortunately stopped" and quits.I am totally new to this.Please help me out to figure this.

    ReplyDelete
  23. I am getting this when I am trying to debug the code

    01-24 19:28:37.559: D/dalvikvm(20748): threadid=1: still suspended after undo (sc=1 dc=1)
    01-24 19:30:53.316: D/dalvikvm(20748): Trying to load lib /data/app-lib/com.official.Try-1/libImageProcessing.so 0x41f48268
    01-24 19:30:53.328: D/dalvikvm(20748): Added shared lib /data/app-lib/com.official.Try-1/libImageProcessing.so 0x41f48268
    01-24 19:30:53.329: D/dalvikvm(20748): No JNI_OnLoad found in /data/app-lib/com.official.Try-1/libImageProcessing.so 0x41f48268, skipping init
    01-24 19:32:30.579: D/AndroidRuntime(20748): Shutting down VM
    01-24 19:32:30.579: W/dalvikvm(20748): threadid=1: thread exiting with uncaught exception (group=0x4167bd40)
    01-24 19:32:30.622: E/AndroidRuntime(20748): FATAL EXCEPTION: main
    01-24 19:32:30.622: E/AndroidRuntime(20748): Process: com.official.Try, PID: 20748
    01-24 19:32:30.622: E/AndroidRuntime(20748): java.lang.RuntimeException: Unable to pause activity {com.official.Try/com.official.Try.MyRealTimeImageProcessing}: java.lang.NullPointerException
    01-24 19:32:30.622: E/AndroidRuntime(20748): at android.app.ActivityThread.performPauseActivity(ActivityThread.java:3129)
    01-24 19:32:30.622: E/AndroidRuntime(20748): at android.app.ActivityThread.performPauseActivity(ActivityThread.java:3072)
    01-24 19:32:30.622: E/AndroidRuntime(20748): at android.app.ActivityThread.handlePauseActivity(ActivityThread.java:3050)
    01-24 19:32:30.622: E/AndroidRuntime(20748): at android.app.ActivityThread.access$1000(ActivityThread.java:139)
    01-24 19:32:30.622: E/AndroidRuntime(20748): at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1221)
    01-24 19:32:30.622: E/AndroidRuntime(20748): at android.os.Handler.dispatchMessage(Handler.java:102)
    01-24 19:32:30.622: E/AndroidRuntime(20748): at android.os.Looper.loop(Looper.java:136)
    01-24 19:32:30.622: E/AndroidRuntime(20748): at android.app.ActivityThread.main(ActivityThread.java:5086)
    01-24 19:32:30.622: E/AndroidRuntime(20748): at java.lang.reflect.Method.invokeNative(Native Method)
    01-24 19:32:30.622: E/AndroidRuntime(20748): at java.lang.reflect.Method.invoke(Method.java:515)
    01-24 19:32:30.622: E/AndroidRuntime(20748): at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:785)
    01-24 19:32:30.622: E/AndroidRuntime(20748): at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:601)
    01-24 19:32:30.622: E/AndroidRuntime(20748): at dalvik.system.NativeStart.main(Native Method)
    01-24 19:32:30.622: E/AndroidRuntime(20748): Caused by: java.lang.NullPointerException
    01-24 19:32:30.622: E/AndroidRuntime(20748): at com.official.Try.CameraPreview.onPause(CameraPreview.java:57)
    01-24 19:32:30.622: E/AndroidRuntime(20748): at com.official.Try.MyRealTimeImageProcessing.onPause(MyRealTimeImageProcessing.java:52)
    01-24 19:32:30.622: E/AndroidRuntime(20748): at android.app.Activity.performPause(Activity.java:5352)
    01-24 19:32:30.622: E/AndroidRuntime(20748): at android.app.Instrumentation.callActivityOnPause(Instrumentation.java:1256)
    01-24 19:32:30.622: E/AndroidRuntime(20748): at android.app.ActivityThread.performPauseActivity(ActivityThread.java:3109)
    01-24 19:32:30.622: E/AndroidRuntime(20748): ... 12 more
    01-24 19:32:30.643: I/Process(20748): Sending signal. PID: 20748 SIG: 9



    Please help me out.I am not understanding what is going wrong.

    ReplyDelete
    Replies
    1. According to these codes.
      -----------------------------------------
      Caused by: java.lang.NullPointerException
      at com.official.Try.CameraPreview.onPause(CameraPreview.java:57)
      -----------------------------------------
      I think you add some codes in CameraPreview.java. It caused some thing become NULL in OnPause() fundction.
      May you post you whole CameraPreview.java ?


      Delete
  24. Its the same CameraPreview.java class of your MyRealTimeImageProcessing android project.I did not change anything.I was trying to run your exact code by replicating your project in a new android project Try and all these errors are coming.But when I import your original project its working perfectly fine without any errors.I don’t know what’s the problem.
    Please let me know if you need to know anything regarding this.Thank You.

    ReplyDelete
    Replies
    1. I think you should check the AndroidManifest.xml, And make should the camera has permission.
      And check the line 57 in CameraPreview.java in you project, there is a null object.

      Delete
    2. This is my AndroidManifest.xml























      ----------------------------------------------------

      And the line 57 in CameraPreview.java is

      55: public void onPause()
      56: {
      57: mCamera.stopPreview();
      58: }


      The code is exact same as your code.

      Delete
    3. This comment has been removed by the author.

      Delete
    4. The camera has permission in AndroidManifest.xml

      Delete
    5. The mCamera in your project is NULL, I think the Camera.open() is failed. You maybe check the result why it being NULL.

      Delete
    6. Hello,
      Thank you,I corrected my NULL error.Now the mCamera object is being created and I am getting the camera View also while debugging but the code is crashing when the ImageProcessing is about to happen.
      This is the Log from debugging


      01-26 02:08:50.741: I/System.out(19083): debugger has settled (1333)
      01-26 02:09:03.405: D/dalvikvm(19083): Trying to load lib /data/app-lib/com.official.Try-2/libImageProcessing.so 0x41f45718
      01-26 02:09:03.408: D/dalvikvm(19083): Added shared lib /data/app-lib/com.official.Try-2/libImageProcessing.so 0x41f45718
      01-26 02:09:03.409: D/dalvikvm(19083): No JNI_OnLoad found in /data/app-lib/com.official.Try-2/libImageProcessing.so 0x41f45718, skipping init
      01-26 02:09:26.242: I/Adreno-EGL(19083): : EGL 1.4 QUALCOMM build: AU_LINUX_ANDROID_LNX.LA.3.5.1_RB1.04.04.02.048.018_msm8226_LNX.LA.3.5.1_RB1__release_AU ()
      01-26 02:09:26.242: I/Adreno-EGL(19083): OpenGL ES Shader Compiler Version: E031.24.00.08
      01-26 02:09:26.242: I/Adreno-EGL(19083): Build Date: 03/07/14 Fri
      01-26 02:09:26.242: I/Adreno-EGL(19083): Local Branch:
      01-26 02:09:26.242: I/Adreno-EGL(19083): Remote Branch: quic/LNX.LA.3.5.1_RB1.1
      01-26 02:09:26.242: I/Adreno-EGL(19083): Local Patches: NONE
      01-26 02:09:26.242: I/Adreno-EGL(19083): Reconstruct Branch: AU_LINUX_ANDROID_LNX.LA.3.5.1_RB1.04.04.02.048.018 + f2fd134 + NOTHING
      01-26 02:09:26.284: D/OpenGLRenderer(19083): Enabling debug mode 0
      01-26 02:10:47.241: I/Choreographer(19083): Skipped 4860 frames! The application may be doing too much work on its main thread.
      01-26 02:10:55.444: I/Choreographer(19083): Skipped 486 frames! The application may be doing too much work on its main thread.
      01-26 02:11:03.137: I/MyRealTimeImageProcessing(19083): DoImageProcessing():
      01-26 02:11:04.591: W/dalvikvm(19083): No implementation found for native Lcom/official/Try/CameraPreview;.ImageProcessing:(II[B[I)Z
      01-26 02:11:09.430: D/AndroidRuntime(19083): Shutting down VM
      01-26 02:11:09.430: W/dalvikvm(19083): threadid=1: thread exiting with uncaught exception (group=0x4167bd40)
      01-26 02:11:09.443: E/AndroidRuntime(19083): FATAL EXCEPTION: main
      01-26 02:11:09.443: E/AndroidRuntime(19083): Process: com.official.Try, PID: 19083
      01-26 02:11:09.443: E/AndroidRuntime(19083): java.lang.UnsatisfiedLinkError: Native method not found: com.official.Try.CameraPreview.ImageProcessing:(II[B[I)Z
      01-26 02:11:09.443: E/AndroidRuntime(19083): at com.official.Try.CameraPreview.ImageProcessing(Native Method)
      01-26 02:11:09.443: E/AndroidRuntime(19083): at com.official.Try.CameraPreview$1.run(CameraPreview.java:137)
      01-26 02:11:09.443: E/AndroidRuntime(19083): at android.os.Handler.handleCallback(Handler.java:733)
      01-26 02:11:09.443: E/AndroidRuntime(19083): at android.os.Handler.dispatchMessage(Handler.java:95)
      01-26 02:11:09.443: E/AndroidRuntime(19083): at android.os.Looper.loop(Looper.java:136)
      01-26 02:11:09.443: E/AndroidRuntime(19083): at android.app.ActivityThread.main(ActivityThread.java:5086)
      01-26 02:11:09.443: E/AndroidRuntime(19083): at java.lang.reflect.Method.invokeNative(Native Method)
      01-26 02:11:09.443: E/AndroidRuntime(19083): at java.lang.reflect.Method.invoke(Method.java:515)
      01-26 02:11:09.443: E/AndroidRuntime(19083): at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:785)
      01-26 02:11:09.443: E/AndroidRuntime(19083): at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:601)
      01-26 02:11:09.443: E/AndroidRuntime(19083): at dalvik.system.NativeStart.main(Native Method)


      Thanks for all your help.I am trying to do this on my own but without your help I had lost all my hope to debug it.I am Grateful to you for your help.

      Delete
    7. Can you please suggest links and sources(book names) to study further on Real Time Image Processing Using Opencv JNI in Android topic.I am really interested to proceed in this field and work on it.Thank You

      Delete
    8. The new error log.
      ---------------------
      java.lang.UnsatisfiedLinkError: Native method not found: com.official.Try.CameraPreview.ImageProcessing:(II[B[I)Z
      at com.official.Try.CameraPreview.ImageProcessing(Native Method)
      ---------------------
      It complaint that can not find the function ImageProcessing.
      Maybe you can upload your project to some cloud storage. I can try to compile it.
      My sources is all on line and Google it. It take time to learn it.

      Delete
    9. I will be really great-full for that.But I don't have any cloud storage to upload my project.This is my email id dawnqueen.03@gmail.com ,Please mail me on this id so that i can send my project as attachment to you in reply.I am really thank full to you for your support and interest in helping me out.

      Delete
  25. Thank you for a great tutorial. But I have a question about using Handler when processing image. Docs say that with post() "The runnable will be run on the thread to which this handler is attached". So heavy operation of image processing will be executed in main thread - is it planned? Or should we process image in a separate thread and use post() only to update MyCameraPreview?

    ReplyDelete
    Replies
    1. Yes, it is better doing heavy processing in separate thread, using main thread in the post() to update UI.

      Delete
  26. Hi thank u for the tutorial. The code working perfect. I doing image processing project. I have a HTC EVO 3d mobile with two camera in back side. i need to pass a left camera data and right camera data into JNI and process in jni and need to display the result in camera preview. i need everything in realtime preview. pls help me. Thanks in advance Emailid: vinom1389@gmail.com

    ReplyDelete
    Replies
    1. And what's the problem? Can you get the Left/Right images from camera?

      Delete
  27. i have splitted the left and right image using opencv library.
    Imgproc.cvtColor(img, gray, Imgproc.COLOR_YUV420sp2RGB);
    Imgproc.cvtColor(img, rgba, Imgproc.COLOR_YUV420sp2RGBA);

    now i need to send the Mat Img and Mat gray to JNI and process in JNI return the result in camera preview. Thanks in advance

    ReplyDelete
    Replies
    1. So, you are using JAVA OpenCV library?

      Delete
    2. yes. currently am doing the splitting process in button click. when the push button clicked in the camera frame the image captured and left and right image splitted and saved in external storage. i need to do pass the left and right image to JNI and display the result in camera preview like your canny image processing example.
      Thanks.

      Delete
    3. Why not just processing image in JAVA? If you want pass image data to JNI, the best way is using the original data(YUV420).

      Delete
    4. am doing the stereo matching project using JNI that's why am using JNI. you said like send a original frame data to JNI and split and process in JNI and display the result to camera preview right.? how to split the left and right image in JNI.

      Delete
    5. How do you split in JAVA? JAVA code? or OpenCV code?

      Delete
    6. Why it can do in JAVA OpenCV, but can not do in JNI OpenCV? Do you have native OpenCV environment? same as my example.

      Delete
    7. yes am having OpenCv environment same as your example.
      Am using Eclipse Juno, NDK, CYGWIN GCC Complier.

      Delete
    8. If you have native opencv and will use it to do the stereo matching, i think the simple procedure is pass the image to JNI and get result from it.
      I am curious, how do you split the stereo images from 3D camera? each camera has single image, or it marge to one image?

      Delete
    9. am using the HTC SDK to split the left and right image.
      i found in this below link
      http://www.htcdev.com/devcenter/opensense-sdk/stereoscopic-3d/s3d-sample-code/

      Delete
    10. The result is RGBA8888 bitmap image.I think the simple way is pass it to JNI and convert to Mat.

      Delete
    11. ok thank u. i will work on it and come back to u. if u find any solution mail me. vinom1389@gmail.com
      Thanks in advance.

      Delete
  28. This comment has been removed by the author.

    ReplyDelete
  29. Hello !! I want to do real time image processing in Android. I use camera2 with API21 or 22. do you can to help me ??

    ReplyDelete
  30. Hi, many thanks for the tut.
    Do you need Android NDK to build that app ? Thanks.

    ReplyDelete
  31. Thank you for sharing very useful information.

    Videocon Telecom has announced its aggressive growth plans to become a preferred National Operator. As part of its network expansion drive, the telco announced 2489 new cell sites and 231 new exclusive stores in existing circles.

    ReplyDelete
  32. Hi the code is work fine.but what i dont want to use camera and i have image in drawable i want to pass that image to jni and get image from jni to display in Imageview with some processing.can u just help me.

    Thannks in Advance

    ReplyDelete
  33. hello, i need to do this title for my final year project, i'm already follow all your steps n download the zip file on github, but i'm still can't do this project, can u help me? :(

    ReplyDelete
  34. Hello all,

    I want to get person object from photo and show. Any idea how can I do it using OpenCV?

    ReplyDelete
  35. Hi,

    I am getting the below error. Could you please help me.


    04-16 13:34:05.021 8197-8197/my.project.MyRealTimeImageProcessing D/dalvikvm﹕ GetMethodID: not returning static method Landroid/os/Process;.getTotalMemory ()J
    04-16 13:34:05.021 8197-8197/my.project.MyRealTimeImageProcessing D/dalvikvm﹕ GetMethodID: not returning static method Landroid/os/Process;.getFreeMemory ()J
    04-16 13:34:05.091 8197-8197/my.project.MyRealTimeImageProcessing W/dalvikvm﹕ Exception Ljava/lang/UnsatisfiedLinkError; thrown while initializing Lmy/project/MyRealTimeImageProcessing/CameraPreview;
    04-16 13:34:05.091 8197-8197/my.project.MyRealTimeImageProcessing D/AndroidRuntime﹕ Shutting down VM
    04-16 13:34:05.091 8197-8197/my.project.MyRealTimeImageProcessing W/dalvikvm﹕ threadid=1: thread exiting with uncaught exception (group=0x41de3d58)
    04-16 13:34:05.101 8197-8197/my.project.MyRealTimeImageProcessing E/AndroidRuntime﹕ FATAL EXCEPTION: main
    Process: my.project.MyRealTimeImageProcessing, PID: 8197
    java.lang.UnsatisfiedLinkError: Couldn't load ImageProcessing from loader dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/my.project.MyRealTimeImageProcessing-1.apk"],nativeLibraryDirectories=[/data/app-lib/my.project.MyRealTimeImageProcessing-1, /vendor/lib, /system/lib]]]: findLibrary returned null
    at java.lang.Runtime.loadLibrary(Runtime.java:358)
    at java.lang.System.loadLibrary(System.java:526)
    at my.project.MyRealTimeImageProcessing.CameraPreview.(CameraPreview.java:111)
    at my.project.MyRealTimeImageProcessing.MyRealTimeImageProcessing.onCreate(MyRealTimeImageProcessing.java:39)
    at android.app.Activity.performCreate(Activity.java:5248)
    at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1087)
    at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2164)
    at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2249)
    at android.app.ActivityThread.access$800(ActivityThread.java:141)
    at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1212)
    at android.os.Handler.dispatchMessage(Handler.java:102)
    at android.os.Looper.loop(Looper.java:136)
    at android.app.ActivityThread.main(ActivityThread.java:5052)
    at java.lang.reflect.Method.invokeNative(Native Method)
    at java.lang.reflect.Method.invoke(Method.java:515)
    at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:793)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:609)
    at dalvik.system.NativeStart.main(Native Method)
    04-16 13:34:05.401 8197-8197/my.project.MyRealTimeImageProcessing I/Process﹕ Sending signal. PID: 8197 SIG: 9

    ReplyDelete
  36. Hi
    And if I need to do a heavier processing, type detect an A4 sheet at runtime for example

    ReplyDelete
  37. Hi, Have you used opencv with react native?

    ReplyDelete
  38. Hi,
    can you tell me how to pass the the image array to detectmultiscale (.....);......?
    how to do face detection using android camera preview ( not using openCV camerabridgeviewbase) using jni.

    ReplyDelete
  39. Hi,
    Nice work !! Ikki Chung :)
    We got a requirement on indicating the LED's status on the preview screen.
    Requirement goes like this there are 4 LED's as you place the camera in front of LED's its status should be shown on the camera preview screen.

    Example
    LED-1 : On
    LED-2 : On
    LED-3 : Off
    LED-4 : Off

    Note: These LED's turn On & Off and regular intervals of time(with in one second).

    How can I do this task. Can you throw some light

    ReplyDelete
  40. Congratulations guys, quality information you have given!!!..Its really useful blog. Thanks for sharing this useful information
    Best Android Training in Velachery | android development course fees in chennai

    ReplyDelete
  41. This information is impressive; I am inspired with your post writing style & how continuously you describe this topic. After reading your post, thanks for taking the time to discuss this, I feel happy about it and I love learning more about this topic.android development course fees in chennai | android app development training in chennai

    ReplyDelete