This post explains how to create in android a feature that lets user organize photos by its content. The feature can be built using Firebase ML kit image labeling API.
To show how to create the feature which categorizes photos based on the content present on the picture, let’s develop an example app which allows user to start the process of extracting the primary content type from the limited number of images present in the photos gallery on the phone, store the image path and content type data in the Firestore database.
The main screen displays the content types in recycler view. On clicking a type, app fetches from the database the list of image paths for the selected content type and displays them in recycler view.
Since we use Firebase ML kit image labeling API, let’s see how it works. Using Firebase ML kit image labeling API, you can find out what content, for example paper, chair, flower, bird, etc.., present on a given picture.
First get the bitmap of an image which needs to be in upright position. Since image labeling API can recognize content on 480x360 size images, it is better to resize larger images to improve the performance.
Then create FirebaseVisionImage object by passing bitmap to FirebaseVisionImage.fromBitmap() method.
FirebaseVisionImage image = FirebaseVisionImage.fromBitmap(bitmap);Then create FirebaseVisionImageLabeler object by calling getOnDeviceImageLabeler() method on FirebaseVision instance. If you want to use cloud version of the api, you need to call getCloudImageLabeler() method.
FirebaseVisionImageLabeler labeler = FirebaseVision.getInstance()
.getOnDeviceImageLabeler();Then call processImage() method on FirebaseVisionImageLabeler object passing FirebaseVisionImage object to it.
Task t = labeler.processImage(image)Then register listeners to the task. Success listener’s onSuccess method receives list of FirebaseVisionImageLabel objects which can be used to get confidence of the label and label text.
t.addOnSuccessListener(new OnSuccessListener<List<FirebaseVisionImageLabel>>() {
@Override
public void onSuccess(List<FirebaseVisionImageLabel> labels) {
//do somthing
}
})
.addOnFailureListener(new OnFailureListener() {
@Override
public void onFailure(@NonNull Exception e) {
//do somthing
}
});Since the app uses Firebase, Firebase setup needs to be done for the project. For steps, please follow firebase setup instructions.
Then add libraries to gradle build files.
implementation 'androidx.recyclerview:recyclerview:1.0.0'
implementation 'com.google.firebase:firebase-core:17.0.0'
implementation 'com.google.firebase:firebase-ml-vision:22.0.0'
implementation 'com.google.firebase:firebase-ml-vision-image-label-model:18.0.0'
implementation 'com.google.android.material:material:1.0.0-alpha1'
implementation 'com.google.firebase:firebase-firestore:20.2.0'
The example has two activities. Main activity provides the option for users to start categorizing photos in the gallery and store the processed data in database. If photos are already categorized and data is available, then the main activity displays the list of content types in recycler view allowing users to click a particular content type to view photos in that type.
Second activity gets the content type that user selected, fetches data from database and displays images in recycler view.
It uses utility class to label images. It contains code to get list of content types for which pictures are available and display them in recycler view.
import android.content.pm.PackageManager;
import android.os.Bundle;
import android.util.Log;
import android.view.View;
import androidx.annotation.NonNull;
import androidx.appcompat.app.AppCompatActivity;
import androidx.core.app.ActivityCompat;
import androidx.core.content.ContextCompat;
import androidx.recyclerview.widget.DividerItemDecoration;
import androidx.recyclerview.widget.LinearLayoutManager;
import androidx.recyclerview.widget.RecyclerView;
import com.google.android.gms.tasks.OnCompleteListener;
import com.google.android.gms.tasks.Task;
import com.google.firebase.firestore.FirebaseFirestore;
import com.google.firebase.firestore.QuerySnapshot;
import com.zoftino.imagecontentfilter.background.ImageContentLabelManager;
import com.zoftino.imagecontentfilter.background.ImageContentLabelTask;
import java.util.List;
public class MainActivity extends AppCompatActivity {
private static final int PHOTOS_LIMIT = 100;
private static final String READ_PERM =
"android.permission.READ_EXTERNAL_STORAGE";
private RecyclerView rv;
private FirebaseFirestore firestoreDB;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
//permission to get photos
if (ContextCompat.checkSelfPermission(this, READ_PERM)
!= PackageManager.PERMISSION_GRANTED) {
ActivityCompat.requestPermissions(this,
new String[]{READ_PERM}, 2);
}
setContentView(R.layout.activity_main);
firestoreDB = FirebaseFirestore.getInstance();
rv = findViewById(R.id.content_type_rv);
LinearLayoutManager recyclerLayoutManager =
new LinearLayoutManager(this);
rv.setLayoutManager(recyclerLayoutManager);
DividerItemDecoration dividerItemDecoration =
new DividerItemDecoration(this,
recyclerLayoutManager.getOrientation());
dividerItemDecoration.setDrawable(getResources()
.getDrawable(android.R.drawable.divider_horizontal_bright, null));
rv.addItemDecoration(dividerItemDecoration);
//get image categories from firestore database
//display image categories in recyclerview
getImageCategories();
}
//process photos from gallery
//categorize them baesd on the content type
//store content type data in firestore db
public void categorizePhotos(View v) {
List<String> imagesUrls = ImageLabelUtil.getCameraImages(this, PHOTOS_LIMIT);
for (String imgUrl : imagesUrls) {
ImageContentLabelTask imageLabelTask = new ImageContentLabelTask(imgUrl);
ImageContentLabelManager.getImageContentManager().labelImageContent(imageLabelTask);
}
}
//get photo content types from db
//already processed data
private void getImageCategories() {
firestoreDB.collection("PhotosCat")
.get()
.addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
@Override
public void onComplete(@NonNull Task<QuerySnapshot> task) {
if (task.isSuccessful()) {
if (task.isSuccessful()) {
List<ImageLabel> typesList =
task.getResult().toObjects(ImageLabel.class);
//display content types in recycler view
PhotoContentTypeRVAdapter contentAdapter = new
PhotoContentTypeRVAdapter(typesList, MainActivity.this);
rv.setAdapter(contentAdapter);
} else {
Log.d("Get Content Types", "Error getting content types");
}
} else {
Log.d("Get Content Types", "Error getting content types");
}
}
});
}
}
<?xml version="1.0" encoding="utf-8"?>
<androidx.constraintlayout.widget.ConstraintLayout
xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:layout_margin="8dp"
tools:context=".MainActivity">
<Button
android:id="@+id/add_prd"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_marginTop="8dp"
android:text="Label 100 Photos from Gallery"
app:layout_constraintLeft_toLeftOf="parent"
app:layout_constraintRight_toRightOf="parent"
app:layout_constraintTop_toTopOf="parent"
android:onClick="categorizePhotos"
style="@style/Widget.AppCompat.Button.Colored"/>
<TextView
android:id="@+id/choose_txt"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_marginTop="8dp"
android:text="Click Content Type to View Photos"
android:textAppearance="@style/TextAppearance.AppCompat.Headline"
app:layout_constraintEnd_toEndOf="parent"
app:layout_constraintStart_toStartOf="parent"
app:layout_constraintTop_toBottomOf="@+id/add_prd" />
<androidx.recyclerview.widget.RecyclerView
android:id="@+id/content_type_rv"
android:scrollbars="vertical"
android:layout_width="match_parent"
android:layout_height="500dp"
app:layout_constraintBottom_toBottomOf="parent"
app:layout_constraintLeft_toLeftOf="parent"
app:layout_constraintRight_toRightOf="parent"
app:layout_constraintTop_toBottomOf="@+id/choose_txt" />
</androidx.constraintlayout.widget.ConstraintLayout>
Recycler view adapter for displaying image content types and starting view image activity.
import android.content.Context;
import android.content.Intent;
import android.view.LayoutInflater;
import android.view.View;
import android.view.ViewGroup;
import android.widget.TextView;
import androidx.recyclerview.widget.RecyclerView;
import java.util.List;
public class PhotoContentTypeRVAdapter extends
RecyclerView.Adapter<PhotoContentTypeRVAdapter.ViewHolder> {
private List<ImageLabel> typeList;
private Context context;
public PhotoContentTypeRVAdapter(List<ImageLabel> list, Context ctx) {
typeList = list;
context = ctx;
}
@Override
public int getItemCount() {
return typeList.size();
}
@Override
public PhotoContentTypeRVAdapter.ViewHolder
onCreateViewHolder(ViewGroup parent, int viewType) {
View view = LayoutInflater.from(parent.getContext())
.inflate(R.layout.photo_content_type_item, parent, false);
PhotoContentTypeRVAdapter.ViewHolder viewHolder =
new PhotoContentTypeRVAdapter.ViewHolder(view);
return viewHolder;
}
@Override
public void onBindViewHolder(PhotoContentTypeRVAdapter.ViewHolder holder, int position) {
final ImageLabel type = typeList.get(position);
holder.textView.setText(type.getImageContentType());
holder.textView.setOnClickListener(new View.OnClickListener() {
@Override
public void onClick(View view) {
//get the selected content type and pass it to activity
getPicturePathsByContentType(type.getImageContentType());
}
});
}
public class ViewHolder extends RecyclerView.ViewHolder {
public TextView textView;
public ViewHolder(View view) {
super(view);
textView = view.findViewById(R.id.photo_content_type);
}
}
//start activity which displays photos in recycler view
private void getPicturePathsByContentType(String type){
Intent i = new Intent();
i.putExtra("contentType", type);
i.setClass(context, PhotosByContentActivity.class);
context.startActivity(i);
}
}
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:orientation="vertical">
<TextView
android:id="@+id/photo_content_type"
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:layout_margin="16dp"
android:textAppearance="@style/TextAppearance.AppCompat.Headline"/>
</LinearLayout>
Activity gets the selected content type from the intent, gets the list of image paths from database, resizes them and displays in recycler view.
package com.zoftino.imagecontentfilter;
import android.os.Bundle;
import android.util.Log;
import android.widget.TextView;
import androidx.annotation.NonNull;
import androidx.appcompat.app.AppCompatActivity;
import androidx.recyclerview.widget.DividerItemDecoration;
import androidx.recyclerview.widget.LinearLayoutManager;
import androidx.recyclerview.widget.RecyclerView;
import com.google.android.gms.tasks.OnCompleteListener;
import com.google.android.gms.tasks.Task;
import com.google.firebase.firestore.FirebaseFirestore;
import com.google.firebase.firestore.QuerySnapshot;
import java.util.List;
public class PhotosByContentActivity extends AppCompatActivity {
private RecyclerView rv;
private FirebaseFirestore firestoreDB;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_photos);
firestoreDB = FirebaseFirestore.getInstance();
rv = findViewById(R.id.photos_rv);
LinearLayoutManager recyclerLayoutManager =
new LinearLayoutManager(this);
rv.setLayoutManager(recyclerLayoutManager);
DividerItemDecoration dividerItemDecoration =
new DividerItemDecoration(this,
recyclerLayoutManager.getOrientation());
dividerItemDecoration.setDrawable(getResources()
.getDrawable(android.R.drawable.divider_horizontal_dim_dark, null));
rv.addItemDecoration(dividerItemDecoration);
//get user selected content type from the intent
String contentType = getIntent().getStringExtra("contentType");
TextView tv = findViewById(R.id.photos_txt);
tv.setText(contentType+" Photos from Gallry");
//display list of photos from the selected type in recycler view
getImageInfo(contentType);
}
//get photo paths from database for the selected content type
private void getImageInfo(String imageLabel) {
firestoreDB.collection("PhotosInfo").document("myPics")
.collection(imageLabel)
.get()
.addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
@Override
public void onComplete(@NonNull Task<QuerySnapshot> task) {
if (task.isSuccessful()) {
List<ImageLabel> imagesInfo = task.getResult()
.toObjects(ImageLabel.class);
//display the selected photos in recycler view
PhotosRecyclerViewAdapter photoAdapter = new
PhotosRecyclerViewAdapter(imagesInfo,
PhotosByContentActivity.this);
rv.setAdapter(photoAdapter);
} else {
Log.d("Get photos by type", "Error getting photos info");
}
}
});
}
}
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:orientation="vertical">
<TextView
android:id="@+id/photos_txt"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_margin="8dp"
android:text="Photos"
android:textAppearance="@style/TextAppearance.AppCompat.Headline"/>
<androidx.recyclerview.widget.RecyclerView
android:id="@+id/photos_rv"
android:scrollbars="vertical"
android:layout_width="match_parent"
android:layout_height="match_parent"/>
</LinearLayout>
import android.content.Context;
import android.graphics.Bitmap;
import android.view.LayoutInflater;
import android.view.View;
import android.view.ViewGroup;
import android.widget.ImageView;
import androidx.recyclerview.widget.RecyclerView;
import java.util.List;
public class PhotosRecyclerViewAdapter extends
RecyclerView.Adapter<PhotosRecyclerViewAdapter.ViewHolder> {
private List<ImageLabel> imageLst;
private Context context;
public PhotosRecyclerViewAdapter(List<ImageLabel> list, Context ctx) {
imageLst = list;
context = ctx;
}
@Override
public int getItemCount() {
return imageLst.size();
}
@Override
public PhotosRecyclerViewAdapter.ViewHolder
onCreateViewHolder(ViewGroup parent, int viewType) {
View view = LayoutInflater.from(parent.getContext())
.inflate(R.layout.photo_item, parent, false);
PhotosRecyclerViewAdapter.ViewHolder viewHolder =
new PhotosRecyclerViewAdapter.ViewHolder(view);
return viewHolder;
}
@Override
public void onBindViewHolder(PhotosRecyclerViewAdapter.ViewHolder holder, int position) {
final ImageLabel imagePath = imageLst.get(position);
holder.image.setImageBitmap(processImageForDisplay(imagePath.getImagePath()));
}
public class ViewHolder extends RecyclerView.ViewHolder {
public ImageView image;
public ViewHolder(View view) {
super(view);
image = view.findViewById(R.id.photo);
}
}
//resize image and display as recycler view item
public Bitmap processImageForDisplay(String imagUrl){
Bitmap bmp = ImageLabelUtil.getUprightImage(imagUrl);
return ImageLabelUtil.resizeImage(bmp, context);
}
}
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:orientation="vertical">
<ImageView
android:id="@+id/photo"
android:layout_width="match_parent"
android:layout_height="200dp"
android:layout_margin="8dp"/>
</LinearLayout>This class contains code to resize and upright images, label images and save data in database.
import android.app.Activity;
import android.content.Context;
import android.database.Cursor;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.graphics.Matrix;
import android.media.ExifInterface;
import android.os.Environment;
import android.provider.MediaStore;
import android.util.DisplayMetrics;
import android.util.Log;
import androidx.annotation.NonNull;
import com.google.android.gms.tasks.OnFailureListener;
import com.google.android.gms.tasks.OnSuccessListener;
import com.google.android.gms.tasks.Task;
import com.google.firebase.firestore.DocumentReference;
import com.google.firebase.firestore.FirebaseFirestore;
import com.google.firebase.ml.vision.FirebaseVision;
import com.google.firebase.ml.vision.common.FirebaseVisionImage;
import com.google.firebase.ml.vision.label.FirebaseVisionImageLabel;
import com.google.firebase.ml.vision.label.FirebaseVisionImageLabeler;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
public class ImageLabelUtil {
private static int IMAGE_HEIGHT = 360;
private static int IMAGE_WIDTH = 480;
private static FirebaseFirestore firestoreDB = FirebaseFirestore.getInstance();
//get paths of images from the gallery,
//restrict the number of item to the limit given
public static List<String> getCameraImages(Context context, int limit) {
final String CAMERA_IMAGES = Environment
.getExternalStorageDirectory().toString() + "/DCIM/Camera";
final String CAMERA_IMAGES_ID = String.valueOf(
CAMERA_IMAGES.toLowerCase().hashCode());
final String[] projection = {MediaStore.Images.Media.DATA};
final String selection = MediaStore.Images.Media.BUCKET_ID + " = ?";
final String[] selectionArgs = {CAMERA_IMAGES_ID};
final Cursor cursor = context.getContentResolver()
.query(MediaStore.Images.Media.EXTERNAL_CONTENT_URI,
projection,
selection,
selectionArgs,
null);
ArrayList<String> result = new ArrayList<String>(cursor.getCount());
if (cursor.moveToFirst()) {
final int dataColumn =
cursor.getColumnIndexOrThrow(MediaStore.Images.Media.DATA);
do {
final String data = cursor.getString(dataColumn);
result.add(data);
} while (cursor.moveToNext() && cursor.getPosition() < limit);
}
cursor.close();
return result;
}
public static Bitmap resizeImage(Bitmap bitmap) {
return Bitmap.createScaledBitmap(bitmap, IMAGE_WIDTH, IMAGE_HEIGHT, true);
}
public static Bitmap resizeImage(Bitmap bitmap, Context ctx){
DisplayMetrics displayMetrics = new DisplayMetrics();
((Activity)ctx).getWindowManager()
.getDefaultDisplay()
.getMetrics(displayMetrics);
int width = displayMetrics.widthPixels;
return Bitmap.createScaledBitmap(bitmap, width, width, true);
}
public static Bitmap getUprightImage(String imgUrl) {
ExifInterface exif = null;
try {
exif = new ExifInterface(imgUrl);
} catch (IOException e) {
}
int orientation = exif.getAttributeInt(ExifInterface.TAG_ORIENTATION, 1);
int rotation = 0;
switch (orientation) {
case 3:
rotation = 180;
break;
case 6:
rotation = 90;
break;
case 8:
rotation = 270;
break;
}
Matrix matrix = new Matrix();
matrix.postRotate(rotation);
Bitmap bitmap = BitmapFactory.decodeFile(imgUrl);
//rotate image
bitmap = Bitmap.createBitmap(bitmap, 0, 0, bitmap.getWidth(),
bitmap.getHeight(), matrix, true);
return bitmap;
}
//For the given image, find the primary content type
//using firebase ml kit image labeling api
public static void labelImage(String imgUrl) {
Bitmap bitmap = getUprightImage(imgUrl);
bitmap = resizeImage(bitmap);
getContentTypePicture(bitmap, imgUrl);
}
public static void getContentTypePicture(Bitmap bitmap, final String path) {
FirebaseVisionImage image = FirebaseVisionImage.fromBitmap(bitmap);
FirebaseVisionImageLabeler labeler = FirebaseVision.getInstance()
.getOnDeviceImageLabeler();
Task r = labeler.processImage(image)
.addOnSuccessListener(new OnSuccessListener<List<FirebaseVisionImageLabel>>() {
@Override
public void onSuccess(List<FirebaseVisionImageLabel> labels) {
String text = "";
float confidence = 0;
float highConfidence = 0;
for (FirebaseVisionImageLabel label : labels) {
confidence = label.getConfidence();
if (highConfidence < confidence) {
highConfidence = confidence;
text = label.getText();
}
Log.d("images content info ",
text + " " + confidence);
}
saveImageContentInfo(path, text);
}
})
.addOnFailureListener(new OnFailureListener() {
@Override
public void onFailure(@NonNull Exception e) {
Log.d("Image Labler", "failed to find photo content type");
}
});
}
private static void saveImageContentInfo(String path, String contentType) {
ImageLabel il = new ImageLabel();
il.setImagePath(path);
il.setImageContentType(contentType);
saveImageInfo(il);
saveImageCategories(contentType);
}
//save image path and content type info in firestore db
private static void saveImageInfo(ImageLabel imageLabel) {
firestoreDB.collection("PhotosInfo").document("myPics")
.collection(imageLabel.getImageContentType())
.add(imageLabel)
.addOnSuccessListener(new OnSuccessListener<DocumentReference>() {
@Override
public void onSuccess(DocumentReference documentReference) {
Log.d("Image Lable", "add photo content info to database");
}
})
.addOnFailureListener(new OnFailureListener() {
@Override
public void onFailure(@NonNull Exception e) {
Log.d("Image Labler", "failed to add photo content info to db");
}
});
}
//save content type in db
private static void saveImageCategories(String imageCat) {
ImageLabel il = new ImageLabel();
il.setImageContentType(imageCat);
firestoreDB.collection("PhotosCat").document(imageCat).set(il)
.addOnSuccessListener(new OnSuccessListener<Void>(){
@Override
public void onSuccess(Void v) {
}
})
.addOnFailureListener(new OnFailureListener() {
@Override
public void onFailure(@NonNull Exception e) {
Log.d("Image Labler", "failed to add photo category to db");
}
});
}
}
public class ImageLabel {
private String imagePath;
private String imageContentType;
public String getImagePath() {
return imagePath;
}
public void setImagePath(String imagePath) {
this.imagePath = imagePath;
}
public String getImageContentType() {
return imageContentType;
}
public void setImageContentType(String imageContentType) {
this.imageContentType = imageContentType;
}
}