Blog - Einleitung

Ancud Blog

Herzlich Willkommen zum Ancud Blog. Hier finden Sie eine Vielzahl von interessanten Artikeln zu verschiedenen Themen. Tauchen Sie ein in unsere Welt des Wissens! 

Blogs

Blogs

Data Engineers : Let’s lift some heavy weights on GCP

Figure 1 : Architecture of the pipeline

This pipeline for you guys , to make you feel more confident in your job and scale up your skills and knowledge , this project is developed within our data science unit at Ancud IT . In this article we are going to build a very solid project which includes :

  • Google container registry (GCR)
  • Cloud build
  • Kubernetes on GCP (GKE)

As you can see above (figure 1) the architecture of the pipeline , okay , confused ? I’m going to explain step by step :

First we are going to write our code in our local machine , it is going to be about leaf disease detection . YES ! you guess it right we are going to glance over some computer vision concepts .

Figure 2 : Leaf disease classification

 

We are going to build CNN model using tensorflow and keras to detect the disease in the leaf . I know I’m being generous with you guys because I’m loading it with the best informations that you need in your journey , yes you need this kind of articles to save your time and money , you don’t have to buy any online courses anymore .

I did mention the services that we’ll be using but I didn’t explain them , so I’m going to give you “La crème de la crème” .

  • Google container registry : it is the place where we are going to send our Docker images .
  • Cloud build : allows us to continuously build, test and deploy . containers.
  • Kubernetes : open-source system to automate the deployment of containers

OKAY ! I know that you don’t like these kind of definitions that are thrown in most of websites without showing where do they fit in real world scenarios , I’am aware of that , I got your backs , don’t worry , my target is to make you feel comfortable with what you learn and read .

 

 

 

 

The steps that we are going to do are as follow :

  1. build the docker file
  2. push the docker image to a container registry (GCR)
  3. kubernetes is going to pull the image from GCR and deploy it

The first step is to glance over the code. 

 

 

Import the librairies :


 

Check which GPU is provided by google colab :

Figure 3 : Output of nvidia-smi command

T4 is a part of the NVIDIA AI inference platform that supports all AI frameworks and provides comprehensive tooling and integrations to drastically simplify the development and deployment of advanced AI

 

Download the dataset :


 

display a sample of the images :


 

plot images of different labels :


 

before proceeding with the article I want to draw your attention to how you can optimise your inputs using tensor flow , so I’m you can click here to read a very good article about Prefetching .

 

Let’s define the model :


 

Compile it and do the training process :


 

Let’s evaluate our model :


 

Right now it’s time for real predictions , we are going to utilize the images within the test folder :


 

Now , it’s time to save the model and move to the cloud :


 

I’m moving to the cloud , see you there …

...

Hey again , glad to see you here . let’s do some fancy job here and flex a little bit while doing it because we deserve it . Los geht’s (here we go in german ) .

The first step is to activate Kubernetes Engine API , it will take some time , but it is ok , just wait for it .


 

The second step is to clone the project from the Github repo :


 

After that , we create a kubernetes cluster (single noded cluster):

gcloud container clusters create project-kube --zone "us-west1-b" --
machine-type "n1-standard-1" --num-nodes "1"

the next step will be to start the pipeline using this command : 

gcloud builds submit --config cloudbuild.yaml

note : make sure to be in the same folder as the cloudbuild.yaml file when you run the command .

after the build job you will get something like that when you access to cloud builder interface

We can see that the image has been build and stored in google container registry :


 

We can also check the pods of our deployment :


 

our app can be accessed using this IP Address. :


 

everything is ready , our app is running in production and this is a screenshot of it :


 

Conclusion :

The IT world is interconnected , you can’t be only passionate by one domain , you have to ask questions about what is going on behind the scenes and be aware of the technical knowledge of the IT projects that you are dealing with , but don’t worry I’m here for you guys , I got you backs , I will do my best to keep you on he right path by providing useful content . AS I SAY ALWAYS : STAY TUNED , GREAT CONTENT IS GOING TO BE COVERED IN THE NEXT ARTICLES . (and I promise you that we are going to be flexing together ) .