Jan 11, 2020 17:00:00

I tried the workflow tool `` Flyte '' that easily visualizes the flow of machine learning and data processing for free

Technology that uses machine learning will be developed, such as

a mechanism to check whether the guest is a psychopath before booking and a handmade kit that can convert a car to a `` fully automatic driving car '' for only 110,000 yen It is becoming more and more active. In such machine learning and data processing, there is a very convenient workflow tool that manages the entire workflow such as the code and data of the development project, and the workflow tool developed independently by the company such as Netflix's Metaflow is open source In some cases, it will be made public. ' Flyte ' is a workflow tool released as open source by Lyft, which develops a ride sharing service, so I actually touched the tool.

Flyte — Flyte 0.0.1 documentation
https://lyft.github.io/flyte/

GitHub-lyft / flyte: develop, execute, and monitor distributed workflows reliably at scale.
https://github.com/lyft/flyte

This time, we will take a look at the functionality of Flyte while following a tutorial using a sample workflow, 'When inputting an image, extract and output the boundaries of the image using a machine learning model.'

It is said that

Kubernetes is required for Flyte operation, so this time I installed Docker and Minikube on Windows.

Install Docker Desktop on Windows | Docker Documentation
https://docs.docker.com/docker-for-windows/install/

Install Minikube-Kubernetes

https://kubernetes.io/docs/tasks/tools/install-minikube/

After installing the required software, start Windows PowerShell with administrator privileges and execute the following command.

[code] kubectl cluster-info [/ code]

When executed, the information of Master of Kubernetes will be displayed, so make a note of the IP address enclosed in the red frame.

The deployment that can install Flyte most easily is 'Sandbox', so execute the following command on PowerShell and deploy Flyte.

[code] kubectl create -f https://raw.githubusercontent.com/lyft/flyte/master/deployment/sandbox/flyte_generated.yaml[/code]

Execute the following command to register the sample workflow in Flyte.

[code] docker run --network host -e FLYTE_PLATFORM_URL = 'IP address enclosed in red frame: 30081' lyft / flytesnacks: v0.1.0 pyflyte -p flytesnacks -d development -c sandbox.config register workflows [/ code]

This will start downloading and installing the components.

After registration, try to access the following URL.

[code] http: // IP address enclosed in red frame: 30081 / console [/ code]

The Flyte screen was displayed. Click 'development' of 'flytesnacks' project.

A list of workflows in the project is displayed. Click the red frame.

Click 'Launch Workflow' to launch the workflow.

This time, we use a sample workflow of `` extracting and outputting the boundaries of images using a machine learning model when inputting images '', and it is necessary to specify the input image, so the sample image is shown in the red frame part Enter the URL and press 'Launch'.

Wait for the process to complete.

When the process was completed, 'SUCCEEDED' was displayed in the status part. Click 'Graph' on the top left tab ...

You can check the workflow in a graph.

The points on the graph change color depending on the execution status of the workflow, 'Unexecuted' is light blue, 'Execution successful' is green, and 'Execution failed' is red, so which part is error You can see at a glance if is out.

In this sample workflow, the graph was simple, but if you register a complicated workflow, the graph will become complicated.

Click 'View Inputs & Outputs' at the top right to see the execution results.

You can check the input information and the output information.

Access the following URL to view the output image.

[code] http: // IP address enclosed in red frame: 30081 / minio [/ code]

The login screen will be displayed. The username is 'minio' and the password is 'miniostorage'.

When you log in, a list of folders will be displayed. Search for the file based on the output information you checked earlier.

Once you have found the output file, select it and click 'Download Output' to download the output file.

This is the image actually output.

This time the sample workflow was activated, but it is of course possible to activate the workflow that you created yourself. It was a useful tool for development sites, because it could clearly visualize the workflow when performing complex machine learning and data processing.

GitHub-lyft / flyte: develop, execute, and monitor distributed workflows reliably at scale.

https://github.com/lyft/flyte