https://github.com/VerizonDigital/vflow

vFlow

The vFlow is not a new innovation, but it’s built with a new architecture and written in a modern language to use resources as efficiently as possible that I’ll talk about it later in this article. All tech people are aware of the value of data flow and the importance of traffic details. Defense of DDoS attacks, intruder detection, route optimization/saving money on the transit bills, real-time traffic visualization, troubleshooting, and e.t.c. are applications that can be built on top of the vFlow. What’s vFlow? the vFlow is a high performance and scalable IPFIX (Internet Protocol Flow Information Export) / sFlow collector that decodes exported UDP packets flow and delivers them to Kafka or NSQ message bus with a low or mid level server. It is designed for terabits or even petabits per second traffic for a global network as a service. It can be able to grow horizontally and handle any traffic level with redundancy. You can integrate it to different application architectures at your company. The only requirements are a messaging bus and your network devices must support IPFIX or sFlow. It supports Apache Kafka and NSQ for the time being but you can write your own plugin for a specific message bus as the vFlow supports the producer’s plugin. An alternative solution for message bus is using docker image. Several docker images are available from the different repository and you can turn it up quickly.

Golang

This is written in pure, Go language; including concurrency, dynamic pooling, memory recycling, and discovery through multicasting. Request another node for a missed IPFIX template through RPC and replicating IPFIX to another 3rd party collector. With respect to all languages, I can say Go is one of the best languages for creating a new service or product. Once the idea comes to your mind, you can design and code it through its simple syntax coding with fewer but strong lines. You can optimize memory and CPU with runtime profiling. I came from C/Perl/Python, but simplicity, concurrency, performance, cross compiling and Go community are the main features that I love this language. if you still haven’t tried it, I suggest you check it out at golang.org and gobyexample.com

Build / Docker

For the time being, it has been tested and runs on Verizon Digital Media Services (more than 5% of the internet provide by VDMS) in an Ubuntu 14.04 operating system with Juniper devices and works with 99.99% uptime under different situations but can be compiled under the other *nix operating system. You can open an issue if you encounter any problem at github.com/verizonDigital/vflow. There isn’t any specific dependency except the Go code it is easy to understand and compiled in a minute. If you don’t like to compile the code, there is a Dockerfile that you can create a docker image with and use a Kafka image to have it up and running instead of compiling. I made a vFlow image at docker hub and you can pull it and run it quickly.

Why we need to implement it in the house, not a commercial product.

After 18 years of experiencing different types of internet service providers, I can say that it would be hard or impossible to integrate real-time flow information using a commercial product with different applications in an enterprise network. Most of them can expose the data through RESTful API that usually returns analyzed data points after a couple of minutes delay (ex. 15-30 minutes). Again the commercial application with visualization and reports are awesome, but not for integration in an enterprise network through RESTful API.


Monitoring

The vFlow has its own monitoring to have visibility of collector, producer, and replicator features. it supports InfluxDB and OpenTSDB back-ends. If you have other back-end or collector, you can collect metrics as it exposes them through RESTful API. The screenshot shows the InfluxDB back-end and Grafana UI.


License

The vFlow is licensed under the Apache License 2.0, Copyright (C) 2017 Verizon.

Thanks for reading! please check github.com/verizonDigital/vflow if you have any questions you can open an issue at GitHub or hit me up at arshad.rad@gmail.com