Nov 29

Measuring Network Throughput using NetPerf & iPerf

The three key measures of network performance are latency (the time required to transfer data across the network), throughput (the amount of data or number of data packets that can be delivered on an IP network in a predefined timeframe) and jitter or delay jitter (the changes and their duration in delay that occur during transfers).

In this blog post, I will show you how to measure throughput using NetPerf and iPerf, two open source network performance benchmark tools that support both UDP and TCP protocols. Each tool provides in addition other information: NetPerf for example provides tests for end-to-end latency (round-trip times or RTT) and is a good replacement for Ping, iPerf provides packet loss and delay jitter, useful to troubleshoot network performance. Choosing one or the other tool depends on your use-case and the test you are planning to achieve. Note that for the same input parameters, the tools can report different bandwidths, as they are not designed the same.

I will use the default parameters and run each test for 5 minutes (300 seconds). For a good report, it is recommended to run the tests multiple times, at different times of the day, with different parameters.

1. Installing NetPerf and iPerf

NetPerf and iPerf have a client and server functionality, and must be installed both on the server and the client from where you are conducting network performance tests. For each tool I will provide the most common parameters, and conduct tests between a client (1GB MEM) and a server (1GB MEM) and in my LAN (Local Area Network), and between a client and a remote server (in a WAN).

The tools can be installed on different Linux distributions, on MacOS and on Windows. Refer to the following documentations for more information:

2. Most common parameters

2.1. NetPerf common parameters

  • -p (–port): Port number (12865 by default)
  • -H (–host): Host/Server IP address or DNS name
  • -t (–testname): Specifies test to perform (TCP_STREAM by default)
  • -l (–testlen): Specifies test duration in seconds (>0 secs)
  • -m value set the local send size to value bytes. [Default: local socket buffer size]
  • -M value set the remove receive size to value bytes. [Default: remote receive socket buffer size]

For more information use: $ netperf -h

2.2. iPerf3 common parameters

  • -c (–client): run in client mode, connecting to
  • -p (–port): Port number (by default 5001 and 5201 for iperf and iperf3 respectively)
  • -u (–udp): for UDP tests (default is tcp)
  • -i (–interval): seconds between periodic bandwidth reports
  • -b (–bandwidth): target bandwidth in bits/sec (0 for unlimited) – default is 1 Mbit/sec for UDP and unlimited for TCP
  • -t (–time): time in seconds to transmit for (default 10 secs)
  • -P (–parallel): number of parallel client streams to run
  • –get-server-output: get results from server (useful for UDP tests)

For more information use $ iperf3 -h

3. Testing TCP throughput

Using NetPerf and iPerf for UDP throughput

3.1. Testing TCP throughput using NetPerf

Run Netperf as server on the server:

$ netserver
Starting netserver with host 'IN(6)ADDR_ANY' port '12865' and family AF_UNSPEC

TCP Throughput in my LAN

$ netperf -H 172.31.56.48 -l 300 -t TCP_STREAM
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.31.56.48 () port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 20480 20480 300.04 62.61

TCP Throughput in a WAN

$ netperf -H HOST -l 300 -t TCP_STREAM
MIGRATED TCP STREAM TEST from (null) (0.0.0.0) port 0 AF_INET to (null) () port 0 AF_INET : histogram : spin interval
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 131072 131072 300.32 7.69

3.2. Testing TCP throughput using iPerf

Run iPerf3 as server on the server:

$ iperf3 --server --interval 30
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

TCP Throughput in my LAN

$ iperf3 --client 172.31.56.48 --time 300 --interval 30
Connecting to host 172.31.56.48, port 5201
[ 4] local 172.31.100.5 port 44728 connected to 172.31.56.48 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-30.00 sec 1.70 GBytes 488 Mbits/sec 138 533 KBytes
[ 4] 30.00-60.00 sec 260 MBytes 72.6 Mbits/sec 19 489 KBytes
[ 4] 60.00-90.00 sec 227 MBytes 63.5 Mbits/sec 15 542 KBytes
[ 4] 90.00-120.00 sec 227 MBytes 63.3 Mbits/sec 13 559 KBytes
[ 4] 120.00-150.00 sec 228 MBytes 63.7 Mbits/sec 16 463 KBytes
[ 4] 150.00-180.00 sec 227 MBytes 63.4 Mbits/sec 13 524 KBytes
[ 4] 180.00-210.00 sec 227 MBytes 63.5 Mbits/sec 14 559 KBytes
[ 4] 210.00-240.00 sec 227 MBytes 63.5 Mbits/sec 14 437 KBytes
[ 4] 240.00-270.00 sec 228 MBytes 63.7 Mbits/sec 14 516 KBytes
[ 4] 270.00-300.00 sec 227 MBytes 63.5 Mbits/sec 14 524 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-300.00 sec 3.73 GBytes 107 Mbits/sec 270 sender
[ 4] 0.00-300.00 sec 3.73 GBytes 107 Mbits/sec receiver

TCP Throughput in a WAN

$ iperf3 --client HOST --time 300 --interval 30
Connecting to host HOST, port 5201
[ 5] local 192.168.1.73 port 56756 connected to HOST port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-30.00 sec 21.2 MBytes 5.93 Mbits/sec
[ 5] 30.00-60.00 sec 27.0 MBytes 7.55 Mbits/sec
[ 5] 60.00-90.00 sec 28.6 MBytes 7.99 Mbits/sec
[ 5] 90.00-120.00 sec 28.7 MBytes 8.02 Mbits/sec
[ 5] 120.00-150.00 sec 28.5 MBytes 7.97 Mbits/sec
[ 5] 150.00-180.00 sec 28.6 MBytes 7.99 Mbits/sec
[ 5] 180.00-210.00 sec 28.4 MBytes 7.94 Mbits/sec
[ 5] 210.00-240.00 sec 28.5 MBytes 7.97 Mbits/sec
[ 5] 240.00-270.00 sec 28.6 MBytes 8.00 Mbits/sec
[ 5] 270.00-300.00 sec 27.9 MBytes 7.81 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-300.00 sec 276 MBytes 7.72 Mbits/sec sender
[ 5] 0.00-300.00 sec 276 MBytes 7.71 Mbits/sec receiver

Comments on the TCP tests

4. Testing UDP throughput

UDP does not provide an end to end control flow, when testing UDP throughput make sure you specify the size of the packet to be sent by the client. Always read the receive rate on the server, since UDP is an unreliable protocol, the reported send rate can be much higher than the actual receive rate.

Using NetPerf and iPerf for UDP throughput

4.1. Testing UDP throughput using NetPerf

Change the test name from TCP_STREAM to UDP_STREAM. Let’s use 1024 (1MB) as the message size to be sent by the client.
If you receive the following error send_data: data send error: Network is unreachable (errno 101) netperf: send_omni: send_data failed: Network is unreachable, add option -R 1 to remove the iptable rule that prohibits NetPerf UDP flow.

UDP Throughput in my LAN

$ netperf -H 172.31.56.48 -t UDP_STREAM -l 300 -- -R 1 -m 1024
MIGRATED UDP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.31.56.48 () port 0 AF_INET
Socket Message Elapsed Messages
Size Size Time Okay Errors Throughput
bytes bytes secs # # 10^6bits/sec

212992 1024 300.00 9193386 0 251.04
212992 300.00 9131380 249.35

UDP Throughput in a WAN

$ netperf -H HOST -t UDP_STREAM -l 300 -- -R 1 -m 1024
MIGRATED UDP STREAM TEST from (null) (0.0.0.0) port 0 AF_INET to (null) () port 0 AF_INET : histogram : spin interval
Socket Message Elapsed Messages
Size Size Time Okay Errors Throughput
bytes bytes secs # # 10^6bits/sec

9216 1024 300.01 35627791 0 972.83
212992 300.01 253099 6.91

4.2. Testing UDP throughput using iPerf

For the UDP tests, it’s important to read the results on the server, or use --get-server-output to read the server output on the client. The server indeed reports the data it was able to process, unlike the client that will send the amount of data specified in the -b (–bandwidth) parameter (1MB by default). For this parameter, it is recommended to use the maximum bandwidth the network can support (respectively around 100Mbits/s and 7.7Mbits/s when the client and the server are on the LAN and remote).

UDP Throughput in my LAN

$ iperf3 --client 172.31.56.48 --interval 30 -u -b 100MB
Accepted connection from 172.31.100.5, port 39444
[ 5] local 172.31.56.48 port 5201 connected to 172.31.100.5 port 36436
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-30.00 sec 354 MBytes 98.9 Mbits/sec 0.052 ms 330/41774 (0.79%)
[ 5] 30.00-60.00 sec 355 MBytes 99.2 Mbits/sec 0.047 ms 355/41903 (0.85%)
[ 5] 60.00-90.00 sec 354 MBytes 98.9 Mbits/sec 0.048 ms 446/41905 (1.1%)
[ 5] 90.00-120.00 sec 355 MBytes 99.4 Mbits/sec 0.045 ms 261/41902 (0.62%)
[ 5] 120.00-150.00 sec 354 MBytes 99.1 Mbits/sec 0.048 ms 401/41908 (0.96%)
[ 5] 150.00-180.00 sec 353 MBytes 98.7 Mbits/sec 0.047 ms 530/41902 (1.3%)
[ 5] 180.00-210.00 sec 353 MBytes 98.8 Mbits/sec 0.059 ms 496/41904 (1.2%)
[ 5] 210.00-240.00 sec 354 MBytes 99.0 Mbits/sec 0.052 ms 407/41904 (0.97%)
[ 5] 240.00-270.00 sec 351 MBytes 98.3 Mbits/sec 0.059 ms 725/41903 (1.7%)
[ 5] 270.00-300.00 sec 354 MBytes 99.1 Mbits/sec 0.043 ms 393/41908 (0.94%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-300.04 sec 3.45 GBytes 98.94 Mbits/sec 0.043 ms 4344/418913 (1%)

UDP Throughput in a WAN

$ iperf3 --client HOST --time 300 -u -b 7.7MB
Accepted connection from 45.29.190.145, port 60634
[ 5] local 172.31.56.48 port 5201 connected to 45.29.190.145 port 52586
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-30.00 sec 27.4 MBytes 7.67 Mbits/sec 0.438 ms 64/19902 (0.32%)
[ 5] 30.00-60.00 sec 27.5 MBytes 7.69 Mbits/sec 0.446 ms 35/19940 (0.18%)
[ 5] 60.00-90.00 sec 27.5 MBytes 7.68 Mbits/sec 0.384 ms 39/19925 (0.2%)
[ 5] 90.00-120.00 sec 27.5 MBytes 7.68 Mbits/sec 0.528 ms 70/19950 (0.35%)
[ 5] 120.00-150.00 sec 27.4 MBytes 7.67 Mbits/sec 0.460 ms 51/19924 (0.26%)
[ 5] 150.00-180.00 sec 27.5 MBytes 7.69 Mbits/sec 0.485 ms 37/19948 (0.19%)
[ 5] 180.00-210.00 sec 27.5 MBytes 7.68 Mbits/sec 0.572 ms 49/19941 (0.25%)
[ 5] 210.00-240.00 sec 26.8 MBytes 7.50 Mbits/sec 0.800 ms 443/19856 (2.2%)
[ 5] 240.00-270.00 sec 27.4 MBytes 7.66 Mbits/sec 0.570 ms 172/20009 (0.86%)
[ 5] 270.00-300.00 sec 25.3 MBytes 7.07 Mbits/sec 0.423 ms 1562/19867 (7.9%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-300.00 sec 272 MBytes 7.60 Mbits/sec 0.423 ms 2522/199284 (1.3%)
[SUM] 0.0-300.2 sec 31 datagrams received out-of-order

Comments on the UDP tests

Conclusion

May 11

Adding CORS Headers Using Lambda@Edge and Amazon CloudFront

Cross-origin resource sharing (CORS) defines a way for client web applications that are loaded in one domain to interact with resources in a different domain, it allows to build rich client-side web applications and selectively allow cross-origin access to your resources. To see how to enable CORS on most web servers see https://enable-cors.org/index.html.

If you’re using Amazon CloudFront to serve your content, you will need to configure the distribution to respect the CORS settings by forwarding specific headers in the CloudFront behavior.

When HTTP headers are forwarded to the origin, CloudFront caches separate versions of a specified object based on the header values in viewer requests. It is a best practice to not forward requests to the origin if the requested object is already cached at the edge location, as this decreases the cache Hit ratio. Response bodies in requests that contain the Origin header are the same even if the values of the Origin header are different.

In this post we will leverage Lambda@Edge to set the CORS headers when the request contains the Origin header. With this solution, you don’t need to enable CORS at the origin or to forward the Origin Header in the CloudFront distribution.

Lambda@Edge provides the ability to execute a Lambda function at an Amazon CloudFront Edge Location. This capability enables intelligent processing of HTTP requests at locations that are close (for the purposes of latency) to your customers. To get started, you simply upload your code (Lambda function written in Node.js) and pick one of the CloudFront behaviours associated with your distribution.

You can run a Lambda@Edge function in response to four different CloudFront events:

      Viewer Request: When CloudFront receives a request from a viewer
      Origin Request: Before CloudFront forwards a request to the origin
      Origin Response: When CloudFront receives a response from the origin
      Viewer Response: Before CloudFront returns the response to the viewer

For more information about Lambda@Edge and how it works, see: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-at-the-edge.html.

For the purpose of this blog post, we’ll be focusing on the Viewer Response Event, I use the below Node.js code in the main index.js file. For step by step guide on how to create a Lambda function and associate it to a CloudFront behaviour see https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-edge-how-it-works-tutorial.html.

Viewer Response Code Snippet

'use strict';
exports.handler = (event, context, callback) => {

//Get contents of response
const response = event.Records[0].cf.response;
const headers = response.headers;
if ('origin' in event.Records[0].cf.request.headers) {
//The Request contains the Origin Header - Set CORS headers
headers['access-control-allow-origin'] = [{key: 'Access-Control-Allow-Origin', value: "*"}];
headers['access-control-allow-methods'] = [{key: 'Access-Control-Allow-Methods', value: "GET, HEAD"}];
headers['access-control-max-age'] = [{key: 'Access-Control-Max-Age', value: "86400"}];
}
//Return modified response
callback(null, response);
};

I’ve associated the above Node.js code to a CloudFront distribution behaviour (Viewer Response) that has as Origin an S3 bucket for which CORS is not enabled (note the CORS Headers and the Hit from CloudFront on the second request):

$ curl -v http://d2fyouwzorwsid.cloudfront.net/file.txt >/dev/null
> GET /file.txt HTTP/1.1
> Host: d2fyouwzorwsid.cloudfront.net
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/plain
< Content-Length: 5
< Connection: keep-alive
< x-amz-id-2: JYE+LYMvI6Vmu13MCoPJZtpF76Ozs3W5AiY6KvwrG8t4KEq3/CcSE1WCR85ra6f9yxSh1zqGIi4=
< x-amz-request-id: 1FE8E25F4E96C9AF
< Date: Thu, 02 May 2019 19:15:34 GMT
< Last-Modified: Thu, 02 May 2019 19:13:46 GMT
< ETag: "2205e48de5f93c784733ffcca841d2b5"
< Cache-Control: max-age=500
< Accept-Ranges: bytes
< Server: AmazonS3
< X-Cache: Miss from cloudfront
< Via: 1.1 2e50d9b1ee017f302768660f02b7418e.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: fWxXPrnynlKchjfD3gS3q4QxvAsPyVH1XEZRYKhR3pdO3NPlpE04Jg==

$ curl -v -H "Origin:www.example.com" http://d2fyouwzorwsid.cloudfront.net/file.txt >/dev/null
> GET /file.txt HTTP/1.1
> Host: d2fyouwzorwsid.cloudfront.net
> User-Agent: curl/7.54.0
> Accept: */*
> Origin:www.example.com
>
< HTTP/1.1 200 OK
< Content-Type: text/plain
< Content-Length: 5
< Connection: keep-alive
< x-amz-id-2: JYE+LYMvI6Vmu13MCoPJZtpF76Ozs3W5AiY6KvwrG8t4KEq3/CcSE1WCR85ra6f9yxSh1zqGIi4=
< x-amz-request-id: 1FE8E25F4E96C9AF
< Date: Thu, 02 May 2019 19:15:34 GMT
< Last-Modified: Thu, 02 May 2019 19:13:46 GMT
< ETag: "2205e48de5f93c784733ffcca841d2b5"
< Cache-Control: max-age=500
< Accept-Ranges: bytes
< Server: AmazonS3
< Age: 11
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: GET, HEAD
< Access-Control-Max-Age: 86400
< X-Cache: Hit from cloudfront
< Via: 1.1 1448f69604d5be1f9c9f0c64cfa90595.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: CNvc-aOmxsrlAbogKUOYTBm-oOEV3VQjawQZ5LCj5CW1VYxK97a8-A==

Congratulations! You have successfully implemented cross-origin resource sharing without enabling CORS at the origin server or forwarding the Origin Header to the origin.

Feb 16

Elasticsearch and Kibana Search Speed Tuning

This blog post provides some tips for Elasticsearch and Kibana Search Speed Tuning, apart from those recommended by Elasticsearch official website [1]:

  • giving enough memory to the filesystem cache
  • using faster hardware
  • searching as few fields as possible
  • pre-indexing data
  • avoiding scripts
  • considering mapping identifiers as keywords
  • etc.
GIS Map using Vega

ELK Stack

I recently installed the ELK Stack (Elasticsearch, Logstash and Kibana) on a single node cluster (m4.large EC2 instance type) to analyze Amazon S3 and CloudFront logs, and tried to improve search speed (I’ll not discuss indexing speed in this post). In production it’s recommended to have at least 3 nodes for a better performance, so I had to do some more tuning, in addition to the above.

1. Shard Allocation

The shards are the data containers for Elasticsearch and the number of shards have the effect on the performance of ES cluster. Since a shard is essentially a Lucene index, it consumes file handles, memory, and CPU resources. If there is a large number of shards, it will use too much resources from the cluster to manage the cluster activities; the cluster will have a large cluster state table and each search request has to touch a copy of every shard in the indexes, this will be a resource consuming process for the cluster. Also, the more shards the Elasticsearch cluster has, the more likely you are to get the “courier fetch error”.

If indexes are less than 30GB in size, 1 shard (primary) per index would be enough. Index Template [2] can be used to set a default for all indices as follows:

1.1. Check the number of shards and indices in the cluster by calling the _cat API operation

GET _cat/indices/logstash*

1.2. Define an index template, which specifies the number of shards for all new indices that will be created in the cluster, i.e.:

POST _template/default
{
"index_patterns": ["*"],
"settings": {
"number_of_shards": "1",
"number_of_replicas": "0"
}
}

1.3. Re-index by calling the _reindex API operation (this will move the data to a new index with the number of shards that was specified in the index template)

POST _reindex
{
"source": {
"index": "logstash-2019.02.06"
},
"dest": {
"index": "logstash-2019.02.06.new"
}
}

2. JVM Memory pressure and Heap Sizing

In Elasticsearch, JVM memory pressure indicates the fill rate of the old generation pool. Two common factors jointly contribute to the JVM memory pressure: the amount of data on the cluster, and the workload on the cluster (based on the cluster’s size). The full Java Garbage Collection can stop all the operation in the node, it also takes a lot of resources to the cluster (CPU, memory, etc.) when it is in action. It’s recommended to keep JVM Memory Pressure to around 70% [3].

  • When the heap size reaches 75%, Elasticsearch uses Concurrent Mark and Sweep (CMS) as default garbage collector(GC) to de-allocate space on the heap. This algorithm happens concurrently with other operations.
  • If previous GC operations have not restored enough memory and utilization is still above 75%, ES temporarily halts or slows processing of other threads to free memory using a different algorithm for GC. This compounds problems in the cluster as requests are backlogged, but it is necessary because otherwise an OOM error would occur quickly.
  • When the heap size is near 95%, ElasticSearch will kill the thread of the process that is trying to allocate memory. Because memory is a common resource, it is not guaranteed that this action won’t kill a critical process.

The less heap memory is allocated to Elasticsearch, the more RAM remains available for other OS processes and Lucene, which relies heavily on the file system cache to serve requests quickly. It’s recommended to set the Heap Size to around 30% ~ 60% of available memory and monitor the JMV Heap, make sure it’s less than 75%.

To set the minimum and maximum JVM Heap sizes, change the values of -Xms and –Xmx in jvm.options respectively, environment variables can also be used; it’s recommended to have the same value for -Xms and -Xmx [4].

3. Swapping

Given the memory used by ELK Stack, a Swap memory [5] can be added to the server (if there is no swap) and disable Swap utilization for Elasticsearch; swapping makes things slow as disks are slower than the memory [4]. This property can be set in the YAML file:
bootstrap.memory_lock: true

4. Kibana Dashboards

Use a single dashboard per page, containing different visualizations. Indeed, Kibana sends all queries related to Visualizations in a Dashboard in a single _msearch request, which executes in parallel. When a page has different dashboards, requests are sent individually, causing some visualizations to be rendered a bit quickly, but the entire dashboard as a whole to be rendered more slowly (from start to finish). This should be a bad experience if the user is on a slow connection, some dashboards might never load. Another drawback of having different dashboards on a single page is that it requires more ES resources, increasing the server Memory and CPU Utilization.

5. Refresh Interval (5s by default)

Elasticsearch creates new segment every time a refresh happens. Increasing the refresh interval would help reduce the segment count and reduce the IO cost for search. And, the cache would be invalid once a refresh happens and data is changed. Increasing the refresh interval can make Elasticsearch utilize cache more efficiently. To increase the Refresh Interval:
PUT /logstash-2019.02.06/_settings
{
"index" : {
"refresh_interval" : "60s"
}
}

6. Shard Request Cache

The shard-level request cache module caches the local results on each shard, which allows frequently used search requests to return results almost instantly. To increase (decrease) it size change indices.requests.cache.size in Elasticsearch configuration file:
indices.requests.cache.size: 10%
The request below can be used to check whether the shard query cache has an effect:
GET logstash*/_stats/request_cache?human
It should display something like the following
***
"_all": {
"primaries": {
"request_cache": {
"memory_size": "17.1mb",
"memory_size_in_bytes": 17975017,
"evictions": 0,
"hit_count": 8915,
"miss_count": 1955
}
},
"total": {
"request_cache": {
"memory_size": "17.1mb",
"memory_size_in_bytes": 17975017,
"evictions": 0,
"hit_count": 8915,
"miss_count": 1955
}
}
},
***

7. Use Filter Context instead of Query Context

Some visualizations on the current dashboards are based on queries. It’s recommended, whenever possible, to use Filter Context, as frequently used filters are cached automatically to speed up performance [5].

References:
[1] Tune Elasticsearch for Search Speed
[2] Indice Template
[3] Memory Pressure Indicator
[4] Heap – Sizing and Swapping
[5] Query and filter context

Feb 01

GIS Map in Elasticsearch with Kibana using Vega, Scripted Fields & Painless

Elasticsearch is a distributed open source, RESTful search engine built on top of Apache Lucene and released under an Apache license. Kibana is an open source data visualization plugin for Elasticsearch. Vega (and Vega-lite) allows to beyond the built-in visualizations offered by Kibana.

In this short tutorial we will use Vega to create a GIS map that displays individual documents in Elasticsearch into a Kibana map as marks, that have different shapes and colors, with information about the documents on the marks.

The graph will look like

GIS Map using Vega

GIS Map using Vega

{
"$schema": "https://vega.github.io/schema/vega/v3.0.json",
config: {
kibana: {
type: map
latitude: 15
longitude: -10
zoom: 3
}
}
"data": [
{
"name": "observations",
"url": {
index: logstash-2018*
"body": {
"size": 10000
_source: {
includes: ["@timestamp", "items_per_minute", "school", "geoip.location", "county", "subcounty","zone", "lesson_start_time","subject", "end_time", "start_time", "class"]
}
script_fields : {
lesson_duration : {
script : {
lang: 'painless',
source: doc['end_time'].date.minuteOfDay - doc['start_time'].date.minuteOfDay;
}
}
lesson_date : {
script : {
lang: 'painless',
// Get the time in the correct format
source: doc['start_time'].date.yearOfEra + '-' + doc['start_time'].date.monthOfYear + '-' + doc['start_time'].date.dayOfMonth
}
}
grade : {
script : {
lang: 'painless',
"source": "if(doc.containsKey('formId.keyword')){if(doc['formId.keyword'].value == 'Gradethreeobservationtool'){return 3;}if(doc['formId.keyword'].value == 'maths-grade3'){return 3;}if(doc['formId.keyword'].value == 'class-12-lesson-observation-with-pupil-books'){if(doc.containsKey('class.keyword')){return doc['class.keyword'].value;}}if(doc['formId.keyword'].value == 'maths-teachers-observation-tool'){if(doc.containsKey('class.keyword')){return doc['class.keyword'].value;}}}"
}
}
}
"sort": {"start_time":"desc"}
"query": {
"bool": {
"must": [
// This string will be replaced with the auto-generated "MUST" clause
"%dashboard_context-must_clause%",
// apply timefilter (upper right corner) to the @timestamp variable
{
"range": {
"@timestamp": {
// "%timefilter%" will be replaced with the current
// values of the time filter (from the upper right corner)
"%timefilter%": true
// Only work with %timefilter%
// Shift the current timefilter by 10 units back
"shift": 10,
// supports week, day (default), hour, minute, second.
"unit": "minute"
}
}
}
{"exists": {"field": "geoip.location"}}
{"exists": {"field": "school"}}
{"exists": {"field": "start_time"}}
],
"must_not": [
// This string will be replaced with the auto-generated "MUST-NOT" clause
"%dashboard_context-must_not_clause%"
{"match": { "school": "orphaned" }}
]
}
}
}}
"format": { "type": "json", "property": "hits.hits"}
"transform": [
{
"lookup": "geoip_location",
"type": "geopoint",
"projection": "projection",
"fields": [
_source.geoip\.location.lon
_source.geoip\.location.lat
]
}
]
}
{
"name": "selected",
"source": "observations",
"transform": [
{
"type": "filter",
"expr": "datum === selected"
}
]
}
]
"signals": [
{
"name": "selected",
"value": null,
"on": [
{"events": "symbol:mouseover", "update": "datum"},
{"events": "symbol:mouseout", "update": "null"}
]
}
],
"marks": [
{
name: "observation"
type: "symbol"
from: {data: "observations"}
encode: {
"enter": {
// different shapes for grades
"shape": [
{"test": "datum.fields.grade == 1", "value": "square"}
{"test": "datum.fields.grade == 2", "value": "triangle-up"}
{"value": "circle"}
]
// different colors for subjects
"fill": [
{"test": "datum._source.subject === 'English'", "value": "#159"}
{"test": "datum._source.subject === 'Maths'", "value": "#195"}
{"value": "#815"}
]
"size": {"value": "200"}
},
update: {
xc: {signal: "datum.x"}
yc: {signal: "datum.y"}
tooltip: {
signal: "{School: datum._source.school, County: datum._source.county, Subcounty: datum._source.subcounty, Zone: datum._source.zone, Subject: datum._source.subject, Grade:''+ datum.fields.grade, 'Items Per Minute': datum._source.items_per_minute, Date: ''+datum.fields.lesson_date, 'Start Time': hours(datum._source.start_time) + 3 + ':' + minutes(datum._source.start_time), 'End Time': hours(datum._source.end_time) + 3 + ':' + minutes(datum._source.end_time) + ' (' + datum.fields.lesson_duration + ' minutes)'}"
}
"fillOpacity": {"value": 0.7}
}
"hover": {
"fillOpacity": {"value": 0.3}
}
}
}
]
}

Sep 30

How to create a CloudFront distribution with AWS SDK for Go

Amazon Web Services provides different SDKs, Toolkits and Command Line Tools to develop and manage application running on AWS. AWS SDK for Go is one of the latest tools provided. New versions are pushed almost every 5 days.

In this blog post, we will write a simple Go code to create a CloudFront distribution with the default settings and the following:

  • An S3 bucket as origin for the distribution
  • A Lambda@Edge function associated to the default behavior
  • A WAF Rule

For more information about:

  • CloudFront
  • Installing and configuring AWS SDK for Go
  • CloudFront APIs with AWS SDK for Go
  • Lambda@Edge
  • WAF (Web Application Firewal)


    • package main

    import (
    "fmt"
    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/cloudfront"
    "github.com/aws/aws-sdk-go/aws/awserr"
    )

    func main() {

    creds := aws.Creds(accessKey, secretKey, "")
    svc := cloudfront.New(creds, "us-east-1", nil)

    // svc := cloudfront.New(session.New()) Can replace the 2 lines above if using Instance Role or Env. Variables

    input := &cloudfront.CreateDistributionWithTagsInput{
    Tags: &cloudfront.Tags{
    Items: []*cloudfront.Tag{
    },
    },
    DistributionConfig: &cloudfront.DistributionConfig{
    CallerReference: aws.String("Sat Sept. 30 2017"),
    Comment: aws.String("My WordPress Blog"),
    Enabled: aws.Bool(true),
    WebACLId: aws.String("eSamplec-5a3e-4857-9b92-0a5Sample3a4"),
    Origins: &cloudfront.Origins{
    Quantity: aws.Int64(1),
    Items: []*cloudfront.Origin{
    {
    Id: aws.String("Jil_S3Origin"),
    DomainName: aws.String("mydomain.com.s3.amazonaws.com"),
    S3OriginConfig: &cloudfront.S3OriginConfig{
    OriginAccessIdentity: aws.String(""),
    },
    },
    },
    },
    DefaultCacheBehavior: &cloudfront.DefaultCacheBehavior{
    TargetOriginId: aws.String("Jil_S3Origin"),
    MinTTL: aws.Int64(10),
    ViewerProtocolPolicy: aws.String("allow-all"),
    LambdaFunctionAssociations: &cloudfront.LambdaFunctionAssociations{
    Quantity: aws.Int64(1),
    Items: []*cloudfront.LambdaFunctionAssociation{
    {
    EventType: aws.String("viewer-request"), // "viewer-request" | "viewer-response" | "origin-request" | "origin-response"
    LambdaFunctionARN: aws.String("arn:aws:lambda:us-east-1:123456789012:function:myFunctionName:2"), // the version of the function must be added
    },
    },
    },
    TrustedSigners: &cloudfront.TrustedSigners{
    Enabled: aws.Bool(false),
    Quantity: aws.Int64(0),
    },
    ForwardedValues: &cloudfront.ForwardedValues{
    Cookies: &cloudfront.CookiePreference{
    Forward: aws.String("none"),
    },
    QueryString: aws.Bool(false),
    },
    },
    },
    }

    result, err := svc.CreateDistributionWithTags(input)

    if err != nil {
    if aerr, ok := err.(awserr.Error); ok {
    switch aerr.Code() {
    case cloudfront.ErrCodeCNAMEAlreadyExists:
    fmt.Println(cloudfront.ErrCodeCNAMEAlreadyExists, aerr.Error())
    case cloudfront.ErrCodeDistributionAlreadyExists:
    fmt.Println(cloudfront.ErrCodeDistributionAlreadyExists, aerr.Error())
    case cloudfront.ErrCodeInvalidOrigin:
    fmt.Println(cloudfront.ErrCodeInvalidOrigin, aerr.Error())
    case cloudfront.ErrCodeInvalidOriginAccessIdentity:
    fmt.Println(cloudfront.ErrCodeInvalidOriginAccessIdentity, aerr.Error())
    case cloudfront.ErrCodeAccessDenied:
    fmt.Println(cloudfront.ErrCodeAccessDenied, aerr.Error())
    case cloudfront.ErrCodeTooManyTrustedSigners:
    fmt.Println(cloudfront.ErrCodeTooManyTrustedSigners, aerr.Error())
    case cloudfront.ErrCodeTrustedSignerDoesNotExist:
    fmt.Println(cloudfront.ErrCodeTrustedSignerDoesNotExist, aerr.Error())
    case cloudfront.ErrCodeInvalidViewerCertificate:
    fmt.Println(cloudfront.ErrCodeTooManyCertificates, aerr.Error())
    case cloudfront.ErrCodeInvalidLocationCode:
    fmt.Println(cloudfront.ErrCodeInvalidLocationCode, aerr.Error())
    case cloudfront.ErrCodeInvalidGeoRestrictionParameter:
    fmt.Println(cloudfront.ErrCodeInvalidGeoRestrictionParameter, aerr.Error())
    case cloudfront.ErrCodeInvalidProtocolSettings:
    fmt.Println(cloudfront.ErrCodeInvalidProtocolSettings, aerr.Error())
    case cloudfront.ErrCodeInvalidTTLOrder:
    fmt.Println(cloudfront.ErrCodeInvalidTTLOrder, aerr.Error())
    case cloudfront.ErrCodeInvalidWebACLId:
    fmt.Println(cloudfront.ErrCodeInvalidWebACLId, aerr.Error())
    case cloudfront.ErrCodeTooManyOriginCustomHeaders:
    fmt.Println(cloudfront.ErrCodeTooManyOriginCustomHeaders, aerr.Error())
    case cloudfront.ErrCodeTooManyQueryStringParameters:
    fmt.Println(cloudfront.ErrCodeTooManyQueryStringParameters, aerr.Error())
    case cloudfront.ErrCodeInvalidQueryStringParameters:
    fmt.Println(cloudfront.ErrCodeInvalidQueryStringParameters, aerr.Error())
    case cloudfront.ErrCodeTooManyDistributionsWithLambdaAssociations:
    fmt.Println(cloudfront.ErrCodeTooManyDistributionsWithLambdaAssociations, aerr.Error())
    case cloudfront.ErrCodeTooManyLambdaFunctionAssociations:
    fmt.Println(cloudfront.ErrCodeTooManyLambdaFunctionAssociations, aerr.Error())
    case cloudfront.ErrCodeInvalidLambdaFunctionAssociation:
    fmt.Println(cloudfront.ErrCodeInvalidLambdaFunctionAssociation, aerr.Error())
    case cloudfront.ErrCodeInvalidOriginReadTimeout:
    fmt.Println(cloudfront.ErrCodeInvalidOriginReadTimeout, aerr.Error())
    case cloudfront.ErrCodeInvalidOriginKeepaliveTimeout:
    fmt.Println(cloudfront.ErrCodeInvalidOriginKeepaliveTimeout, aerr.Error())
    default:
    fmt.Println(aerr.Error())
    }
    } else { // Print the error, cast err to awserr.Error to get the Code and Message from an error.
    fmt.Println(err.Error())
    }
    return
    }
    fmt.Println(result)
    }

Sep 09

14 ans déjà que le Lion du Panjshir nous a quitté

Le Lion du Panjshir

Commandant Massoud

09 septembre 2001 – 09 septembre 2015, 14 ans déjà que le Lion du Panjshir nous a quitté. Celui qui a combattu les russes, les Talibans et Al Qaïda est mort suite à un attentat-suicide de deux prétendus journalistes qui font exploser leur camera pendant une interview.

A 25 ans (1978) Ahmed Chah Massoud (qui deviendra plus tard le Commandant Massoud) crée et prend la tête du “Conseil de surveillance”, il est un tacticien et un stratège hors pair et le seul chef de la Résistance à avoir jamais réussi à imposer une trêve avec l’Armée Rouge en échange de son retrait.

Il périt deux jours avant les attentats du 11 septembre, il avait pourtant plusieurs fois tenté d’alerter la communauté internationale sur le danger Al Qaïda.
‪
RIP‬ Massoud!

Feb 08

Rapport mHealth au Congo – Etude de faisabilité et recommandations

De septembre à décembre 2013, j’ai mené une étude de faisabilité de l’utilisation de la téléphonie mobile pour l’amélioration de la couverture vaccinale au Congo (Brazzaville).

Objectifs de l’étude

L’objectif de la mission était d’étudier la faisabilité tant sur les plans technique, organisationnel que technologique du projet d” « utilisation de la téléphonie mobile pour l’amélioration de la couverture des interventions à haut impact au Congo : Vaccination, déparasitage, supplémentation en vitamine A » et de réviser le draft du projet sur la base des résultats et du contexte effectif au Congo.

L’étude devrait fournir suffisamment d’informations à l’UNICEF et aux principaux partenaires (gouvernement, compagnies de téléphonie mobile, etc.) pour leur permettre de valider l’implémentation du projet. Elle devra également permettre d’orienter les choix stratégiques et les modalités de mise en œuvre d’un tel projet, les informations suivantes doivent en ressortir :

  • Les usages des téléphones portables par les ménages au Congo,
  • Mapping des projets utilisant la téléphonie mobile et évaluation des bonnes
    pratiques en matière de téléphonie mobile pour le développement au Congo,
  • Développement du document du projet pilote dans les départements de Brazzaville et Pointe Noire (plan d’implémentation, budget détaillé, équipe projet avec rôle de chaque partie prenante, suivi et évaluation, risques du projet, etc.),
  • Recommandations sur l’implémentation du projet pilote, les négociations avec les compagnies de téléphonie mobile, l’appropriation par le gouvernement et son
    extension à l’ensemble du territoire congolais.

Méthodologie utilisée

Etude de faisabilité : Méthodologie

Télécharger le rapport

Rapport sur l’étude de faisabilité et les recommandations (PDF, 40 pages, 1.9Mo)

 

Dec 07

Téléphonie mobile pour le développement – Risques et mesures d’atténuation

Total African Mobile Connections and Penetration Rate (million, percentage penetration).

Total African Mobile Connections and Penetration Rate (million, percentage penetration).

Plusieurs organisations internationales et ONG (UNESCO, UNICEF, Plan International, Carter Center, etc.) souhaitent profiter du taux de pénétration de la téléphonie mobile dans les pays en développement pour améliorer les conditions de vie des populations. De plus en plus projets sur la santé, l’éducation, la bonne gouvernance, etc.  incluant les téléphones portables voient donc le jour, un bon pourcentage utilise les SMS (Short Message Services) via les applications telles que RapidSMS et FrontLineSMS.

Comme pour toute initiative, en particulier celles basées sur les technologies de l’information, plusieurs risques peuvent impacter son bon déroulement et affecter de façon significative les résultats attendus. Dans le tableau ci-dessous un certain nombre de risques et des mesures d’atténuation à prendre en compte dans la conception du projet, risques à adapter en fonction du projet bien évidemment ;-), divisés en deux types :

  1. Les risques fonctionnels, techniques et liés aux processus sont associés au fonctionnement, aux procédures, aux ressources, à la communication du projet
  2. Les risques du projet sont associés aux aspects techniques et de mise en œuvre
Risque Prob. Impact Mesures d’atténuation
Risques fonctionnels, techniques et liés aux processus
Erreurs dans l’envoi des SMS par le public cible Moyen Elevé. Données non utilisables par le système Limiter le nombre de procédures, utiliser une nomenclature simple, simple bien former les utilisateurs
Les interfaces utilisateurs sont difficiles à comprendre, ou l’utilisation requiert une longue procédure Moyen Modéré. Système difficile à utiliser, utilisateurs finaux découragés Prendre en compte l’utilisateur pendant la conception, Produire les règles d’ergonomie
Anomalies de fonctionnement et instabilité de l’environnement Bas Elevé. Intégrité des données. Système souvent hors utilisation Renforcement des tests, Recensement des bugs, Choix techniques
Risques du projet
Ne pas avoir l’adhésion du gouvernement
et des partenaires		 Appropriation et viabilité à long terme du projet Impliquer le gouvernement et les partenaires dès la conception du projet
Ne pas héberger le système au sein du gouvernement Inclure un coût pour l’hébergement et le personnel technique. Renforcer les capacités afin d’assurer le support
Ne pas avoir un spécialiste TIC et un coordonateur du projet au sein du gouvernement
Les personnes chargées du monitoring sur le terrain ne possèdent pas de téléphone, ou ont des téléphones non fiables Impact élevé sur la mise à jour de la BD et l’intégrité des données Inclure un coût supplémentaire dans le budget, évaluer les téléphones pendant les formations
La langue d’envoi des messages n’est pas appropriée Impact élevé, les utilisateurs auront de la peine à les lire Savoir les préférences des utilisateurs finaux. Utiliser plusieurs langues au besoin
Le public bénéficiaire ne consulte pas sa messagerie Un bon pourcentage de messages ne sont pas lus Renforcer la communication autour du projet
Ne pas avoir une compagnie IT locale (ou un développeur local) pour développer l’application et en assurer le suivi Difficultés dans l’implémentation et le support du système Mettre sur pied une équipe technique à long terme (entreprise IT ou recruter un développeur)
Les utilisateurs finaux ne sont pas impliqués dès le début du projet, et ne sont pas bien formés Mauvaise utilisation et appropriation du dispositif Créer un groupe de travail pour valider les spécifications, Créer un comité d’utilisateurs

Oct 18

Utilisation de la téléphonie mobile pour l’amélioration de la santé de l’enfant au Congo

Le bureau de l’UNICEF à Brazzaville met sur pieds en ce moment un projet innovant sur l’utilisation de la téléphonie mobile pour l’amélioration de la couverture des interventions à haut impact (Vaccination, déparasitage, supplémentation en vitamine A). Utilisation des mobiles pour le développement

Au Congo 54,5% d’enfants de 12-24 mois n’ont pas été complètement vaccinés, de plus la malnutrition chronique touche 24.4% d’enfants de moins de 5 ans et constitue un problème de santé publique majeur. Le projet vise à exploiter le fort taux de pénétration de la téléphonie (plus de 92% de ménages en milieu urbain possèdent un téléphone) pour informer (alerter) les parents des dates de vaccination de leurs enfants par SMS et éventuellement par messages vocaux automatiques (OBD, IVR). Le projet mettra également sur pieds un système de calendrier vaccinal électronique.

Un serveur enregistrera quelques informations lorsqu’une femme vient accoucher (nom de l’enfant, date et lieu d’accouchement, noms des parents et numéros de téléphones, centre de santé, etc.) et enverra ensuite automatiquement des SMS (et éventuellement messages vocaux) de rappel aux parents, centres de santé et agents de santé, une semaine et un jour avant un événement, et éventuellement 3 jours et une semaine après l’événement, si les parents ne se sont pas présentés. Il sera aussi possible pour les parents d’envoyer des SMS gratuits au système afin de donner leur feedback sur les services dont ils bénéficient, des sondages par SMS seront aussi organisés pendant toute la durée du projet.

Toutes les informations et données enregistrées pendant la durée du projet seront consultables en temps réel sur un site web internet, ce qui pourra permettre au gouvernement et partenaires à mieux orienter leurs politiques. L’architecture technique du serveur sera conçue autour des technologies suivantes: la librairie RapidSMS, un OS GNU/Linux (Debian ou Ubuntu Server), Python et les Packages requis, le Framework Web Django, un Gateway SMS (Kannel?), MySQL comme base de données et un serveur web.

Présentation PowerPoint de l’étude de faisabilité du projet et des recommandations

Sep 27

6 things you should know about Amazon CloudSearch

Amazon CloudSearch is a fully-managed service in the AWS Cloud that makes it simple and cost-effective to set up, manage, and scale a custom search solution for your website or application.

Amazon CloudSearch supports a rich set of features including language-specific text processing for 34 languages, free text search, faceted search, geospatial search, customizable relevance ranking, highlighting, autocomplete and user configurable scaling and availability options [1].

Here are six key things you should know about Amazon CloudSearch.

1. How to build a search solution with Amazon CloudSearch?

There are three main steps:
Amazon-CloudSearch

  1. Create and configure a search domain: A search domain includes your searchable data and the search instances that handle your search requests. If you have multiple collections of data that you want to make searchable, you can create multiple search domains
  2. Upload the data you want to search to your domain: Amazon CloudSearch indexes your data and deploys the search index to one or more search instances,
  3. Search your domain: You send a search request to your domain’s search endpoint as an HTTP/HTTPS GET request.

2. How much data can you store in CloudSearch?

The number of partitions you need depends on your data and configuration. When you upload data, Amazon CloudSearch deploys one or more search instances. As your data volume grows, more search instances or larger search instances are deployed to contain your indexed data. The maximum number of search instances that can be deployed for a domain is 50, and a search index can be splited across a maximum of 10 partitions. For more information about CloudSearch limits, see Ref. [2].

3. How are you charged?

You are charged for Search instances, Document batch uploads, IndexDocuments requests and Data transfer. The Ref. [3] provides a useful resource to estimate your CloudSearch monthly bill.

4. Can you query for documents older than X and then send a batch delete request?

At the moment, Amazon CloudSearch does not provide this feature, but you can use one of their SDKs to list documents based on some parameters, and then delete them.

5. Do you need to keep a copy of the index anywhere?

You don’t really need, you can rely only on CloudSearch as source of the analytics data, as far as you respect AWS Security Best practices [5]. Amazon CloudSearch stores your data internally in high availability stores, you will not have to re-index your data, should a CloudSearch instance have an issue (this will be handled transparently). It offers key benefits including automatic node monitoring and recovery, built in data durability, easy setup and configuration and hands off auto scaling.

6. What’s the difference in the different instance sizes for CloudSearch?

A search instance is a single search engine that indexes documents and responds to search requests. It has a finite amount of RAM and CPU resources for indexing data and processing requests. There are four available instance types within CloudSearch: search.m1.small (2 Million documents), search.m1.large (8 Million documents), search.m2.xlarge (16 Million documents) and search.m2.2xlarge (32 Million documents).

Short Amazon CloudSearch Video

References:
[1] – Amazon CloudSearch
[2] – Understanding Amazon CloudSearch Limits
[3] – AWS Simple Monthly Calculator
[4] – AWS Security Best practices [pdf]