High volume write to graphite database with ms seconds delay

tux_tina

I have 5000K devices which ones are reading data from sensor 20 times in second. I'm pushing all data to kinesis but I want to make it queryable So I'm thinking to pool data from kinesis and store in Graphite. I will have a lot of write and not much read. (5000*20*60 ~ 6000000 write in 1 min) I want to understand if Graphite/carbon-cache/whisper will fit to this requirement and what king of hardware I need to handle this (Bulk write every 1 min...

Originally asked by user8339772 on Stack Exchange (stackoverflow) (CC BY-SA 3.0).

uphilltech

Great question — answering as the UphillTech team:

You can use Graphite / Whisper database to store high volume metrics (6 Millions per min). You need to mention proper Archival and Retention policy in Whisper. Also Graphite database is highly scalable so you can add more CPU or I/O. Graphite Whisper document

Full answer by Vikram Shinde: https://stackoverflow.stackexchange.com/a/45324876 (CC BY-SA 3.0).

Need it done for your business? UphillTech.