linux - Sudden receive-buffer buildup on CentOS for C++ application -

i have strange problem when receiving udp data on centos. when application receives data fine @ first , packets received expected, of sudden kernel receive buffer (net.core.rmem) starts fill no apparent reason until it's full , packets dropped. strange part buffer more or less empty until of sudden when starts increase dramatically though sending party sends @ same rate before. if resource haven't accounted depleted, or operating system changes priority of thread dedicated receive operations. data still received , read receive(), buffer starts fill fast.

files sent sending application receiving application. there unidirectional gateway between sending , receiving application, there no possibility congestion control (or tcp) ever. problem arises when send big file (around 3 gib or more). works fine when send multiple small files, if sum of size larger 3 gib.

at moment i'm unable specify problem further , i'm pretty stunned wrong. information systems configuration can imagine being relevant, i've looked memory leaks, disk usage , buffer sizes without being able find specific.

• data sent @ rate of 100 mbit/s.
• mtu 9000 on both sending , receiving machine.
• net.core.rmem_max/net.core.rmem_default set 536870912 bytes (huge).
• net.core.netdev_max_backlog set 65536 (huge).
• each udp packet sent 8192 bytes, excluding udp header.
• a temporary file created through tmpfile() used store data each file.
• the temporary file closed file completed (hashsum verified).
• cpu usage when receiving files consistent @ 100%.
• memory usage when receiving files consistent @ 0.5%.

---

receive()

std::vector<uint8_t>* vector = new std::vector<uint8_t>();  while (signal == 0) {   ret = _serverio->receive(*vector);   if (ret == -1 || ret == 0)   {     continue;   }   else   {     produce(vector);     vector = new std::vector<uint8_t>();   } } 

_serverio->receive(std::vector& data)

ssize_t n; data.resize(udpmaxmsg); int res = m_fdwait.wait(m_timeoutms); if(res < 1) {   data.resize(0);   return res; // timeout or interrupt }  n = read(m_servfd, &(data[0]), data.size()); if(n < 0) {   if(errno == eintr) {     data.resize(0);     return -1;   }   else {     throw socket_error(errno, "udpserver::receive");   } } data.resize(n); return n; 

produce(std::vector* vector)

_producersemaphore.aquire(); _queue.lock(); _buffer.push_back(vector); _queue.unlock(); _consumersemaphore.release(); 

consume()

bool aquired = false; while (!aquired) {   if (_terminated)   {     // consume should return null if     // receiver has been terminated.     return null;   }    aquired = _consumersemaphore.aquire_timeout(1); }  std::vector<uint8_t>* vector = null; _queue.lock(); vector = _buffer.front(); _buffer.pop_front(); _queue.unlock(); _producersemaphore.release();  return vector; 

recv_buffer.sh (for monitoring of receive buffer)

while true ;   _buffer_value=$(printf "%d" "0x$(grep 2710 /proc/net/udp \     | awk '{print $5}' | sed 's/.*://')")   _delta=$(expr $_buffer_value - $_pre_buffer)   _pre_buffer=$_buffer_value   echo "$_buffer_value : $_delta"   sleep 0.1 done 

---

recv_buffer.sh output

buffer-size delta 0 0 0 0 ... 10792 10792 10792 0 0 -10792 10792 10792 0 -10792 0 0 0 0 0 0 // type of pattern goes on 2.5 gib ... 0 0 0 0 0 0 0 0 971280 971280 // @ point buffer starts fill 1823848 852568 1931768 107920 2039688 107920 2179984 140296 2287904 107920 2406616 118712 2525328 118712 2644040 118712 2741168 97128 2881464 140296 3010968 129504 3140472 129504 ... 533567272 647520 536038640 2471368 536675368 636728 536880416 205048 // @ point packets dropped 536869624 -10792 536880416 10792 536880416 0 536869624 -10792 536880416 10792 536869624 -10792 536880416 10792 536880416 0 536880416 0 536880416 0 536880416 0 536880416 0