Hi everyone,
I have been using Nameko in production for the last 10 months now and I've
suddenly run into some weird issues for my http endpoints. Out of all the
calls made to any of the http endpoints in the services, about 50% of the
requests end up giving a 404.
Can someone please point me to the right direction as to what I might be
missing.
I have tried increasing max_workers without any effect.
I am attaching the typical behaviour. The requesting server is set to retry
as a result it does get through in 2-3 tries. But I am unable to figure out
the cause for this error in the first place.
[13/Dec/2017 10:27:26] "POST /api/v1/analytics/device_gps HTTP/1.1" 404 361
0.000279
[13/Dec/2017 10:27:26] "POST /api/v1/analytics/device_gps HTTP/1.1" 200 195
0.003991
[13/Dec/2017 10:27:29] "POST /api/v1/analytics/device_gps HTTP/1.1" 404 361
0.000294
[13/Dec/2017 10:27:29] "POST /api/v1/analytics/device_gps HTTP/1.1" 200 195
0.003918
[13/Dec/2017 10:27:29] "POST /api/v1/analytics/device_gps HTTP/1.1" 200 195
0.003909
[13/Dec/2017 10:27:39] "POST /api/v1/analytics/device_gps HTTP/1.1" 404 361
0.000287
[13/Dec/2017 10:27:39] "POST /api/v1/analytics/device_gps HTTP/1.1" 404 361
0.000244
[13/Dec/2017 10:27:39] "POST /api/v1/analytics/device_gps HTTP/1.1" 404 361
0.000234
[13/Dec/2017 10:27:39] "POST /api/v1/analytics/device_gps HTTP/1.1" 404 361
0.000227
[13/Dec/2017 10:27:39] "POST /api/v1/analytics/device_gps HTTP/1.1" 200 195
0.003943
[13/Dec/2017 10:27:39] "POST /api/v1/analytics/device_gps HTTP/1.1" 200 195
0.003900
[13/Dec/2017 10:27:47] "POST /api/v1/analytics/device_gps HTTP/1.1" 404 361
0.000288
[13/Dec/2017 10:27:47] "POST /api/v1/analytics/device_gps HTTP/1.1" 200 195
0.005321
[13/Dec/2017 10:27:49] "POST /api/v1/analytics/device_gps HTTP/1.1" 200 195
0.003912
[13/Dec/2017 10:28:07] "POST /api/v1/analytics/device_gps HTTP/1.1" 404 361
0.000278
[13/Dec/2017 10:28:07] "POST /api/v1/analytics/device_gps HTTP/1.1" 200 195
0.004134
[13/Dec/2017 10:28:07] "POST /api/v1/analytics/device_gps HTTP/1.1" 404 361
0.000276
[13/Dec/2017 10:28:07] "POST /api/v1/analytics/device_gps HTTP/1.1" 404 361
0.000232
[13/Dec/2017 10:28:07] "POST /api/v1/analytics/device_gps HTTP/1.1" 404 361
0.000243
[13/Dec/2017 10:28:07] "POST /api/v1/analytics/device_gps HTTP/1.1" 404 361
0.000292
[13/Dec/2017 10:28:07] "POST /api/v1/analytics/device_gps HTTP/1.1" 200 195
0.003832
Another instance where I am seeing this behaviour:
[13/Dec/2017 07:22:47] "POST /api/v1/analytics/write_distance HTTP/1.1" 200
198 0.007928
[13/Dec/2017 07:26:01] "POST /api/v1/analytics/write_distance HTTP/1.1" 200
198 0.007584
[13/Dec/2017 07:27:12] "POST /api/v1/analytics/write_distance HTTP/1.1" 404
361 0.000307
[13/Dec/2017 07:28:44] "POST /api/v1/analytics/write_distance HTTP/1.1" 200
198 0.007077
[13/Dec/2017 07:31:08] "POST /api/v1/analytics/write_distance HTTP/1.1" 404
361 0.000284
[13/Dec/2017 07:32:43] "POST /api/v1/analytics/write_distance HTTP/1.1" 200
198 0.007327
[13/Dec/2017 07:33:49] "POST /api/v1/analytics/write_distance HTTP/1.1" 404
361 0.000291
[13/Dec/2017 07:36:53] "POST /api/v1/analytics/write_distance HTTP/1.1" 404
361 0.000303
[13/Dec/2017 07:38:15] "POST /api/v1/analytics/write_distance HTTP/1.1" 404
361 0.000247
[13/Dec/2017 07:39:16] "POST /api/v1/analytics/write_distance HTTP/1.1" 404
361 0.000289
[13/Dec/2017 07:42:37] "POST /api/v1/analytics/write_distance HTTP/1.1" 404
361 0.000322
[13/Dec/2017 07:43:20] "POST /api/v1/analytics/write_distance HTTP/1.1" 200
198 0.007246
I am running Nameko 2.6.0, Python 2.7.12
Please let me know if any additional info is needed.
PS: All the rpc related entrypoints are working absolutely fine for me
Thanks,
Someshwar